shrine 2.10.1 → 2.11.0

data/doc/design.md

@@ -206,4 +206,4 @@ automatically:
 * deletes the uploaded file if attachment was replaced/removed or the record
   destroyed
 
-[Using Attacher]: http://shrinerb.com/rdoc/files/doc/attacher_md.html
+[Using Attacher]: https://shrinerb.com/rdoc/files/doc/attacher_md.html

data/doc/direct_s3.md

@@ -23,11 +23,12 @@ storage service is beneficial for several reasons:
   request-response lifecycle might not be able to finish before the request
   times out.
 
-You can start by setting both temporary and permanent storage to S3 with
-different prefixes (or even different buckets):
+To start, let's set both temporary and permanent storage to S3, with the
+temporary storage uploading to the `cache/` directory:
 
 ```rb
 # Gemfile
+gem "shrine", "~> 2.11"
 gem "aws-sdk-s3", "~> 1.2"
 ```
 ```rb
@@ -42,7 +43,7 @@ s3_options = {
 
 Shrine.storages = {
   cache: Shrine::Storage::S3.new(prefix: "cache", **s3_options),
-  store: Shrine::Storage::S3.new(prefix: "store", **s3_options),
+  store: Shrine::Storage::S3.new(**s3_options),
 }
 ```
 
@@ -69,7 +70,7 @@ client.put_bucket_cors(
   cors_configuration: {
     cors_rules: [{
       allowed_headers: ["Authorization", "Content-Type", "Origin"],
-      allowed_methods: ["GET", "POST"],
+      allowed_methods: ["GET", "POST", "PUT"],
       allowed_origins: ["*"],
       max_age_seconds: 3000,
     }]
@@ -80,27 +81,6 @@ client.put_bucket_cors(
 Note that due to DNS propagation it may take some time for the CORS update to
 be applied.
 
-## File hash
-
-After direct S3 uploads we'll need to manually construct Shrine's JSON
-representation of an uploaded file:
-
-```rb
-{
-  "id": "349234854924394", # requied
-  "storage": "cache", # required
-  "metadata": {
-    "size": 45461, # optional, but recommended
-    "filename": "foo.jpg", # optional
-    "mime_type": "image/jpeg" # optional
-  }
-}
-```
-
-* `id` – location of the file on S3 (minus the `:prefix`)
-* `storage` – direct uploads typically use the `:cache` storage
-* `metadata` – hash of metadata extracted from the file
-
 ## Strategy A (dynamic)
 
 * Best user experience
@@ -113,7 +93,7 @@ upload the file to S3. The `presign_endpoint` plugin gives us this presign
 route, so we just need to mount it in our application:
 
 ```rb
-Shrine.plugin :presign_endpoint
+Shrine.plugin :presign_endpoint, presign_options: { method: :put }
 ```
 ```rb
 # config.ru (Rack)
@@ -129,37 +109,31 @@ Rails.application.routes.draw do
 end
 ```
 
-The above will create a `GET /presign` route, which returns the S3 URL which
-the file should be uploaded to, along with the required POST parameters and
-request headers.
+The above will create a `GET /presign` route, which internally calls
+[`Shrine::Storage::S3#presign`], returning the HTTP verb (PUT) and the S3 URL
+to which the file should be uploaded, along with the required parameters (will
+only be present for POST presigns) and request headers.
 
 ```rb
 # GET /presign
 {
-  "url": "https://my-bucket.s3-eu-west-1.amazonaws.com",
-  "fields": {
-    "key": "cache/b7d575850ba61b44c8a9ff889dfdb14d88cdc25f8dd121004c8",
-    "policy": "eyJleHBpcmF0aW9uIjoiMjAxNS0QwMToxMToyOVoiLCJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJzaHJpbmUtdGVzdGluZyJ9LHsia2V5IjoiYjdkNTc1ODUwYmE2MWI0NGU3Y2M4YTliZmY4OGU5ZGZkYjE2NTQ0ZDk4OGNkYzI1ZjhkZDEyMTAwNGM4In0seyJ4LWFtei1jcmVkZW50aWFsIjoiQUtJQUlKRjU1VE1aWlk0NVVUNlEvMjAxNTEwMjQvZXUtd2VzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LHsieC1hbXotYWxnb3JpdGhtIjoiQVdTNC1ITUFDLVNIQTI1NiJ9LHsieC1hbXotZGF0ZSI6IjIwMTUxMDI0VDAwMTEyOVoifV19",
-    "x-amz-credential": "AKIAIJF55TMZYT6Q/20151024/eu-west-1/s3/aws4_request",
-    "x-amz-algorithm": "AWS4-HMAC-SHA256",
-    "x-amz-date": "20151024T001129Z",
-    "x-amz-signature": "c1eb634f83f96b69bd675f535b3ff15ae184b102fcba51e4db5f4959b4ae26f4"
-  },
+  "method": "put",
+  "url": "https://my-bucket.s3.eu-central-1.amazonaws.com/cache/my-key?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMDH2HTSB3RKB4WQ%2F20180424%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20180424T212022Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=1036b9cefe52f0b46c1f257f6817fc3c55cd8d9004f87a38cf86177762359375",
+  "fields": {},
   "headers": {}
 }
 ```
 
-On the client side you can then make a request to the presign endpoint as soon
-as the user selects a file, and use the returned request information to upload
+On the client side you can make it so that, when the user selects a file,
+upload parameters are fetched from presign endpoint, and are used to upload
 the selected file directly to S3. It's recommended to use [Uppy] for this.
 
 Once the file has been uploaded, you can generate a JSON representation of the
-uploaded file on the client-side, and write it to the hidden attachment field.
-The `id` field needs to be equal to the `key` presign field minus the storage
-`:prefix`.
+uploaded file on the client-side, and write it to the hidden attachment field
+(or send it directly in an AJAX request).
 
-```html
-<input type='hidden' name='photo[image]' value='{
+```rb
+{
   "id": "302858ldg9agjad7f3ls.jpg",
   "storage": "cache",
   "metadata": {
@@ -167,12 +141,18 @@ The `id` field needs to be equal to the `key` presign field minus the storage
     "filename": "nature.jpg",
     "mime_type": "image/jpeg",
   }
-}'>
+}
 ```
 
-This JSON string will now be submitted and assigned to the attachment attribute
-instead of the raw file. See the [demo app] for an example JavaScript
-implementation of multiple direct S3 uploads.
+* `id` – location of the file on S3 (minus the `:prefix`)
+* `storage` – direct uploads typically use the `:cache` storage
+* `metadata` – hash of metadata extracted from the file
+
+Once submitted this JSON will then be assigned to the attachment attribute
+instead of the raw file. See [this walkthrough][direct S3 upload walkthrough]
+for adding dynamic direct S3 uploads from scratch using [Uppy], as well as the
+[Roda][roda demo] or [Rails][rails demo] demo app for a complete example of
+multiple direct S3 uploads.
 
 ## Strategy B (static)
 
@@ -182,22 +162,22 @@ implementation of multiple direct S3 uploads.
 
 An alternative to the previous strategy is to generate an S3 upload form on
 page render. The user can then select a file and submit it directly to S3. For
-generating the form we can use `Shrine::Storage::S3#presign`, which returns a
-[`Aws::S3::PresignedPost`] object with `#url` and `#fields` attributes:
-
-```erb
-<%
-  presign = Shrine.storages[:cache].presign SecureRandom.hex,
-                                            success_action_redirect: new_album_url
-%>
-
-<form action="<%= presign.url %>" method="post" enctype="multipart/form-data">
-  <% presign.fields.each do |name, value| %>
-    <input type="hidden" name="<%= name %>" value="<%= value %>">
-  <% end %>
-  <input type="file" name="file">
-  <input type="submit" value="Upload">
-</form>
+generating the form can use [`Shrine::Storage::S3#presign`], which returns URL
+and form fields that should be used for the upload.
+
+```rb
+presigned_data = Shrine.storages[:cache].presign(
+  SecureRandom.hex,
+  success_action_redirect: new_album_url
+)
+
+Forme.form(action: presigned_data[:url], method: "post", enctype: "multipart/form-data") do |f|
+  presigned_data[:fields].each do |name, value|
+    f.input :hidden, name: name, value: value
+  end
+  f.input :file, name: "file"
+  f.input :submit, value: "Upload"
+end
 ```
 
 Note the additional `:success_action_redirect` option which tells S3 where to
@@ -206,30 +186,30 @@ builder to generate this form, you might need to also tell S3 to ignore the
 additional `utf8` and `authenticity_token` fields that Rails generates:
 
 ```rb
-<%
-  presign = Shrine.storages[:cache].presign SecureRandom.hex,
-                                            allow_any: ["utf8", "authenticity_token"],
-                                            success_action_redirect: new_album_url
-%>
+presigned_data = Shrine.storages[:cache].presign(
+  SecureRandom.hex,
+  allow_any: ["utf8", "authenticity_token"],
+  success_action_redirect: new_album_url
+)
+
+# ...
 ```
 
 Let's assume we specified the redirect URL to be a page which renders the form
 for a new record. S3 will include some information about the upload in form of
 GET parameters in the URL, out of which we only need the `key` parameter:
 
-```erb
-<%
-  cached_file = {
-    storage: "cache",
-    id: params[:key][/cache\/(.+)/, 1], # we subtract the storage prefix
-    metadata: {},
-  }
-%>
+```rb
+cached_file = {
+  storage: "cache",
+  id: request.params[:key][/cache\/(.+)/, 1], # we subtract the storage prefix
+  metadata: {},
+}
 
-<form action="/albums" method="post">
-  <input type="hidden" name="album[image]" value="<%= cached_file.to_json %>">
-  <input type="submit" value="Save">
-</form>
+Forme.form(@album, action: "/albums", method: "post") do |f|
+  f.input :image, type: :hidden, value: cached_file.to_json
+  f.button "Save"
+end
 ```
 
 ## Object data
@@ -278,15 +258,35 @@ following trick:
 ```rb
 class MyUploader < Shrine
   plugin :processing
+  plugin :refresh_metadata
 
   process(:store) do |io, context|
-    real_metadata = io.open { |opened_io| extract_metadata(opened_io, context) }
-    io.metadata.update(real_metadata)
+    io.refresh_metadata!
     io # return the same cached IO
   end
 end
 ```
 
+## Checksum
+
+To have AWS S3 verify the integrity of the uploaded data, you can use a
+checksum. For that you first need to tell AWS S3 that you're going to be
+including the `Content-MD5` request header in the upload request, by adding
+the `:content_md5` presign option.
+
+```rb
+Shrine.plugin :presign_endpoint, presign_options: -> (request) do
+  {
+    content_md5: request.params["checksum"],
+    method: :put,
+  }
+end
+```
+
+With the above setup, you can pass the MD5 hash of the file via the `checksum`
+query parameter in the request to the presign endpoint. See [this
+walkthrough][checksum walkthrough] for a complete JavaScript solution.
+
 ## Clearing cache
 
 Directly uploaded files won't automatically be deleted from your temporary
@@ -353,11 +353,24 @@ Shrine::Attacher.promote do |data|
 end
 ```
 
+## Testing
+
+To avoid network requests in your test and development environment, you can use
+[Minio]. Minio is an open source object storage server with AWS S3 compatible
+API which you can run locally. See how to set it up in the [Testing][minio
+setup] guide.
+
+[`Shrine::Storage::S3#presign`]: https://shrinerb.com/rdoc/classes/Shrine/Storage/S3.html#method-i-presign
 [`Aws::S3::PresignedPost`]: http://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Bucket.html#presigned_post-instance_method
-[demo app]: https://github.com/shrinerb/shrine/tree/master/demo
+[direct S3 upload walkthrough]: https://gist.github.com/janko-m/9aea154d72eb85b1fbfa16e1d77946e5#adding-direct-s3-uploads-to-a-roda--sequel-app-with-shrine
+[checksum walkthrough]: https://gist.github.com/janko-m/4470b5fb0737c5c1f8bcfe8cdc3fd296#using-checksums-to-verify-integrity-of-direct-uploads-with-shrine--uppy
+[roda demo]: https://github.com/shrinerb/shrine/tree/master/demo
+[rails demo]: https://github.com/erikdahlstrand/shrine-rails-example
 [Uppy]: https://uppy.io
 [Amazon S3 Data Consistency Model]: http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyMode
 [CORS guide]: http://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html
 [CORS API]: https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#put_bucket_cors-instance_method
 [lifecycle Console]: http://docs.aws.amazon.com/AmazonS3/latest/UG/lifecycle-configuration-bucket-no-versioning.html
 [lifecycle API]: https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#put_bucket_lifecycle_configuration-instance_method
+[Minio]: https://minio.io
+[minio setup]: https://shrinerb.com/rdoc/files/doc/testing_md.html#label-Minio

data/doc/metadata.md

@@ -0,0 +1,213 @@
+# Extracting Metadata
+
+Before a file is uploaded, Shrine automatically extracts metadata from it, and
+stores them in the `Shrine::UploadedFile` object. By default it extracts
+`size`, `filename` and `mime_type`.
+
+```rb
+uploaded_file = uploader.upload(file)
+uploaded_file.metadata #=>
+# {
+#   "size" => 345993,
+#   "filename" => "matrix.mp4",
+#   "mime_type" => "video/mp4",
+# }
+```
+
+You can also use `Shrine#extract_metadata` directly to extract metadata from
+any IO object.
+
+```rb
+uploader.extract_metadata(io) #=>
+# {
+#   "size" => 345993,
+#   "filename" => "matrix.mp4",
+#   "mime_type" => "video/mp4",
+# }
+```
+
+## MIME type
+
+By default, the `mime_type` metadata will be copied over from the
+`#content_type` attribute of the input file, if present. However, since
+`#content_type` value comes from the `Content-Type` header of the upload
+request, it's *not guaranteed* to hold the actual MIME type of the file (browser
+determines this header based on file extension). Moreover, only
+`ActionDispatch::Http::UploadedFile` and `Shrine::Plugins::RackFile::UploadedFile`
+objects have `#content_type` defined, so when uploading simple file objects
+`mime_type` will be nil. That makes relying on `#content_type` both a security
+risk and limiting.
+
+To remedy that, Shrine comes with a `determine_mime_type` plugin which is able
+to extract the MIME type from IO *content*. When you load it, the `mime_type`
+plugin will now be determined using the UNIX [`file`] command.
+
+```rb
+Shrine.plugin :determine_mime_type
+```
+```rb
+uploaded_file = uploader.upload StringIO.new("<?php ... ?>")
+uploaded_file.mime_type #=> "text/x-php"
+```
+
+The `file` command won't correctly determine the MIME type in all cases, that's
+why the `determine_mime_type` plugin comes with different MIME type analyzers.
+So, instead of the `file` command you can use gems like [MimeMagic] or
+[Marcel], as well as mix-and-match the analyzers to suit your needs. See the
+plugin documentation for more details.
+
+## Image Dimensions
+
+Shrine comes with a `store_dimensions` plugin for extracting image dimensions.
+It adds `width` and `height` metadata values, and also adds `#width`,
+`#height`, and `#dimensions` methods to the `Shrine::UploadedFile` object. By
+default, the plugin uses [FastImage] to analyze dimensions, but you can also
+have it use [MiniMagick] or [ruby-vips]:
+
+```rb
+Shrine.plugin :store_dimensions, analyzer: :mini_magick
+```
+```rb
+uploaded_file = uploader.upload(image)
+uploaded_file.metadata["width"]  #=> 1600
+uploaded_file.metadata["height"] #=> 900
+
+# convenience methods
+uploaded_file.width      #=> 1600
+uploaded_file.height     #=> 900
+uploaded_file.dimensions #=> [1600, 900]
+```
+
+## Custom metadata
+
+In addition to the built-in metadata, Shrine allows you to extract and store
+any custom metadata, using the `add_metadata` plugin (which extends
+`Shrine#extract_metadata`). For example, you might want to extract EXIF data
+from images:
+
+```rb
+require "mini_magick"
+
+class ImageUploader < Shrine
+  plugin :add_metadata
+
+  add_metadata :exif do |io|
+    Shrine.with_file(io) do |file|
+      begin
+        MiniMagick::Image.new(file.path).exif
+      rescue MiniMagick::Error
+        # not a valid image
+      end
+    end
+  end
+end
+```
+```rb
+uploaded_file = uploader.upload(image)
+uploaded_file.metadata["exif"] #=> {...}
+uploaded_file.exif             #=> {...}
+```
+
+Of, if you're uploading videos, you might want to extract some video-specific
+meatadata:
+
+```rb
+require "streamio-ffmpeg"
+
+class VideoUploader < Shrine
+  plugin :add_metadata
+
+  add_metadata do |io, context|
+    movie = Shrine.with_file(io) { |file| FFMPEG::Movie.new(file.path) }
+
+    { "duration"   => movie.duration,
+      "bitrate"    => movie.bitrate,
+      "resolution" => movie.resolution,
+      "frame_rate" => movie.frame_rate }
+  end
+end
+```
+```rb
+uploaded_file = uploader.upload(video)
+uploaded_file.metadata #=>
+# {
+#   ...
+#   "duration" => 7.5,
+#   "bitrate" => 481,
+#   "resolution" => "640x480",
+#   "frame_rate" => 16.72
+# }
+```
+
+The yielded `io` object will not always be an object that responds to `#path`.
+If you're using the `data_uri` plugin, the `io` will be a `StringIO` wrapper.
+When the `restore_cached_data` plugin is loaded, any assigned cached file will
+get their metadata extracted, and `io` will be a `Shrine::UploadedFile` object.
+If you're using a metadata analyzer that requires the source file to be on
+disk, you can use `Shrine.with_file` to ensure you have a file object.
+
+Also, be aware that metadata is extracted before file validation, so you'll
+need to handle the cases where the file is not of expected type.
+
+## Metadata columns
+
+If you want to write any of the metadata values into a separate database column
+on the record, you can use the `metadata_attributes` plugin.
+
+```rb
+Shrine.plugin :metadata_attributes, :mime_type => :type
+```
+```rb
+photo = Photo.new(image: file)
+photo.image_type #=> "image/jpeg"
+```
+
+## Refreshing metadata
+
+When uploading directly to the cloud, the metadata of the original file by
+default won't get extracted on the server side, because your application never
+received the file content.
+
+To have Shrine extra metadata when a cached file is assigned to the attachment
+attribute, it's recommended to load the `restore_cached_data` plugin.
+
+```rb
+Shrine.plugin :restore_cached_data # extract metadata from cached files on assingment
+```
+```rb
+photo.image = '{"id":"ks9elsd.jpg","storage":"cache","metadata":{}}' # metadata is extracted
+photo.image.metadata #=>
+# {
+#   "size" => 4593484,
+#   "filename" => "nature.jpg",
+#   "mime_type" => "image/jpeg"
+# }
+```
+
+Extracting metadata from a cached file requires retrieving file content from
+the storage, which might not be desirable depending on your case, that's why
+`restore_cached_data` plugin is not loaded by default. However, Shrine will not
+download the whole file from the storage, instead, it will open a connection to
+the storage, and the metadata analyzers will download how much of the file they
+need. Most MIME type analyzers and the FastImage dimensions analyzer need only
+the first few kilobytes.
+
+You can also extract metadata from an uploaded file explicitly using the
+`refresh_metadata` plugin (which the `restore_cached_data` plugin uses
+internally).
+
+```rb
+Shrine.plugin :refresh_metadata
+```
+```rb
+uploaded_file.metadata #=> {}
+uploaded_file.refresh_metadata!
+uploaded_file.metadata #=> {"filename"=>"nature.jpg","size"=>532894,"mime_type"=>"image/jpeg"}
+```
+
+[`file`]: http://linux.die.net/man/1/file
+[MimeMagic]: https://github.com/minad/mimemagic
+[Marcel]: https://github.com/basecamp/marcel
+[FastImage]: https://github.com/sdsykes/fastimage
+[MiniMagick]: https://github.com/minimagick/minimagick
+[ruby-vips]: https://github.com/jcupitt/ruby-vips