shrine 2.12.0 → 2.13.0

data/doc/attacher.md

@@ -66,16 +66,32 @@ cached file in form of a JSON string, and assigns the cached result to record's
 attacher.assign(io)
 
 # writes the given cached file to the data column
-attacher.assign '{
-  "id": "9260ea09d8effd.jpg",
-  "storage": "cache",
-  "metadata": { ... }
-}'
+attacher.assign('{"id":"9260ea09d8effd.jpg","storage":"cache","metadata":{ ... }}')
+```
+
+When assigning an IO object, any additional options passed to `#assign` will be
+forwarded to `Shrine#upload`. This allows you to do things like overriding
+metadata, setting upload location, or passing upload options:
+
+```rb
+attacher.assign io,
+  metadata:       { "filename" => "myfile.txt" },
+  location:       "custom/location",
+  upload_options: { acl: "public-read" }
+```
+
+If you're attaching a cached file and want to override its metadata before
+assignment, you can do it like so:
+
+```rb
+cached_file = Shrine.uploaded_file('{"id":"9260ea09d8effd.jpg","storage":"cache","metadata":{ ... }}')
+cached_file.metadata["filename"] = "myfile.txt"
+
+attacher.assign(cached_file.to_json)
 ```
 
 For security reasons `#assign` doesn't accept files uploaded to permanent
-storage, but you can also use `#set` to attach any `Shrine::UploadedFile`
-object.
+storage, but you can use `#set` to attach any `Shrine::UploadedFile` object.
 
 ```rb
 uploaded_file #=> #<Shrine::UploadedFile>
@@ -160,9 +176,20 @@ The `:action` parameter is optional; it can be used for triggering a certain
 processing block, and it is also automatically printed by the `logging` plugin
 to aid in debugging.
 
-Internally this calls `#swap`, which will update the record with any uploaded
-file, but will reload the record to check if the current attachment hasn't
-changed (if the `backgrounding` plugin is loaded).
+As a matter of fact, all additional options passed to `#promote` will be
+forwarded to `Shrine#upload`. So unless you're generating versions, you can do
+things like override metadata, set upload location, or pass upload options:
+
+```rb
+attacher.promote cached_file,
+  metadata:       { "filename" => "myfile.txt" },
+  location:       "custom/location",
+  upload_options: { acl: "public-read" }
+```
+
+Internally `#promote` calls `#swap`, which will update the record with any
+uploaded file, but will reload the record to check if the current attachment
+hasn't changed (if the `backgrounding` plugin is loaded).
 
 ```rb
 attacher.swap(uploaded_file)
@@ -218,19 +245,29 @@ Normally you can upload and delete directly by using the uploader.
 ```rb
 uploader = ImageUploader.new(:store)
 uploaded_file = uploader.upload(image) # uploads the file to `:store` storage
-uploader.delete(uploaded_file)         # deletes the file uploaded to `:store`
+uploader.delete(uploaded_file)         # deletes the uploaded file from `:store`
 ```
 
-The attacher has methods for "caching", "storing" and "deleting" files, which
-delegate to these uploader methods, but also pass in the `#context`:
+But the attacher also has wrapper methods for uploading and deleting, which
+also automatically pass in the attacher `#context` (which includes `:record`
+and `:name`):
 
 ```rb
-cached_file = attacher.cache!(image) # delegates to `Shrine#upload`
-stored_file = attacher.store!(image) # delegates to `Shrine#upload`
-attacher.delete!(stored_file)        # delegates to `Shrine#delete`
+attacher.cache!(file) # uploads file to temporary storage
+# => #<Shrine::UploadedFile: @data={"storage" => "cache", ...}>
+attacher.store!(file) # uploads file to permanent storage
+# => #<Shrine::UploadedFile: @data={"storage" => "store", ...}>
+attacher.delete!(uploaded_file) # deletes uploaded file from storage
 ```
 
-The `#cache!` and `#store!` only upload the file to the storage, they don't
-write to record's data column.
+These methods only upload/delete files, they don't write to record's data
+column. You can also pass additional options for `Shrine#upload` and
+`Shrine#delete`:
+
+```rb
+attacher.cache!(file, upload_options: { acl: "public-read" })
+attacher.store!(file, location: "custom/location")
+attacher.delete!(uploaded_file, foo: "bar")
+```
 
 [file migrations]: https://shrinerb.com/rdoc/files/doc/migrating_storage_md.html

data/doc/design.md

@@ -1,5 +1,7 @@
 # The Design of Shrine
 
+*If you want an in-depth walkthrough through the Shrine codebase, see [Notes on study of shrine implementation] article by Jonathan Rochkind.*
+
 There are five main types of objects that you deal with in Shrine:
 
 * Storage
@@ -56,8 +58,9 @@ Storages are typically not used directly, but through `Shrine`.
 
 ## `Shrine`
 
-A `Shrine` object (also called an "uploader") acts as a wrapper around a
-storage. First the storage needs to be registered under a name:
+A `Shrine` object (also called an "uploader") is essentially a wrapper around
+the `#upload` storage method. First the storage needs to be registered under a
+name:
 
 ```rb
 Shrine.storages[:file_system] = Shrine::Storage::FileSystem.new("uploads")
@@ -76,12 +79,17 @@ following:
 
 * generates a unique location
 * extracts metadata
-* uploads the file
+* uploads the file (calls `Storage#upload`)
 * closes the file
 * creates a `Shrine::UploadedFile` from the data
 
-In applications it's common to create subclasses of `Shrine`, in order to allow
-having different uploading logic for different types of files.
+`Shrine` class and subclasses are also used for loading plugins that extend all
+core classes. Each `Shrine` subclass has its own subclass of each of the core
+classes (`Shrine::UploadedFile`, `Shrine::Attacher`, and `Shrine::Attachment`),
+which makes it possible to have different `Shrine` subclasses with differently
+customized attachment logic. See [Creating a New Plugin] guide and the [Plugin
+system of Sequel and Roda] article for more details on the design of Shrine's
+plugin system.
 
 ## `Shrine::UploadedFile`
 
@@ -207,3 +215,6 @@ automatically:
   destroyed
 
 [Using Attacher]: https://shrinerb.com/rdoc/files/doc/attacher_md.html
+[Notes on study of shrine implementation]: https://bibwild.wordpress.com/2018/09/12/notes-on-study-of-shrine-implementation/
+[Creating a New Plugin]: https://shrinerb.com/rdoc/files/doc/creating_plugins_md.html
+[Plugin system of Sequel and Roda]: https://twin.github.io/the-plugin-system-of-sequel-and-roda/

data/doc/direct_s3.md

@@ -24,7 +24,7 @@ storage service is beneficial for several reasons:
   times out.
 
 To start, let's set both temporary and permanent storage to S3, with the
-temporary storage uploading to the `cache/` directory:
+temporary storage uploading to the `cache/` prefix:
 
 ```rb
 # Gemfile
@@ -47,39 +47,41 @@ Shrine.storages = {
 }
 ```
 
-## Enabling CORS
-
-In order to be able upload files directly to your S3 bucket, you need enable
-CORS. You can do that from the AWS S3 Console by going to your bucket, clicking
-on the "Permissions" tab, then on "CORS Configuration", and following the
-[guide for configuring CORS][CORS guide].
-
-Alternatively you can configure CORS via an [API call][CORS API]:
-
-```rb
-require "aws-sdk-s3"
-
-client = Aws::S3::Client.new(
-  access_key_id:     "<YOUR KEY>",
-  secret_access_key: "<YOUR SECRET>",
-  region:            "<REGION>",
-)
-
-client.put_bucket_cors(
-  bucket: "<YOUR BUCKET>",
-  cors_configuration: {
-    cors_rules: [{
-      allowed_headers: ["Authorization", "Content-Type", "Origin"],
-      allowed_methods: ["GET", "POST", "PUT"],
-      allowed_origins: ["*"],
-      max_age_seconds: 3000,
-    }]
-  }
-)
+## Bucket CORS configuration
+
+In order to be able upload files directly to your S3 bucket, you'll need to
+update your bucket's CORS configuration, as public uploads are not allowed by
+default. You can do that from the AWS S3 Console by going to your bucket,
+clicking on the "Permissions" tab and then on "CORS Configuration".
+
+If you're using [Uppy], this is the recommended CORS configuration for the
+[Aws S3 plugin] that should work for both POST and PUT uploads:
+
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
+  <CORSRule>
+    <AllowedOrigin>https://my-app.com</AllowedOrigin>
+    <AllowedMethod>GET</AllowedMethod>
+    <AllowedMethod>POST</AllowedMethod>
+    <AllowedMethod>PUT</AllowedMethod>
+    <MaxAgeSeconds>3000</MaxAgeSeconds>
+    <AllowedHeader>Authorization</AllowedHeader>
+    <AllowedHeader>x-amz-date</AllowedHeader>
+    <AllowedHeader>x-amz-content-sha256</AllowedHeader>
+    <AllowedHeader>content-type</AllowedHeader>
+  </CORSRule>
+  <CORSRule>
+    <AllowedOrigin>*</AllowedOrigin>
+    <AllowedMethod>GET</AllowedMethod>
+    <MaxAgeSeconds>3000</MaxAgeSeconds>
+  </CORSRule>
+</CORSConfiguration>
 ```
 
-Note that due to DNS propagation it may take some time for the CORS update to
-be applied.
+Replace `https://my-app.com` with the URL to your app (in development you can
+set this to `*`). Once you've hit "Save", it may take some time for the
+new CORS settings to be applied.
 
 ## Strategy A (dynamic)
 
@@ -87,13 +89,28 @@ be applied.
 * Single or multiple file uploads
 * Some JavaScript needed
 
-When the user selects a file in the form, on the client-side we asynchronously
-fetch the presign information from the server, and use this information to
-upload the file to S3. The `presign_endpoint` plugin gives us this presign
-route, so we just need to mount it in our application:
+When the user selects a file in the form, on the client side we asynchronously
+fetch the upload parameters from the server, and use it to upload the file to
+S3. It's recommended to use [Uppy] for client side uploads.
+
+The `presign_endpoint` plugin provides a Rack application that generates these
+upload parameters, which we can just mount in our application. We'll make our
+presign endpoint also use the additional `type` and `filename` query parameters
+to set `Content-Type` and `Content-Disposition` for the uploaded file, as well
+as limit the upload size to 10 MB (see [`Shrine::Storage::S3#presign`] for the
+list of available options).
 
 ```rb
-Shrine.plugin :presign_endpoint, presign_options: { method: :put }
+Shrine.plugin :presign_endpoint, presign_options: -> (request) {
+  filename = request.params["filename"]
+  type     = request.params["type"]
+
+  {
+    content_disposition:    "inline; filename=\"#{filename}\"", # set download filename
+    content_type:           type,                               # set content type (required if using DigitalOcean Spaces)
+    content_length_range:   0..(10*1024*1024),                  # limit upload size to 10 MB
+  }
+}
 ```
 ```rb
 # config.ru (Rack)
@@ -110,27 +127,32 @@ end
 ```
 
 The above will create a `GET /presign` route, which internally calls
-[`Shrine::Storage::S3#presign`], returning the HTTP verb (PUT) and the S3 URL
-to which the file should be uploaded, along with the required parameters (will
-only be present for POST presigns) and request headers.
+[`Shrine::Storage::S3#presign`] to return the HTTP verb (POST) and the S3 URL
+to which the file should be uploaded, along with the required POST parameters
+and request headers.
 
 ```rb
 # GET /presign
 {
-  "method": "put",
-  "url": "https://my-bucket.s3.eu-central-1.amazonaws.com/cache/my-key?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMDH2HTSB3RKB4WQ%2F20180424%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20180424T212022Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=1036b9cefe52f0b46c1f257f6817fc3c55cd8d9004f87a38cf86177762359375",
-  "fields": {},
+  "method": "post",
+  "url": "https://my-bucket.s3-eu-west-1.amazonaws.com",
+  "fields": {
+    "key": "b7d575850ba61b44c8a9ff889dfdb14d88cdc25f8dd121004c8",
+    "policy": "eyJleHBpcmF0aW9uIjoiMjAxNS0QwMToxMToyOVoiLCJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJ...",
+    "x-amz-credential": "AKIAIJF55TMZYT6Q/20151024/eu-west-1/s3/aws4_request",
+    "x-amz-algorithm": "AWS4-HMAC-SHA256",
+    "x-amz-date": "20151024T001129Z",
+    "x-amz-signature": "c1eb634f83f96b69bd675f535b3ff15ae184b102fcba51e4db5f4959b4ae26f4"
+  },
   "headers": {}
 }
 ```
 
-On the client side you can make it so that, when the user selects a file,
-upload parameters are fetched from presign endpoint, and are used to upload
-the selected file directly to S3. It's recommended to use [Uppy] for this.
-
-Once the file has been uploaded, you can generate a JSON representation of the
-uploaded file on the client-side, and write it to the hidden attachment field
-(or send it directly in an AJAX request).
+Uppy's [AWS S3][uppy aws s3] plugin would then make a request to this endpoint and use these
+parameters to upload the file directly to S3. Once the file has been uploaded,
+you can generate a JSON representation of the uploaded file on the client side,
+and write it to the hidden attachment field (or send it directly in an AJAX
+request).
 
 ```rb
 {
@@ -148,11 +170,11 @@ uploaded file on the client-side, and write it to the hidden attachment field
 * `storage` – direct uploads typically use the `:cache` storage
 * `metadata` – hash of metadata extracted from the file
 
-Once submitted this JSON will then be assigned to the attachment attribute
-instead of the raw file. See [this walkthrough][direct S3 upload walkthrough]
-for adding dynamic direct S3 uploads from scratch using [Uppy], as well as the
-[Roda][roda demo] or [Rails][rails demo] demo app for a complete example of
-multiple direct S3 uploads.
+Once the form is submitted, this JSON data will then be assigned to the
+attachment attribute instead of the raw file. See [this walkthrough][direct S3
+upload walkthrough] for adding dynamic direct S3 uploads from scratch, as well
+as the [Roda][roda demo] / [Rails][rails demo] demo app for a complete example
+of multiple direct S3 uploads.
 
 ## Strategy B (static)
 
@@ -166,13 +188,13 @@ generating the form can use [`Shrine::Storage::S3#presign`], which returns URL
 and form fields that should be used for the upload.
 
 ```rb
-presigned_data = Shrine.storages[:cache].presign(
+presign_data = Shrine.storages[:cache].presign(
   SecureRandom.hex,
   success_action_redirect: new_album_url
 )
 
-form action: presigned_data[:url], method: "post", enctype: "multipart/form-data" do |f|
-  presigned_data[:fields].each do |name, value|
+form action: presign_data[:url], method: "post", enctype: "multipart/form-data" do |f|
+  presign_data[:fields].each do |name, value|
     f.input :hidden, name: name, value: value
   end
   f.input :file, name: "file"
@@ -186,7 +208,7 @@ builder to generate this form, you might need to also tell S3 to ignore the
 additional `utf8` and `authenticity_token` fields that Rails generates:
 
 ```rb
-presigned_data = Shrine.storages[:cache].presign(
+presign_data = Shrine.storages[:cache].presign(
   SecureRandom.hex,
   allow_any: ["utf8", "authenticity_token"],
   success_action_redirect: new_album_url
@@ -202,7 +224,7 @@ GET parameters in the URL, out of which we only need the `key` parameter:
 ```rb
 cached_file = {
   storage: "cache",
-  id: request.params[:key][/cache\/(.+)/, 1], # we subtract the storage prefix
+  id: params["key"][/^cache\/(.+)/, 1], # we subtract the storage prefix
   metadata: {},
 }
 
@@ -212,6 +234,17 @@ form @album, action: "/albums" do |f|
 end
 ```
 
+## Shrine metadata
+
+When attaching a file that was uploaded directly to S3, by default Shrine will
+not extract metadata from the file, instead it will simply copy over any
+metadata assigned on the client side. This is the default behaviour because
+extracting metadata requires retrieving file content, which in this case means
+additional HTTP requests.
+
+See [this section][metadata direct uploads] or the rationale and instructions
+on how to opt in.
+
 ## Object data
 
 When the cached S3 object is copied to permanent storage, the destination S3
@@ -234,59 +267,6 @@ plugin :upload_options, store: -> (io, context) do
 end
 ```
 
-## Shrine metadata
-
-With direct uploads any metadata has to be extracted on the client-side, since
-the file upload doesn't touch the application, so the Shrine uploader doesn't
-get a chance to extract the metadata. When directly uploaded file is promoted
-to permanent storage, Shrine's default behaviour is to just copy the received
-metadata.
-
-If you want to re-extract metadata on the server before file validation, you
-can load the `restore_cached_data`. That will make Shrine open the S3 file for
-reading, pass it for metadata extraction, and then override the metadata
-received from the client with the extracted ones.
-
-```rb
-plugin :restore_cached_data
-```
-
-Note that if you don't need this metadata before file validation, and you would
-like to have it extracted in a background job, you can do that with the
-following trick:
-
-```rb
-class MyUploader < Shrine
-  plugin :processing
-  plugin :refresh_metadata
-
-  process(:store) do |io, context|
-    io.refresh_metadata!
-    io # return the same cached IO
-  end
-end
-```
-
-## Checksum
-
-To have AWS S3 verify the integrity of the uploaded data, you can use a
-checksum. For that you first need to tell AWS S3 that you're going to be
-including the `Content-MD5` request header in the upload request, by adding
-the `:content_md5` presign option.
-
-```rb
-Shrine.plugin :presign_endpoint, presign_options: -> (request) do
-  {
-    content_md5: request.params["checksum"],
-    method: :put,
-  }
-end
-```
-
-With the above setup, you can pass the MD5 hash of the file via the `checksum`
-query parameter in the request to the presign endpoint. See [this
-walkthrough][checksum walkthrough] for a complete JavaScript solution.
-
 ## Clearing cache
 
 Directly uploaded files won't automatically be deleted from your temporary
@@ -353,6 +333,42 @@ Shrine::Attacher.promote do |data|
 end
 ```
 
+## Checksums
+
+You can have AWS S3 verify the integrity of the uploaded data by including a
+checksum generated on the client side in the upload request. For that we'll
+need to include the checksum in the presign request, which we can pass in via
+the `checksum` query parameter. The `:content_md5` parameter is not supported
+in POST presigns, so for this we'll need to switch to PUT.
+
+```rb
+Shrine.plugin :presign_endpoint, presign_options: -> (request) do
+  {
+    method: :put,
+    content_md5: request.params["checksum"],
+  }
+end
+```
+
+See [this walkthrough][checksum walkthrough] for a complete JavaScript
+implementation of checksums.
+
+Note that PUT presigns don't support the `:content_length_range` option, but
+they support `:content_length` instead. So, if you want to limit the upload
+size during direct uploads, you can pass an additional `size` query parameter
+to the presign request on the client side, and require it when generating
+presign options:
+
+```rb
+Shrine.plugin :presign_endpoint, presign_options: -> (request) do
+  {
+    method: :put,
+    content_length: request.params.fetch("size"),
+    content_md5: request.params["checksum"],
+  }
+end
+```
+
 ## Testing
 
 To avoid network requests in your test and development environment, you can use
@@ -367,6 +383,8 @@ setup] guide.
 [roda demo]: https://github.com/shrinerb/shrine/tree/master/demo
 [rails demo]: https://github.com/erikdahlstrand/shrine-rails-example
 [Uppy]: https://uppy.io
+[uppy aws s3]: https://uppy.io/docs/aws-s3/
+[uppy aws-s3 cors]: https://uppy.io/docs/aws-s3/#S3-Bucket-configuration
 [Amazon S3 Data Consistency Model]: http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyMode
 [CORS guide]: http://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html
 [CORS API]: https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#put_bucket_cors-instance_method
@@ -374,3 +392,4 @@ setup] guide.
 [lifecycle API]: https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#put_bucket_lifecycle_configuration-instance_method
 [Minio]: https://minio.io
 [minio setup]: https://shrinerb.com/rdoc/files/doc/testing_md.html#label-Minio
+[metadata direct uploads]: https://github.com/shrinerb/shrine/blob/master/doc/metadata.md#direct-uploads