shrine 2.8.0 → 2.9.0

data/README.md

@@ -37,8 +37,8 @@ Shrine.plugin :rack_file # for non-Rails apps
 ```
 
 Next decide how you will name the attachment attribute on your model, and run a
-migration that adds an `<attachment>_data` text column, which Shrine will use
-to store all information about the attachment:
+migration that adds an `<attachment>_data` text or JSON column, which Shrine
+will use to store all information about the attachment:
 
 ```rb
 Sequel.migration do                           # class AddImageDataToPhotos < ActiveRecord::Migration
@@ -340,20 +340,39 @@ The model attachment attributes and callbacks just delegate the behaviour
 to a `Shrine::Attacher` object.
 
 ```rb
-attacher = ImageUploader::Attacher.new(photo, :image) # returned by `photo.image_attacher`
+photo.image_attacher #=> #<Shrine::Attacher>
+```
+
+The `Shrine::Attacher` object can be instantiated and used directly:
+
+```rb
+attacher = ImageUploader::Attacher.new(photo, :image)
 
 attacher.assign(file) # equivalent to `photo.image = file`
 attacher.get          # equivalent to `photo.image`
 attacher.url          # equivalent to `photo.image_url`
 ```
 
-The attacher is what drives attaching files to models, and it functions
+The attacher is what drives attaching files to model instances, and it functions
 independently from models' attachment interface. This means that you can use it
 as an alternative, in case you prefer not to add additional attributes to the
 model, or prefer explicitness over callbacks. It's also useful when you need
 something more advanced which isn't available through the attachment
 attributes.
 
+The `Shrine::Attacher` by default uses `:cache` for temporary and `:store` for
+permanent storage, but you can specify a different storage:
+
+```rb
+ImageUploader::Attacher.new(photo, :image, cache: :other_cache, store: :other_store)
+
+# OR
+
+photo.image_attacher(cache: :other_cache, store: :other_store)
+photo.image = file # uploads to :other_cache storage
+photo.save         # promotes to :other_store storage
+```
+
 Whenever the attacher uploads or deletes files, it sends a `context` hash
 which includes `:record`, `:name`, and `:action` keys, so that you can perform
 processing or generate location differently depending on this information. See
@@ -775,12 +794,11 @@ Rails.application.routes.draw do
 end
 ```
 
-The above created a `POST /images/upload` endpoint. You can now use a
-client-side file upload library like [FineUploader], [Dropzone] or
-[jQuery-File-Upload] to upload files asynchronously to the `/images/upload`
-endpoint the moment they are selected. Once the file has been uploaded, the
-endpoint will return JSON data of the uploaded file, which the client can then
-write to a hidden attachment field, to be submitted instead of the raw file.
+The above created a `POST /images/upload` endpoint. You can now use [Uppy] to
+upload files asynchronously to the `/images/upload` endpoint the moment they
+are selected. Once the file has been uploaded, the endpoint will return JSON
+data of the uploaded file, which the client can then write to a hidden
+attachment field, to be submitted instead of the raw file.
 
 Many popular storage services can accept file uploads directly from the client
 ([Amazon S3], [Google Cloud Storage], [Microsoft Azure Storage] etc), which
@@ -846,14 +864,49 @@ libraries are:
 
 ## Clearing cache
 
-From time to time you'll want to clean your temporary storage from old files.
-Amazon S3 provides [a built-in solution][S3 lifecycle], and for FileSystem you
-can run something like this periodically:
+Shrine doesn't automatically delete files uploaded to temporary storage, instead
+you should set up a separate recurring task that will automatically delete old
+cached files.
+
+Most of Shrine storage objects come with a `#clear!` method, which you can call
+in a recurring script. For FileSystem and S3 storage it would look like this:
 
 ```rb
+# FileSystem storage
 file_system = Shrine.storages[:cache]
 file_system.clear!(older_than: Time.now - 7*24*60*60) # delete files older than 1 week
 ```
+```rb
+# S3 storage
+s3 = Shrine.storages[:cache]
+s3.clear! { |object| object.last_modified < Time.now - 7*24*60*60 } # delete files older than 1 week
+```
+
+Note that for S3 you can also configure bucket lifecycle rules to do this for
+you. This can be done either from the [AWS Console][S3 lifecycle console] or
+via an [API call][S3 lifecycle API]:
+
+```rb
+require "aws-sdk-s3"
+
+client = Aws::S3::Client.new(
+  access_key_id:     "<YOUR KEY>",
+  secret_access_key: "<YOUR SECRET>",
+  region:            "<REGION>",
+)
+
+client.put_bucket_lifecycle_configuration(
+  bucket: "<YOUR BUCKET>",
+  lifecycle_configuration: {
+    rules: [{
+      expiration: { days: 7 },
+      filter: { prefix: "cache/" },
+      id: "cache-clear",
+      status: "Enabled"
+    }]
+  }
+)
+```
 
 ## Logging
 
@@ -903,9 +956,9 @@ generic image servers.
 Shrine has integrations for many commercial on-the-fly processing services,
 including [Cloudinary], [Imgix] and [Uploadcare].
 
-If you don't want to use a commercial service, [Attache] and [Dragonfly] are
-great open-source image servers. For Attache a Shrine integration is in
-progress, while for Dragonfly it is not needed.
+If you don't want to use a commercial service, [Dragonfly] is a great
+open-source image server. See [this blog post][processing post] on how you can
+integrate Dragonfly with Shrine.
 
 ## Chunked & Resumable uploads
 
@@ -960,9 +1013,7 @@ The gem is available as open source under the terms of the [MIT License].
 [Context]: https://github.com/janko-m/shrine#context
 [image_processing]: https://github.com/janko-m/image_processing
 [ffmpeg]: https://github.com/streamio/streamio-ffmpeg
-[FineUploader]: https://github.com/FineUploader/fine-uploader
-[Dropzone]: https://github.com/enyo/dropzone
-[jQuery-File-Upload]: https://github.com/blueimp/jQuery-File-Upload
+[Uppy]: https://uppy.io
 [Amazon S3]: https://aws.amazon.com/s3/
 [Google Cloud Storage]: https://cloud.google.com/storage/
 [Microsoft Azure Storage]: https://azure.microsoft.com/en-us/services/storage/
@@ -971,7 +1022,6 @@ The gem is available as open source under the terms of the [MIT License].
 [Cloudinary]: https://github.com/janko-m/shrine-cloudinary
 [Imgix]: https://github.com/janko-m/shrine-imgix
 [Uploadcare]: https://github.com/janko-m/shrine-uploadcare
-[Attache]: https://github.com/choonkeat/attache
 [Dragonfly]: http://markevans.github.io/dragonfly/
 [tus]: http://tus.io
 [tus-ruby-server]: https://github.com/janko-m/tus-ruby-server
@@ -985,4 +1035,6 @@ The gem is available as open source under the terms of the [MIT License].
 [roda_demo]: https://github.com/janko-m/shrine/tree/master/demo
 [rails_demo]: https://github.com/erikdahlstrand/shrine-rails-example
 [backgrounding libraries]: https://github.com/janko-m/shrine/wiki/Backgrounding-libraries
-[S3 lifecycle]: http://docs.aws.amazon.com/AmazonS3/latest/UG/lifecycle-configuration-bucket-no-versioning.html
+[S3 lifecycle Console]: http://docs.aws.amazon.com/AmazonS3/latest/UG/lifecycle-configuration-bucket-no-versioning.html
+[S3 lifecycle API]: https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#put_bucket_lifecycle_configuration-instance_method
+[processing post]: https://twin.github.io/better-file-uploads-with-shrine-processing/

data/doc/carrierwave.md

@@ -239,7 +239,7 @@ Shrine. Let's assume we have a `Photo` model with the "image" attachment. First
 we need to create the `image_data` column for Shrine:
 
 ```rb
-add_column :photos, :image_data, :text
+add_column :photos, :image_data, :text # or :json or :jsonb if supported
 ```
 
 Afterwards we need to make new uploads write to the `image_data` column. This
@@ -247,9 +247,6 @@ can be done by including the below module to all models that have CarrierWave
 attachments:
 
 ```rb
-require "fastimage"
-require "mime/types"
-
 module CarrierwaveShrineSynchronization
   def self.included(model)
     model.before_save do
@@ -272,6 +269,8 @@ module CarrierwaveShrineSynchronization
         end
       end
 
+      # Remove the `.to_json` if you're using a JSON column, otherwise the JSON
+      # object will be saved as an escaped string.
       write_attribute(:"#{name}_data", data.to_json)
     else
       write_attribute(:"#{name}_data", nil)
@@ -283,22 +282,13 @@ module CarrierwaveShrineSynchronization
   # If you'll be using `:prefix` on your Shrine storage, make sure to
   # subtract it from the path assigned as `:id`.
   def uploader_to_shrine_data(uploader)
-    path = uploader.store_path(read_attribute(uploader.mounted_as))
-
-    size   = uploader.file.size if changes.key?(uploader.mounted_as)
-    size ||= FastImage.new(uploader.url).content_length # OPTIONAL (makes an HTTP request)
-    size ||= File.size(File.join(uploader.root, path)) if File.exist?(path)
-    filename = File.basename(path)
-    mime_type = MIME::Types.type_for(path).first.to_s.presence
+    filename = read_attribute(uploader.mounted_as)
+    path     = uploader.store_path(filename)
 
     {
       storage: :store,
       id: path,
-      metadata: {
-        size: size,
-        filename: filename,
-        mime_type: mime_type,
-      },
+      metadata: { filename: filename }
     }
   end
 end
@@ -326,6 +316,19 @@ instead of CarrierWave, using equivalent Shrine storages. For help with
 translating the code from CarrierWave to Shrine, you can consult the reference
 below.
 
+You'll notice that Shrine metadata will be absent from the migrated files'
+data. You can run a script that will fill in any missing metadata defined in
+your Shrine uploader:
+
+```rb
+Shrine.plugin :refresh_metadata
+
+Photo.find_each do |photo|
+  attachment = ImageUploader.uploaded_file(photo.image, &:refresh_metadata!)
+  photo.update(image_data: attachment.to_json)
+end
+```
+
 ## CarrierWave to Shrine direct mapping
 
 ### `CarrierWave::Uploader::Base`
@@ -591,10 +594,6 @@ As mentioned before, in Shrine you register storages through `Shrine.storages`,
 and the attachment storages will automatically be `:cache` and `:store`, but
 you can change this with the `default_storage` plugin.
 
-#### `fog_*`
-
-These options are set on the [shrine-fog] storage.
-
 #### `delete_tmp_file_after_storage`, `remove_previously_stored_file_after_update`
 
 By default Shrine deletes cached and replaced files, but you can choose to keep
@@ -651,8 +650,64 @@ You can just add conditionals in processing code.
 No equivalent, it depends on your application whether you need the form to be
 multipart or not.
 
+### `CarrierWave::Storage::Fog`
+
+You can use [`Shrine::Storage::S3`] \(built-in\),
+[`Shrine::Storage::GoogleCloudStorage`], or generic [`Shrine::Storage::Fog`]
+storage. The reference will assume you're using S3 storage.
+
+#### `:fog_credentials`, `:fog_directory`
+
+The S3 Shrine storage accepts `:access_key_id`, `:secret_access_key`, `:region`,
+and `:bucket` options in the initializer:
+
+```rb
+Shrine::Storage::S3.new(
+  access_key_id:     "...",
+  secret_access_key: "...",
+  region:            "...",
+  bucket:            "...",
+)
+```
+
+#### `:fog_attributes`
+
+The object data can be configured via the `:upload_options` hash:
+
+```rb
+Shrine::Storage::S3.new(upload_options: {content_disposition: "attachment"}, **options)
+```
+
+#### `:fog_public`
+
+The object permissions can be configured with the `:acl` upload option:
+
+```rb
+Shrine::Storage::S3.new(upload_options: {acl: "private"}, **options)
+```
+
+#### `:fog_authenticated_url_expiration`
+
+The `#url` method accepts the `:expires_in` option, you can set the default
+expiration with the `default_url_options` plugin:
+
+```rb
+plugin :default_url_options, store: {expires_in: 600}
+```
+
+#### `:fog_use_ssl_for_aws`, `:fog_aws_accelerate`
+
+Shrine allows you to override the S3 endpoint:
+
+```rb
+Shrine::Storage::S3.new(endnpoint: "https://s3-accelerate.amazonaws.com", **options)
+```
+
 [image_processing]: https://github.com/janko-m/image_processing
 [demo app]: https://github.com/janko-m/shrine/tree/master/demo
 [Reprocessing versions]: http://shrinerb.com/rdoc/files/doc/regenerating_versions_md.html
 [shrine-fog]: https://github.com/janko-m/shrine-fog
 [direct uploads]: http://shrinerb.com/rdoc/files/doc/direct_s3_md.html
+[`Shrine::Storage::S3`]: http://shrinerb.com/rdoc/classes/Shrine/Storage/S3.html
+[`Shrine::Storage::GoogleCloudStorage`]: https://github.com/renchap/shrine-google_cloud_storage
+[`Shrine::Storage::Fog`]: https://github.com/janko-m/shrine-fog

data/doc/creating_storages.md

@@ -37,9 +37,21 @@ end
 ## Upload
 
 The job of `Storage#upload` is to upload the given IO object to the storage.
+It's recommended to use [HTTP.rb] for uploading, as it accepts any IO object
+that responds to `#read` (not just file objects), and it streams the IO data
+directly to the socket, making it suitable for large uploads.
+
+```rb
+require "http"
+
+# streaming raw upload
+HTTP.post("http://example.com/upload", body: io)
+# streaming multipart upload
+HTTP.post("http://example.com/upload", form: { file: HTTP::FormData::File.new(io) })
+```
+
 It's good practice to test the storage with a [fake IO] object which responds
-only to required methods. Some HTTP libraries don't support uploading non-file
-IOs, although for [Faraday] and [REST client] you can work around that.
+only to required methods, as not all received IO objects will be file objects.
 
 If your storage doesn't control which id the uploaded file will have, you
 can modify the `id` variable before returning:
@@ -135,28 +147,6 @@ class Shrine
 end
 ```
 
-## Multi delete
-
-If your storage supports deleting multiple files at the same time, you can
-implement an additional method, which will automatically get picked up by the
-`multi_delete` plugin:
-
-```rb
-class Shrine
-  module Storage
-    class MyStorage
-      # ...
-
-      def multi_delete(ids)
-        # deletes multiple files at once
-      end
-
-      # ...
-    end
-  end
-end
-```
-
 ## Clearing
 
 While this method is not used by Shrine, it is good to give users the
@@ -235,6 +225,5 @@ Note that using the linter doesn't mean that you shouldn't write any manual
 tests for your storage. There will likely be some edge cases that won't be
 tested by the linter.
 
+[HTTP.rb]: https://github.com/httprb/http
 [fake IO]: https://github.com/janko-m/shrine-cloudinary/blob/ca587c580ea0762992a2df33fd590c9a1e534905/test/test_helper.rb#L20-L27
-[REST client]: https://github.com/janko-m/shrine-cloudinary/blob/ca587c580ea0762992a2df33fd590c9a1e534905/lib/shrine/storage/cloudinary.rb#L138-L141
-[Faraday]: https://github.com/janko-m/shrine-uploadcare/blob/2038781ace0f54d82fa06cc04c4c2958919208ad/lib/shrine/storage/uploadcare.rb#L140

data/doc/direct_s3.md

@@ -24,7 +24,7 @@ storage service is beneficial for several reasons:
   times out.
 
 You can start by setting both temporary and permanent storage to S3 with
-different prefixes (or even buckets):
+different prefixes (or even different buckets):
 
 ```rb
 # Gemfile
@@ -34,10 +34,10 @@ gem "aws-sdk-s3", "~> 1.2"
 require "shrine/storage/s3"
 
 s3_options = {
-  access_key_id:     "abc",
-  secret_access_key: "123",
-  region:            "my-region",
-  bucket:            "my-bucket",
+  access_key_id:     "<YOUR KEY>",
+  secret_access_key: "<YOUR SECRET>",
+  bucket:            "<YOUR BUCKET>",
+  region:            "<REGION>",
 }
 
 Shrine.storages = {
@@ -49,14 +49,36 @@ Shrine.storages = {
 ## Enabling CORS
 
 In order to be able upload files directly to your S3 bucket, you need enable
-CORS. You can do that in the AWS S3 Console by clicking on "Properties >
-Permissions > Add CORS Configuration", and then just follow the Amazon
-documentation on how to write a CORS file.
+CORS. You can do that from the AWS S3 Console by going to your bucket, clicking
+on the "Permissions" tab, then on "CORS Configuration", and following the
+[guide for configuring CORS][CORS guide].
 
-http://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html
+Alternatively you can configure CORS via an [API call][CORS API]:
 
-Note that due to DNS propagation it may take some time for update of the CORS
-settings to be applied.
+```rb
+require "aws-sdk-s3"
+
+client = Aws::S3::Client.new(
+  access_key_id:     "<YOUR KEY>",
+  secret_access_key: "<YOUR SECRET>",
+  region:            "<REGION>",
+)
+
+client.put_bucket_cors(
+  bucket: "<YOUR BUCKET>",
+  cors_configuration: {
+    cors_rules: [{
+      allowed_headers: ["Authorization", "Content-Type", "Origin"],
+      allowed_methods: ["GET", "POST"],
+      allowed_origins: ["*"],
+      max_age_seconds: 3000,
+    }]
+  }
+)
+```
+
+Note that due to DNS propagation it may take some time for the CORS update to
+be applied.
 
 ## File hash
 
@@ -127,11 +149,9 @@ request headers.
 }
 ```
 
-You can now use a client-side file upload library like [FineUploader],
-[Dropzone] or [jQuery-File-Upload] to upload selected files directly to S3.
-When the user selects a file, the client can make a request to the presign
-endpoint, and use the returned request information to upload the selected file
-directly to S3.
+On the client side you can then make a request to the presign endpoint as soon
+as the user selects a file, and use the returned request information to upload
+the selected file directly to S3. It's recommended to use [Uppy] for this.
 
 Once the file has been uploaded, you can generate a JSON representation of the
 uploaded file on the client-side, and write it to the hidden attachment field.
@@ -168,22 +188,30 @@ generating the form we can use `Shrine::Storage::S3#presign`, which returns a
 ```erb
 <%
   presign = Shrine.storages[:cache].presign SecureRandom.hex,
-    success_action_redirect: new_album_url,
-    allow_any: ['utf8', 'authenticity_token']
+                                            success_action_redirect: new_album_url
 %>
 
 <form action="<%= presign.url %>" method="post" enctype="multipart/form-data">
-  <input type="file" name="file">
   <% presign.fields.each do |name, value| %>
     <input type="hidden" name="<%= name %>" value="<%= value %>">
   <% end %>
+  <input type="file" name="file">
   <input type="submit" value="Upload">
 </form>
 ```
 
-Note the additional `success_action_redirect` option which tells S3 where to
-redirect to after the file has been uploaded. We also tell S3 to exclude the
-`utf8` and `authenticity_token` fields that the Rails form builder generates.
+Note the additional `:success_action_redirect` option which tells S3 where to
+redirect to after the file has been uploaded. If you're using the Rails form
+builder to generate this form, you might need to also tell S3 to ignore the
+additional `utf8` and `authenticity_token` fields that Rails generates:
+
+```rb
+<%
+  presign = Shrine.storages[:cache].presign SecureRandom.hex,
+                                            allow_any: ["utf8", "authenticity_token"],
+                                            success_action_redirect: new_album_url
+%>
+```
 
 Let's assume we specified the redirect URL to be a page which renders the form
 for a new record. S3 will include some information about the upload in form of
@@ -193,7 +221,7 @@ GET parameters in the URL, out of which we only need the `key` parameter:
 <%
   cached_file = {
     storage: "cache",
-    id: params[:key][/cache\/(.+)/, 1], # we have to remove the prefix part
+    id: params[:key][/cache\/(.+)/, 1], # we subtract the storage prefix
     metadata: {},
   }
 %>
@@ -204,7 +232,29 @@ GET parameters in the URL, out of which we only need the `key` parameter:
 </form>
 ```
 
-## Metadata
+## Object data
+
+When the cached S3 object is copied to permanent storage, the destination S3
+object will by default inherit any object data that was assigned to the cached
+object via presign parameters. However, S3 will by default also ignore any new
+object parameters that are given to the copy request.
+
+Whether object data will be copied or replaced depends on the value of the
+`:metadata_directive` parameter:
+
+* `"COPY"` - destination object will inherit source object data and any new data will be ignored (default)
+* `"REPLACE"` - destination object will not inherit any of the source object data and will accept new data
+
+You can use the `upload_options` plugin to change the `:metadata_directive`
+option when S3 objects are copied:
+
+```rb
+plugin :upload_options, store: -> (io, context) do
+  { metadata_directive: "REPLACE" } if io.is_a?(Shrine::UploadedFile)
+end
+```
+
+## Shrine metadata
 
 With direct uploads any metadata has to be extracted on the client-side, since
 the file upload doesn't touch the application, so the Shrine uploader doesn't
@@ -239,9 +289,40 @@ end
 
 ## Clearing cache
 
-Since directly uploaded files will stay in your temporary storage, you will
-want to periodically delete the old ones that were already promoted. Luckily,
-Amazon provides [a built-in solution][object lifecycle] for that.
+Directly uploaded files won't automatically be deleted from your temporary
+storage, so you'll want to periodically clear them. One way to do that is
+by setting up recurring script which calls `Shrine::Storage::S3#clear!`:
+
+```rb
+s3 = Shrine.storages[:cache]
+s3.clear! { |object| object.last_modified < Time.now - 7*24*60*60 } # delete files older than 1 week
+```
+
+Alternatively you can add a bucket lifeycle rule to do this for you. This can
+be done either from the [AWS Console][lifecycle console] or via an [API
+call][lifecycle API]:
+
+```rb
+require "aws-sdk-s3"
+
+client = Aws::S3::Client.new(
+  access_key_id:     "<YOUR KEY>",
+  secret_access_key: "<YOUR SECRET>",
+  region:            "<REGION>",
+)
+
+client.put_bucket_lifecycle_configuration(
+  bucket: "<YOUR BUCKET>",
+  lifecycle_configuration: {
+    rules: [{
+      expiration: { days: 7 },
+      filter: { prefix: "cache/" },
+      id: "cache-clear",
+      status: "Enabled"
+    }]
+  }
+)
+```
 
 ## Eventual consistency
 
@@ -274,8 +355,9 @@ end
 
 [`Aws::S3::PresignedPost`]: http://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Bucket.html#presigned_post-instance_method
 [demo app]: https://github.com/janko-m/shrine/tree/master/demo
-[Dropzone]: https://github.com/enyo/dropzone
-[jQuery-File-Upload]: https://github.com/blueimp/jQuery-File-Upload
-[FineUploader]: https://github.com/FineUploader/fine-uploader
+[Uppy]: https://uppy.io
 [Amazon S3 Data Consistency Model]: http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyMode
-[object lifecycle]: http://docs.aws.amazon.com/AmazonS3/latest/UG/lifecycle-configuration-bucket-no-versioning.html
+[CORS guide]: http://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html
+[CORS API]: https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#put_bucket_cors-instance_method
+[lifecycle Console]: http://docs.aws.amazon.com/AmazonS3/latest/UG/lifecycle-configuration-bucket-no-versioning.html
+[lifecycle API]: https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#put_bucket_lifecycle_configuration-instance_method