shrine 3.0.1 → 3.3.0

data/doc/securing_uploads.md

@@ -1,6 +1,6 @@
 ---
 id: securing-uploads
-title: Securing uploads
+title: Securing Uploads
 ---
 
 Shrine does a lot to make your file uploads secure, but there are still a lot
@@ -63,7 +63,7 @@ in the `:max_size` option to reject files that are larger than the specified
 limit:
 
 ```rb
-plugin :upload_endpoint, max_size: 100*1024*1024 # 20 MB
+plugin :upload_endpoint, max_size: 100*1024*1024 # 100 MB
 ```
 
 If you're doing direct uploads to Amazon S3 using the `presign_endpoint`
@@ -151,7 +151,7 @@ processing:
 ```rb
 class ImageUploader < Shrine
   # ...
-  Attacher.derivatives_processor do |original|
+  Attacher.derivatives do |original|
     width, height = Shrine.dimensions(original)
 
     fail ImageBombError if width > 5000 || height > 5000

data/doc/storage/file_system.md

@@ -1,6 +1,6 @@
 ---
 id: file-system
-title: Shrine::Storage::FileSystem
+title: File System
 ---
 
 The FileSystem storage handles uploads to the filesystem, and it is most

data/doc/storage/memory.md

@@ -0,0 +1,19 @@
+---
+title: Memory
+---
+
+The Memory storage stores uploaded files in memory, which is suitable for
+testing.
+
+```rb
+Shrine.storages[:store] = Shrine::Storage::Memory.new
+```
+
+By default, each storage instance uses a new Hash object for storing files,
+but you can pass your own:
+
+```rb
+my_store = Hash.new
+
+Shrine.storages[:store] = Shrine::Storage::Memory.new(my_store)
+```

data/doc/storage/s3.md

@@ -1,31 +1,41 @@
 ---
-title: Shrine::Storage::S3
+title: AWS S3
 ---
 
-The S3 storage handles uploads to Amazon S3 service, using the [aws-sdk-s3]
+The S3 storage handles uploads to [AWS S3] service (or any s3-compatible
+service such as [DigitalOcean Spaces] or [MinIO]). It requires the [aws-sdk-s3]
 gem:
 
 ```rb
+# Gemfile
 gem "aws-sdk-s3", "~> 1.14"
 ```
 
-It can be initialized by providing the bucket name and credentials:
+## Initialization
+
+The storage is initialized by providing your bucket name, region and
+credentials:
 
 ```rb
 require "shrine/storage/s3"
 
 s3 = Shrine::Storage::S3.new(
   bucket: "my-app", # required
+  region: "eu-west-1", # required
   access_key_id: "abc",
   secret_access_key: "xyz",
-  region: "eu-west-1",
 )
 ```
 
-The core features of this storage require the following AWS permissions:
-`s3:ListBucket`, `s3:PutObject`, `s3:GetObject`, and `s3:DeleteObject`. If you
-have additional upload options configured such as setting object ACLs, then
-additional permissions may be required.
+> The storage requires the following AWS S3 permissions:
+>
+>  * `s3:ListBucket` for the bucket resource
+>  * `s3:GetObject`, `s3:PutObject`, `s3:PutObjectAcl`, `s3:DeleteObject`,
+>    `s3:ListMultipartUploadParts` and `s3:AbortMultipartUpload` for the object
+>    resources
+
+> The `:access_key_id` and `:secret_access_key` options is just one form of
+> authentication, see the [AWS SDK docs][credentials] for more options.
 
 The storage exposes the underlying Aws objects:
 
@@ -45,14 +55,29 @@ s3.object("key") #=> #<Aws::S3::Object>
 
 By default, uploaded S3 objects will have private visibility, meaning they can
 only be accessed via signed expiring URLs generated using your private S3
-credentials. If you would like to generate public URLs, you can tell S3 storage
-to make uploads public:
+credentials.
 
 ```rb
-s3 = Shrine::Storage::S3.new(public: true, **s3_options)
+s3 = Shrine::Storage::S3.new(**s3_options)
+s3.upload(io, "key") # uploads with default "private" ACL
+s3.url("key")        # https://my-bucket.s3.amazonaws.com/key?X-Amz-Expires=900&X-Amz-Signature=b22d37c37d...
+```
 
+If you would like to generate public URLs, you can tell S3 storage to make
+uploads public:
+
+```rb
+s3 = Shrine::Storage::S3.new(public: true, **s3_options)
 s3.upload(io, "key") # uploads with "public-read" ACL
-s3.url("key")        # returns public (unsigned) object URL
+s3.url("key")        # https://my-bucket.s3.amazonaws.com/key
+```
+
+If you want to make only *some* uploads public, you can conditionally apply the
+`:acl` upload option and `:public` URL option:
+
+```rb
+Shrine.plugin :upload_options, store: -> (io, **) { { acl: "public-read" } }
+Shrine.plugin :url_options,    store: -> (io, **) { { public: true } }
 ```
 
 ## Prefix
@@ -80,13 +105,11 @@ You can also generate upload options per upload with the `upload_options`
 plugin
 
 ```rb
-class MyUploader < Shrine
-  plugin :upload_options, store: -> (io, derivative: nil, **) do
-    if derivative == :thumb
-      { acl: "public-read" }
-    else
-      { acl: "private" }
-    end
+Shrine.plugin :upload_options, store: -> (io, derivative: nil, **) do
+  if derivative == :thumb
+    { acl: "public-read" }
+  else
+    { acl: "private" }
   end
 end
 ```
@@ -97,23 +120,9 @@ or when using the uploader directly
 uploader.upload(file, upload_options: { acl: "private" })
 ```
 
-Note that, unlike the `:upload_options` storage option, upload options given on
-the uploader level won't be forwarded for generating presigns, since presigns
-are generated using the storage directly.
-
-## URL options
-
-Other than [`:host`](#url-host) and [`:public`](#public-uploads) URL options,
-all additional options are forwarded to [`Aws::S3::Object#presigned_url`].
-
-```rb
-s3.url(
-  expires_in: 15,
-  response_content_disposition: ContentDisposition.attachment("my-filename"),
-  response_content_type: "foo/bar",
-  # ...
-)
-```
+> Unlike the `:upload_options` storage option, upload options given on
+  the uploader level won't be forwarded for generating presigns, since presigns
+  are generated using the storage directly.
 
 ## URL Host
 
@@ -133,12 +142,14 @@ s3.url("image.jpg", host: "https://your-s3-host.com/prefix/") # needs to end wit
 ```
 
 To have the `:host` option passed automatically for every URL, use the
-`url_options` plugin.
+`url_options` plugin:
 
 ```rb
 plugin :url_options, store: { host: "http://abc123.cloudfront.net" }
 ```
 
+### Signer
+
 If you would like to [serve private content via CloudFront], you need to sign
 the object URLs with a special signer, such as [`Aws::CloudFront::UrlSigner`]
 provided by the `aws-sdk-cloudfront` gem. The S3 storage initializer accepts a
@@ -157,13 +168,27 @@ Shrine::Storage::S3.new(signer: signer.method(:signed_url))
 Shrine::Storage::S3.new(signer: -> (url, **options) { signer.signed_url(url, **options) })
 ```
 
+## URL options
+
+Other than `:host` and `:public` URL options, all additional `S3#url` options
+are forwarded to [`Aws::S3::Object#presigned_url`].
+
+```rb
+s3.url(
+  expires_in: 15,
+  response_content_disposition: ContentDisposition.attachment("my-filename"),
+  response_content_type: "foo/bar",
+  # ...
+)
+```
+
 ## Presigns
 
-The `#presign` method can be used for generating paramters for direct uploads
-to Amazon S3:
+The `S3#presign` method can be used for generating parameters for direct upload
+to S3:
 
 ```rb
-s3.presign("/path/to/file") #=>
+s3.presign("key") #=>
 # {
 #   url: "https://my-bucket.s3.amazonaws.com/...",
 #   fields: { ... },  # blank for PUT presigns
@@ -172,11 +197,25 @@ s3.presign("/path/to/file") #=>
 # }
 ```
 
-Additional presign options can be given in three places:
+By default, parameters for a POST upload is generated, but you can also
+generate PUT upload parameters:
+
+```rb
+s3.presign("key", method: :put)
+```
 
-* in `Storage::S3#presign` by forwarding options
-* in `:upload_options` option on this storage
-* in `presign_endpoint` plugin through `:presign_options`
+Any additional options are forwarded to [`Aws::S3::Object#presigned_post`]
+(for POST uploads) and [`Aws::S3::Object#presigned_url`] (for PUT uploads).
+
+```rb
+s3.presign("key", method: :put, content_disposition: "attachment; filename=my-file.txt") #=>
+# {
+#   url: "https://my-bucket.s3.amazonaws.com/...",
+#   fields: {},
+#   headers: { "Content-Disposition" => "attachment; filename=my-file.txt" },
+#   method :put,
+# }
+```
 
 ## Large files
 
@@ -184,33 +223,24 @@ The aws-sdk-s3 gem has the ability to automatically use multipart upload/copy
 for larger files, splitting the file into multiple chunks and uploading/copying
 them in parallel.
 
-By default any files that are uploaded will use the multipart upload if they're
-larger than 15MB, and any files that are copied will use the multipart copy if
-they're larger than 150MB, but you can change the thresholds via
-`:multipart_threshold`.
-
-```rb
-thresholds = { upload: 30*1024*1024, copy: 200*1024*1024 }
-Shrine::Storage::S3.new(multipart_threshold: thresholds, **s3_options)
-```
-
-If you want to change how many threads aws-sdk-s3 will use for multipart
-upload/copy, you can use the `upload_options` plugin to specify
-`:thread_count`.
+By default, multipart upload will be used for files larger than 15MB, and
+multipart copy for files larger than 100MB, but you can change the thresholds
+via `:multipart_threshold`:
 
 ```rb
-plugin :upload_options, store: -> (io, context) do
-  { thread_count: 5 }
-end
+Shrine::Storage::S3.new(
+  multipart_threshold: { upload: 30*1024*1024, copy: 200*1024*1024 },
+  **s3_options,
+)
 ```
 
 ## Encryption
 
-The easiest way to use server-side encryption for uploaded S3 objects is to
+The easiest way to use **server-side** encryption for uploaded S3 objects is to
 configure default encryption for your S3 bucket. Alternatively, you can pass
 server-side encryption parameters to the API calls.
 
-The `#upload` method accepts `:sse_*` options:
+The `S3#upload` method accepts `:sse_*` options:
 
 ```rb
 s3.upload(io, "key", sse_customer_algorithm: "AES256",
@@ -219,7 +249,7 @@ s3.upload(io, "key", sse_customer_algorithm: "AES256",
                      ssekms_key_id:          "key_id")
 ```
 
-The `#presign` method accepts `:server_side_encryption_*` options for POST
+The `S3#presign` method accepts `:server_side_encryption_*` options for POST
 presigns, and the same `:sse_*` options as above for PUT presigns.
 
 ```rb
@@ -232,24 +262,19 @@ When downloading encrypted S3 objects, the same server-side encryption
 parameters need to be passed in.
 
 ```rb
-s3.download("key", sse_customer_algorithm: "AES256",
-                   sse_customer_key:       "secret_key",
-                   sse_customer_key_md5:   "secret_key_md5")
-
 s3.open("key", sse_customer_algorithm: "AES256",
                sse_customer_key:       "secret_key",
                sse_customer_key_md5:   "secret_key_md5")
 ```
 
-If you want to use client-side encryption instead, you can instantiate the
-storage with an `Aws::S3::Encryption::Client` instance.
+**Client-side** encryption is supported as well:
 
 ```rb
-client = Aws::S3::Encryption::Client.new(
-  kms_key_id: "alias/my-key"
-)
+encryption_client = Aws::S3::EncryptionV2::Client.new(...)
+s3 = Shrine::Storage::S3.new(client: encryption_client, **other_options)
 
-Shrine::Storage::S3(client: client, bucket: "my-bucket")
+s3.upload(io, "key") # encrypts on upload
+s3.open("key")       # decrypts on download
 ```
 
 ## Accelerate endpoint
@@ -283,20 +308,18 @@ Alternatively you can periodically call the `#clear!` method:
 s3.clear! { |object| object.last_modified < Time.now - 7*24*60*60 }
 ```
 
-## Request Rate and Performance Guidelines
-
-Amazon S3 automatically scales to high request rates. For example, your
-application can achieve at least 3,500 PUT/POST/DELETE and 5,500 GET requests
-per second per prefix in a bucket (a prefix is a top-level "directory" in the
-bucket). If your app needs to support higher request rates to S3 than that, you
-can scale exponentially by using more prefixes.
-
+[AWS S3]: https://aws.amazon.com/s3/
+[MinIO]: https://min.io/
+[DigitalOcean Spaces]: https://www.digitalocean.com/products/spaces/
+[aws-sdk-s3]: https://rubygems.org/gems/aws-sdk-s3
 [uploading]: http://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#put-instance_method
 [copying]: http://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#copy_from-instance_method
 [presigning]: http://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_post-instance_method
 [`Aws::S3::Object#presigned_url`]: https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_url-instance_method
-[aws-sdk-s3]: https://github.com/aws/aws-sdk-ruby/tree/master/gems/aws-sdk-s3
 [Transfer Acceleration]: http://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html
 [object lifecycle]: http://docs.aws.amazon.com/AmazonS3/latest/UG/lifecycle-configuration-bucket-no-versioning.html
 [serve private content via CloudFront]: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/PrivateContent.html
 [`Aws::CloudFront::UrlSigner`]: https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/CloudFront/UrlSigner.html
+[credentials]: https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/setup-config.html
+[`Aws::S3::Object#presigned_post`]: https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_post-instance_method
+[`Aws::S3::Object#presigned_url`]: https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_url-instance_method

data/doc/testing.md

@@ -119,7 +119,7 @@ module TestData
       small:  uploaded_image,
     )
 
-    attacher.column_data
+    attacher.column_data # or attacher.data in case of postgres jsonb column
   end
 
   def uploaded_image
@@ -128,7 +128,7 @@ module TestData
     # for performance we skip metadata extraction and assign test metadata
     uploaded_file = Shrine.upload(file, :store, metadata: false)
     uploaded_file.metadata.merge!(
-      "size"      => file.size,
+      "size"      => File.size(file.path),
       "mime_type" => "image/jpeg",
       "filename"  => "test.jpg",
     )

data/doc/upgrading_to_3.md

@@ -283,8 +283,9 @@ attacher.destroy_background # calls destroy block
 ## Versions
 
 The `versions`, `processing`, `recache`, and `delete_raw` plugins have been
-deprecated in favour of the new [`derivatives`][derivatives] plugin. Let's
-assume you have the following `versions` code:
+deprecated in favour of the new **[`derivatives`][derivatives]** plugin.
+
+Let's assume you have the following `versions` configuration:
 
 ```rb
 class ImageUploader < Shrine
@@ -307,27 +308,35 @@ class ImageUploader < Shrine
   end
 end
 ```
+
+When an attached file is promoted to permanent storage, the versions would
+automatically get generated:
+
 ```rb
 photo = Photo.new(photo_params)
 
 if photo.valid?
-  photo.save # automatically calls processing block
+  photo.save # generates versions on promotion
   # ...
 else
   # ...
 end
 ```
 
-With `derivatives` it becomes this:
+With `derivatives`, the original file is automatically downloaded and retained
+during processing, so the setup is simpler:
 
 ```rb
-Shrine.plugin :derivatives, versions_compatibility: true # handle versions column format
+Shrine.plugin :derivatives,
+  create_on_promote:      true, # automatically create derivatives on promotion
+  versions_compatibility: true  # handle versions column format
 ```
 ```rb
 class ImageUploader < Shrine
-  Attacher.derivatives_processor do |original|
+  Attacher.derivatives do |original|
     magick = ImageProcessing::MiniMagick.source(original)
 
+    # the :original file should NOT be included anymore
     {
       large:  magick.resize_to_limit!(800, 800),
       medium: magick.resize_to_limit!(500, 500),
@@ -340,28 +349,13 @@ end
 photo = Photo.new(photo_params)
 
 if photo.valid?
-  photo.image_derivatives! if photo.image_changed? # create derivatives
-  photo.save # automatically calls processing block
+  photo.save # creates derivatives on promotion
   # ...
 else
   # ...
 end
 ```
 
-If you have multiple places where you need to generate derivatives, and want it
-to happen automatically like it did with the `versions` plugin, you can
-override `Attacher#promote` to call `Attacher#create_derivatives` before
-promotion:
-
-```rb
-class Shrine::Attacher
-  def promote(*)
-    create_derivatives
-    super
-  end
-end
-```
-
 ### Accessing derivatives
 
 The derivative URLs are accessed in the same way as versions:
@@ -370,7 +364,7 @@ The derivative URLs are accessed in the same way as versions:
 photo.image_url(:small)
 ```
 
-But the derivatives themselves are accessed differently:
+But the files themselves are accessed differently:
 
 ```rb
 # versions
@@ -424,8 +418,8 @@ database column in different formats:
 
 The `:versions_compatibility` flag to the `derivatives` plugin enables it to
 read the `versions` format, which aids in transition. Once the `derivatives`
-plugin has been deployed to production, you can switch existing records to the
-new column format:
+plugin has been deployed to production, you can update existing records with
+the new column format:
 
 ```rb
 Photo.find_each do |photo|
@@ -465,11 +459,11 @@ creating another derivatives processor that you will trigger in the controller:
 
 ```rb
 class ImageUploader < Shrine
-  Attacher.derivatives_processor do |original|
+  Attacher.derivatives do |original|
     # this will be triggered in the background job
   end
 
-  Attacher.derivatives_processor :foreground do |original|
+  Attacher.derivatives :foreground do |original|
     # this will be triggered in the controller
   end
 end
@@ -486,48 +480,77 @@ else
 end
 ```
 
-#### Parallelize
+### Default URL
 
-The `parallelize` plugin has been removed. The `derivatives` plugin is
-thread-safe, so you can parallelize uploading processed files manually:
+If you were using the `default_url` plugin, the `Attacher.default_url` now
+receives a `:derivative` option:
 
 ```rb
-# Gemfile
-gem "concurrent-ruby"
+Attacher.default_url do |derivative: nil, **|
+  "https://my-app.com/fallbacks/#{derivative}.jpg" if derivative
+end
 ```
-```rb
-require "concurrent"
 
-derivatives = attacher.process_derivatives
+#### Fallback to original
 
-tasks = derivatives.map do |name, file|
-  Concurrent::Promises.future(name, file) do |name, file|
-    attacher.add_derivative(name, file)
-  end
-end
+With the `versions` plugin, a missing version URL would automatically fall back
+to the original file. The `derivatives` plugin has no such fallback, but you
+can configure it manually:
 
-Concurrent::Promises.zip(*tasks).wait!
+```rb
+Attacher.default_url do |derivative: nil, **|
+  file&.url if derivative
+end
 ```
 
-#### Default URL
+#### Fallback to version
 
-The `derivatives` plugin integrates with the `default_url` plugin:
+The `versions` plugin had the ability to fall back missing version URL to
+another version that already exists. The `derivatives` plugin doesn't have this
+built in, but you can implement it as follows:
 
 ```rb
+DERIVATIVE_FALLBACKS = { foo: :bar, ... }
+
 Attacher.default_url do |derivative: nil, **|
-  "https://my-app.com/fallbacks/#{derivative}.jpg" if derivative
+  derivatives[DERIVATIVE_FALLBACKS[derivative]]&.url if derivative
+end
+```
+
+### Location
+
+The `Shrine#generate_location` method will now receive a `:derivative`
+parameter instead of `:version`:
+
+```rb
+class MyUploader < Shrine
+  def generate_location(io, derivative: nil, **)
+    derivative #=> :large, :medium, :small, ...
+    # ...
+  end
 end
 ```
 
-However, it doesn't implement any other URL fallbacks that the `versions`
-plugin has for missing derivatives.
+### Overwriting original
+
+With the `derivatives` plugin, saving processed files separately from the
+original file, so the original file is automatically kept. This means it's not
+possible anymore to overwrite the original file as part of processing.
+
+However, **it's highly recommended to always keep the original file**, even if
+you don't plan to use it. That way, if there is ever a need to reprocess
+derivatives, you have the original file to use as a base.
+
+That being said, if you still want to overwrite the original file, [this
+thread][overwriting original] has some tips.
 
 ## Other
 
 ### Processing
 
 The `processing` plugin has been deprecated over the new
-[`derivatives`][derivatives] plugin. If you were modifying the original file:
+[`derivatives`][derivatives] plugin. If you were previously replacing the
+original file:
 
 ```rb
 class MyUploader < Shrine
@@ -547,7 +570,7 @@ you should now add the processed file as a derivative:
 class MyUploader < Shrine
   plugin :derivatives
 
-  Attacher.derivatives_processor do |original|
+  Attacher.derivatives do |original|
     magick = ImageProcessing::MiniMagick.source(original)
 
     { normalized: magick.resize_to_limit!(1600, 1600) }
@@ -555,6 +578,30 @@ class MyUploader < Shrine
 end
 ```
 
+### Parallelize
+
+The `parallelize` plugin has been removed. With `derivatives` plugin you can
+parallelize uploading processed files manually:
+
+```rb
+# Gemfile
+gem "concurrent-ruby"
+```
+```rb
+require "concurrent"
+
+attacher    = photo.image_attacher
+derivatives = attacher.process_derivatives
+
+tasks = derivatives.map do |name, file|
+  Concurrent::Promises.future(name, file) do |name, file|
+    attacher.add_derivative(name, file)
+  end
+end
+
+Concurrent::Promises.zip(*tasks).wait!
+```
+
 ### Logging
 
 The `logging` plugin has been removed in favour of the
@@ -602,7 +649,7 @@ attacher.copy(other_attacher)
 with
 
 ```rb
-attacher.attach other_attacher.file
+attacher.set attacher.upload(other_attacher.file)
 attacher.add_derivatives other_attacher.derivatives # if using derivatives
 ```
 
@@ -660,3 +707,4 @@ end
 [derivatives]: https://shrinerb.com/docs/plugins/derivatives
 [instrumentation]: https://shrinerb.com/docs/plugins/instrumentation
 [mirroring]: https://shrinerb.com/docs/plugins/mirroring
+[overwriting original]: https://discourse.shrinerb.com/t/keep-original-file-after-processing/50/4