shrine 2.13.0 → 2.14.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9f51ec7d13c7f830e1cebb7d12c6444aad0b0c2df6c7d57e2cec61da7e8bc8b1
-  data.tar.gz: 9c75a7232d0e10910582a418646fa850305ef2cc164a6a8bc6e021ec52ac5de7
+  metadata.gz: 705f49f8a110b3b50eb00c0fdf17da95dce2fa7782aaadfe20808af178a46405
+  data.tar.gz: 41789005e3146dc56a327d7d0c4fe6de12153cb8135789a9a3cf56212fdd1d13
 SHA512:
-  metadata.gz: 03c0e1e29b8ba35b64fc6e33ab1a297a7f843726e3e7d21bca57449d1b36ee14c2781c3a2ddc6aa1f5bf36aeff9c7f8e77ee1eec8658cc3177bf701b9e942362
-  data.tar.gz: 3d2e11839c487fbd1782bc41751718a6d8e601070aa4c2a03f469f79a9b6fd176d2f5da9db37f2d1236ec1c64bb480c580217061f1d67c87eec0a2f86ae46d74
+  metadata.gz: 7f9ed520831fb14e199ef8ee9bddeaac47f7cf4151c5eb66ad68620c8c45db6c01d4fc4d62c6ff87a9bed2a771d48c97cc234ba903f1c3bf4005e79bcfa52d33
+  data.tar.gz: e70bbbd5f5731778112c4ce700df42b08b207368b5898625dbff8123faae40e383e7429b134d3850c485bfc5cab1e142a6c123f6e1a4adc6bb3d430484377a82

data/CHANGELOG.md

@@ -1,3 +1,73 @@
+## 2.14.0 (2018-12-27)
+
+* Add `tempfile` plugin for easier reusing of the same uploaded file copy on disk (@janko-m)
+
+* Don't re-open the uploaded file if it's already open in `refresh_metadata` plugin (@janko-m)
+
+* Drop support for MRI 2.1 and 2.2 (@janko-m)
+
+* Fix `backgrounding` not working when default storage was changed with `Attachment.new` (@janko-m)
+
+* Don't clear existing metadata definitions when loading `add_metadata` plugin (@janko-m)
+
+* Don't clear existing processing blocks when loading `processing` plugin (@janko-m)
+
+* Deprecate automatic escaping of `:content_disposition` in `Shrine::Storage::S3` (@janko-m)
+
+* Use `content_disposition` gem in `Shrine::Storage::S3` and `rack_response` plugin (@janko-m)
+
+* Make `FileSystem#clear!` work correctly when the storage directory is a symlink (@janko-m)
+
+* Don't abort promotion in `backgrounding` plugin when original metadata was updated (@janko-m)
+
+* Don't mutate the `UploadedFile` data hash in `refresh_metadata` plugin (@janko-m)
+
+* Deprecate `Storage::S3#download` (@janko-m)
+
+* Stop using `Storage#download` in `UploadedFile#download` for peformance (@janko-m)
+
+* Remove `#download` from the Shrine storage specification (@janko-m)
+
+* Keep `context` argument in `#extract_metadata` optional after loading `add_metadata` plugin (@janko-m)
+
+* Include metadata key with `nil` value when `nil` is returned in `add_metadata` block (@janko-m)
+
+* Strip query params in upload location when re-uploading from `shrine-url` storage (@jrochkind)
+
+* Inline Base plugin into core classes, extract them to separate files (@printercu)
+
+* Make `rack_response` plugin work with `Rack::Sendfile` for `FileSystem` storage (@janko-m)
+
+* Add `:filename` and `:type` options to `rack_response` plugin (@janko-m)
+
+* Add `:host` option to `UploadedFile#download_url` in `download_endpoint` plugin (@janko-m)
+
+* Add support for client-side encryption to S3 storage (@janko-m)
+
+* Don't look up the attachment class in each new model instance (@printercu)
+
+* Allow `Attacher#cached?` and `Attacher#stored?` to take an `UploadedFile` object (@jrochkind)
+
+* Allow assigning a filename to the `DataFile` object in `Shrine.data_uri` (@janko-m)
+
+* Don't strip media type parameters for the `DataFile` object in `data_uri` plugin (@janko-m)
+
+* Add `:content_type` analyzer to `Shrine.mime_type_analyzers` in `determine_mime_type` plugin (@janko-m)
+
+* Rename `:default` analyzer to `:content_type` in `determine_mime_type` plugin (@janko-m)
+
+* Don't display a warning when `determine_mime_type` plugin is loaded with `:default` analyzer (@janko-m)
+
+* Exclude media type parameters when copying `IO#content_type` into `mime_type` metadata (@janko-m)
+
+* Remove superfluous `#head_object` S3 API call in `S3#download` (@janko-m)
+
+* Make `S3#download` and `S3#open` work with server side encryption options (@janko-m)
+
+* Make previously extracted metadata available under `:metadata` in `add_metadata` plugin (@jrochkind)
+
+* Use a guard raise cause for `bucket` argument in S3 for an appropriate error message (@ardecvz)
+
 ## 2.13.0 (2018-11-04)
 
 * Specify UTF-8 charset in `Content-Type` response header in `presign_endpoint` plugin (@janko-m)
@@ -6,6 +76,8 @@
 
 * Force UTF-8 encoding on filenames coming from Rack's multipart request params in `rack_file` plugin (@janko-m)
 
+* Raise `Shrine::Error` if `file` command returns error in stdout in `determine_mime_type` plugin (@janko-m)
+
 * Allow `:host` in `S3#url` to specify a host URL with an additional path prefix (@janko-m)
 
 * Revert adding bucket name to URL path in `S3#url` when `:host` is used with `:force_path_style` (@janko-m)

data/README.md

@@ -226,12 +226,12 @@ plugin is loaded.
 Here are some examples of IO objects that can be uploaded:
 
 ```rb
-uploader.upload File.open("/path/to/file", "rb")             # upload from disk
-uploader.upload StringIO.new("file content")                 # upload from memory
-uploader.upload ActionDispatch::Http::UploadedFile.new       # upload from Rails controller
-uploader.upload Shrine.rack_file({ tempfile: Tempfile.new }) # upload from Rack controller
-uploader.upload Rack::Test::UploadedFile.new                 # upload from rack-test
-uploader.upload Down.open("https://example.org/file")        # upload from internet
+uploader.upload File.open("/path/to/file", binmode: true) # upload from disk
+uploader.upload StringIO.new("file content")              # upload from memory
+uploader.upload ActionDispatch::Http::UploadedFile.new    # upload from Rails controller
+uploader.upload Shrine.rack_file({ tempfile: tempfile })  # upload from Rack controller
+uploader.upload Rack::Test::UploadedFile.new              # upload from rack-test
+uploader.upload Down.open("https://example.org/file")     # upload from internet
 ```
 
 `Shrine::UploadedFile`, the object returned after upload, is itself an IO-like
@@ -285,12 +285,8 @@ uploaded_file.rewind # rewinds the IO
 uploaded_file.close  # closes the IO
 ```
 
-If you want to retrieve the content of the uploaded file, you can use a
-combination of `#open` and `#read`:
-
-```rb
-uploaded_file.open(&:read) #=> "..." (binary content of the uploaded file)
-```
+For more details on these `Shrine::UploadedFile` methods, see the [Retrieving
+Uploads] guide.
 
 ## Attachment
 
@@ -465,6 +461,13 @@ uploaded_file.mime_type         #=> "video/mp4"
 uploaded_file.size              #=> 345993
 ```
 
+By default these values are determined from the following attributes on the IO
+object:
+
+* `filename` – `io.original_filename` or `io.path`
+* `mime_type` – `io.content_type`
+* `size` – `io.size`
+
 ### MIME type
 
 By default `mime_type` will be inherited from `#content_type` attribute of the
@@ -765,7 +768,7 @@ Shrine.plugin :presign_endpoint
 ```
 ```rb
 # config.ru (Rack)
-map "/presign" do
+map "/s3/params" do
   run Shrine.presign_endpoint(:cache)
 end
 
@@ -773,11 +776,11 @@ end
 
 # config/routes.rb (Rails)
 Rails.application.routes.draw do
-  mount Shrine.presign_endpoint(:cache) => "/presign"
+  mount Shrine.presign_endpoint(:cache) => "/s3/params"
 end
 ```
 
-The above will add a `GET /presign` route to your app. You can now hook Uppy's
+The above will add a `GET /s3/params` route to your app. You can now hook Uppy's
 [AWS S3][uppy aws s3] plugin to this endpoint and have it upload directly to
 S3. See [this walkthrough][direct S3 uploads walkthrough] that shows adding
 direct S3 uploads from scratch, as well as the [Direct Uploads to S3][direct S3
@@ -789,7 +792,7 @@ If you wanted to implement this enpdoint yourself, this is how it could roughly
 look like for S3 storage in Sinatra:
 
 ```rb
-get "/presign" do
+get "/s3/params" do
   storage  = Shrine.storages[:cache]
   location = SecureRandom.hex + File.extname(params["filename"].to_s)
 
@@ -990,6 +993,7 @@ The gem is available as open source under the terms of the [MIT License].
 [external storages]: https://shrinerb.com/#external
 [creating storage]: https://shrinerb.com/rdoc/files/doc/creating_storages_md.html
 [creating plugin]: https://shrinerb.com/rdoc/files/doc/creating_plugins_md.html
+[Retrieving Uploads]: https://shrinerb.com/rdoc/files/doc/retrieving_uploads_md.html
 [Using Attacher]: https://shrinerb.com/rdoc/files/doc/attacher_md.html
 [plugins]: https://shrinerb.com/#plugins
 [`file`]: http://linux.die.net/man/1/file

data/doc/creating_storages.md

@@ -122,27 +122,6 @@ The storage can support additional options to customize how the file will be
 opened, `Shrine::UploadedFile#open` and `Shrine::UploadedFile#download` will
 forward any given options to `#open`.
 
-## Download
-
-`Shrine::UploadedFile#download` by default uses the `#open` storage method to
-stream file content to a Tempfile. However, if you would like to use your own
-custom way of downloading to a file, you can define `#download` on the storage
-and `Shrine::UploadedFile#download` will automatically call that instead.
-
-```rb
-class MyStorage
-  # ...
-  def download(id, **options)
-    # download the uploaded file to a Tempfile
-  end
-  # ...
-end
-```
-
-The storage can support additional options to customize how the file will be
-downloaded, `Shrine::UploadedFile#download` will forward any given options to
-`#download`.
-
 ## Url
 
 The `#url` storage method is called by `Shrine::UploadedFile#url`, it accepts a

data/doc/design.md

@@ -125,6 +125,7 @@ uploaded_file.size
 # storage methods
 uploaded_file.url
 uploaded_file.exists?
+uploaded_file.open
 uploaded_file.download
 uploaded_file.delete
 # ...

data/doc/direct_s3.md

@@ -96,25 +96,26 @@ S3. It's recommended to use [Uppy] for client side uploads.
 The `presign_endpoint` plugin provides a Rack application that generates these
 upload parameters, which we can just mount in our application. We'll make our
 presign endpoint also use the additional `type` and `filename` query parameters
-to set `Content-Type` and `Content-Disposition` for the uploaded file, as well
-as limit the upload size to 10 MB (see [`Shrine::Storage::S3#presign`] for the
-list of available options).
+to set `Content-Type` header, `Content-Disposition` header (using the
+[content_disposition] gem), as well as limit the upload size to 10 MB (see
+[`Shrine::Storage::S3#presign`] for the list of available options).
 
 ```rb
 Shrine.plugin :presign_endpoint, presign_options: -> (request) {
+  # Uppy will send the "filename" and "type" query parameters
   filename = request.params["filename"]
   type     = request.params["type"]
 
   {
-    content_disposition:    "inline; filename=\"#{filename}\"", # set download filename
-    content_type:           type,                               # set content type (required if using DigitalOcean Spaces)
-    content_length_range:   0..(10*1024*1024),                  # limit upload size to 10 MB
+    content_disposition:    ContentDisposition.inline(filename), # set download filename
+    content_type:           type,                                # set content type (required if using DigitalOcean Spaces)
+    content_length_range:   0..(10*1024*1024),                   # limit upload size to 10 MB
   }
 }
 ```
 ```rb
 # config.ru (Rack)
-map "/presign" do
+map "/s3/params" do
   run Shrine.presign_endpoint(:cache)
 end
 
@@ -122,17 +123,17 @@ end
 
 # config/routes.rb (Rails)
 Rails.application.routes.draw do
-  mount Shrine.presign_endpoint(:cache) => "/presign"
+  mount Shrine.presign_endpoint(:cache) => "/s3/params"
 end
 ```
 
-The above will create a `GET /presign` route, which internally calls
+The above will create a `GET /s3/params` route, which internally calls
 [`Shrine::Storage::S3#presign`] to return the HTTP verb (POST) and the S3 URL
 to which the file should be uploaded, along with the required POST parameters
 and request headers.
 
 ```rb
-# GET /presign
+# GET /s3/params
 {
   "method": "post",
   "url": "https://my-bucket.s3-eu-west-1.amazonaws.com",
@@ -148,11 +149,11 @@ and request headers.
 }
 ```
 
-Uppy's [AWS S3][uppy aws s3] plugin would then make a request to this endpoint and use these
-parameters to upload the file directly to S3. Once the file has been uploaded,
-you can generate a JSON representation of the uploaded file on the client side,
-and write it to the hidden attachment field (or send it directly in an AJAX
-request).
+Uppy's [AWS S3][uppy aws s3] plugin would then make a request to this endpoint
+and use these parameters to upload the file directly to S3. Once the file has
+been uploaded, you can generate a JSON representation of the uploaded file on
+the client side, and write it to the hidden attachment field (or send it
+directly in an AJAX request).
 
 ```rb
 {
@@ -176,6 +177,13 @@ upload walkthrough] for adding dynamic direct S3 uploads from scratch, as well
 as the [Roda][roda demo] / [Rails][rails demo] demo app for a complete example
 of multiple direct S3 uploads.
 
+Also, if you're dealing with larger files, you may want to make the uploads
+resumable by using the [Aws S3 Multipart][uppy aws s3 multipart] Uppy plugin
+instead, with the [uppy-s3_multipart] gem on the backend. Your back-end
+implementation is similar, just using `Shrine.uppy_s3_multipart` in place of
+`Shrine.presign_endpoint`. Instructions can be found in uppy-s3_multipart
+README.
+
 ## Strategy B (static)
 
 * Basic user experience
@@ -385,6 +393,8 @@ setup] guide.
 [Uppy]: https://uppy.io
 [uppy aws s3]: https://uppy.io/docs/aws-s3/
 [uppy aws-s3 cors]: https://uppy.io/docs/aws-s3/#S3-Bucket-configuration
+[uppy aws s3 multipart]: https://uppy.io/docs/aws-s3/
+[uppy-s3_multipart]: https://github.com/janko-m/uppy-s3_multipart
 [Amazon S3 Data Consistency Model]: http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyMode
 [CORS guide]: http://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html
 [CORS API]: https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#put_bucket_cors-instance_method
@@ -393,3 +403,4 @@ setup] guide.
 [Minio]: https://minio.io
 [minio setup]: https://shrinerb.com/rdoc/files/doc/testing_md.html#label-Minio
 [metadata direct uploads]: https://github.com/shrinerb/shrine/blob/master/doc/metadata.md#direct-uploads
+[content_disposition]: https://github.com/shrinerb/content_disposition

data/doc/metadata.md

@@ -26,6 +26,13 @@ uploader.extract_metadata(io) #=>
 # }
 ```
 
+By default these values are determined from the following attributes on the IO
+object:
+
+* `filename` – `io.original_filename` or `io.path`
+* `mime_type` – `io.content_type`
+* `size` – `io.size`
+
 Note that you can also manually add or override metadata on upload by passing
 the `:metadata` option to `Shrine#upload`:
 
@@ -181,21 +188,15 @@ least partially) retrieving file content from the storage, which could
 potentially be expensive depending on the storage and the type of metadata
 being extracted.
 
-If you're just attaching files uploaded directly to local disk, there wouldn't
-be any additional performance penalty for extracting metadata. However, if
-you're attaching files uploaded directly to a cloud service like S3, retrieving
-file content for metadata extraction would require an HTTP download. If you're
-using just the `determine_mime_type` plugin, only a small portion of the file
-will be downloaded, so the performance impact might not be so big. But in other
-cases you might have to download the whole file.
-
 There are two ways of extracting metadata from directly uploaded files. If you
 want metadata to be automatically extracted on assignment (which is useful if
-you want to validate the extracted metadata or have it immediately available),
-you can load the `restore_cached_data` plugin:
+you want to validate the extracted metadata or have it immediately available
+for any other reason), you can load the `restore_cached_data` plugin:
 
 ```rb
-Shrine.plugin :restore_cached_data # automatically extract metadata from cached files on assingment
+class ImageUploader < Shrine
+  plugin :restore_cached_data # automatically extract metadata from cached files on assignment
+end
 ```
 ```rb
 photo.image = '{"id":"ks9elsd.jpg","storage":"cache","metadata":{}}' # metadata is extracted
@@ -212,15 +213,13 @@ during background promotion using the `refresh_metadata` plugin (which the
 `restore_cached_data` plugin uses internally):
 
 ```rb
-Shrine.plugin :refresh_metadata
-```
-```rb
-class MyUploader < Shrine
+class ImageUploader < Shrine
+  plugin :refresh_metadata
   plugin :processing
 
   # this will be called in the background if using backgrounding plugin
   process(:store) do |io, context|
-    io.refresh_metadata! # extracts metadata and updates `io.metadata`
+    io.refresh_metadata!(context) # extracts metadata and updates `io.metadata`
     io
   end
 end
@@ -232,17 +231,24 @@ two approaches. For example, if you're attaching video files, you might want to
 extract MIME type upfront and video-specific metadata in a background job, which
 can be done as follows (provided that `backgrounding` plugin is used):
 
-```rb
-Shrine.plugin :restore_cached_data
-```
 ```rb
 class MyUploader < Shrine
   plugin :determine_mime_type # this will be called in the foreground
+  plugin :restore_cached_data
+  plugin :refresh_metadata
+  plugin :add_metadata
   plugin :processing
 
   # this will be called in the background if using backgrounding plugin
   process(:store) do |io, context|
-    additional_metadata = io.download do |file|
+    io.refresh_metadata!(context)
+    io
+  end
+
+  add_metadata do |io, context|
+    next unless context[:action] == :store # this will be the case during promotion
+
+    Shrine.with_file(io) do |file|
       # example of metadata extraction
       movie = FFMPEG::Movie.new(file.path) # uses the streamio-ffmpeg gem
 
@@ -251,10 +257,49 @@ class MyUploader < Shrine
         "resolution" => movie.resolution,
         "frame_rate" => movie.frame_rate }
     end
+  end
+end
+```
 
-    io.metadata.merge!(additional_metadata)
+If you want to do both metadata extraction and file processing during
+promotion, you can wrap both in an `UploadedFile#open` block to make
+sure the file content is retrieved from the storage only once.
 
-    io
+```rb
+class MyUploader < Shrine
+  plugin :refresh_metadata
+  plugin :processing
+
+  process(:store) do |io, context|
+    io.open do |io, context|
+      io.refresh_metadata!(context)
+
+      original = io.download # reuses already open uploaded file
+      # ... processing ...
+    end
+  end
+end
+```
+
+If you're dealing with large files, it's recommended to also use the `tempfile`
+plugin to make sure the same copy of the uploaded file is used for metadata
+extraction (`Shrine.with_file`) and processing (`UploadedFile#tempfile`).
+
+```rb
+Shrine.plugin :tempfile # load it globally so that it overrides `Shrine.with_file`
+```
+```rb
+class MyUploader < Shrine
+  plugin :refresh_metadata
+  plugin :processing
+
+  process(:store) do |io, context|
+    io.open do |io, context|
+      io.refresh_metadata!(context)
+
+      original = io.tempfile # used the cached tempfile
+      # ... processing ...
+    end
   end
 end
 ```