shrine 2.5.0 → 2.6.0

data/doc/securing_uploads.md

@@ -8,8 +8,7 @@ the obvious ones to not-so-obvious ones, and try to provide solutions.
 ## Validate file type
 
 Almost always you will be accepting certain types of files, and it's a good
-idea to create a whitelist (or blaclist) of supported extensions and MIME
-types.
+idea to create a whitelist (or a blacklist) of extensions and MIME types.
 
 By default Shrine stores the MIME type derived from the extension, which means
 it's not guaranteed to hold the actual MIME type of the the file. However, you
@@ -22,8 +21,8 @@ class MyUploader < Shrine
   plugin :determine_mime_type
 
   Attacher.validate do
-    validate_extension_inclusion [/jpe?g/, "png", "gif"]
-    validate_mime_type_inclusion ["image/jpeg", "image/png", "image/gif"]
+    validate_extension_inclusion %w[jpg jpeg png gif]
+    validate_mime_type_inclusion %w[image/jpeg image/png image/gif]
   end
 end
 ```
@@ -49,7 +48,7 @@ end
 In the following sections we talk about various strategies to prevent files from
 being uploaded to cache and the temporary directory.
 
-### Direct uploads
+### Limiting filesize in direct uploads
 
 If you're doing direct uploads with the `direct_upload` plugin, you can pass
 in the `:max_size` option, which will refuse too large files and automatically
@@ -68,34 +67,32 @@ plugin :direct_upload, presign: ->(request) do
 end
 ```
 
-### Regular uploads
+### Limiting filesize at application level
 
-If you're simply accepting uploads synchronously in the form, you can prevent
-large files from getting into cache by loading the `remove_invalid` plugin:
+If your application is accepting file uploads, it's good practice to limit the
+maximum allowed `Content-Length` before calling `params` for the first time,
+to avoid Rack parsing the multipart request parameters and creating a Tempfile
+for uploads that are obviously attempts of attacks.
 
 ```rb
-plugin :remove_invalid
-```
-
-### Limiting at application level
+if request.content_length >= 100*1024*1024 # 100MB
+  response.status = 413 # Request Entity Too Large
+  response.body = "The uploaded file was too large (maximum is 100MB)"
+  request.halt
+end
 
-If your application is accepting file uploads directly (either through direct
-uploads or regular ones), you can limit the maximum request body size in your
-application server (nginx or apache):
+request.params # Rack parses the multipart request params
+```
 
-```sh
-# nginx.conf
+Alternatively you can allow uploads of any size to temporary Shrine storage,
+but tell Shrine to immediately delete the file if it failed validations by
+loading the `remove_invalid` plugin.
 
-http {
-  # ...
-  server {
-    # ...
-    client_max_body_size 20M;
-  }
-}
+```rb
+plugin :remove_invalid
 ```
 
-### Paranoid limiting
+### Paranoid filesize limiting
 
 If you want to make sure that no large files ever get to your storages, and
 you don't really care about the error message, you can use the `hooks` plugin
@@ -140,6 +137,25 @@ end
 If you're doing processing on caching, you can use the fastimage gem directly
 in a conditional.
 
+## Prevent metadata tampering
+
+When cached file is retained on validation errors or it was direct uploaded,
+the uploaded file representation is assigned to the attacher. This also
+includes any file metadata. By default Shrine won't attempt to re-extract
+metadata, because for remote storages that requires an additional HTTP request,
+which might not be feasible depending on the application requirements.
+
+However, this means that the attacker can directly upload a malicious file
+(because direct uploads aren't validated), and then modify the metadata hash so
+that it passes Shrine validations, before submitting the cached file to your
+app. To guard yourself from such attacks, you can load the
+`restore_cached_data` plugin, which will automatically re-extract metadata from
+cached files on assignment and override the received metadata.
+
+```rb
+plugin :restore_cached_data
+```
+
 ## Limit number of files
 
 When doing direct uploads, it's a good idea to apply some kind of throttling to

data/doc/testing.md

@@ -80,20 +80,11 @@ factory :photo do
 end
 ```
 
-On the other hand, if you're setting up test data using YAML fixtures, you
-aren't that flexible, because you can only use primitive data types that are
-part of the YAML language. In that case you can load the `data_uri` Shrine
-plugin, and assign files in form of data URI strings through the
-`<attachment>_data_uri` accessor provided by the plugin.
-
-```rb
-Shrine.plugin :data_uri
-```
-```yml
-# test/fixtures/photos.yml
-photo:
-  image_data_uri: "data:image/png;base64,<%= Base64.encode64(File.binread("test/files/image.png")) %>"
-```
+On the other hand, if you're setting up test data using Rails' YAML fixtures,
+you unfortunately won't be able to use them for assigning files. This is
+because Rails fixtures only allow assigning primitive data types, and don't
+allow you to specify Shrine attributes, you can only assign to columns
+directly.
 
 ## Background jobs
 
@@ -222,7 +213,7 @@ return a hash of unprocessed original files:
 class ImageUploader
   def process(io, context)
     if context[:action] == :store
-      {small: io, medium: io, large: io}
+      {small: io.download, medium: io.download, large: io.download}
     end
   end
 end

data/lib/shrine.rb

@@ -2,6 +2,7 @@ require "shrine/version"
 
 require "securerandom"
 require "json"
+require "tempfile"
 
 class Shrine
   # A generic exception used by Shrine.
@@ -137,7 +138,7 @@ class Shrine
           self::Attacher.include(plugin::AttacherMethods) if defined?(plugin::AttacherMethods)
           self::Attacher.extend(plugin::AttacherClassMethods) if defined?(plugin::AttacherClassMethods)
           plugin.configure(self, *args, &block) if plugin.respond_to?(:configure)
-          nil
+          plugin
         end
 
         # Retrieves the storage under the given identifier (can be a Symbol or
@@ -151,7 +152,7 @@ class Shrine
         # model class. Example:
         #
         #     class Photo
-        #       include Shrine[:image] # creates a Shrine::Attachment object
+        #       include Shrine.attachment(:image) # creates a Shrine::Attachment object
         #     end
         def attachment(name)
           self::Attachment.new(name)
@@ -166,7 +167,7 @@ class Shrine
         def uploaded_file(object, &block)
           case object
           when String
-            warn "Giving a string to Shrine.uploaded_file is deprecated and won't be possible in Shrine 3. Use Attacher#uploaded_file instead."
+            deprecation("Giving a string to Shrine.uploaded_file is deprecated and won't be possible in Shrine 3. Use Attacher#uploaded_file instead.")
             uploaded_file(JSON.parse(object), &block)
           when Hash
             uploaded_file(self::UploadedFile.new(object), &block)
@@ -176,6 +177,11 @@ class Shrine
             raise Error, "cannot convert #{object.inspect} to a #{self}::UploadedFile"
           end
         end
+
+        # Prints a deprecation warning to standard error.
+        def deprecation(message)
+          warn "SHRINE DEPRECATION WARNING: #{message}"
+        end
       end
 
       module InstanceMethods
@@ -340,7 +346,8 @@ class Shrine
         # Retrieves the location for the given IO and context. First it looks
         # for the `:location` option, otherwise it calls #generate_location.
         def get_location(io, context)
-          context[:location] || generate_location(io, context)
+          location = context[:location] || generate_location(io, context)
+          location or raise Error, "location generated for #{io.inspect} was nil (context = #{context})"
         end
 
         # If the IO object is a Shrine::UploadedFile, it simply copies over its
@@ -492,7 +499,7 @@ class Shrine
         # is then written to the attachment attribute in the JSON format.
         def assign(value)
           if value.is_a?(String)
-            return if value == "" || value == read || !cache.uploaded?(uploaded_file(value))
+            return if value == "" || !cache.uploaded?(uploaded_file(value))
             assign_cached(uploaded_file(value))
           else
             uploaded_file = cache!(value, action: :cache) if value
@@ -504,7 +511,7 @@ class Shrine
         # attribute. It then runs file validations, and records that the
         # attachment has changed.
         def set(uploaded_file)
-          @old = get
+          @old = get unless uploaded_file == get
           _set(uploaded_file)
           validate
         end
@@ -516,9 +523,10 @@ class Shrine
         end
 
         # Returns true if a new file has been attached.
-        def attached?
+        def changed?
           instance_variable_defined?(:@old)
         end
+        alias attached? changed?
 
         # Plugins can override this if they want something to be done before
         # save.
@@ -528,6 +536,7 @@ class Shrine
         # Deletes the old file and promotes the new one. Typically this should
         # be called after saving the model instance.
         def finalize
+          return if !instance_variable_defined?(:@old)
           replace
           remove_instance_variable(:@old)
           _promote(action: :store) if cached?
@@ -601,19 +610,19 @@ class Shrine
 
         # Uploads the file using the #cache uploader, passing the #context.
         def cache!(io, **options)
-          warn "Sending :phase to Shrine::Attacher#cache! is deprecated and will not be supported in Shrine 3. Use :action instead." if options[:phase]
+          Shrine.deprecation("Sending :phase to Attacher#cache! is deprecated and will not be supported in Shrine 3. Use :action instead.") if options[:phase]
           cache.upload(io, context.merge(_equalize_phase_and_action(options)))
         end
 
         # Uploads the file using the #store uploader, passing the #context.
         def store!(io, **options)
-          warn "Sending :phase to Shrine::Attacher#store! is deprecated and will not be supported in Shrine 3. Use :action instead." if options[:phase]
+          Shrine.deprecation("Sending :phase to Attacher#store! is deprecated and will not be supported in Shrine 3. Use :action instead.") if options[:phase]
           store.upload(io, context.merge(_equalize_phase_and_action(options)))
         end
 
         # Deletes the file using the uploader, passing the #context.
         def delete!(uploaded_file, **options)
-          warn "Sending :phase to Shrine::Attacher#delete! is deprecated and will not be supported in Shrine 3. Use :action instead." if options[:phase]
+          Shrine.deprecation("Sending :phase to Attacher#delete! is deprecated and will not be supported in Shrine 3. Use :action instead.") if options[:phase]
           store.delete(uploaded_file, context.merge(_equalize_phase_and_action(options)))
         end
 

data/lib/shrine/plugins/activerecord.rb

@@ -29,7 +29,7 @@ class Shrine
     # saves again with a stored attachment, you can detect this in callbacks:
     #
     #     class User < ActiveRecord::Base
-    #       include ImageUploader[:avatar]
+    #       include ImageUploader::Attachment.new(:avatar)
     #
     #       before_save do
     #         if avatar_data_changed? && avatar_attacher.cached?
@@ -53,7 +53,7 @@ class Shrine
     # of the attachment, you can do it directly on the model.
     #
     #     class User < ActiveRecord::Base
-    #       include ImageUploader[:avatar]
+    #       include ImageUploader::Attachment.new(:avatar)
     #       validates_presence_of :avatar
     #     end
     #
@@ -85,11 +85,11 @@ class Shrine
 
           model.class_eval <<-RUBY, __FILE__, __LINE__ + 1 if opts[:activerecord_callbacks]
             before_save do
-              #{@name}_attacher.save if #{@name}_attacher.attached?
+              #{@name}_attacher.save if #{@name}_attacher.changed?
             end
 
             after_commit on: [:create, :update] do
-              #{@name}_attacher.finalize if #{@name}_attacher.attached?
+              #{@name}_attacher.finalize if #{@name}_attacher.changed?
             end
 
             after_commit on: [:destroy] do

data/lib/shrine/plugins/add_metadata.rb

@@ -17,7 +17,7 @@ class Shrine
     #     document.pages
     #
     # You can also extract multiple metadata values at once, by using
-    # `add_metadata` without an argument.
+    # `add_metadata` without an argument, and returning a hash of metadata.
     #
     #     add_metadata do |io, context|
     #       movie = FFMPEG::Movie.new(io.path)
@@ -73,9 +73,13 @@ class Shrine
           metadata = super
 
           opts[:metadata].each do |metadata_block|
-            custom_metadata = instance_exec(io, context, &metadata_block)
+            custom_metadata = instance_exec(io, context, &metadata_block) || {}
             io.rewind
-            metadata.merge!(custom_metadata) unless custom_metadata.nil?
+            # convert symbol keys to strings
+            custom_metadata.keys.each do |key|
+              custom_metadata[key.to_s] = custom_metadata.delete(key) if key.is_a?(Symbol)
+            end
+            metadata.merge!(custom_metadata)
           end
 
           metadata

data/lib/shrine/plugins/background_helpers.rb

@@ -1,3 +1,3 @@
-warn "The background_helpers Shrine plugin has been renamed to \"backgrounding\". Loading the plugin through \"background_helpers\" will stop working in Shrine 3."
+Shrine.deprecation("The background_helpers plugin has been renamed to \"backgrounding\". Loading the plugin through \"background_helpers\" will stop working in Shrine 3.")
 require "shrine/plugins/backgrounding"
 Shrine::Plugins.register_plugin(:background_helpers, Shrine::Plugins::Backgrounding)

data/lib/shrine/plugins/backgrounding.rb

@@ -5,17 +5,22 @@ class Shrine
     # useful if you're doing processing and/or you're storing files on an
     # external storage service.
     #
-    #     plugin :backgrounding
-    #
-    # ## Usage
-    #
     # The plugin provides `Attacher.promote` and `Attacher.delete` methods,
     # which allow you to hook up to promoting and deleting and spawn background
     # jobs, by passing a block.
     #
+    #     Shrine.plugin :backgrounding
     #     Shrine::Attacher.promote { |data| PromoteJob.perform_async(data) }
     #     Shrine::Attacher.delete { |data| DeleteJob.perform_async(data) }
     #
+    # If you don't want to apply backgrounding for all uploaders, you can
+    # declare the hooks only for specific uploaders.
+    #
+    #     class MyUploader < Shrine
+    #       Attacher.promote { |data| PromoteJob.perform_async(data) }
+    #       Attacher.delete { |data| DeleteJob.perform_async(data) }
+    #     end
+    #
     # The yielded `data` variable is a serializable hash containing all context
     # needed for promotion/deletion. Now you just need to declare the job
     # classes, and inside them call `Attacher.promote` or `Attacher.delete`,
@@ -23,7 +28,6 @@ class Shrine
     #
     #     class PromoteJob
     #       include Sidekiq::Worker
-    #
     #       def perform(data)
     #         Shrine::Attacher.promote(data)
     #       end
@@ -31,7 +35,6 @@ class Shrine
     #
     #     class DeleteJob
     #       include Sidekiq::Worker
-    #
     #       def perform(data)
     #         Shrine::Attacher.delete(data)
     #       end
@@ -45,6 +48,16 @@ class Shrine
     # the foreground before kicking off a background job, you can use the
     # `recache` plugin.
     #
+    # In your application you can use `Attacher#cached?` and `Attacher#stored?`
+    # to differentiate between your background job being in progress and
+    # having completed.
+    #
+    #     if user.avatar_attacher.cached? # background job is still in progress
+    #       # ...
+    #     elsif user.avatar_attacher.stored? # background job has finished
+    #       # ...
+    #     end
+    #
     # ## `Attacher.promote` and `Attacher.delete`
     #
     # In background jobs, `Attacher.promote` and `Attacher.delete` will resolve

data/lib/shrine/plugins/cached_attachment_data.rb

@@ -10,11 +10,11 @@ class Shrine
     # the cached file as JSON, and should be used to set the value of the
     # hidden form field.
     #
-    #     @user.cached_avatar_data #=> '{"storage":"cache","id":"...","metadata":{...}}'
+    #     @user.cached_avatar_data #=> '{"id":"38k25.jpg","storage":"cache","metadata":{...}}'
     #
     # This method delegates to `Attacher#read_cached`:
     #
-    #     attacher.read_cached #=> '{"storage":"cache","id":"...","metadata":{...}}'
+    #     attacher.read_cached #=> '{"id":"38k25.jpg","storage":"cache","metadata":{...}}'
     module CachedAttachmentData
       module AttachmentMethods
         def initialize(*)
@@ -26,7 +26,7 @@ class Shrine
             end
 
             def cached_#{@name}_data=(value)
-              warn "Calling #cached_#{@name}_data= is deprecated and will be removed in Shrine 3. You should use the original field name: `f.hidden_field :#{@name}, value: record.cached_#{@name}_data`."
+              Shrine.deprecation("Calling #cached_#{@name}_data= is deprecated and will be removed in Shrine 3. You should use the original field name: `f.hidden_field :#{@name}, value: record.cached_#{@name}_data`.")
               #{@name}_attacher.assign(value)
             end
           RUBY
@@ -35,7 +35,7 @@ class Shrine
 
       module AttacherMethods
         def read_cached
-          get.to_json if cached? && attached?
+          get.to_json if cached? && changed?
         end
       end
     end

data/lib/shrine/plugins/data_uri.rb

@@ -1,5 +1,8 @@
 require "base64"
+require "strscan"
+require "cgi/util"
 require "stringio"
+require "forwardable"
 
 class Shrine
   module Plugins
@@ -19,42 +22,105 @@ class Shrine
     #     user.avatar.size              #=> 43423
     #
     # You can also use `#data_uri=` and `#data_uri` methods directly on the
-    # `Shrine::Attacher`:
+    # `Shrine::Attacher` (which the model methods just delegate to):
     #
     #     attacher.data_uri = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA"
     #
-    # If you want the uploaded file to have an extension, you can generate a
-    # filename based on the content type of the data URI:
-    #
-    #     plugin :data_uri, filename: ->(content_type) do
-    #       extension = MIME::Types[content_type].first.preferred_extension
-    #       "data_uri.#{extension}"
-    #     end
-    #
     # If the data URI wasn't correctly parsed, an error message will be added to
     # the attachment column. You can change the default error message:
     #
     #     plugin :data_uri, error_message: "data URI was invalid"
     #     plugin :data_uri, error_message: ->(uri) { I18n.t("errors.data_uri_invalid") }
     #
+    # If you just want to parse the data URI and create an IO object from it,
+    # you can do that with `Shrine.data_uri`. If the data URI cannot be parsed,
+    # a `Shrine::Plugins::DataUri::ParseError` will be raised.
+    #
+    #     # or YourUploader.data_uri("...")
+    #     io = Shrine.data_uri("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA")
+    #     io.content_type #=> "image/png"
+    #     io.size         #=> 21
+    #
+    # When the content type is ommited, `text/plain` is assumed. The parser
+    # also supports raw data URIs which aren't base64-encoded.
+    #
+    #     # or YourUploader.data_uri("...")
+    #     io = Shrine.data_uri("data:,raw%20content")
+    #     io.content_type #=> "text/plain"
+    #     io.size         #=> 11
+    #     io.read         #=> "raw content"
+    #
+    # The created IO object won't convey any file extension (because it doesn't
+    # have a filename), but you can generate a filename based on the content
+    # type of the data URI:
+    #
+    #     require "mime/types"
+    #
+    #     plugin :data_uri, filename: ->(content_type) do
+    #       extension = MIME::Types[content_type].first.preferred_extension
+    #       "data_uri.#{extension}"
+    #     end
+    #
     # This plugin also adds a `UploadedFile#data_uri` method (and `#base64`),
     # which returns a base64-encoded data URI of any UploadedFile:
     #
-    #     user.avatar.data_uri #=> "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA"
-    #     user.avatar.base64   #=> "iVBORw0KGgoAAAANSUhEUgAAAAUA"
+    #     uploaded_file.data_uri #=> "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA"
+    #     uploaded_file.base64   #=> "iVBORw0KGgoAAAANSUhEUgAAAAUA"
     #
     # [data URIs]: https://tools.ietf.org/html/rfc2397
     # [HTML5 Canvas]: https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API
     module DataUri
-      DEFAULT_ERROR_MESSAGE = "data URI was invalid"
+      class ParseError < Error; end
+
+      DATA_REGEXP          = /data:/
+      MEDIA_TYPE_REGEXP    = /[-\w.+]+\/[-\w.+]+(;[-\w.+]+=[^;,]+)*/
+      BASE64_REGEXP        = /;base64/
+      CONTENT_SEPARATOR    = /,/
       DEFAULT_CONTENT_TYPE = "text/plain"
-      DATA_URI_REGEXP = /\Adata:([-\w.+]+\/[-\w.+]+)?(;base64)?,(.*)\z/m
 
       def self.configure(uploader, opts = {})
         uploader.opts[:data_uri_filename] = opts.fetch(:filename, uploader.opts[:data_uri_filename])
         uploader.opts[:data_uri_error_message] = opts.fetch(:error_message, uploader.opts[:data_uri_error_message])
       end
 
+      module ClassMethods
+        # Parses the given data URI and creates an IO object from it.
+        #
+        #     Shrine.data_uri("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA")
+        #     #=> #<Shrine::Plugins::DataUri::DataFile>
+        def data_uri(uri)
+          info = parse_data_uri(uri)
+
+          content_type = info[:content_type] || DEFAULT_CONTENT_TYPE
+          content      = info[:base64] ? Base64.decode64(info[:data]) : CGI.unescape(info[:data])
+          filename     = opts[:data_uri_filename]
+          filename     = filename.call(content_type) if filename
+
+          data_file = DataFile.new(content, content_type: content_type, filename: filename)
+          info[:data].clear
+
+          data_file
+        end
+
+        private
+
+        def parse_data_uri(uri)
+          scanner = StringScanner.new(uri)
+          scanner.scan(DATA_REGEXP) or raise ParseError, "data URI has invalid format"
+          media_type = scanner.scan(MEDIA_TYPE_REGEXP)
+          base64 = scanner.scan(BASE64_REGEXP)
+          scanner.scan(CONTENT_SEPARATOR) or raise ParseError, "data URI has invalid format"
+
+          content_type = media_type[/^[^;]+/] if media_type
+
+          {
+            content_type: content_type,
+            base64:       !!base64,
+            data:         scanner.post_match,
+          }
+        end
+      end
+
       module AttachmentMethods
         def initialize(*)
           super
@@ -79,19 +145,13 @@ class Shrine
         def data_uri=(uri)
           return if uri == ""
 
-          if match = uri.match(DATA_URI_REGEXP)
-            content_type = match[1] || DEFAULT_CONTENT_TYPE
-            content      = match[2] ? Base64.decode64(match[3]) : match[3]
-            filename     = shrine_class.opts[:data_uri_filename]
-            filename     = filename.call(content_type) if filename
-
-            assign DataFile.new(content, content_type: content_type, filename: filename)
-          else
-            message = shrine_class.opts[:data_uri_error_message] || DEFAULT_ERROR_MESSAGE
-            message = message.call(uri) if message.respond_to?(:call)
-            errors.replace [message]
-            @data_uri = uri
-          end
+          data_file = shrine_class.data_uri(uri)
+          assign(data_file)
+        rescue ParseError => error
+          message = shrine_class.opts[:data_uri_error_message] || error.message
+          message = message.call(uri) if message.respond_to?(:call)
+          errors.replace [message]
+          @data_uri = uri
         end
 
         # Form builders require the reader as well.
@@ -108,18 +168,32 @@ class Shrine
 
         # Returns contents of the file base64-encoded.
         def base64
-          content = open { |io| io.read }
-          Base64.encode64(content).chomp
+          binary = open { |io| io.read }
+          result = Base64.encode64(binary).chomp
+          binary.clear # deallocate string
+          result
         end
       end
 
-      class DataFile < StringIO
+      class DataFile
         attr_reader :content_type, :original_filename
 
         def initialize(content, content_type: nil, filename: nil)
-          @content_type = content_type
+          @content_type      = content_type
           @original_filename = filename
-          super(content)
+          @io                = StringIO.new(content)
+        end
+
+        def to_io
+          @io
+        end
+
+        extend Forwardable
+        delegate Shrine::IO_METHODS.keys => :@io
+
+        def close
+          @io.close
+          @io.string.clear # deallocate string
         end
       end
     end