shrine 1.0.0 → 1.1.0

data/doc/securing_uploads.md

@@ -0,0 +1,172 @@
+# Securing uploads
+
+Shrine does a lot to make your file uploads secure, but there are still a lot
+of security measures that could be added by the user on the application's side.
+This guide will try to cover all the well-known security issues, ranging from
+the obvious ones to not-so-obvious ones, and try to provide solutions.
+
+## Validate file type
+
+Almost always you will be accepting certain types of files, and it's a good
+idea to create a whitelist (or blaclist) of supported extensions and MIME
+types.
+
+By default Shrine stores the MIME type derived from the extension, which means
+it's not guaranteed to hold the actual MIME type of the the file. However, you
+can load the `determine_mime_type` plugin which by default uses the [file]
+utility to determine the MIME type from magic file headers.
+
+```rb
+class MyUploader < Shrine
+  plugin :validation_helpers
+  plugin :determine_mime_type
+
+  Attacher.validate do
+    validate_extension_inclusion [/jpe?g/, "png", "gif"]
+    validate_mime_type_inclusion ["image/jpeg", "image/png", "image/gif"]
+  end
+end
+```
+
+## Limit filesize
+
+It's a good idea to generally limit the filesize of uploaded files, so that
+attackers cannot easily flood your storage. There are various layers at which
+you can apply filesize limits, depending on how you're accepting uploads.
+Firstly, you should probably add a filesize validation to prevent large files
+from being uploaded to `:store`:
+
+```rb
+class MyUploader < Shrine
+  plugin :validation_helpers
+
+  Attacher.validate do
+    validate_max_size 20*1024*1024 # 20 MB
+  end
+end
+```
+
+In the following sections we talk about various strategies to prevent files from
+being uploaded to cache and the temporary directory.
+
+### Direct uploads
+
+If you're doing direct uploads with the `direct_upload` plugin, you can pass
+in the `:max_size` option, which will refuse too large files and automatically
+delete it from temporary storage.
+
+```rb
+plugin :direct_upload, max_size: 20*1024*1024 # 20 MB
+```
+
+This option doesn't apply to presigned uploads, if you're using S3 you can
+limit the filesize on presigning:
+
+```rb
+plugin :direct_upload, presign: ->(request) do
+  {content_length_range: 0..20*1024*1024}
+end
+```
+
+### Regular uploads
+
+If you're simply accepting uploads synchronously in the form, you can prevent
+large files from getting into cache by loading the `remove_invalid` plugin:
+
+```rb
+plugin :remove_invalid
+```
+
+### Limiting at application level
+
+If your application is accepting file uploads directly (either through direct
+uploads or regular ones), you can limit the maximum request body size in your
+application server (nginx or apache):
+
+```sh
+# nginx.conf
+
+http {
+  # ...
+  server {
+    # ...
+    client_max_body_size 20M;
+  }
+}
+```
+
+### Paranoid limiting
+
+If you want to make sure that no large files ever get to your storages, and
+you don't really care about the error message, you can use the `hooks` plugin
+and raise an error:
+
+```rb
+class MyUploader
+  plugin :hooks
+
+  def before_upload(io, context)
+    if io.respond_to?(:read)
+      raise FileTooLarge if io.size >= 20*1024*1024
+    end
+  end
+end
+```
+
+## Limit image dimensions
+
+It's possible to create so-called [image bombs], which are images that have a
+small filesize but very large dimensions. These are dangerous if you're doing
+image processing, since processing them can take a lot of time and memory. This
+makes it trivial to DoS the application which doesn't have any protection
+against them.
+
+Shrine uses the [fastimage] gem for determining image dimensions which has
+built-in protection against image bombs (ImageMagick for example doesn't), but
+you still need to prevent those files from being attached and processed:
+
+```rb
+class MyUploader < Shrine
+  plugin :store_dimensions
+  plugin :validation_helpers
+
+  Attacher.validate do
+    validate_max_width  2500
+    validate_max_height 2500
+  end
+end
+```
+
+If you're doing processing on caching, you can use the fastimage gem directly
+in a conditional.
+
+## Limit number of files
+
+When doing direct uploads, it's a good idea to apply some kind of throttling to
+the endpoint, to ensure the attacker cannot upload an unlimited number files,
+because even with a filesize limit it would allow flooding the storage. A good
+library for throttling requests is [rack-attack].
+
+Also, it's generally a good idea to limit the *minimum* filesize as well as
+maximum, to prevent uploading large amounts of small files:
+
+```rb
+class MyUploader < Shrine
+  plugin :validation_helpers
+
+  Attacher.validate do
+    validate_min_size 10*1024 # 10 KB
+  end
+end
+```
+
+## References
+
+* [Nvisium: Secure File Uploads](https://nvisium.com/blog/2015/10/13/secure-file-uploads/)
+* [OWASP: Unrestricted File Upload](https://www.owasp.org/index.php/Unrestricted_File_Upload)
+* [AppSec: 8 Basic Rules to Implement Secure File Uploads](https://software-security.sans.org/blog/2009/12/28/8-basic-rules-to-implement-secure-file-uploads/)
+
+[image bombs]: https://www.bamsoftware.com/hacks/deflate.html
+[fastimage]: https://github.com/sdsykes/fastimage
+[file]: http://linux.die.net/man/1/file
+[rack-attack]: https://github.com/kickstarter/rack-attack

data/lib/shrine.rb

@@ -207,7 +207,8 @@ class Shrine
 
         # The main method for uploading files.  Takes in an IO object and an
         # optional context (used internally by Shrine::Attacher).  It calls
-        # user-defined #process, and aferwards it calls #store.
+        # user-defined #process, and aferwards it calls #store.  The `io` is
+        # closed after upload.
         def upload(io, context = {})
           io = processed(io, context) || io
           store(io, context)
@@ -309,15 +310,15 @@ class Shrine
         # the metadata, stores the file, and returns a Shrine::UploadedFile.
         def _store(io, context)
           _enforce_io(io)
-          location = context[:location] || generate_location(io, context)
-          metadata = extract_metadata(io, context)
+          context[:location] ||= get_location(io, context)
+          context[:metadata] ||= extract_metadata(io, context)
 
-          put(io, context.merge(location: location, metadata: metadata))
+          put(io, context)
 
           self.class::UploadedFile.new(
-            "id"       => location,
+            "id"       => context[:location],
             "storage"  => storage_key.to_s,
-            "metadata" => metadata,
+            "metadata" => context[:metadata],
           )
         end
 
@@ -334,12 +335,7 @@ class Shrine
         # Does the actual uploading, calling `#upload` on the storage.
         def copy(io, context)
           storage.upload(io, context[:location], context[:metadata])
-        end
-
-        # Some storages support moving, so we provide this method for plugins
-        # to use, but by default the file will be copied.
-        def move(io, context)
-          storage.move(io, context[:location], context[:metadata])
+          io.close rescue nil
         end
 
         # Does the actual deletion, calls `UploadedFile#delete`.
@@ -352,6 +348,12 @@ class Shrine
           process(io, context)
         end
 
+        # Retrieves the location for the given io and context. First it looks
+        # for the `:location` option, otherwise it calls #generate_location.
+        def get_location(io, context)
+          generate_location(io, context)
+        end
+
         # Checks if the object is a valid IO by checking that it responds to
         # `#read`, `#eof?`, `#rewind`, `#size` and `#close`, otherwise raises
         # Shrine::InvalidFile.
@@ -598,7 +600,9 @@ class Shrine
 
         # Delegates to `Shrine#default_url`.
         def default_url(**options)
-          store.default_url(options.merge(context))
+          url = store.default_url(options.merge(context))
+          warn "Overriding Shrine#default_url is deprecated and will be removed in Shrine 2. You should use the default_url plugin." if url
+          url
         end
 
         # The validation block provided by `Shrine.validate`.
@@ -630,7 +634,7 @@ class Shrine
         # The context that's sent to Shrine on upload and delete. It holds the
         # record and the name of the attachment.
         def context
-          @context ||= {name: name, record: record}.freeze
+          @context ||= {name: name, record: record}
         end
       end
 
@@ -676,8 +680,7 @@ class Shrine
 
         # The extension derived from `#original_filename`.
         def extension
-          extname = File.extname(id)
-          extname[1..-1] unless extname.empty?
+          File.extname(id)[1..-1] || File.extname(original_filename.to_s)[1..-1]
         end
 
         # The filesize of the original file.
@@ -689,6 +692,7 @@ class Shrine
         def mime_type
           metadata.fetch("mime_type")
         end
+        alias content_type mime_type
 
         # Part of Shrine::UploadedFile's complying to the IO interface.  It
         # delegates to the internally downloaded file.
@@ -706,6 +710,7 @@ class Shrine
         # delegates to the internally downloaded file.
         def close
           io.close
+          io.delete if io.class.name == "Tempfile"
         end
 
         # Part of Shrine::UploadedFile's complying to the IO interface.  It

data/lib/shrine/plugins/activerecord.rb

@@ -26,7 +26,7 @@ class Shrine
     # you should first disable these transactions for those tests.
     #
     # If you want to put some parts of this lifecycle into a background job, see
-    # the background_helpers plugin.
+    # the backgrounding plugin.
     #
     # Additionally, any Shrine validation errors will added to ActiveRecord's
     # errors upon validation. Note that if you want to validate presence of the
@@ -64,7 +64,7 @@ class Shrine
       end
 
       module AttacherClassMethods
-        # Needed by the background_helpers plugin.
+        # Needed by the backgrounding plugin.
         def find_record(record_class, record_id)
           record_class.find(record_id)
         end

data/lib/shrine/plugins/background_helpers.rb

@@ -1,148 +1,2 @@
-class Shrine
-  module Plugins
-    # The background_helpers plugin enables you to intercept phases of
-    # uploading and put them into background jobs. This doesn't require any
-    # additional columns.
-    #
-    #     plugin :background_helpers
-    #
-    # ## Promoting
-    #
-    # If you're doing processing, or your `:store` is something other than
-    # Storage::FileSystem, it's recommended to put promoting (moving to store)
-    # into a background job. This plugin allows you to do that by calling
-    # `Shrine::Attacher.promote`:
-    #
-    #     Shrine::Attacher.promote { |data| UploadJob.perform_async(data) }
-    #
-    # When you call `Shrine::Attacher.promote` with a block, it will save the
-    # block and call it on every promotion. Then in your background job you can
-    # again call `Shrine::Attacher.promote` with the data, and internally it
-    # will resolve all necessary objects, do the promoting and update the
-    # record.
-    #
-    #     class UploadJob
-    #       include Sidekiq::Worker
-    #
-    #       def perform(data)
-    #         Shrine::Attacher.promote(data)
-    #       end
-    #     end
-    #
-    # Shrine automatically handles all concurrency issues, such as canceling
-    # promoting if the attachment has changed in the meanwhile.
-    #
-    # ## Deleting
-    #
-    # If your `:store` is something other than Storage::FileSystem, it's
-    # recommended to put deleting files into a background job. This plugin
-    # allows you to do that by calling `Shrine::Attacher.delete`:
-    #
-    #     Shrine::Attacher.delete { |data| DeleteJob.perform_async(data) }
-    #
-    # When you call `Shrine::Attacher.delete` with a block, it will save the
-    # block and call it on every delete. Then in your background job you can
-    # again call `Shrine::Attacher.delete` with the data, and internally it
-    # will resolve all necessary objects, and delete the file.
-    #
-    #     class DeleteJob
-    #       include Sidekiq::Worker
-    #
-    #       def perform(data)
-    #         Shrine::Attacher.delete(data)
-    #       end
-    #     end
-    #
-    # ## Conclusion
-    #
-    # The examples above used Sidekiq, but obviously you can just as well use
-    # any other backgrounding library. Also, if you want you can use
-    # backgrounding just for certain uploaders:
-    #
-    #     class ImageUploader < Shrine
-    #       Attacher.promote { |data| UploadJob.perform_async(data) }
-    #       Attacher.delete { |data| DeleteJob.perform_async(data) }
-    #     end
-    #
-    # If you would like to speed up your uploads and deletes, you can use the
-    # parallelize plugin, either as a replacement or an addition to
-    # background_helpers.
-    module BackgroundHelpers
-      module AttacherClassMethods
-        # If block is passed in, stores it to be called on promotion. Otherwise
-        # resolves data into objects and calls Attacher#promote.
-        def promote(data = nil, &block)
-          if block
-            shrine_class.opts[:background_promote] = block
-          else
-            record_class, record_id = data["record"]
-            record_class = Object.const_get(record_class)
-            record = find_record(record_class, record_id)
-
-            name = data["attachment"]
-            attacher = record.send("#{name}_attacher")
-            cached_file = attacher.uploaded_file(data["uploaded_file"])
-
-            attacher.promote(cached_file)
-          end
-        end
-
-        # If block is passed in, stores it to be called on deletion. Otherwise
-        # resolves data into objects and calls `Shrine#delete`.
-        def delete(data = nil, &block)
-          if block
-            shrine_class.opts[:background_delete] = block
-          else
-            record_class, record_id = data["record"]
-            record = Object.const_get(record_class).new
-            record.id = record_id
-
-            name, phase = data["attachment"], data["phase"]
-            attacher = record.send("#{name}_attacher")
-            uploaded_file = attacher.uploaded_file(data["uploaded_file"])
-            context = {name: name.to_sym, record: record, phase: phase.to_sym}
-
-            attacher.store.delete(uploaded_file, context)
-          end
-        end
-      end
-
-      module AttacherMethods
-        # Calls the promoting block with the data if it's been registered.
-        def _promote
-          if background_promote = shrine_class.opts[:background_promote]
-            data = {
-              "uploaded_file" => get.to_json,
-              "record"        => [record.class.to_s, record.id],
-              "attachment"    => name,
-            }
-
-            instance_exec(data, &background_promote) if promote?(get)
-          else
-            super
-          end
-        end
-
-        private
-
-        # Calls the deleting block with the data if it's been registered.
-        def delete!(uploaded_file, phase:)
-          if background_delete = shrine_class.opts[:background_delete]
-            data = {
-              "uploaded_file" => uploaded_file.to_json,
-              "record"        => [record.class.to_s, record.id],
-              "attachment"    => name,
-              "phase"         => phase,
-            }
-
-            instance_exec(data, &background_delete)
-          else
-            super(uploaded_file, phase: phase)
-          end
-        end
-      end
-    end
-
-    register_plugin(:background_helpers, BackgroundHelpers)
-  end
-end
+require "shrine/plugins/backgrounding"
+Shrine::Plugins.register_plugin(:background_helpers, Shrine::Plugins::Backgrounding)

data/lib/shrine/plugins/backgrounding.rb

@@ -0,0 +1,148 @@
+class Shrine
+  module Plugins
+    # The background_helpers plugin enables you to intercept phases of
+    # uploading and put them into background jobs. This doesn't require any
+    # additional columns.
+    #
+    #     plugin :backgrounding
+    #
+    # ## Promoting
+    #
+    # If you're doing processing, or your `:store` is something other than
+    # Storage::FileSystem, it's recommended to put promoting (moving to store)
+    # into a background job. This plugin allows you to do that by calling
+    # `Shrine::Attacher.promote`:
+    #
+    #     Shrine::Attacher.promote { |data| UploadJob.perform_async(data) }
+    #
+    # When you call `Shrine::Attacher.promote` with a block, it will save the
+    # block and call it on every promotion. Then in your background job you can
+    # again call `Shrine::Attacher.promote` with the data, and internally it
+    # will resolve all necessary objects, do the promoting and update the
+    # record.
+    #
+    #     class UploadJob
+    #       include Sidekiq::Worker
+    #
+    #       def perform(data)
+    #         Shrine::Attacher.promote(data)
+    #       end
+    #     end
+    #
+    # Shrine automatically handles all concurrency issues, such as canceling
+    # promoting if the attachment has changed in the meanwhile.
+    #
+    # ## Deleting
+    #
+    # If your `:store` is something other than Storage::FileSystem, it's
+    # recommended to put deleting files into a background job. This plugin
+    # allows you to do that by calling `Shrine::Attacher.delete`:
+    #
+    #     Shrine::Attacher.delete { |data| DeleteJob.perform_async(data) }
+    #
+    # When you call `Shrine::Attacher.delete` with a block, it will save the
+    # block and call it on every delete. Then in your background job you can
+    # again call `Shrine::Attacher.delete` with the data, and internally it
+    # will resolve all necessary objects, and delete the file.
+    #
+    #     class DeleteJob
+    #       include Sidekiq::Worker
+    #
+    #       def perform(data)
+    #         Shrine::Attacher.delete(data)
+    #       end
+    #     end
+    #
+    # ## Conclusion
+    #
+    # The examples above used Sidekiq, but obviously you can just as well use
+    # any other backgrounding library. Also, if you want you can use
+    # backgrounding just for certain uploaders:
+    #
+    #     class ImageUploader < Shrine
+    #       Attacher.promote { |data| UploadJob.perform_async(data) }
+    #       Attacher.delete { |data| DeleteJob.perform_async(data) }
+    #     end
+    #
+    # If you would like to speed up your uploads and deletes, you can use the
+    # parallelize plugin, either as a replacement or an addition to
+    # background_helpers.
+    module Backgrounding
+      module AttacherClassMethods
+        # If block is passed in, stores it to be called on promotion. Otherwise
+        # resolves data into objects and calls `Attacher#promote`.
+        def promote(data = nil, &block)
+          if block
+            shrine_class.opts[:backgrounding_promote] = block
+          else
+            record_class, record_id = data["record"]
+            record_class = Object.const_get(record_class)
+            record = find_record(record_class, record_id)
+
+            name = data["attachment"]
+            attacher = record.send("#{name}_attacher")
+            cached_file = attacher.uploaded_file(data["uploaded_file"])
+
+            attacher.promote(cached_file)
+          end
+        end
+
+        # If block is passed in, stores it to be called on deletion. Otherwise
+        # resolves data into objects and calls `Shrine#delete`.
+        def delete(data = nil, &block)
+          if block
+            shrine_class.opts[:backgrounding_delete] = block
+          else
+            record_class, record_id = data["record"]
+            record = Object.const_get(record_class).new
+            record.id = record_id
+
+            name, phase = data["attachment"], data["phase"]
+            attacher = record.send("#{name}_attacher")
+            uploaded_file = attacher.uploaded_file(data["uploaded_file"])
+            context = {name: name.to_sym, record: record, phase: phase.to_sym}
+
+            attacher.store.delete(uploaded_file, context)
+          end
+        end
+      end
+
+      module AttacherMethods
+        # Calls the promoting block with the data if it's been registered.
+        def _promote
+          if background_promote = shrine_class.opts[:backgrounding_promote]
+            data = {
+              "uploaded_file" => get.to_json,
+              "record"        => [record.class.to_s, record.id],
+              "attachment"    => name.to_s,
+            }
+
+            instance_exec(data, &background_promote) if promote?(get)
+          else
+            super
+          end
+        end
+
+        private
+
+        # Calls the deleting block with the data if it's been registered.
+        def delete!(uploaded_file, phase:)
+          if background_delete = shrine_class.opts[:backgrounding_delete]
+            data = {
+              "uploaded_file" => uploaded_file.to_json,
+              "record"        => [record.class.to_s, record.id],
+              "attachment"    => name.to_s,
+              "phase"         => phase.to_s,
+            }
+
+            instance_exec(data, &background_delete)
+          else
+            super(uploaded_file, phase: phase)
+          end
+        end
+      end
+    end
+
+    register_plugin(:backgrounding, Backgrounding)
+  end
+end