paperclip 6.0.0 → 6.1.0

data/NEWS

@@ -1,3 +1,20 @@
+6.1.0 (2018-07-27):
+
+* BUGFIX: Don't double-encode URLs (Roderick Monje).
+* BUGFIX: Only use the content_type when it exists (Jean-Philippe Doyle).
+* STABILITY: Better handling of the content-disposition header. Now supports
+  file name that is either enclosed or not in double quotes and is case
+  insensitive as per RC6266 grammar (Hasan Kumar, Yves Riel).
+* STABILITY: Change database column type of attachment file size from unsigned 4-byte
+  `integer` to unsigned 8-byte `bigint`. The former type limits attachment size
+  to just over 2GB, which can easily be exceeded by a large video file (Laurent
+  Arnoud, Alen Zamanyan).
+* STABILITY: Better error message when thumbnail processing errors (Hayden Ball).
+* STABILITY: Fix file linking issues around Windows (Akihiko Odaki).
+* STABILITY: Files without an extension will now be checked for spoofing attempts
+  (George Walters II).
+* STABILITY: Manually close Tempfiles when we are done with them (Erkki Eilonen).
+
 6.0.0 (2018-03-09):
 
 * Improvement: Depend only on `aws-sdk-s3` instead of `aws-sdk` (https://github.com/thoughtbot/paperclip/pull/2481)

data/README.md

@@ -1,6 +1,28 @@
 Paperclip
 =========
 
+# Deprecated
+
+**[Paperclip is deprecated]**.
+
+For new projects, we recommend Rails' own [ActiveStorage].
+
+For existing projects, please consult and contribute to [the migration guide] ([en español]).
+
+
+We will leave the Issues open as a discussion forum _only_. We do _not_
+guarantee a response from us in the Issues.
+
+We are no longer accepting pull requests _except_ pull requests against the
+migration guide. All other pull requests will be closed without merging.
+
+[Paperclip is deprecated]: https://robots.thoughtbot.com/closing-the-trombone
+[ActiveStorage]: http://guides.rubyonrails.org/active_storage_overview.html
+[the migration guide]: https://github.com/thoughtbot/paperclip/blob/master/MIGRATING.md
+[en español]: https://github.com/thoughtbot/paperclip/blob/master/MIGRATING-ES.md
+
+# Existing documentation
+
 ## Documentation valid for `master` branch
 
 Please check the documentation for the paperclip version you are using:
@@ -167,7 +189,7 @@ Paperclip is distributed as a gem, which is how it should be used in your app.
 Include the gem in your Gemfile:
 
 ```ruby
-gem "paperclip", "~> 5.2.1"
+gem "paperclip", "~> 6.0.0"
 ```
 
 Or, if you want to get the latest, you can get master from the main paperclip repository:
@@ -341,7 +363,7 @@ Lastly, you can also define multiple validations on a single attachment using `v
 
 ```ruby
 validates_attachment :avatar, presence: true,
-  content_type: { content_type: "image/jpeg" },
+  content_type: "image/jpeg",
   size: { in: 0..10.kilobytes }
 ```
 
@@ -368,7 +390,7 @@ afterwards, then assign manually:
 ```ruby
 class Book < ActiveRecord::Base
   has_attached_file :document, styles: { thumbnail: "60x60#" }
-  validates_attachment :document, content_type: { content_type: "application/pdf" }
+  validates_attachment :document, content_type: "application/pdf"
   validates_something_else # Other validations that conflict with Paperclip's
 end
 
@@ -400,7 +422,7 @@ image-y ones:
 
 ```ruby
 validates_attachment :avatar,
-  content_type: { content_type: ["image/jpeg", "image/gif", "image/png"] }
+  content_type: ["image/jpeg", "image/gif", "image/png"]
 ```
 
 `Paperclip::ContentTypeDetector` will attempt to match a file's extension to an

data/UPGRADING

@@ -2,9 +2,9 @@
 #  NOTE FOR UPGRADING FROM 4.3.0 OR EARLIER      #
 ##################################################
 
-Paperclip is now compatible with aws-sdk >= 2.0.0.
+Paperclip is now compatible with aws-sdk-s3.
 
-If you are using S3 storage, aws-sdk >= 2.0.0 requires you to make a few small
+If you are using S3 storage, aws-sdk-s3 requires you to make a few small
 changes:
 
 * You must set the `s3_region`
@@ -13,5 +13,5 @@ changes:
   using a hyphen. For example, `:public_read` needs to be changed to
   `public-read`.
 
-For a walkthrough of upgrading from 4 to 5 and aws-sdk >= 2.0 you can watch
+For a walkthrough of upgrading from 4 to *5* (not 6) and aws-sdk >= 2.0 you can watch
 http://rubythursday.com/episodes/ruby-snack-27-upgrade-paperclip-and-aws-sdk-in-prep-for-rails-5

data/features/step_definitions/attachment_steps.rb

@@ -84,12 +84,12 @@ end
 
 Then /^I should have attachment columns for "([^"]*)"$/ do |attachment_name|
   cd(".") do
-    columns = eval(`bundle exec rails runner "puts User.columns.map{ |column| [column.name, column.type] }.inspect"`.strip)
+    columns = eval(`bundle exec rails runner "puts User.columns.map{ |column| [column.name, column.sql_type] }.inspect"`.strip)
     expect_columns = [
-      ["#{attachment_name}_file_name", :string],
-      ["#{attachment_name}_content_type", :string],
-      ["#{attachment_name}_file_size", :integer],
-      ["#{attachment_name}_updated_at", :datetime]
+      ["#{attachment_name}_file_name", "varchar"],
+      ["#{attachment_name}_content_type", "varchar"],
+      ["#{attachment_name}_file_size", "bigint"],
+      ["#{attachment_name}_updated_at", "datetime"]
     ]
     expect(columns).to include(*expect_columns)
   end
@@ -97,12 +97,12 @@ end
 
 Then /^I should not have attachment columns for "([^"]*)"$/ do |attachment_name|
   cd(".") do
-    columns = eval(`bundle exec rails runner "puts User.columns.map{ |column| [column.name, column.type] }.inspect"`.strip)
+    columns = eval(`bundle exec rails runner "puts User.columns.map{ |column| [column.name, column.sql_type] }.inspect"`.strip)
     expect_columns = [
-      ["#{attachment_name}_file_name", :string],
-      ["#{attachment_name}_content_type", :string],
-      ["#{attachment_name}_file_size", :integer],
-      ["#{attachment_name}_updated_at", :datetime]
+      ["#{attachment_name}_file_name", "varchar"],
+      ["#{attachment_name}_content_type", "varchar"],
+      ["#{attachment_name}_file_size", "bigint"],
+      ["#{attachment_name}_updated_at", "datetime"]
     ]
 
     expect(columns).not_to include(*expect_columns)

data/lib/paperclip.rb

@@ -98,6 +98,7 @@ module Paperclip
       swallow_stderr: true,
       use_exif_orientation: true,
       whiny: true,
+      is_windows: Gem.win_platform?
     }
   end
 

data/lib/paperclip/attachment.rb

@@ -1,4 +1,3 @@
-# encoding: utf-8
 require 'uri'
 require 'paperclip/url_generator'
 require 'active_support/deprecation'
@@ -338,8 +337,15 @@ module Paperclip
     # inconsistencies in timing of S3 commands. It's possible that calling
     # #reprocess! will lose data if the files are not kept.
     def reprocess!(*style_args)
-      saved_only_process, @options[:only_process] = @options[:only_process], style_args
-      saved_preserve_files, @options[:preserve_files] = @options[:preserve_files], true
+      saved_flags = @options.slice(
+        :only_process,
+        :preserve_files,
+        :check_validity_before_processing
+      )
+      @options[:only_process] = style_args
+      @options[:preserve_files] = true
+      @options[:check_validity_before_processing] = false
+
       begin
         assign(self)
         save
@@ -348,8 +354,7 @@ module Paperclip
         warn "#{e} - skipping file."
         false
       ensure
-        @options[:only_process] = saved_only_process
-        @options[:preserve_files] = saved_preserve_files
+        @options.merge!(saved_flags)
       end
     end
 
@@ -532,6 +537,10 @@ module Paperclip
           reduce(original) do |file, processor|
           file = Paperclip.processor(processor).make(file, style.processor_options, self)
           intermediate_files << file unless file == @queued_for_write[:original]
+          # if we're processing the original, close + unlink the source tempfile
+          if name == :original
+            @queued_for_write[:original].close(true)
+          end
           file
         end
 
@@ -591,7 +600,11 @@ module Paperclip
     def unlink_files(files)
       Array(files).each do |file|
         file.close unless file.closed?
-        file.unlink if file.respond_to?(:unlink) && file.path.present? && File.exist?(file.path)
+
+        begin
+          file.unlink if file.respond_to?(:unlink)
+        rescue Errno::ENOENT
+        end
       end
     end
 

data/lib/paperclip/filename_cleaner.rb

@@ -1,4 +1,3 @@
-# encoding: utf-8
 module Paperclip
   class FilenameCleaner
     def initialize(invalid_character_regex)

data/lib/paperclip/geometry_detector_factory.rb

@@ -17,7 +17,7 @@ module Paperclip
         orientation = Paperclip.options[:use_exif_orientation] ?
           "%[exif:orientation]" : "1"
         Paperclip.run(
-          "identify",
+          Paperclip.options[:is_windows] ? "magick identify" : "identify",
           "-format '%wx%h,#{orientation}' :file", {
             :file => "#{path}[0]"
           }, {

data/lib/paperclip/interpolations.rb

@@ -5,6 +5,7 @@ module Paperclip
   # Paperclip.interpolates method.
   module Interpolations
     extend self
+    ID_PARTITION_LIMIT = 1_000_000_000
 
     # Hash assignment of interpolations. Included only for compatibility,
     # and is not intended for normal use.
@@ -175,7 +176,11 @@ module Paperclip
     def id_partition attachment, style_name
       case id = attachment.instance.id
       when Integer
-        ("%09d".freeze % id).scan(/\d{3}/).join("/".freeze)
+        if id < ID_PARTITION_LIMIT
+          ("%09d".freeze % id).scan(/\d{3}/).join("/".freeze)
+        else
+          ("%012d".freeze % id).scan(/\d{3}/).join("/".freeze)
+        end
       when String
         id.scan(/.{3}/).first(3).join("/".freeze)
       else

data/lib/paperclip/io_adapters/abstract_adapter.rb

@@ -4,7 +4,7 @@ module Paperclip
   class AbstractAdapter
     OS_RESTRICTED_CHARACTERS = %r{[/:]}
 
-    attr_reader :content_type, :original_filename, :size
+    attr_reader :content_type, :original_filename, :size, :tempfile
     delegate :binmode, :binmode?, :close, :close!, :closed?, :eof?, :path, :readbyte, :rewind, :unlink, :to => :@tempfile
     alias :length :size
 
@@ -57,15 +57,16 @@ module Paperclip
     end
 
     def link_or_copy_file(src, dest)
-      Paperclip.log("Trying to link #{src} to #{dest}")
-      FileUtils.ln(src, dest, force: true) # overwrite existing
-      @destination.close
-      @destination.open.binmode
-    rescue Errno::EXDEV, Errno::EPERM, Errno::ENOENT, Errno::EEXIST => e
-      Paperclip.log(
-        "Link failed with #{e.message}; copying link #{src} to #{dest}"
-      )
-      FileUtils.cp(src, dest)
+      begin
+        Paperclip.log("Trying to link #{src} to #{dest}")
+        FileUtils.ln(src, dest, force: true) # overwrite existing
+      rescue Errno::EXDEV, Errno::EPERM, Errno::ENOENT, Errno::EEXIST => e
+        Paperclip.log(
+          "Link failed with #{e.message}; copying link #{src} to #{dest}"
+        )
+        FileUtils.cp(src, dest)
+      end
+
       @destination.close
       @destination.open.binmode
     end

data/lib/paperclip/io_adapters/attachment_adapter.rb

@@ -31,7 +31,13 @@ module Paperclip
       if source.staged?
         link_or_copy_file(source.staged_path(@style), destination.path)
       else
-        source.copy_to_local_file(@style, destination.path)
+        begin
+          source.copy_to_local_file(@style, destination.path)
+        rescue Errno::EACCES
+          # clean up lingering tempfile if we cannot access source file
+          destination.close(true)
+          raise
+        end
       end
       destination
     end

data/lib/paperclip/io_adapters/http_url_proxy_adapter.rb

@@ -9,7 +9,8 @@ module Paperclip
     REGEXP = /\Ahttps?:\/\//
 
     def initialize(target, options = {})
-      super(URI(URI.escape(target)), options)
+      escaped = URI.escape(target)
+      super(URI(target == URI.unescape(target) ? escaped : target), options)
     end
   end
 end

data/lib/paperclip/io_adapters/uri_adapter.rb

@@ -28,15 +28,17 @@ module Paperclip
     end
 
     def content_type_from_content
-      if @content.respond_to?(:content_type)
-        @content.content_type
-      end
+      @content.meta["content-type"].presence
     end
 
     def filename_from_content_disposition
-      if @content.meta.key?("content-disposition")
-        matches = @content.meta["content-disposition"].match(/filename="([^"]*)"/)
-        matches[1] if matches
+      if @content.meta.key?("content-disposition") && @content.meta["content-disposition"].match(/filename/i)
+        # can include both filename and filename* values according to RCF6266. filename should come first
+        _, filename = @content.meta["content-disposition"].split(/filename\*?\s*=\s*/i)
+
+        # filename can be enclosed in quotes or not
+        matches = filename.match(/"(.*)"/)
+        matches ? matches[1] : filename.split(';')[0]
       end
     end
 

data/lib/paperclip/logger.rb

@@ -2,7 +2,7 @@ module Paperclip
   module Logger
     # Log a paperclip-specific line. This will log to STDOUT
     # by default. Set Paperclip.options[:log] to false to turn off.
-    def log message
+    def log(message)
       logger.info("[paperclip] #{message}") if logging?
     end
 

data/lib/paperclip/media_type_spoof_detector.rb

@@ -11,7 +11,7 @@ module Paperclip
     end
 
     def spoofed?
-      if has_name? && has_extension? && media_type_mismatch? && mapping_override_mismatch?
+      if has_name? && media_type_mismatch? && mapping_override_mismatch?
         Paperclip.log("Content Type Spoof: Filename #{File.basename(@name)} (#{supplied_content_type} from Headers, #{content_types_from_name.map(&:to_s)} from Extension), content type discovered from file command: #{calculated_content_type}. See documentation to allow this combination.")
         true
       else
@@ -30,15 +30,18 @@ module Paperclip
     end
 
     def media_type_mismatch?
-      supplied_type_mismatch? || calculated_type_mismatch?
+      extension_type_mismatch? || calculated_type_mismatch?
     end
 
-    def supplied_type_mismatch?
-      supplied_media_type.present? && !media_types_from_name.include?(supplied_media_type)
+    def extension_type_mismatch?
+      supplied_media_type.present? &&
+        has_extension? &&
+        !media_types_from_name.include?(supplied_media_type)
     end
 
     def calculated_type_mismatch?
-      !media_types_from_name.include?(calculated_media_type)
+      supplied_media_type.present? &&
+        !calculated_content_type.include?(supplied_media_type)
     end
 
     def mapping_override_mismatch?

data/lib/paperclip/processor.rb

@@ -37,13 +37,21 @@ module Paperclip
     # The convert method runs the convert binary with the provided arguments.
     # See Paperclip.run for the available options.
     def convert(arguments = "", local_options = {})
-      Paperclip.run('convert', arguments, local_options)
+      Paperclip.run(
+        Paperclip.options[:is_windows] ? "magick convert" : "convert",
+        arguments,
+        local_options,
+      )
     end
 
     # The identify method runs the identify binary with the provided arguments.
     # See Paperclip.run for the available options.
     def identify(arguments = "", local_options = {})
-      Paperclip.run('identify', arguments, local_options)
+      Paperclip.run(
+        Paperclip.options[:is_windows] ? "magick identify" : "identify",
+        arguments,
+        local_options,
+      )
     end
   end
 end

data/lib/paperclip/schema.rb

@@ -5,7 +5,7 @@ module Paperclip
   module Schema
     COLUMNS = {:file_name    => :string,
                :content_type => :string,
-               :file_size    => :integer,
+               :file_size    => :bigint,
                :updated_at   => :datetime}
 
     def self.included(base)

data/lib/paperclip/storage/fog.rb

@@ -19,7 +19,7 @@ module Paperclip
     #   store your files.  Remember that the bucket must be unique across
     #   all of Amazon S3. If the bucket does not exist, Paperclip will
     #   attempt to create it.
-    # * +fog_file*: This can be hash or lambda returning hash. The
+    # * +fog_file+: This can be hash or lambda returning hash. The
     #   value is used as base properties for new uploaded file.
     # * +path+: This is the key under the bucket in which the file will
     #   be stored. The URL will be constructed from the bucket and the

data/lib/paperclip/style.rb

@@ -1,4 +1,3 @@
-# encoding: utf-8
 module Paperclip
   # The Style class holds the definition of a thumbnail style,  applying
   # whatever processing is required to normalize the definition and delaying

data/lib/paperclip/thumbnail.rb

@@ -85,7 +85,10 @@ module Paperclip
           dest: File.expand_path(dst.path),
         )
       rescue Terrapin::ExitStatusError => e
-        raise Paperclip::Error, "There was an error processing the thumbnail for #{@basename}" if @whiny
+        if @whiny
+          message = "There was an error processing the thumbnail for #{@basename}:\n" + e.message
+          raise Paperclip::Error, message
+        end
       rescue Terrapin::CommandNotFoundError => e
         raise Paperclip::Errors::CommandNotFoundError.new("Could not run the `convert` command. Please install ImageMagick.")
       end

data/lib/paperclip/validators/media_type_spoof_detection_validator.rb

@@ -8,6 +8,10 @@ module Paperclip
         if Paperclip::MediaTypeSpoofDetector.using(adapter, value.original_filename, value.content_type).spoofed?
           record.errors.add(attribute, :spoofed_media_type)
         end
+
+        if adapter.tempfile
+          adapter.tempfile.close(true)
+        end
       end
     end
 

data/lib/paperclip/version.rb

@@ -1,5 +1,5 @@
 module Paperclip
   unless defined?(Paperclip::VERSION)
-    VERSION = "6.0.0".freeze
+    VERSION = "6.1.0".freeze
   end
 end