paperclip 4.3.7 → 6.1.0

data/MIGRATING.md

@@ -0,0 +1,375 @@
+# Migrating from Paperclip to ActiveStorage
+
+Paperclip and ActiveStorage solve similar problems with similar solutions, so
+transitioning from one to the other is straightforward data re-writing.
+
+The process of going from Paperclip to ActiveStorage is as follows:
+
+1. Apply the ActiveStorage database migrations.
+2. Configure storage.
+3. Copy the database data over.
+4. Copy the files over.
+5. Update your tests.
+6. Update your views.
+7. Update your controllers.
+8. Update your models.
+
+## Apply the ActiveStorage database migrations
+
+Follow [the instructions for installing ActiveStorage]. You'll very likely want
+to add the `mini_magick` gem to your Gemfile.
+
+```sh
+rails active_storage:install
+```
+
+[the instructions for installing ActiveStorage]: https://github.com/rails/rails/blob/master/activestorage/README.md#installation
+
+## Configure storage
+
+Again, follow [the instructions for configuring ActiveStorage].
+
+[the instructions for configuring ActiveStorage]: http://edgeguides.rubyonrails.org/active_storage_overview.html#setup
+
+## Copy the database data over
+
+The `active_storage_blobs` and `active_storage_attachments` tables are where
+ActiveStorage expects to find file metadata. Paperclip stores the file metadata
+directly on the associated object's table.
+
+You'll need to write a migration for this conversion. Because the models for
+your domain are involved, it's tricky to supply a simple script. But we'll try!
+
+Here's how it would go for a `User` with an `avatar`, that is this in
+Paperclip:
+
+```ruby
+class User < ApplicationRecord
+  has_attached_file :avatar
+end
+```
+
+Your Paperclip migrations will produce a table like so:
+
+```ruby
+create_table "users", force: :cascade do |t|
+  t.string "avatar_file_name"
+  t.string "avatar_content_type"
+  t.integer "avatar_file_size"
+  t.datetime "avatar_updated_at"
+end
+```
+
+And you'll be converting into these tables:
+
+```ruby
+create_table "active_storage_attachments", force: :cascade do |t|
+  t.string "name", null: false
+  t.string "record_type", null: false
+  t.integer "record_id", null: false
+  t.integer "blob_id", null: false
+  t.datetime "created_at", null: false
+  t.index ["blob_id"], name: "index_active_storage_attachments_on_blob_id"
+  t.index ["record_type", "record_id", "name", "blob_id"], name: "index_active_storage_attachments_uniqueness", unique: true
+end
+```
+
+```ruby
+create_table "active_storage_blobs", force: :cascade do |t|
+  t.string "key", null: false
+  t.string "filename", null: false
+  t.string "content_type"
+  t.text "metadata"
+  t.bigint "byte_size", null: false
+  t.string "checksum", null: false
+  t.datetime "created_at", null: false
+  t.index ["key"], name: "index_active_storage_blobs_on_key", unique: true
+end
+```
+
+So, assuming you want to leave the files in the exact same place,  _this is
+your migration_. Otherwise, see the next section first and modify the migration
+to taste.
+
+```ruby
+Dir[Rails.root.join("app/models/**/*.rb")].sort.each { |file| require file }
+
+class ConvertToActiveStorage < ActiveRecord::Migration[5.2]
+  require 'open-uri'
+
+  def up
+    # postgres
+    get_blob_id = 'LASTVAL()'
+    # mariadb
+    # get_blob_id = 'LAST_INSERT_ID()'
+    # sqlite
+    # get_blob_id = 'LAST_INSERT_ROWID()'
+
+    active_storage_blob_statement = ActiveRecord::Base.connection.raw_connection.prepare(<<-SQL)
+      INSERT INTO active_storage_blobs (
+        `key`, filename, content_type, metadata, byte_size, checksum, created_at
+      ) VALUES (?, ?, ?, '{}', ?, ?, ?)
+    SQL
+
+    active_storage_attachment_statement = ActiveRecord::Base.connection.raw_connection.prepare(<<-SQL)
+      INSERT INTO active_storage_attachments (
+        name, record_type, record_id, blob_id, created_at
+      ) VALUES (?, ?, ?, #{get_blob_id}, ?)
+    SQL
+
+    models = ActiveRecord::Base.descendants.reject(&:abstract_class?)
+
+    transaction do
+      models.each do |model|
+        attachments = model.column_names.map do |c|
+          if c =~ /(.+)_file_name$/
+            $1
+          end
+        end.compact
+
+        model.find_each.each do |instance|
+          attachments.each do |attachment|
+            active_storage_blob_statement.execute(
+              key(instance, attachment),
+              instance.send("#{attachment}_file_name"),
+              instance.send("#{attachment}_content_type"),
+              instance.send("#{attachment}_file_size"),
+              checksum(instance.send(attachment)),
+              instance.updated_at.iso8601
+            )
+
+            active_storage_attachment_statement.
+              execute(attachment, model.name, instance.id, instance.updated_at.iso8601)
+          end
+        end
+      end
+    end
+
+    active_storage_attachment_statement.close
+    active_storage_blob_statement.close
+  end
+
+  def down
+    raise ActiveRecord::IrreversibleMigration
+  end
+
+  private
+
+  def key(instance, attachment)
+    SecureRandom.uuid
+    # Alternatively:
+    # instance.send("#{attachment}_file_name")
+  end
+
+  def checksum(attachment)
+    # local files stored on disk:
+    url = attachment.path
+    Digest::MD5.base64digest(File.read(url))
+
+    # remote files stored on another person's computer:
+    # url = attachment.url
+    # Digest::MD5.base64digest(Net::HTTP.get(URI(url)))
+  end
+end
+```
+
+## Copy the files over
+
+The above migration leaves the files as they are. However, the default
+Paperclip and ActiveStorage storage services use different locations.
+
+By default, Paperclip looks like this:
+
+```
+public/system/users/avatars/000/000/004/original/the-mystery-of-life.png
+```
+
+And ActiveStorage looks like this:
+
+```
+storage/xM/RX/xMRXuT6nqpoiConJFQJFt6c9
+```
+
+That `xMRXuT6nqpoiConJFQJFt6c9` is the `active_storage_blobs.key` value. In the
+migration above we simply used the filename but you may wish to use a UUID
+instead.
+
+
+### Moving local storage files
+
+```ruby
+#!bin/rails runner
+
+class ActiveStorageBlob < ActiveRecord::Base
+end
+
+class ActiveStorageAttachment < ActiveRecord::Base
+  belongs_to :blob, class_name: 'ActiveStorageBlob'
+  belongs_to :record, polymorphic: true
+end
+
+ActiveStorageAttachment.find_each do |attachment|
+  name = attachment.name
+
+  source = attachment.record.send(name).path
+  dest_dir = File.join(
+    "storage",
+    attachment.blob.key.first(2),
+    attachment.blob.key.first(4).last(2))
+  dest = File.join(dest_dir, attachment.blob.key)
+
+  FileUtils.mkdir_p(dest_dir)
+  puts "Moving #{source} to #{dest}"
+  FileUtils.cp(source, dest)
+end
+```
+
+### Moving files on a remote host (S3, Azure Storage, GCS, etc.)
+
+One of the most straightforward ways to move assets stored on a remote host is
+to use a rake task that regenerates the file names and places them in the
+proper file structure/hierarchy.
+
+Assuming you have a model configured similarly to the example below:
+
+```ruby
+class Organization < ApplicationRecord
+  # New ActiveStorage declaration
+  has_one_attached :logo
+
+  # Old Paperclip config
+  # must be removed BEFORE to running the rake task so that
+  # all of the new ActiveStorage goodness can be used when
+  # calling organization.logo
+  has_attached_file :logo,
+                    path: "/organizations/:id/:basename_:style.:extension",
+                    default_url: "https://s3.amazonaws.com/xxxxx/organizations/missing_:style.jpg",
+                    default_style: :normal,
+                    styles: { thumb: "64x64#", normal: "400x400>" },
+                    convert_options: { thumb: "-quality 100 -strip", normal: "-quality 75 -strip" }
+end
+```
+
+The following rake task would migrate all of your assets:
+
+```ruby
+namespace :organizations do
+  task migrate_to_active_storage: :environment do
+    Organization.where.not(logo_file_name: nil).find_each do |organization|
+      # This step helps us catch any attachments we might have uploaded that
+      # don't have an explicit file extension in the filename
+      image = organization.logo_file_name
+      ext = File.extname(image)
+      image_original = URI.unescape(image.gsub(ext, "_original#{ext}"))
+
+      # this url pattern can be changed to reflect whatever service you use
+      logo_url = "https://s3.amazonaws.com/xxxxx/organizations/#{organization.id}/#{image_original}"
+      organization.logo.attach(io: open(logo_url),
+                                   filename: organization.logo_file_name,
+                                   content_type: organization.logo_content_type)
+    end
+  end
+end
+```
+
+An added advantage of this method is that you're creating a copy of all assets,
+which is handy in the event you need to rollback your deploy.
+
+This also means that you can run the rake task from your development machine
+and completely migrate the assets before your deploy, minimizing the chances
+that you'll have a timed-out deployment.
+
+The main drawback of this method is the same as its benefit - you are
+essentially duplicating all of your assets. These days storage and bandwidth
+are relatively cheap, but in some instances where you have a huge volume of
+files, or very large file sizes, this might get a little less feasible.
+
+In my experience I was able to move tens of thousands of images in a matter of
+a couple of hours, just by running the migration overnight on my MacBook Pro.
+
+Once you've confirmed that the migration and deploy have gone successfully you
+can safely delete the old assets from your remote host.
+
+## Update your tests
+
+Instead of the `have_attached_file` matcher, you'll need to write your own.
+Here's one that is similar in spirit to the Paperclip-supplied matcher:
+
+```ruby
+RSpec::Matchers.define :have_attached_file do |name|
+  matches do |record|
+    file = record.send(name)
+    file.respond_to?(:variant) && file.respond_to?(:attach)
+  end
+end
+```
+
+## Update your views
+
+In Paperclip it looks like this:
+
+```ruby
+image_tag @user.avatar.url(:medium)
+```
+
+In ActiveStorage it looks like this:
+
+```ruby
+image_tag @user.avatar.variant(resize: "250x250")
+```
+
+## Update your controllers
+
+This should _require_ no update. However, if you glance back at the database
+schema above, you may notice a join.
+
+For example, if your controller has
+
+```ruby
+def index
+  @users = User.all.order(:name)
+end
+```
+
+And your view has
+
+```
+<ul>
+  <% @users.each do |user| %>
+    <li><%= image_tag user.avatar.variant(resize: "10x10"), alt: user.name %></li>
+  <% end %>
+</ul>
+```
+
+Then you'll end up with an n+1 as you load each attachment in the loop.
+
+So while the controller and model will work without change, you will want to
+double-check your loops and add `includes` as needed. ActiveStorage adds an
+`avatar_attachment` and `avatar_blob` relationship to has-one relations, and
+`avatar_attachments` and `avatar_blobs` to has-many:
+
+```ruby
+def index
+  @users = User.all.order(:name).includes(:avatar_attachment)
+end
+```
+
+## Update your models
+
+Follow [the guide on attaching files to records]. For example, a `User` with an
+`avatar` is represented as:
+
+```ruby
+class User < ApplicationRecord
+  has_one_attached :avatar
+end
+```
+
+Any resizing is done in the view as a variant.
+
+[the guide on attaching files to records]: http://edgeguides.rubyonrails.org/active_storage_overview.html#attaching-files-to-records
+
+## Remove Paperclip
+
+Remove the Gem from your `Gemfile` and run `bundle`. Run your tests because
+you're done!

data/NEWS

@@ -1,39 +1,134 @@
-4.3.7 (7/1/2016):
-
-* Add deprecation warnings
-* Improvement: Add `fog_options` configuration to send options to fog when storing files.
-* Improvement: the `URI adapter` now uses the content-disposition header to name the downloaded file.
-
-4.3.6 (3/13/2016):
-
-* Bug Fix: When a spoofed media type is detected, megabytes of mime-types info are added to logs. See https://cwe.mitre.org/data/definitions/779.html.
-
-4.3.5 (2/8/2016):
-
-* Bug Fix: Remove deprecation warnings for v5.0 for now. Will re-add once the version has landed.
-
-4.3.4 (2/2/2016):
-
-* Bug Fix: Fix incompatible change from v4.3.3.
-
-4.3.3 (1/29/2016):
+6.1.0 (2018-07-27):
+
+* BUGFIX: Don't double-encode URLs (Roderick Monje).
+* BUGFIX: Only use the content_type when it exists (Jean-Philippe Doyle).
+* STABILITY: Better handling of the content-disposition header. Now supports
+  file name that is either enclosed or not in double quotes and is case
+  insensitive as per RC6266 grammar (Hasan Kumar, Yves Riel).
+* STABILITY: Change database column type of attachment file size from unsigned 4-byte
+  `integer` to unsigned 8-byte `bigint`. The former type limits attachment size
+  to just over 2GB, which can easily be exceeded by a large video file (Laurent
+  Arnoud, Alen Zamanyan).
+* STABILITY: Better error message when thumbnail processing errors (Hayden Ball).
+* STABILITY: Fix file linking issues around Windows (Akihiko Odaki).
+* STABILITY: Files without an extension will now be checked for spoofing attempts
+  (George Walters II).
+* STABILITY: Manually close Tempfiles when we are done with them (Erkki Eilonen).
+
+6.0.0 (2018-03-09):
+
+* Improvement: Depend only on `aws-sdk-s3` instead of `aws-sdk` (https://github.com/thoughtbot/paperclip/pull/2481)
+
+5.3.0 (2018-03-09):
+
+* Improvement: Use `FactoryBot` instead of `FactoryGirl` (https://github.com/thoughtbot/paperclip/pull/2501)
+* Improvement: README updates (https://github.com/thoughtbot/paperclip/pull/2411, https://github.com/thoughtbot/paperclip/pull/2433, https://github.com/thoughtbot/paperclip/pull/2374, https://github.com/thoughtbot/paperclip/pull/2417, https://github.com/thoughtbot/paperclip/pull/2536)
+* Improvement: Remove Ruby 2.4 deprecation warning (https://github.com/thoughtbot/paperclip/pull/2401)
+* Improvement: Rails 5 migration compatibility (https://github.com/thoughtbot/paperclip/pull/2470)
+* Improvement: Documentation around post processing (https://github.com/thoughtbot/paperclip/pull/2381)
+* Improvement: S3 hostname example documentation (https://github.com/thoughtbot/paperclip/pull/2379)
+* Bugfix: Allow paperclip to load in IRB (https://github.com/thoughtbot/paperclip/pull/2369)
+* Bugfix: MIME type detection (https://github.com/thoughtbot/paperclip/issues/2527)
+* Bugfix: Bad tempfile state after symlink failure (https://github.com/thoughtbot/paperclip/pull/2540)
+* Bugfix: Rewind file after Fog bucket creation (https://github.com/thoughtbot/paperclip/pull/2572)
+* Improvement: Use `Terrapin` instead of `Cocaine` (https://github.com/thoughtbot/paperclip/pull/2553)
+
+5.2.1 (2018-01-25):
+
+* Bugfix: Fix copying files on Windows. (#2532)
+
+5.2.0 (2018-01-23):
+
+* Security: Remove the automatic loading of URI adapters. Some of these
+  adapters can be specially crafted to expose your network topology. (#2435)
+* Bugfix: The rake task no longer rescues `Exception`. (#2476)
+* Bugfix: Handle malformed `Content-Disposition` headers (#2283)
+* Bugfix: The `:only_process` option works when passed a lambda again. (#2289)
+* Improvement: Added `:use_accelerate_endpoint` option when using S3 to enable
+  [Amazon S3 Transfer Acceleration](http://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html)
+  (#2291)
+* Improvement: Make the fingerprint digest configurable per attachment. The
+  default remains MD5. Making this configurable means it can change in a future
+  version because it is not considered secure anymore against intentional file
+  corruption. For more info, see https://en.wikipedia.org/wiki/MD5#Security
+
+  You can change the digest used for an attachment by adding the
+  `:adapter_options` parameter to the `has_attached_file` options like this:
+  `has_attached_file :avatar, adapter_options: { hash_digest: Digest::SHA256 }`
+
+  Use the rake task to regenerate fingerprints with the new digest for a given
+  class. Note that this does **not** check the file integrity using the old
+  fingerprint. Run the following command to regenerate fingerprints for all
+  User attachments:
+  `CLASS=User rake paperclip:refresh:fingerprints`
+  You can optionally limit the attachment that will be processed, e.g:
+  `CLASS=User ATTACHMENT=avatar rake paperclip:refresh:fingerprints` (#2229)
+* Improvement: The new `frame_index` option on the thumbnail processor allows
+  you to select a specific frame from an animated upload to use as a thumbnail.
+  Initial support is for mkv, avi, MP4, mov, MPEG, and GIF. (#2155)
+* Improvement: Instead of copying files, use hard links. This is an
+  optimization. (#2120)
+* Improvement: S3 storage option `:s3_prefixes_in_alias`. (#2287)
+* Improvement: Fog option `:fog_public` can be a lambda. (#2302)
+* Improvement: One fewer warning on JRuby. (#2352)
+* Ruby 2.4.0 compatibility (doesn't use Fixnum anymore)
+
+5.1.0 (2016-08-19):
+
+* Add default `content_type_detector` to `UploadedFileAdapter` (#2270)
+* Default S3 protocol to empty string (#2038)
+* Don't write original file if it wasn't reprocessed (#1993)
+* Disallow trailing newlines in regular expressions (#2266)
+* Support for readbyte in Paperclip attachments (#2034)
+* (port from 4.3) Uri io adapter uses the content-disposition filename (#2250)
+* General refactors and documentation improvements
+
+5.0.0 (2016-07-01):
+
+* Improvement: Add `read_timeout` configuration for URI Adapter download_content method.
+* README adjustments for Ruby beginners (add links, elucidate model in Quick Start)
+* Bugfix: Now it's possible to save images from URLs with special characters [#1932]
+* Bugfix: Return false when file to copy is not present in cloud storage [#2173]
+* Automatically close file while checking mime type [#2016]
+* Add `read_timeout` option to `UriAdapter#download_content` method [#2232]
+* Fix a nil error in content type validation matcher [#1910]
+* Documentation improvements
+
+5.0.0.beta2 (2016-04-01):
+
+* Bugfix: Dynamic fog directory option is now respected
+* Bugfix: Fixes cocaine duplicated paths [#2169]
+* Removal of dead code (older versions of Rails and AWS SDK)
+* README adjustments
 
-* Improvement: Add deprecation warnings in preparation for release of v5.0
+5.0.0.beta1 (2016-03-13):
 
-4.3.2 (11/18/2015):
+* Bug Fix: megabytes of mime-types info in logs when a spoofed media type is detected.
+* Drop support to end-of-life'd ruby 2.0.
+* Drop support for end-of-life'd Rails 3.2 and 4.1
+* Drop support for AWS v1
+* Remove tests for JRuby and Rubinius from Travis CI (they were failing)
+* Improvement: Add `fog_options` configuration to send options to fog when
+  storing files.
+* Extracted repository for locales only:  https://github.com/thoughtbot/paperclip-i18n
+* Bugfix: Original file could be unlinked during `post_process_style`, producing failures
+* Bugfix for image magick scaling images up
+* Memory consumption improvements
+* `url` on a unpersisted record returns `default_url` rather than `nil`
+* Improvement: aws-sdk v2 support
+  https://github.com/thoughtbot/paperclip/pull/1903
 
-* Performance: Reduce memory allocations (#2056)
+  If your Gemfile contains aws-sdk (>= 2.0.0) and aws-sdk-v1, paperclip will use
+  aws-sdk v2. With aws-sdk v2, S3 storage requires you to set the s3_region.
+  s3_region may be nested in s3_credentials, and (if not nested in
+  s3_credentials) it may be a Proc.
 
-4.3.1 (9/9/2015):
+4.3
 
-* Backport of bugfix to `remove_column`, so it works in Rails 3 and 4
-  c740fb171fe2f88c60b999d2a1c2122f2b8f43e9
-* Fix GeometryParser regex for usage of '@>' flag
-* `url` on a unpersisted record returns default_url
-* spec deprecation warnings and failures
-* README adjustments
+See patch versions in v4.3 NEWS:
+https://github.com/thoughtbot/paperclip/blob/v4.3/NEWS
 
-4.3.0 (6/18/2015):
+4.3.0 (2015-06-18):
 
 * Improvement: Update aws-sdk and cucumber gem versions.
 * Improvement: Add `length` alias for `size` method in AbstractAdapter.
@@ -46,7 +141,7 @@
 * Ruby Versioning: Drop support for 1.9.3 (EOL'ed)
 * Rails Versioning: Drop support for 4.0.0 (EOL'ed)
 
-4.2.4:
+4.2.4 (2015-06-05):
 
 * Rollback backwards incompatible change, allowing paperclip to run on
   Ruby >= 1.9.2.