shrine 2.2.0 → 2.3.0

data/doc/direct_s3.md

@@ -1,13 +1,30 @@
 # Direct Uploads to S3
 
-Shrine gives you the ability to upload files directly to S3, which frees your
-server from accepting file uploads. If on saving the record you need to do some
-file processing, you can kick that into a background job using the
-`backgrounding` plugin. If you're not doing any processing and your permanent
-storage is also S3, saving the record will perform an S3 COPY request from
-cache to store, without any downloading and uploading (which is both fast and
-memory-efficient).
+Shrine gives you the ability to upload files directly to Amazon S3, which is
+beneficial for several use cases:
+
+* accepting uploads is resource-intensive for the server, and delegating it to
+  an external service makes scaling easier
+
+* if both temporary and permanent storage are S3, promoting an S3 file to
+  permanent storage will simply issue an S3 copy request, without any
+  downloading and reuploading
+
+* with multiple servers it's generally not possible to cache files to the disk,
+  unless you're using a distibuted filesystem that's shared between servers
+
+* Heroku restricts file uploads to disk, allowing you to save files only in
+  the temporary folder, which gets wiped out between deploys
+
+* Heroku has a 30-second request limit, so if the client has a slow connection
+  and/or your files are larger, uploads to your app can easily hit that limit
+
+You can start by setting both temporary and permanent storage to S3 with
+different prefixes (or even buckets):
 
+```rb
+gem "aws-sdk", "~> 2.1"
+```
 ```rb
 require "shrine/storage/s3"
 
@@ -21,9 +38,10 @@ Shrine.storages = {
 
 ## Enabling CORS
 
-First thing that you need to do is enable CORS on your S3 bucket. You can do
-that by clicking on "Properties > Permissions > Add CORS Configuration", and
-then just follow the Amazon documentation on how to write a CORS file.
+In order to be able upload files directly to your S3 bucket, you need enable
+CORS. You can do that in the AWS S3 Console by clicking on "Properties >
+Permissions > Add CORS Configuration", and then just follow the Amazon
+documentation on how to write a CORS file.
 
 http://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html
 
@@ -32,7 +50,8 @@ DNS propagation.
 
 ## File hash
 
-Shrine's JSON representation of an uploaded file looks like this:
+After direct S3 uploads we'll need to manually construct Shrine's
+representation of an uploaded file:
 
 ```rb
 {
@@ -46,10 +65,9 @@ Shrine's JSON representation of an uploaded file looks like this:
 }
 ```
 
-The `id`, `storage` fields are optional, while the `metadata` values are
-optional (`metadata.size` is only required to later upload that file to a
-non-S3 storage). After uploading the file to S3, you need to construct this
-JSON, and then you can assign it to the hidden attachment field in the form.
+* `id` – location of the file on S3 (minus the `:prefix`)
+* `storage` – direct uploads typically use the `:cache` storage
+* `metadata` – hash of metadata extracted from the file
 
 ## Strategy A (dynamic)
 
@@ -57,17 +75,20 @@ JSON, and then you can assign it to the hidden attachment field in the form.
 * Single or multiple file uploads
 * Some JavaScript needed
 
-When the user selects the file, we dynamically request the presign from the
+When the user selects the file, we dynamically fetch the presign from the
 server, and use this information to start uploading the file to S3. The
-direct_upload plugin gives us this presign route, so we just need to mount it
+`direct_upload` plugin gives us this presign route, so we just need to mount it
 in our application:
 
+```rb
+gem "roda"
+```
 ```rb
 plugin :direct_upload
 ```
 ```rb
 Rails.application.routes.draw do
-  mount ImageUploader::UploadEndpoint => "/image"
+  mount ImageUploader::UploadEndpoint => "/images"
 end
 ```
 
@@ -91,27 +112,24 @@ necessary request parameters:
 ```
 
 For uploading to S3 you'll probably want to use a JavaScript file upload
-library like [jQuery-File-Upload] or [Dropzone]. After the upload you should
-create a JSON representation of the uploaded file, which you can write to
-the hidden attachment field:
-
-```js
-var image = {
-  id: key.match(/cache\/(.+)/)[1], // we have to remove the prefix part
-  storage: 'cache',
-  metadata: {
-    size:      data.files[0].size,
-    filename:  data.files[0].name.match(/[^\/\\]+$/)[0], // IE returns full path
-    mime_type: data.files[0].type
+library like [jQuery-File-Upload], [Dropzone] or [FineUploader]. After the
+upload you should create a JSON representation of the uploaded file, which you
+can write to the hidden attachment field:
+
+```html
+<input type='hidden' name='photo[image]' value='{
+  "id": "302858ldg9agjad7f3ls.jpg",
+  "storage": "cache",
+  "metadata": {
+    "size": 943483,
+    "filename": "nature.jpg",
+    "mime_type": "image/jpeg",
   }
-}
-
-$('input[type=file]').prev().val(JSON.stringify(image))
+}'>
 ```
 
-It's generally a good idea to disable the submit button until the file is
-uploaded, as well as display a progress bar. See the [example app] for a
-working implementation of multiple direct S3 uploads.
+See the [demo app] for an example JavaScript implementation of multiple direct
+S3 uploads.
 
 ## Strategy B (static)
 
@@ -163,7 +181,7 @@ caching the file doesn't touch your application. When the cached file is stored,
 Shrine's default behaviour is to simply copy over cached file's metadata.
 
 If you want to extract metadata on the server before storing, you can just
-load the restore_cached_data plugin.
+load the `restore_cached_data` plugin.
 
 ```rb
 plugin :restore_cached_data
@@ -201,12 +219,12 @@ backgrounding library to perform the job with a delay:
 Shrine.plugin :backgrounding
 
 Shrine::Attacher.promote do |data|
-  PromoteJob.perform_in(60, data) # tells a Sidekiq worker to perform in 1 minute
+  PromoteJob.perform_in(3, data) # tells a Sidekiq worker to perform in 3 seconds
 end
 ```
 
 [`Aws::S3::PresignedPost`]: http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Bucket.html#presigned_post-instance_method
-[example app]: https://github.com/janko-m/shrine-example
+[demo app]: https://github.com/janko-m/shrine/tree/master/demo
 [Dropzone]: https://github.com/enyo/dropzone
 [jQuery-File-Upload]: https://github.com/blueimp/jQuery-File-Upload
 [Amazon S3 Data Consistency Model]: http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyMode

data/doc/paperclip.md

@@ -1,158 +1,251 @@
 # Shrine for Paperclip Users
 
-This guide is aimed at helping Paperclip users transition to Shrine. We will
-first generally mention what are the key differences. Afterwards there is a
-complete reference of Paperclip's interface and what is the equivalent in
-Shrine.
+This guide is aimed at helping Paperclip users transition to Shrine, and it
+consists of three parts:
 
-## Uploaders
-
-While in Paperclip you write your uploading logic as a list of options inside
-your models, in Shrine you instead have "uploader" classes where you put all
-your uploading logic.
+1. Explanation of the key differences in design between Paperclip and Shrine
+2. Instructions how to migrate and existing app that uses Paperclip to Shrine
+3. Extensive reference of Paperclip's interface with Shrine equivalents
 
-```rb
-class ImageUploader < Shrine
-  plugin :validation_helpers
+## Storages
 
-  Attacher.validate do
-    validate_mime_type_inclusion [/^image/]
-  end
+While in Paperclip you configure storage in the model, a Shrine storage is just
+a class which you configure individually:
 
-  def process(io, context)
-    # processing
-  end
+```rb
+class Photo < ActiveRecord::Base
+  has_attached_file :image,
+    storage: :s3,
+    s3_credentials: {
+      bucket:            "my-bucket",
+      access_key_id:     "abc",
+      secret_access_key: "xyz",
+    },
+    s3_host_alias: "http://abc123.cloudfront.net",
 end
 ```
+```rb
+Shrine.storages[:store] = Shrine::Storage::S3.new(
+  bucket:            "my-bucket",
+  access_key_id:     "abc",
+  secret_access_key: "xyz",
+  host:              "http://abc123.cloudfront.net",
+)
+```
 
-Unlike Paperclip, in Shrine you can use these uploaders directly if you have
-to do some lower-level logic. First you need to register storages, and then
-you can instantiate uploaders with a specific storage:
+Paperclip doesn't have a concept of "temporary" storage, so it cannot retain
+uploaded files in case of validation errors, and [direct S3 uploads] cannot be
+implemented in a safe way. Shrine conceptually separates a "temporary" and
+"permanent" storage:
 
 ```rb
-require "shrine/storage/file_system"
-
 Shrine.storages = {
   cache: Shrine::Storage::FileSystem.new("public", prefix: "uploads/cache"),
-  store: Shrine::Storage::FileSystem.new("public", prefix: "uploads/store"),
+  store: Shrine::Storage::S3.new(bucket: "my-bucket", **s3_options),
 }
 ```
+
+## Uploaders
+
+While in Paperclip you define all your uploading logic inside your models,
+Shrine takes a more object-oriented approach and lets you define uploading logic
+inside "uploader" classes:
+
+```rb
+class Photo < ActiveRecord::Base
+  has_attached_file :image
+end
+```
+
+```rb
+class ImageUploader < Shrine
+  # ...
+end
+
+class Photo < ActiveRecord::Base
+  include ImageUploader[:image]
+end
+```
+
+Among other things, this allows you to use uploader classes standalone, which
+gives you more power:
+
 ```rb
-uploader = Shrine.new(:cache)
+uploader = ImageUploader.new(:store)
 uploaded_file = uploader.upload(File.open("nature.jpg"))
-uploaded_file.path #=> "/uploads/cache/s9ffdkfd02kd.jpg"
-uploaded_file.original_filename #=> "nature.jpg"
+uploaded_file     #=> #<Shrine::UploadedFile>
+uploaded_file.url #=> "https://my-bucket.s3.amazonaws.com/store/kfds0lg9rer.jpg"
 ```
 
 ### Processing
 
-Unlike Paperclip, in Shrine you define and perform processing on
-instance-level, which gives a lot of flexibility. As the result you can return
-a single file or a hash of versions:
+In contrast to Paperclip's static options, in Shrine you define and perform
+processing on instance-level. The result of processing can be a single file
+or a hash of versions:
 
 ```rb
-require "image_processing/mini_magick" # part of the "image_processing" gem
+class Photo < ActiveRecord::Base
+  has_attached_file :image,
+    styles: {
+      large:  "800x800>",
+      medium: "500x500>",
+      small:  "300x300>",
+    }
+end
+```
 
+```rb
 class ImageUploader < Shrine
   include ImageProcessing::MiniMagick
   plugin :processing
   plugin :versions
 
   process(:store) do |io, context|
-    thumbnail = resize_to_limit(io.download, 300, 300)
-    {original: io, thumbnail: thumbnail}
+    size_800 = resize_to_limit(io.download, 800, 800)
+    size_500 = resize_to_limit(size_800,    500, 500)
+    size_300 = resize_to_limit(size_500,    300, 300)
+
+    {large: size_800, medium: size_500, small: size_300}
   end
 end
 ```
 
-#### Regenerating versions
+This allows you to fully optimize processing, because you can easily specify
+which files are processed from which, and even add parallelization.
 
-Shrine doesn't have a built-in way of regenerating versions, because that's
-very individual and depends on what versions you want regenerated, what ORM are
-you using, how many records there are in your database etc. The [Regenerating
-versions] guide provides some useful tips on this task.
+#### Reprocessing versions
 
-### Logging
+Shrine doesn't have a built-in way of regenerating versions, because that has
+to be written and optimized differently depending on whether you're adding or
+removing a version, what ORM are you using, how many records there are in the
+database etc. The [Reprocessing versions] guide provides some useful tips on
+this task.
 
-In Paperclip you enable logging by setting `Paperclip.options[:log] = true`.
-Shrine also provides logging with the `logging` plugin:
+### Validations
+
+Validations are also defined inside the uploader on the instance-level, which
+allows you to do conditional validations:
 
 ```rb
-Shrine.plugin :logging
+class Photo < ActiveRecord::Base
+  has_attached_file :image
+  validates_attachment :image,
+    content_type: {content_type: %w[image/jpeg image/png image/gif]},
+    size: {in: 0..10.megabytes}
+end
 ```
 
-## Attachments
+```rb
+class ImageUploader < Shrine
+  plugin :validation_helpers
+
+  Attacher.validate do
+    validate_mime_type_inclusion %w[image/jpeg image/gif image/png]
+    validate_max_size 10*1024*1024 unless record.admin?
+  end
+end
+```
+
+#### MIME type spoofing
+
+Paperclip detects MIME type spoofing, in the way that it extracts the MIME type
+from file contents using the `file` command and MimeMagic, compares it to the
+value that the `mime-types` gem determined from file extension, and raises a
+validation error if these two values mismatch.
 
-The uploaders can then integrate with models by generating attachment modules
-which are included into the models. Shrine ships with plugins for Sequel and
-ActiveRecord ORMs, so you first have to load the one for your ORM:
+However, this turned out to be very problematic, leading to a lot of valid
+files being classified as "spoofed", because of the differences of MIME
+type databases between the `mime-types` gem, `file` command, and MimeMagic.
+
+Shrine takes a different approach here. By default it will extract MIME
+type from file extension, but it has a plugin for determining MIME type from
+file contents, which by default uses the `file` command:
 
 ```rb
-Shrine.plugin :sequel       # If you're using Sequel
-Shrine.plugin :activerecord # If you're using ActiveRecord
+Shrine.plugin :determine_mime_type
 ```
 
-Now you use your uploaders to generate "attachment modules", which you can then
-include in your models:
+However, it doesn't try to compare this value with the one from file extension,
+it just means that now this value will be used for your MIME type validations.
+With this approach you can still prevent malicious files from being attached,
+but without the possibility of false negatives.
+
+### Logging
+
+In Paperclip you enable logging by setting `Paperclip.options[:log] = true`,
+however, this only logs ImageMagick commands. Shrine has full logging support,
+which measures processing, uploading and deleting individually, along with
+context for debugging:
 
 ```rb
-class User < Sequel::Model
-  include ImageUploader[:avatar] # adds `avatar`, `avatar=` and `avatar_url` methods
-end
+Shrine.plugin :logging
+```
+```
+2015-10-09T20:06:06.676Z #25602: STORE[cache] ImageUploader[:avatar] User[29543] 1 file (0.1s)
+2015-10-09T20:06:06.854Z #25602: PROCESS[store]: ImageUploader[:avatar] User[29543] 1-3 files (0.22s)
+2015-10-09T20:06:07.133Z #25602: DELETE[destroyed]: ImageUploader[:avatar] User[29543] 3 files (0.07s)
 ```
 
-Unlike in Paperclip which requires you to have 4 `<attachment>_*` columns, in
-Shrine you only need to have an `<attachment>_data` text column, and all
-information will be stored there (in the above case `avatar_data`).
+## Attachments
 
-The attachments use `:store` for storing the files, and `:cache` for caching.
-The latter is something Paperclip doesn't do, but caching before storing is
-really great because the file then persists on validation errors, and also in
-backgrounding you can show the users the cached version before the file is
-finished storing.
+While Paperclip is designed to only integrate with ActiveRecord, Shrine is
+designed to be completely generic and integrate with any ORM. It ships with
+plugins for ActiveRecord and Sequel:
 
-### Validations
+```rb
+Shrine.plugin :activerecord # if you're using ActiveRecord
+Shrine.plugin :sequel       # if you're using Sequel
+```
 
-In Shrine validations are done inside uploader classes, and validation methods
-are provided by the `validation_helpers` plugin:
+Instead of giving you class methods for defining attachments, in Shrine you
+generate attachment modules which you simply include in your models, which
+gives your models similar set of methods that Paperclip gives:
 
 ```rb
-class ImageUploader < Shrine
-  plugin :validation_helpers
-
-  Attacher.validate do
-    validate_max_size 5*1024*1024
-    validate_mime_type_inclusion [/^image/]
-  end
+class Photo < Sequel::Model
+  include ImageUploader[:image]
 end
 ```
 
-For presence validation you should use the one provided by your ORM:
+### Attachment column
+
+Unlike in Paperclip which requires you to have 4 `<attachment>_*` columns, in
+Shrine you only need to have a single `<attachment>_data` text column (in the
+above case `image_data`), and all information will be stored there.
 
 ```rb
-class User < Sequel::Model
-  include ImageUploader[:avatar]
+photo.image_data #=>
+# {
+#   "storage" => "store",
+#   "id" => "photo/1/image/0d9o8dk42.png",
+#   "metadata" => {
+#     "filename"  => "nature.png",
+#     "size"      => 49349138,
+#     "mime_type" => "image/png"
+#   }
+# }
 
-  def validate
-    validates_presence [:avatar]
-  end
-end
+photo.image.original_filename #=> "nature.png"
+photo.image.size              #=> 49349138
+photo.image.mime_type         #=> "image/png"
 ```
 
-#### MIME type spoofing
-
-By default Shrine will extract the MIME type from the `Content-Type` header of
-the uploaded file, which is solely determined from the file extension, so it's
-prone to spoofing. Shrine provides the `determine_mime_type` plugin which
-determines the MIME type from the file *contents* instead:
+Unlike Paperclip, Shrine will store this information for each processed
+version, making them first-class citizens:
 
 ```rb
-Shrine.plugin :determine_mime_type
+photo.image[:original]       #=> #<Shrine::UploadedFile>
+photo.image[:original].width #=> 800
+
+photo.image[:thumb]          #=> #<Shrine::UploadedFile>
+photo.image[:thumb].width    #=> 300
 ```
 
-By default the UNIX [file] utility is used, but you can choose other analyzers.
-Unlike Paperclip, you won't get any errors if the MIME type is "spoofed",
-instead it's better if you simply validate allowed MIME types.
+Also, since Paperclip stores only the filename, it has to recalculate the full
+location each time it wants to generate the URL. That makes it really difficult
+to move files to a new location, because changing how the location is generated
+will now cause incorrect URLs to be generated for all existing files. Shrine
+calculates the whole location only once and saves it to the column.
 
 ### Hooks/Callbacks
 
@@ -312,8 +405,6 @@ In Shrine attachments will automatically use `:cache` and `:store` storages
 which you have to register:
 
 ```rb
-require "shrine/storage/file_system"
-
 Shrine.storages = {
   cache: Shrine::Storage::FileSystem.new("public", prefix: "uploads/cache"),
   store: Shrine::Storage::FileSystem.new("public", prefix: "uploads/store"),
@@ -411,4 +502,5 @@ Shrine doesn't have an equivalent to this, but the [Regenerating versions]
 guide provides some useful tips on how to do this.
 
 [file]: http://linux.die.net/man/1/file
-[Regenerating versions]: http://shrinerb.com/rdoc/files/doc/regenerating_versions_md.html
+[Reprocessing versions]: http://shrinerb.com/rdoc/files/doc/regenerating_versions_md.html
+[direct S3 uploads]: http://shrinerb.com/rdoc/files/doc/direct_s3_md.html