shrine 1.0.0 → 1.1.0

data/doc/changing_location.md

@@ -0,0 +1,50 @@
+# Changing Location of Files
+
+You have a production app with already uploaded attachments. However, you've
+realized that the existing store folder structure for attachments isn't working
+for you.
+
+The first step is to change the location, either by using the `pretty_location`
+plugin:
+
+```rb
+Shrine.plugin :pretty_location
+```
+
+Or by overriding `#generate_location`:
+
+```rb
+class MyUploader < Shrine
+  def generate_location(io, context)
+    "#{context[:record].class}/#{context[:record.id]}/#{io.original_filename}"
+  end
+end
+```
+
+After you've deployed this change, all existing attachments on old locations
+will continue to work properly. The next step is to run a script that will
+move those to new locations. The easiest way to do that is to reupload them:
+
+```rb
+Shrine.plugin :migration_helpers # before the model is loaded
+Shrine.plugin :multi_delete # for deleting multiple files at once
+```
+```rb
+old_avatars = []
+
+User.paged_each do |user|
+  user.update_avatar do |avatar|
+    old_avatars << avatar
+    user.avatar_store.upload(avatar)
+  end
+end
+
+if old_avatars.any?
+  # you'll have to change this code slightly if you're using versions
+  uploader = old_avatars.first.uploader
+  uploader.delete(old_avatars)
+end
+```
+
+And now all your existing attachments should be happily living on new
+locations.

data/doc/creating_plugins.md

@@ -1,4 +1,4 @@
-# Creating a new plugin
+# Creating a New Plugin
 
 Shrine has a lot of plugins built-in, but you can also easily create your own.
 Simply put, a plugin is a module:
@@ -68,7 +68,7 @@ these modules, you can also make your plugin configurable:
 Shrine.plugin :my_plugin, foo: "bar"
 ```
 
-You can do this my adding a `.configure` method to your plugin, which will be
+You can do this by adding a `.configure` method to your plugin, which will be
 given any passed in arguments or blocks. Typically you'll want to save these
 options into Shrine's `opts`, so that you can access them inside of Shrine's
 methods.

data/doc/creating_storages.md

@@ -1,4 +1,6 @@
-# Creating a new storage
+# Creating a New Storage
+
+## Essentials
 
 Shrine ships with the FileSystem and S3 storages, but it's also easy to create
 your own. A storage is a class which has at least the following methods:
@@ -47,18 +49,46 @@ class Shrine
 end
 ```
 
-To check that your storage implements all these methods correctly, you can use
-`Shrine::Storage::Linter` in tests:
+If your storage doesn't control which id the uploaded file will have, you
+can modify the `id` variable:
 
 ```rb
-require "shrine/storage/linter"
+def upload(io, id, metadata = {})
+  actual_id = do_upload(io, id, metadata)
+  id.replace(actual_id)
+end
+```
 
-storage = Shrine::Storage::MyStorage.new(*args)
-Shrine::Storage::Linter.call(storage)
+Likewise, if you need to save some information into the metadata after upload,
+you can modify the metadata hash:
+
+```rb
+def upload(io, id, metadata = {})
+  additional_metadata = do_upload(io, id, metadata)
+  metadata.merge!(additional_metadata)
+end
 ```
 
-The linter will pass real files through your storage, and raise an error with
-an appropriate message if a part of the specification isn't satisfied.
+## Streaming
+
+If your storage can stream files by yielding chunks, you can add an additional
+`#stream` method:
+
+```rb
+class Shrine
+  module Storage
+    class MyStorage
+      # ...
+
+      def stream(id)
+        # yields chunks of the file
+      end
+
+      # ...
+    end
+  end
+end
+```
 
 ## Moving
 
@@ -76,7 +106,7 @@ class Shrine
       end
 
       def movable?(io, id)
-        # whether the given `io` is movable, to the location `id`
+        # whether the given `io` is movable to the location `id`
       end
 
       # ...
@@ -106,3 +136,34 @@ class Shrine
   end
 end
 ```
+
+## Linter
+
+To check that your storage implements all these methods correctly, you can use
+`Shrine::Storage::Linter`:
+
+```rb
+require "shrine/storage/linter"
+
+storage = Shrine::Storage::MyStorage.new(*args)
+linter = Shrine::Storage::Linter.new(storage)
+linter.call
+```
+
+The linter will test your methods with simple IO objects, and raise an error
+with an appropriate message if a part of the specification isn't satisfied.
+
+If you want to specify the IO object to use for testing (e.g. you need the IO
+to be an actual image), you can pass in a lambda which returns the IO when
+called:
+
+```rb
+linter.call(->{File.open("test/fixtures/image.jpg")})
+```
+
+If you don't want errors to be raised but rather only warnings, you can
+pass `action: :warn` when initializing
+
+```rb
+linter = Shrine::Storage::Linter.new(storage, action: :warn)
+```

data/doc/direct_s3.md

@@ -1,13 +1,23 @@
-# Direct uploads to S3
+# Direct Uploads to S3
 
 Probably the best way to do file uploads is to upload them directly to S3, and
-afterwards do processing in a background job. Direct S3 uploads are a bit more
-involved, so we'll explain the process.
+then upon saving the record when file is moved to a permanent place, put that
+and any additional file processing in the background. The goal of this guide
+is to provide instructions, as well as evaluate possible ways of doing this.
+
+```rb
+require "shrine/storage/s3"
+
+Shrine.storages = {
+  cache: Shrine::Storage::S3.new(prefix: "cache", **s3_options),
+  store: Shrine::Storage::S3.new(prefix: "store", **s3_options),
+}
+```
 
 ## Enabling CORS
 
-First thing that we need to do is enable CORS on our S3 bucket. You can do that
-by clicking on "Properties > Permissions > Add CORS Configuration", and
+First thing that you need to do is enable CORS on your S3 bucket. You can do
+that by clicking on "Properties > Permissions > Add CORS Configuration", and
 then just follow the Amazon documentation on how to write a CORS file.
 
 http://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html
@@ -15,50 +25,50 @@ http://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html
 Note that it may take some time for the CORS settings to be applied, due to
 DNS propagation.
 
-## Static upload
-
-If you're doing just a single file upload in your form, you can generate
-upfront the fields necessary for direct S3 uploads using
-`Shrine::Storage::S3#presign`. This method returns a [`Aws::S3::PresignedPost`]
-object, which has `#url` and `#fields`, which you could use like this:
+## File hash
 
-```erb
-<% presign = Shrine.storages[:cache].presign(SecureRandom.hex) %>
+Shrine's JSON representation of an uploaded file looks like this:
 
-<form action="<%= presign.url %>" method="post" enctype="multipart/form-data">
-  <input type="file" name="file">
-  <% presign.fields.each do |name, value| %>
-    <input type="hidden" name="<%= name %>" value="<%= value %>">
-  <% end %>
-</form>
+```rb
+{
+  "id": "349234854924394", # requied
+  "storage": "cache", # required
+  "metadata": {
+    "size": 45461, # required
+    "filename": "foo.jpg", # optional
+    "mime_type": "image/jpeg", # optional
+  }
+}
 ```
 
-You can also pass additional options to `#presign`:
+The `id`, `storage` and `metadata.size` fields are required, and the rest of
+the metadata is optional. After uploading the file to S3, you need to construct
+this JSON and assign it to the hidden attachment field in the form.
 
-```rb
-Shrine.storages[:cache].presign(SecureRandom.hex,
-  content_length_range: 0..(5*1024*1024), # Limit of 5 MB
-  success_action_redirect: webhook_url,   # Tell S3 where to redirect
-  # ...
-)
-```
+## Strategy A (dynamic)
 
-## Dynamic upload
+* Best user experience
+* Single or multiple file uploads
+* Some JavaScript needed
 
-If the frontend is separate from the backend, or you want to do multiple file
-uploads, you need to generate these presigns dynamically. The `direct_upload`
-plugins provides a route just for that:
+You can configure the `direct_upload` plugin to expose the presign route, and
+mount the endpoint:
 
 ```rb
 plugin :direct_upload, presign: true
 ```
+```rb
+Rails.application.routes.draw do
+  mount ImageUploader::UploadEndpoint => "attachments/image"
+end
+```
 
 This gives the endpoint a `GET /:storage/presign` route, which generates a
 presign object and returns it as JSON:
 
 ```rb
 {
-  "url" => "https://shrine-testing.s3-eu-west-1.amazonaws.com",
+  "url" => "https://my-bucket.s3-eu-west-1.amazonaws.com",
   "fields" => {
     "key" => "b7d575850ba61b44c8a9ff889dfdb14d88cdc25f8dd121004c8",
     "policy" => "eyJleHBpcmF0aW9uIjoiMjAxNS0QwMToxMToyOVoiLCJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJzaHJpbmUtdGVzdGluZyJ9LHsia2V5IjoiYjdkNTc1ODUwYmE2MWI0NGU3Y2M4YTliZmY4OGU5ZGZkYjE2NTQ0ZDk4OGNkYzI1ZjhkZDEyMTAwNGM4In0seyJ4LWFtei1jcmVkZW50aWFsIjoiQUtJQUlKRjU1VE1aWlk0NVVUNlEvMjAxNTEwMjQvZXUtd2VzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LHsieC1hbXotYWxnb3JpdGhtIjoiQVdTNC1ITUFDLVNIQTI1NiJ9LHsieC1hbXotZGF0ZSI6IjIwMTUxMDI0VDAwMTEyOVoifV19",
@@ -70,47 +80,108 @@ presign object and returns it as JSON:
 }
 ```
 
-You can use this data in a similar way as with static upload. See
-the [example app] for how multiple file upload to S3 can be done using
-[jQuery-File-Upload].
-
-If you want to pass additional options to `Storage::S3#presign`, you can pass
-a block to `:presign`:
+When the user attaches a file, you should first request the presign object from
+the direct endpoint, and then upload the file to the given URL with the given
+fields. For uploading to S3 you can use any of the great JavaScript libraries
+out there, [jQuery-File-Upload] for example.
+
+After the upload you create a JSON representation of the uploaded file and
+usually write it to the hidden attachment field in the form:
+
+```js
+var image = {
+  id: /cache\/(.+)/.exec(key)[1], # we have to remove the prefix part
+  storage: 'cache',
+  metadata: {
+    size:      data.files[0].size,
+    filename:  data.files[0].name,
+    mime_type: data.files[0].type,
+  }
+}
 
-```rb
-plugin :direct_upload, presign: ->(request) do # yields a Roda request object
-  {success_action_redirect: "http://example.com/webhook"}
-end
+$('input[type=file]').prev().value(JSON.stringify(image))
 ```
 
-## File hash
+It's generally a good idea to disable the submit button until the file is
+uploaded, as well as display a progress bar. See the [example app] for the
+working implementation of multiple direct S3 uploads.
 
-Once you've uploaded the file to S3, you need to create the representation of
-the uploaded file which Shrine will understand. This is how a Shrine's uploaded
-file looks like:
+## Strategy B (static)
 
-```rb
-{
-  "id" => "349234854924394",
-  "storage" => "cache",
-  "metadata" => {
-    "size" => 45461,
-    "filename" => "foo.jpg",     # optional
-    "mime_type" => "image/jpeg", # optional
+* Basic user experience
+* Only for single uploads
+* No JavaScript needed
+
+An alternative to the previous strategy is generating a file upload form that
+submits synchronously to S3, and then redirects back to your application.
+For that you can use `Shrine::Storage::S3#presign`, which returns a
+[`Aws::S3::PresignedPost`] object, which has `#url` and `#fields`:
+
+```erb
+<% presign = Shrine.storages[:cache].presign(SecureRandom.hex, success_action_redirect: new_album_url) %>
+
+<form action="<%= presign.url %>" method="post" enctype="multipart/form-data">
+  <input type="file" name="file">
+  <% presign.fields.each do |name, value| %>
+    <input type="hidden" name="<%= name %>" value="<%= value %>">
+  <% end %>
+  <input type="submit" value="Upload">
+</form>
+```
+
+After the file is submitted, S3 will redirect to the URL you specified and
+include the object key as a query param:
+
+```erb
+<%
+  cached_file = {
+    storage: "cache",
+    id: params[:key][/cache\/(.+)/, 1], # we have to remove the prefix part,
+    metadata: {
+      size: Shrine.storages[:cache].bucket.object(params[:key]).size,
+    }
   }
-}
+%>
+
+<form action="/albums" method="post">
+  <input type="hidden" name="album[image]" value="<%= cached_file.to_json %>">
+  <input type="submit" value="Save">
+</form>
 ```
 
-The `id`, `storage` and `metadata.size` fields are required, and the rest of
-the metadata is optional. You need to assign a JSON representation of this
-hash to the model in place of the attachment.
+Notice that we needed to fetch and assign the size of the uploaded file. This
+is because this hash is later transformed into an IO which requires `#size`
+to be non-nil (and it is read from the metadata field).
+
+## Eventual consistency
+
+When uploading objects to Amazon S3, sometimes they may not be available
+immediately. This can be a problem when using direct S3 uploads, because
+usually in this case you're using S3 for both cache and store, so the S3 object
+is moved to store soon after caching.
+
+> Amazon S3 provides eventual consistency for some operations, so it is
+> possible that new data will not be available immediately after the upload,
+> which could result in an incomplete data load or loading stale data. COPY
+> operations where the cluster and the bucket are in different regions are
+> eventually consistent. All regions provide read-after-write consistency for
+> uploads of new objects with unique object keys. For more information about
+> data consistency, see [Amazon S3 Data Consistency Model] in the *Amazon Simple
+> Storage Service Developer Guide*.
+
+This means that in certain cases copying from cache to store can fail if it
+happens immediately after uploading to cache. If you start noticing these
+errors, and you're using `backgrounding` plugin, you can tell your
+backgrounding library to perform the job with a delay:
 
 ```rb
-user.avatar = '{"id":"43244656","storage":"cache",...}'
+Shrine.plugin :backgrounding
+Shrine::Attacher.promote do |data|
+  UploadJob.perform_in(60, data) # tells a Sidekiq worker to perform in 1 minute
+end
 ```
 
-In a form you can assign this to an appropriate "hidden" field.
-
 [`Aws::S3::PresignedPost`]: http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Bucket.html#presigned_post-instance_method
 [example app]: https://github.com/janko-m/shrine-example
 [jQuery-File-Upload]: https://github.com/blueimp/jQuery-File-Upload
+[Amazon S3 Data Consistency Model]: http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyMode

data/doc/migrating_storage.md

@@ -1,4 +1,4 @@
-# Migrating to another storage
+# Migrating to Another Storage
 
 While your application is live in production and performing uploads, it may
 happen that you decide you want to change your storage (the `:store`). Shrine
@@ -12,8 +12,8 @@ current store (let's say that you're migrating from FileSystem to S3):
 
 ```rb
 Shrine.storages = {
-  cache: Shrine::Storage::FileSystem.new("public", subdirectory: "uploads/cache"),
-  store: Shrine::Storage::FileSystem.new("public", subdirectory: "uploads/store"),
+  cache: Shrine::Storage::FileSystem.new("public", prefix: "uploads/cache"),
+  store: Shrine::Storage::FileSystem.new("public", prefix: "uploads/store"),
   new_store: Shrine::Storage::S3.new(**s3_options),
 }
 
@@ -30,11 +30,12 @@ files to the new storage, and update the records. This is how you can do it
 if you're using Sequel:
 
 ```rb
-Shrine.plugin :migration_helpers
-
+Shrine.plugin :migration_helpers # before the model is loaded
+```
+```rb
 User.paged_each do |user|
   user.update_avatar do |avatar|
-    user.avatar_store.upload(avatar)
+    user.avatar_store.upload(avatar, {record: user, name: :avatar})
   end
 end
 
@@ -53,7 +54,7 @@ be `:store` again):
 
 ```rb
 Shrine.storages = {
-  cache: Shrine::Storage::FileSystem.new("public", subdirectory: "uploads/cache"),
+  cache: Shrine::Storage::FileSystem.new("public", prefix: "uploads/cache"),
   store: Shrine::Storage::S3.new(**s3_options),
 }
 
@@ -64,11 +65,12 @@ Shrine.storages[:new_store] = Shrine.storages[:store]
 Sequel it would be something like:
 
 ```rb
-Shrine.plugin :migration_helpers
-
+Shrine.plugin :migration_helper # before the model is loaded
+```
+```rb
 User.paged_each do |user|
   user.update_avatar do |avatar|
-    avatar.to_json.gsub('new_store', 'store')
+    avatar.to_json.gsub('"new_store"', '"store"')
   end
 end