shrine 3.1.0 → 3.4.0

data/doc/plugins/atomic_helpers.md

@@ -3,14 +3,48 @@ title: Atomic Helpers
 ---
 
 The [`atomic_helpers`][atomic_helpers] plugin provides API for retrieving and
-persisting attachments in a concurrency-safe way, which is useful when using
-the `backgrounding` plugin. The database plugins (`activerecord` and `sequel`)
-implement atomic promotion and atomic persistence on top of this plugin.
+persisting attachments in a concurrency-safe way, which is especially useful
+when using the `backgrounding` plugin. The database plugins (`activerecord`
+and `sequel`) implement atomic promotion and atomic persistence on top of this
+plugin.
 
 ```rb
 plugin :atomic_helpers
 ```
 
+## Problem Statement
+
+What happens if two different processors (web workers, background jobs,
+command-line executions, whatever) try to edit a shrine attachment
+concurrently? The kinds of edits typically made include: "promoting a file",
+moving it to a different storage and persisting that change in the model;
+adding or changing a derivative; adding or changing a metadata element.
+
+There are two main categories of "race condition":
+
+1. The file could be switched out from under you. If you were promoting a file,
+but some other process has *changed* the attachment, you don't want to
+overwrite it with the promomoted version of the *prior* attacchment. Likewise,
+if you were adding metadata or a derivative, they would be corresponding to a
+certain attachment, and you don't want to accidentally add them to a now changed
+attacchment for which they are inappropriate.
+
+2. Overwriting each other's edits. Since all shrine (meta)data is stored in a
+single JSON hash, standard implementations will write the entire JSON hash at
+once to a rdbms column or other store. If two processes both read in the hash,
+make a change to different keys in it, and then write it back out, the second
+process to write will 'win' and overwrite changes made by the first.
+
+The atomic helpers give you tools to avoid both of these sorts of race
+conditions, under conditions of concurrent editing.
+
+## High-level ORM helpers
+
+If you are using the `sequel` or `activerecord` plugins, they give you two
+higher-level helpers: `atomic_persist` and `atomic_promote`. See the
+[persistence]  documentation for more.
+
+
 ## Retrieving
 
 The `Attacher.retrieve` method provided by the plugin instantiates an attacher
@@ -177,3 +211,7 @@ end
 ```
 
 [atomic_helpers]: https://github.com/shrinerb/shrine/blob/master/lib/shrine/plugins/atomic_helpers.rb
+
+[persistence]: https://shrinerb.com/docs/plugins/persistence
+
+[backgrounding]: https://shrinerb.com/docs/plugins/backgrounding

data/doc/plugins/backgrounding.md

@@ -187,12 +187,22 @@ and make the execution synchronous, you can override them on the attacher level
 and call the default behaviour:
 
 ```rb
-photo.image_attacher.promote_block(&:promote)
-photo.image_attacher.destroy_block(&:destroy)
+photo.image_attacher.promote_block { promote } # promote synchronously
+photo.image_attacher.destroy_block { destroy } # destroy synchronously
 
 # ... now promotion and deletion will be synchronous ...
 ```
 
+You can also do this on the class level if you want to disable backgrounding
+that was set up by a superclass:
+
+```rb
+class MyUploader < Shrine
+  Attacher.promote_block { promote } # promote synchronously
+  Attacher.destroy_block { destroy } # destroy synchronously
+end
+```
+
 [backgrounding]: https://github.com/shrinerb/shrine/blob/master/lib/shrine/plugins/backgrounding.rb
 [derivatives]: https://shrinerb.com/docs/plugins/derivatives
 [atomic_helpers]: https://shrinerb.com/docs/plugins/atomic_helpers

data/doc/plugins/column.md

@@ -66,23 +66,48 @@ If you want to load attachment from a Hash, use `Attacher.from_data` or
 
 ## Serializer
 
-By default the `JSON` standard library is used as the serializer, but you can
-use your own serializer. The serializer object needs to implement `#dump` and
-`#load` methods.
+By default, the `JSON` standard library is used for serializing hash data. With
+the [`model`][model] and [`entity`][entity] plugin, the data is serialized
+before writing to and deserialized after reading from the data attribute.
+
+You can also use your own serializer via the `:serializer` option. The
+serializer object needs to implement `#dump` and `#load` methods:
+
+```rb
+class MyDataSerializer
+  def self.dump(data)
+    data #=> { "id" => "...", "storage" => "...", "metadata" => { ... } }
+
+    JSON.generate(data) # serialize data, e.g. into JSON
+  end
+
+  def self.load(data)
+    data #=> '{"id":"...", "storage":"...", "metadata": {...}}'
+
+    JSON.parse(data) # deserialize data, e.g. from JSON
+  end
+end
+
+plugin :column, serializer: MyDataSerializer
+```
+
+Some serialization libraries such as [Oj] and [MessagePack] already implement
+this interface, which simplifies the configuration:
 
 ```rb
-require "oj"
+require "oj" # https://github.com/ohler55/oj
 
-plugin :column, serializer: Oj # use custom serializer
+plugin :column, serializer: Oj
 ```
 
-If you want to disable serialization, you can set serializer to `nil`.
+If you want to disable serialization and work with hashes directly, you can set
+`:serializer` to `nil`:
 
 ```rb
 plugin :column, serializer: nil # disable serialization
 ```
 
-You can also change the serializer on the attacher level:
+The serializer can also be changed for a particular attacher instance:
 
 ```rb
 Shrine::Attacher.new(column_serializer: Oj)  # use custom serializer
@@ -90,3 +115,7 @@ Shrine::Attacher.new(column_serializer: nil) # disable serialization
 ```
 
 [column]: https://github.com/shrinerb/shrine/blob/master/lib/shrine/plugins/column.rb
+[model]: https://shrinerb.com/docs/plugins/model
+[entity]: https://shrinerb.com/docs/plugins/entity
+[Oj]: https://github.com/ohler55/oj
+[MessagePack]: https://github.com/msgpack/msgpack-ruby

data/doc/plugins/default_url.md

@@ -8,7 +8,7 @@ returned when there is no attached file.
 ```rb
 plugin :default_url
 
-Attacher.default_url do |options|
+Attacher.default_url do |**options|
   "/#{name}/missing.jpg"
 end
 ```
@@ -28,7 +28,7 @@ Any URL options passed will be available in the default URL block:
 attacher.url(foo: "bar")
 ```
 ```rb
-Attacher.default_url do |options|
+Attacher.default_url do |**options|
   options #=> { foo: "bar" }
 end
 ```
@@ -37,12 +37,15 @@ The default URL block is evaluated in the context of an instance of
 `Shrine::Attacher`.
 
 ```rb
-Attacher.default_url do |options|
+Attacher.default_url do |**options|
   self    #=> #<Shrine::Attacher>
 
+  file    #=> #<Shrine::UploadedFile>
   name    #=> :avatar
   record  #=> #<User>
   context #=> { ... }
+
+  # ...
 end
 ```
 

data/doc/plugins/derivatives.md

@@ -2,7 +2,7 @@
 title: Derivatives
 ---
 
-The derivatives plugin allows storing processed files ("derivatives") alongside
+The [`derivatives`][derivatives] plugin allows storing processed files ("derivatives") alongside
 the main attached file. The processed file data will be saved together with the
 main attachment data in the same record attribute.
 
@@ -38,49 +38,43 @@ class ImageUploader < Shrine
 end
 ```
 ```rb
-class Photo < Model(:image_data)
-  include ImageUploader::Attachment(:image)
-end
-```
-```rb
 photo = Photo.new(image: file)
-photo.image_derivatives! # calls derivatives processor and uploads results
+photo.image_derivatives! # creates derivatives
 photo.save
 ```
 
-If you're allowing the attached file to be updated later on, in your update
-route make sure to create derivatives for new attachments:
+You can then retrieve the URL of a processed derivative:
 
 ```rb
-photo.image_derivatives! if photo.image_changed?
-```
-
-You can then retrieve created derivatives as follows:
-
-```rb
-photo.image(:large)           #=> #<Shrine::UploadedFile ...>
-photo.image(:large).url       #=> "https://s3.amazonaws.com/path/to/large.jpg"
-photo.image(:large).size      #=> 43843
-photo.image(:large).mime_type #=> "image/jpeg"
+photo.image_url(:large) #=> "https://s3.amazonaws.com/path/to/large.jpg"
 ```
 
-The derivatives data is stored in the `#<name>_data` record attribute, alongside
-the main file data:
+The derivatives data is stored in the `<attachment>_data` column alongside the
+main file:
 
 ```rb
 photo.image_data #=>
 # {
-#   "id": "original.jpg",
+#   "id": "path/to/original.jpg",
 #   "store": "store",
 #   "metadata": { ... },
 #   "derivatives": {
-#     "small": { "id": "small.jpg", "storage": "store", "metadata": { ... } },
-#     "medium": { "id": "medium.jpg", "storage": "store", "metadata": { ... } },
-#     "large": { "id": "large.jpg", "storage": "store", "metadata": { ... } },
+#     "small": { "id": "path/to/small.jpg", "storage": "store", "metadata": { ... } },
+#     "medium": { "id": "path/to/medium.jpg", "storage": "store", "metadata": { ... } },
+#     "large": { "id": "path/to/large.jpg", "storage": "store", "metadata": { ... } },
 #   }
 # }
 ```
 
+And they can be retrieved as `Shrine::UploadedFile` objects:
+
+```rb
+photo.image(:large)           #=> #<Shrine::UploadedFile id="path/to/large.jpg" storage=:store metadata={...}>
+photo.image(:large).url       #=> "https://s3.amazonaws.com/path/to/large.jpg"
+photo.image(:large).size      #=> 5825949
+photo.image(:large).mime_type #=> "image/jpeg"
+```
+
 ## Retrieving derivatives
 
 The list of stored derivatives can be retrieved with `#<name>_derivatives`:
@@ -135,7 +129,7 @@ You can use the [`default_url`][default_url] plugin to set up URL fallbacks:
 
 ```rb
 Attacher.default_url do |derivative: nil, **|
-  "https://my-app.com/fallbacks/#{derivative}.jpg" if derivative
+  "/fallbacks/#{derivative}.jpg" if derivative
 end
 ```
 ```rb
@@ -203,6 +197,19 @@ attacher.create_derivatives(different_source) # pass a different source file
 attacher.create_derivatives(foo: "bar")       # pass custom options to the processor
 ```
 
+### Create on promote
+
+You can also have derivatives created automatically on promotion:
+
+```rb
+Shrine.plugin :derivatives, create_on_promote: true
+```
+```rb
+attacher.assign(file)
+attacher.finalize # creates derivatives on promotion
+attacher.derivatives #=> { small: ..., medium: ..., large: ... }
+```
+
 ### Naming processors
 
 If you want to have multiple processors for an uploader, you can assign each
@@ -279,11 +286,14 @@ The storage block is evaluated in the context of a `Shrine::Attacher` instance:
 
 ```rb
 Attacher.derivatives_storage do |derivative|
-  self   #=> #<Shrine::Attacher>
+  self    #=> #<Shrine::Attacher>
 
+  file    #=> #<Shrine::UploadedFile>
   record  #=> #<Photo>
   name    #=> :image
   context #=> { ... }
+
+  # ...
 end
 ```
 
@@ -352,6 +362,7 @@ which allows you to change your processing logic based on the record data.
 Attacher.derivatives :my_processor do |original|
   self    #=> #<Shrine::Attacher>
 
+  file    #=> #<Shrine::UploadedFile>
   record  #=> #<Photo>
   name    #=> :image
   context #=> { ... }
@@ -390,7 +401,8 @@ attacher.process_derivatives(:my_processor) # downloads attached file and passes
 
 If you want to use a different source file, you can pass it in to the process
 call. Typically you'd pass a local file on disk. If you pass a
-`Shrine::UploadedFile` object, it will be automatically downloaded to disk.
+`Shrine::UploadedFile` object or another IO-like object, it will be
+automatically downloaded/copied to a local TempFile on disk.
 
 ```rb
 # named processor:
@@ -410,17 +422,31 @@ attacher.file.download do |original|
 end
 ```
 
+If a processor might not always need a local source file, you avoid a
+potentially expensive download/copy by registering the processor with
+`download: false`, in which case the source file will be passed to the
+processor as is.
+
+```rb
+Attacher.derivatives :my_processor, download: false do |source|
+  source #=> Could be File, Shrine::UploadedFile, or other IO-like object
+  shrine_class.with_file(source) do |file|
+    # can force download/copy if necessary with `with_file`,
+  end
+end
+```
+
 ## Adding derivatives
 
 If you already have processed files that you want to save, you can do that with
 `Attacher#add_derivatives`:
 
 ```rb
-attacher.add_derivatives(
+attacher.add_derivatives({
   one: file_1,
   two: file_2,
   # ...
-)
+})
 
 attacher.derivatives #=>
 # {
@@ -434,7 +460,7 @@ New derivatives will be merged with existing ones:
 
 ```rb
 attacher.derivatives #=> { one: #<Shrine::UploadedFile> }
-attacher.add_derivatives(two: two_file)
+attacher.add_derivatives({ two: two_file })
 attacher.derivatives #=> { one: #<Shrine::UploadedFile>, two: #<Shrine::UploadedFile> }
 ```
 
@@ -442,7 +468,7 @@ The merging is deep, so the following will work as well:
 
 ```rb
 attacher.derivatives #=> { nested: { one: #<Shrine::UploadedFile> } }
-attacher.add_derivatives(nested: { two: two_file })
+attacher.add_derivatives({ nested: { two: two_file } })
 attacher.derivatives #=> { nested: { one: #<Shrine::UploadedFile>, two: #<Shrine::UploadedFile> } }
 ```
 
@@ -453,6 +479,11 @@ For adding a single derivative, you can also use the singular
 attacher.add_derivative(:thumb, thumbnail_file)
 ```
 
+> Note that new derivatives will replace any existing derivatives living under
+the same key, but won't delete them. If this is your case, make sure to save a
+reference to the old derivatives before assigning new ones, and then delete
+them after persisting the change.
+
 Any options passed to `Attacher#add_derivative(s)` will be forwarded to
 [`Attacher#upload_derivatives`](#uploading-derivatives).
 
@@ -469,11 +500,11 @@ If you want to upload processed files without setting them, you can use
 `Attacher#upload_derivatives`:
 
 ```rb
-derivatives = attacher.upload_derivatives(
+derivatives = attacher.upload_derivatives({
   one: file_1,
   two: file_2,
   # ...
-)
+})
 
 derivatives #=>
 # {
@@ -549,7 +580,7 @@ If you want to save already uploaded derivatives, you can use
 
 ```rb
 attacher.derivatives #=> { one: #<Shrine::UploadedFile> }
-attacher.merge_derivatives attacher.upload_derivatives(two: two_file)
+attacher.merge_derivatives attacher.upload_derivatives({ two: two_file })
 attacher.derivatives #=> { one: #<Shrine::UploadedFile>, two: #<Shrine::UploadedFile> }
 ```
 
@@ -557,10 +588,15 @@ This does a deep merge, so the following will work as well:
 
 ```rb
 attacher.derivatives #=> { nested: { one: #<Shrine::UploadedFile> } }
-attacher.merge_derivatives attacher.upload_derivatives(nested: { two: two_file })
+attacher.merge_derivatives attacher.upload_derivatives({ nested: { two: two_file } })
 attacher.derivatives #=> { nested: { one: #<Shrine::UploadedFile>, two: #<Shrine::UploadedFile> } }
 ```
 
+> Note that new derivatives will replace any existing derivatives living under
+the same key, but won't delete them. If this is your case, make sure to save a
+reference to the old derivatives before assigning new ones, and then delete
+them after persisting the change.
+
 The `Attacher#merge_derivatives` method is thread-safe.
 
 ### Setting derivatives
@@ -570,7 +606,7 @@ If instead of adding you want to *override* existing derivatives, you can use
 
 ```rb
 attacher.derivatives #=> { one: #<Shrine::UploadedFile> }
-attacher.set_derivatives attacher.upload_derivatives(two: two_file)
+attacher.set_derivatives attacher.upload_derivatives({ two: two_file })
 attacher.derivatives #=> { two: #<Shrine::UploadedFile> }
 ```
 
@@ -678,7 +714,7 @@ You can store derivatives even if there is no main attached file:
 
 ```rb
 attacher.file #=> nil
-attacher.add_derivatives(one: one_file, two: two_file)
+attacher.add_derivatives({ one: one_file, two: two_file })
 attacher.data #=>
 # {
 #   "derivatives" => {
@@ -756,11 +792,13 @@ plugin :derivatives
 Processing derivatives will trigger a `derivatives.shrine` event with the
 following payload:
 
-| Key                  | Description                            |
-| :--                  | :----                                  |
-| `:processor`         | Name of the derivatives processor      |
-| `:processor_options` | Any options passed to the processor    |
-| `:uploader`          | The uploader class that sent the event |
+| Key                  | Description                                |
+| :--                  | :----                                      |
+| `:processor`         | Name of the derivatives processor          |
+| `:processor_options` | Any options passed to the processor        |
+| `:io`                | The source file passed to the processor    |
+| `:attacher`          | The attacher instance doing the processing |
+| `:uploader`          | The uploader class that sent the event     |
 
 A default log subscriber is added as well which logs these events:
 
@@ -776,7 +814,7 @@ plugin :derivatives, log_subscriber: -> (event) {
 }
 ```
 ```
-{"name":"derivatives","duration":2133,"processor":"thumbnails","processor_options":{},"uploader":"ImageUploader"}
+{"name":"derivatives","duration":2133,"processor":"thumbnails","processor_options":{},"io":"#<File:...>","uploader":"ImageUploader"}
 ```
 
 Or disable logging altogether:
@@ -785,6 +823,7 @@ Or disable logging altogether:
 plugin :derivatives, log_subscriber: nil
 ```
 
+[derivatives]: https://github.com/shrinerb/shrine/blob/master/lib/shrine/plugins/derivatives.rb
 [default_url]: https://shrinerb.com/docs/plugins/default_url
 [entity]: https://shrinerb.com/docs/plugins/entity
 [model]: https://shrinerb.com/docs/plugins/model