shrine 2.19.3 → 3.6.0

data/doc/plugins/atomic_helpers.md

@@ -0,0 +1,217 @@
+---
+title: Atomic Helpers
+---
+
+The [`atomic_helpers`][atomic_helpers] plugin provides API for retrieving and
+persisting attachments in a concurrency-safe way, which is especially useful
+when using the `backgrounding` plugin. The database plugins (`activerecord`
+and `sequel`) implement atomic promotion and atomic persistence on top of this
+plugin.
+
+```rb
+plugin :atomic_helpers
+```
+
+## Problem Statement
+
+What happens if two different processors (web workers, background jobs,
+command-line executions, whatever) try to edit a shrine attachment
+concurrently? The kinds of edits typically made include: "promoting a file",
+moving it to a different storage and persisting that change in the model;
+adding or changing a derivative; adding or changing a metadata element.
+
+There are two main categories of "race condition":
+
+1. The file could be switched out from under you. If you were promoting a file,
+but some other process has *changed* the attachment, you don't want to
+overwrite it with the promomoted version of the *prior* attacchment. Likewise,
+if you were adding metadata or a derivative, they would be corresponding to a
+certain attachment, and you don't want to accidentally add them to a now changed
+attacchment for which they are inappropriate.
+
+2. Overwriting each other's edits. Since all shrine (meta)data is stored in a
+single JSON hash, standard implementations will write the entire JSON hash at
+once to a rdbms column or other store. If two processes both read in the hash,
+make a change to different keys in it, and then write it back out, the second
+process to write will 'win' and overwrite changes made by the first.
+
+The atomic helpers give you tools to avoid both of these sorts of race
+conditions, under conditions of concurrent editing.
+
+## High-level ORM helpers
+
+If you are using the `sequel` or `activerecord` plugins, they give you two
+higher-level helpers: `atomic_persist` and `atomic_promote`. See the
+[persistence]  documentation for more.
+
+
+## Retrieving
+
+The `Attacher.retrieve` method provided by the plugin instantiates an attacher
+from a record instance, attachment name and attachment data, asserting that the
+given attachment data matches the attached file on the record.
+
+```rb
+# with a model instance
+Shrine::Attacher.retrieve(
+  model: photo,
+  name:  :image,
+  file:  { "id" => "abc123", "storage" => "cache" },
+)
+#=> #<Shrine::Attacher ...>
+
+# with an entity instance
+Shrine::Attacher.retrieve(
+  entity: photo,
+  name:   :image,
+  file:   { "id" => "abc123", "storage" => "cache" },
+)
+#=> #<Shrine::Attacher ...>
+```
+
+If the record has `Shrine::Attachment` included, the `#<name>_attacher` method
+will be called on the record, which will return the correct attacher class.
+
+```rb
+class Photo
+  include ImageUploader::Attachment(:image)
+end
+```
+```rb
+Shrine::Attacher.retrieve(model: photo, name: :image, file: { ... })
+#=> #<ImageUploader::Attacher ...>
+```
+
+Otherwise it will call `Attacher.from_model`/`Attacher.from_entity` from the
+`model`/`entity` plugin, in which case you need to make sure to call
+`Attacher.retrieve` on the appropriate attacher class.
+
+```rb
+ImageUploader::Attacher.retrieve(entity: photo, name: :image, file: { ... })
+#=> #<ImageUploader::Attacher ...>
+```
+
+If the attached file on the record doesn't match the provided attachment data,
+a `Shrine::AttachmentChanged` exception is raised. Note that metadata is
+allowed to differ, Shrine will only compare location and storage of the file.
+
+```rb
+photo.image_data #=> '{"id":"foo","storage":"store","metadata":{...}}'
+
+Shrine::Attacher.retrieve(
+  model: photo,
+  name: :image,
+  file: { "id" => "bar", "storage" => "store" },
+)
+# ~> Shrine::AttachmentChanged: attachment has changed
+```
+
+### File data
+
+The `Attacher#file_data` method can be used for sending the attached file data
+into a background job. It returns only location and storage of the attached
+file, leaving out any metadata or derivatives data that `Attacher#data` would
+return. This way the background job payload is kept light.
+
+```rb
+attacher.file_data #=> { "id" => "abc123", "storage" => "store" }
+```
+
+This value can then be passed as the `:file` argument to
+`Shrine::Attacher.retrieve`.
+
+## Promoting
+
+The `Attacher#abstract_atomic_promote` method provided by the plugin promotes
+the cached file to permanent storage, reloads the record to check whether the
+attachment hasn't changed, and if not persists the promoted file.
+
+Internally it calls `Attacher#abstract_atomic_persist` to do the persistence,
+forwarding `:reload` and `:persist` options as well as a given block to it (see
+the next section for more details).
+
+```rb
+# in the controller
+attacher.attach_cached(io)
+attacher.cached? #=> true
+```
+```rb
+# in a background job
+attacher.abstract_atomic_promote(reload: -> (&block) { ... }, persist: -> { ... })
+attacher.stored? #=> true
+```
+
+If the attachment has changed during promotion, the promoted file is deleted and
+a `Shrine::AttachmentChanged` exception is raised.
+
+If you want to execute some code after the attachment change check but before
+persistence, you can pass a block:
+
+```rb
+attacher.abstract_atomic_promote(**options) do |reloaded_attacher|
+  # this will be executed before persistence
+end
+```
+
+Any additional options to `Attacher#abstract_atomic_promote` are forwarded to
+`Attacher#promote`.
+
+## Persisting
+
+The `Attacher#abstract_atomic_persist` method reloads the record to check
+whether the attachment hasn't changed, and if not persists the attachment.
+
+It requires reloader and persister to be passed in, as they will be specific to
+the database library you're using. The reloader needs to call the given block
+with the reloaded record, while the persister needs to persist the promoted
+file.
+
+```rb
+attacher.abstract_atomic_persist(
+  reload:  -> (&block) { ... }, # call the block with reloaded record
+  persist: -> { ... },          # persist promoted file
+)
+```
+
+To illustrate, this is how the `Attacher#atomic_promote` method provided by the
+`sequel` plugin is implemented:
+
+```rb
+attacher.abstract_atomic_persist(
+  reload: -> (&block) {
+    attacher.record.db.transaction do
+      block.call attacher.record.dup.lock! # the DB lock ensures no changes
+    end
+  },
+  persist: -> {
+    attacher.record.save_changes(validate: false)
+  }
+)
+```
+
+By default, the file currently set on the attacher will be considered the
+original file, and will be compared to the reloaded record. You can specify a
+different original file:
+
+```rb
+original_file = attacher.file
+
+attacher.set(new_file)
+
+attacher.abstract_atomic_persist(original_file, **options)
+```
+
+If you want to execute some code after the attachment change check but before
+persistence, you can pass a block:
+
+```rb
+attacher.abstract_atomic_persist(**options) do |reloaded_attacher|
+  # this will be executed before persistence
+end
+```
+
+[atomic_helpers]: https://github.com/shrinerb/shrine/blob/master/lib/shrine/plugins/atomic_helpers.rb
+
+[persistence]: https://shrinerb.com/docs/plugins/persistence
+
+[backgrounding]: https://shrinerb.com/docs/plugins/backgrounding

data/doc/plugins/backgrounding.md

@@ -1,150 +1,208 @@
-# Backgrounding
+---
+title: Backgrounding
+---
 
 The [`backgrounding`][backgrounding] plugin enables you to move promoting and
-deleting of files from record's lifecycle into background jobs. This is
-especially useful if you're doing processing and/or you're storing files on an
-external storage service.
+deleting of files into background jobs. This is especially useful if you're
+processing [derivatives] and storing files to a remote storage service.
 
-The plugin provides `Attacher.promote` and `Attacher.delete` methods, which
-allow you to hook up to promoting and deleting and spawn background jobs, by
-passing a block.
+```rb
+Shrine.plugin :backgrounding # load the plugin globally
+```
+
+## Setup
+
+Define background jobs that will promote and destroy attachments:
 
 ```rb
-Shrine.plugin :backgrounding
+class PromoteJob
+  include Sidekiq::Worker
+
+  def perform(attacher_class, record_class, record_id, name, file_data)
+    attacher_class = Object.const_get(attacher_class)
+    record         = Object.const_get(record_class).find(record_id) # if using Active Record
 
-# makes all uploaders use background jobs
-Shrine::Attacher.promote { |data| PromoteJob.perform_async(data) }
-Shrine::Attacher.delete { |data| DeleteJob.perform_async(data) }
+    attacher = attacher_class.retrieve(model: record, name: name, file: file_data)
+    attacher.atomic_promote
+  rescue Shrine::AttachmentChanged, ActiveRecord::RecordNotFound
+    # attachment has changed or record has been deleted, nothing to do
+  end
+end
 ```
+```rb
+class DestroyJob
+  include Sidekiq::Worker
 
-If you don't want to apply backgrounding for all uploaders, you can declare the
-hooks only for specific uploaders (in this case it's still recommended to keep
-the plugin loaded globally).
+  def perform(attacher_class, data)
+    attacher_class = Object.const_get(attacher_class)
 
-```rb
-class MyUploader < Shrine
-  # makes this uploader use background jobs
-  Attacher.promote { |data| PromoteJob.perform_async(data) }
-  Attacher.delete { |data| DeleteJob.perform_async(data) }
+    attacher = attacher_class.from_data(data)
+    attacher.destroy
+  end
 end
 ```
 
-The yielded `data` variable is a serializable hash containing all context
-needed for promotion/deletion. Now you just need to declare the job classes,
-and inside them call `Attacher.promote` or `Attacher.delete`, this time with
-the received data.
+Then, in your initializer, you can configure all uploaders to use these jobs:
 
 ```rb
-class PromoteJob
-  include Sidekiq::Worker
-  def perform(data)
-    Shrine::Attacher.promote(data)
-  end
+Shrine::Attacher.promote_block do
+  PromoteJob.perform_async(self.class.name, record.class.name, record.id, name.to_s, file_data)
+end
+Shrine::Attacher.destroy_block do
+  DestroyJob.perform_async(self.class.name, data)
 end
+```
 
-class DeleteJob
-  include Sidekiq::Worker
-  def perform(data)
-    Shrine::Attacher.delete(data)
+Alternatively, you can setup backgrounding only for specific uploaders:
+
+```rb
+class MyUploader < Shrine
+  Attacher.promote_block do
+    PromoteJob.perform_async(self.class.name, record.class.name, record.id, name.to_s, file_data)
+  end
+  Attacher.destroy_block do
+    DestroyJob.perform_async(self.class.name, data)
   end
 end
 ```
 
-This example used Sidekiq, but obviously you could just as well use any other
-backgrounding library. This setup will be applied globally for all uploaders.
+## How it works
+
+If backgrounding blocks are registered, they will be automatically called on
+`Attacher#promote_cached` and `Attacher#destroy_previous` (called by
+`Attacher#finalize`), and `Attacher#destroy_attached`.
+
+```rb
+attacher.assign(file)
+attacher.finalize         # spawns promote job
+attacher.destroy_attached # spawns destroy job
+```
+
+These methods are automatically called as part of the attachment flow if you're
+using `Shrine::Attachment` with a persistence plugin such as `activerecord` or
+`sequel`.
 
-If you're generating versions, and you want to process some versions in the
-foreground before kicking off a background job, you can use the `recache`
-plugin.
+```rb
+photo = Photo.new
+photo.image = file
+photo.save    # spawns promote job
+photo.destroy # spawns destroy job
+```
 
-In your application you can use `Attacher#cached?` and `Attacher#stored?`
-to differentiate between your background job being in progress and
-having completed.
+### Atomic promotion
+
+Inside the promote job, we use `Attacher.retrieve` and
+`Attacher#atomic_promote` for concurrency safety. These methods are provided
+by the [`atomic_helpers`][atomic_helpers] plugin, which is loaded automatically
+by your persistence plugin (`activerecord`, `sequel`).
 
 ```rb
-if user.avatar_attacher.cached? # background job is still in progress
-  # ...
-elsif user.avatar_attacher.stored? # background job has finished
-  # ...
-end
+attacher = Shrine::Attacher.retrieve(model: record, name: name, file: file_data)
+attacher.atomic_promote
 ```
 
-## `Attacher.promote` and `Attacher.delete`
+Without concurrency safety, promotion would look like this:
 
-In background jobs, `Attacher.promote` and `Attacher.delete` will resolve all
-necessary objects, and do the promotion/deletion. If `Attacher.find_record` is
-defined (which comes with ORM plugins), model instances will be treated as
-database records, with the `#id` attribute assumed to represent the primary
-key. Then promotion will have the following behaviour:
+```rb
+attacher = record.send(:"#{name}_attacher")
+attacher.promote
+attacher.persist
+```
 
-1. retrieves the database record
-  * if record is not found, it finishes
-  * if record is found but attachment has changed, it finishes
-2. uploads cached file to permanent storage
-3. reloads the database record
-  * if record is not found, it deletes the promoted files and finishes
-  * if record is found but attachment has changed, it deletes the promoted files and finishes
-4. updates the record with the promoted files
+## Registering backgrounding blocks
 
-Both `Attacher.promote` and `Attacher.delete` return a `Shrine::Attacher`
-instance (if the action hasn't aborted), so you can use it to perform
-additional tasks:
+The blocks registered by `Attacher.promote_block` and `Attacher#destroy_block`
+are by default evaluated in context of a `Shrine::Attacher` instance. You can
+also use the explicit version by declaring an attacher argument:
 
 ```rb
-def perform(data)
-  attacher = Shrine::Attacher.promote(data)
-  attacher.record.update(published: true) if attacher && attacher.record.is_a?(Post)
+Shrine::Attacher.promote_block do |attacher|
+  PromoteJob.perform_async(
+    attacher.class.name,
+    attacher.record.class.name,
+    attacher.record.id,
+    attacher.name.to_s,
+    attacher.file_data,
+  )
+end
+
+Shrine::Attacher.destroy_block do |attacher|
+  DestroyJob.perform_async(
+    attacher.class.name,
+    attacher.data,
+  )
 end
 ```
 
-### Plain models
+You can also register backgrounding blocks on attacher *instances* for more
+flexibility:
 
-You can also do backgrounding with plain models which don't represent database
-records; the plugin will use that mode if `Attacher.find_record` is not
-defined. In that case promotion will have the following behaviour:
+```rb
+photo.image_attacher.promote_block do |attacher|
+  PromoteJob.perform_async(
+    attacher.class.name,
+    attacher.record.class.name,
+    attacher.record.id,
+    attacher.name.to_s,
+    attacher.file_data,
+    current_user.id, # pass arguments known at the controller level
+  )
+end
 
-1. instantiates the model
-2. uploads cached file to permanent storage
-3. writes promoted files to the model instance
+photo.image = file
+photo.save # executes the promote block above
+```
 
-You can then retrieve the promoted files via the attacher object that
-`Attacher.promote` returns, and do any additional tasks if you need to.
+## Calling backgrounding blocks
 
-## `Attacher#_promote` and `Attacher#_delete`
+If you want to call backgrounding blocks directly, you can do that by calling
+`Attacher#promote_background` and `Attacher#destroy_background`.
 
-The plugin modifies `Attacher#_promote` and `Attacher#_delete` to call the
-registered blocks with serializable attacher data, and these methods are
-internally called by the attacher. `Attacher#promote` and `Attacher#delete!`
-remain synchronous.
+```rb
+attacher.promote_background # calls promote block directly
+attacher.destroy_background # calls destroy block directly
+```
+
+Any options passed to these methods will be forwarded to the background block:
 
 ```rb
-# asynchronous (spawn background jobs)
-attacher._promote
-attacher._delete(attachment)
+attacher.promote_background(foo: "bar")
+```
+```rb
+# with instance eval
+Shrine::Attacher.promote_block do |**options|
+  options #=> { foo: "bar" }
+end
 
-# synchronous
-attacher.promote
-attacher.delete!(attachment)
+# without instance eval
+Shrine::Attacher.promote_block do |attacher, **options|
+  options #=> { foo: "bar" }
+end
 ```
 
-## `Attacher.dump` and `Attacher.load`
+## Disabling backgrounding
 
-The plugin adds `Attacher.dump` and `Attacher.load` methods for serializing
-attacher object and loading it back up. You can use them to spawn background
-jobs for custom tasks.
+If you've registered backgrounding blocks, but want to temporarily disable them
+and make the execution synchronous, you can override them on the attacher level
+and call the default behaviour:
 
 ```rb
-data = Shrine::Attacher.dump(attacher)
-SomethingJob.perform_async(data)
+photo.image_attacher.promote_block { promote } # promote synchronously
+photo.image_attacher.destroy_block { destroy } # destroy synchronously
 
-# ...
+# ... now promotion and deletion will be synchronous ...
+```
 
-class SomethingJob
-  def perform(data)
-    attacher = Shrine::Attacher.load(data)
-    # ...
-  end
+You can also do this on the class level if you want to disable backgrounding
+that was set up by a superclass:
+
+```rb
+class MyUploader < Shrine
+  Attacher.promote_block { promote } # promote synchronously
+  Attacher.destroy_block { destroy } # destroy synchronously
 end
 ```
 
-[backgrounding]: /lib/shrine/plugins/backgrounding.rb
+[backgrounding]: https://github.com/shrinerb/shrine/blob/master/lib/shrine/plugins/backgrounding.rb
+[derivatives]: https://shrinerb.com/docs/plugins/derivatives
+[atomic_helpers]: https://shrinerb.com/docs/plugins/atomic_helpers

data/doc/plugins/cached_attachment_data.md

@@ -1,4 +1,6 @@
-# Cached Attachment Data
+---
+title: Cached Attachment Data
+---
 
 The [`cached_attachment_data`][cached_attachment_data] plugin adds the ability
 to retain the cached file across form redisplays, which means the file doesn't
@@ -13,13 +15,13 @@ cached file as JSON, and should be used to set the value of the hidden form
 field.
 
 ```rb
-@user.cached_avatar_data #=> '{"id":"38k25.jpg","storage":"cache","metadata":{...}}'
+photo.cached_image_data #=> '{"id":"38k25.jpg","storage":"cache","metadata":{...}}'
 ```
 
-This method delegates to `Attacher#read_cached`:
+This method delegates to `Attacher#cached_data`:
 
 ```rb
-attacher.read_cached #=> '{"id":"38k25.jpg","storage":"cache","metadata":{...}}'
+attacher.cached_data #=> '{"id":"38k25.jpg","storage":"cache","metadata":{...}}'
 ```
 
-[cached_attachment_data]: /lib/shrine/plugins/cached_attachment_data.rb
+[cached_attachment_data]: https://github.com/shrinerb/shrine/blob/master/lib/shrine/plugins/cached_attachment_data.rb

data/doc/plugins/column.md

@@ -0,0 +1,121 @@
+---
+title: Column
+---
+
+The [`column`][column] plugin provides interface for serializing and
+deserializing attachment data in format suitable for persisting in a database
+column (JSON by default).
+
+```rb
+plugin :column
+```
+
+## Serializing
+
+The `Attacher#column_data` method returns attached file data in serialized
+format, ready to be persisted into a database column.
+
+```rb
+attacher.attach(io)
+attacher.column_data #=> '{"id":"...","storage":"...","metadata":{...}}'
+```
+
+If there is no attached file, `nil` is returned.
+
+```rb
+attacher.column_data #=> nil
+```
+
+If you want to retrieve this data as a Hash, use `Attacher#data` instead.
+
+## Deserializing
+
+The `Attacher.from_column` method instantiates the attacher from serialized
+attached file data.
+
+```rb
+attacher = Shrine::Attacher.from_column('{"id":"...","storage":"...","metadata":{...}}')
+attacher.file #=> #<Shrine::UploadedFile>
+```
+
+If `nil` is given, it means no attached file.
+
+```rb
+attacher = Shrine::Attacher.from_column(nil)
+attacher.file #=> nil
+```
+
+Any additional options are forwarded to `Attacher#initialize`.
+
+```rb
+attacher = Shrine::Attacher.from_column('{...}', store: :other_store)
+attacher.store_key #=> :other_store
+```
+
+If you want to load attachment data into an existing attacher, use
+`Attacher#load_column`.
+
+```rb
+attacher.file #=> nil
+attacher.load_column('{"id":"...","storage":"...","metadata":{...}}')
+attacher.file #=> #<Shrine::UploadedFile>
+```
+
+If you want to load attachment from a Hash, use `Attacher.from_data` or
+`Attacher#load_data` instead.
+
+## Serializer
+
+By default, the `JSON` standard library is used for serializing hash data. With
+the [`model`][model] and [`entity`][entity] plugin, the data is serialized
+before writing to and deserialized after reading from the data attribute.
+
+You can also use your own serializer via the `:serializer` option. The
+serializer object needs to implement `#dump` and `#load` methods:
+
+```rb
+class MyDataSerializer
+  def self.dump(data)
+    data #=> { "id" => "...", "storage" => "...", "metadata" => { ... } }
+
+    JSON.generate(data) # serialize data, e.g. into JSON
+  end
+
+  def self.load(data)
+    data #=> '{"id":"...", "storage":"...", "metadata": {...}}'
+
+    JSON.parse(data) # deserialize data, e.g. from JSON
+  end
+end
+
+plugin :column, serializer: MyDataSerializer
+```
+
+Some serialization libraries such as [Oj] and [MessagePack] already implement
+this interface, which simplifies the configuration:
+
+```rb
+require "oj" # https://github.com/ohler55/oj
+
+plugin :column, serializer: Oj
+```
+
+If you want to disable serialization and work with hashes directly, you can set
+`:serializer` to `nil`:
+
+```rb
+plugin :column, serializer: nil # disable serialization
+```
+
+The serializer can also be changed for a particular attacher instance:
+
+```rb
+Shrine::Attacher.new(column_serializer: Oj)  # use custom serializer
+Shrine::Attacher.new(column_serializer: nil) # disable serialization
+```
+
+[column]: https://github.com/shrinerb/shrine/blob/master/lib/shrine/plugins/column.rb
+[model]: https://shrinerb.com/docs/plugins/model
+[entity]: https://shrinerb.com/docs/plugins/entity
+[Oj]: https://github.com/ohler55/oj
+[MessagePack]: https://github.com/msgpack/msgpack-ruby