dis 1.2.0 → 1.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cf00e607df1b1aa62ffa3e025c22ac42e3363d14fb0a06cb6b8f41d3b7757db5
-  data.tar.gz: 2cfe7f5a47d87bdd5e6c59e7f98446bec2918a27c8e3476c123ef1164842d437
+  metadata.gz: bf532ee41185ec1d1fb96f36e9cd697a7a1c0ece44da5a30b2ff01d8a6c77010
+  data.tar.gz: 8d338d164af992ca38b9af28840c97eabc33f6434791a5505b5a602dd1aa0b3f
 SHA512:
-  metadata.gz: e2428a8aa2299b730e1db9702fbec9e725cae6a38bfd3e04bce0ac1ba41484f45840f52cdfe0f998dfe5ffa3c5c5a5fa31bc01c380a1aa8a307392e4a4c9a6a4
-  data.tar.gz: d1e9bfbc53cdd753fa0b52730b4c6ac88ec00ca1ba3dd519086fdea58f117947f30d0eb5bef297d952a8fb8deab6d642ce939d5a045d19f70f0586c578b3111e
+  metadata.gz: 8db87ae2b81b5639fc9312a3dbad3580e6b96886bcb22ddcd6d834efa7452e7348738c37d656ea86c343041cbde683b68f4277a28125f0467de4878b3ca2d3b0
+  data.tar.gz: bde07caf197b9a2ae72f3403c3b0f4ad189dbca90259768354eaa96ae7f026edcb3bc6fa86c3b2378361b9eb76ef8a9f5a8446b8584dd5b375861f0de546c740

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright 2014 Inge Jørgensen
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -3,39 +3,20 @@
 
 # Dis
 
-Dis is a content-adressable store for file uploads in your Rails app.
+Dis is a content-addressable store for file uploads in your Rails app.
 
-Data can be stored either on disk or in the cloud - anywhere that
-[Fog](http://fog.io) knows how to talk to.
+Data can be stored either on disk or in the cloud — anywhere
+[Fog](http://fog.io) can connect to.
 
-It doesn't do any processing, but it's a simple foundation to roll
-your own on. If you're looking to handle image uploads, check out
+It doesn't do any processing, but provides a foundation for
+building your own. If you're looking to handle image uploads, check out
 [DynamicImage](https://github.com/elektronaut/dynamic_image). It's
 built on top of Dis and handles resizing, cropping and more on demand.
 
-Requires Rails 5+
+## Requirements
 
-## Layers
-
-The underlaying storage consists of one or more layers. A layer is a
-unit of storage location, which can either be a local path, or a cloud
-provider like Amazon S3 or Google Cloud Storage.
-
-There are two types of layers, immediate and delayed. Files are
-written to immediate layers and then replicated to the rest in the
-background using ActiveJob.
-
-Reads are performed from the first available layer. In case of a read
-miss, the file is backfilled from the next layer.
-
-An example configuration could be to have a local layer first, and
-then for example an Amazon S3 bucket. This provides you with an
-on-disk cache backed by cloud storage. You can also add additional
-layers if you want fault tolerance across regions or even providers.
-
-Layers can be configured as read-only. This can be useful if you want
-to read from your staging or production environment while developing
-locally, or if you're transitioning away from a provider.
+- Ruby >= 3.2
+- Rails >= 7.1
 
 ## Installation
 
@@ -51,10 +32,10 @@ Now, run the generator to install the initializer:
 bin/rails generate dis:install
 ```
 
-By default, files will be stored in `db/dis`. You can edit
-`config/initializers/dis.rb` if you want to change the path or add
-additional layers. Note that you also need the corresponding
-[Fog gem](https://github.com/fog) if you want to use cloud storage:
+By default, files will be stored in `db/dis`. Edit
+`config/initializers/dis.rb` to change the path or add
+additional layers. Cloud storage requires the corresponding
+[Fog gem](https://github.com/fog):
 
 ```ruby
 gem "fog-aws"
@@ -70,9 +51,9 @@ bin/rails generate dis:model Document
 
 This will create a model along with a migration.
 
-Here's what your model might look like. Note that Dis does not
-validate any data by default, but you can use the standard Rails validators.
-A validator for validating presence of data is provided.
+Here's what your model might look like. Dis does not validate any data
+by default, but you can use standard Rails validators. A presence
+validator for data is also provided.
 
 ```ruby
 class Document < ActiveRecord::Base
@@ -84,30 +65,32 @@ class Document < ActiveRecord::Base
 end
 ```
 
-To save your document, simply set the `file` attribute.
+To save your document, set the `file` attribute. This extracts
+`content_type` and `filename` from the upload automatically.
 
 ```ruby
 document_params = params.require(:document).permit(:file)
 @document = Document.create(document_params)
 ```
 
-You can also pass a file directly:
+You can also assign `data` directly, but you'll need to set
+`content_type` and `filename` yourself:
 
 ```ruby
-Document.create(data: File.open('document.pdf'),
-                content_type: 'application/pdf',
-                filename: 'document.pdf')
+Document.create(data: File.open("document.pdf"),
+                content_type: "application/pdf",
+                filename: "document.pdf")
 ```
 
-..or even a string:
+...or even a string:
 
 ```ruby
-Document.create(data: 'foo', content_type: 'text/plain', filename: 'foo.txt')
+Document.create(data: "foo", content_type: "text/plain", filename: "foo.txt")
 ```
 
-Getting your file back out is straightforward:
+Reading the file back out:
 
-``` ruby
+```ruby
 class DocumentsController < ApplicationController
   def show
     @document = Document.find(params[:id])
@@ -115,15 +98,83 @@ class DocumentsController < ApplicationController
       send_data(@document.data,
                 filename: @document.filename,
                 type: @document.content_type,
-                disposition: "attachment)
+                disposition: "attachment")
     end
   end
 end
 ```
 
-## Behind the scenes
+## Layers
+
+The underlying storage consists of one or more layers. Each layer
+targets either a local path or a cloud provider like Amazon S3 or
+Google Cloud Storage.
+
+There are three types of layers:
+
+- **Immediate** layers are written to synchronously during the
+  request cycle.
+- **Delayed** layers are replicated in the background using ActiveJob.
+- **Cache** layers are bounded, immediate layers with LRU eviction.
+  They act as both a read cache and an upload buffer.
+
+Reads are performed from the first available layer. On a miss, the
+file is backfilled from the next layer.
+
+A typical multi-layer configuration has a local layer first and an
+Amazon S3 bucket second. This gives you an on-disk cache backed by
+cloud storage. Additional layers can provide fault tolerance across
+regions or providers.
+
+```ruby
+# config/initializers/dis.rb
+
+# Fast local layer (immediate, synchronous writes)
+Dis::Storage.layers << Dis::Layer.new(
+  Fog::Storage.new(provider: "Local", local_root: Rails.root.join("db/dis")),
+  path: Rails.env
+)
+
+# Cloud layer (delayed, replicated via ActiveJob)
+Dis::Storage.layers << Dis::Layer.new(
+  Fog::Storage.new(
+    provider: "AWS",
+    aws_access_key_id: ENV["AWS_ACCESS_KEY_ID"],
+    aws_secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"]
+  ),
+  path: "my-bucket",
+  delayed: true
+)
+```
+
+Layers can be configured as read-only — useful for reading from
+staging or production while developing locally, or when transitioning
+away from a provider.
+
+### Cache layers
+
+A cache layer provides bounded local storage with automatic eviction.
+Files are evicted in LRU order, but only after they have been
+replicated to at least one non-cache writeable layer. This ensures
+unreplicated uploads are never lost.
+
+The cache size is a soft limit: the cache may temporarily exceed it
+if no files are safe to evict, and will shrink back once delayed
+replication jobs complete.
+
+```ruby
+Dis::Storage.layers << Dis::Layer.new(
+  Fog::Storage.new(provider: "Local", local_root: Rails.root.join("tmp/dis")),
+  path: Rails.env,
+  cache: 1.gigabyte
+)
+```
+
+Cache layers cannot be combined with `delayed` or `readonly`.
+
+## Low-level API
 
-You can interact directly with the store if you want.
+You can also interact with the store directly.
 
 ```ruby
 file = File.open("foo.txt")
@@ -139,23 +190,5 @@ See the [generated documentation on RubyDoc.info](https://www.rubydoc.info/gems/
 
 ## License
 
-Copyright 2014 Inge Jørgensen
-
-Permission is hereby granted, free of charge, to any person obtaining
-a copy of this software and associated documentation files (the
-"Software"), to deal in the Software without restriction, including
-without limitation the rights to use, copy, modify, merge, publish,
-distribute, sublicense, and/or sell copies of the Software, and to
-permit persons to whom the Software is furnished to do so, subject to
-the following conditions:
-
-The above copyright notice and this permission notice shall be
-included in all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
-LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
-OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+Copyright 2014-2026 Inge Jørgensen. Released under the
+[MIT License](LICENSE).

data/lib/dis/errors.rb

@@ -1,15 +1,23 @@
 # frozen_string_literal: true
 
 module Dis
+  # Namespace for all Dis error classes.
   module Errors
+    # Base error class for all Dis errors.
     class Error < StandardError; end
 
+    # Raised when attempting to write to a readonly layer.
     class ReadOnlyError < Dis::Errors::Error; end
 
+    # Raised when no storage layers are configured, or no writeable
+    # immediate layers exist for a write operation.
     class NoLayersError < Dis::Errors::Error; end
 
+    # Raised when a file cannot be found in any storage layer.
     class NotFoundError < Dis::Errors::Error; end
 
+    # Raised when attempting to store a record that has no data
+    # assigned.
     class NoDataError < Dis::Errors::Error; end
   end
 end

data/lib/dis/jobs/change_type.rb

@@ -4,14 +4,21 @@ module Dis
   module Jobs
     # = Dis ChangeType Job
     #
-    # Handles delayed object type change.
+    # Handles delayed object type change. Stores the content under
+    # the new type in delayed layers, then deletes it under the
+    # old type. Retries up to 10 times on failure.
     #
-    #   Dis::Jobs::ChangeType.perform_later("old_things", "new_things", key)
+    # @example
+    #   Dis::Jobs::ChangeType.perform_later("old", "new", key)
     class ChangeType < ActiveJob::Base
       queue_as :dis
 
       retry_on StandardError, attempts: 10, wait: :polynomially_longer
 
+      # @param prev_type [String] the current type scope
+      # @param new_type [String] the new type scope
+      # @param key [String] the content hash
+      # @return [void]
       def perform(prev_type, new_type, key)
         Dis::Storage.delayed_store(new_type, key)
         Dis::Storage.delayed_delete(prev_type, key)

data/lib/dis/jobs/delete.rb

@@ -4,14 +4,19 @@ module Dis
   module Jobs
     # = Dis Delete Job
     #
-    # Handles delayed deletion of objects.
+    # Handles delayed deletion of objects from all delayed layers.
+    # Retries up to 10 times on failure.
     #
+    # @example
     #   Dis::Jobs::Delete.perform_later("documents", key)
     class Delete < ActiveJob::Base
       queue_as :dis
 
       retry_on StandardError, attempts: 10, wait: :polynomially_longer
 
+      # @param type [String] the type scope
+      # @param key [String] the content hash
+      # @return [void]
       def perform(type, key)
         Dis::Storage.delayed_delete(type, key)
       end

data/lib/dis/jobs/evict.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module Dis
+  module Jobs
+    # = Dis Evict Job
+    #
+    # Handles cache eviction for cache layers. Evicts files in LRU
+    # order, but only after they have been replicated to a
+    # non-cache writeable layer. Retries up to 10 times on failure.
+    #
+    # @example
+    #   Dis::Jobs::Evict.perform_later
+    class Evict < ActiveJob::Base
+      queue_as :dis
+
+      retry_on StandardError, attempts: 10, wait: :polynomially_longer
+
+      # @return [void]
+      def perform
+        Dis::Storage.evict_caches
+      end
+    end
+  end
+end

data/lib/dis/jobs/store.rb

@@ -4,8 +4,12 @@ module Dis
   module Jobs
     # = Dis Store Job
     #
-    # Handles delayed storage of objects.
+    # Handles delayed storage of objects. Replicates content from
+    # immediate layers to all delayed layers. Retries up to 10
+    # times on failure. Discarded if the source file no longer
+    # exists.
     #
+    # @example
     #   Dis::Jobs::Store.perform_later("documents", key)
     class Store < ActiveJob::Base
       queue_as :dis
@@ -14,6 +18,9 @@ module Dis
 
       retry_on StandardError, attempts: 10, wait: :polynomially_longer
 
+      # @param type [String] the type scope
+      # @param key [String] the content hash
+      # @return [void]
       def perform(type, key)
         Dis::Storage.delayed_store(type, key)
       end

data/lib/dis/jobs.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
 require "dis/jobs/delete"
+require "dis/jobs/evict"
 require "dis/jobs/store"
 require "dis/jobs/change_type"