dis 0.9.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 0d9ec3b8500386221e40c1ea04e0b38e0d416f6d
+  data.tar.gz: 1b7ae3618b80c6d95b27a5ba832292c3cd035a45
+SHA512:
+  metadata.gz: 186c74e43812f0e71ddfc36856106decb829607854bd819d5a51467cfeb3e19811fabd9bd755975bfdc50ee11971e96f19a847612da6e0aaf456078a453916fe
+  data.tar.gz: 54bc48092d38eca8edb24d727b5295c56ea9a7d31aed80432ac8ea4a9316eb14162d2333797c76ae8b280a5d55e5fcb4ab27c7652e221d2de1b486e6385b21cf

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright 2014 Inge Jørgensen
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,165 @@
+# Dis [![Build Status](https://travis-ci.org/elektronaut/dis.png)](https://travis-ci.org/elektronaut/dis) [![Code Climate](https://codeclimate.com/github/elektronaut/dis.png)](https://codeclimate.com/github/elektronaut/dis) [![Code Climate](https://codeclimate.com/github/elektronaut/dis/coverage.png)](https://codeclimate.com/github/elektronaut/dis)
+
+Dis handles file uploads for your Rails app.
+It's similar to [Paperclip](https://github.com/thoughtbot/paperclip)
+and [Carrierwave](https://github.com/carrierwaveuploader/carrierwave),
+but different in a few ways. Chiefly, it's much, much simpler.
+
+Your files are stored in one or more layers, either on disk or in
+a cloud somewhere. [Fog](http://fog.io) and
+[Active Job](https://github.com/rails/activejob) does most of the
+heavy lifting.
+
+Files are indexed by the SHA1 hash of their contents. This means you get
+deduplication for free. This also means you run the (very slight) risk of
+hash collisions. There is no concept of updates in the data store,
+a file with changed content is by definition a different file.
+
+It does not do any processing. The idea is to provide a simple foundation
+other gems can build on. If you are looking to handle uploaded images,
+check out [DynamicImage](https://github.com/elektronaut/dynamic_image).
+
+Requires Rails 4.1+ and Ruby 1.9.3+.
+
+## Documentation
+
+[Documentation on RubyDoc.info](http://rdoc.info/github/elektronaut/dis)
+
+## Installation
+
+Add the gem to your Gemfile and run `bundle install`:
+
+```ruby
+gem "dis"
+```
+
+Now, run the generator to install the initializer:
+
+```sh
+bin/rails generate dis:install
+```
+
+## Usage
+
+Run the generator to create your model.
+
+```sh
+bin/rails generate dis:model Document
+```
+
+This will create a model along with a migration.
+
+Here's what your model might look like. Note that Dis does not
+validate any data by default, you are expected to use the Rails validators.
+A validator for validating presence of data is provided.
+
+```ruby
+class Document < ActiveRecord::Base
+  include Dis::Model
+  validates_data_presence
+  validates :content_type, presence: true, format: /\Aapplication\/(x\-)?pdf\z/
+  validates :filename, presence: true, format: /\A[\w_\-\.]+\.pdf\z/i
+  validates :content_length, numericality: { less_than: 5.megabytes }
+end
+```
+
+To save your document, simply set the `file` attribute.
+
+```ruby
+document_params = params.require(:document).permit(:file)
+@document = Document.create(document_params)
+```
+
+You can also assign the data directly.
+
+```ruby
+Document.create(
+  data:         File.open('document.pdf'),
+  content_type: 'application/pdf',
+  filename:     'document.pdf'
+)
+```
+
+## Defining layers
+
+The install generator will set you up with a local storage layer on disk,
+but this is configurable in `config/initializers/dis.rb`.
+
+You can have as many layers as you want, any storage provider
+[supported by Fog](http://fog.io/storage/) should work in theory.
+Having a local layer first is a good idea, this will provide you
+with a cache on disk. Any misses will be filled from the next layer.
+
+```ruby
+Dis::Storage.layers << Dis::Layer.new(
+  Fog::Storage.new({provider: 'Local', local_root: Rails.root.join('db', 'binaries')}),
+  path: Rails.env
+)
+```
+
+Delayed layers will be processed out of the request cycle using
+whatever adapter you've configured
+[Active Job](https://github.com/rails/activejob) to use.
+Note: You must have at least one non-delayed layer.
+
+```ruby
+if Rails.env.production?
+  Dis::Storage.layers << Dis::Layer.new(
+    Fog::Storage.new({
+      provider:              'AWS',
+      aws_access_key_id:     YOUR_AWS_ACCESS_KEY_ID,
+      aws_secret_access_key: YOUR_AWS_SECRET_ACCESS_KEY
+    }),
+    path: "my_bucket",
+    delayed: true
+  )
+end
+```
+
+You can also set layers to be read only. This is handy if you want to
+access production data from your development environment, or if you
+are in the process of migration from one provider to another.
+
+```ruby
+if Rails.env.development?
+  Dis::Storage.layers << Dis::Layer.new(
+    Fog::Storage.new(...),
+    readonly: true
+  )
+end
+```
+
+## Interacting with the store
+
+You can interact directly with the store if you want.
+
+```ruby
+file = File.open("foo.txt")
+hash = Dis::Storage.store("stuff", file) # => "8843d7f92416211de9ebb963ff4ce28125932878"
+Dis::Storage.exists?("stuff", hash)      # => true
+Dis::Storage.get("stuff", hash).body     # => "foobar"
+Dis::Storage.delete("stuff", hash)       # => true
+```
+
+## License
+
+Copyright 2014 Inge Jørgensen
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/lib/dis.rb

@@ -0,0 +1,15 @@
+# encoding: utf-8
+
+require "digest/sha1"
+require "fog"
+require "active_job"
+require "dis/errors"
+require "dis/jobs"
+require "dis/layer"
+require "dis/layers"
+require "dis/model"
+require "dis/storage"
+require "dis/validations"
+
+module Dis
+end

data/lib/dis/errors.rb

@@ -0,0 +1,11 @@
+# encoding: utf-8
+
+module Dis
+  module Errors
+    class Error < StandardError; end
+    class ReadOnlyError < Dis::Errors::Error; end
+    class NoLayersError < Dis::Errors::Error; end
+    class NotFoundError < Dis::Errors::Error; end
+    class NoDataError < Dis::Errors::Error; end
+  end
+end

data/lib/dis/jobs.rb

@@ -0,0 +1,4 @@
+# encoding: utf-8
+
+require "dis/jobs/delete"
+require "dis/jobs/store"

data/lib/dis/jobs/delete.rb

@@ -0,0 +1,18 @@
+# encoding: utf-8
+
+module Dis
+  module Jobs
+    # = Dis Delete Job
+    #
+    # Handles delayed deletion of objects.
+    #
+    #   Dis::Jobs::Delete.enqueue("documents", hash)
+    class Delete < ActiveJob::Base
+      queue_as :dis
+
+      def perform(type, hash)
+        Dis::Storage.delayed_delete(type, hash)
+      end
+    end
+  end
+end

data/lib/dis/jobs/store.rb

@@ -0,0 +1,18 @@
+# encoding: utf-8
+
+module Dis
+  module Jobs
+    # = Dis Store Job
+    #
+    # Handles delayed storage of objects.
+    #
+    #   Dis::Jobs::Store.enqueue("documents", hash)
+    class Store < ActiveJob::Base
+      queue_as :dis
+
+      def perform(type, hash)
+        Dis::Storage.delayed_store(type, hash)
+      end
+    end
+  end
+end

data/lib/dis/layer.rb

@@ -0,0 +1,174 @@
+# encoding: utf-8
+
+module Dis
+  # = Dis Layer
+  #
+  # Represents a layer of storage. It's a wrapper around
+  # <tt>Fog::Storage</tt>, any provider supported by Fog should be usable.
+  #
+  # ==== Options
+  #
+  # * <tt>:delayed</tt> - Delayed layers will be processed outside of
+  #   the request cycle by ActiveJob.
+  # * <tt>:readonly</tt> - Readonly layers can only be read from,
+  #   not written to.
+  # * <tt>:public</tt> - Objects stored in public layers will have the
+  #   public readable flag set if supported by the storage provider.
+  # * <tt>:path</tt> - Directory name to use for the store. For Amazon S3,
+  #   this will be the name of the bucket.
+  #
+  # ==== Examples
+  #
+  # This creates a local storage layer. It's a good idea to have a local layer
+  # first, this provides you with a cache on disk that will be faster than
+  # reading from the cloud.
+  #
+  #   Dis::Layer.new(
+  #     Fog::Storage.new({
+  #       provider: 'Local',
+  #       local_root: Rails.root.join('db', 'dis')
+  #     }),
+  #     path: Rails.env
+  #   )
+  #
+  # This creates a delayed layer on Amazon S3. ActiveJob will kick in and
+  # and transfer content from one of the immediate layers later at it's
+  # leisure.
+  #
+  #   Dis::Layer.new(
+  #     Fog::Storage.new({
+  #       provider:              'AWS',
+  #       aws_access_key_id:     YOUR_AWS_ACCESS_KEY_ID,
+  #       aws_secret_access_key: YOUR_AWS_SECRET_ACCESS_KEY
+  #     }),
+  #     path: "my_bucket",
+  #     delayed: true
+  #   )
+  class Layer
+    attr_reader :connection
+
+    def initialize(connection, options={})
+      options     = default_options.merge(options)
+      @connection = connection
+      @delayed    = options[:delayed]
+      @readonly   = options[:readonly]
+      @public     = options[:public]
+      @path       = options[:path]
+    end
+
+    # Returns true if the layer is a delayed layer.
+    def delayed?
+      @delayed
+    end
+
+    # Returns true if the layer isn't a delayed layer.
+    def immediate?
+      !delayed?
+    end
+
+    # Returns true if the layer isn't a delayed layer.
+    def public?
+      @public
+    end
+
+    # Returns true if the layer is read only.
+    def readonly?
+      @readonly
+    end
+
+    # Returns true if the layer is writeable.
+    def writeable?
+      !readonly?
+    end
+
+    # Stores a file.
+    #
+    #   hash = Digest::SHA1.file(file.path).hexdigest
+    #   layer.store("documents", hash, path)
+    #
+    # Hash must be a hex digest of the file content. If an object with the
+    # supplied hash already exists, no action will be performed. In other
+    # words, no data will be overwritten if a hash collision occurs.
+    #
+    # Returns an instance of Fog::Model, or raises an error if the layer
+    # is readonly.
+    def store(type, hash, file)
+      raise Dis::Errors::ReadOnlyError if readonly?
+      store!(type, hash, file)
+    end
+
+    # Returns true if a object with the given hash exists.
+    #
+    #    layer.exists?("documents", hash)
+    def exists?(type, hash)
+      (directory(type, hash) &&
+      directory(type, hash).files.head(key_component(type, hash))) ? true : false
+    end
+
+    # Retrieves a file from the store.
+    #
+    #    layer.get("documents", hash)
+    def get(type, hash)
+      if dir = directory(type, hash)
+        dir.files.get(key_component(type, hash))
+      end
+    end
+
+    # Deletes a file from the store.
+    #
+    #   layer.delete("documents", hash)
+    #
+    # Returns true if the file was deleted, or false if it could not be found.
+    # Raises an error if the layer is readonly.
+    def delete(type, hash)
+      raise Dis::Errors::ReadOnlyError if readonly?
+      delete!(type, hash)
+    end
+
+    private
+
+    def default_options
+      { delayed: false, readonly: false, public: false, path: nil }
+    end
+
+    def directory_component(type, hash)
+      path || ""
+    end
+
+    def key_component(type, hash)
+      [type, hash[0...2], hash[2..hash.length]].compact.join('/')
+    end
+
+    def delete!(type, hash)
+      return false unless exists?(type, hash)
+      get(type, hash).destroy
+    end
+
+    def directory(type, hash)
+      connection.directories.get(directory_component(type, hash))
+    end
+
+    def directory!(type, hash)
+      dir = directory(type, hash)
+      dir ||= connection.directories.create(
+        key:    directory_component(type, hash),
+        public: public?
+      )
+      dir
+    end
+
+    def store!(type, hash, file)
+      return get(type, hash) if exists?(type, hash)
+      file.rewind if file.respond_to?(:rewind)
+      directory!(type, hash).files.create(
+        key:    key_component(type, hash),
+        body:   (file.kind_of?(Fog::Model) ? file.body : file),
+        public: public?
+      )
+    end
+
+    def path
+      @path && !@path.empty? ? @path : nil
+    end
+  end
+end

data/lib/dis/layers.rb

@@ -0,0 +1,69 @@
+# encoding: utf-8
+
+module Dis
+  # = Dis Layers
+  #
+  # Represents a collection of layers.
+  class Layers
+    include Enumerable
+
+    def initialize(layers=[])
+      @layers = layers
+    end
+
+    # Adds a layer to the collection.
+    def <<(layer)
+      @layers << layer
+    end
+
+    # Clears all layers from the collection.
+    def clear!
+      @layers = []
+    end
+
+    # Iterates over the layers.
+    def each
+      @layers.each { |layer| yield layer }
+    end
+
+    # Returns a new instance containing only the delayed layers.
+    def delayed
+      self.class.new select { |layer| layer.delayed? }
+    end
+
+    # Returns true if one or more delayed layers exist.
+    def delayed?
+      delayed.any?
+    end
+
+    # Returns a new instance containing only the immediate layers.
+    def immediate
+      self.class.new select { |layer| layer.immediate? }
+    end
+
+    # Returns true if one or more immediate layers exist.
+    def immediate?
+      immediate.any?
+    end
+
+    # Returns a new instance containing only the readonly layers.
+    def readonly
+      self.class.new select { |layer| layer.readonly? }
+    end
+
+    # Returns true if one or more readonly layers exist.
+    def readonly?
+      readonly.any?
+    end
+
+    # Returns a new instance containing only the writeable layers.
+    def writeable
+      self.class.new select { |layer| layer.writeable? }
+    end
+
+    # Returns true if one or more writeable layers exist.
+    def writeable?
+      writeable.any?
+    end
+  end
+end

data/lib/dis/model.rb

@@ -0,0 +1,159 @@
+# encoding: utf-8
+
+require 'dis/model/class_methods'
+require 'dis/model/data'
+
+module Dis
+  # = Dis Model
+  #
+  # ActiveModel extension for the model holding your data. To use it,
+  # simply include the module in your model:
+  #
+  #   class Document < ActiveRecord::Base
+  #     include Dis::Model
+  #   end
+  #
+  # You'll need to define a few attributes in your database table.
+  # Here's a minimal migration:
+  #
+  #   create_table :documents do |t|
+  #     t.string  :content_hash
+  #     t.string  :content_type
+  #     t.integer :content_length
+  #     t.string  :filename
+  #   end
+  #
+  # You can override the names of any of these by setting
+  # <tt>dis_attributes</tt>.
+  #
+  #   class Document < ActiveRecord::Base
+  #     include Dis::Model
+  #     self.dis_attributes = {
+  #       filename:       :my_filename,
+  #       content_length: :filesize
+  #     }
+  #   end
+  #
+  # == Usage
+  #
+  # To save a file, simply assign to the <tt>file</tt> attribute.
+  #
+  #   document = Document.create(file: params.permit(:file))
+  #
+  # <tt>content_type</tt> and <tt>filename</tt> will automatically be set if
+  # the supplied object quacks like a file. <tt>content_length</tt> will always
+  # be set. <tt>content_hash</tt> won't be set until the record is being saved.
+  #
+  # If you don't care about filenames and content types and just want to store
+  # a binary blob, you can also just set the <tt>data</tt> attribute.
+  #
+  #   my_data = File.read('document.pdf')
+  #   document.update(data: my_data)
+  #
+  # The data won't be stored until the record is saved, and not unless
+  # the record is valid.
+  #
+  # To retrieve your data, simply read the <tt>data</tt> attribute. The file
+  # will be lazily loaded from the store on demand and cached in memory as long
+  # as the record stays in scope.
+  #
+  #   my_data = document.data
+  #
+  # Destroying a record will delete the file from the store, unless another
+  # record also refers to the same hash. Similarly, stale files will be purged
+  # when content changes.
+  #
+  # == Validations
+  #
+  # No validation is performed by default. If you want to ensure that data is
+  # present, use the <tt>validates_data_presence</tt> method.
+  #
+  #   class Document < ActiveRecord::Base
+  #     include Dis::Model
+  #     validates_data_presence
+  #   end
+  #
+  # If you want to validate content types, size or similar, simply use standard
+  # Rails validations on the metadata attributes:
+  #
+  #   validates :content_type, presence: true, format: /\Aapplication\/(x\-)?pdf\z/
+  #   validates :filename, presence: true, format: /\A[\w_\-\.]+\.pdf\z/i
+  #   validates :content_length, numericality: { less_than: 5.megabytes }
+  module Model
+    extend ActiveSupport::Concern
+
+    included do
+      before_save :store_data
+      after_save :cleanup_data
+      after_destroy :delete_data
+    end
+
+    # Returns the data as a binary string, or nil if no data has been set.
+    def data
+      dis_data.read
+    end
+
+    # Returns true if data is set.
+    def data?
+      dis_data.any?
+    end
+
+    # Assigns new data. This also sets <tt>content_length</tt>, and resets
+    # <tt>content_hash</tt> to nil.
+    def data=(new_data)
+      new_data = Dis::Model::Data.new(self, new_data)
+      attribute_will_change!('data') unless new_data == dis_data
+      @dis_data = new_data
+      dis_set :content_hash, nil
+      dis_set :content_length, dis_data.content_length
+    end
+
+    # Returns true if the data has been changed since the object was last saved.
+    def data_changed?
+      changes.include?('data')
+    end
+
+    # Assigns new data from an uploaded file. In addition to the actions
+    # performed by <tt>data=</tt>, this will set <tt>content_type</tt> and
+    # <tt>filename</tt>.
+    def file=(file)
+      self.data = file
+      dis_set :content_type, file.content_type
+      dis_set :filename, file.original_filename
+    end
+
+    private
+
+    def cleanup_data
+      if previous_hash = changes[dis_attribute(:content_hash)].try(&:first)
+        dis_data.expire(previous_hash)
+      end
+    end
+
+    def delete_data
+      dis_data.expire(dis_get(:content_hash))
+    end
+
+    def store_data
+      if dis_data.changed?
+        dis_set :content_hash, dis_data.store!
+      end
+    end
+
+    def dis_get(attribute_name)
+      self[dis_attribute(attribute_name)]
+    end
+
+    def dis_data
+      @dis_data ||= Dis::Model::Data.new(self)
+    end
+
+    def dis_set(attribute_name, value)
+      self[dis_attribute(attribute_name)] = value
+    end
+
+    def dis_attribute(attribute_name)
+      self.class.dis_attributes[attribute_name]
+    end
+  end
+end

data/lib/dis/model/class_methods.rb

@@ -0,0 +1,63 @@
+# encoding: utf-8
+
+module Dis
+  module Model
+    module ClassMethods
+      # Returns the mapping of attribute names.
+      def dis_attributes
+        default_dis_attributes.merge(@dis_attributes || {})
+      end
+
+      # Sets the current mapping of attribute names. Use this if you want to
+      # override the attributes and database columns that Dis will use.
+      #
+      #   class Document < ActiveRecord::Base
+      #     include Dis::Model
+      #     self.dis_attributes = { filename: :my_custom_filename }
+      #   end
+      def dis_attributes=(new_attributes)
+        @dis_attributes = new_attributes
+      end
+
+      # Returns the storage type name, which Dis will use for
+      # directory scoping. Defaults to the name of the database table.
+      #
+      #   class Document < ActiveRecord::Base; end
+      #   Document.dis_type # => "documents"
+      def dis_type
+        @dis_type || self.table_name
+      end
+
+      # Sets the storage type name.
+      #
+      # Take care not to set the same name for multiple models, this will
+      # cause data loss when a record is destroyed.
+      def dis_type=(new_type)
+        @dis_type = new_type
+      end
+
+      # Adds a presence validation on the +data+ attribute.
+      #
+      # This is better than using `validates :data, presence: true`, since
+      # that would cause it to load the data from storage on each save.
+      #
+      #   class Document < ActiveRecord::Base
+      #     include Dis::Model
+      #     validates_data_presence
+      #   end
+      def validates_data_presence
+        validates_with Dis::Validations::DataPresence
+      end
+
+      # Returns the default attribute names.
+      def default_dis_attributes
+        {
+          content_hash: :content_hash,
+          content_length: :content_length,
+          content_type: :content_type,
+          filename: :filename
+        }
+      end
+    end
+  end
+end

data/lib/dis/model/data.rb

@@ -0,0 +1,118 @@
+# encoding: utf-8
+
+module Dis
+  module Model
+    # = Dis Model Data
+    #
+    # Facilitates communication between the model and the storage,
+    # and holds any newly assigned data before the record is saved.
+    class Data
+      def initialize(record, raw=nil)
+        @record = record
+        @raw = raw
+      end
+
+      # Returns true if two Data objects represent the same data.
+      def ==(comp)
+        # TODO: This can be made faster by
+        # comparing hashes for stored objects.
+        comp.read == read
+      end
+
+      # Returns true if data exists either in memory or in storage.
+      def any?
+        raw? || stored?
+      end
+
+      # Returns the data as a binary string.
+      def read
+        @cached ||= read_from(closest)
+      end
+
+      # Will be true if data has been explicitely set.
+      #
+      #   Dis::Model::Data.new(record).changed? # => false
+      #   Dis::Model::Data.new(record, new_file).changed? # => true
+      def changed?
+        raw?
+      end
+
+      # Returns the length of the data.
+      def content_length
+        if raw? && raw.respond_to?(:length)
+          raw.length
+        else
+          read.try(&:length).to_i
+        end
+      end
+
+      # Expires a data object from the storage if it's no longer being used
+      # by existing records. This is triggered from callbacks on the record
+      # whenever they are changed or destroyed.
+      def expire(hash)
+        unless @record.class.where(
+          @record.class.dis_attributes[:content_hash] => hash
+        ).any?
+          Dis::Storage.delete(storage_type, hash)
+        end
+      end
+
+      # Stores the data. Returns a hash of the content for reference.
+      def store!
+        raise Dis::Errors::NoDataError unless raw?
+        Dis::Storage.store(storage_type, raw)
+      end
+
+      private
+
+      def closest
+        if raw?
+          raw
+        elsif stored?
+          stored
+        end
+      end
+
+      def content_hash
+        @record[@record.class.dis_attributes[:content_hash]]
+      end
+
+      def raw?
+        raw ? true : false
+      end
+
+      def read_from(object)
+        return nil unless object
+        if object.respond_to?(:body)
+          object.body
+        elsif object.respond_to?(:read)
+          object.rewind
+          response = object.read
+          object.rewind
+          response
+        else
+          object
+        end
+      end
+
+      def storage_type
+        @record.class.dis_type
+      end
+
+      def stored?
+        !content_hash.blank?
+      end
+
+      def stored
+        Dis::Storage.get(
+          storage_type,
+          content_hash
+        )
+      end
+
+      def raw
+        @raw
+      end
+    end
+  end
+end

data/lib/dis/storage.rb

@@ -0,0 +1,153 @@
+# encoding: utf-8
+
+module Dis
+  # = Dis Storage
+  #
+  # This is the interface for interacting with the storage layers.
+  #
+  # All queries are scoped by object type, which will default to the table
+  # name of the model. Take care to use your own scope if you interact with
+  # the store directly, as models will purge expired content when they change.
+  #
+  # Files are stored with a SHA1 digest of the file contents as the key.
+  # This ensures data is deduplicated per scope. Hash collisions will be
+  # silently ignored.
+  #
+  # Layers should be added to <tt>Dis::Storage.layers</tt>. At least
+  # one writeable, non-delayed layer must exist.
+  class Storage
+    class << self
+      # Exposes the layer set, which is an instance of
+      # <tt>Dis::Layers</tt>.
+      def layers
+        @layers ||= Dis::Layers.new
+      end
+
+      # Stores a file and returns a digest. Kicks off a
+      # <tt>Dis::Jobs::Store</tt> job if any delayed layers are defined.
+      #
+      #   hash = Dis::Storage.store("things", File.open('foo.bin'))
+      #   # => "8843d7f92416211de9ebb963ff4ce28125932878"
+      def store(type, file)
+        require_writeable_layers!
+        hash = store_immediately!(type, file)
+        if layers.delayed.writeable.any?
+          Dis::Jobs::Store.enqueue(type, hash)
+        end
+        hash
+      end
+
+      # Transfers files from immediate layers to all delayed layers.
+      #
+      #   Dis::Storage.delayed_store("things", hash)
+      def delayed_store(type, hash)
+        file = get(type, hash)
+        layers.delayed.writeable.each do |layer|
+          layer.store(type, hash, file)
+        end
+      end
+
+      # Returns true if the file exists in any layer.
+      #
+      #   Dis::Storage.exists?("things", hash) # => true
+      def exists?(type, hash)
+        require_layers!
+        layers.each do |layer|
+          return true if layer.exists?(type, hash)
+        end
+        false
+      end
+
+      # Retrieves a file from the store.
+      #
+      #   stuff = Dis::Storage.get("things", hash)
+      #
+      # If any misses are detected, it will try to fetch the file from the
+      # first available layer, then store it in all immediate layer.
+      #
+      # Returns an instance of Fog::Model.
+      def get(type, hash)
+        require_layers!
+        miss = false
+        layers.each do |layer|
+          if result = layer.get(type, hash)
+            store_immediately!(type, result) if miss
+            return result
+          else
+            miss = true
+          end
+        end
+        raise Dis::Errors::NotFoundError
+      end
+
+      # Deletes a file from all layers. Kicks off a
+      # <tt>Dis::Jobs::Delete</tt> job if any delayed layers are defined.
+      # Returns true if the file existed in any immediate layers,
+      # or false if not.
+      #
+      #   Dis::Storage.delete("things", hash)
+      #   # => true
+      #   Dis::Storage.delete("things", hash)
+      #   # => false
+      def delete(type, hash)
+        require_writeable_layers!
+        deleted = false
+        layers.immediate.writeable.each do |layer|
+          deleted = true if layer.delete(type, hash)
+        end
+        if layers.delayed.writeable.any?
+          Dis::Jobs::Delete.enqueue(type, hash)
+        end
+        deleted
+      end
+
+      # Deletes content from all delayed layers.
+      #
+      #   Dis::Storage.delayed_delete("things", hash)
+      def delayed_delete(type, hash)
+        layers.delayed.writeable.each do |layer|
+          layer.delete(type, hash)
+        end
+      end
+
+      private
+
+      def store_immediately!(type, file)
+        hash_file(file) do |hash|
+          layers.immediate.writeable.each do |layer|
+            layer.store(type, hash, file)
+          end
+        end
+      end
+
+      def require_layers!
+        unless layers.any?
+          raise Dis::Errors::NoLayersError
+        end
+      end
+
+      def require_writeable_layers!
+        unless layers.immediate.writeable.any?
+          raise Dis::Errors::NoLayersError
+        end
+      end
+
+      def digest
+        Digest::SHA1
+      end
+
+      def hash_file(file, &block)
+        hash = case file
+        when Fog::Model
+          digest.hexdigest(file.body)
+        when String
+          digest.hexdigest(file)
+        else
+          digest.file(file.path).hexdigest
+        end
+        yield hash if block_given?
+        hash
+      end
+    end
+  end
+end

data/lib/dis/validations.rb

@@ -0,0 +1,3 @@
+# encoding: utf-8
+
+require 'dis/validations/data_presence'

data/lib/dis/validations/data_presence.rb

@@ -0,0 +1,17 @@
+# encoding: utf-8
+
+module Dis
+  module Validations
+    # = Dis Data Presence Validation
+    #
+    class DataPresence < ActiveModel::Validator
+      # Validates that a record has data, either freshly assigned or
+      # persisted in the storage. Adds a `:blank` error on `:data`if not.
+      def validate(record)
+        unless record.data?
+          record.errors.add(:data, :blank)
+        end
+      end
+    end
+  end
+end

data/lib/dis/version.rb

@@ -0,0 +1,5 @@
+# encoding: utf-8
+
+module Dis
+  VERSION = "0.9.0"
+end

data/lib/rails/generators/dis/install/install_generator.rb

@@ -0,0 +1,17 @@
+# encoding: utf-8
+
+require 'rails/generators'
+require 'rails/generators/rails/model/model_generator'
+
+module Dis
+  module Generators
+    class InstallGenerator < Rails::Generators::Base
+      desc "Creates the Dis initializer"
+      source_root File.expand_path("../templates", __FILE__)
+
+      def create_initializer
+        template 'initializer.rb', File.join('config', 'initializers', 'dis.rb')
+      end
+    end
+  end
+end

data/lib/rails/generators/dis/install/templates/initializer.rb

@@ -0,0 +1,18 @@
+# Be sure to restart your server when you modify this file.
+
+# Creates a local storage layer in db/dis:
+Dis::Storage.layers << Dis::Layer.new(
+  Fog::Storage.new({ provider: 'Local', local_root: Rails.root.join('db', 'dis') }),
+  path: Rails.env
+)
+
+# You can also add cloud storage.
+# Dis::Storage.layers << Dis::Layer.new(
+#   Fog::Storage.new({
+#     provider:              'AWS',
+#     aws_access_key_id:     AWS_ACCESS_KEY_ID,
+#     aws_secret_access_key: AWS_SECRET_ACCESS_KEY
+#   }),
+#   path: "my_bucket",
+#   delayed: true
+# )

data/lib/rails/generators/dis/model/model_generator.rb

@@ -0,0 +1,41 @@
+# encoding: utf-8
+
+require 'rails/generators'
+require 'rails/generators/rails/model/model_generator'
+
+module Dis
+  module Generators
+    class ModelGenerator < Rails::Generators::ModelGenerator
+      desc "Creates a Dis model"
+
+      def initialize(args, *options)
+        super(inject_dis_attributes(args), *options)
+      end
+
+      def add_model_extension
+        inject_into_file File.join('app/models', class_path, "#{file_name}.rb"), after: "ActiveRecord::Base\n" do
+          "  include Dis::Model\n"
+        end
+      end
+
+      private
+
+      def inject_dis_attributes(args)
+        if args.any?
+          args = [args[0]] + dis_attributes + args[1..args.length]
+        else
+          args
+        end
+      end
+
+      def dis_attributes
+        %w{
+          content_hash:string
+          content_type:string
+          content_length:integer
+          filename:string
+        }
+      end
+    end
+  end
+end

metadata

@@ -0,0 +1,134 @@
+--- !ruby/object:Gem::Specification
+name: dis
+version: !ruby/object:Gem::Version
+  version: 0.9.0
+platform: ruby
+authors:
+- Inge Jørgensen
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-09-23 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 4.1.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 4.1.0
+- !ruby/object:Gem::Dependency
+  name: fog
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.22.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.22.1
+- !ruby/object:Gem::Dependency
+  name: activejob
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: sqlite3
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec-rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+description: Dis is a Rails plugin that stores your file uploads and other binary
+  blobs.
+email:
+- inge@elektronaut.no
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- MIT-LICENSE
+- README.md
+- lib/dis.rb
+- lib/dis/errors.rb
+- lib/dis/jobs.rb
+- lib/dis/jobs/delete.rb
+- lib/dis/jobs/store.rb
+- lib/dis/layer.rb
+- lib/dis/layers.rb
+- lib/dis/model.rb
+- lib/dis/model/class_methods.rb
+- lib/dis/model/data.rb
+- lib/dis/storage.rb
+- lib/dis/validations.rb
+- lib/dis/validations/data_presence.rb
+- lib/dis/version.rb
+- lib/rails/generators/dis/install/install_generator.rb
+- lib/rails/generators/dis/install/templates/initializer.rb
+- lib/rails/generators/dis/model/model_generator.rb
+homepage: https://github.com/elektronaut/dis
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 1.9.2
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.1
+signing_key: 
+specification_version: 4
+summary: A file store for your Rails app
+test_files: []