fragmenter 0.5.1 → 1.0.0.rc1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 0d849a0a1482e71d93c6c90ea5a9336e2ced2be6
+  data.tar.gz: 1d87cae0a38cc7dddce3d76a08a5181a7d11587b
+SHA512:
+  metadata.gz: 14a4a61707cf76f83ceff137473dd8343b2ec7a1a0f5c645900c78dbb472516a943075cdae551be02efdf2e7e527e87e7a01afe62e21299738d23f04914382e0
+  data.tar.gz: 3720a5b353199f1be826f663b758873d0311d0fa4a032a9ebdb99bcfeedcad5059bf4e9b180fea48e2b359dd5726551e6093a89bd718a3bb9b9c6c1b0ebaa47d

data/.gitignore

@@ -7,3 +7,4 @@ doc/
 bin/
 vendor/ruby
 tmp
+TODO

data/.rspec

@@ -0,0 +1,3 @@
+--color
+--format progress
+-I spec/spec_helper.rb

data/.travis.yml

@@ -0,0 +1,7 @@
+language: ruby
+rvm:
+  - 1.9.3
+  - 2.0.0
+services:
+  - redis-server
+script: bundle exec rspec spec

data/HISTORY.md

@@ -1,3 +1,14 @@
+# 1.0.0.rc1
+
+* Add modules for easiy integration with Rails controllers and models.
+* Add validators for fragment storage.
+* Add services to help handle requests.
+* Add a request object that wraps the various resource, fragment, and rack
+  request objects.
+* Rename `Fragmenter::Base` to `Fragmenter::Wrapper`.
+* Enhanced documentation so that people can actually see what to do with the
+  libary.
+
 # 0.5.1
 
 * Initial release!

data/README.md

@@ -1,51 +1,195 @@
+[![Build Status](https://travis-ci.org/dscout/fragmenter.png?branch=master)](https://travis-ci.org/dscout/fragmenter)
+[![Code Climate](https://codeclimate.com/github/dscout/fragmenter.png)](https://codeclimate.com/github/dscout/fragmenter)
+
 # Fragmenter
 
-Fragmenter stores and rebuilds binary data in a distributed fashion. The only
-engine currently provided is Redis.
+Fragmenter is a library for multipart upload support backed by Redis.
+Fragmenter handles storing multiple parts of a larger binary and rebuilding it
+back into the original after all parts have been stored.
 
-## Why Fragments?
+## Why Fragment?
 
 It alleviates the problems posed by uploading large blocks of data from slow
 clients, notably mobile apps, by allowing the device to send multiple smaller
 blocks of data independently. Once all of the smaller blocks have been received
 they can quickly be rebuilt into the original file on the server.
 
+Think of multipart uploading as blocky streaming. Nginx, Rack and Rails all
+make it impossible to stream binary uploads. Breaking them into manageable
+pieces is the simplest workaround.
+
+Fragments are intended to be rather small, anywhere from 10-50k depending on
+the underlying data size. There is a balance between connection overhead from
+repeated server calls, being connection error tollerant, and not blocking the
+server from handling other connections.
+
 ## Requirements
 
-Fragmenter is tested on Ruby 1.9.3, but any ruby implementation with 1.9 syntax
+Fragmenter is tested on Ruby 2.0, but any ruby implementation with 1.9 syntax
 should be supported.
 
 Redis 2.0 or greater is required and version 2.6 is recommended.
 
 ## Installation
 
-    $ gem install fragmenter
+Add this to your Gemfile:
 
-## Getting Started
+```ruby
+gem 'fragmenter'
+```
 
-### Configuration
+## Configuration
 
-    Fragmenter.configure do |config|
-      config.redis      = $redis
-      config.logger     = Rails.logger
-      config.expiration = 2.days.to_i
-    end
+You can configure the following components of `Fragmenter`:
 
-### Usage
+* **redis**      - Redis instance to use for IO. Defaults to a new instance connected to `localhost`.
+* **logger**     - Logger instance to write out to. Defaults to `STDOUT` at the `INFO` level.
+* **expiration** - The number of seconds until fragments will expire. Defaults to 86400, or 1 day.
 
-    fragmenter = Fragmenter::Base.new(record)
+```ruby
+Fragmenter.configure do |config|
+  config.redis      = $redis
+  config.logger     = Rails.logger
+  config.expiration = 2.days.to_i
+end
+```
 
-    fragmenter.store(binary_data, number: 1, total: 12, content_type: 'image/jpeg')
-    fragmenter.complete? # => false
+## Using Fragmenter with Rails
 
-    fragmenter.store(binary_data, number: 12, total: 12, content_type: 'image/jpeg')
-    fragmenter.complete? # => true
+However, it is designed to be used from within a Rails controller. Include the
+provided `Fragmenter::Controller` module into any controller you wish to have
+process uploads:
 
-    rebuilt = fragmenter.rebuild # => binary data
-    fragmenter.clean!
+```ruby
+class UploadControler < ApplicationController
+  include Fragmenter::Rails::Controller
 
-More detailed examples will be added soon.
+  private
 
-## License
+  def resource
+    @resource ||= Avatar.find(:avatar_id)
+  end
+end
+```
+
+The module adds methods for handling the GET, PUT, and DELETE requests needed
+for handling fragment uploads. You must define a `resource` method that returns
+an object implementing `fragmenter`. In the example above the `resource` is an
+instance of the `Avatar` model, which could look something like this:
+
+```ruby
+class Avatar < ActiveRecord::Base
+  include Fragmenter::Rails::Model
+
+  def rebuild_fragments
+    self.avatar = Fragmenter::DummyIO.new(fragmenter.rebuild).tap do |io|
+      io.content_type = fragmenter.meta['content_type']
+    end
 
-Please see LICENSE for licensing details.
+    save!
+  end
+end
+```
+
+You **must** provide a concrete `rebuild_fragments` method that will perform
+rebuilding, saving, persisting etc. Without overriding `rebuild_fragments` a
+`Fragmenter::AbstractMethodError` will be raised when storage is complete and
+it attempts to rebuild.
+
+The example above synchronous storage using a mounted CarrierWave style
+uploader. You may want to perform rebuilding with a background worker instead
+to keep response times speedy.
+
+After you have configured your routes to map `show`, `update` and `destroy` to
+the uploads controller:
+
+```ruby
+MyApp::Application.routes.draw do
+  resource :avatar do
+    resource :upload, only: [:show, :update, :destroy]
+  end
+end
+```
+
+Then you can start sending `PUT` requests with successive fragments of data.
+Each fragment will be stored uniquely to the parent object, an instance of
+Avatar in this case. For each fragment that is stored the response will be the
+JSON representation of the fragments along with a `200 OK` status code:
+
+```bash
+curl -i
+     -X PUT                    /
+     -H 'X-Fragment-Number: 1' /
+     -H 'X-Fragment-Total: 2'  /
+     --data-binary @blob-1     /
+     http://example.com/avatar/1/upload
+
+#=> HTTP/1.1 200 OK
+#=> { "content_type": "image/jpeg", "fragments": [1], "total": 2 }
+```
+
+When the final part is uploaded the status code will be `202 Accepted` if the
+fragment is valid and can be rebuilt:
+
+```bash
+curl -i
+     -X PUT                    /
+     -H 'X-Fragment-Number: 2' /
+     -H 'X-Fragment-Total: 2'  /
+     --data-binary @blob-2     /
+     http://example.com/avatar/1/upload
+
+#=> HTTP/1.1 202 Accepted
+#=> { "content_type": "image/jpeg", "fragments": [1,2], "total": 2 }
+```
+
+### Validation
+
+Often you will want to be sure that all of the data is being stored without any
+bytes missing. A standard way to handle this is by sending a checksum that is
+verified after transfer. Fragmenter handles checksum matching using a validator
+that verifies each fragment that is uploaded. Validation is handled for any request
+where the `Content-MD5` header has been sent:
+
+```bash
+curl -X PUT                                             /
+     -H 'Content-MD5: ceba1b1ffc89e99abb54c1f8ab0c4157' /
+     -H 'X-Fragment-Number: 1'                          /
+     -H 'X-Fragment-Total: 1'                           /
+     --data-binary @blob                                /
+     http://example.com/avatar/1/upload
+```
+
+Failure to match the checksum will result in a `422 Unprocessable Entity`
+response with an accompanying message and errors:
+
+```json
+{ "message": "Upload of part failed.",
+  "errors":  [
+    "Expected checksum {{expected}} to match {{calculated}}"
+  ]
+}
+```
+As images uploads are a common use-case for fragmented uploading an
+ImageValidator is included, but not one of the default validators. You can
+control with validators are used by overriding the `validators` method within
+the controller:
+
+```ruby
+class AvatarUploader < ApplicationController
+  ...
+
+  private
+
+  def validators
+    super + [ImageValidator, CustomValidator]
+  end
+end
+```
+
+To add a custom validator you must add it at some point in the validator chain.
+A validator can be any class that responds to `valid?` with a boolean value and
+provides a list of errors. See the [ImageValidator][1] for an example validator
+that only performs validation when all fragments are complete.
+
+[1]:lib/fragmenter/validators/image

data/fragmenter.gemspec

@@ -4,20 +4,22 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 require 'fragmenter/version'
 
 Gem::Specification.new do |gem|
-  gem.name          = 'fragmenter'
-  gem.version       = Fragmenter::VERSION
-  gem.authors       = ['Parker Selbert']
-  gem.email         = ['parker@sorentwo.com']
-  gem.description   = %q{Fragmentize and rebuild data}
-  gem.summary       = %q{Fragmentize and rebuild data}
-
-  gem.homepage      = 'https://github.com/dscout/fragmenter'
+  gem.name         = 'fragmenter'
+  gem.version      = Fragmenter::VERSION
+  gem.authors      = ['Parker Selbert']
+  gem.email        = ['parker@sorentwo.com']
+  gem.homepage     = 'https://github.com/dscout/fragmenter'
+  gem.description  = %q{Fragmentize and rebuild data}
+  gem.summary      = <<-SUMMARY
+    Multipart upload support backed by Redis. Fragmenter handles storing
+    multiple parts of a larger binary and rebuilding it back into the original
+    after all parts have been stored.
+  SUMMARY
 
   gem.files         = `git ls-files`.split($/)
   gem.test_files    = gem.files.grep(%r{^(spec)/})
   gem.require_paths = ['lib']
 
   gem.add_dependency 'redis', '~> 3.0.0'
-
-  gem.add_development_dependency 'rspec', '~> 2.11.0'
+  gem.add_development_dependency 'rspec', '~> 2.14.0'
 end

data/lib/fragmenter.rb

@@ -1,37 +1,44 @@
 require 'logger'
 require 'redis'
-require 'fragmenter/version'
-require 'fragmenter/base'
 require 'fragmenter/redis'
+require 'fragmenter/version'
+require 'fragmenter/wrapper'
+require 'fragmenter/rails/controller'
+require 'fragmenter/rails/model'
+require 'fragmenter/services/uploader'
+require 'fragmenter/services/storer'
+require 'fragmenter/validators/checksum_validator'
 
 module Fragmenter
-  def self.configure(&block)
-    yield self
-  end
+  class << self
+    def configure(&block)
+      yield self
+    end
 
-  def self.logger
-    @logger ||= Logger.new(STDOUT).tap do |logger|
-      logger.level = Logger::INFO
+    def logger
+      @logger ||= Logger.new(STDOUT).tap do |logger|
+        logger.level = Logger::INFO
+      end
     end
-  end
 
-  def self.logger=(logger)
-    @logger = logger
-  end
+    def logger=(logger)
+      @logger = logger
+    end
 
-  def self.redis
-    @redis ||= ::Redis.new
-  end
+    def redis
+      @redis ||= ::Redis.new
+    end
 
-  def self.redis=(redis)
-    @redis = redis
-  end
+    def redis=(redis)
+      @redis = redis
+    end
 
-  def self.expiration=(expiration)
-    @expiration = expiration
-  end
+    def expiration=(expiration)
+      @expiration = expiration
+    end
 
-  def self.expiration
-    @expiration || 60 * 60 * 24
+    def expiration
+      @expiration || 60 * 60 * 24
+    end
   end
 end

data/lib/fragmenter/rails/controller.rb

@@ -0,0 +1,50 @@
+module Fragmenter
+  module Rails
+    module Controller
+      def show
+        render json: fragmenter.as_json
+      end
+
+      def update
+        if uploader.store
+          render json: fragmenter.as_json, status: update_status
+        else
+          render json: {
+            message: 'Upload of part failed.', errors: uploader.errors
+          }, status: :unprocessable_entity
+        end
+      end
+
+      def destroy
+        fragmenter.clean!
+
+        render nothing: true, status: :no_content
+      end
+
+      private
+
+      def fragmenter
+        resource.fragmenter
+      end
+
+      def validators
+        [Fragmenter::Validators::ChecksumValidator]
+      end
+
+      def uploader
+        @uploader ||= Fragmenter::Services::Uploader.new(
+          Fragmenter::Request.new(
+            resource:   resource,
+            fragmenter: fragmenter,
+            body:       request.body,
+            headers:    request.headers
+          ), validators
+        )
+      end
+
+      def update_status
+        uploader.complete? ? :accepted : :ok
+      end
+    end
+  end
+end

data/lib/fragmenter/rails/model.rb

@@ -0,0 +1,19 @@
+module Fragmenter
+  class AbstractMethodError < StandardError; end
+
+  module Rails
+    module Model
+      def fragmenter
+        @fragmenter ||= Fragmenter::Wrapper.new(self)
+      end
+
+      def store_fragment(*args)
+        fragmenter.store(*args)
+      end
+
+      def rebuild_fragments
+        raise Fragmenter::AbstractMethodError.new('This must be overriden on your model')
+      end
+    end
+  end
+end

data/lib/fragmenter/request.rb

@@ -0,0 +1,20 @@
+module Fragmenter
+  class Request
+    attr_accessor :fragmenter, :headers, :resource
+
+    def initialize(options = {})
+      @body       = options.fetch(:body, StringIO.new(''))
+      @fragmenter = options.fetch(:fragmenter, nil)
+      @headers    = options.fetch(:headers, {})
+      @resource   = options.fetch(:resource, nil)
+    end
+
+    def body
+      if @body.respond_to?(:read)
+        @body.read
+      else
+        @body
+      end
+    end
+  end
+end

data/lib/fragmenter/services/storer.rb

@@ -0,0 +1,41 @@
+require 'fragmenter/request'
+
+module Fragmenter
+  module Services
+    class Storer
+      attr_reader :request, :errors
+
+      def initialize(request)
+        @request = request
+        @errors  = []
+      end
+
+      def store
+        stored = fragmenter.store(request.body, extracted_options)
+
+        unless stored
+          errors << 'Unable to store fragment'
+        end
+
+        stored
+      end
+
+      private
+
+      def fragmenter
+        request.fragmenter
+      end
+
+      def extracted_options
+        headers = request.headers
+
+        headers['HTTP_X_FRAGMENT_NUMBER'] ||= headers['HTTP_PART_NUMBER']
+        headers['HTTP_X_FRAGMENT_TOTAL']  ||= headers['HTTP_PARTS_TOTAL']
+
+        { content_type: headers.fetch('CONTENT_TYPE'),
+          number:       headers.fetch('HTTP_X_FRAGMENT_NUMBER'),
+          total:        headers.fetch('HTTP_X_FRAGMENT_TOTAL') }
+      end
+    end
+  end
+end

data/lib/fragmenter/services/uploader.rb

@@ -0,0 +1,58 @@
+require 'fragmenter/services/storer'
+
+module Fragmenter
+  module Services
+    class Uploader
+      attr_reader :request, :validators
+      attr_writer :storer
+
+      def initialize(request, validators = [])
+        @request    = request
+        @validators = validators
+      end
+
+      def storer
+        @storer ||= Fragmenter::Services::Storer.new(request)
+      end
+
+      def store
+        stored    = valid? && storer.store
+        @complete = fragmenter.complete?
+
+        if stored && complete?
+          rebuild_fragments
+        end
+
+        stored
+      end
+
+      def errors
+        validator_instances.map(&:errors).flatten
+      end
+
+      def complete?
+        !!@complete
+      end
+
+      def valid?
+        validator_instances.all?(&:valid?)
+      end
+
+      private
+
+      def fragmenter
+        request.fragmenter
+      end
+
+      def rebuild_fragments
+        request.resource.rebuild_fragments
+      end
+
+      def validator_instances
+        @validator_instances ||= validators.map do |validator|
+          validator.new(request)
+        end
+      end
+    end
+  end
+end

data/lib/fragmenter/validators/checksum_validator.rb

@@ -0,0 +1,34 @@
+require 'digest/md5'
+
+module Fragmenter
+  module Validators
+    class ChecksumValidator
+      attr_reader :errors, :request
+
+      def initialize(request)
+        @request = request
+        @errors  = []
+      end
+
+      def valid?
+        matches = expected.nil? || expected == calculated
+
+        unless matches
+          errors << "Expected checksum #{expected} to match #{calculated}"
+        end
+
+        matches
+      end
+
+      private
+
+      def expected
+        request.headers['HTTP_CONTENT_MD5']
+      end
+
+      def calculated
+        @calculated ||= Digest::MD5.hexdigest(request.body)
+      end
+    end
+  end
+end

data/lib/fragmenter/version.rb

@@ -1,3 +1,3 @@
 module Fragmenter
-  VERSION = '0.5.1'
+  VERSION = '1.0.0.rc1'
 end

data/lib/fragmenter/{base.rb → wrapper.rb}

@@ -1,5 +1,5 @@
 module Fragmenter
-  class Base
+  class Wrapper
     extend Forwardable
 
     attr_reader :object, :engine

data/spec/fragmenter/fragment_spec.rb

@@ -5,54 +5,54 @@ describe Fragmenter::Fragment do
 
   describe '#number' do
     it 'defaults the number to 1' do
-      described_class.new(blob, {}).number.should == 1
+      Fragmenter::Fragment.new(blob, {}).number.should == 1
     end
   end
 
   describe '#total' do
     it 'defaults the total to 1' do
-      described_class.new(blob, {}).total.should == 1
+      Fragmenter::Fragment.new(blob, {}).total.should == 1
     end
   end
 
   describe '#content_type' do
     it 'defaults the content_type to a binary format' do
-      described_class.new(blob, {}).content_type.should == 'application/octet-stream'
+      Fragmenter::Fragment.new(blob, {}).content_type.should == 'application/octet-stream'
     end
   end
 
   describe '#padded_number' do
     it 'zero pads the number with as many zeros as the total has places' do
-      described_class.new(blob, number: 1, total: 2000).padded_number.should == '0001'
+      Fragmenter::Fragment.new(blob, number: 1, total: 2000).padded_number.should == '0001'
     end
   end
 
   describe '#valid?' do
     it 'is valid with a complete blob and sensible options' do
-      described_class.new(blob, number: 1, total: 2).should be_valid
+      Fragmenter::Fragment.new(blob, number: 1, total: 2).should be_valid
     end
 
     it 'is not valid with an empty blob' do
-      described_class.new('', number: 1, total: 2).should_not be_valid
+      Fragmenter::Fragment.new('', number: 1, total: 2).should_not be_valid
     end
 
     it 'is not valid without an integer part number greater than 1' do
-      described_class.new(blob, number: -1, total: 2).should_not be_valid
-      described_class.new(blob, number: 'one', total: 2).should_not be_valid
+      Fragmenter::Fragment.new(blob, number: -1, total: 2).should_not be_valid
+      Fragmenter::Fragment.new(blob, number: 'one', total: 2).should_not be_valid
     end
 
     it 'is not valid without an integer part total' do
-      described_class.new(blob, number: 1, total: -2).should_not be_valid
-      described_class.new(blob, number: 1, total: 'two').should_not be_valid
+      Fragmenter::Fragment.new(blob, number: 1, total: -2).should_not be_valid
+      Fragmenter::Fragment.new(blob, number: 1, total: 'two').should_not be_valid
     end
 
     it 'is not valid when the number is greater the total' do
-      described_class.new(blob, number: 2, total: 1).should_not be_valid
-      described_class.new(blob, number: 2, total: 2).should be_valid
+      Fragmenter::Fragment.new(blob, number: 2, total: 1).should_not be_valid
+      Fragmenter::Fragment.new(blob, number: 2, total: 2).should be_valid
     end
 
     it 'is not valid without a content type resembling a mime type' do
-      described_class.new(blob, content_type: 'jpg').should_not be_valid
+      Fragmenter::Fragment.new(blob, content_type: 'jpg').should_not be_valid
     end
   end
 end

data/spec/fragmenter/rails/controller_spec.rb

@@ -0,0 +1,85 @@
+require 'spec_helper'
+require 'fragmenter/rails/controller'
+
+describe Fragmenter::Rails::Controller do
+  UploadController = Struct.new(:resource) do
+    include Fragmenter::Rails::Controller
+  end
+
+  Resource = Struct.new(:id) do
+    def fragmenter
+      @fragmenter ||= Fragmenter::Wrapper.new(self)
+    end
+  end
+
+  describe '#show' do
+    it 'renders the JSON representation of the associated fragmenter' do
+      resource   = Resource.new(100)
+      controller = UploadController.new(resource)
+
+      controller.stub(:render)
+
+      controller.show
+
+      expect(controller).to have_received(:render).with(
+        json: { 'fragments' => [] }
+      )
+    end
+  end
+
+  describe '#destroy' do
+    it 'commands the fragmenter to clean' do
+      resource   = Resource.new(100)
+      controller = UploadController.new(resource)
+
+      controller.stub(:render)
+      resource.fragmenter.stub(:clean!)
+
+      controller.destroy
+
+      expect(resource.fragmenter).to have_received(:clean!)
+      expect(controller).to have_received(:render).with(
+        nothing: true,
+        status: :no_content
+      )
+    end
+  end
+
+  describe '#update' do
+    it 'stores the request body' do
+      resource   = Resource.new(100)
+      controller = UploadController.new(resource)
+      uploader   = double(:uploader, store: true, complete?: false)
+
+      controller.stub(:render)
+      controller.stub(uploader: uploader)
+
+      controller.update
+
+      expect(uploader).to have_received(:store)
+      expect(controller).to have_received(:render).with(
+        json: { 'fragments' => [] },
+        status: :ok
+      )
+    end
+
+    it 'renders error messages if storage fails' do
+      resource   = Resource.new(100)
+      controller = UploadController.new(resource)
+      uploader   = double(:uploader, store: false, errors: [], complete?: false)
+
+      controller.stub(:render)
+      controller.stub(uploader: uploader)
+
+      controller.update
+
+      expect(controller).to have_received(:render).with(
+        json: {
+          message: 'Upload of part failed.',
+          errors:  []
+        },
+        status: :unprocessable_entity
+      )
+    end
+  end
+end

data/spec/fragmenter/rails/model_spec.rb

@@ -0,0 +1,18 @@
+require 'spec_helper'
+require 'fragmenter/rails/model'
+
+describe Fragmenter::Rails::Model do
+  let(:model) do
+    double(:model).extend(Fragmenter::Rails::Model)
+  end
+
+  it 'adds a fragmenter wrapper around the underlying model' do
+    expect(model).to respond_to(:fragmenter)
+    expect(model.fragmenter).to be_instance_of(Fragmenter::Wrapper)
+  end
+
+  it 'adds an abstract rebuild_fragments method for compatibility' do
+    expect(model).to respond_to(:rebuild_fragments)
+    expect { model.rebuild_fragments }.to raise_error(Fragmenter::AbstractMethodError)
+  end
+end

data/spec/fragmenter/redis_spec.rb

@@ -4,10 +4,10 @@ require 'fragmenter/redis'
 describe Fragmenter::Redis do
   let(:blob_1)     { '00010110' }
   let(:blob_2)     { '11101110' }
-  let(:fragmenter) { mock(:fragmenter, key: 'abcdefg') }
+  let(:fragmenter) { double(:fragmenter, key: 'abcdefg') }
   let(:redis)      { Fragmenter.redis }
 
-  subject(:engine) { described_class.new(fragmenter) }
+  subject(:engine) { Fragmenter::Redis.new(fragmenter) }
 
   before do
     Fragmenter.logger = Logger.new('/dev/null')

data/spec/fragmenter/request_spec.rb

@@ -0,0 +1,15 @@
+require 'fragmenter/request'
+
+describe Fragmenter::Request do
+  describe '#body' do
+    it 'attempts to read from the body if it is an IO object' do
+      request = Fragmenter::Request.new(body: StringIO.new('blob'))
+      expect(request.body).to eq('blob')
+    end
+
+    it 'does not attempt to read the body if it is not IO' do
+      request = Fragmenter::Request.new(body: 'blob')
+      expect(request.body).to eq('blob')
+    end
+  end
+end

data/spec/fragmenter/services/storer_spec.rb

@@ -0,0 +1,49 @@
+require 'fragmenter/services/storer'
+
+describe Fragmenter::Services::Storer do
+  Storer = Fragmenter::Services::Storer
+
+  describe '#store' do
+    it 'stores the request with the fragmenter' do
+      fragmenter = double(:fragmenter)
+
+      request = Fragmenter::Request.new(
+        fragmenter: fragmenter,
+        body: '00100',
+        headers: {
+          'CONTENT_TYPE'           => 'application/octet-stream',
+          'HTTP_X_FRAGMENT_NUMBER' => 1,
+          'HTTP_X_FRAGMENT_TOTAL'  => 2
+        }
+      )
+
+      expect(fragmenter).to receive(:store).with(
+        '00100',
+        content_type: 'application/octet-stream',
+        number: 1,
+        total:  2
+      ).and_return(true)
+
+      expect(Storer.new(request).store).to be_true
+    end
+
+    it 'records an error if the body could not be stored' do
+      fragmenter = double(:fragmenter, store: false)
+
+      request = Fragmenter::Request.new(
+        fragmenter: fragmenter,
+        body: '00100',
+        headers: {
+          'CONTENT_TYPE'           => 'application/octet-stream',
+          'HTTP_X_FRAGMENT_NUMBER' => 1,
+          'HTTP_X_FRAGMENT_TOTAL'  => 2
+        }
+      )
+
+      storer = Storer.new(request)
+
+      expect(storer.store).to be_false
+      expect(storer.errors.length).to be_nonzero
+    end
+  end
+end

data/spec/fragmenter/services/uploader_spec.rb

@@ -0,0 +1,73 @@
+require 'fragmenter/services/uploader'
+
+describe Fragmenter::Services::Uploader do
+  Uploader = Fragmenter::Services::Uploader
+
+  describe '#store' do
+    it 'attempts to store the fragments without any validators' do
+      fragmenter = double(:fragmenter, complete?: false)
+      request    = Fragmenter::Request.new(fragmenter: fragmenter)
+      uploader   = Uploader.new(request)
+
+      expect(uploader.storer).to receive(:store).and_return(true)
+      expect(uploader.store).to be_true
+    end
+
+    it 'does not attempt to store fragments if any validators are invalid' do
+      validator = Struct.new(:request) do
+        def valid?
+          false
+        end
+      end
+
+      fragmenter = double(:fragmenter, complete?: false)
+      request    = Fragmenter::Request.new(fragmenter: fragmenter)
+      uploader   = Uploader.new(request, [validator])
+
+      expect(uploader.storer).to_not receive(:store)
+
+      expect(uploader.store).to be_false
+    end
+
+    it 'instructs the resource to rebuild if storage is complete' do
+      resource   = double(:resource)
+      fragmenter = double(:fragmenter, complete?: true)
+      storer     = double(:storer, store: true)
+      request    = Fragmenter::Request.new(resource: resource, fragmenter: fragmenter)
+      uploader   = Uploader.new(request)
+
+      uploader.storer = storer
+
+      expect(resource).to receive(:rebuild_fragments)
+
+      uploader.store
+    end
+  end
+
+  describe '#complete?' do
+    it 'is incomplete without store having been called' do
+      uploader = Uploader.new(Object.new)
+
+      expect(uploader).to_not be_complete
+    end
+  end
+
+  describe '#errors' do
+    it 'merges the errors from all validators' do
+      validator_a = Struct.new(:request) do
+        def valid?; false; end
+        def errors; ['bad']; end
+      end
+
+      validator_b = Struct.new(:request) do
+        def valid?; false; end
+        def errors; ['invalid']; end
+      end
+
+      uploader = Uploader.new({}, [validator_a, validator_b])
+      uploader.valid?
+
+      expect(uploader.errors).to eq(%w[bad invalid])
+    end
+  end
+end

data/spec/fragmenter/validators/checksum_validator_spec.rb

@@ -0,0 +1,41 @@
+require 'fragmenter/validators/checksum_validator'
+
+describe Fragmenter::Validators::ChecksumValidator do
+  Validator = Fragmenter::Validators::ChecksumValidator
+
+  describe '#valid?' do
+    it 'is always valid if no expected checksum was given' do
+      expect(Validator.new(double(headers: {}))).to be_valid
+    end
+
+    it 'is valid if the expected checksum matches the body checksum' do
+      request = double(
+        body:    '0010001000',
+        headers: { 'HTTP_CONTENT_MD5' => '4ac8660969d304047daa9c3539f63682' }
+      )
+
+      expect(Validator.new(request)).to be_valid
+    end
+
+    it 'is not valid if the expected checksum does not match the body checksum' do
+      request = double(
+        body:    '0010001000',
+        headers: { 'HTTP_CONTENT_MD5' => 'a9c3539f636824ac8660969d304047da' }
+      )
+
+      expect(Validator.new(request)).to_not be_valid
+    end
+
+    it 'records an error when the checksums do not match' do
+      request = double(
+        body:    '0010001000',
+        headers: { 'HTTP_CONTENT_MD5' => 'a9c3539f636824ac8660969d304047da' }
+      )
+
+      validator = Validator.new(request)
+      validator.valid?
+
+      expect(validator.errors.length).to be_nonzero
+    end
+  end
+end

data/spec/fragmenter/{base_spec.rb → wrapper_spec.rb}

@@ -1,15 +1,15 @@
-require 'fragmenter/base'
+require 'fragmenter/wrapper'
 
-describe Fragmenter::Base do
-  let(:object)       { mock('object', id: 1001) }
-  let(:engine_class) { mock('engine_class', new: engine) }
-  let(:engine)       { mock('engine') }
+describe Fragmenter::Wrapper do
+  let(:object)       { double(:object, id: 1001) }
+  let(:engine_class) { double(:engine_class, new: engine) }
+  let(:engine)       { double(:engine) }
 
-  subject { described_class.new(object, engine_class) }
+  subject(:base) { Fragmenter::Wrapper.new(object, engine_class) }
 
   describe '#key' do
     it 'composes a key from the object class and id value' do
-      subject.key.should match(/[a-z]+-\d+/)
+      base.key.should match(/[a-z]+-\d+/)
     end
   end
 
@@ -20,12 +20,12 @@ describe Fragmenter::Base do
     it 'delegates #store to the storage engine' do
       engine.should_receive(:store).with(blob, headers)
 
-      subject.store(blob, headers)
+      base.store(blob, headers)
     end
 
     it 'delegates #fragments to the storage engine' do
       engine.should_receive(:fragments)
-      subject.fragments
+      base.fragments
     end
   end
 
@@ -34,7 +34,7 @@ describe Fragmenter::Base do
       engine.stub('meta' => { 'content_type' => 'application/octet-stream' },
                   'fragments' => ['1', '2'])
 
-      subject.as_json.tap do |json|
+      base.as_json.tap do |json|
         json.should have_key('content_type')
         json.should have_key('fragments')
       end

data/spec/fragmenter_spec.rb

@@ -10,13 +10,12 @@ describe Fragmenter do
   describe '.logger' do
     it 'attempts to instantiate a standard logger to STDOUT' do
       Fragmenter.logger.should be_instance_of(Logger)
-      Fragmenter.logger.level.should == Logger::INFO
     end
   end
 
   describe '.logger=' do
     it 'stores the logger instance on the module' do
-      logger = mock(:logger)
+      logger = double(:logger)
 
       Fragmenter.logger = logger
       Fragmenter.logger.should be(logger)
@@ -31,7 +30,7 @@ describe Fragmenter do
 
   describe '.redis=' do
     it 'stores the redis instance on the module' do
-      redis = mock(:redis)
+      redis = double(:redis)
 
       Fragmenter.redis = redis
       Fragmenter.redis.should be(redis)
@@ -52,8 +51,8 @@ describe Fragmenter do
   end
 
   describe '.configure' do
-    let(:redis)  { mock(:redis) }
-    let(:logger) { mock(:logger) }
+    let(:redis)  { double(:redis) }
+    let(:logger) { double(:logger) }
 
     it 'allows customization via passing a block' do
       Fragmenter.configure do |config|

data/spec/spec_helper.rb

@@ -0,0 +1,8 @@
+require 'fragmenter'
+
+RSpec.configure do |config|
+  config.treat_symbols_as_metadata_keys_with_true_values = true
+  config.run_all_when_everything_filtered = true
+  config.filter_run :focus
+  config.order = 'random'
+end

metadata

@@ -1,20 +1,18 @@
 --- !ruby/object:Gem::Specification
 name: fragmenter
 version: !ruby/object:Gem::Version
-  version: 0.5.1
-  prerelease: 
+  version: 1.0.0.rc1
 platform: ruby
 authors:
 - Parker Selbert
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-10-29 00:00:00.000000000 Z
+date: 2013-07-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: redis
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
@@ -22,7 +20,6 @@ dependencies:
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
@@ -30,19 +27,17 @@ dependencies:
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 2.11.0
+        version: 2.14.0
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 2.11.0
+        version: 2.14.0
 description: Fragmentize and rebuild data
 email:
 - parker@sorentwo.com
@@ -51,6 +46,8 @@ extensions: []
 extra_rdoc_files: []
 files:
 - .gitignore
+- .rspec
+- .travis.yml
 - Gemfile
 - HISTORY.md
 - LICENSE.txt
@@ -58,48 +55,61 @@ files:
 - Rakefile
 - fragmenter.gemspec
 - lib/fragmenter.rb
-- lib/fragmenter/base.rb
 - lib/fragmenter/fragment.rb
+- lib/fragmenter/rails/controller.rb
+- lib/fragmenter/rails/model.rb
 - lib/fragmenter/redis.rb
+- lib/fragmenter/request.rb
+- lib/fragmenter/services/storer.rb
+- lib/fragmenter/services/uploader.rb
+- lib/fragmenter/validators/checksum_validator.rb
 - lib/fragmenter/version.rb
-- spec/fragmenter/base_spec.rb
+- lib/fragmenter/wrapper.rb
 - spec/fragmenter/fragment_spec.rb
+- spec/fragmenter/rails/controller_spec.rb
+- spec/fragmenter/rails/model_spec.rb
 - spec/fragmenter/redis_spec.rb
+- spec/fragmenter/request_spec.rb
+- spec/fragmenter/services/storer_spec.rb
+- spec/fragmenter/services/uploader_spec.rb
+- spec/fragmenter/validators/checksum_validator_spec.rb
+- spec/fragmenter/wrapper_spec.rb
 - spec/fragmenter_spec.rb
 - spec/spec_helper.rb
 homepage: https://github.com/dscout/fragmenter
 licenses: []
+metadata: {}
 post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
-  - - ! '>='
+  - - '>='
     - !ruby/object:Gem::Version
       version: '0'
-      segments:
-      - 0
-      hash: -1322051639440920079
 required_rubygems_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
-  - - ! '>='
+  - - '>'
     - !ruby/object:Gem::Version
-      version: '0'
-      segments:
-      - 0
-      hash: -1322051639440920079
+      version: 1.3.1
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.23
+rubygems_version: 2.0.0
 signing_key: 
-specification_version: 3
-summary: Fragmentize and rebuild data
+specification_version: 4
+summary: Multipart upload support backed by Redis. Fragmenter handles storing multiple
+  parts of a larger binary and rebuilding it back into the original after all parts
+  have been stored.
 test_files:
-- spec/fragmenter/base_spec.rb
 - spec/fragmenter/fragment_spec.rb
+- spec/fragmenter/rails/controller_spec.rb
+- spec/fragmenter/rails/model_spec.rb
 - spec/fragmenter/redis_spec.rb
+- spec/fragmenter/request_spec.rb
+- spec/fragmenter/services/storer_spec.rb
+- spec/fragmenter/services/uploader_spec.rb
+- spec/fragmenter/validators/checksum_validator_spec.rb
+- spec/fragmenter/wrapper_spec.rb
 - spec/fragmenter_spec.rb
 - spec/spec_helper.rb