defile 0.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: d2bbb1e9c99cb337359ac4aa0e5c27974e7fb6f0
+  data.tar.gz: 03df667144adc0818f2377b97b74e28d4e27713b
+SHA512:
+  metadata.gz: e9b48b6fa1da665fe19ff206f7575659b424ae0caab182f0d66d6bf7008ea5efc9fb7252eaac793b4306fb5f9659f2aa56ecf70e3e9db56fce617eba29b609c9
+  data.tar.gz: 2df044121ef0ed6821897500ff194fc832ae91911b05a0b8c13f499a942a9f09652275fc49a0eab58e79b98d522a8aa45205fd3d660a4401f1e86751ee80e5d2

data/.gitignore

@@ -0,0 +1,27 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+*.bundle
+*.so
+*.o
+*.a
+mkmf.log
+
+s3.yml
+spec/defile/test_app/log
+spec/defile/test_app/tmp
+spec/defile/test_app/db/*.sqlite

data/.rspec

@@ -0,0 +1 @@
+-r defile/spec_helper

data/.travis.yml

@@ -0,0 +1,8 @@
+language: ruby
+rvm:
+  - 2.1.5
+gemfile:
+  - Gemfile
+before_script:
+  - export DISPLAY=:99.0
+  - sh -e /etc/init.d/xvfb start

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in defile.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2014 Jonas Nicklas
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,466 @@
+# Defile
+
+Defile is a modern file upload library for Ruby applications. It is simple, yet
+powerful. Defile is an attempt by CarrierWave's original author to fix the
+design mistakes and overengineering in CarrierWave.
+
+Features:
+
+- Configurable backends, file system, S3, etc...
+- Convenient integration with ORMs
+- On the fly manipulation of images and other files
+- Streaming IO for fast and memory friendly uploads
+- Works across form redisplays, i.e. when validations fail, even on S3
+- Effortless direct uploads, even to S3
+
+## Quick start, Rails
+
+Add the gem:
+
+``` ruby
+gem "mini_magick"
+gem "defile", require: ["defile/rails", "defile/image_processing"]
+```
+
+Use the `attachment` method to use Defile in a model:
+
+``` ruby
+class User < ActiveRecord::Base
+  attachment :profile_image
+end
+```
+
+Generate a migration:
+
+``` sh
+rails generate migration add_profile_image_to_users profile_image_id:string
+rake db:migrate
+```
+
+Add an attachment field to your form:
+
+``` erb
+<%= form_for @user do |form| %>
+  <%= form.attachment_field :profile_image %>
+<% end %>
+```
+
+Set up strong parameters:
+
+``` ruby
+def user_params
+  params.require(:user).permit(:profile_image, :profile_image_cache_id)
+end
+```
+
+And start uploading! Finally show the file in your view:
+
+``` erb
+<%= image_tag attachment_url(@user, :profile_image, :fill, 300, 300) %>
+```
+
+## How it works
+
+Defile consists of several parts:
+
+1. Backends: cache and persist files
+2. Model attachments: map files to model columns
+3. A Rack application: streams files and accepts uploads
+4. Rails helpers: conveniently generate markup in your views
+4. A JavaScript library: facilitates direct uploads
+
+Let's look at each of these in more detail!
+
+## 1. Backend
+
+Files are uploaded to a backend. The backend assigns an ID to this file, which
+will be unique for this file within the backend.
+
+Let's look at a simple example of using the backend:
+
+``` ruby
+backend = Defile::Backend::FileSystem.new("tmp")
+
+file = backend.upload(StringIO.new("hello"))
+file.id # => "b205bc..."
+file.read # => "hello"
+
+backend.get(file.id).read # => "hello"
+```
+
+As you may notice, backends are "flat". Files do not have directories, nor do
+they have names or permissions, they are only identified by their ID.
+
+Defile has a global registry of backends, accessed through `Defile.backends`.
+
+There are two "special" backends, which are only really special in that they
+are the default backends for attachments. They are `cache` and `store`.
+
+The cache is intended to be transient. Files are added here before they are
+meant to be permanently stored. Usually files are then moved to the store for
+permanent storage, but this isn't always the case.
+
+Suppose for example that a user uploads a file in a form and receives a
+validation error. In that case the file has been temporarily stored in the
+cache. The user might decide to fix the error and resubmit, at which point the
+file will be promoted to the store. On the other hand, the user might simply
+give up and leave, now the file is left in the cache for later cleanup.
+
+Defile has convenient accessors for setting the `cache` and `store`, so for
+example you can switch to the S3 backend like this:
+
+``` ruby
+# config/initializers/defile.rb
+require "defile/backend/s3"
+
+aws = {
+  access_key_id: "xyz",
+  secret_access_key: "abc",
+  bucket: "my-bucket",
+}
+Defile.cache = Defile::Backend::S3.new(prefix: "cache", **aws)
+Defile.store = Defile::Backend::S3.new(prefix: "store", **aws)
+```
+
+Try this in the quick start example above and your files are now uploaded to
+S3.
+
+Backends also provide the option of restricting the size of files they accept.
+For example:
+
+``` ruby
+Defile.cache = Defile::Backend::S3.new(max_size: 10.megabytes, ...)
+```
+
+### Uploadable
+
+The `upload` method on backends can be called with a variety of objects. It
+requires that the object passed to it behaves similarly to Ruby IO objects, in
+particular it must implement the methods `size`, `read(length = nil, buffer =
+nil)`, `eof?` and `close`. All of `File`, `Tempfile`,
+`ActionDispath::UploadedFile` and `StringIO` implement this interface, however
+`String` does not. If you want to upload a file from a `String` you must wrap
+it in a `StringIO` first.
+
+## 2. Attachments
+
+You've already seen the `attachment` method:
+
+``` ruby
+class User < ActiveRecord::Base
+  attachment :profile_image
+end
+```
+
+Calling `attachment` generates a getter and setter with the given name. When
+you assign a file to the setter, it is uploaded to the cache:
+
+``` ruby
+User.new
+
+# with a ActionDispatch::UploadedFile
+user.profile_image = params[:file]
+
+# with a regular File object
+File.open("/some/path", "rb") do |file|
+  user.profile_image = file
+end
+
+# or a StringIO
+user.profile_image = StringIO.new("hello world")
+
+user.profile_image.id # => "fec421..."
+user.profile_image.read # => "hello world"
+```
+
+When you call `save` on the record, the uploaded file is transferred from the
+cache to the store. Where possible, Defile does this move efficiently. For example
+if both `cache` and `store` are on the same S3 account, instead of downloading
+the file and uploading it again, Defile will simply issue a copy command to S3.
+
+### Other ORMs
+
+Defile is built to integrate with ORMs other than ActiveRecord, but this being
+a very young gem, such integrations do not yet exist. Take a look at the [ActiveRecord
+integration](lib/defile/attachment/active_record.rb), building your own should
+not be too difficult.
+
+### Pure Ruby classes
+
+You can also use attachments in pure Ruby classes like this:
+
+``` ruby
+class User
+  extend Defile::Attachment
+
+  attr_accessor :profile_image_id
+
+  attachment :profile_image
+end
+```
+
+## 3. Rack Application
+
+Defile includes a Rack application (an endpoint, not a middleware). This application
+streams files from backends and can even accept file uploads and upload them to
+backends.
+
+**Important:** Unlike other file upload solutions, Defile always streams your files thorugh your
+application. It cannot generate URLs to your files. This means that you should
+**always** put a CDN or other HTTP cache in front of your application. Serving
+files through your app takes a lot of resources and you want it to happen rarely.
+
+This restriction, while onerous, is at the same time what makes Defile so easy
+and pleasant to use.
+
+### Mounting
+
+If you are using Rails and have required [defile/rails.rb](lib/defile/rails.rb),
+then the Rack application is mounted for you at `/attachments`. You should be able
+to see this when you run `rake routes`.
+
+You could also run the application on its own, it doesn't need to be mounted to
+work.
+
+### Retrieving files
+
+Files can be retrieved from the application by calling:
+
+```
+GET /attachments/:backend_name/:id/:filename
+```
+
+The `:filename` serves no other purpose than generating a nice name when the user
+downloads the file, it does not in any way affect the downloaded file. For caching
+purposes you should always use the same filename for the same file. The Rails helpers
+default this to the name of the column.
+
+### Processing
+
+Defile provides on the fly processing of files. You can trigger it by calling
+a URL like this:
+
+```
+GET /attachments/:backend_name/:processor_name/*args/:id/:filename
+```
+
+Suppose we have uploaded a file:
+
+``` ruby
+Defile.cache.upload(StringIO.new("hello")).id # => "a4e8ce"
+```
+
+And we've defined a processor like this:
+
+``` ruby
+Defile.processor :reverse do |file|
+  StringIO.new(file.read.reverse)
+end
+```
+
+Then you could do the following.
+
+``` sh
+curl http://127.0.0.1:3000/attachments/cache/reverse/a4e8ce/some_file.txt
+elloh
+```
+
+Defile calls `call` on the processor and passes in the retrieved file, as well
+as all additional arguments sent through the URL. See the
+[built in image processors](lib/defile/image_processing.rb) for a more advanced
+example.
+
+## 4. Rails helpers
+
+Defile provides the `attachment_field` form helper which generates a file field
+as well as a hidden field, suffixed with `cache_id`. This field keeps track of
+the file in case it is not yet permanently stored, for example if validations
+fail. It is also used for direct and presigned uploads. For this reason it is
+highly recommended to use `attachment_field` instead of `file_field`.
+
+``` erb
+<%= form_for @user do |form| %>
+  <%= form.attachment_field :profile_image %>
+<% end %>
+```
+
+Will generate something like:
+
+``` html
+<form action="/users" enctype="multipart/form-data" method="post">
+  <input name="user[profile_image_cache_id]" type="hidden">
+  <input name="user[profile_image]" type="file">
+</form>
+```
+
+The `attachment_url` helper can then be used for generating URLs for the uploaded
+files:
+
+``` erb
+<%= image_tag attachment_url(@user, :profile_image) %>
+```
+
+Any additional arguments to it are included in the URL as processor arguments:
+
+``` erb
+<%= image_tag attachment_url(@user, :profile_image, :fill, 300, 300) %>
+```
+
+## 5. JavaScript library
+
+Defile's JavaScript library is small but powerful.
+
+Uploading files is slow, so anything we can do to speed up the process is going
+to lead to happier users. One way to cheat is to start uploading files directly
+after the user has chosen a file, instead of waiting until they hit the submit
+button. This provides a significantly better user experience. Implementing this
+is usually tricky, but thankfully Defile makes it very easy.
+
+First, load the JavaScript file. If you're using the asset pipeline, you can
+simply include it like this:
+
+``` javascript
+//= require defile
+```
+
+Otherwise you can grab a copy [here](https://raw.githubusercontent.com/jnicklas/defile/master/app/assets/javascripts/defile.js).
+
+Now mark the field for direct upload:
+
+``` erb
+<%= form.attachment_field :profile_image, direct: true %>
+```
+
+There is no step 3 ;)
+
+The file is now uploaded to the `cache` immediately after the user chooses a file.
+If you try this in the browser, you'll notice that an AJAX request is fired as
+soon as you choose a file. Then when you submit to the server, the file is no
+longer submitted, only its id.
+
+If you want to improve the experience of this, the JavaScript library fires
+a couple of custom DOM events. These events bubble, so you can also listen for
+them on the form for example:
+
+``` javascript
+form.addEventListener("upload:start", function() {
+  // ...
+});
+
+form.addEventListener("upload:success", function() {
+  // ...
+});
+
+input.addEventListener("upload:progress", function() {
+  // ...
+});
+```
+
+You can also listen for them with jQuery, even with event delegation:
+
+``` javascript
+$(document).on("upload:start", "form", function(e) {
+  // ...
+});
+```
+
+This way you could for example disable the submit button until all files have
+uploaded:
+
+``` javascript
+$(document).on("upload:start", "form", function(e) {
+  $(this).find("input[type=submit]").attr("disabled", true)
+});
+
+$(document).on("upload:complete", "form", function(e) {
+  if(!$(this).find("input.uploading").length) {
+    $(this).find("input[type=submit]").removeAttr("disabled")
+  }
+});
+```
+
+### Presigned uploads
+
+Amazon S3 supports uploads directly from the browser to S3 buckets. With this
+feature you can bypass your application entirely; uploads never hit your application
+at all. Unfortunately the default configuration of S3 buckets does not allow
+cross site AJAX requests from posting to buckets. Fixing this is easy though.
+
+- Open the AWS S3 console and locate your bucket
+- Right click on it and choose "Properties"
+- Open the "Permission" section
+- Click "Add CORS Configuration"
+
+The default configuration only allows "GET", you'll want to allow "POST" as well.
+It could look something like this:
+
+``` xml
+<CORSConfiguration>
+    <CORSRule>
+        <AllowedOrigin>*</AllowedOrigin>
+        <AllowedMethod>GET</AllowedMethod>
+        <AllowedMethod>POST</AllowedMethod>
+        <MaxAgeSeconds>3000</MaxAgeSeconds>
+        <AllowedHeader>Authorization</AllowedHeader>
+    </CORSRule>
+</CORSConfiguration>
+```
+
+If you're paranoid you can restrict the allowed origin to only your domain, but
+since your bucket is only writable with authentication anyway, this shouldn't
+be necessary.
+
+Note that you do not need to, and in fact you shouldn't, make your bucket world
+writable.
+
+Once you've put in the new configuration, click "Save".
+
+Now you can enable presigned uploads:
+
+``` erb
+<%= form.attachment_field :profile_image, presigned: true %>
+```
+
+You can also enable both direct and presigned uploads, and it'll fall back to
+direct uploads if presigned uploads aren't available. This is useful if you're
+using the FileSystem backend in development or test mode and the S3 backend in
+production mode.
+
+``` erb
+<%= form.attachment_field :profile_image, direct: true, presigned: true %>
+```
+
+### Browser compatibility
+
+Defile's JavaScript library requires HTML5 features which are unavailable on
+IE9 and earlier versions. All other major browsers are supported. Note though
+that it has not yet been extensively tested.
+
+## Cache expiry
+
+Files will accumulate in your cache, and you'll probably want to remove them
+after some time.
+
+The FileSystem backend does not currently provide any method of doing this. PRs
+welcome ;)
+
+On S3 this can be conveniently handled through lifecycle rules. Exactly how
+depends a bit on your setup. If you are using the suggested setup of having
+one bucket with `cache` and `store` being directories in that bucket (or prefixes
+in S3 parlance), then follow the following steps, otherwise adapt them to your
+needs:
+
+- Open the AWS S3 console and locate your bucket
+- Right click on it and choose "Properties"
+- Open the "Lifecycle" section
+- Click "Add rule"
+- Choose "Apply the rule to: A prefix"
+- Enter "cache/" as the prefix (trailing slash!)
+- Click "Configure rule"
+- For "Action on Objects" you'll probably want to choose "Permanently Delete Only"
+- Choose whatever number of days you're comfortable with, I chose "1"
+- Click "Review" and finally "Create and activate Rule"
+
+## License
+
+[MIT](LICENSE.txt)