rack-cache 0.2.0

data/CHANGES

@@ -0,0 +1,27 @@
+## 0.2.0 / 2008-10-24 / Initial Release
+
+  * Document events and transitions in `rack/cache/config/default.rb`
+  * Basic logging support (`trace`, `warn`, `info`, `error` from within Context)
+  * EntityStore: store entity bodies keyed by SHA
+  * MetaStore: store response headers keyed by URL
+  * Last-Modified/ETag validation
+  * Vary support
+  * Implement error! transition
+  * New Rack::Cache::Core
+  * memcached meta and entity store implementations
+  * URI based storage configuration
+  * Read options from Rack env if present (rack-cache.XXX keys)
+  * `object` is now `entry`
+  * Documentation framework and website
+  * Document storage areas and implementations
+  * Document configuration/events
+
+## 0.1.0 / 2008-07-21 / Proof of concept (unreleased)
+
+  * Basic core with event support
+  * `#import` method for bringing in config files
+  * Freshness based expiration
+  * RFC 2616 If-Modified-Since based validation
+  * A horribly shitty storage back-end (Hash in mem)
+  * Don't cache hop-by-hop headers: Connection, Keep-Alive, Proxy-Authenticate,
+    Proxy-Authorization, TE, Trailers, Transfer-Encoding, Upgrade

data/COPYING

@@ -0,0 +1,18 @@
+Copyright (c) 2008 Ryan Tomayko <http://tomayko.com/about>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to
+deal in the Software without restriction, including without limitation the
+rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+sell copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README

@@ -0,0 +1,96 @@
+Rack::Cache
+===========
+
+Rack::Cache is suitable as a quick drop-in component to enable HTTP caching for
+Rack-based applications that produce freshness (Expires, Cache-Control) and/or
+validation (Last-Modified, ETag) information.
+
+We strive to implement those portions of RFC 2616 Section 13 relevant to gateway
+(i.e., "reverse proxy") cache scenarios with a system for specifying cache
+policy:
+
+  * Standards-based (RFC 2616)
+  * Freshness/expiration based caching
+  * Validation
+  * Vary support
+  * Portable: 100% Ruby / works with any Rack-enabled framework
+  * Configuration language for advanced caching policies
+  * Disk, memcached, and heap memory storage backends
+
+For more information about Rack::Cache features and usage, see:
+
+http://tomayko.com/src/rack-cache/
+
+Rack::Cache is not overly optimized for performance. The main goal of the
+project is to provide a portable, easy-to-configure, and standards-based
+caching solution for small to medium sized deployments. More sophisticated /
+high-performance caching systems (e.g., Varnish, Squid, httpd/mod-cache) may be
+more appropriate for large deployments with significant throughput requirements.
+
+Status
+------
+
+Rack::Cache is a young and experimental project that is likely to change
+substantially and may not be wholly functional, consistent, fast, or correct.
+The current focus is on reaching basic compliance with RFC 2616 and providing
+good documentation.
+
+Installation
+------------
+
+From Gem:
+
+    $ sudo gem install rack-cache
+
+With a local working copy:
+
+    $ git clone git://github.com/rtomayko/rack-cache.git
+    $ rake package && sudo rake install
+
+Basic Usage
+-----------
+
+Rack::Cache is implemented as a piece of Rack middleware and can be used with
+any Rack-based application. If your application includes a rackup (`.ru`) file
+or uses Rack::Builder to construct the application pipeline, simply require
+and use as follows:
+
+    require 'rack/cache'
+
+    use Rack::Cache,
+      :metastore   => 'file:/var/cache/rack/meta',
+      :entitystore => 'file:/var/cache/rack/body',
+      :verbose     => true
+
+    run app
+
+Assuming you've designed your backend application to take advantage of HTTP's
+caching features, no further code or configuration is required for basic
+caching.
+
+Documentation
+-------------
+
+http://tomayko.com/src/rack-cache/
+
+License
+-------
+
+Copyright (c) 2008 Ryan Tomayko <http://tomayko.com/about>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to
+deal in the Software without restriction, including without limitation the
+rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+sell copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,144 @@
+require 'rake/clean'
+
+task :default => :test
+
+CLEAN.include %w[coverage/ doc/api tags]
+CLOBBER.include %w[dist]
+
+# load gemspec like github's gem builder to surface any SAFE issues.
+Thread.new do
+  require 'rubygems/specification'
+  $spec = eval("$SAFE=3\n#{File.read('rack-cache.gemspec')}")
+end.join
+
+# SPECS =====================================================================
+
+desc 'Run specs with story style output'
+task :spec do
+  sh 'specrb --specdox -Ilib:test test/*_test.rb'
+end
+
+desc 'Run specs with unit test style output'
+task :test => FileList['test/*_test.rb'] do |t|
+  suite = t.prerequisites
+  sh "testrb -Ilib:test #{suite.join(' ')}", :verbose => false
+end
+
+desc 'Generate test coverage report'
+task :rcov do
+  sh "rcov -Ilib:test test/*_test.rb"
+end
+
+# DOC =======================================================================
+desc 'Build all documentation'
+task :doc => %w[doc:api doc:graphs doc:markdown]
+
+# requires the hanna gem:
+#   gem install mislav-hanna --source=http://gems.github.com
+desc 'Build API documentation (doc/api)'
+task 'doc:api' => 'doc/api/index.html' 
+file 'doc/api/index.html' => FileList['lib/**/*.rb'] do |f|
+  rm_rf 'doc/api'
+  sh((<<-SH).gsub(/[\s\n]+/, ' ').strip)
+  hanna
+    --op doc/api
+    --promiscuous
+    --charset utf8
+    --fmt html
+    --inline-source
+    --line-numbers
+    --accessor option_accessor=RW
+    --main Rack::Cache
+    --title 'Rack::Cache API Documentation'
+    #{f.prerequisites.join(' ')}
+  SH
+end
+CLEAN.include 'doc/api'
+
+desc 'Build graphviz graphs'
+task 'doc:graphs'
+%w[pdf png svg].each do |filetype|
+  FileList["doc/*.dot"].each do |source|
+    dest = source.sub(/dot$/, filetype)
+    file dest => source do |f|
+      sh "dot -T#{filetype} #{source} -o #{f.name}"
+    end
+    task 'doc:graphs' => dest
+    CLEAN.include dest
+  end
+end
+
+desc 'Build markdown documentation files'
+task 'doc:markdown'
+FileList['doc/*.markdown'].each do |source|
+  dest = "doc/#{File.basename(source, '.markdown')}.html"
+  file dest => source do |f|
+    puts "markdown: #{source} -> #{dest}" if verbose
+    require 'erb'
+    require 'rdiscount'
+    template = File.read(source)
+    content = Markdown.new(ERB.new(template, 0, "%<>").result(binding)).to_html
+    title = content.match("<h1>(.*)</h1>")[1] rescue ''
+    layout = ERB.new(File.read("doc/layout.html.erb"), 0, "%<>")
+    output = layout.result(binding)
+    File.open(dest, 'w') { |io| io.write(output) }
+  end
+  task 'doc:markdown' => dest
+  CLEAN.include dest
+end
+
+task 'doc:publish' => :doc do
+  sh 'rsync -avz doc/ gus@tomayko.com:/src/rack-cache'
+end
+
+# PACKAGING =================================================================
+
+def package(ext='')
+  "dist/rack-cache-#{$spec.version}" + ext
+end
+
+desc 'Build packages'
+task :package => %w[.gem .tar.gz].map {|e| package(e)}
+
+desc 'Build and install as local gem'
+task :install => package('.gem') do
+  sh "gem install #{package('.gem')}"
+end
+
+directory 'dist/'
+
+file package('.gem') => %w[dist/ rack-cache.gemspec] + $spec.files do |f|
+  sh "gem build rack-cache.gemspec"
+  mv File.basename(f.name), f.name
+end
+
+file package('.tar.gz') => %w[dist/] + $spec.files do |f|
+  sh "git archive --format=tar HEAD | gzip > #{f.name}"
+end
+
+desc 'Upload gem and tar.gz distributables to rubyforge'
+task :release => [package('.gem'), package('.tar.gz')] do |t|
+  sh <<-SH
+    rubyforge add_release wink rack-cache #{$spec.version} #{package('.gem')} &&
+    rubyforge add_file    wink rack-cache #{$spec.version} #{package('.tar.gz')}
+  SH
+end
+
+# GEMSPEC ===================================================================
+
+file 'rack-cache.gemspec' => FileList['{lib,test}/**','Rakefile'] do |f|
+  # read spec file and split out manifest section
+  spec = File.read(f.name)
+  parts = spec.split("  # = MANIFEST =\n")
+  fail 'bad spec' if parts.length != 3
+  # determine file list from git ls-files
+  files = `git ls-files`.
+    split("\n").sort.reject{ |file| file =~ /^\./ }.
+    map{ |file| "    #{file}" }.join("\n")
+  # piece file back together and write...
+  parts[1] = "  s.files = %w[\n#{files}\n  ]\n"
+  spec = parts.join("  # = MANIFEST =\n")
+  spec.sub!(/s.date = '.*'/, "s.date = '#{Time.now.strftime("%Y-%m-%d")}'")
+  File.open(f.name, 'w') { |io| io.write(spec) }
+  puts "updated #{f.name}"
+end

data/TODO

@@ -0,0 +1,40 @@
+## 0.3
+
+  - BUG: meta store hits but entity misses
+  - BUG: HEAD request on invalid entry caches zero-length response
+  - BUG: Response body written to cache each time validation succeeds
+  - Are we doing HEAD properly?
+  - liberal, conservative, sane caching configs
+  - Sample app
+  - busters.rb doc and tests
+  - no-cache.rb doc and tests
+  - Canonicalized URL for cache key:
+    - sorts params by key, then value
+    - urlencodes /[^ A-Za-z0-9_.-]/ host, path, and param key/value
+  - Support server-specific X-Sendfile (or similar) for delivering cached
+    bodies (file entity stores only).
+  - Sqlite3 (meta store)
+  - Cache invalidation on PUT, POST, DELETE.
+    - Invalidate at the request URI; or, anything that's "near" the request URI.
+    - Invalidate at the URI of the Location or Content-Location response header.
+
+## Backlog
+
+  - Purge/invalidate specific cache entries
+  - Purge/invalidate everything
+  - Maximum size of cached entity
+  - Last-Modified factor: requests that have a Last-Modified header but no Expires
+    header have a TTL assigned based on the last modified age of the response:
+    TTL = (Age * Factor), or, 1h  = (10h * 0.1)
+  - I wonder if it would be possible to run in threaded mode but with an
+    option to lock before making requests to the backend. The idea is to be
+    able to serve requests from cache in separate threads. This should
+    probably be implemented as a separate middleware component.
+  - stale-while-revalidate
+  - Serve cached copies when down (see: stale-if-error) - e.g., database
+    connection drops and the cache takes over what it can. 
+  - When a cache misses due to Vary, try to validate using the best match. Note
+    that you can't do this with a weak validator, so only strong etags can be
+    used.
+  - Consider implementing ESI (http://www.w3.org/TR/esi-lang). This should
+    probably be implemented as a separate middleware component.

data/doc/configuration.markdown

@@ -0,0 +1,224 @@
+Configuration Language
+======================
+
+__Rack::Cache__ includes a configuration system that can be used to specify
+fairly sophisticated cache policy on a global or per-request basis.
+
+  - [Synopsis](#synopsis)
+  - [Setting Cache Options](#setopt)
+  - [Cache Option Reference](#options)
+  - [Configuration Machinery - Events and Transitions](#machinery)
+  - [Importing Configuration](#import)
+  - [Default Configuration Machinery](#default)
+  - [Notes](#notes)
+
+<a id='synopsis'></a>
+
+Synopsis
+--------
+
+    use Rack::Cache do
+      # set cache related options
+      set :verbose, true
+      set :metastore,   'memcached://localhost:11211'
+      set :entitystore, 'file:/var/cache/rack/body'
+
+      # override events / transitions
+      on :receive do
+        pass! if request.url =~ %r|/dontcache/|
+        error! 402 if request.referrer =~ /digg.com/
+      end
+
+      on :miss do
+        trace 'missed: %s', request.url
+      end
+
+      # bring in other configuration machinery
+      import 'rack/cache/config/breakers'
+      import 'mycacheconfig'
+    end
+
+<a id='setopt'></a>
+
+Setting Cache Options
+---------------------
+
+Cache options can be set when the __Rack::Cache__ object is created; or by using
+the `set` method within a configuration block; or by setting a
+`rack-cache.<option>` variable in __Rack__'s __Environment__.
+
+When the __Rack::Cache__ object is instantiated:
+
+    use Rack::Cache,
+      :verbose => true,
+      :metastore => 'memcached://localhost:11211/',
+      :entitystore => 'file:/var/cache/rack'
+
+Using the `set` method within __Rack::Cache__'s configuration context:
+
+    use Rack::Cache do
+      set :verbose, true
+      set :metastore, 'memcached://localhost:11211/'
+      set :entitystore, 'file:/var/cache/rack'
+    end
+
+Using __Rack__'s __Environment__:
+
+    env.merge!(
+      'rack-cache.verbose' => true,
+      'rack-cache.metastore' => 'memcached://localhost:11211/',
+      'rack-cache.entitystore' => 'file:/var/cache/rack'
+    )
+
+<a id='options'></a>
+
+Cache Option Reference
+----------------------
+
+Use the following options to customize __Rack::Cache__:
+
+### `verbose`
+
+Boolean specifying whether verbose trace logging is enabled. This option is
+currently enabled (`true`) by default but is likely to be disabled (`false`) in
+a future release. All log output is written to the `rack.errors` stream, which
+is typically set to `STDERR`.
+
+The `trace`, `info`, `warn`, and `error` methods can be used within the
+configuration context to write messages to the errors stream.
+
+### `default_ttl`
+
+An integer specifying the number of seconds a cached object should be considered
+"fresh" when no explicit freshness information is provided in a response.
+Explicit `Cache-Control` or `Expires` response headers always override this
+value. The `default_ttl` option defaults to `0`, meaning responses without
+explicit freshness information are considered immediately "stale" and will not
+be served from cache without validation.
+
+### `metastore`
+
+A URI specifying the __MetaStore__ implementation used to store request/response
+meta information. See the [Rack::Cache Storage Documentation](storage.html)
+for detailed information on different storage implementations.
+
+If no metastore is specified, the `heap:/` store is assumed. This implementation
+has significant draw-backs so explicit configuration is recommended.
+
+### `entitystore`
+
+A URI specifying the __EntityStore__ implementation used to store
+response bodies. See the [Rack::Cache Storage Documentation](storage.html)
+for detailed information on different storage implementations.
+
+If no entitystore is specified, the `heap:/` store is assumed. This
+implementation has significant draw-backs so explicit configuration is
+recommended.
+
+<a id='machinery'></a>
+
+Configuration Machinery - Events and Transitions
+------------------------------------------------
+
+The configuration machinery is built around a series of interceptable events and
+transitions controlled by a simple configuration language. The following diagram
+shows each state (interceptable event) along with their possible transitions:
+
+<p class='center'>
+<img src='events.png' alt='Events and Transitions Diagram' />
+</p>
+
+Custom logic can be layered onto the `receive`, `hit`, `miss`, `fetch`, `store`,
+`deliver`, and `pass` events by passing a block to the `on` method:
+
+    on :fetch do
+      trace 'fetched %p from backend application', request.url
+    end
+
+Here, the `trace` method writes a message to the `rack.errors` stream when a
+response is fetched from the backend application. The `request` object is a
+[__Rack::Cache::Request__](./api/classes/Rack/Cache/Request) that can be
+inspected (and modified) to determine what action should be taken next.
+
+Event blocks are capable of performing more interesting operations:
+
+  * Transition to a different event or override default caching logic.
+  * Modify the request, response, cache entry, or Rack environment options.
+  * Set the `metastore` or `entitystore` options to select a different storage
+    mechanism / location dynamically.
+  * Collect statistics or log request/response/cache information.
+
+When an event is triggered, the blocks associated with the event are executed in
+reverse/FILO order (i.e., the block declared last runs first) until a
+_transitioning statement_ is encountered. Transitioning statements are suffixed
+with a bang character (e.g, `pass!`, `store!`, `error!`) and cause the current
+event to halt and the machine to transition to the subsequent event; control is
+not returned to the original event. The [default configuration](#default)
+includes documentation on available transitions for each event.
+
+The `next` statement can be used to exit an event block without transitioning
+to another event. Subsequent event blocks are executed until a transitioning
+statement is encountered:
+
+    on :fetch do
+      next if response.freshness_information?
+
+      if request.url =~ /\/feed$/
+        trace 'feed will expire in fifteen minutes'
+        response.ttl = 15 * 60
+      end
+    end
+
+<a id='import'></a>
+
+Importing Configuration
+-----------------------
+
+Since caching logic can be layered, it's possible to separate various bits of
+cache policy into files for organization and reuse.
+
+    use Rack::Cache do
+      import 'rack/cache/config/busters'
+      import 'mycacheconfig'
+
+      # more stuff here
+    end
+
+The `breakers` and `mycacheconfig` configuration files are normal Ruby source
+files (i.e., they have a `.rb` extension) situated on the `$LOAD_PATH` - the
+`import` statement works like Ruby's `require` statement but the contents of the
+files are evaluated in the context of the configuration machinery, as if
+specified directly in the configuration block.
+
+The `rack/cache/config/busters.rb` file makes a good example. It hooks into the
+`fetch` event and adds an impractically long expiration lifetime to any response
+that includes a cache busting query string:
+
+<%= File.read('lib/rack/cache/config/busters.rb').gsub(/^/, '    ') %>
+
+
+<a id='default'></a>
+
+Default Configuration Machinery
+-------------------------------
+
+The `rack/cache/config/default.rb` file is imported when the __Rack::Cache__
+object is instantiated and before any custom configuration code is executed.
+It's useful to understand this configuration because it drives the default
+transitioning logic.
+
+<%= File.read('lib/rack/cache/config/default.rb').gsub(/^/, '    ') %>
+
+<a id='notes'></a>
+
+Notes
+-----
+
+The configuration language was inspired by [Varnish][var]'s
+[VCL configuration language][vcl].
+
+[var]: http://varnish.projects.linpro.no/
+  "Varnish HTTP accelerator"
+
+[vcl]: http://tomayko.com/man/vcl
+  "VCL(7) -- Varnish Configuration Language Manual Page"