rtomayko-rack-cache 0.2.0 → 0.3.0

data/CHANGES

@@ -1,23 +1,58 @@
-## 0.3.0 / ...
-
-  * Cache responses that have a Transfer-Encoding. The current implementation
-    caches the encoded value and reproduces the message with the same
-    Transfer-Encoding header on subsequent cache hits. This goes against
-    RFC 2616, which says that intermediaries must fully decode responses with
-    a Transfer-Encoding header before passing them on (though they may re-encode
-    or apply a different encoding). This seems less relevant to Rack::Cache,
-    however, since we're not a true "hop". We'll see how it goes and act
-    accordingly.
+## 0.3.0 / December 2008
+
+  * Add support for public and private cache control directives. Responses
+    marked as explicitly public are cached even when the request includes
+    an Authorization or Cookie header. Responses marked as explicitly private
+    are considered uncacheable.
+
+  * Added a "private_headers" option that dictates which request headers
+    trigger default "private" cache control processing. By default, the
+    Cookie and Authorization headers are included. Headers may be added or
+    removed as necessary to change the default private logic.
+
+  * Adhere to must-revalidate/proxy-revalidate cache control directives by
+    not assigning the default_ttl to responses that don't include freshness
+    information. This should let us begin using default_ttl more liberally
+    since we can control it using the must-revalidate/proxy-revalidate directives.
+
+  * Use the s-maxage Cache-Control value in preference to max-age when
+    present. The ttl= method now sets the s-maxage value instead of max-age.
+    Code that used ttl= to control freshness at the client needs to change
+    to set the max-age directive explicitly.
+
+  * Enable support for X-Sendfile middleware by responding to #to_path on
+    bodies served from disk storage. Adding the Rack::Sendfile component
+    upstream from Rack::Cache will result in cached bodies being served
+    directly by the web server (instead of being read in Ruby).
+
+  * BUG: MetaStore hits but EntityStore misses. This would 500 previously; now
+    we detect it and act as if the MetaStore missed as well.
+
+  * Implement low level #purge method on all concrete entity store
+    classes -- removes the entity body corresponding to the SHA1 key
+    provided and returns nil.
+
+  * Basically sane handling of HEAD requests. A HEAD request is never passed
+    through to the backend except when transitioning with pass!. This means
+    that the cache responds to HEAD requests without invoking the backend at
+    all when the cached entry is fresh. When no cache entry exists, or the
+    cached entry is stale and can be validated, the backend is invoked with
+    a GET request and the HEAD is handled right before the response
+    is delivered upstream.
+
   * BUG: The Age response header was not being set properly when a stale
     entry was validated. This would result in Age values that exceeded
     the freshness lifetime in responses.
+
   * BUG: A cached entry in a heap meta store could be unintentionally
     modified by request processing since the cached objects were being
     returned directly. The result was typically missing/incorrect header
     values (e.g., missing Content-Type header). [dkubb]
+
   * BUG: 304 responses should not include entity headers (especially
     Content-Length). This is causing Safari/WebKit weirdness on 304
     responses.
+
   * BUG: The If-None-Match header was being ignored, causing the cache
     to send 200 responses to matching conditional GET requests.
 

data/README

@@ -3,16 +3,14 @@ Rack::Cache
 
 Rack::Cache is suitable as a quick drop-in component to enable HTTP caching for
 Rack-based applications that produce freshness (Expires, Cache-Control) and/or
-validation (Last-Modified, ETag) information.
-
-We strive to implement those portions of RFC 2616 Section 13 relevant to gateway
-(i.e., "reverse proxy") cache scenarios with a system for specifying cache
-policy:
+validation (Last-Modified, ETag) information:
 
   * Standards-based (RFC 2616)
   * Freshness/expiration based caching
-  * Validation
+  * Validation (If-Modified-Since / If-None-Match)
   * Vary support
+  * Cache-Control: public, private, max-age, s-maxage, must-revalidate,
+    and proxy-revalidate.
   * Portable: 100% Ruby / works with any Rack-enabled framework
   * Configuration language for advanced caching policies
   * Disk, memcached, and heap memory storage backends
@@ -68,10 +66,17 @@ Assuming you've designed your backend application to take advantage of HTTP's
 caching features, no further code or configuration is required for basic
 caching.
 
-Documentation
--------------
+Links
+-----
 
-http://tomayko.com/src/rack-cache/
+Documentation:
+    http://tomayko.com/src/rack-cache/
+
+Mailing List:
+    http://groups.google.com/group/rack-cache
+
+GitHub:
+    http://github.com/rtomayko/rack-cache/
 
 License
 -------

data/Rakefile

@@ -21,7 +21,7 @@ end
 desc 'Run specs with unit test style output'
 task :test => FileList['test/*_test.rb'] do |t|
   suite = t.prerequisites
-  sh "testrb -Ilib:test #{suite.join(' ')}", :verbose => false
+  sh "specrb -Ilib:test #{suite.join(' ')}", :verbose => false
 end
 
 desc 'Generate test coverage report'
@@ -72,10 +72,10 @@ desc 'Build markdown documentation files'
 task 'doc:markdown'
 FileList['doc/*.markdown'].each do |source|
   dest = "doc/#{File.basename(source, '.markdown')}.html"
-  file dest => source do |f|
+  file dest => [source, 'doc/layout.html.erb'] do |f|
     puts "markdown: #{source} -> #{dest}" if verbose
-    require 'erb'
-    require 'rdiscount'
+    require 'erb' unless defined? ERB
+    require 'rdiscount' unless defined? RDiscount
     template = File.read(source)
     content = Markdown.new(ERB.new(template, 0, "%<>").result(binding), :smart).to_html
     title = content.match("<h1>(.*)</h1>")[1] rescue ''
@@ -87,10 +87,16 @@ FileList['doc/*.markdown'].each do |source|
   CLEAN.include dest
 end
 
+desc 'Publish documentation'
 task 'doc:publish' => :doc do
   sh 'rsync -avz doc/ gus@tomayko.com:/src/rack-cache'
 end
 
+desc 'Start the documentation development server (requires thin)'
+task 'doc:server' do
+  sh 'cd doc && thin --rackup server.ru --port 3035 start'
+end
+
 # PACKAGING =================================================================
 
 def package(ext='')

data/TODO

@@ -1,42 +1,32 @@
-## 0.3
+## 0.4
 
-  - BUG: HEAD request on invalid entry caches zero-length response
-  - BUG: meta store hits but entity misses
-  - BUG: Response body written to cache each time validation succeeds
-    (actually, I'm not positive whether this is happening or not but
-    it looks like it is).
-  - Are we doing HEAD properly?
   - liberal, conservative, sane caching configs
-  - Sample app
-  - busters.rb doc and tests
-  - no-cache.rb doc and tests
+  - Sample apps: Rack, Rails, Sinatra, Merb, etc.
+  - busters.rb and no-cache.rb doc and tests
   - Canonicalized URL for cache key:
     - sorts params by key, then value
     - urlencodes /[^ A-Za-z0-9_.-]/ host, path, and param key/value
-  - Support server-specific X-Sendfile (or similar) for delivering cached
-    bodies (file entity stores only).
-  - Sqlite3 (meta store)
+  - Custom cache keys
   - Cache invalidation on PUT, POST, DELETE.
     - Invalidate at the request URI; or, anything that's "near" the request URI.
     - Invalidate at the URI of the Location or Content-Location response header.
 
 ## Backlog
 
+  - Add missing Expires header if we have a max-age.
   - Purge/invalidate specific cache entries
   - Purge/invalidate everything
   - Maximum size of cached entity
   - Last-Modified factor: requests that have a Last-Modified header but no Expires
     header have a TTL assigned based on the last modified age of the response:
     TTL = (Age * Factor), or, 1h  = (10h * 0.1)
-  - I wonder if it would be possible to run in threaded mode but with an
-    option to lock before making requests to the backend. The idea is to be
-    able to serve requests from cache in separate threads. This should
+  - Run under multiple-threads with an option to lock before making requests
+    to the backend. The idea is to be able to serve requests from cache in
+    separate threads. This should probably be implemented as a separate
+    middleware component.
+  - Consider implementing ESI (http://www.w3.org/TR/esi-lang). This should
     probably be implemented as a separate middleware component.
+  - Sqlite3 (meta store)
   - stale-while-revalidate
   - Serve cached copies when down (see: stale-if-error) - e.g., database
     connection drops and the cache takes over what it can. 
-  - When a cache misses due to Vary, try to validate using the best match. Note
-    that you can't do this with a weak validator, so only strong etags can be
-    used.
-  - Consider implementing ESI (http://www.w3.org/TR/esi-lang). This should
-    probably be implemented as a separate middleware component.

data/doc/configuration.markdown

@@ -115,6 +115,14 @@ If no entitystore is specified, the `heap:/` store is assumed. This
 implementation has significant draw-backs so explicit configuration is
 recommended.
 
+### `private_headers`
+
+An array of request header names that cause the response to be treated with
+private cache control semantics. The default value is `['Authorization', 'Cookie']`.
+If any of these headers are present in the request, the response is considered
+private and will not be cached _unless_ the response is explicitly marked public
+(e.g., `Cache-Control: public`).
+
 <a id='machinery'></a>
 
 Configuration Machinery - Events and Transitions

data/doc/index.markdown

@@ -2,10 +2,10 @@ __Rack::Cache__ is suitable as a quick drop-in component to enable HTTP caching
 for [Rack][]-based applications that produce freshness (`Expires`,
 `Cache-Control`) and/or validation (`Last-Modified`, `ETag`) information.
 
-  * Standards-based ([RFC 2616][rfc] / [Section 13][s13]).
+  * Standards-based (see [RFC 2616][rfc] / [Section 13][s13]).
   * Freshness/expiration based caching
   * Validation
-  * Vary Support
+  * Vary support
   * Portable: 100% Ruby / works with any [Rack][]-enabled framework.
   * [Configuration language][config] for advanced caching policies.
   * Disk, memcached, and heap memory [storage backends][storage].
@@ -47,29 +47,35 @@ simply `require` and `use` as follows:
 
 Assuming you've designed your backend application to take advantage of HTTP's
 caching features, no further code or configuration is required for basic
-caching. More sophisticated stuff is possible with [Rack::Cache's Configuration
-Language][config].
+caching.
 
-Advanced Usage
---------------
+More
+----
 
-  * [Configuration Language Documentation][config] - How to customize cache
+  * [Configuration Language Documentation][config] - how to customize cache
     policy using the simple event-based configuration system.
 
-  * [Cache Storage Documentation][storage] - Detailed information on the various
+  * [Cache Storage Documentation][storage] - detailed information on the various
     storage implementations available in __Rack::Cache__ and how to choose the one
     that's best for your application.
 
-  * [FAQ](./faq) - Frequently Asked Questions about __Rack::Cache__.
+  * [Things Caches Do][things] - an illustrated guide to how HTTP gateway
+    caches work with pointers to other useful resources on HTTP caching.
 
-  * [GitHub Repository](http://github.com/rtomayko/rack-cache/) - Get your
+  * [GitHub Repository](http://github.com/rtomayko/rack-cache/) - get your
     fork on.
 
+  * [Mailing List](http://groups.google.com/group/rack-cache) - for hackers
+    and users (`rack-cache@groups.google.com`).
+
+  * [FAQ](./faq) - Frequently Asked Questions about __Rack::Cache__.
+
   * [RDoc API Documentation](./api/) - Mostly worthless if you just want to use
     __Rack::Cache__ in your application but mildly insightful if you'd like to
     get a feel for how the system has been put together; I recommend
     [reading the source](http://github.com/rtomayko/rack-cache/master/lib/rack/cache).
 
+
 See Also
 --------
 
@@ -99,6 +105,7 @@ and is provided under [the MIT license](./license)
 
 [config]:  ./configuration "Rack::Cache Configuration Language Documentation"
 [storage]: ./storage       "Rack::Cache Storage Documentation"
+[things]:  http://tomayko.com/writings/things-caches-do
 
 [rfc]: http://tools.ietf.org/html/rfc2616
   "RFC 2616 - Hypertext Transfer Protocol -- HTTP/1.1 [ietf.org]"

data/doc/layout.html.erb

@@ -14,6 +14,7 @@
         <p>
           <a href="./configuration" title='Configuration Language Documentation'>Config</a> |
           <a href="./storage" title='Cache Storage Documentation'>Storage</a> |
+          <a href="http://tomayko.com/writings/things-caches-do" title="Things Caches Do">Things</a> |
           <a href="./faq" title='Frequently Asked Questions'>FAQ</a> |
           <a href="./api/" title='Fucking Sucks.'>RDoc</a>
         </p>

data/doc/server.ru

@@ -0,0 +1,34 @@
+# Rackup config that serves the contents of Rack::Cache's
+# doc directory. The documentation is rebuilt on each request.
+
+# Rewrites URLs like conventional web server configs.
+class Rewriter < Struct.new(:app)
+  def call(env)
+    if env['PATH_INFO'] =~ /\/$/
+      env['PATH_INFO'] += 'index.html'
+    elsif env['PATH_INFO'] !~ /\.\w+$/
+      env['PATH_INFO'] += '.html'
+    end
+    app.call(env)
+  end
+end
+
+# Rebuilds documentation on each request.
+class DocBuilder < Struct.new(:app)
+  def call(env)
+    if env['PATH_INFO'] !~ /\.(css|js|gif|jpg|png|ico)$/
+      env['rack.errors'] << "*** rebuilding documentation (rake -s doc)\n"
+      system "rake -s doc"
+    end
+    app.call(env)
+  end
+end
+
+use Rack::CommonLogger
+use DocBuilder
+use Rewriter
+use Rack::Static, :root => File.dirname(__FILE__), :urls => ["/"]
+
+run(lambda{|env| [404,{},'<h1>Not Found</h1>']})
+
+# vim: ft=ruby

data/lib/rack/cache/config/default.rb

@@ -17,7 +17,7 @@
 #
 on :receive do
   pass! unless request.method? 'GET', 'HEAD'
-  pass! if request.header? 'Cookie', 'Authorization', 'Expect'
+  pass! if request.header? 'Expect'
   lookup!
 end
 
@@ -109,7 +109,6 @@ end
 #   * error! - return the error code specified and abandon request.
 #
 on :store do
-  entry.ttl = default_ttl if entry.ttl.nil?
   trace 'store backend response in cache (ttl: %ds)', entry.ttl
   persist!
 end

data/lib/rack/cache/core.rb

@@ -74,9 +74,9 @@ module Rack::Cache
       @triggered.include?(event)
     end
 
-  private
     # Event handlers.
     attr_reader :events
+    private :events
 
   public
     # Attach custom logic to one or more events.
@@ -101,6 +101,13 @@ module Rack::Cache
     end
 
   private
+    # Does the request include authorization or other sensitive information
+    # that should cause the response to be considered private by default?
+    # Private responses are not stored in the cache.
+    def private_request?
+      request.header?(*private_headers)
+    end
+
     # Determine if the #response validators (ETag, Last-Modified) matches
     # a conditional value specified in #original_request.
     def not_modified?
@@ -119,6 +126,7 @@ module Rack::Cache
   private
     def perform_receive
       @original_request = Request.new(@env.dup.freeze)
+      @env['REQUEST_METHOD'] = 'GET' if @original_request.head?
       @request = Request.new(@env)
       info "%s %s", @original_request.request_method, @original_request.fullpath
       transition(from=:receive, to=[:pass, :lookup, :error])
@@ -126,6 +134,7 @@ module Rack::Cache
 
     def perform_pass
       trace 'passing'
+      request.env['REQUEST_METHOD'] = @original_request.request_method
       fetch_from_backend
       transition(from=:pass, to=[:pass, :finish, :error]) do |event|
         if event == :pass
@@ -191,6 +200,20 @@ module Rack::Cache
       request.env.delete('HTTP_IF_MODIFIED_SINCE')
       request.env.delete('HTTP_IF_NONE_MATCH')
       fetch_from_backend
+
+      # mark the response as explicitly private if any of the private
+      # request headers are present and the response was not explicitly
+      # declared public.
+      if private_request? && !@response.public?
+        @response.private = true
+      else
+        # assign a default TTL for the cache entry if none was specified in
+        # the response; the must-revalidate cache control directive disables
+        # default ttl assigment.
+        if default_ttl > 0 && @response.ttl.nil? && !@response.must_revalidate?
+          @response.ttl = default_ttl
+        end
+      end
       transition(from=:fetch, to=[:store, :deliver, :error])
     end
 
@@ -198,7 +221,11 @@ module Rack::Cache
       @entry = @response
       transition(from=:store, to=[:persist, :deliver, :error]) do |event|
         if event == :persist
-          trace "writing response to cache"
+          if @response.private?
+            warn 'forced to store response marked as private.'
+          else
+            trace "storing response in cache"
+          end
           metastore.store(original_request, @entry, entitystore)
           @response = @entry
           :deliver
@@ -211,6 +238,7 @@ module Rack::Cache
     def perform_deliver
       trace "delivering response ..."
       response.not_modified! if not_modified?
+      response.body = [] if @original_request.head?
       transition(from=:deliver, to=[:finish, :error])
     end
 

data/lib/rack/cache/entitystore.rb

@@ -58,6 +58,12 @@ module Rack::Cache
         [key, size]
       end
 
+      # Remove the body corresponding to key; return nil.
+      def purge(key)
+        @hash.delete(key)
+        nil
+      end
+
       def self.resolve(uri)
         new
       end
@@ -89,16 +95,19 @@ module Rack::Cache
         nil
       end
 
-      # Open the entity body and return an IO object. The IO object's
-      # each method is overridden to read 8K chunks instead of lines.
-      def open(key)
-        io = File.open(body_path(key), 'rb')
-        def io.each
+      class Body < ::File #:nodoc:
+        def each
           while part = read(8192)
             yield part
           end
         end
-        io
+        alias_method :to_path, :path
+      end
+
+      # Open the entity body and return an IO object. The IO object's
+      # each method is overridden to read 8K chunks instead of lines.
+      def open(key)
+        Body.open(body_path(key), 'rb')
       rescue Errno::ENOENT
         nil
       end
@@ -121,6 +130,13 @@ module Rack::Cache
         [key, size]
       end
 
+      def purge(key)
+        File.unlink body_path(key)
+        nil
+      rescue Errno::ENOENT
+        nil
+      end
+
     protected
       def storage_path(stem)
         File.join root, stem
@@ -190,6 +206,13 @@ module Rack::Cache
         [key, size]
       end
 
+      def purge(key)
+        cache.delete(key)
+        nil
+      rescue Memcached::NotFound
+        nil
+      end
+
       extend Rack::Utils
 
       # Create MemCache store for the given URI. The URI must specify

data/lib/rack/cache/headers.rb

@@ -21,7 +21,7 @@ module Rack::Cache
     # header is present.
     def cache_control
       @cache_control ||=
-        (headers['Cache-Control'] || '').split(/\s*,\s*/).inject({}) {|hash,token|
+        headers['Cache-Control'].to_s.split(/\s*[,;]\s*/).inject({}) {|hash,token|
           name, value = token.split(/\s*=\s*/, 2)
           hash[name.downcase] = (value || true) unless name.empty?
           hash
@@ -107,16 +107,15 @@ module Rack::Cache
       !fresh?
     end
 
-    # Determine if the response is worth caching under any circumstance. An
-    # object that is cacheable may not necessary be served from cache without
-    # first validating the response with the origin.
+    # Determine if the response is worth caching under any circumstance. Responses
+    # marked "private" with an explicit Cache-Control directive are considered
+    # uncacheable
     #
-    # An object that includes no freshness lifetime (Expires, max-age) and that
-    # does not include a validator (Last-Modified, Etag) serves no purpose in a
-    # cache that only serves fresh or valid objects.
+    # Responses with neither a freshness lifetime (Expires, max-age) nor cache
+    # validator (Last-Modified, Etag) are considered uncacheable.
     def cacheable?
       return false unless CACHEABLE_RESPONSE_CODES.include?(status)
-      return false if no_store?
+      return false if no_store? || private?
       validateable? || fresh?
     end
 
@@ -124,7 +123,8 @@ module Rack::Cache
     # a +Cache-Control+ header with +max-age+ value is present or when the
     # +Expires+ header is set.
     def freshness_information?
-      header?('Expires') || !cache_control['max-age'].nil?
+      header?('Expires') ||
+        !!(cache_control['s-maxage'] || cache_control['max-age'])
     end
 
     # Determine if the response includes headers that can be used to validate
@@ -137,7 +137,7 @@ module Rack::Cache
     # revalidating with the origin. Note that this does not necessary imply that
     # a caching agent ought not store the response in its cache.
     def no_cache?
-      !cache_control['no-cache'].nil?
+      cache_control['no-cache']
     end
 
     # Indicates that the response should not be stored under any circumstances.
@@ -145,6 +145,42 @@ module Rack::Cache
       cache_control['no-store']
     end
 
+    # True when the response has been explicitly marked "public".
+    def public?
+      cache_control['public']
+    end
+
+    # Mark the response "public", making it eligible for other clients. Note
+    # that responses are considered "public" by default unless the request
+    # includes private headers (Authorization, Cookie).
+    def public=(value)
+      value = value ? true : nil
+      self.cache_control = cache_control.
+        merge('public' => value, 'private' => !value)
+    end
+
+    # True when the response has been marked "private" explicitly.
+    def private?
+      cache_control['private']
+    end
+
+    # Mark the response "private", making it ineligible for serving other
+    # clients.
+    def private=(value)
+      value = value ? true : nil
+      self.cache_control = cache_control.
+        merge('public' => !value, 'private' => value)
+    end
+
+    # Indicates that the cache must not serve a stale response in any
+    # circumstance without first revalidating with the origin. When present,
+    # the TTL of the response should not be overriden to be greater than the
+    # value provided by the origin.
+    def must_revalidate?
+      cache_control['must-revalidate'] ||
+      cache_control['proxy-revalidate']
+    end
+
     # The date, as specified by the Date header. When no Date header is present,
     # set the Date header to Time.now and return.
     def date
@@ -163,29 +199,36 @@ module Rack::Cache
 
     # The number of seconds after the time specified in the response's Date
     # header when the the response should no longer be considered fresh. First
-    # check for a Cache-Control max-age value, and fall back on an expires
-    # header; return nil when no maximum age can be established.
+    # check for a s-maxage directive, then a max-age directive, and then fall
+    # back on an expires header; return nil when no maximum age can be
+    # established.
     def max_age
-      if age = cache_control['max-age']
+      if age = (cache_control['s-maxage'] || cache_control['max-age'])
         age.to_i
       elsif headers['Expires']
         Time.httpdate(headers['Expires']) - date
       end
     end
 
-    # Sets the number of seconds after which the response should no longer
-    # be considered fresh. This sets the Cache-Control max-age value.
+    # The number of seconds after which the response should no longer
+    # be considered fresh. Sets the Cache-Control max-age directive.
     def max_age=(value)
       self.cache_control = cache_control.merge('max-age' => value.to_s)
     end
 
+    # Like #max_age= but sets the s-maxage directive, which applies only
+    # to shared caches.
+    def shared_max_age=(value)
+      self.cache_control = cache_control.merge('s-maxage' => value.to_s)
+    end
+
     # The Time when the response should be considered stale. With a
     # Cache-Control/max-age value is present, this is calculated by adding the
     # number of seconds specified to the responses #date value. Falls back to
     # the time specified in the Expires header or returns nil if neither is
     # present.
     def expires_at
-      if max_age = cache_control['max-age']
+      if max_age = (cache_control['s-maxage'] || cache_control['max-age'])
         date + max_age.to_i
       elsif time = headers['Expires']
         Time.httpdate(time)
@@ -200,9 +243,15 @@ module Rack::Cache
       max_age - age if max_age
     end
 
-    # Set the response's time-to-live to the specified number of seconds. This
-    # adjusts the Cache-Control/max-age value.
+    # Set the response's time-to-live for shared caches to the specified number
+    # of seconds. This adjusts the Cache-Control/s-maxage directive.
     def ttl=(seconds)
+      self.shared_max_age = age + seconds
+    end
+
+    # Set the response's time-to-live for private/client caches. This adjusts
+    # the Cache-Control/max-age directive.
+    def client_ttl=(seconds)
       self.max_age = age + seconds
     end
 
@@ -250,8 +299,7 @@ module Rack::Cache
       nil
     end
 
-    # The literal value of the Vary header, or nil when no Vary header is
-    # present.
+    # The literal value of the Vary header, or nil when no header is present.
     def vary
       headers['Vary']
     end