redis_template_resolver 0.1.1

data/.gitignore

@@ -0,0 +1,2 @@
+.*.sw[po]
+coverage/

data/CHANGELOG

@@ -0,0 +1,6 @@
+v0.1.1
+  - Fix dependency in gemspec
+  - Add CHANGELOG file
+
+v0.1.0
+  - Initial independent release

data/README.rdoc

@@ -0,0 +1,84 @@
+= Redis Template Resolver
+
+== Overview
+This gem handles remote templates - fetching via http and caching the
+result locally in the process (first cache level), as well as in a redis
+instance (second cache level), as well as fallback handling if an error
+was encountered during the fetching of the template.
+
+The caches obviously allow faster access to the fetched result, where the
+second level redis cache allows a centralized, authoritative result to be
+shared between multiple server processes.
+
+By default, the process-local cache will be refreshed from the redis cached
+result every 60 seconds (configurable).
+
+If an error was encountered during the fetching of the template via http, 
+the library will fall back onto a configurable default template to serve, which
+will be refreshed after a configurable timeout.
+
+Once a template has been fetched successfully, it will not time out or be 
+re-fetched. To fetch the template anew, the redis cache entry needs to be 
+removed, which will trigger a new http fetch the next time the first level
+cache will be refreshed from the second level cache.
+
+== Configuration
+
+The following options can be set on the class in a config/initializer/ file:
+
++RedisTemplateResolver.default_template+:: 
+  *mandatory* Body of the default template to use if the remote template cannot
+  be retrieved. Note that this is the actual template, not the path to a
+  template file.
++RedisTemplateResolver.redis_connector+::
+  *mandatory* The redis connector to use for storing templates in the second
+  level cache. Note that this is an already configured instance of Redis, not
+  the connection parameters.
++RedisTemplateResolver.local_cache_ttl+:: 
+  The number of seconds between refreshing the in-process cache from redis
+  (defaults to 60).
++RedisTemplateResolver.local_cache_negative_ttl+:: 
+  The number of seconds to wait before retrying to fetch the template via http,
+  if anything went wrong (defaults to 10).
++RedisTemplateResolver.http_timeout+:: 
+  Seconds to wait for the fetch via http to complete (defaults to 5 seconds).
++RedisTemplateResolver.template_handler_name+:: 
+  Name of the template handler that should render the fetched template
+  (defaults to "erb").
+
+= Usage
+Create a class that derives from RedisTemplateResolver.
+
+This class *must* override the following method: +lookup_template_url+. The
+method will receive the name of the template to be retrieved in +@template_name+
+and should return a full URL to the template.
+
+Optionally the following two methods can also be overridden:
+
++postprocess_template( template_body )+:: 
+  Called right after successfully retrieving the template file, which is passed
+  in as an argument. The method should return the template body to be actually
+  stored and used. If you want to perform sanity checks on the returned
+  template, you can raise an +TemplateRejectedError+ exception, which will
+  refuse the retrieved template and pretend that template retrieval failed.
++resolver_guard( name, prefix, partial, details)+:: 
+  If this method is present and returns false or nil, Redis Template Resolver
+  will not handle this template. Return true to process the template resolution
+  with this library.
+
+In your application controller, add the following line:
+
+<tt>append_view_path MyRedisTemplateResolver.new</tt>
+
+where MyRedisTemplateResolver is the name of the class that derives from the
+RedisTemplateResolver class. Supply a template name with a prefix of "redis:"
+to route the lookup request to the redis template resolver.
+
+E.g. to fetch your layouts via http, use the following:
+
+<tt>  layout lambda { |args| "redis:some_template_name}" }</tt>
+
+
+
+
+

data/lib/redis_template_resolver.rb

@@ -0,0 +1,230 @@
+=begin rdoc
+  Looks for templates in a redis cache, The Rails Way(tm). 
+  The lookup procedure is as follows:
+
+  1. Determine current consuming application name. If unknown, use default 
+     template.
+
+  2. look up template in local cache. 
+     If found in local cache, and not expired: Use this template.
+     If found in local cache, but expired: delete from cache, continue to 
+     next step.
+     If not found in local cache: continue to next step.
+
+  3. Attempt to retrieve from redis cache.
+     If found, write back to local cache, with a lifetime of 
+     +local_cache_ttl+ seconds.
+     If not found, continue to next step.
+  
+  4. Attempt to retrieve from remote URL, configured for the consuming 
+     application. 
+     If successful, write back to redis (indefinitely) and local cache
+     with an expiration of +local_cache_ttl+ seconds.
+     If not successful, continue to next step.
+
+     Note that there is not mutex for the HTTP retrieval, so ALL rails 
+     processes could theoretically perform this step at the same time.
+     Fixme: Add Mutex handling with redis.
+
+  5. Write default template to local cache with a lifetime of 
+     +local_cache_negative_ttl+ seconds to avoid hammering redis and the
+     remote server.
+     Use the default template.
+
+  NOTE: This class has some stuff hard wired into it that makes it (as is)
+        only useful to look up mustache templates that will be used for a
+        view layout. 
+=end
+
+class RedisTemplateResolver < ActionView::Resolver
+  class TemplateRejectedError < StandardError ; end
+
+  @@local_cache_ttl = 60 # seconds
+  cattr_accessor :local_cache_ttl
+
+  @@local_cache_negative_ttl = 10 # seconds
+  cattr_accessor :local_cache_negative_ttl
+
+  @@http_timeout = 5 # seconds
+  cattr_accessor :http_timeout
+
+  @@template_handler_name = "erb"
+  cattr_accessor :template_handler_name
+
+  cattr_accessor :cache
+  cattr_accessor :default_template
+  cattr_accessor :redis_connector
+
+  self.caching = false
+
+=begin rdoc
+  Clears the local cache completely.
+=end
+  def self.clear_cache
+    @@cache = {}
+  end
+
+=begin rdoc
+  Called by rails to fetch the template.
+  Returns an array of ActionView::Template instances. 
+  If successful, only one template is returned inside the returned array.
+
+  Note that to indicate lookup failure, this function MUST return an empty
+  array!
+=end
+  def find_templates( name, prefix, partial, details )
+    Rails.logger.debug "Called RedisTemplateResolver"
+
+    return [] unless name.start_with?( "redis:" ) 
+    if respond_to?( :resolver_guard )
+      return [] unless resolver_guard( name, prefix, partial, details )
+    end
+
+
+    _, @template_name = name.split(':', 2 )
+    Rails.logger.debug "RedisTemplateResolver fetching template with name: #{@template_name}"
+
+    template_body = fetch_template
+
+    path = ActionView::Resolver::Path.build( name, prefix, nil )
+    handler = ActionView::Template.handler_for_extension( self.template_handler_name )
+  
+    template = ActionView::Template.new( template_body, path, handler,
+                                         :virtual_path => path.virtual,
+                                         :format       => [ :html ],
+                                         :updated_at   => Time.now )
+    return [ template ]
+  end
+
+  protected
+=begin rdoc
+  Performs the lookup steps described in the class description.
+  Handles the fallbacks on lookup failures in the caches.
+=end
+  def fetch_template
+    template_body = fetch_template_from_local_cache
+    template_body ||= fetch_template_from_redis
+    template_body ||= fetch_remote_template_and_store
+    template_body ||= store_template_to_local_cache( self.default_template,
+                                                     self.local_cache_negative_ttl )
+
+    return template_body 
+  end
+
+=begin rdoc
+  Triggers the fetching of the template from the remote URL, and writes
+  it back to redis and the local cache on success.
+
+  Returns the template contents as a string, or nil on failure.
+=end
+  def fetch_remote_template_and_store
+    template_body = fetch_template_from_url
+
+    if template_body
+      store_template_to_local_cache( template_body )
+      store_template_to_redis( template_body )
+    end
+
+    return template_body
+  end
+
+=begin rdoc
+  Returns the redis key used for template lookups.
+=end
+  def redis_key
+    return "rlt:#{@template_name}"
+  end
+
+=begin rdoc
+  Performs the HTTP retrieval of a template from the remote url configured.
+  Validates the consuming application name and encoding of the returned 
+  template, if the HTTP request was answered by a 200 code.
+
+  Returns the template contents as a string, or nil if any of the steps fail.
+=end
+  def fetch_template_from_url
+    layout_url = lookup_template_url
+    return nil if layout_url == nil
+
+    Rails.logger.info "Fetching remote template from #{layout_url.inspect}"
+
+    response = HTTParty.get( layout_url, :timeout => self.http_timeout )
+    response_body = response.body
+
+    Rails.logger.info "Got remote template response code #{response.code} with body #{response_body.inspect}"
+
+    return nil if response.code == 404
+    
+    response_body = postprocess_template( response_body ) if respond_to?( :postprocess_template )
+
+    return response_body
+    
+  rescue Timeout::Error, SocketError, Errno::ECONNREFUSED, TemplateRejectedError => e
+    Rails.logger.error e.message
+    return nil
+  end
+
+  def lookup_template_url
+    # :nocov:
+    raise "Override me! Expect to find the template name in @template_name"
+    # :nocov:
+  end
+
+=begin rdoc
+  Returns the template contents from the local cache, checking the expiration 
+  time as it does so.
+
+  Returns the template content on success, or nil on failure. 
+
+  Also removes the template from the cache if the entry is expired.
+=end
+  def fetch_template_from_local_cache
+    @@cache ||= {}
+    
+    return nil unless @@cache[@template_name]
+
+    if @@cache[@template_name][:expiration].to_i <= Time.now.to_i
+      Rails.logger.debug( "Local cache entry is too old, removing!" )
+      @@cache.delete( @template_name )
+      return nil
+    end
+
+    return @@cache[@template_name][:template]
+  end
+
+=begin rdoc
+  Fetches the template from redis, and writes the retrieved value to the
+  local cache on success.
+
+  Returns the template contents as a string, or nil on failure.
+=end
+  def fetch_template_from_redis
+    result = self.redis_connector.get( redis_key )
+    store_template_to_local_cache( result ) if result
+
+    return result
+  end
+
+=begin rdoc
+  Does the actual storing of the template contents in the local cache.
+  The ttl value defaults to the +local_cache_ttl+ and is interpreted as being
+  given in seconds.
+
+  Returns the passed in template body.
+=end
+  def store_template_to_local_cache( template_body, ttl = self.local_cache_ttl )
+    @@cache ||= {}
+
+    @@cache[@template_name] = { :template   => template_body,
+                                :expiration => Time.now.to_i + ttl }
+
+    return template_body
+  end
+
+=begin rdoc
+  Stores the given template body to the redis cache.
+=end
+  def store_template_to_redis( template_body )
+    return self.redis_connector.set( redis_key, template_body )
+  end
+end

data/redis_template_resolver.gemspec

@@ -0,0 +1,28 @@
+Gem::Specification.new do |s|
+  s.name = "redis_template_resolver"
+  s.version = "0.1.1"
+  s.homepage = "https://github.com/psd/redis_template_resolver"
+  s.summary = "A template resolver for rails that retrieves templates via http and caches them in redis"
+
+  s.description = <<-EOS
+    A template resolver for rails 3.2. It retrieves templates via HTTP 
+    and stores them in a redis instance, as well as in local caches. 
+  EOS
+
+  s.authors = [ "ProSiebenSat.1 Ditital, Central Systems Architecture" ]
+  s.email = ["sven.riedel@prosiebensat1digital.de"]
+
+  s.files         = `git ls-files`.split("\n")
+  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+
+  s.add_runtime_dependency( "redis", "> 3.0.0" )
+  s.add_runtime_dependency( "httparty", "> 0.8.0" )
+  s.add_runtime_dependency( "rails", "~> 3.2.6" )
+  s.add_runtime_dependency( "activesupport", ">3.2.0" )
+
+  s.add_development_dependency( "rspec", ">2.10.0" )
+  s.add_development_dependency( "simplecov" )
+  s.add_development_dependency( "timecop" )
+end
+

data/spec/lib/redis_template_resolver_spec.rb

@@ -0,0 +1,192 @@
+require 'spec_helper'
+
+class ExampleClass < RedisTemplateResolver
+  def lookup_template_url
+    return "http://www.example.com/some/path"
+  end
+end
+
+describe ExampleClass do
+
+  let( :dummy_remote_template ) { "dummy template {{{flash}}} {{{body}}} {{{head}}} {{{headline}}} {{{title}}}" }
+  let( :dummy_remote_template_with_html_escaped_tags ) { "dummy template {{flash}} {{body}} {{head}} {{headline}} {{title}}" }
+
+  let( :fake_httparty_response ) do
+    OpenStruct.new( :code => 200, :body => dummy_remote_template )
+  end
+
+  let( :fake_httparty_response_with_html_escaped_tags ) do
+    OpenStruct.new( :code => 200, :body => dummy_remote_template_with_html_escaped_tags )
+  end
+
+  let( :redis_connector ) do
+    double( Redis, :get => nil, :set => nil )
+  end
+
+  before( :each ) do
+    described_class.redis_connector = redis_connector
+    described_class.clear_cache
+  end
+
+  describe "#find_templates" do
+    it "should return an empty array if the template isn't a layout" do
+      result = subject.find_templates( "foo", "not_a_layout", "", {} )
+      result.should == []
+    end
+
+    it "should return an empty array if the template name doesn't start with redis:" do
+      result = subject.find_templates( "not_redis:template", "layouts", "", {} )
+      result.should == []
+    end
+
+    context "with a guard function supplied" do
+      it 'should abort processing if the guard function returns false' do
+        ExampleClass.any_instance.stub( :resolver_guard ).and_return( false )
+        result = subject.find_templates( "redis:template", "layouts", "", {} )
+        result.should == []
+      end
+
+      it 'should continue processing if the guard function returns true' do
+        HTTParty.stub!( :get => fake_httparty_response )
+        ExampleClass.any_instance.stub( :resolver_guard ).and_return( true )
+        result = subject.find_templates( "redis:template", "layouts", "", {} )
+        result.first.source.should == dummy_remote_template
+      end
+    end
+
+    context "when retrieving a relevant template" do
+      let( :name ) { "redis:kabeleins_local" }
+      let( :prefix ) { "layouts" }
+      let( :partial ) { "" }
+      let( :details ) { {} }
+    
+      context "when retrieving the template from the local cache" do
+        it "should fall back when the template is unknown" do
+          described_class.redis_connector.should_receive( :get ).once.and_return( nil )
+          subject.find_templates( name, prefix, partial, details )
+        end
+
+        it "should fall back when the consuming application is unknown" do
+          described_class.redis_connector.should_receive( :get ).once.and_return( nil )
+          subject.find_templates( "redis:no_such_application", prefix, partial, details )
+        end
+
+        context "if the template in the local cache is too old" do
+          before( :each ) do
+            Timecop.travel( described_class.local_cache_ttl.seconds.ago ) do
+              HTTParty.stub!( :get ).and_return( fake_httparty_response )
+              subject.find_templates( name, prefix, partial, details )
+            end
+          end
+
+          it "should fall back and remove the template if it is older than the local cache TTL" do
+            described_class.redis_connector.should_receive( :get ).once.and_return( nil ) 
+            subject.find_templates( name, prefix, partial, details )
+          end
+        end
+        
+        context "if the template is in the local cache less than the local cache TTL" do
+          before( :each ) do
+            target_time = described_class.local_cache_ttl - 2
+            Timecop.travel( target_time.seconds.ago ) do
+              HTTParty.stub!( :get => fake_httparty_response )
+              subject.find_templates( name, prefix, partial, details )
+            end
+          end
+
+          it "should use the found template and not fall back" do
+            redis_connector.should_not_receive( :get )
+            result = subject.find_templates( name, prefix, partial, details )
+            result.first.source.should == dummy_remote_template
+          end
+        end
+      end
+
+      context "when the template cannot be found in the local cache" do
+        before( :each ) do
+          described_class.clear_cache
+        end
+
+        context "if a template could be found in the redis cache" do
+          before( :each ) do
+            redis_connector.stub!( :get ).and_return( dummy_remote_template )
+          end
+
+          it "should store the template in the local cache for 60 seconds" do
+            subject.find_templates( name, prefix, partial, details )
+            described_class.cache.should have_key( "kabeleins_local" )
+
+            cache_entry = described_class.cache["kabeleins_local"]
+            cache_entry.should have_key( :template )
+            cache_entry.should have_key( :expiration )
+            cache_entry[:template].should == dummy_remote_template
+            expected_expiration_time = Time.now.to_i + described_class.local_cache_ttl
+            cache_entry[:expiration].should be_between( expected_expiration_time - 2,
+                                                        expected_expiration_time )
+          end
+
+          it "should not fall back to http" do
+            HTTParty.should_not_receive( :get )
+            subject.find_templates( name, prefix, partial, details )
+          end
+
+        end
+
+        context "if no template could be found in the redis cache" do
+          before( :each ) do
+            redis_connector.stub!( :get )
+          end
+
+          context "if the template could be retrieved via HTTP" do
+            before( :each ) do
+              HTTParty.stub!( :get => fake_httparty_response )
+            end
+
+            it "should store the template in redis" do
+              redis_connector.should_receive( :set ).once.with( "rlt:kabeleins_local",
+                                                       dummy_remote_template )
+              subject.find_templates( name, prefix, partial, details )
+            end
+
+            it "should store the template in the local cache" do
+              subject.find_templates( name, prefix, partial, details )
+              described_class.cache.should have_key( "kabeleins_local" )
+
+              cache_entry = described_class.cache["kabeleins_local"]
+              cache_entry.should have_key( :template )
+              cache_entry.should have_key( :expiration )
+              cache_entry[:template].should == dummy_remote_template
+              expected_expiration_time = Time.now.to_i + described_class.local_cache_ttl
+              cache_entry[:expiration].should be_between( expected_expiration_time - 2,
+                                                          expected_expiration_time )
+            end
+          end
+
+          context "if the template could not be retrieved via HTTP" do
+            before( :each ) do
+              HTTParty.stub!( :get ).and_raise( Timeout::Error )
+            end
+
+            it "should use the default template" do
+              result = subject.find_templates( name, prefix, partial, details )
+              result.first.source.should == described_class.default_template
+            end
+
+            it "should store the default template in the local cache for 10 seconds" do
+              subject.find_templates( name, prefix, partial, details )
+              described_class.cache.should have_key( "kabeleins_local" )
+
+              cache_entry = described_class.cache["kabeleins_local"]
+              cache_entry.should have_key( :template )
+              cache_entry.should have_key( :expiration )
+              cache_entry[:template].should == described_class.default_template
+              expected_expiration_time = Time.now.to_i + described_class.local_cache_negative_ttl
+              cache_entry[:expiration].should be_between( expected_expiration_time - 2,
+                                                          expected_expiration_time )
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/spec/spec_helper.rb

@@ -0,0 +1,21 @@
+require 'simplecov'
+require 'redis'
+require 'rails'
+require 'action_view'
+require 'active_support/core_ext/numeric/time'
+require 'httparty'
+require 'timecop'
+require 'ostruct'
+
+SimpleCov.start
+require_relative "../lib/redis_template_resolver"
+
+RSpec.configure do |config|
+  config.order = "random"
+  config.mock_with :rspec
+
+  config.before( :each ) do
+    logger = double( "Logger" ).as_null_object
+    Rails.logger = logger
+  end
+end

metadata

@@ -0,0 +1,166 @@
+--- !ruby/object:Gem::Specification
+name: redis_template_resolver
+version: !ruby/object:Gem::Version
+  version: 0.1.1
+  prerelease: 
+platform: ruby
+authors:
+- ProSiebenSat.1 Ditital, Central Systems Architecture
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-11-08 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: redis
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>'
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>'
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+- !ruby/object:Gem::Dependency
+  name: httparty
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>'
+      - !ruby/object:Gem::Version
+        version: 0.8.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>'
+      - !ruby/object:Gem::Version
+        version: 0.8.0
+- !ruby/object:Gem::Dependency
+  name: rails
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 3.2.6
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 3.2.6
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>'
+      - !ruby/object:Gem::Version
+        version: 3.2.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>'
+      - !ruby/object:Gem::Version
+        version: 3.2.0
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>'
+      - !ruby/object:Gem::Version
+        version: 2.10.0
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>'
+      - !ruby/object:Gem::Version
+        version: 2.10.0
+- !ruby/object:Gem::Dependency
+  name: simplecov
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: timecop
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: ! "    A template resolver for rails 3.2. It retrieves templates via
+  HTTP \n    and stores them in a redis instance, as well as in local caches. \n"
+email:
+- sven.riedel@prosiebensat1digital.de
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- CHANGELOG
+- README.rdoc
+- lib/redis_template_resolver.rb
+- redis_template_resolver.gemspec
+- spec/lib/redis_template_resolver_spec.rb
+- spec/spec_helper.rb
+homepage: https://github.com/psd/redis_template_resolver
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.23
+signing_key: 
+specification_version: 3
+summary: A template resolver for rails that retrieves templates via http and caches
+  them in redis
+test_files: []