primer 0.1.0

data/README.rdoc

@@ -0,0 +1,213 @@
+= Primer
+
+This is an experiment to bring automatic cache expiry and regeneration to Rails.
+At Songkick[http://www.songkick.com], we have a ton of code that deals with caches
+and denormalization and messages and offline processing and it's hard to maintain.
+I want to get rid of it. All of it.
+
+
+== What?
+
+Inspired by LunaScript[http://asana.com/luna] and Fun[http://marcuswest.in/read/fun-intro/],
+I figured Ruby could figure out which values a computation uses, and use that to
+expire caches for you without having to write any expiry code. This turns out to be
+possible, at least for typical ActiveRecord usage, and Primer includes an engine
+for that.
+
+Primer currently lets you do the following:
+
+* Mark up ERB templates with cache keys on Rails and Sinatra
+* Monitor what attributes a cache value depends on
+* Automatically expire a cache when its dependencies change
+* Declare how caches should be recalculated for eager cache population
+* Update pages in real time when their data is updated
+
+It does all this without you having to write a single cache sweeper. You
+just declare how to render your site, Primer does the rest.
+
+
+== Enough waffle, show me the code!
+
+The following is the minimal, 'hello world' use case. Get your ActiveRecord
+model, put a mixn in it (<em>after</em> declaring the model's associations):
+
+  class BlogPost < ActiveRecord::Base
+    include Primer::Watcher
+  end
+
+Set up a cache for your app (you need Redis[http://code.google.com/p/redis/]):
+
+  Primer.cache = Primer::Cache::Redis.new(:host => "10.0.1.1", :port => 6380)
+
+Throw a helper in your views:
+
+  # Rails
+  module ApplicationHelper
+    include Primer::Helpers::ERB
+  end
+  
+  # Sinatra
+  helpers { include Primer::Helpers::ERB }
+
+Wrap cache blocks around your markup for expensive bits:
+
+  # views/posts/show.html.erb
+  
+  <% primer "/posts/#{@post.id}/title" do %>
+    <%= @post.title.upcase %>
+  <% end %>
+
+The output of the block gets cached to Redis using the given key. Once the
+output is cached, the block will not be called again. The cache is invalidated
+when (and only when) the title of <tt>@post</tt> changes; Primer figures this
+out and you don't need to write any cache sweeping code.
+
+Finally you need to bind the cache to the event bus, unless you want to run the
+cache monitoring work in a background process (see below):
+
+  Primer.cache.bind_to_bus
+  
+  # If you're using ActiveRecord
+  Primer::Worker::ActiveRecordAgent.bind_to_bus
+
+
+=== Declaring cache generators
+
+You may have noticed that Primer forces the use of path-style keys for your
+cache. Instead of wrapping code you want to memoize in a block, you can
+declare how to calculate it elsewhere and use a router to map cache keys
+to calculations. For example we could rewrite our post title example like
+this:
+
+  # views/posts/show.html.erb
+  # note '=' sign here, not used with block form
+  
+  <%= primer "/posts/#{@post.id}/title" %>
+
+Then you can declare how to calculate this in a router attached to your
+cache object:
+
+  Primer.cache.routes do
+    get "/posts/:id/title" do
+      post = BlogPost.find(params[:id])
+      post.title.upcase
+    end
+  end
+
+The advantage of this is that the cache now has a way to generate cache values
+outside of your rendering stack, meaning that instead of just invalidating the
+cache it can actually calculate the new value so the cache is always ready
+for incoming requests.
+
+It also means you can generate cache content offline; running the following
+will generate the cache of the first post's title:
+
+  Primer.cache.compute("/posts/1/title")
+
+
+=== Throttling
+
+Let's say you have a cache value that depends on many little bits of model
+data, a common situation in web front-ends. For example:
+
+  Primer.cache.routes do
+    get "/posts/:id/summary" do
+      post = BlogPost.find(params[:id])
+      <<-HTML
+        <div class="post-summary">
+          <h2>#{ post.title }</h2>
+          <ul class="post-meta">
+            <li>Posted #{ post.strftime('%A %e %B %Y') } by #{ post.author.name }</li>
+            <li>Tagged with #{ post.tags.map { |t| link_to(t.name, t) }.join(', ') }</li>
+          </ul>
+        </div>
+      HTML
+    end
+  end
+
+We've got a few domain objects in use here: the post itself, its author, the
+tags attached to the post. We'd want this value regenerating whenever any of
+this data changes, but what if many values that affect this template change at
+around the same time? We might not want to regenerate it for every single change,
+we just want to make sure it looks okay after all the changes have been applied.
+Primer lets you throttle cache regeneration, for example this makes sure each
+key is never regenerated twice within a 5-second interval:
+
+  Primer.cache.throttle = 5
+
+When a value affecting a key changes, Primer will wait 5 seconds before
+regenerating it, allowing other data changes to accrue before we update the
+cache.
+
+
+=== Background workers
+
+You'll probably want to move a lot of the work Primer does out of your front-end
+process. Primer includes an AMQP message bus to support this, and setting it up
+is easy - put this somewhere in your app's setup:
+
+  Primer.bus = Primer::Bus::AMQP.new(:queue => 'my_app_events')
+
+To make a background worker, you just need a file like this:
+
+  # worker.rb
+  
+  # load your models, config, Primer routes etc
+  require 'path/to/app/environment'
+  
+  Primer.worker!
+
+Running <tt>ruby worker.rb</tt> will start a process in the shell that listens
+for change notifications and updates the cache for you. You can start as many
+of these as you like to spread the load out.
+
+
+=== Real-time page updates
+
+If you want to be properly web-scale, you'll need to be updating your pages
+in real time as your data changes. Primer lets you update any fragment generated
+by a block-less <tt>primer</tt> call in your view automatically.
+
+All you need to do is place some middleware in your Rack config:
+
+  # config.ru
+  
+  require 'path/to/sinatra/app'
+  
+  use Primer::RealTime
+  run Sinatra::Application
+
+Add the client-side script to your templates (this must be in the <tt>HEAD</tt>):
+
+  <script type="text/javascript" src="/primer.js"></script>
+
+Then configure it wherever your data model gets used to tell it you want to
+use real-time updates and where the messaging server is. You should also set
+a password - this will stop third parties being able to publish to the message
+bus and inject arbitrary HTML into your pages.
+
+  Primer.real_time = true
+  Primer::RealTime.bayeux_server = 'http://localhost:9292'
+  Primer::RealTime.password = 'super_secret_password'
+
+
+== Examples
+
+See <tt>example/README.rdoc</tt>, a little Sinatra blog with a Redis cache,
+offline cache workers and real-time page updates.
+
+
+== Anything else?
+
+I've tested it on Ruby 1.8.7 and 1.9.2 with Rails 3 and Sinatra 1.1. I've
+briefly tried using it in Rails 2.2 and it looked okay-ish.
+
+I'm NOT using this in production, and neither should you. Ideas and feedback
+welcome, pull requests considered, bug reports likely to gather dust.
+
+
+== License
+
+Copyright (c) 2010 Songkick.com, James Coglan. Named by the inimitable
+grillpanda[http://github.com/grillpanda]. Released under the MIT license.
+

data/example/README.rdoc

@@ -0,0 +1,69 @@
+= Primer example app
+
+This is a simple app that demonstrates how you can use Primer. It's based on
+ActiveRecord for the model and Sinatra for the front-end. To start it up,
+install some gems, build the database then use Rack:
+
+  gem install redis amqp faye sinatra activerecord sqlite3-ruby
+  ruby script/setup_database.rb
+  rackup -s thin -E production config.ru
+
+
+=== models
+
+Let's take a look at the internals. In the +models+ directory, you'll see a
+simple BlogPost model:
+
+  class BlogPost < ActiveRecord::Base
+    include Primer::Watcher
+  end
+
+That's all we need for Primer to monitor how our data changes.
+
+
+=== public, views
+
+The +public+ and +views+ directories are used by the Sinatra app, and should
+be pretty self-explanatory. Note which bits of the views are cached, and which
+parts don't use blocks - we set these keys up in <tt>environment.rb</tt>.
+
+
+=== application.rb
+
+The <tt>application.rb</tt> file is a simple Sinatra app - it loads the
+model from <tt>environment.rb</tt> and just sets up a couple of pages.
+
+
+=== environment.rb
+
+The meat of the system is in <tt>environment.rb</tt>. This file loads the
+gems we need, loads our models, and configures Primer. See that file for more
+details, but note that it uses an AMQP bus - this means we can compute cache
+changes outside the app process. You'll need to start <tt>worker.rb</tt> so
+that your changes get processed.
+
+
+=== worker.rb
+
+This file just loads <tt>environment.rb</tt>, then tells Primer to start a
+worker. The call to <tt>Primer.worker!</tt> is blocking, and makes Primer
+pick up messages from the AMQP bus and process them. You can start as many
+of these as you like to distribute the work. Just run:
+
+  ruby worker.rb
+
+If you don't run any of these, messages will build up in AMQP until you start
+a worker. Each data change message goes to only one worker process so you
+won't duplicate any work.
+
+
+=== console
+
+You can interact with the model by starting IRB with the app's environment:
+
+  irb -r ./environment.rb
+
+Go and create posts and change their data to see how the front-end reacts.
+To get the real-time updates to work you must have a worker running to
+update caches; this is not done by the web app process.
+

data/example/application.rb

@@ -0,0 +1,26 @@
+require 'rubygems'
+require 'sinatra'
+
+class Application < Sinatra::Base
+  ROOT = File.expand_path(File.dirname(__FILE__))
+  
+  require ROOT + '/environment'
+  
+  set :reload_templates, true
+  set :static, true
+  set :public, ROOT + '/public'
+  set :views,  ROOT + '/views'
+  
+  helpers { include Primer::Helpers::ERB }
+  
+  get '/' do
+    @posts = BlogPost.all
+    erb :index
+  end
+  
+  get '/posts/:id' do
+    @post = BlogPost.find(params[:id])
+    erb :show
+  end
+end
+

data/example/config.ru

@@ -0,0 +1,5 @@
+require File.expand_path(File.dirname(__FILE__)) + '/application'
+
+use Primer::RealTime
+run Application
+

data/example/environment.rb

@@ -0,0 +1,31 @@
+dir = File.expand_path(File.dirname(__FILE__))
+
+# Load gems: ActiveRecord and Primer
+require 'rubygems'
+require 'active_record'
+require dir + '/../lib/primer'
+
+# Load database config and models
+require dir + '/models/connection'
+require dir + '/models/blog_post'
+
+# Configure Primer with a Redis cache and AMQP bus
+Primer.cache = Primer::Cache::Redis.new
+Primer.bus   = Primer::Bus::AMQP.new(:queue => 'blog_changes')
+
+# Enable real-time page updates
+Primer.real_time = true
+Primer::RealTime.bayeux_server = 'http://0.0.0.0:9292'
+Primer::RealTime.password = 'omg_rofl_scale'
+
+# Set up cache generation routes
+Primer.cache.routes do
+  get '/posts/:id/date' do
+    BlogPost.find(params[:id]).created_at.strftime('%A %e %B %Y')
+  end
+  
+  get '/posts/:id/title' do
+    BlogPost.find(params[:id]).title.upcase
+  end
+end
+

data/example/models/blog_post.rb

@@ -0,0 +1,4 @@
+class BlogPost < ActiveRecord::Base
+  include Primer::Watcher
+end
+

data/example/models/connection.rb

@@ -0,0 +1,10 @@
+require 'fileutils'
+require 'rubygems'
+require 'active_record'
+
+dir = File.expand_path(File.dirname(__FILE__))
+
+FileUtils.mkdir_p(dir + '/../db')
+dbfile = dir + '/../db/blog.sqlite3'
+ActiveRecord::Base.establish_connection(:adapter  => 'sqlite3', :database => dbfile)
+

data/example/public/style.css

@@ -0,0 +1,75 @@
+body {
+  font: 16px/1.4 FreeSans, Helvetica, Arial, sans-serif;
+  background: #353e4b;
+}
+
+.sub {
+  width: 800px;
+  margin: 0 auto;
+  padding: 1em 2em;
+}
+
+.header {
+  text-shadow: #23282e 0px -2px 0px;
+}
+
+.header h1 {
+  color: #e5dec7;
+  font-size: 4em;
+  letter-spacing: -0.06em;
+  margin: 0;
+}
+
+.header h2 {
+  color: #b3a784;
+  font-size: 1.5em;
+  font-weight: normal;
+  letter-spacing: -0.06em;
+  margin: 0 0 0.5em;
+}
+
+.content .sub {
+  background: #fff;
+  color: #333;
+  -webkit-border-radius: 16px;
+  -moz-border-radius: 16px;
+  border-radius: 16px;
+}
+
+ul {
+  font-size: 1.5em;
+  list-style: none;
+}
+
+ul .date {
+  font-size: 0.8em;
+  letter-spacing: -0.06em;
+  color: #888;
+}
+
+h3 {
+  color: #9ba749;
+  font-size: 3em;
+  letter-spacing: -0.06em;
+  margin: 0;
+}
+
+h4 {
+  color: #888;
+  font-size: 1.5em;
+  font-weight: normal;
+  margin: 0 0 1em;
+}
+
+a {
+  color: #9ba749;
+  font-weight: bold;
+  text-decoration: none;
+}
+
+.footer {
+  font-size: 0.8em;
+  color: #999;
+  text-shadow: #23282e 0px -1px 0px;
+}
+

data/example/script/setup_database.rb

@@ -0,0 +1,11 @@
+require File.expand_path(File.dirname(__FILE__)) + '/../models/connection'
+
+ActiveRecord::Schema.define do |version|
+  create_table :blog_posts, :force => true do |t|
+    t.timestamps
+    t.string :title
+    t.text   :body
+    t.string :author
+  end
+end
+

data/example/views/index.erb

@@ -0,0 +1,13 @@
+<p>Look at all the awesome posts omg</p>
+
+<ul>
+  <% @posts.each do |post| %>
+    <li>
+      <span class="date"><%= post.created_at.strftime '%d %b %Y' %></span>
+      <a href="/posts/<%= post.id %>">
+        <%= primer "/posts/#{post.id}/title", :span %>
+      </a>
+    </li>
+  <% end %>
+</ul>
+

data/example/views/layout.erb

@@ -0,0 +1,26 @@
+<!doctype html>
+<html>
+  <head>
+    <meta http-equiv="Content-type" content="text/html; charset=utf-8">
+    <title>My ROFLscale blog</title>
+    <link rel="stylesheet" href="/style.css">
+    <script type="text/javascript" src="/primer.js"></script>
+  </head>
+  <body>
+    
+    <div class="header"><div class="sub">
+      <h1>My ROFLscale blog</h1>
+      <h2>If you can&rsquo;t store it in <tt>/dev/null</tt> it isn&rsquo;t worth reading</h2>
+    </div></div>
+    
+    <div class="content"><div class="sub">
+      <%= yield %>
+    </div></div>
+    
+    <div class="footer"><div class="sub">
+      <p>Copyright &copy; 2010 That&rsquo;s What She Said</p>
+    </div></div>
+    
+  </body>
+</html>
+

data/example/views/show.erb

@@ -0,0 +1,7 @@
+<h3><%= @post.title.upcase %></h3>
+<h4>By <%= @post.author %> &mdash; <%= primer "/posts/#{@post.id}/date", :span %></h4>
+
+<% primer "/posts/#{@post.id}/body" do %>
+  <p><%= @post.body %></p>
+<% end %>
+

data/example/worker.rb

@@ -0,0 +1,3 @@
+require File.expand_path(File.dirname(__FILE__)) + '/environment'
+Primer.worker!
+

data/lib/javascript/primer.js

@@ -0,0 +1,36 @@
+(function() {
+  window.PRIMER_CHANNELS = window.PRIMER_CHANNELS || [];
+  
+  var BAYEUX_CLIENT = null;
+  
+  var connect = function() {
+    BAYEUX_CLIENT = new Faye.Client('/primer/bayeux');
+    var i = PRIMER_CHANNELS.length;
+    while (i--) listen(PRIMER_CHANNELS.pop());
+    PRIMER_CHANNELS = {push: listen};
+  };
+  
+  var listen = function(channel) {
+    BAYEUX_CLIENT.subscribe(channel, function(message) {
+      var node = document.getElementById(message.dom_id);
+      if (node) node.innerHTML = message.content;
+    });
+  };
+  
+  var script = document.createElement('script'),
+      head   = document.getElementsByTagName('head')[0];
+  
+  script.type = 'text/javascript';
+  script.src  = '/primer/bayeux.js';
+  
+  script.onload = script.onreadystatechange = function() {
+    var state = script.readyState;
+    if (!state || state === 'loaded' || state === 'complete') {
+      script.onload = script.onreadystatechange = null;
+      head.removeChild(script);
+      connect();
+    }
+  };
+  head.appendChild(script);
+})();
+

data/lib/primer/bus/amqp.rb

@@ -0,0 +1,43 @@
+require 'mq'
+
+module Primer
+  class Bus
+    
+    class AMQP < Bus
+      def initialize(config = {})
+        @config = config
+        super()
+      end
+      
+      def publish(topic, message)
+        tuple = [topic, message]
+        queue.publish(YAML.dump(tuple))
+      end
+      
+      def subscribe(*args, &block)
+        bind
+        super
+      end
+      
+    private
+      
+      def queue
+        Faye.ensure_reactor_running!
+        return @queue if defined?(@queue)
+        raise "I need a queue name!" unless @config[:queue]
+        @queue = MQ.new.queue(@config[:queue])
+      end
+      
+      def bind
+        return if @bound
+        queue.subscribe do |message|
+          data = YAML.load(message)
+          distribute(*data)
+        end
+        @bound = true
+      end
+    end
+    
+  end
+end
+

data/lib/primer/bus/memory.rb

@@ -0,0 +1,12 @@
+module Primer
+  class Bus
+    
+    class Memory < Bus
+      def publish(topic, message)
+        distribute(topic, message)
+      end
+    end
+    
+  end
+end
+

data/lib/primer/bus.rb

@@ -0,0 +1,30 @@
+module Primer
+  class Bus
+    
+    autoload :Memory, ROOT + '/primer/bus/memory'
+    autoload :AMQP,   ROOT + '/primer/bus/amqp'
+    
+    def initialize
+      unsubscribe_all
+    end
+    
+    def distribute(topic, message)
+      return unless @listeners.has_key?(topic)
+      @listeners[topic].each { |cb| cb.call(message) }
+    end
+    
+    def subscribe(topic, &listener)
+      @listeners[topic].add(listener)
+    end
+    
+    def unsubscribe(topic, &listener)
+      @listeners[topic].delete(listener)
+    end
+    
+    def unsubscribe_all
+      @listeners = Hash.new { |h,k| h[k] = Set.new }
+    end
+    
+  end
+end
+

data/lib/primer/cache/memory.rb

@@ -0,0 +1,60 @@
+module Primer
+  class Cache
+    
+    class Memory < Cache
+      def initialize
+        clear
+      end
+      
+      def clear
+        @data_store   = {}
+        @relations    = {}
+        @dependencies = {}
+      end
+      
+      def put(cache_key, value)
+        validate_key(cache_key)
+        @data_store[cache_key] = value
+        publish_change(cache_key)
+      end
+      
+      def get(cache_key)
+        validate_key(cache_key)
+        @data_store[cache_key]
+      end
+      
+      def has_key?(cache_key)
+        @data_store.has_key?(cache_key)
+      end
+      
+      def invalidate(cache_key)
+        @data_store.delete(cache_key)
+        return unless deps = @dependencies[cache_key]
+        deps.each do |attribute|
+          @relations[attribute].delete(cache_key)
+        end
+        @dependencies.delete(cache_key)
+      end
+      
+      def relate(cache_key, attributes)
+        deps = @dependencies[cache_key] ||= Set.new
+        attributes.each do |attribute|
+          deps.add(attribute)
+          list = @relations[attribute] ||= Set.new
+          list.add(cache_key)
+        end
+      end
+      
+      def keys_for_attribute(attribute)
+        return [] unless keys = @relations[attribute]
+        keys.to_a
+      end
+      
+      def timeout(cache_key, &block)
+        add_timeout(cache_key, @throttle, &block)
+      end
+    end
+    
+  end
+end
+