primer 0.1.0 → 0.2.0

data/History.txt

@@ -0,0 +1,17 @@
+=== 0.2.0 / 2011-07-14
+
+* New Lazyness module to avoid ActiveRecord database calls when not needed
+* Handle namespaced ActiveRecord classes when traversing associations
+* Allow Watcher to be added to ActiveRecord classes before associations
+* Ensure compatibility with latest Sinatra router
+* Use AMQP::Channel instead of MQ when using newer amqp gems
+* Add EventMachine-compatible console for platforms with blocking IRBs
+* Move cache updating logic into Worker::ChangesAgent
+
+
+=== 0.1.0 / 2010-11-25
+
+* Initial release. Includes caching, method logging, cache expiry and
+  regeneration, offline workers, real-time page updates and ActiveRecord
+  integration for updating object graphs
+

data/README.rdoc

@@ -1,18 +1,18 @@
 = Primer
 
 This is an experiment to bring automatic cache expiry and regeneration to Rails.
-At Songkick[http://www.songkick.com], we have a ton of code that deals with caches
-and denormalization and messages and offline processing and it's hard to maintain.
-I want to get rid of it. All of it.
+At Songkick[http://www.songkick.com], we have a ton of code that deals with
+caches and denormalization and messages and offline processing and it's hard to
+maintain. I want to get rid of it. All of it.
 
 
 == What?
 
 Inspired by LunaScript[http://asana.com/luna] and Fun[http://marcuswest.in/read/fun-intro/],
 I figured Ruby could figure out which values a computation uses, and use that to
-expire caches for you without having to write any expiry code. This turns out to be
-possible, at least for typical ActiveRecord usage, and Primer includes an engine
-for that.
+expire caches for you without having to write any expiry code. This turns out to
+be possible, at least for typical ActiveRecord usage, and Primer includes an
+engine for that.
 
 Primer currently lets you do the following:
 
@@ -22,14 +22,14 @@ Primer currently lets you do the following:
 * Declare how caches should be recalculated for eager cache population
 * Update pages in real time when their data is updated
 
-It does all this without you having to write a single cache sweeper. You
-just declare how to render your site, Primer does the rest.
+It does all this without you having to write a single cache sweeper. You just
+declare how to render your site, Primer does the rest.
 
 
 == Enough waffle, show me the code!
 
 The following is the minimal, 'hello world' use case. Get your ActiveRecord
-model, put a mixn in it (<em>after</em> declaring the model's associations):
+model, put a mixn in it:
 
   class BlogPost < ActiveRecord::Base
     include Primer::Watcher
@@ -62,30 +62,29 @@ output is cached, the block will not be called again. The cache is invalidated
 when (and only when) the title of <tt>@post</tt> changes; Primer figures this
 out and you don't need to write any cache sweeping code.
 
-Finally you need to bind the cache to the event bus, unless you want to run the
-cache monitoring work in a background process (see below):
+Finally you need to run the cache agent, unless you want to run the cache
+monitoring work in a background process (see below):
 
-  Primer.cache.bind_to_bus
+  Primer::Worker::ChangesAgent.run!
   
   # If you're using ActiveRecord
-  Primer::Worker::ActiveRecordAgent.bind_to_bus
+  Primer::Worker::ActiveRecordAgent.run!
 
 
 === Declaring cache generators
 
 You may have noticed that Primer forces the use of path-style keys for your
-cache. Instead of wrapping code you want to memoize in a block, you can
-declare how to calculate it elsewhere and use a router to map cache keys
-to calculations. For example we could rewrite our post title example like
-this:
+cache. Instead of wrapping code you want to memoize in a block, you can declare
+how to calculate it elsewhere and use a router to map cache keys to calculations.
+For example we could rewrite our post title example like this:
 
   # views/posts/show.html.erb
   # note '=' sign here, not used with block form
   
   <%= primer "/posts/#{@post.id}/title" %>
 
-Then you can declare how to calculate this in a router attached to your
-cache object:
+Then you can declare how to calculate this in a router attached to your cache
+object:
 
   Primer.cache.routes do
     get "/posts/:id/title" do
@@ -96,19 +95,59 @@ cache object:
 
 The advantage of this is that the cache now has a way to generate cache values
 outside of your rendering stack, meaning that instead of just invalidating the
-cache it can actually calculate the new value so the cache is always ready
-for incoming requests.
+cache it can actually calculate the new value so the cache is always ready for
+incoming requests.
 
-It also means you can generate cache content offline; running the following
-will generate the cache of the first post's title:
+It also means you can generate cache content offline; running the following will
+generate the cache of the first post's title:
 
   Primer.cache.compute("/posts/1/title")
 
 
+=== Lazy loading
+
+A common problem when caching view fragments is not knowing whether you need to
+load database objects in the controller. If the parts of the view that use that
+object are already cached, there's no point loading the object since it won't be
+needed to render the page.
+
+Rails has some lazy-loading capability built in, for example if you have a model
+that says <tt>BlogPost.has_many :comments</tt>, a call to <tt>blog_post.comments</tt>
+won't actually load the comments until you call <tt>each</tt> on that collection
+to read the data. Primer introduces the same idea for all database calls.
+
+If you <tt>include Primer::Lazyness</tt> in a model, then <tt>Model.find(id)</tt>
+will not call the database. It just returns an object with the ID you asked for,
+and won't actually load the model from the database until you try to access
+other properties that we cannot infer from the <tt>find()</tt> call.
+
+For example, this makes it easy to load a model in the controller and use its ID
+to key cache fragments, safe in the knowledge that no unnecessary database calls
+will be made.
+
+  BlogPost < ActiveRecord::Base
+    include Primer::Lazyness
+  end
+  
+  # Does not call the database
+  post = BlogPost.find(1)
+  
+  # Does not call the database unless the title is not cached
+  primer "/posts/#{post.id}/title" do
+    post.title
+  end
+
+Primer also lazy-loads on other properties, for example if I call
+<tt>BlogPost.find_by_title("Hello World")</tt> then Primer will create an object
+whose <tt>title</tt> is <tt>"Hello World"</tt> without calling the database, but
+then if I ask for the object's <tt>id</tt> or other properties the real model is
+loaded.
+
+
 === Throttling
 
-Let's say you have a cache value that depends on many little bits of model
-data, a common situation in web front-ends. For example:
+Let's say you have a cache value that depends on many little bits of model data,
+a common situation in web front-ends. For example:
 
   Primer.cache.routes do
     get "/posts/:id/summary" do
@@ -158,15 +197,15 @@ To make a background worker, you just need a file like this:
   Primer.worker!
 
 Running <tt>ruby worker.rb</tt> will start a process in the shell that listens
-for change notifications and updates the cache for you. You can start as many
-of these as you like to spread the load out.
+for change notifications and updates the cache for you. You can start as many of
+these as you like to spread the load out.
 
 
 === Real-time page updates
 
-If you want to be properly web-scale, you'll need to be updating your pages
-in real time as your data changes. Primer lets you update any fragment generated
-by a block-less <tt>primer</tt> call in your view automatically.
+If you want to be properly web-scale, you'll need to be updating your pages in
+real time as your data changes. Primer lets you update any fragment generated by
+a block-less <tt>primer</tt> call in your view automatically.
 
 All you need to do is place some middleware in your Rack config:
 
@@ -181,10 +220,10 @@ Add the client-side script to your templates (this must be in the <tt>HEAD</tt>)
 
   <script type="text/javascript" src="/primer.js"></script>
 
-Then configure it wherever your data model gets used to tell it you want to
-use real-time updates and where the messaging server is. You should also set
-a password - this will stop third parties being able to publish to the message
-bus and inject arbitrary HTML into your pages.
+Then configure it wherever your data model gets used to tell it you want to use
+real-time updates and where the messaging server is. You should also set a
+password - this will stop third parties being able to publish to the message bus
+and inject arbitrary HTML into your pages.
 
   Primer.real_time = true
   Primer::RealTime.bayeux_server = 'http://localhost:9292'
@@ -208,6 +247,6 @@ welcome, pull requests considered, bug reports likely to gather dust.
 
 == License
 
-Copyright (c) 2010 Songkick.com, James Coglan. Named by the inimitable
+Copyright (c) 2010-2011 Songkick.com, James Coglan. Named by the inimitable
 grillpanda[http://github.com/grillpanda]. Released under the MIT license.
 

data/example/README.rdoc

@@ -4,17 +4,19 @@ This is a simple app that demonstrates how you can use Primer. It's based on
 ActiveRecord for the model and Sinatra for the front-end. To start it up,
 install some gems, build the database then use Rack:
 
-  gem install redis amqp faye sinatra activerecord sqlite3-ruby
+  bundle install
   ruby script/setup_database.rb
+  redis-server
+  rabbitmq-server
   rackup -s thin -E production config.ru
 
 
 === models
 
 Let's take a look at the internals. In the +models+ directory, you'll see a
-simple BlogPost model:
+simple blog post model:
 
-  class BlogPost < ActiveRecord::Base
+  class Post < ActiveRecord::Base
     include Primer::Watcher
   end
 
@@ -23,47 +25,55 @@ That's all we need for Primer to monitor how our data changes.
 
 === public, views
 
-The +public+ and +views+ directories are used by the Sinatra app, and should
-be pretty self-explanatory. Note which bits of the views are cached, and which
+The +public+ and +views+ directories are used by the Sinatra app, and should be
+pretty self-explanatory. Note which bits of the views are cached, and which
 parts don't use blocks - we set these keys up in <tt>environment.rb</tt>.
 
 
 === application.rb
 
-The <tt>application.rb</tt> file is a simple Sinatra app - it loads the
-model from <tt>environment.rb</tt> and just sets up a couple of pages.
+The <tt>application.rb</tt> file is a simple Sinatra app - it loads the model
+from <tt>environment.rb</tt> and just sets up a couple of pages.
 
 
 === environment.rb
 
-The meat of the system is in <tt>environment.rb</tt>. This file loads the
-gems we need, loads our models, and configures Primer. See that file for more
-details, but note that it uses an AMQP bus - this means we can compute cache
-changes outside the app process. You'll need to start <tt>worker.rb</tt> so
-that your changes get processed.
+The meat of the system is in <tt>environment.rb</tt>. This file loads the gems
+we need, loads our models, and configures Primer. See that file for more details,
+but note that it uses an AMQP bus - this means we can compute cache changes
+outside the app process. You'll need to start <tt>worker.rb</tt> so that your
+changes get processed.
 
 
 === worker.rb
 
 This file just loads <tt>environment.rb</tt>, then tells Primer to start a
-worker. The call to <tt>Primer.worker!</tt> is blocking, and makes Primer
-pick up messages from the AMQP bus and process them. You can start as many
-of these as you like to distribute the work. Just run:
+worker. The call to <tt>Primer.worker!</tt> is blocking, and makes Primer pick
+up messages from the AMQP bus and process them. You can start as many of these
+as you like to distribute the work. Just run:
 
   ruby worker.rb
 
 If you don't run any of these, messages will build up in AMQP until you start
-a worker. Each data change message goes to only one worker process so you
-won't duplicate any work.
+a worker. Each data change message goes to only one worker process so you won't
+duplicate any work.
 
 
-=== console
+=== console.rb
 
-You can interact with the model by starting IRB with the app's environment:
+You can interact with the model by starting IRB with the app's environment.
+Because EventMachine does not play nice with IRB, you may need to use the custom
+console to script the application:
 
-  irb -r ./environment.rb
+  # in one shell:
+  ruby console.rb server
+  
+  # in another shell
+  ruby console.rb client
+  >> require './environment'
+  >> etc.
 
-Go and create posts and change their data to see how the front-end reacts.
-To get the real-time updates to work you must have a worker running to
-update caches; this is not done by the web app process.
+Go and create posts and change their data to see how the front-end reacts. To
+get the real-time updates to work you must have a worker running to update
+caches; this is not done by the web app process.
 

data/example/application.rb

@@ -14,12 +14,12 @@ class Application < Sinatra::Base
   helpers { include Primer::Helpers::ERB }
   
   get '/' do
-    @posts = BlogPost.all
+    @posts = Post.all
     erb :index
   end
   
   get '/posts/:id' do
-    @post = BlogPost.find(params[:id])
+    @post = Post.find(params[:id])
     erb :show
   end
 end

data/example/console.rb

@@ -0,0 +1,133 @@
+# EventMachine-aware console by Aman Gupta
+# https://gist.github.com/145471
+
+require 'rubygems'
+require 'eventmachine'
+require 'stringio'
+
+module Console
+  PROMPT = "\n>> ".freeze
+
+  def post_init
+    send_data PROMPT
+    send_data "\0"
+  end
+
+  def receive_data data
+    return close_connection if data =~ /exit|quit/
+
+    begin
+      @ret, @out, $stdout = :exception, $stdout, StringIO.new
+      @ret = eval(data, scope, '(rirb)')
+    rescue StandardError, ScriptError, Exception, SyntaxError
+      $! = RuntimeError.new("unknown exception raised") unless $!
+      print $!.class, ": ", $!, "\n"
+
+      trace = []
+      $!.backtrace.each do |line|
+        trace << "\tfrom #{line}"
+        break if line =~ /\(rirb\)/
+      end
+      
+      puts trace
+    ensure
+      $stdout, @out = @out, $stdout
+      @out.rewind
+      @out = @out.read
+    end
+
+    send_data @out unless @out.empty?
+    send_data "=> #{@ret.inspect}" unless @ret == :exception
+    send_data "\0\n>> \0"
+  end
+
+  # def send_data data
+  #   p ['server send', data]
+  #   super
+  # end
+
+  def scope
+    @scope ||= instance_eval{ binding }
+  end
+
+  def handle_error
+    $! = RuntimeError.new("unknown exception raised") unless $!
+    print $!.class, ": ", $!, "\n"
+
+    trace = []
+    $!.backtrace.each do |line|
+      trace << "\\tfrom \#{line}"
+      break if line =~ /\(rirb\)/
+    end
+    
+    puts trace
+  end
+
+  def self.start port = 7331
+    EM.run{
+      @server ||= EM.start_server '127.0.0.1', port, self
+    }
+  end
+
+  def self.stop
+    @server.close_connection if @server
+    @server = nil
+  end
+end
+
+module RIRB
+  def connection_completed
+    p 'connected to console'
+  end
+
+  def receive_data data
+    # p ['receive', data]
+    (@buffer ||= BufferedTokenizer.new("\0")).extract(data).each do |d|
+      process(d)
+    end
+  end
+
+  def process data
+    if data.strip == '>>'
+      while l = Readline.readline('>> ')
+        unless l.nil? or l.strip.empty?
+          Readline::HISTORY.push(l)
+          send_data l
+          break
+        end
+      end
+    else
+      puts data
+    end
+  end
+
+  def unbind
+    p 'disconnected'
+    EM.stop_event_loop
+  end
+  
+  def self.connect host = 'localhost', port = 7331
+    require 'readline'
+    EM.run{
+      trap('INT'){ exit }
+      @connection.close_connection if @connection
+      @connection = EM.connect host, port, self do |c|
+        c.instance_eval{ @host, @port = host, port }
+      end
+    }
+  end
+  attr_reader :host, :port
+end
+
+if __FILE__ == $0
+  EM.run{
+    if ARGV[0] == 'server'
+      Console.start
+    elsif ARGV[0] == 'client'
+      RIRB.connect
+    else
+      puts "#{$0} <server|client>"
+      EM.stop
+    end
+  }
+end

data/example/environment.rb

@@ -6,13 +6,19 @@ require 'active_record'
 require dir + '/../lib/primer'
 
 # Load database config and models
-require dir + '/models/connection'
-require dir + '/models/blog_post'
+require dir + '/db/connection'
+require dir + '/models/post'
+require dir + '/models/comment'
 
 # Configure Primer with a Redis cache and AMQP bus
 Primer.cache = Primer::Cache::Redis.new
 Primer.bus   = Primer::Bus::AMQP.new(:queue => 'blog_changes')
 
+# If you don't want to use AMQP, delete the above bus config
+# and add these lines to run invalidation work in this process:
+# Primer::Worker::ActiveRecordAgent.run!
+# Primer::Worker::ChangesAgent.run!
+
 # Enable real-time page updates
 Primer.real_time = true
 Primer::RealTime.bayeux_server = 'http://0.0.0.0:9292'
@@ -21,11 +27,18 @@ Primer::RealTime.password = 'omg_rofl_scale'
 # Set up cache generation routes
 Primer.cache.routes do
   get '/posts/:id/date' do
-    BlogPost.find(params[:id]).created_at.strftime('%A %e %B %Y')
+    post = Post.find(params[:id])
+    post.created_at.strftime('%A %e %B %Y')
   end
   
   get '/posts/:id/title' do
-    BlogPost.find(params[:id]).title.upcase
+    post = Post.find(params[:id])
+    post.title.upcase
+  end
+  
+  get '/posts/:id/comments' do
+    @post = Post.find(params[:id])
+    ERB.new(File.read(dir + '/views/comments.erb')).result(binding)
   end
 end
 

data/example/models/comment.rb

@@ -0,0 +1,5 @@
+class Comment < ActiveRecord::Base
+  include Primer::Watcher
+  belongs_to :post
+end
+

data/example/models/post.rb

@@ -0,0 +1,5 @@
+class Post < ActiveRecord::Base
+  include Primer::Watcher
+  has_many :comments
+end
+

data/example/script/setup_database.rb

@@ -1,11 +1,17 @@
-require File.expand_path(File.dirname(__FILE__)) + '/../models/connection'
+require File.expand_path(File.dirname(__FILE__)) + '/../db/connection'
 
 ActiveRecord::Schema.define do |version|
-  create_table :blog_posts, :force => true do |t|
+  create_table :posts, :force => true do |t|
     t.timestamps
     t.string :title
     t.text   :body
     t.string :author
   end
+  
+  create_table :comments, :force => true do |t|
+    t.timestamps
+    t.belongs_to :post
+    t.text       :body
+  end
 end
 

data/example/views/comments.erb

@@ -0,0 +1,6 @@
+<h4>Comments</h4>
+
+<% @post.comments.each do |comment| %>
+  <h5><%= comment.created_at.strftime('%A %e %B %Y') %></h5>
+  <p><%= comment.body %></p>
+<% end %>

data/example/views/layout.erb

@@ -18,7 +18,7 @@
     </div></div>
     
     <div class="footer"><div class="sub">
-      <p>Copyright &copy; 2010 That&rsquo;s What She Said</p>
+      <p>Copyright &copy; 2011 I Can&rsquo;t Even Words</p>
     </div></div>
     
   </body>

data/example/views/show.erb

@@ -1,7 +1,11 @@
 <h3><%= @post.title.upcase %></h3>
-<h4>By <%= @post.author %> &mdash; <%= primer "/posts/#{@post.id}/date", :span %></h4>
+<h4>
+  By <%= @post.author %> &mdash;
+  <%= primer "/posts/#{@post.id}/date", :span %>
+</h4>
 
 <% primer "/posts/#{@post.id}/body" do %>
   <p><%= @post.body %></p>
 <% end %>
 
+<%= primer "/posts/#{@post.id}/comments" %>

data/lib/primer/bus/amqp.rb

@@ -1,4 +1,5 @@
-require 'mq'
+require 'amqp'
+require 'mq' unless defined? AMQP::Channel
 
 module Primer
   class Bus
@@ -10,8 +11,8 @@ module Primer
       end
       
       def publish(topic, message)
-        tuple = [topic, message]
-        queue.publish(YAML.dump(tuple))
+        tuple = [topic.to_s, message]
+        queue.publish(Primer.serialize(tuple))
       end
       
       def subscribe(*args, &block)
@@ -25,13 +26,14 @@ module Primer
         Faye.ensure_reactor_running!
         return @queue if defined?(@queue)
         raise "I need a queue name!" unless @config[:queue]
-        @queue = MQ.new.queue(@config[:queue])
+        amqp_klass = defined?(MQ) ? MQ : ::AMQP::Channel
+        @queue = amqp_klass.new.queue(@config[:queue])
       end
       
       def bind
         return if @bound
         queue.subscribe do |message|
-          data = YAML.load(message)
+          data = Primer.deserialize(message)
           distribute(*data)
         end
         @bound = true