vanity 0.2.0

data/CHANGELOG

@@ -0,0 +1,8 @@
+0.1.1
+* Added: experiment method on object, used to define and access experiments.
+* Added: playground configuration (Vanity.playground.namespace = , etc).
+* Added: use_vanity now accepts block instead of symbol.
+* Changed: Vanity::Helpers becomes Vanity::Rails.
+* Changed: A/B test experiments alternatives now handled using Alternatives
+object.
+* Removed: A/B test measure method no longer in use.

data/README.rdoc

@@ -0,0 +1,153 @@
+Vanity is an Experience Driven-Development framework for Rails.
+
+Requires Ruby 1.9 and Redis 1.0 or later.
+
+
+== A/B Testing with Rails (in 5 easy steps)
+
+Add Vanity to your Rails app:
+
+  class ApplicationController < ActionController::Base
+    use_vanity :current_user
+  end
+
+Define an A/B test.  This test compares three pricing options:
+
+  experiment "Price options" do
+    description "Mirror, mirror on the wall, who's the better price of them all?"
+    alternatives 19, 25, 29
+  end
+
+Present different options to the user:
+
+  <h2>Get started for only $<%= ab_test :pricing_options %> a month!</h2>
+
+Measure conversion:
+
+  class SignupController < ApplicationController
+    def signup
+      @account = Account.new(params[:account])
+      if @account.save
+        ab_goal! :pricing_options # <- conversion
+        redirect_to @acccount
+      else
+        render action: :offer
+      end
+    end
+  end
+
+Check the report:
+
+  vanity
+
+
+== A/B Tests
+
+Each A/B experiment represents several (two or more) alternatives.  Use the
+ab_test method to choose an alternative. Call ab_test without a block to return
+the value of the chosen alternative.  Call ab_test with a block to yield with
+the value.
+
+Here are some examples:
+
+  def index
+    if ab_test(:new_page) # classic true/false test
+      render action: "new_page"
+    else
+      render action: "index"
+    end
+  end
+    
+  def index
+    # alternatives are names of templates 
+    render template: ab_test(:new_page)
+  end
+   
+  <%= if ab_test(:banner) %>100% less complexity!<% end %>
+  
+  <%= ab_test(:greeting) %> <%= current_user.name %>
+   
+  <% ab_test :features do |count| %>
+    <%= count %> features to choose from!
+  <% end %>
+
+To measure conversion, call ab_goal! with the experiment's name.  Typically,
+you would do that from a controller action, for example:
+
+  def create
+    ab_goal! :new_page
+    ...
+  end
+
+To measure conversion, simply call ab_goal! with the experiment name.  From the
+Vanity identity set by the filter we know which alternative was presented by
+ab_test, and can correlate conversions to alternative.  It's that simple!
+
+
+== Managing Identity
+
+For effective A/B tests, you want to:
+- Randomly show different alternatives to different people
+- Consistently show the same alternatives to the same person
+- Know which alternative caused a conversion
+- When running multiple tests at once, keep them independent
+
+If you don't use any other mechanism, Vanity will assign a random value to a
+persistent cookie and use it to track the same visitor on subsequent visits.
+Cookie tracking is enabled by use_vanity.
+
+If you keep track of users, you would want to use the user's identity instead.
+Using user identity is more reliable than a cookie tied to a single Web
+browser.
+
+To do that, call use_vanity with the name of a method which returns an object
+with the desired id attribute.  Alternatively, you can use a proc.  These two
+examples are equivalent:
+
+  use_vanity :current_user
+  use_vanity { |controller| controller.current_user.id }
+
+There are times when you would want to use a different identity to distinguish
+test alternatives.  For example, your application may have groups and you may
+want to A/B test an option that will be available (or not) to all people in the
+same group.
+
+You can tell Vanity to use a different identity on a particular controller
+using use_vanity.  Alternatively, you can configure the experiment to extract
+the identity.  The following example will apply to all controllers that have a
+project attribute (without affecting other experiments):
+
+  example "New feature" do
+    description "New feature only available to some groups"
+    identify { |controller| controller.project.id }
+  end
+
+
+== Configuring Vanity
+
+Vanity will work out of the box on a default configuration.  Assuming you're
+using Redis on localhost, post 6379, there's nothing special to do.
+
+If you run a different setup, use the playground object to configure Vanity.
+For example:
+
+  Vanity.playground.host = "redis.local"
+  Vanity.playground.password = "supersecret"
+
+
+== Credits
+
+EDD was all Nathaniel Talbott's idea, I had experience tests to finish for
+Apartly, there was coffee involved and out came the idea for Vanity.
+
+First experiment, A/B tests, heavily influenced by Patrick McKenzie's awesome
+A/Bingo (http://www.bingocardcreator.com/abingo)
+
+Pain points courtesy of Google Analytics's stylish graphs and too-many-clicks
+goal tracking process.
+
+
+
+== License
+
+Vanity, copyright (C) 2009 Assaf Arkin, released under the "Use for good, not evil" license (www.json.org/license.html)

data/lib/vanity.rb

@@ -0,0 +1,23 @@
+require "redis"
+require "openssl"
+
+# All the cool stuff happens in other places:
+# - Vanity::Helpers
+# - Vanity::Playground
+# - Experiment::AbTest
+module Vanity
+  # Version number.
+  module Version
+    version = Gem::Specification.load(File.expand_path("../vanity.gemspec", File.dirname(__FILE__))).version.to_s.split(".").map { |i| i.to_i }
+    MAJOR = version[0]
+    MINOR = version[1]
+    PATCH = version[2]
+    STRING = "#{MAJOR}.#{MINOR}.#{PATCH}"
+  end
+end
+
+
+require File.join(File.dirname(__FILE__), "vanity/playground")
+require File.join(File.dirname(__FILE__), "vanity/experiment/base")
+require File.join(File.dirname(__FILE__), "vanity/experiment/ab_test")
+require File.join(File.dirname(__FILE__), "vanity/rails") if defined?(Rails)

data/lib/vanity/experiment/ab_test.rb

@@ -0,0 +1,170 @@
+module Vanity
+  module Experiment
+
+    # Experiment alternative.  See AbTest#alternatives.
+    class Alternative
+
+      def initialize(experiment, id, value) #:nodoc:
+        @experiment = experiment
+        @id = id
+        @value = value
+      end
+
+      # Alternative id, only unique for this experiment.
+      attr_reader :id
+     
+      # Alternative value.
+      attr_reader :value
+
+      # Number of participants who viewed this alternative.
+      def participants
+        redis.scard(key("participants")).to_i
+      end
+
+      # Number of participants who converted on this alternative.
+      def converted
+        redis.scard(key("converted")).to_i
+      end
+
+      # Number of conversions for this alternative (same participant may be counted more than once).
+      def conversions
+        redis.get(key("conversions")).to_i
+      end
+
+      def participating!(identity)
+        redis.sadd key("participants"), identity
+      end
+
+      def conversion!(identity)
+        if redis.sismember(key("participants"), identity)
+          redis.sadd key("converted"), identity
+          redis.incr key("conversions")
+        end
+      end
+
+    protected
+
+      def key(name)
+        @experiment.key("alts:#{id}:#{name}")
+      end
+
+      def redis
+        @experiment.redis
+      end
+
+    end
+
+    # The meat.
+    class AbTest < Base
+      def initialize(*args) #:nodoc:
+        super
+      end
+
+      # Chooses a value for this experiment.
+      #
+      # This method returns different values for different identity (see
+      # #identify), and consistenly the same value for the same
+      # expriment/identity pair.
+      #
+      # For example:
+      #   color = experiment(:which_blue).choose
+      def choose
+        identity = identify
+        alt = alternative_for(identity)
+        alt.participating! identity
+        alt.value
+      end
+
+      # Records a conversion.
+      #
+      # For example:
+      #   experiment(:which_blue).conversion!
+      def conversion!
+        identity = identify
+        alt = alternative_for(identity)
+        alt.conversion! identity
+        alt.id
+      end
+
+      # Call this method once to specify values for the A/B test.  At least two
+      # values are required.
+      #
+      # Call without argument to previously defined alternatives (see Alternative).
+      #
+      # For example:
+      #   experiment "Background color" do
+      #     alternatives "red", "blue", "orange"
+      #   end
+      #
+      #   alts = experiment(:background_color).alternatives
+      #   puts "#{alts.count} alternatives, with the colors: #{alts.map(&:value).join(", ")}"
+      def alternatives(*args)
+        args = [true, false] if args.empty?
+        @alternatives = []
+        args.each_with_index do |arg, i|
+          @alternatives << Alternative.new(self, i, arg)
+        end
+        class << self ; self ; end.send(:define_method, :alternatives) { @alternatives }
+        alternatives
+      end
+
+      # Sets this test to two alternatives: true and false.
+      def true_false
+        alternatives true, false
+      end
+
+      def report
+        alts = alternatives.map { |alt|
+          "<dt>Option #{(65 + alt.id).chr}</dt><dd><code>#{CGI.escape_html alt.value.inspect}</code> viewed #{alt.participants} times, converted #{alt.conversions}<dd>"
+        }
+        %{<dl class="data">#{alts.join}</dl>}
+      end
+
+      # Forces this experiment to use a particular alternative. Useful for
+      # tests, e.g.
+      #
+      #   setup do
+      #     experiment(:green_button).select(true)
+      #   end
+      #
+      #   def test_shows_green_button
+      #     . . .
+      #   end
+      #
+      # Use nil to clear out selection:
+      #   teardown do
+      #     experiment(:green_button).select(nil)
+      #   end
+      def chooses(value)
+        alternative = alternatives.find { |alt| alt.value == value }
+        raise ArgumentError, "No alternative #{value.inspect} for #{name}" unless alternative
+        Vanity.context.session[:vanity] ||= {}
+        Vanity.context.session[:vanity][id] = alternative.id
+      end
+
+      def humanize
+        "A/B Test" 
+      end
+
+      def save #:nodoc:
+        fail "Experiment #{name} needs at least two alternatives" unless alternatives.count >= 2
+        super
+      end
+
+    private
+
+      # Chooses an alternative for the identity and returns its index. This
+      # method always returns the same alternative for a given experiment and
+      # identity, and randomly distributed alternatives for each identity (in the
+      # same experiment).
+      def alternative_for(identity)
+        session = Vanity.context.session[:vanity]
+        index = session && session[id]
+        index ||= Digest::MD5.hexdigest("#{name}/#{identity}").to_i(17) % alternatives.count
+        alternatives[index]
+      end
+
+    end
+  end
+end
+

data/lib/vanity/experiment/base.rb

@@ -0,0 +1,98 @@
+module Vanity
+  module Experiment
+
+    # Base class that all experiment types are derived from.
+    class Base
+
+      class << self
+        
+        # Returns the type of this class as a symbol (e.g. AbTest becomes
+        # ab_test).
+        def type
+          name.split("::").last.gsub(/([a-z])([A-Z])/) { "#{$1}_#{$2}" }.gsub(/([A-Z])([A-Z][a-z])/) { "#{$1}_#{$2}" }.downcase
+        end
+
+      end
+
+      def initialize(playground, id, name, &block)
+        @playground = playground
+        @id, @name = id.to_sym, name
+        @namespace = "#{@playground.namespace}:#{@id}"
+        created = redis.get(key(:created_at)) || (redis.setnx(key(:created_at), Time.now.to_i) ; redis.get(key(:created_at))) 
+        @created_at = Time.at(created.to_i)
+        @identify_block = ->(context){ context.vanity_identity }
+      end
+
+      # Human readable experiment name, supplied during creation.
+      attr_reader :name
+
+      # Unique identifier, derived from name, e.g. "Green Button" become :green_button.
+      attr_reader :id
+      
+      # Experiment creation timestamp.
+      attr_reader :created_at
+     
+      # Sets or returns description. For example
+      #   experiment :simple do
+      #     description "Simple experiment"
+      #   end
+      #
+      #   puts "Just defined: " + experiment(:simple).description
+      def description(text = nil)
+        @description = text if text
+        @description
+      end
+
+      def report
+        fail "Implement me"
+      end
+      
+      # Called to save the experiment definition.
+      def save #:nodoc:
+      end
+
+      # Call this method with no argument or block to return an identity.  Call
+      # this method with a block to define how to obtain an identity for the
+      # current experiment.
+      #
+      # For example, this experiment use the identity of the project associated
+      # with the controller:
+      #
+      #   class ProjectController < ApplicationController
+      #     before_filter :set_project
+      #     attr_reader :project
+      #
+      #     . . .
+      #   end
+      #
+      #   experiment "Project widget" do
+      #     alternatives :small, :medium, :large
+      #     identify do |controller|
+      #       controller.project.id
+      #     end
+      #   end
+      def identify(&block)
+        if block_given?
+          @identify_block = block
+          self
+        else
+          @identify_block.call(Vanity.context) or fail "No identity found"
+        end
+      end
+
+      # Returns key for this experiment, or with an argument, return a key
+      # using the experiment as the namespace.  Examples:
+      #   key => "vanity:experiments:green_button"
+      #   key("participants") => "vanity:experiments:green_button:participants"
+      def key(name = nil) #:nodoc:
+        name ? "#{@namespace}:#{name}" : @namespace
+      end
+
+      # Shortcut for Vanity.playground.redis
+      def redis #:nodoc:
+        @playground.redis
+      end
+    end
+  end
+end
+

data/lib/vanity/playground.rb

@@ -0,0 +1,127 @@
+require "active_support"
+
+module Vanity
+
+  # Vanity.playground.configuration
+  class Configuration
+  end
+
+  # Playground catalogs all your experiments, holds the Vanity configuration.
+  # For example:
+  #   Vanity.playground.logger = my_logger
+  #   puts Vanity.playground.map(&:name)
+  class Playground
+
+    # Created new Playground. Unless you need to, use the global Vanity.playground.
+    def initialize
+      @experiments = {}
+      @namespace = "vanity:#{Vanity::Version::MAJOR}"
+      @load_path = "experiments"
+    end
+    
+    # Redis host name.  Default is 127.0.0.1
+    attr_accessor :host
+
+    # Redis port number.  Default is 6379.
+    attr_accessor :port
+
+    # Redis database number. Default is 0.
+    attr_accessor :db
+
+    # Redis database password.
+    attr_accessor :password
+
+    # Namespace for database keys.  Default is vanity:n, where n is the major release number, e.g. vanity:1 for 1.0.3.
+    attr_accessor :namespace
+
+    # Path to load experiment files from.
+    attr_accessor :load_path
+
+    # Logger.
+    attr_accessor :logger
+
+    # Defines a new experiment. Generally, do not call this directly,
+    # use #experiment instead.
+    def define(name, options = nil, &block)
+      id = name.to_s.downcase.gsub(/\W/, "_")
+      raise "Experiment #{id} already defined once" if @experiments[id]
+      options ||= {}
+      type = options[:type] || :ab_test
+      klass = Experiment.const_get(type.to_s.gsub(/\/(.?)/) { "::#{$1.upcase}" }.gsub(/(?:^|_)(.)/) { $1.upcase })
+      experiment = klass.new(self, id, name)
+      experiment.instance_eval &block
+      experiment.save
+      @experiments[id] = experiment
+    end
+
+    # Returns the named experiment. You may not have guessed, but this method
+    # raises an exception if it cannot load the experiment's definition.
+    #
+    # Experiment names are always mapped by downcasing them and replacing
+    # non-word characters with underscores, so "Green call to action" becomes
+    # "green_call_to_action". You can also use a symbol if you feel like it.
+    def experiment(name)
+      id = name.to_s.downcase.gsub(/\W/, "_")
+      unless @experiments.has_key?(id)
+        require File.join(load_path, id)
+      end
+      @experiments[id] or fail LoadError, "Expected experiments/#{id}.rb to define experiment #{name}"
+    end
+
+    # Returns list of all loaded experiments.
+    def experiments
+      Dir[File.join(load_path, "*.rb")].each do |file|
+        require file
+      end
+      @experiments.values
+    end
+
+    # Use this instance to access the Redis database.
+    def redis
+      redis = Redis.new(host: self.host, port: self.port, db: self.db,
+                        password: self.password, logger: self.logger)
+      class << self ; self ; end.send(:define_method, :redis) { redis }
+      redis
+    end
+
+  end
+
+  @playground = Playground.new
+  # Returns the playground instance.
+  def self.playground
+    @playground
+  end
+
+  # Returns the Vanity context.  For example, when using Rails this would be
+  # the current controller, which can be used to get/set the vanity identity.
+  def self.context
+    Thread.current[:vanity_context]
+  end
+
+  # Sets the Vanity context.  For example, when using Rails this would be
+  # set by the set_vanity_context before filter (via use_vanity).
+  def self.context=(context)
+    Thread.current[:vanity_context] = context
+  end
+
+end
+
+class Object
+
+  # Use this method to define or access an experiment.
+  # 
+  # To define an experiment, call with a name, options and a block.  For
+  # example:
+  #   experiment "Text size" do
+  #     alternatives :small, :medium, :large
+  #   end
+  #
+  #   puts experiment(:text_size).alternatives
+  def experiment(name, options = nil, &block)
+    if block
+      Vanity.playground.define(name, options, &block)
+    else
+      Vanity.playground.experiment(name)
+    end
+  end
+end

data/lib/vanity/rails.rb

@@ -0,0 +1,13 @@
+require File.join(File.dirname(__FILE__), "../vanity")
+require File.join(File.dirname(__FILE__), "rails/helpers")
+require File.join(File.dirname(__FILE__), "rails/testing")
+
+# Use Rails logger by default.
+Vanity.playground.logger ||= ActionController::Base.logger
+Vanity.playground.load_path = "#{RAILS_ROOT}/experiments"
+
+# Include in controller, add view helper methods.
+ActionController::Base.class_eval do
+  include Vanity::Rails
+  helper Vanity::Rails
+end

data/lib/vanity/rails/helpers.rb

@@ -0,0 +1,121 @@
+module Vanity  
+  # Helper methods for use in your controllers.
+  #
+  # 1) Use Vanity from within your controller:
+  #
+  #   class ApplicationController < ActionController::Base
+  #     use_vanity :current_user end
+  #   end
+  #
+  # 2) Present different options for an A/B test:
+  #
+  #   Get started for only $<%= ab_test :pricing %> a month!
+  #
+  # 3) Measure conversion:
+  #
+  #   def signup
+  #     ab_goal! :pricing
+  #     . . .
+  #   end
+  module Rails
+    module ClassMethods
+
+      # Defines the vanity_identity method, and the set_identity_context before
+      # filter.
+      #
+      # Single argument names a method that returns an object whose identity is
+      # the vanity identity.  Identity is used to present an experiment
+      # consistently to the same person or people.  It can be the user's
+      # identity, group, project.  The object must provide its identity in
+      # response to the method +id+.
+      #
+      # For example, if +current_user+ returns a +User+ object, then to use the
+      # user's id:
+      #   class ApplicationController < ActionController::Base
+      #     use_vanity :current_user
+      #   end
+      #
+      # If that method returns nil (e.g. the user has not signed in), a random
+      # value will be used, instead.  That random value is maintained using a
+      # cookie.
+      #
+      # Alternatively, if you call use_vanity with a block, it will yield to the
+      # block with controller.
+      #
+      # If there is no identity you can use, call use_vanity with the value +nil+.
+      #
+      # For example:
+      #   class ApplicationController < ActionController::Base
+      #     use_vanity :current_user
+      #   end
+      #
+      #   class ProjectController < ApplicationController
+      #     use_vanity { |controller| controller.project_id }
+      #   end
+      def use_vanity(symbol = nil, &block)
+        define_method :vanity_identity do
+          return @vanity_identity if @vanity_identity
+          if block
+            @vanity_identity = block.call(self)
+          elsif symbol && object = send(symbol)
+            @vanity_identity = object.id
+          else
+            @vanity_identity = cookies["vanity_id"] || OpenSSL::Random.random_bytes(16).unpack("H*")[0]
+            cookies["vanity_id"] = { value: @vanity_identity, expires: 1.month.from_now }
+            @vanity_identity
+          end
+        end
+        define_method :set_vanity_context do
+          Vanity.context = self
+        end
+        before_filter :set_vanity_context
+      end
+    end
+
+    def self.included(base) #:nodoc:
+      base.extend ClassMethods
+    end
+
+    # This method returns one of the alternative values in the named A/B test.
+    #
+    # Examples using ab_test inside controller:
+    #   def index
+    #     if ab_test(:new_page) # true/false test
+    #       render action: "new_page"
+    #     else
+    #       render action: "index"
+    #     end
+    #   end
+    #
+    #   def index
+    #     render action: ab_test(:new_page) # alternatives are page names
+    #   end
+    # 
+    # Examples using ab_test inside view:
+    #   <%= if ab_test(:banner) %>100% less complexity!<% end %>
+    #
+    #   <%= ab_test(:greeting) %> <%= current_user.name %>
+    #
+    #   <% ab_test :features do |count| %>
+    #     <%= count %> features to choose from!
+    #   <% end %>
+    def ab_test(name, &block)
+      value = Vanity.playground.experiment(name).choose
+      if block
+        content = capture(value, &block)
+        block_called_from_erb?(block) ? concat(content) : content
+      else
+        value
+      end
+    end
+
+    # This method records conversion on the named A/B test. For example:
+    #   def create
+    #     ab_goal! :call_to_action
+    #     Acccount.create! params[:account]
+    #   end
+    def ab_goal!(name)
+      Vanity.playground.experiment(name).conversion!
+    end
+  end
+end

data/lib/vanity/rails/testing.rb

@@ -0,0 +1,9 @@
+module ActionController #:nodoc:
+  class TestCase
+    alias :setup_controller_request_and_response_without_vanity :setup_controller_request_and_response
+    def setup_controller_request_and_response
+      setup_controller_request_and_response_without_vanity 
+      Vanity.context = @request
+    end
+  end
+end

data/lib/vanity/report.erb

@@ -0,0 +1,22 @@
+<style>
+  ol.experiments { list-style: none; margin: 1em 0; padding: 0; border-bottom: 1px solid #ccc }
+  ol.experiments .hide { display: none }
+  li.experiment h3 { margin: 0 }
+  li.experiment blockquote { margin: 0 }
+  li.experiment .meta { color: #444 }
+  li.experiment .data dt { float: left }
+  li.experiment .green { color: green }
+  li.experiment .red { color: red }
+  li.experiment .data dd { margin: 0 0 .3em 6em }
+</style>
+<h2>Experiments</h2>
+<ol class="experiments">
+  <% experiments.sort_by(&:created_at).each do |exp| %>
+    <li class="experiment" id="experiment_<%= CGI.escape exp.id.to_s %>">
+      <h3><%= CGI.escape_html exp.name %></h3>
+      <blockquote><%= CGI.escape_html exp.description.to_s %></blockquote>
+      <%= exp.report %>
+      <p class="meta"><%= exp.humanize %> started <%= exp.created_at.strftime("%a, %b %-d %Y") %></p>
+    </li>
+  <% end %>
+</ol>

data/test/ab_test_template.erb

@@ -0,0 +1,3 @@
+<% ab_test :simple_ab do |value| %>
+  <%= value %>
+<% end %>

data/test/ab_test_test.rb

@@ -0,0 +1,190 @@
+require "test/test_helper"
+
+class AbTestController < ActionController::Base
+  use_vanity :current_user
+  attr_accessor :current_user
+
+  def test_render
+    render text: ab_test(:simple_ab)
+  end
+
+  def test_view
+    render inline: "<%= ab_test(:simple_ab) %>"
+  end
+
+  def test_capture
+    render file: File.join(File.dirname(__FILE__), "ab_test_template.erb")
+  end
+
+  def goal
+    ab_goal! :simple_ab
+    render text: ""
+  end
+end
+
+
+class AbTestTest < ActionController::TestCase
+  tests AbTestController
+  def setup
+    experiment(:simple_ab) { }
+  end
+
+  # Experiment definition
+
+  def uses_ab_test_when_type_is_ab_test
+    experiment(:ab, type: :ab_test) { }
+    assert_instance_of Vanity::Experiment::AbTest, experiment(:ab)
+  end
+
+  def requires_at_least_two_alternatives_per_experiment
+    assert_raises RuntimeError do
+      experiment :none, type: :ab_test do
+        alternatives []
+      end
+    end
+    assert_raises RuntimeError do
+      experiment :one, type: :ab_test do
+        alternatives "foo"
+      end
+    end
+    experiment :two, type: :ab_test do
+      alternatives "foo", "bar"
+    end
+  end
+
+  # Running experiment
+
+  def returns_the_same_alternative_consistently
+    experiment :foobar do
+      alternatives "foo", "bar"
+      identify { "6e98ec" }
+    end
+    assert value = experiment(:foobar).choose
+    assert_match /foo|bar/, value
+    1000.times do
+      assert_equal value, experiment(:foobar).choose
+    end
+  end
+
+  def returns_different_alternatives_for_each_participant
+    experiment :foobar do
+      alternatives "foo", "bar"
+      identify { rand(1000).to_s }
+    end
+    alts = Array.new(1000) { experiment(:foobar).choose }
+    assert_equal %w{bar foo}, alts.uniq.sort
+    assert_in_delta alts.select { |a| a == "foo" }.count, 500, 100 # this may fail, such is propability
+  end
+
+  def records_all_participants_in_each_alternative
+    ids = (Array.new(200) { |i| i.to_s } * 5).shuffle
+    experiment :foobar do
+      alternatives "foo", "bar"
+      identify { ids.pop }
+    end
+    1000.times { experiment(:foobar).choose }
+    alts = experiment(:foobar).alternatives
+    assert_equal 200, alts.inject(0) { |total,alt| total + alt.participants }
+    assert_in_delta alts.first.participants, 100, 20
+  end
+
+  def records_each_converted_participant_only_once
+    ids = (Array.new(100) { |i| i.to_s } * 5).shuffle
+    test = self
+    experiment :foobar do
+      alternatives "foo", "bar"
+      identify { test.identity ||= ids.pop }
+    end
+    500.times do
+      test.identity = nil
+      experiment(:foobar).choose
+      experiment(:foobar).conversion!
+    end
+    alts = experiment(:foobar).alternatives
+    assert_equal 100, alts.inject(0) { |total,alt| total + alt.converted }
+  end
+
+  def test_records_conversion_only_for_participants
+    test = self
+    experiment :foobar do
+      alternatives "foo", "bar"
+      identify { test.identity ||= rand(100).to_s }
+    end
+    1000.times do
+      test.identity = nil
+      experiment(:foobar).choose
+      experiment(:foobar).conversion!
+      test.identity << "!"
+      experiment(:foobar).conversion!
+    end
+    alts = experiment(:foobar).alternatives
+    assert_equal 100, alts.inject(0) { |t,a| t + a.converted }
+  end
+
+
+  # A/B helper methods
+
+  def test_fail_if_no_experiment
+    new_playground
+    assert_raise MissingSourceFile do
+      get :test_render
+    end
+  end
+
+  def test_ab_test_chooses_in_render
+    responses = Array.new(100) do
+      @controller = nil ; setup_controller_request_and_response
+      get :test_render
+      @response.body
+    end
+    assert_equal %w{false true}, responses.uniq.sort
+  end
+
+  def test_ab_test_chooses_view_helper
+    responses = Array.new(100) do
+      @controller = nil ; setup_controller_request_and_response
+      get :test_view
+      @response.body
+    end
+    assert_equal %w{false true}, responses.uniq.sort
+  end
+
+  def test_ab_test_with_capture
+    responses = Array.new(100) do
+      @controller = nil ; setup_controller_request_and_response
+      get :test_capture
+      @response.body
+    end
+    assert_equal %w{false true}, responses.map(&:strip).uniq.sort
+  end
+
+  def test_ab_test_goal
+    responses = Array.new(100) do
+      @controller.send(:cookies).clear
+      get :goal
+      @response.body
+    end
+  end
+
+
+  # Testing with tests
+  
+  def test_with_given_choice
+    100.times do
+      @controller = nil ; setup_controller_request_and_response
+      experiment(:simple_ab).chooses(true)
+      get :test_render
+      post :goal
+    end
+    alts = experiment(:simple_ab).alternatives
+    assert_equal [100,0], alts.map { |alt| alt.participants }
+    assert_equal [100,0], alts.map { |alt| alt.conversions }
+  end
+
+  def test_which_chooses_non_existent_alternative
+    assert_raises ArgumentError do
+      experiment(:simple_ab).chooses(404)
+    end
+  end
+
+end

data/test/experiment_test.rb

@@ -0,0 +1,45 @@
+require "test/test_helper"
+
+class ExperimentTest < MiniTest::Spec
+  it "creates ID from name" do
+    exp = experiment("Green Button/Alert") { }
+    assert_equal "Green Button/Alert", exp.name
+    assert_equal :green_button_alert, exp.id
+  end
+
+  it "evalutes definition block at creation" do
+    experiment :green_button do
+      expects(:xmts).returns("x")
+    end
+    assert_equal "x", experiment(:green_button).xmts
+  end
+
+  it "saves experiments after defining it" do
+    experiment :green_button do
+      expects(:save)
+    end
+  end
+
+  it "stores when experiment created" do
+    experiment(:simple) { }
+    assert_instance_of Time, experiment(:simple).created_at
+    assert_in_delta experiment(:simple).created_at.to_i, Time.now.to_i, 1
+  end
+
+  it "keeps creation timestamp across definitions" do
+    early = Time.now - 1.day
+    Time.expects(:now).once.returns(early)
+    experiment(:simple) { }
+    assert_equal early.to_i, experiment(:simple).created_at.to_i
+    new_playground
+    experiment(:simple) { }
+    assert_equal early.to_i, experiment(:simple).created_at.to_i
+  end
+
+  it "has description" do
+    experiment :simple do
+      description "Simple experiment"
+    end
+    assert_equal "Simple experiment", experiment(:simple).description
+  end
+end

data/test/playground_test.rb

@@ -0,0 +1,67 @@
+require "test/test_helper"
+
+class PlaygroundTest < MiniTest::Spec
+  before do
+    @namespace = "vanity:0"
+  end
+
+  it "has one global instance" do
+    assert instance = Vanity.playground
+    assert_equal instance, Vanity.playground
+  end
+
+  it "use vanity-{major} as default namespace" do
+    assert @namespace, Vanity.playground.namespace
+  end
+
+  it "fails if it cannot load named experiment from file" do
+    assert_raises MissingSourceFile do
+      experiment("Green button")
+    end
+  end
+
+  it "loads named experiment from experiments directory" do
+    Vanity.playground.expects(:require).with("experiments/green_button")
+    begin
+      experiment("Green button")
+    rescue LoadError=>ex
+    end
+  end
+
+  it "complains if experiment not defined in expected filed" do
+    Vanity.playground.expects(:require).with("experiments/green_button")
+    assert_raises LoadError do
+      experiment("Green button")
+    end
+  end
+
+  it "returns experiment defined in file" do
+    playground = class << Vanity.playground ; self ; end
+    playground.send :define_method, :require do |file|
+      Vanity.playground.define "Green Button" do
+        def xmts ; "x" ; end
+      end
+    end
+    assert_equal "x", experiment("Green button").xmts
+  end
+
+  it "can define and access experiment using symbol" do
+    assert green = experiment("Green Button") { }
+    assert_equal green, experiment(:green_button)
+    assert red = experiment(:red_button) { }
+    assert_equal red, experiment("Red Button")
+  end
+
+  it "detect and fail when defining the same experiment twice" do
+    experiment("Green Button") { }
+    assert_raises RuntimeError do
+      experiment(:green_button) { }
+    end
+  end
+
+  it "uses playground namespace in experiment" do
+    experiment(:green_button) { }
+    assert_equal "#{@namespace}:green_button", experiment(:green_button).send(:key)
+    assert_equal "#{@namespace}:green_button:participants", experiment(:green_button).send(:key, "participants")
+  end
+end

data/test/rails_test.rb

@@ -0,0 +1,76 @@
+require "test/test_helper"
+
+class UseVanityController < ActionController::Base
+  attr_accessor :current_user
+
+  def index
+    render text: ab_test(:simple)
+  end
+end
+
+# Pages accessible to everyone, e.g. sign in, community search.
+class UseVanityTest < ActionController::TestCase
+  tests UseVanityController
+
+  def setup
+    experiment :simple do
+    end
+    UseVanityController.class_eval do
+      use_vanity :current_user
+    end
+  end
+
+  def test_vanity_cookie_is_persistent
+    get :index
+    assert cookie = @response["Set-Cookie"].find { |c| c[/^vanity_id=/] }
+    assert expires = cookie[/vanity_id=[a-f0-9]{32}; path=\/; expires=(.*)(;|$)/, 1]
+    assert_in_delta Time.parse(expires), Time.now + 1.month, 1.minute
+  end
+
+  def test_vanity_cookie_default_id
+    get :index
+    assert_match cookies['vanity_id'], /^[a-f0-9]{32}$/
+  end
+
+  def test_vanity_cookie_retains_id
+    @request.cookies['vanity_id'] = "from_last_time"
+    get :index
+    assert_equal "from_last_time", cookies['vanity_id']
+  end
+
+  def test_vanity_identity_set_from_cookie
+    @request.cookies['vanity_id'] = "from_last_time"
+    get :index
+    assert_equal "from_last_time", @controller.send(:vanity_identity)
+  end
+
+  def test_vanity_identity_set_from_user
+    @controller.current_user = mock("user", id: "user_id")
+    get :index
+    assert_equal "user_id", @controller.send(:vanity_identity)
+  end
+
+  def test_vanity_identity_with_no_user_model
+    UseVanityController.class_eval do
+      use_vanity nil
+    end
+    @controller.current_user = Object.new
+    get :index
+    assert_match cookies['vanity_id'], /^[a-f0-9]{32}$/
+  end
+
+  def test_vanity_identity_set_with_block
+    UseVanityController.class_eval do
+      attr_accessor :project_id
+      use_vanity { |controller| controller.project_id }
+    end
+    @controller.project_id = "576"
+    get :index
+    assert_equal "576", @controller.send(:vanity_identity)
+  end
+
+  def teardown
+    UseVanityController.send(:filter_chain).clear
+    nuke_playground
+  end
+end

data/test/test_helper.rb

@@ -0,0 +1,35 @@
+$LOAD_PATH.delete_if { |path| path[/gems\/vanity-\d/] }
+$LOAD_PATH.unshift File.expand_path("../lib", File.dirname(__FILE__))
+RAILS_ROOT = File.expand_path("..")
+require "minitest/spec"
+require "mocha"
+require "action_controller"
+require "action_controller/test_case"
+require "lib/vanity/rails"
+MiniTest::Unit.autorun
+
+class MiniTest::Unit::TestCase
+  # Call this on teardown. It wipes put the playground and any state held in it
+  # (mostly experiments), resets vanity ID, and clears Redis of all experiments.
+  def nuke_playground
+    Vanity.playground.redis.flushdb
+    new_playground
+    self.identity = nil
+  end
+
+  # Call this if you need a new playground, e.g. to re-define the same experiment,
+  # or reload an experiment (saved by the previous playground).
+  def new_playground
+    Vanity.instance_variable_set :@playground, Vanity::Playground.new
+  end
+
+  attr_accessor :identity # pass identity to/from experiment/test case
+
+  def teardown
+    nuke_playground
+  end
+end
+
+ActionController::Routing::Routes.draw do |map|
+  map.connect ':controller/:action/:id'
+end

data/vanity.gemspec

@@ -0,0 +1,19 @@
+Gem::Specification.new do |spec|
+  spec.name           = "vanity"
+  spec.version        = "0.2.0"
+  spec.author         = "Assaf Arkin"
+  spec.email          = "assaf@labnotes.org"
+  spec.homepage       = "http://github.com/assaf/vanity"
+  spec.summary        = "Experience Driven Development framework for Rails"
+  spec.description    = ""
+  #spec.post_install_message = "To get started run vanity --help"
+
+  spec.files          = Dir["{bin,lib,rails,test}/**/*", "CHANGELOG", "README.rdoc", "vanity.gemspec"]
+
+  spec.has_rdoc         = true
+  spec.extra_rdoc_files = "README.rdoc", "CHANGELOG"
+  spec.rdoc_options     = "--title", "Vanity #{spec.version}", "--main", "README.rdoc",
+                          "--webcvs", "http://github.com/assaf/#{spec.name}"
+
+  spec.add_dependency "redis", "0.1"
+end

metadata

@@ -0,0 +1,86 @@
+--- !ruby/object:Gem::Specification 
+name: vanity
+version: !ruby/object:Gem::Version 
+  version: 0.2.0
+platform: ruby
+authors: 
+- Assaf Arkin
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2009-11-10 00:00:00 -08:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: redis
+  type: :runtime
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - "="
+      - !ruby/object:Gem::Version 
+        version: "0.1"
+    version: 
+description: ""
+email: assaf@labnotes.org
+executables: []
+
+extensions: []
+
+extra_rdoc_files: 
+- README.rdoc
+- CHANGELOG
+files: 
+- lib/vanity/experiment/ab_test.rb
+- lib/vanity/experiment/base.rb
+- lib/vanity/playground.rb
+- lib/vanity/rails/helpers.rb
+- lib/vanity/rails/testing.rb
+- lib/vanity/rails.rb
+- lib/vanity/report.erb
+- lib/vanity.rb
+- test/ab_test_template.erb
+- test/ab_test_test.rb
+- test/experiment_test.rb
+- test/playground_test.rb
+- test/rails_test.rb
+- test/test_helper.rb
+- CHANGELOG
+- README.rdoc
+- vanity.gemspec
+has_rdoc: true
+homepage: http://github.com/assaf/vanity
+licenses: []
+
+post_install_message: 
+rdoc_options: 
+- --title
+- Vanity 0.2.0
+- --main
+- README.rdoc
+- --webcvs
+- http://github.com/assaf/vanity
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.3.5
+signing_key: 
+specification_version: 3
+summary: Experience Driven Development framework for Rails
+test_files: []
+