vanity 1.5.3 → 1.6.0

data/CHANGELOG

@@ -1,3 +1,51 @@
+== 1.6.0 (2011-07-18)
+
+If robots or spiders make up a significant portion of your sites traffic they
+can affect your conversion rate.  Vanity can optionally add participants to the
+experiments using asynchronous JavaScript callbacks, which will keep almost all
+robots out.  To set this up simply do the following:
+
+1. Vanity.playground.use_js!
+2. Set Vanity.playground.add_participant_path = '/path/to/vanity/action'
+3. Add <%= Vanity.vanity_js %> to the bottom of any view that needs to set up
+   an ab_test
+
+
+Fix for metrics on rails 3 models (Esteban Pastorino).
+
+Use JavaScript to report participants, useful for ignoring bots on publicly
+accessible pages (Doug Cole).
+
+AbTest#choose returns an Alternative rather than just the value (Doug Cole).
+
+Add warnings instead of swallowing errors (Anthony Eden).
+
+Fixing broken test for mongodb adapter (Joshua Krall).
+
+Fix returning correct experiment when in test mode and manually set via
+#chooses (Ryan Sonnek).
+
+Don't round the conversion rate before using it, it affects the test results,
+making them less accurate (Doug Cole).
+
+Fixed loading config from yml when using other than redis adapter (Arttu Tervo)
+
+Default to localhost unless host in config file (Arttu Tervo)
+
+Fixed mongo connection adapter connect! when called after disconnect! (Arttu
+Tervo)
+
+Use mongo replica set connection if multiple hosts were given in YAML
+configuration file (Arttu Tervo)
+
+Cookie domain from rails configuration (Arttu Tervo)
+
+Add bson_ext to Gemfile to load C extension for mongodb ruby driver, and
+prevent Notice messages as the tests run (tenaciousflea)
+
+Update redis-namespace dependency to 1.0 (Ville Lautanala)
+
+
 == 1.5.3 (2011-04-11)
 
 Added number of participants and number of converted participants to ab_test
@@ -59,9 +107,9 @@ to multiple adapters (not just Redis). The easiest is to use the configuration
 file config/vanity.yml. For example:
 
   development:
-	  adapter: redis
-	production:
-	  adapter: mongodb
+    adapter: redis
+  production:
+    adapter: mongodb
 
 We get to keep Redis, add new MongoDB adapter, but lose Mock. It's still there,
 but there's a new way to use Vanity outside production: you can turn data
@@ -117,7 +165,7 @@ Next, authenticate using your account credentials.  For example, in your
 config/environments/production.rb:
 
   require "garb"
-	Garb::Session.login('..email..', '..password..', account_type: "GOOGLE") rescue nil
+  Garb::Session.login('..email..', '..password..', account_type: "GOOGLE") rescue nil
 
 Last, define Vanity metrics that tap to Google Analytics metrics.  For example:
 
@@ -179,22 +227,22 @@ This release introduces metrics.  Metrics are the gateway drug to better softwar
 
 It’s as simple as defining a metric:
 
-	metric "Cheers" do
-		description "They love us, don't they?"
-	end
+  metric "Cheers" do
+    description "They love us, don't they?"
+  end
 
 Tracking it from your code:
 
-	track! :cheers
+  track! :cheers
 
 And watching the graph from the Dashboard.
 
 You can (should) also use metrics with your A/B tests, for example:
 
   ab_test "Pricing options" do
-	  metrics :signup
-		alternatives 15, 25, 29
-	end
+    metrics :signup
+    alternatives 15, 25, 29
+  end
 
 This new usage may become requirement in a future release.
 
@@ -213,8 +261,8 @@ Much thanks to Ian Sefferman for fixing issues with Ruby 1.8.7 and Rails support
 This release changes the way you define a new experiment.  You can use a method suitable for the type of experiment you want to define, or call the generic define method (previously: experiment method).  For example:
 
   ab_test "My A/B test" do
-	  alternatives :a, :b
-	end
+    alternatives :a, :b
+  end
 
 The experiment method is no longer overloaded: it looks up an experiment (loading its definition if necessary), but does not define an experiment.  The ab_test method is overloaded, though this may change in the future.
 

data/Gemfile

@@ -13,12 +13,12 @@ group :test do
   gem "garb"
   gem "mocha"
   gem "mongo"
-  gem "passenger"
-  gem "rails", "2.3.8"
-  gem "rack", "1.1.0"
+  gem "bson_ext"
+  gem "mysql"
+  gem "passenger", "~>2.0"
+  gem "rails", "~>2.3.8"
+  gem "rack"
   gem "shoulda"
-  gem "sqlite3-ruby", "1.2.5" # 1.3.0 doesn't like Ruby 1.9.1
   gem "timecop"
-  #gem "SystemTimer"
   gem "webmock"
 end

data/README.rdoc

@@ -62,7 +62,8 @@ And:
 
   vanity report --output vanity.html
   
-  == Rails 3
+
+== Rails 3
   
   There is currently an issue with report generation. The vanity-talk Google Group has a couple posts that outline the issue for now. This is one of the posts: http://groups.google.com/group/vanity-talk/browse_thread/thread/343081a72a0cefb6
   
@@ -73,6 +74,14 @@ And:
     `match '/vanity(/:action(/:id(.:format)))', :controller=>:vanity`
 
 
+== Registering participants with Javascript
+
+If robots or spiders make up a significant portion of your sites traffic they can affect your conversion rate.  Vanity can optionally add participants to the experiments using asynchronous javascript callbacks, which will keep almost all robots out.  To set this up simply do the following:
+
+* Add Vanity.playground.use_js!
+* Set Vanity.playground.add_participant_path = '/path/to/vanity/action', this should point to the add_participant path that is added with Vanity::Rails::Dashboard, make sure that this action is available to all users
+* Add <%= vanity_js %> to any page that needs uses an ab_test. vanity_js needs to be included after your call to ab_test so that it knows which version of the experiment the participant is a member of.  The helper will render nothing if the there are no ab_tests running on the current page, so adding vanity_js to the bottom of your layouts is a good option.  Keep in mind that if you call use_js! and don't include vanity_js in your view no participants will be recorded.
+
 == Contributing
 
 * Fork the project

data/lib/vanity/adapters/mongodb_adapter.rb

@@ -15,13 +15,25 @@ module Vanity
     #
     # @since 1.4.0
     class MongodbAdapter < AbstractAdapter
+      attr_reader :mongo
+
       def initialize(options)
-        @mongo = Mongo::Connection.new(options[:host], options[:port], :connect=>false)
+        setup_connection(options)
         @options = options.clone
         @options[:database] ||= (@options[:path] && @options[:path].split("/")[1]) || "vanity"
         connect!
       end
 
+      def setup_connection(options = {})
+        if options[:hosts]
+          args = (options[:hosts].map{|host| [host, options[:port]] } << {:connect => false})
+          @mongo = Mongo::ReplSetConnection.new(*args)
+        else
+          @mongo = Mongo::Connection.new(options[:host], options[:port], :connect => false)
+        end
+        @mongo
+      end
+
       def active?
         @mongo.connected?
       end
@@ -38,6 +50,7 @@ module Vanity
       end
 
       def connect!
+        @mongo ||= setup_connection(@options)
         @mongo.connect
         database = @mongo.db(@options[:database])
         database.authenticate @options[:username], @options[:password], true if @options[:username]
@@ -45,11 +58,14 @@ module Vanity
         @experiments = database.collection("vanity.experiments")
         @participants = database.collection("vanity.participants")
         @participants.create_index [[:experiment, 1], [:identity, 1]], :unique=>true
+        @participants.create_index [[:experiment, 1], [:seen, 1]]
+        @participants.create_index [[:experiment, 1], [:converted, 1]]
+        @mongo
       end
 
       def to_s
         userinfo = @options.values_at(:username, :password).join(":") if @options[:username]
-        URI::Generic.build(:scheme=>"mongodb", :userinfo=>userinfo, :host=>@options[:host], :port=>@options[:port], :path=>"/#{@options[:database]}").to_s
+        URI::Generic.build(:scheme=>"mongodb", :userinfo=>userinfo, :host=>(@mongo.host || @options[:host]), :port=>(@mongo.port || @options[:port]), :path=>"/#{@options[:database]}").to_s
       end
 
       def flushdb
@@ -112,8 +128,8 @@ module Vanity
       def ab_counts(experiment, alternative)
         record = @experiments.find_one({ :_id=>experiment }, { :fields=>[:conversions] })
         conversions = record && record["conversions"]
-        { :participants => @participants.find({ :experiment=>experiment, :seen=>alternative }, { :fields=>[] }).count,
-          :converted    => @participants.find({ :experiment=>experiment, :converted=>alternative }, { :fields=>[] }).count,
+        { :participants => @participants.find({ :experiment=>experiment, :seen=>alternative }).count,
+          :converted    => @participants.find({ :experiment=>experiment, :converted=>alternative }).count,
           :conversions  => conversions && conversions[alternative.to_s] || 0 }
       end
 
@@ -131,16 +147,16 @@ module Vanity
       end
 
       def ab_add_participant(experiment, alternative, identity)
-        @participants.update({ :experiment=>experiment, :identity=>identity }, { "$set"=>{ :seen=>alternative } }, :upsert=>true)
+        @participants.update({ :experiment=>experiment, :identity=>identity }, { "$push"=>{ :seen=>alternative } }, :upsert=>true)
       end
 
       def ab_add_conversion(experiment, alternative, identity, count = 1, implicit = false)
         if implicit
-          @participants.update({ :experiment=>experiment, :identity=>identity }, { "$set"=>{ :seen=>alternative } }, :upsert=>true)
+          @participants.update({ :experiment=>experiment, :identity=>identity }, { "$push"=>{ :seen=>alternative } }, :upsert=>true)
         else
           participating = @participants.find_one(:experiment=>experiment, :identity=>identity, :seen=>alternative)
         end
-        @participants.update({ :experiment=>experiment, :identity=>identity }, { "$set"=>{ :converted=>alternative } }, :upsert=>true) if implicit || participating
+        @participants.update({ :experiment=>experiment, :identity=>identity }, { "$push"=>{ :converted=>alternative } }, :upsert=>true) if implicit || participating
         @experiments.update({ :_id=>experiment }, { "$inc"=>{ "conversions.#{alternative}"=>count } }, :upsert=>true)
       end
 

data/lib/vanity/adapters/redis_adapter.rb

@@ -26,7 +26,13 @@ module Vanity
       end
 
       def disconnect!
-        @redis.quit rescue nil if @redis
+        if redis
+          begin
+            redis.quit
+          rescue Exception => e
+            warn("Error while disconnecting from redis: #{e.message}")
+          end
+        end
         @redis = nil
       end
 
@@ -42,7 +48,7 @@ module Vanity
       end
 
       def to_s
-        @redis.id
+        redis.id
       end
 
       def redis

data/lib/vanity/experiment/ab_test.rb

@@ -6,12 +6,11 @@ module Vanity
     # One of several alternatives in an A/B test (see AbTest#alternatives).
     class Alternative
 
-      def initialize(experiment, id, value, participants, converted, conversions)
+      def initialize(experiment, id, value) #, participants, converted, conversions)
         @experiment = experiment
         @id = id
         @name = "option #{(@id + 65).chr}"
         @value = value
-        @participants, @converted, @conversions = participants, converted, conversions
       end
 
       # Alternative id, only unique for this experiment.
@@ -27,13 +26,22 @@ module Vanity
       attr_reader :experiment
 
       # Number of participants who viewed this alternative.
-      attr_reader :participants
+      def participants
+        load_counts unless @participants
+        @participants
+      end
 
       # Number of participants who converted on this alternative (a participant is counted only once).
-      attr_reader :converted
+      def converted
+        load_counts unless @converted
+        @converted
+      end
 
       # Number of conversions for this alternative (same participant may be counted more than once).
-      attr_reader :conversions
+      def conversions
+        load_counts unless @conversions
+        @conversions
+      end
 
       # Z-score for this alternative, related to 2nd-best performing alternative. Populated by AbTest#score.
       attr_accessor :z_score
@@ -44,9 +52,9 @@ module Vanity
       # Difference from least performing alternative. Populated by AbTest#score.
       attr_accessor :difference
 
-      # Conversion rate calculated as converted/participants, rounded to 3 places.
+      # Conversion rate calculated as converted/participants
       def conversion_rate
-        @conversion_rate ||= (participants > 0 ? (converted.to_f/participants.to_f * 1000).round / 1000.0 : 0.0)
+        @conversion_rate ||= (participants > 0 ? converted.to_f/participants.to_f  : 0.0)
       end
 
       # The measure we use to order (sort) alternatives and decide which one is better (by calculating z-score).
@@ -71,25 +79,32 @@ module Vanity
         "#{name}: #{value} #{converted}/#{participants}"
       end
 
+      def load_counts
+        if @experiment.playground.collecting?
+          @participants, @converted, @conversions = @experiment.playground.connection.ab_counts(@experiment.id, id).values_at(:participants, :converted, :conversions)
+        else
+          @participants = @converted = @conversions = 0
+        end
+      end
     end
 
 
-    # The meat.
-    class AbTest < Base
-      class << self
+      # The meat.
+      class AbTest < Base
+        class << self
 
-        # Convert z-score to probability.
-        def probability(score)
-          score = score.abs
-          probability = AbTest::Z_TO_PROBABILITY.find { |z,p| score >= z }
-          probability ? probability.last : 0
-        end
+          # Convert z-score to probability.
+          def probability(score)
+            score = score.abs
+            probability = AbTest::Z_TO_PROBABILITY.find { |z,p| score >= z }
+            probability ? probability.last : 0
+          end
 
-        def friendly_name
-          "A/B Test" 
-        end
+          def friendly_name
+            "A/B Test" 
+          end
 
-      end
+        end
 
       def initialize(*args)
         super
@@ -139,8 +154,7 @@ module Vanity
       def _alternatives
         alts = []
         @alternatives.each_with_index do |value, i|
-          counts = @playground.collecting? ? connection.ab_counts(@id, i) : Hash.new(0)
-          alts << Alternative.new(self, i, value, counts[:participants], counts[:converted], counts[:conversions])
+          alts << Alternative.new(self, i, value)
         end
         alts
       end
@@ -187,8 +201,10 @@ module Vanity
             index = connection.ab_showing(@id, identity)
             unless index
               index = alternative_for(identity)
-              connection.ab_add_participant @id, index, identity
-              check_completion!
+              if !@playground.using_js?
+                connection.ab_add_participant @id, index, identity
+                check_completion!
+              end
             end
           else
             index = connection.ab_get_outcome(@id) || alternative_for(identity)
@@ -197,8 +213,9 @@ module Vanity
           identity = identity()
           @showing ||= {}
           @showing[identity] ||= alternative_for(identity)
+          index = @showing[identity]
         end
-        @alternatives[index.to_i]
+        alternatives[index.to_i]
       end
 
       # Returns fingerprint (hash) for given alternative.  Can be used to lookup
@@ -233,8 +250,16 @@ module Vanity
             connection.ab_not_showing @id, identity
           else
             index = @alternatives.index(value)
+            #add them to the experiment unless they are already in it
+            unless index == connection.ab_showing(@id, identity)
+              connection.ab_add_participant @id, index, identity
+              check_completion!
+            end
             raise ArgumentError, "No alternative #{value.inspect} for #{name}" unless index
-            connection.ab_show @id, identity, index
+            if (connection.ab_showing(@id, identity) && connection.ab_showing(@id, identity) != index) || 
+         alternative_for(identity) != index
+              connection.ab_show @id, identity, index
+            end
           end
         else
           @showing ||= {}
@@ -247,7 +272,7 @@ module Vanity
       def showing?(alternative)
         identity = identity()
         if @playground.collecting?
-          connection.ab_showing(@id, identity) == alternative.id
+          (connection.ab_showing(@id, identity) || alternative_for(identity)) == alternative.id
         else
           @showing ||= {}
           @showing[identity] == alternative.id
@@ -379,9 +404,9 @@ module Vanity
         if @outcome_is
           begin
             result = @outcome_is.call
-            outcome = result.id if result && result.experiment == self
-          rescue
-            # TODO: logging
+            outcome = result.id if Alternative === result && result.experiment == self
+          rescue 
+            warn "Error in AbTest#complete!: #{$!}"
           end
         else
           best = score.best

data/lib/vanity/experiment/base.rb

@@ -80,6 +80,8 @@ module Vanity
       # Button" becomes :green_button.
       attr_reader :id
 
+      attr_reader :playground
+
       # Time stamp when experiment was created.
       def created_at
         @created_at ||= connection.get_experiment_created_at(@id)
@@ -124,7 +126,7 @@ module Vanity
         @description = text if text
         @description
       end
-
+ 
 
       # -- Experiment completion --
 
@@ -189,7 +191,7 @@ module Vanity
           begin
             complete! if @complete_block.call
           rescue
-            # TODO: logging
+            warn "Error in Vanity::Experiment::Base: #{$!}"
           end
         end
       end

data/lib/vanity/frameworks/rails.rb

@@ -46,7 +46,11 @@ module Vanity
               @vanity_identity = object.id
             elsif response # everyday use
               @vanity_identity = cookies["vanity_id"] || ActiveSupport::SecureRandom.hex(16)
-              cookies["vanity_id"] = { :value=>@vanity_identity, :expires=>1.month.from_now }
+              cookie = { :value=>@vanity_identity, :expires=>1.month.from_now }
+              # Useful if application and admin console are on separate domains.
+              # This only works in Rails 3.x.
+              cookie_options[:domain] ||= ::Rails.application.config.session_options[:domain] if ::Rails.respond_to?(:application)
+              cookies["vanity_id"] = cookie
               @vanity_identity
             else # during functional testing
               @vanity_identity = "test"
@@ -140,7 +144,14 @@ module Vanity
       #     <%= count %> features to choose from!
       #   <% end %>
       def ab_test(name, &block)
-        value = Vanity.playground.experiment(name).choose
+        if Vanity.playground.using_js?
+          @_vanity_experiments ||= {}
+          @_vanity_experiments[name] ||= Vanity.playground.experiment(name).choose
+          value = @_vanity_experiments[name].value
+        else
+          value = Vanity.playground.experiment(name).choose.value
+        end
+ 
         if block
           content = capture(value, &block)
           block_called_from_erb?(block) ? concat(content) : content
@@ -149,6 +160,13 @@ module Vanity
         end
       end
 
+      def vanity_js
+        return if @_vanity_experiments.nil?
+        javascript_tag do
+          render Vanity.template("vanity.js.erb")
+        end
+      end
+
       def vanity_h(text)
         h(text)
       end
@@ -186,6 +204,16 @@ module Vanity
         exp.chooses(exp.alternatives[params[:a].to_i].value)
         render :file=>Vanity.template("_experiment"), :locals=>{:experiment=>exp}
       end
+
+      def add_participant
+        if params[:e].nil? || params[:e].empty?
+          render :status => 404, :nothing => true
+          return
+        end
+        exp = Vanity.playground.experiment(params[:e])
+        exp.chooses(exp.alternatives[params[:a].to_i].value)
+        render :status => 200, :nothing => true
+      end
     end
   end
 end

data/lib/vanity/helpers.rb

@@ -34,7 +34,14 @@ module Vanity
     #   end
     # @since 1.2.0
     def ab_test(name, &block)
-      value = Vanity.playground.experiment(name).choose
+      if Vanity.playground.using_js?
+        @_vanity_experiments ||= {}
+        @_vanity_experiments[name] ||= Vanity.playground.experiment(name).choose
+        value = @_vanity_experiments[name].value
+      else
+        value = Vanity.playground.experiment(name).choose.value
+      end
+
       if block
         content = capture(value, &block)
         block_called_from_erb?(block) ? concat(content) : content

data/lib/vanity/playground.rb

@@ -10,6 +10,7 @@ module Vanity
   class Playground
 
     DEFAULTS = { :collecting => true, :load_path=>"experiments" }
+    DEFAULT_ADD_PARTICIPANT_PATH = '/vanity/add_participant'
 
     # Created new Playground. Unless you need to, use the global
     # Vanity.playground.
@@ -38,7 +39,7 @@ module Vanity
       end
 
       @options = defaults.merge(config).merge(options)
-      if @options.values_at(:host, :port, :db).any?
+      if @options[:host] == 'redis' && @options.values_at(:host, :port, :db).any?
         warn "Deprecated: please specify Redis connection as URL (\"redis://host:port/db\")"
         establish_connection :adapter=>"redis", :host=>@options[:host], :port=>@options[:port], :database=>@options[:db] || @options[:database]
       elsif @options[:redis]
@@ -58,6 +59,8 @@ module Vanity
         @logger.level = Logger::ERROR
       end
       @loading = []
+      @use_js = false
+      self.add_participant_path = DEFAULT_ADD_PARTICIPANT_PATH
       @collecting = !!@options[:collecting]
     end
    
@@ -70,6 +73,9 @@ module Vanity
     # Logger.
     attr_accessor :logger
 
+    # Path to the add_participant action, necessary if you have called use_js!
+    attr_accessor :add_participant_path
+
     # Defines a new experiment. Generally, do not call this directly,
     # use one of the definition methods (ab_test, measure, etc).
     #
@@ -95,6 +101,34 @@ module Vanity
       experiments[id.to_sym] or raise NameError, "No experiment #{id}"
     end
 
+
+    # -- Robot Detection --
+
+    # Call to indicate that participants should be added via js
+    # This helps keep robots from participating in the ab test
+    # and skewing results.
+    #
+    # If you use this, there are two more steps:
+    # - Set Vanity.playground.add_participant_path = '/path/to/vanity/action',
+    #   this should point to the add_participant path that is added with
+    #   Vanity::Rails::Dashboard, make sure that this action is available
+    #   to all users
+    # - Add <%= vanity_js %> to any page that needs uses an ab_test. vanity_js
+    #   needs to be included after your call to ab_test so that it knows which
+    #   version of the experiment the participant is a member of.  The helper
+    #   will render nothing if the there are no ab_tests running on the current
+    #   page, so adding vanity_js to the bottom of your layouts is a good
+    #   option.  Keep in mind that if you call use_js! and don't include
+    #   vanity_js in your view no participants will be recorded.
+    def use_js!
+      @use_js = true
+    end
+
+    def using_js?
+      @use_js
+    end
+
+
     # Returns hash of experiments (key is experiment id).
     #
     # @see Vanity::Experiment
@@ -329,7 +363,6 @@ module Vanity
       path << ".erb" unless name["."]
       path
     end
-
   end
 end
 

data/lib/vanity/templates/_vanity.js.erb

@@ -0,0 +1,20 @@
+<% unless @_vanity_experiments.empty? %>
+  var httpRequest;
+  <%
+    @_vanity_experiments.each do |name, alternative| 
+  %>
+    var params = "e=<%= name %>&a=<%= alternative.id %>&authenticity_token=" + encodeURIComponent("<%= form_authenticity_token %>");
+    if (window.XMLHttpRequest) { // Mozilla, Safari, ...
+      httpRequest = new XMLHttpRequest();
+    } else if (window.ActiveXObject) { // IE
+      try { httpRequest = new ActiveXObject("Msxml2.XMLHTTP"); }
+      catch (e) { }
+    }
+    if (httpRequest) {
+      httpRequest.open('POST', "<%= Vanity.playground.add_participant_path %>", true);
+      httpRequest.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
+      httpRequest.setRequestHeader("X-Requested-With", "XMLHttpRequest");
+      httpRequest.send(params);
+    }
+  <% end %>
+<% end %>

data/lib/vanity/version.rb

@@ -1,5 +1,5 @@
 module Vanity
-  VERSION = "1.5.3"
+  VERSION = "1.6.0"
 
   module Version
     version = VERSION.to_s.split(".").map { |i| i.to_i }

data/test/adapters/redis_adapter_test.rb

@@ -0,0 +1,17 @@
+require 'test/test_helper'
+
+class RedisAdapterTest < Test::Unit::TestCase
+  def test_warn_on_disconnect_error
+    if defined?(Redis)
+      assert_nothing_raised do
+        Redis.any_instance.stubs(:connect!)
+        mocked_redis = stub("Redis")
+        mocked_redis.expects(:quit).raises(RuntimeError)
+        redis_adapter = Vanity::Adapters::RedisAdapter.new({})
+        redis_adapter.expects(:warn).with("Error while disconnecting from redis: RuntimeError")
+        redis_adapter.stubs(:redis).returns(mocked_redis)
+        redis_adapter.disconnect!
+      end
+    end
+  end
+end