vanity 1.8.4 → 1.9.0.beta

data/.travis.yml

@@ -20,8 +20,9 @@ gemfile:
   - gemfiles/rails32.gemfile
   - gemfiles/rails4.gemfile
 before_script:
-- if [[ "`basename $BUNDLE_GEMFILE`" == "Gemfile" ]]; then rvm rubygems --force 1.8.25; fi # Rubygems 2.0.x fails with Rails 2.3
-- "mysql -e 'create database vanity_test;' >/dev/null"
+- 'echo ''gem: --no-ri --no-rdoc'' > ~/.gemrc' # skip installing docs for gems
+- if [[ "`basename $BUNDLE_GEMFILE`" == "Gemfile" ]]; then rvm rubygems 1.8.25; fi # Rubygems 2.0.x fails with Rails 2.3
+- if [[ "$DB" == "mysql" ]]; then mysql -e 'create database vanity_test;' >/dev/null; fi
 #- if [[ "$DB" == "pgsql" ]]; then psql -c 'create database vanity_test;' -U postgres >/dev/null; fi
 matrix:
   exclude:

data/CHANGELOG

@@ -1,3 +1,15 @@
+== 1.9.0 (2014-01-18)
+
+Change order of preference for vanity identity, from highest to lowest: URL param, cookie, method call, assignment. (@phillbaker)
+Add Playground#request_filter for AbTest#choose(s) (@phillbaker)
+Call on_assignment and perform rebalance when using js (@tmandry)
+Add Adapter#experiment_persisted? and check before showing Dashboard. (@phillbaker)
+Show info message at top of report if not collection (@hellochar)
+Add gracefull failover when redis is down/unreachable. (@phillbaker)
+Update README and documentation for clarity. (@nextmat)
+Remove hard Redis gem dependency from gemspec. (@phillbaker)
+Update appraisal to v1.0.0. (@phillbaker)
+
 == 1.8.3 (2013-11-20)
 
 Add support for Rails 4. (@phillbaker)

data/Gemfile

@@ -9,10 +9,13 @@ gem "rails", "~>2.3.8"
 gem "passenger", "~>2.0"
 
 # Persistence
+gem "redis", ">= 2.1"
+gem "redis-namespace", ">= 1.1.0"
 gem "bson_ext"
 gem "mongo"
 gem "mysql"
-gem "pg"
+gem "sqlite3"
+# gem "pg"
 
 # Math libraries
 gem "backports", :platforms => :mri_18
@@ -26,14 +29,14 @@ gem "garb"
 gem "SystemTimer", "1.2.3", :platforms => :mri_18
 
 # Testing
-gem "appraisal"
-
 gem "mocha", :require=>false
 gem "shoulda", :require=>false # Requires test/unit
 gem "timecop", :require=>false
 gem "webmock", :require=>false
 
 group :development do
+  gem "appraisal", ">= 1.0.0.beta2" # For setting up test Gemfiles
+
   gem "jekyll"
   gem "rake"
   gem "RedCloth"

data/Gemfile.lock

@@ -1,9 +1,7 @@
 PATH
   remote: .
   specs:
-    vanity (1.8.4)
-      redis (>= 2.1)
-      redis-namespace (>= 1.1.0)
+    vanity (1.8.3)
 
 GEM
   remote: https://rubygems.org/
@@ -23,9 +21,10 @@ GEM
     addressable (2.2.7)
     albino (1.3.3)
       posix-spawn (>= 0.3.6)
-    appraisal (0.5.2)
+    appraisal (1.0.0.beta2)
       bundler
       rake
+      thor (>= 0.14.0)
     backports (3.3.5)
     bson (1.6.0)
     bson_ext (1.6.0)
@@ -61,7 +60,6 @@ GEM
       fastthread (>= 1.0.1)
       rack
       rake (>= 0.8.1)
-    pg (0.13.2)
     posix-spawn (0.3.6)
     rack (1.1.3)
     rails (2.3.14)
@@ -72,16 +70,18 @@ GEM
       activesupport (= 2.3.14)
       rake (>= 0.8.3)
     rake (10.1.0)
-    redis (3.2.0)
-    redis-namespace (1.5.1)
-      redis (~> 3.0, >= 3.0.4)
+    redis (3.0.6)
+    redis-namespace (1.3.2)
+      redis (~> 3.0.4)
     rubystats (0.2.3)
     shoulda (3.0.1)
       shoulda-context (~> 1.0.0)
       shoulda-matchers (~> 1.0.0)
     shoulda-context (1.0.0)
     shoulda-matchers (1.0.0)
+    sqlite3 (1.3.8)
     syntax (1.0.0)
+    thor (0.18.1)
     timecop (0.3.5)
     webmock (1.8.0)
       addressable (>= 2.2.7)
@@ -94,7 +94,7 @@ PLATFORMS
 DEPENDENCIES
   RedCloth
   SystemTimer (= 1.2.3)
-  appraisal
+  appraisal (>= 1.0.0.beta2)
   backports
   bson_ext
   garb
@@ -104,12 +104,14 @@ DEPENDENCIES
   mongo
   mysql
   passenger (~> 2.0)
-  pg
   rack
   rails (~> 2.3.8)
   rake
+  redis (>= 2.1)
+  redis-namespace (>= 1.1.0)
   rubystats
   shoulda
+  sqlite3
   timecop
   vanity!
   webmock

data/README.rdoc

@@ -1,7 +1,7 @@
 = Vanity
 {<img src="https://travis-ci.org/assaf/vanity.png?branch=master" alt="Build Status" />}[https://travis-ci.org/assaf/vanity]
 
-Vanity is an Experiment Driven Development framework for Rails.
+Vanity is an A/B testing framework for Rails that is datastore agnostic.
 
 * All about Vanity: http://vanity.labnotes.org
 * On Github: http://github.com/assaf/vanity
@@ -9,12 +9,18 @@ Vanity is an Experiment Driven Development framework for Rails.
 http://farm3.static.flickr.com/2540/4099665871_497f274f68_o.jpg
 
 
-== A/B Testing With Rails (In 5 Easy Steps)
+== A/B Testing With Rails
 
 === <b>Step 1:</b> Start using Vanity in your Rails application
 
 ==== Step 1.1
 
+===== Rails 3 & Rails 4 installation
+
+Add to your Gemfile:
+
+  gem "vanity"
+
 ===== Rails 2.x installation
 
   Rails::Initializer.run do |config|
@@ -25,27 +31,46 @@ http://farm3.static.flickr.com/2540/4099665871_497f274f68_o.jpg
     end
   end
 
-===== Rails 3 & Rails 4 installation
+==== Step 1.2
+
+Choose a datastore that best fits your needs and preferences for storing experiment results. Choose one of: Redis, MongoDB or an SQL database. While Redis is usually faster, it may add additional complexity to your stack.
+
+===== Redis Setup
 
 Add to your Gemfile:
 
-  gem "vanity"
+  gem "redis", ">= 2.1"
+  gem "redis-namespace", ">= 1.1.0"
 
-==== Step 1.2
+===== MongoDB Setup
 
-Choose a method to store experiment results: for Redis or an external tracking mechanism, migrations aren't needed. Mongo is also available. To use a relational database, via ActiveRecord, run the generator and migrations to create the database schema:
+Add to your Gemfile:
+
+  gem "bson_ext"
+  gem "mongo"
+
+===== SQL Database Setup
+
+Vanity supports multiple SQL stores (like MySQL, MariaDB, Postgres, Sqlite, etc.) using ActiveRecord, which is built into Rails. If you're using DataMapper, Sequel or another persistence framework, add to your Gemfile:
+
+   gem "active_record"
+
+If you're going to store data in the database, run the generator and migrations to create the database schema:
 
   $ rails generate vanity
   $ rake db:migrate
 
 ==== Step 1.3
 
-add to your application controller:
+Turn Vanity on, and pass a reference to a method that identifies a user. For example:
 
   class ApplicationController < ActionController::Base
     use_vanity :current_user
+    layout false  # exclude this if you want to use your application layout
   end
 
+For more information, please see the {identity documentation}[http://vanity.labnotes.org/identity.html].
+
 === <b>Step 2:</b> Define your first A/B test
 
 This experiment goes in the file <code>experiments/price_options.rb</code>:
@@ -56,7 +81,11 @@ This experiment goes in the file <code>experiments/price_options.rb</code>:
     metrics :signups
   end
 
-NOTE: If using a metric as above ("signups"), there needs to be a corresponding ruby file for that metric. Inside the "experiments" directory create a "metrics" directory with a file called "signups.rb". The contents of the file can describe the signup metric, refer to the "Metrics" Vanity documentation page for an example.
+If the experiment uses a metric as above ("signups"), there needs to be a corresponding ruby file for that metric, <code>experiments/metrics/signups.rb</code>.
+
+  metric "Signup (Activation)" do
+    description "Measures how many people signed up for our awesome service."
+  end
 
 === <b>Step 3:</b> Present the different options to your users
 
@@ -88,26 +117,26 @@ To view metrics and experiment results with the dashboard in Rails 3 & Rails 4:
 
 In <code>config/routes.rb</code>, add:
 
-  match '/vanity(/:action(/:id(.:format)))', :controller => :vanity, :via => :get
+  match '/vanity(/:action(/:id(.:format)))', :controller => :vanity, :via => [:get, :post]
 
-Where the controller should looks like:
+The controller should look like:
 
   class VanityController < ApplicationController
     include Vanity::Rails::Dashboard
   end
 
-NOTE: By default, Vanity turns collection off in development mode. To turn it on, in <code>config/environments/development.rb</code> add:
-
-  Vanity.playground.collecting = true
-
 == Registering participants with Javascript
 
-If robots or spiders make up a significant portion of your sites traffic they can affect your conversion rate.  Vanity can optionally add participants to the experiments using asynchronous javascript callbacks, which will keep almost all robots out.  To set this up simply do the following:
+If robots or spiders make up a significant portion of your sites traffic they can affect your conversion rate. Vanity can optionally add participants to the experiments using asynchronous javascript callbacks, which will keep almost all robots out. To set this up simply do the following:
 
 * Add <code>Vanity.playground.use_js!</code>
-* Set <code>Vanity.playground.add_participant_path = '/path/to/vanity/action'</code>, this should point to the add_participant path that is added with Vanity::Rails::Dashboard, make sure that this action is available to all users
+* Set <code>Vanity.playground.add_participant_path = '/path/to/vanity/action'</code>, this should point to the add_participant path that is added with Vanity::Rails::Dashboard, make sure that this action is accessible by all users (ie does not require authentication).
 * Add <code><%= vanity_js %></code> to any page that needs uses an ab_test. <code>vanity_js</code> needs to be included after your call to ab_test so that it knows which version of the experiment the participant is a member of. The helper will render nothing if the there are no ab_tests running on the current page, so adding <code>vanity_js</code> to the bottom of your layouts is a good option. Keep in mind that if you call <code>use_js!</code> and don't include <code>vanity_js</code> in your view no participants will be recorded.
 
+=== <b>Update regarding using AJAX registration to avoid robots</b>
+
+As of January 2014, as reported in {issue 175}[https://github.com/assaf/vanity/issues/175], Googlebot is now executing AJAX requests to the original domain of the page, which includes the current implementation of the JS callback. This means that Googlebot-originating requests may be included in experiments, greatly skewing results. A fix is being worked on, please follow the issue for further updates.
+
 == Compatibility
 
 Here's what's tested and known to work:

data/Rakefile

@@ -1,5 +1,4 @@
 require "rake/testtask"
-require 'appraisal'
 
 # -- Building stuff --
 
@@ -75,10 +74,16 @@ ADAPTERS = %w{redis mongodb mysql}
 
 desc "Test using different back-ends"
 task "test:adapters", :adapter do |t, args|
-  adapters = args.adapter ? [args.adapter] : ADAPTERS
-  adapters.each do |adapter|
-    puts "** Testing #{adapter} adapter"
-    sh "rake appraisal test DB=#{adapter} #{'--trace' if Rake.application.options.trace}"
+  begin # Make sure we have appraisal installed and available
+    require "appraisal"
+
+    adapters = args.adapter ? [args.adapter] : ADAPTERS
+    adapters.each do |adapter|
+      puts "** Testing #{adapter} adapter"
+      sh "rake appraisal test DB=#{adapter} #{'--trace' if Rake.application.options.trace}"
+    end
+  rescue LoadError
+    warn "The appraisal gem must be available"
   end
 end
 
@@ -150,10 +155,10 @@ task :report do
   Vanity.playground.metrics.values.each(&:destroy!)
   Vanity.playground.reload!
 
-  # Control	182	35	19.23%	N/A
-  # Treatment A	180	45	25.00%	1.33
-  # Treatment B	189	28	14.81%	-1.13
-  # Treatment C	188	61	32.45%	2.94
+  # Control 182 35  19.23%  N/A
+  # Treatment A 180 45  25.00%  1.33
+  # Treatment B 189 28  14.81%  -1.13
+  # Treatment C 188 61  32.45%  2.94
   Vanity.playground.experiment(:null_abc).instance_eval do
     fake nil=>[182,35], :red=>[180,45], :green=>[189,28], :blue=>[188,61]
     @created_at = (Date.today - 40).to_time

data/doc/_layouts/page.html

@@ -16,11 +16,11 @@
     <div id="links">
       <a href="http://github.com/assaf/vanity">Source code</a> |
       <a href="http://groups.google.com/group/vanity-talk" title="vanity-talk google group">Google Group</a> |
-      <a href="http://vanity.labnotes.org/api/index.html">API reference</a>
+      <a href="http://rdoc.info/gems/vanity">API reference</a>
     </div>
     <div id="sidebar">
       <ul>
-        <li><a href="index.html#intro" title="A/B Testing with Rails in 5 easy steps">2 Minute Demo</a></li>
+        <li><a href="index.html#intro" title="A/B Testing with Rails">Intro</a></li>
         <li><a href="metrics.html">Metrics</a></li>
         <li><a href="ab_testing.html" title="Everything you need to know">A/B Testing</a></li>
         <li><a href="rails.html">Using with Rails</a></li>
@@ -29,11 +29,9 @@
         <li><a href="configuring.html">Configuring</a></li>
         <li><a href="contributing.html">Contributing</a></li>
         <li><a href="experimental.html">Experimental</a></li>
-        <li>Download: <a href="vanity-api.zip">API</a>/<a href="vanity.pdf">PDF</a></li>
-      </ul> 
+      </ul>
       <ul id="stats">
-        <li><a href="http://travis-ci.org/assaf/vanity"><img src="http://travis-ci.org/assaf/vanity.png"></a></li>
-        <li><a href="http://github.com/assaf/vanity/graphs/traffic">Traffic</a></li>
+        <li><a href="http://travis-ci.org/assaf/vanity"><img src="https://api.travis-ci.org/assaf/vanity.png?branch=master"></a></li>
         <li><a href="http://wiki.github.com/assaf/vanity/whos-using-vanity">Who's using it?</a></li>
       </ul>
     </div>

data/doc/ab_testing.textile

@@ -75,7 +75,7 @@ Vanity splits the audience randomly -- using "cookies and other mechanisms":iden
 
 Vanity will show the conversion rate for each alternative, and how that conversion compares to the worst performing alternative.  In the example above, option A has 80.6% conversion rate, 11% more than option B's 72.6% conversion rate (72.6 * 111% ~ 80.6%). 
 
-(These large numbers are easily explained by the fact that this report was generated form made up data)
+(These large numbers are easily explained by the fact that this report was generated from made up data)
 
 It takes only a handful of visits before you'll see one alternative clearly performing better than all others.  That's a sign that you should continue running the experiment.  You see, small sample size tend to give out random results.
 

data/doc/configuring.textile

@@ -11,6 +11,8 @@ Database connection information is loaded from @config/vanity.yml@, based on the
 development:
   adapter: redis
   connection: redis://localhost:6379/0
+test:
+  collecting: false
 production:
   adapter: mongodb
   database: analytics
@@ -24,10 +26,6 @@ The available database adapters are:
 * +mongodb+ -- Available options are host, port, database (defaults to "vanity"), username and password.
 * +active_record+ -- Uses existing ActiveRecord configuration, by you can over-ride by supplying different options. To pick different underlying adapter, set +active_record_adapter+.
 
-You want Vanity to collect information (metrics, experiments, etc) in production, but there's no point collecting data in other environments. You can turn data collection on and off by setting @Vanity.playground.collecting@. Under Rails, collection is turned off in all environments expect production.
-
-You may want to turn data collection on for integration tests, depending on what you're testing. Also, you may need to turn it on if your development server runs more than one process, e.g. if you're using Passenger for development and want to use the Vanity console to pick a particular alternative in an A/B test.
-
 Available configuration options are:
 
 |_. name      |_. Is all about ...                      |_. Default      |

data/doc/email.textile

@@ -48,7 +48,7 @@ end
 Create an ActionMailer class, for example:
 
 <pre>
-class UserMailer < ActionMailer::Base 
+class UserMailer < ActionMailer::Base
   def invite_email(user)
     use_vanity_mailer user
     mail :to => user.email, :subject =>ab_test(:invite_subject)
@@ -125,5 +125,3 @@ A/B test your email content:
 </pre>
 
 Here we use the text from the "invite_text" experiment and then use the @vanity_track_url_for@ helper to add the identity and the metric to track into the url so that Vanity can track the click-throughs.
-
-Remember: By default, Vanity only collects data in production mode.

data/doc/index.textile

@@ -10,7 +10,7 @@ title: Welcome to Vanity
 
 h3. Reading Order
 
-* "2 Minute Demo(A/B Testing with Rails in 5 easy steps)":#intro
+* "Introduction & Setup":#intro
 * "Metrics":metrics.html
 * "A/B Testing(Everything you need to know)":ab_testing.html
 * "Using with Rails":rails.html
@@ -24,68 +24,8 @@ Also:
 * "Join vanity-talk list":http://groups.google.com/group/vanity-talk
 * "Contributing to Vanity":contributing.html
 
+h3(#intro). Introduction & setup
 
-h3(#intro). A/B Testing with Rails (in 5 easy steps)
-
-*Step 1:* Start using Vanity in your Rails application:
-
-<pre>
-Rails::Initializer.run do |config|
-  gem.config "vanity"
-
-  config.after_initialize do
-    require "vanity"
-  end
-end
-</pre>
-
-And:
-
-<pre>
-class ApplicationController < ActionController::Base
-  use_vanity :current_user
-end
-</pre>
-
-*Step 2:* Define your first A/B test.  This experiment goes in the file <code>experiments/price_options.rb</code>:
-
-<pre>
-ab_test "Price options" do
-  description "Mirror, mirror on the wall, who's the better price of all?"
-  alternatives 19, 25, 29
-  metrics :signup
-end
-</pre>
-
-*Step 3:* Present the different options to your users:
-
-<pre>
-<h2>Get started for only $<%= ab_test :price_options %> a month!</h2>
-</pre>
-
-*Step 4:* Measure conversion:
-
-<pre>
-class SignupController < ApplicationController
-  def signup
-    @account = Account.new(params[:account])
-    if @account.save
-      track! :signup
-      redirect_to @acccount
-    else
-      render action: :offer
-    end
-  end
-end
-</pre>
-
-*Step 5:* Check the report:
-
-<pre>
-vanity report --output vanity.html
-</pre>
-
-!images/price_options.png!
+Please see the "setup instructions in the README":https://github.com/assaf/vanity#vanity
 
 Read more about "A/B Testing ...":ab_testing.html
-