moses-vanity 1.7.1

data/MIT-LICENSE

@@ -0,0 +1,21 @@
+Copyright (c) 2010 Assaf Arkin
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+

data/README.rdoc

@@ -0,0 +1,108 @@
+Vanity is an Experiment Driven Development framework for Rails.
+
+* All about Vanity: http://vanity.labnotes.org
+* On github: http://github.com/assaf/vanity
+
+http://farm3.static.flickr.com/2540/4099665871_497f274f68_o.jpg
+
+
+== A/B Testing With Rails (In 5 Easy Steps)
+
+<b>Step 1:</b> Start using Vanity in your Rails application:
+
+== Rails 2.x configuration
+
+  Rails::Initializer.run do |config|
+    gem.config "vanity"
+
+    config.after_initialize do
+      require "vanity"
+    end
+  end
+  
+== Rails 3 configuration
+
+Add to your Gemfile:
+    
+  gem "vanity"
+
+If using a relational database, run the generator and migrations to create the database schema:
+
+  $ rails generate vanity
+  $ rake db:migrate
+  
+Add to your application controller:
+
+  class ApplicationController < ActionController::Base
+    use_vanity :current_user
+  end
+
+<b>Step 2:</b> Define your first A/B test.  This experiment goes in the file <code>experiments/price_options.rb</code>:
+
+  ab_test "Price options" do
+    description "Mirror, mirror on the wall, who's the better price of all?"
+    alternatives 19, 25, 29
+    metrics :signups
+  end
+  
+  NOTE: If using a metric as above ("signups"), there needs to be a corresponding ruby file for that metric. Inside the "experiments" directory create a "metrics" directory with a file called "signup.rb". The contents of the file can describe the signup metric, refer to the "Metrics" Vanity documentation page for an example.
+
+<b>Step 3:</b> Present the different options to your users:
+
+  <h2>Get started for only $<%= ab_test :price_options %> a month!</h2>
+
+<b>Step 4:</b> Measure conversion:
+
+  class SignupController < ApplicationController
+    def signup
+      @account = Account.new(params[:account])
+      if @account.save
+        track! :signups
+        redirect_to @acccount
+      else
+        render action: :offer
+      end
+    end
+  end
+
+<b>Step 5:</b> Check the report:
+
+  vanity report --output vanity.html
+  
+
+== Rails 3
+  
+  There is currently an issue with report generation. The vanity-talk Google Group has a couple posts that outline the issue for now. This is one of the posts: http://groups.google.com/group/vanity-talk/browse_thread/thread/343081a72a0cefb6
+  
+  If you are collecting data (in development you need to opt-in to this by setting Vanity.playground.collecting = true in environments/development.rb) you can view experiment results with the vanity dashboard instead of the report. 
+  
+  The vanity dashboard setup instructions with Vanity work for Rails 3.x except the route is different. A Rails 3.x-style route would look like this:
+  
+    `match '/vanity(/:action(/:id(.:format)))', :controller=>:vanity`
+
+
+== Registering participants with Javascript
+
+If robots or spiders make up a significant portion of your sites traffic they can affect your conversion rate.  Vanity can optionally add participants to the experiments using asynchronous javascript callbacks, which will keep almost all robots out.  To set this up simply do the following:
+
+* Add Vanity.playground.use_js!
+* Set Vanity.playground.add_participant_path = '/path/to/vanity/action', this should point to the add_participant path that is added with Vanity::Rails::Dashboard, make sure that this action is available to all users
+* Add <%= vanity_js %> to any page that needs uses an ab_test. vanity_js needs to be included after your call to ab_test so that it knows which version of the experiment the participant is a member of.  The helper will render nothing if the there are no ab_tests running on the current page, so adding vanity_js to the bottom of your layouts is a good option.  Keep in mind that if you call use_js! and don't include vanity_js in your view no participants will be recorded.
+
+== Contributing
+
+* Fork the project
+* Please use a topic branch to make your changes, it's easier to test them that way
+* Fix, patch, enhance, document, improve, sprinkle pixie dust
+* At minimum run rake test, if possible, please run rake test:all
+* Tests. Please. Run rake test, of if you can, rake test:all
+* Send a pull request on GitHub
+
+
+== Credits/License
+
+Original code, copyright of Assaf Arkin, released under the MIT license.  
+
+Documentation available under the Creative Commons Attribution license.
+
+For full list of credits and licenses: http://vanity.labnotes.org/credits.html.

data/Rakefile

@@ -0,0 +1,189 @@
+require "rake/testtask"
+
+# -- Building stuff --
+
+spec = Gem::Specification.load(Dir["*.gemspec"].first)
+
+desc "Build the Gem"
+task :build do
+  sh "gem build #{spec.name}.gemspec"
+end
+
+desc "Install #{spec.name} locally"
+task :install=>:build do
+  sudo = "sudo" unless File.writable?( Gem::ConfigMap[:bindir])
+  sh "#{sudo} gem install #{spec.name}-#{spec.version}.gem"
+end
+
+desc "Push new release to gemcutter and git tag"
+task :push=>["test:all", "build"] do
+  sh "git push"
+  puts "Tagging version #{spec.version} .."
+  sh "git tag v#{spec.version}"
+  sh "git push --tag"
+  puts "Building and pushing gem .."
+  sh "gem push #{spec.name}-#{spec.version}.gem"
+end
+
+
+# -- Testing stuff --
+
+desc "Test everything"
+task "test:all"=>"test:adapters"
+
+# Ruby versions we're testing with.
+RUBIES = %w{1.8.7 1.9.2}
+
+# Use rake test:rubies to run all combination of tests (see test:adapters) using
+# all the versions of Ruby specified in RUBIES. Or to test a specific version of
+# Ruby, rake test:rubies[1.8.7].
+#
+# This task uses RVM to install all the Ruby versions it needs, and creates a
+# vanity gemset in each one that includes Bundler and all the gems specified in
+# Gemfile. If anything goes south you can always wipe these gemsets or uninstall
+# these Rubies and start over.
+desc "Test using multiple versions of Ruby"
+task "test:rubies", :ruby do |t, args|
+  rubies = args.ruby ? [args.ruby] : RUBIES
+  rubies.each do |ruby|
+    puts "** Setup #{ruby}"
+    sh "env rvm_install_on_use_flag=1 rvm_gemset_create_on_use_flag=1 rvm use #{ruby}@vanity"
+    sh "rvm #{ruby}@vanity rake test:setup"
+    puts
+    puts "** Test using #{ruby}"
+    sh "rvm #{ruby}@vanity -S bundle exec rake test:adapters #{'--trace' if Rake.application.options.trace}"
+  end
+end
+
+task "test:setup" do
+  # Intended to be used from test:rubies, within specific RVM context.
+  begin # Make sure we got Bundler installed.
+    require "bundler"
+  rescue LoadError
+    sh "gem install bundler"
+  end
+  begin # Make sure we got all the dependencies
+    sh "bundle exec ruby -e puts > /dev/null"
+  rescue
+    sh "bundle install"
+  end
+end
+
+# These are all the adapters we're going to test with.
+ADAPTERS = %w{redis mongodb mysql}
+
+desc "Test using different back-ends"
+task "test:adapters", :adapter do |t, args|
+  adapters = args.adapter ? [args.adapter] : ADAPTERS
+  adapters.each do |adapter|
+    puts "** Testing #{adapter} adapter"
+    sh "rake test DB=#{adapter} #{'--trace' if Rake.application.options.trace}"
+  end
+end
+
+# Run the test suit.
+
+task :default=>:test
+desc "Run all tests"
+Rake::TestTask.new do |task|
+  task.test_files = FileList['test/**/*_test.rb']
+  if Rake.application.options.trace
+    #task.warning = true
+    task.verbose = true
+  elsif Rake.application.options.silent
+    task.ruby_opts << "-W0"
+  else
+    task.verbose = true
+  end
+    task.ruby_opts << "-I."
+end
+
+task(:clobber) { rm_rf "tmp" }
+
+
+# -- Documenting stuff --
+
+begin
+  require "yard"
+  YARD::Rake::YardocTask.new(:yardoc) do |task|
+    task.files  = FileList["lib/**/*.rb"].exclude("lib/vanity/backport.rb")
+    task.options = "--output", "html/api", "--title", "Vanity #{spec.version}", "--main", "README.rdoc", "--files", "CHANGELOG"
+  end
+rescue LoadError
+end
+
+desc "Jekyll generates the main documentation (sans API)"
+task(:jekyll) { sh "jekyll", "doc", "html" }
+file "html/vanity.pdf"=>:jekyll do |t|
+  pages = %w{index metrics ab_testing rails email identity configuring contributing}.map{ |p| "html/#{p}.html" }
+  args = %w{--disable-javascript --outline --title Vanity --header-html doc/_layouts/_header.html --print-media-type}
+  args.concat %w{--margin-left 20 --margin-right 20 --margin-top 20 --margin-bottom 20 --header-spacing 5}
+  args.concat pages << t.name
+  sh "wkhtmltopdf", *args
+end
+
+file "html/vanity-api.zip"=>:yardoc do |t|
+  Dir.chdir "html" do
+    sh "zip vanity-api.zip -r api"
+  end
+end
+desc "Create documentation in docs directory (including API)"
+task :docs=>[:jekyll, :yardoc, "html/vanity-api.zip", "html/vanity.pdf"]
+desc "Remove temporary files and directories"
+task(:clobber) { rm_rf "html" ; rm_rf ".yardoc" }
+
+desc "Publish documentation to vanity.labnotes.org"
+task :publish=>[:clobber, :docs] do
+  sh "rsync -cr --del --progress html/ labnotes.org:/var/www/vanity/"
+end
+
+
+# -- Misc --
+
+task :report do
+  $LOAD_PATH.unshift "lib"
+  require "vanity"
+  require "timecop"
+  Vanity.playground.load_path = "test/experiments"
+  Vanity.playground.experiments.values.each(&:destroy)
+  Vanity.playground.metrics.values.each(&:destroy!)
+  Vanity.playground.reload!
+
+  # Control	182	35	19.23%	N/A
+  # Treatment A	180	45	25.00%	1.33
+  # Treatment B	189	28	14.81%	-1.13
+  # Treatment C	188	61	32.45%	2.94
+  Vanity.playground.experiment(:null_abc).instance_eval do
+    fake nil=>[182,35], :red=>[180,45], :green=>[189,28], :blue=>[188,61]
+    @created_at = (Date.today - 40).to_time
+    @completed_at = (Date.today - 35).to_time
+  end
+
+  Vanity.playground.experiment(:age_and_zipcode).instance_eval do
+    fake false=>[80,35], true=>[84,32]
+    @created_at = (Date.today - 30).to_time
+    @completed_at = (Date.today - 15).to_time
+  end
+
+  Vanity.context = Object.new
+  Vanity.context.instance_eval { def vanity_identity ; 0 ; end }
+  signups = 50
+  (Date.today - 90..Date.today).each do |date|
+    Timecop.travel date do
+      signups += rand(15) - 5
+      Vanity.playground.track! :signups, signups
+    end
+  end
+
+  cheers, yawns = 0, 0
+  (Date.today - 80..Date.today).each do |date|
+    Timecop.travel date do
+      cheers = cheers - 5 + rand(20)
+      Vanity.playground.track! :yawns, cheers
+      yawns = yawns - 5 + rand(30)
+      Vanity.playground.track! :cheers, yawns
+    end
+  end
+
+  Vanity::Commands.report ENV["OUTPUT"]
+end

data/bin/vanity

@@ -0,0 +1,16 @@
+#!/usr/bin/env ruby
+#
+# This file was generated by Bundler.
+#
+# The application 'vanity' is installed as part of a gem, and
+# this file is here to facilitate running it.
+#
+
+require 'pathname'
+ENV['BUNDLE_GEMFILE'] ||= File.expand_path("../../Gemfile",
+  Pathname.new(__FILE__).realpath)
+
+require 'rubygems'
+require 'bundler/setup'
+
+load Gem.bin_path('vanity', 'vanity')

data/doc/_config.yml

@@ -0,0 +1,2 @@
+# http://wiki.github.com/mojombo/jekyll/configuration
+pygments: false

data/doc/_layouts/_header.html

@@ -0,0 +1,34 @@
+<html>
+  <head>
+    <script>
+      function subst() {
+        var vars={};
+        var x=document.location.search.substring(1).split('&');
+        for(var i in x) {var z=x[i].split('=',2);vars[z[0]] = unescape(z[1]);}
+        var x=['frompage','topage','page','webpage','section','subsection','subsubsection'];
+        for(var i in x) {
+          var y = document.getElementsByClassName(x[i]);
+          for(var j=0; j<y.length; ++j) y[j].textContent = vars[x[i]];
+        }
+      }
+    </script>
+    <style>
+      body { margin: 0; padding: 0; height: 2em }
+      .header {
+        font-size: 10pt;
+        margin: 0;
+        padding: 0;
+        border-bottom: 1px solid #ccc;
+        width: 100%;
+      }
+    </style>
+  </head>
+  <body onload="subst()">
+    <table class="header">
+      <tr>
+        <td class="section"></td>
+        <td style="text-align:right"><span class="page"></span></td>
+      </tr>
+    </table>
+  </body>
+</html>

data/doc/_layouts/page.html

@@ -0,0 +1,47 @@
+<!DOCTYPE html>
+<html>
+  <head>
+    <title>Vanity &mdash; {{ page.title }}</title>
+    <link href="css/page.css" media="screen,print" rel="stylesheet" type="text/css">
+    <link href="css/print.css" media="print" rel="stylesheet" type="text/css">
+    <link href="images/favicon.png" rel="shortcut icon">
+    <script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js"></script>
+    <script type="text/javascript" src="site.js"></script>
+  </head>
+  <body>
+    <div id="header">
+      <div class="title"><a href="http://vanity.labnotes.org" title="Mirror, mirror, on the wall &hellip;">Vanity</a></div>
+      <div class="tagline">Experiment<br>Driven Development</div>
+    </div>
+    <div id="links">
+      <a href="http://github.com/assaf/vanity">Source code</a> |
+      <a href="http://groups.google.com/group/vanity-talk" title="vanity-talk google group">Google Group</a> |
+      <a href="http://vanity.labnotes.org/api/index.html">API reference</a>
+    </div>
+    <div id="sidebar">
+      <ul>
+        <li><a href="index.html#intro" title="A/B Testing with Rails in 5 easy steps">2 Minute Demo</a></li>
+        <li><a href="metrics.html">Metrics</a></li>
+        <li><a href="ab_testing.html" title="Everything you need to know">A/B Testing</a></li>
+        <li><a href="rails.html">Using with Rails</a></li>
+        <li><a href="email.html">Testing emails</a></li>
+        <li><a href="identity.html">Managing Identity</a></li>
+        <li><a href="configuring.html">Configuring</a></li>
+        <li><a href="contributing.html">Contributing</a></li>
+        <li><a href="experimental.html">Experimental</a></li>
+        <li>Download: <a href="vanity-api.zip">API</a>/<a href="vanity.pdf">PDF</a></li>
+      </ul> 
+      <ul id="stats">
+        <li><a href="http://travis-ci.org/assaf/vanity"><img src="http://travis-ci.org/assaf/vanity.png"></a></li>
+        <li><a href="http://github.com/assaf/vanity/graphs/traffic">Traffic</a></li>
+        <li><a href="http://wiki.github.com/assaf/vanity/whos-using-vanity">Who's using it?</a></li>
+      </ul>
+    </div>
+    <div id="content">
+      <h1 id="{{ page.title | downcase | replace(' ', '_') }}">{{ page.title }}</h1>
+      {{ content }}
+    </div>
+    <div id="footer"><a href="credits.html">Credits / License</a></div>
+    <script type="text/javascript">var _gaq=_gaq||[];_gaq.push(["_setAccount", "UA-1828623-6"], ["_trackPageview"]);(function(){var ga=document.createElement("script");ga.src=("https:"==document.location.protocol?"https://ssl":"http://www")+".google-analytics.com/ga.js";ga.setAttribute("async", "true");document.documentElement.firstChild.appendChild(ga);})();</script>
+  </body>
+</html>

data/doc/_metrics.textile

@@ -0,0 +1,12 @@
+Metrics and experiments both measure, the main difference is scope.  Typically you would use metrics for long-term measurements of general indicators you are interested in.  For example, metrics for number of sign ups, comments submitted, videos uploaded, etc.  You would then run short-term experiments to improve on these metrics.  For example, experiment with different landing page to see how it affect sign ups.  You may run several experiments in attempt to improve the same metric, an experiment may also measure against multiple metrics (e.g. it may increase uploaded videos, but decrease comments).  A successful experiment is one that teaches what to do in order to improve a metric.
+
+Start collecting metrics early.  The longer you collect them, the better you'll be able to spot trends.
+
+You can collect different kinds of metrics.  Web applications focus on visits, but you may want to look deeper into the application and collect metrics that gauge engagement and growth from within your user base.  Use metrics to spot usage patterns and see which direction you should take your application in.  Metrics could point you in the direction of features that get all the usage, and are the basis for your application.  They'll also tell you which features are dead-weight so you can trim the fat.
+
+Watching metrics is fun, but not where the action is.  The action is making changes that improve these metrics.  Once you get into the practice of changing code and measuring impact you will notice that your intuition is often right, but not always.  People will surprise you.  You'll realize that, rather than brainstorming five headlines for a page and using one, you can use all five and pick the one that works best.  You'll notice that more options you try out, the faster and further you can improve your metrics.  You'll want to make your experiments lightweight, quick to setup, measure and teardown (or officiate into code).  Welcome to EDD.
+
+Metrics collect historical data which you can use to predict future behavior.  Classical example is looking at growth patterns to determine when it's time for an infrastructure upgrade and how large will it be.  You can also use historical data to predict probable range for future behavior (bounds), and issue alerts when system gets out of these bounds.  For example, if you get 50~75 registrations a day, and all of a sudden than number drops to zero, you might have a problem with the registration system.  You'll want an alert for that.  If it shoots into 200, you might have an awesome PR opportunity going on.
+
+Metrics are not just for business people.  They can help you improve usability by measuring -- rather than guessing -- which actions have a positive affect.  Was it speeding up database queries or JavaScript code?  The new edit-in-place feature?  IE6 support?  Just kidding.  Metrics can help you find broken code that slipped past the test suite, although it won't tell you where the bug is, you'll notice when a broken feature gets no usage.   Metrics can also help you with content improvement (micro-copy).
+

data/doc/ab_testing.textile

@@ -0,0 +1,210 @@
+---
+layout: page
+title: A/B Testing
+---
+
+<div id="toc">
+# "True or False":#tf
+# "Interpreting the Results":#interpret
+# "Multiple Alternatives":#multiple
+# "A/B Testing and Code Testing":#test
+# "Let the Experiment Decide":#decide
+</div>
+
+
+"A/B testing":http://en.wikipedia.org/wiki/A/B_testing (or "split testing") are experiments you can run to compare the performance of different alternatives.  A classical example is using an A/B test to compare two versions of a landing page, to find out which alternative leads to more registrations.
+
+You can use A/B tests to gauge interest in a new feature, response to a feature change, improve the site's design and copy, and so forth.  In spite of the name, you can use A/B tests to check out more than two alternatives.
+
+bq. "If you are not embarrassed by the first version of your product, you’ve launched too late" -- Reid Hoffman, founder of LinkedIn
+
+
+h3(#tf). True or False
+
+Let's start with a simple experiment.  We have this idea that a bigger sign-up link will increase the number of people who sign up for our service.  Let's see how well our hypothesis holds.
+
+We already have a "metric":metrics.html we're monitoring, and our experiment will measure against it:
+
+<pre>
+ab_test "Big singup link" do
+  description "Testing to see if a bigger sign-up link increases number of signups."
+  metrics :signup
+end
+</pre>
+
+Next, we're going to show some of our visitors a bigger sign-up link:
+
+<pre>
+<% bigger = "font:14pt;font-weight:bold" if ab_test(:big_signup_link) %>
+<%= link_to "Sign up", signup_url, style: bigger %>
+</pre>
+
+Approximately half the visitors to our site will see this link:
+
+<notextile>
+<a href="#">Sign up</a>
+</notextile>
+
+The other half will see this one:
+
+<notextile>
+<a href="#" style="font:14pt;font-weight:bold">Sign up</a>
+</notextile>
+
+An A/B test has two parts, we just covered the part which decides which alternative to show.  The second part measures the effectiveness of each alternative.  This happens as result of measuring the metric.
+
+Remember that we're measuring signups, so we already have this in the code:
+
+<pre>
+class SignupController < ApplicationController
+  def signup
+    Account.create(params[:account])
+    track! :signup
+  end
+end
+</pre>
+
+
+h3(#interpret). Interpreting the Results
+
+We're going to let the experiment run for a while and track the results using "the dashboard":rails.html#dashboard, or by running the command @vanity report@.
+
+Vanity splits the audience randomly -- using "cookies and other mechanisms":identity.html -- and records who got to see each alternative, and how many in each group converted (in our case, signed up).  Dividing conversions by participants gives you the conversion rate.
+
+!images/clear_winner.png!
+
+Vanity will show the conversion rate for each alternative, and how that conversion compares to the worst performing alternative.  In the example above, option A has 80.6% conversion rate, 11% more than option B's 72.6% conversion rate (72.6 * 111% ~ 80.6%). 
+
+(These large numbers are easily explained by the fact that this report was generated form made up data)
+
+It takes only a handful of visits before you'll see one alternative clearly performing better than all others.  That's a sign that you should continue running the experiment.  You see, small sample size tend to give out random results.
+
+To get actionable results, you want a large enough sample, more specifically, you want to look at the probability.  Vanity picks the top two alternatives and "calculates a z-score":http://20bits.com/articles/statistical-analysis-and-ab-testing/ to determine the probability that the best alternative performed better than second best.  It presents that probability which should tell you when is a good time to wrap up the experiment.
+
+This is the part that gets most people confused about A/B testing.  Let's say we ran an experiment with two alternatives and we notice that option B performs 50% better than option A (A * 150% = B).  We calculate from the z-score a 90% probability.
+
+"With 90% probability" does not mean 90% of the difference (50%), it does not mean B performs 45% better than A (90% * 50% = 45%).  In fact, it doesn't tell us how well B performs relative to A.  Option B may perform exceptionally well during the experiment, not so well later on.
+
+The only thing "with 90% probability" tells us is the probability that option B is somewhat better than option A.  And that means 10% probability that the results we're seeing are totally random and mean nothing in the long run.  In other words: 9 out of 10 times, B is indeed better than A.
+
+If you run the test longer to collect a larger sample size you'll see the probability increase to 95%, then 99% and finally 99.9%.  That's big confidence in the outcome of the experiment, but it might take a long time to get there.
+
+You might want to instead decide on some target probability (which could change from one experiment to another).  For example, if you pick 95% as the target, you're going to act on the wrong conclusion 1 out of 20 times, but you're going to finish your experiments faster, which means you'll get to iterate quickly and more often.  Fast iterations are one way to improve the quality of your software.
+
+You'll want to read more about "A/B testing and statistical significance":http://www.cennydd.co.uk/2009/statistical-significance-other-ab-test-pitfalls/
+
+
+h3(#multiple).  Multiple Alternatives
+
+Your A/B tests can have as many alternatives as you care, with two caveats.  The more alternatives you have the larger the sample size you need, and so the longer it will take to find out the outcome of your experiment.  You want alternatives that are significantly different from each other, testing two pricing options at $5 and $25 is fast, testing all the prices between $5 and $25 at $1 increments will take a long time to reach any conclusive result.
+
+The second caveat is that right now Vanity only scores the two best performing alternatives.  This may be an issue in some experiments, it may also be fixed in a future release.
+
+To define an experiment with multiple alternatives:
+
+<pre>
+ab_test "Price options" do
+  description "Mirror, mirror on the wall, who's the better price of them all?"
+  alternatives 5, 15, 25
+  metrics :signup
+end
+</pre>
+
+The @ab_test@ method returns the value of one of the chosen alternatives, so in your views you can write:
+
+<pre>
+<h2>Get started for only $<%= ab_test :price_options %> a month!</h2>
+</pre>
+
+!images/price_options.png!
+
+If you don't given any values, Vanity will run your experiment with the values false and true.  Here are other examples for rendering A/B tests with multiple values:
+
+<pre>
+def index
+  # alternatives are names of templates 
+  render template: ab_test(:new_page)
+end
+</pre>
+   
+<pre>
+<%= ab_test(:greeting) %> <%= current_user.name %>
+</pre>
+   
+<pre>
+<% ab_test :features do |count| %>
+  <%= count %> features to choose from!
+<% end %>
+</pre>
+
+
+h3(#test).  A/B Testing and Code Testing
+
+If you're presenting more than one alternative to visitors of your site, you'll want to test more than one alternative.  Don't let A/B testing become A/broken.
+
+You can force a functional/integration test to choose a given alternative:
+
+<pre>
+def test_big_signup_link
+  experiment(:big_signup_link).chooses(true)
+  get :index
+  assert_select "a[href=/signup][style^=font:14pt]", "Sign up"
+end
+</pre>
+
+Here's another example using Webrat:
+
+<pre>
+def test_price_option
+  [19, 25, 29].each do |price| 
+    experiment(:price_options).chooses(price)
+    visit root_path
+    assert_contain "Get started for only $#{price} a month!"
+  end
+end
+</pre>
+
+You'll also want to test each alternative visually, from your Web browser.  For that you'll have to install the "the Dashboard":rails.html#dashboard, which lets you pick which alternative is shown to you:
+
+!images/ab_in_dashboard.png!
+
+Once the experiment is over, simply remove its definition from the experiments directory and run the test suite again.  You'll see errors in all the places that touch the experiment (from failing to load it), pointing you to what parts of the code you need to remove/change.
+
+
+h3(#decide). Let the Experiment Decide
+
+Sample size and probability help you interpret the results, you can also use them to configure an experiment to automatically complete itself.
+
+This experiment will conclude once it has 1000 participants for each alternative, or a leading alternative with probability of 95% or higher:
+
+<pre>
+ab_test "Self completed" do
+  description "This experiment will self-complete."
+  metrics :coolness
+  complete_if do
+    alternatives.all? { |alt| alt.participants >= 1000 } ||
+      (score.choice && score.choice.probability >= 95)
+  end
+end
+</pre>
+
+When it reaches its end, the experiment will stop recording conversions, chose one of its alternatives as the outcome and switch every usage of @ab_test@ to that alternative.
+
+By default Vanity will choose the alternative with the highest conversion rate.  This is most often, but not always, the best outcome.  Imagine an experiment where option B results in less conversions, but higher quality conversion that option A.  Perhaps you're interested in option B conversions if you're losing no more than 20% compared to option A.  Here's a way to write that outcome:
+
+<pre>
+ab_test "Self completed" do
+  description "This experiment will self-complete."
+  metrics :coolness
+
+  complete_if do
+    score(95).choice # only return choice with probability >= 95
+  end
+
+  outcome_is do
+    a, b = alternatives
+    b.conversion_rate >= 0.8 * a.conversion_rate ? b : a
+  end
+end
+</pre>
+