split 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/.gitignore +4 -0
- data/Gemfile +3 -0
- data/README.mdown +177 -0
- data/Rakefile +8 -0
- data/lib/split.rb +50 -0
- data/lib/split/alternative.rb +77 -0
- data/lib/split/dashboard.rb +35 -0
- data/lib/split/dashboard/public/reset.css +48 -0
- data/lib/split/dashboard/public/style.css +34 -0
- data/lib/split/dashboard/views/index.erb +56 -0
- data/lib/split/dashboard/views/layout.erb +22 -0
- data/lib/split/experiment.rb +59 -0
- data/lib/split/helper.rb +28 -0
- data/lib/split/version.rb +3 -0
- data/spec/experiment_spec.rb +67 -0
- data/spec/helper_spec.rb +86 -0
- data/spec/spec_helper.rb +7 -0
- data/split.gemspec +27 -0
- metadata +141 -0
data/.gitignore
ADDED
data/Gemfile
ADDED
data/README.mdown
ADDED
@@ -0,0 +1,177 @@
|
|
1
|
+
# Split
|
2
|
+
## Rack based split testing framework
|
3
|
+
|
4
|
+
Split is a rack based ab testing framework designed to work with Rails, Sinatra or any other rack based app.
|
5
|
+
|
6
|
+
Split is heavily inspired by the Abingo and Vanity rails ab testing plugins and Resque in its use of Redis.
|
7
|
+
|
8
|
+
## Requirements
|
9
|
+
|
10
|
+
Split uses redis as a datastore.
|
11
|
+
|
12
|
+
If you're on OS X, Homebrew is the simplest way to install Redis:
|
13
|
+
|
14
|
+
$ brew install redis
|
15
|
+
$ redis-server /usr/local/etc/redis.conf
|
16
|
+
|
17
|
+
You now have a Redis daemon running on 6379.
|
18
|
+
|
19
|
+
## Setup
|
20
|
+
|
21
|
+
If you are using bundler add split to your Gemfile:
|
22
|
+
|
23
|
+
gem 'split'
|
24
|
+
|
25
|
+
Then run:
|
26
|
+
|
27
|
+
bundle install
|
28
|
+
|
29
|
+
Otherwise install the gem:
|
30
|
+
|
31
|
+
gem install split
|
32
|
+
|
33
|
+
and require it in your project:
|
34
|
+
|
35
|
+
require 'split'
|
36
|
+
|
37
|
+
### Rails
|
38
|
+
|
39
|
+
Split is autoloaded when rails starts up, as long as you've configured redis it will 'just work'.
|
40
|
+
|
41
|
+
### Sinatra
|
42
|
+
|
43
|
+
To configure sinatra with Split you need to enable sessions and mix in the helper methods. Add the following lines at the top of your sinatra app:
|
44
|
+
|
45
|
+
class MySinatraApp < Sinatra::Base
|
46
|
+
enable :sessions
|
47
|
+
helpers Split::Helper
|
48
|
+
|
49
|
+
get '/' do
|
50
|
+
...
|
51
|
+
end
|
52
|
+
|
53
|
+
## Usage
|
54
|
+
|
55
|
+
To begin your ab test use the `ab_test` method, naming your experiment with the first argument and then the different variants which you wish to test on as the other arguments.
|
56
|
+
|
57
|
+
`ab_test` returns one of the alternatives, if a user has already seen that test they will get the same alternative as before, which you can use to split your code on.
|
58
|
+
|
59
|
+
It can be used to render different templates, show different text or any other case based logic.
|
60
|
+
|
61
|
+
`finished` is used to make a completion of an experiment, or conversion.
|
62
|
+
|
63
|
+
Example: View
|
64
|
+
|
65
|
+
<% ab_test("login_button", "/images/button1.jpg", "/images/button2.jpg") do |button_file| %>
|
66
|
+
<%= img_tag(button_file, :alt => "Login!") %>
|
67
|
+
<% end %>
|
68
|
+
|
69
|
+
Example: Controller
|
70
|
+
|
71
|
+
def register_new_user
|
72
|
+
#See what level of free points maximizes users' decision to buy replacement points.
|
73
|
+
@starter_points = ab_test("new_user_free_points", 100, 200, 300)
|
74
|
+
end
|
75
|
+
|
76
|
+
Example: Conversion tracking (in a controller!)
|
77
|
+
|
78
|
+
def buy_new_points
|
79
|
+
#some business logic
|
80
|
+
finished("buy_new_points") #Either a conversion named with :conversion or a test name.
|
81
|
+
end
|
82
|
+
|
83
|
+
Example: Conversion tracking (in a view)
|
84
|
+
|
85
|
+
Thanks for signing up, dude! <% finished("signup_page_redesign") >
|
86
|
+
|
87
|
+
## Web Interface
|
88
|
+
|
89
|
+
Split comes with a Sinatra-based front end to get an overview of how your experiments are doing.
|
90
|
+
|
91
|
+
You can mount this inside your app using Rack::URLMap in your `config.ru`
|
92
|
+
|
93
|
+
require 'split/dashboard'
|
94
|
+
|
95
|
+
run Rack::URLMap.new \
|
96
|
+
"/" => Your::App.new,
|
97
|
+
"/split" => Split::Dashboard.new
|
98
|
+
|
99
|
+
You may want to password protect that page, you can do so with `Rack::Auth::Basic`
|
100
|
+
|
101
|
+
Split::Dashboard.use Rack::Auth::Basic do |username, password|
|
102
|
+
username == 'admin' && password == 'p4s5w0rd'
|
103
|
+
end
|
104
|
+
|
105
|
+
## Configuration
|
106
|
+
|
107
|
+
You may want to change the Redis host and port Split connects to, or
|
108
|
+
set various other options at startup.
|
109
|
+
|
110
|
+
Split has a `redis` setter which can be given a string or a Redis
|
111
|
+
object. This means if you're already using Redis in your app, Split
|
112
|
+
can re-use the existing connection.
|
113
|
+
|
114
|
+
String: `Split.redis = 'localhost:6379'`
|
115
|
+
|
116
|
+
Redis: `Split.redis = $redis`
|
117
|
+
|
118
|
+
For our rails app we have a `config/initializers/split.rb` file where
|
119
|
+
we load `config/split.yml` by hand and set the Redis information
|
120
|
+
appropriately.
|
121
|
+
|
122
|
+
Here's our `config/split.yml`:
|
123
|
+
|
124
|
+
development: localhost:6379
|
125
|
+
test: localhost:6379
|
126
|
+
staging: redis1.example.com:6379
|
127
|
+
fi: localhost:6379
|
128
|
+
production: redis1.example.com:6379
|
129
|
+
|
130
|
+
And our initializer:
|
131
|
+
|
132
|
+
rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
|
133
|
+
rails_env = ENV['RAILS_ENV'] || 'development'
|
134
|
+
|
135
|
+
split_config = YAML.load_file(rails_root + '/config/split.yml')
|
136
|
+
Split.redis = split_config[rails_env]
|
137
|
+
|
138
|
+
## Namespaces
|
139
|
+
|
140
|
+
If you're running multiple, separate instances of Split you may want
|
141
|
+
to namespace the keyspaces so they do not overlap. This is not unlike
|
142
|
+
the approach taken by many memcached clients.
|
143
|
+
|
144
|
+
This feature is provided by the [redis-namespace][rs] library, which
|
145
|
+
Split uses by default to separate the keys it manages from other keys
|
146
|
+
in your Redis server.
|
147
|
+
|
148
|
+
Simply use the `Split.redis.namespace` accessor:
|
149
|
+
|
150
|
+
Split.redis.namespace = "resque:GitHub"
|
151
|
+
|
152
|
+
We recommend sticking this in your initializer somewhere after Redis
|
153
|
+
is configured.
|
154
|
+
|
155
|
+
## Contributors
|
156
|
+
|
157
|
+
Special thanks to the following people for submitting patches:
|
158
|
+
|
159
|
+
* Lloyd Pick
|
160
|
+
* Jeffery Chupp
|
161
|
+
|
162
|
+
## Note on Patches/Pull Requests
|
163
|
+
|
164
|
+
* Fork the project.
|
165
|
+
* Make your feature addition or bug fix.
|
166
|
+
* Add tests for it. This is important so I don't break it in a
|
167
|
+
future version unintentionally.
|
168
|
+
* Commit, do not mess with rakefile, version, or history.
|
169
|
+
(if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
|
170
|
+
* Send me a pull request. Bonus points for topic branches.
|
171
|
+
|
172
|
+
## Copyright
|
173
|
+
|
174
|
+
Copyright (c) 2011 Andrew Nesbitt. See LICENSE for details.
|
175
|
+
|
176
|
+
|
177
|
+
n.b don't pass the same alternative twice!
|
data/Rakefile
ADDED
data/lib/split.rb
ADDED
@@ -0,0 +1,50 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'split/experiment'
|
3
|
+
require 'split/alternative'
|
4
|
+
require 'split/helper'
|
5
|
+
require 'redis/namespace'
|
6
|
+
|
7
|
+
module Split
|
8
|
+
extend self
|
9
|
+
# Accepts:
|
10
|
+
# 1. A 'hostname:port' string
|
11
|
+
# 2. A 'hostname:port:db' string (to select the Redis db)
|
12
|
+
# 3. A 'hostname:port/namespace' string (to set the Redis namespace)
|
13
|
+
# 4. A redis URL string 'redis://host:port'
|
14
|
+
# 5. An instance of `Redis`, `Redis::Client`, `Redis::DistRedis`,
|
15
|
+
# or `Redis::Namespace`.
|
16
|
+
def redis=(server)
|
17
|
+
if server.respond_to? :split
|
18
|
+
if server =~ /redis\:\/\//
|
19
|
+
redis = Redis.connect(:url => server, :thread_safe => true)
|
20
|
+
else
|
21
|
+
server, namespace = server.split('/', 2)
|
22
|
+
host, port, db = server.split(':')
|
23
|
+
redis = Redis.new(:host => host, :port => port,
|
24
|
+
:thread_safe => true, :db => db)
|
25
|
+
end
|
26
|
+
namespace ||= :split
|
27
|
+
|
28
|
+
@redis = Redis::Namespace.new(namespace, :redis => redis)
|
29
|
+
elsif server.respond_to? :namespace=
|
30
|
+
@redis = server
|
31
|
+
else
|
32
|
+
@redis = Redis::Namespace.new(:split, :redis => server)
|
33
|
+
end
|
34
|
+
end
|
35
|
+
|
36
|
+
# Returns the current Redis connection. If none has been created, will
|
37
|
+
# create a new one.
|
38
|
+
def redis
|
39
|
+
return @redis if @redis
|
40
|
+
self.redis = 'localhost:6379'
|
41
|
+
self.redis
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
45
|
+
if defined?(Rails)
|
46
|
+
class ActionController::Base
|
47
|
+
ActionController::Base.send :include, Split::Helper
|
48
|
+
ActionController::Base.helper Split::Helper
|
49
|
+
end
|
50
|
+
end
|
@@ -0,0 +1,77 @@
|
|
1
|
+
module Split
|
2
|
+
class Alternative
|
3
|
+
attr_accessor :name
|
4
|
+
attr_accessor :participant_count
|
5
|
+
attr_accessor :completed_count
|
6
|
+
attr_accessor :experiment_name
|
7
|
+
|
8
|
+
def initialize(name, experiment_name, counters = {})
|
9
|
+
@experiment_name = experiment_name
|
10
|
+
@name = name
|
11
|
+
@participant_count = counters['participant_count'].to_i
|
12
|
+
@completed_count = counters['completed_count'].to_i
|
13
|
+
end
|
14
|
+
|
15
|
+
def increment_participation
|
16
|
+
@participant_count +=1
|
17
|
+
self.save
|
18
|
+
end
|
19
|
+
|
20
|
+
def increment_completion
|
21
|
+
@completed_count +=1
|
22
|
+
self.save
|
23
|
+
end
|
24
|
+
|
25
|
+
def conversion_rate
|
26
|
+
return 0 if participant_count.zero?
|
27
|
+
(completed_count.to_f/participant_count.to_f)
|
28
|
+
end
|
29
|
+
|
30
|
+
def z_score
|
31
|
+
# CTR_E = the CTR within the experiment split
|
32
|
+
# CTR_C = the CTR within the control split
|
33
|
+
# E = the number of impressions within the experiment split
|
34
|
+
# C = the number of impressions within the control split
|
35
|
+
|
36
|
+
experiment = Split::Experiment.find(@experiment_name)
|
37
|
+
control = experiment.alternatives[0]
|
38
|
+
alternative = self
|
39
|
+
|
40
|
+
return 'N/A' if control.name == alternative.name
|
41
|
+
|
42
|
+
ctr_e = alternative.conversion_rate
|
43
|
+
ctr_c = control.conversion_rate
|
44
|
+
|
45
|
+
e = alternative.participant_count
|
46
|
+
c = control.participant_count
|
47
|
+
|
48
|
+
standard_deviation = ((ctr_e / ctr_c**3) * ((e*ctr_e)+(c*ctr_c)-(ctr_c*ctr_e)*(c+e))/(c*e)) ** 0.5
|
49
|
+
|
50
|
+
z_score = ((ctr_e / ctr_c) - 1) / standard_deviation
|
51
|
+
end
|
52
|
+
|
53
|
+
def save
|
54
|
+
if Split.redis.hgetall("#{experiment_name}:#{name}")
|
55
|
+
Split.redis.hset "#{experiment_name}:#{name}", 'participant_count', @participant_count
|
56
|
+
Split.redis.hset "#{experiment_name}:#{name}", 'completed_count', @completed_count
|
57
|
+
else
|
58
|
+
Split.redis.hmset "#{experiment_name}:#{name}", 'participant_count', 'completed_count', @participant_count, @completed_count
|
59
|
+
end
|
60
|
+
end
|
61
|
+
|
62
|
+
def self.find(name, experiment_name)
|
63
|
+
counters = Split.redis.hgetall "#{experiment_name}:#{name}"
|
64
|
+
self.new(name, experiment_name, counters)
|
65
|
+
end
|
66
|
+
|
67
|
+
def self.find_or_create(name, experiment_name)
|
68
|
+
self.find(name, experiment_name) || self.create(name, experiment_name)
|
69
|
+
end
|
70
|
+
|
71
|
+
def self.create(name, experiment_name)
|
72
|
+
alt = self.new(name, experiment_name)
|
73
|
+
alt.save
|
74
|
+
alt
|
75
|
+
end
|
76
|
+
end
|
77
|
+
end
|
@@ -0,0 +1,35 @@
|
|
1
|
+
require 'sinatra/base'
|
2
|
+
require 'split'
|
3
|
+
|
4
|
+
module Split
|
5
|
+
class Dashboard < Sinatra::Base
|
6
|
+
dir = File.dirname(File.expand_path(__FILE__))
|
7
|
+
|
8
|
+
set :views, "#{dir}/dashboard/views"
|
9
|
+
set :public, "#{dir}/dashboard/public"
|
10
|
+
set :static, true
|
11
|
+
|
12
|
+
helpers do
|
13
|
+
def url(*path_parts)
|
14
|
+
[ path_prefix, path_parts ].join("/").squeeze('/')
|
15
|
+
end
|
16
|
+
|
17
|
+
def path_prefix
|
18
|
+
request.env['SCRIPT_NAME']
|
19
|
+
end
|
20
|
+
end
|
21
|
+
|
22
|
+
get '/' do
|
23
|
+
@experiments = Split::Experiment.all
|
24
|
+
erb :index
|
25
|
+
end
|
26
|
+
|
27
|
+
post '/:experiment' do
|
28
|
+
@experiment = Split::Experiment.find(params[:experiment])
|
29
|
+
@alternative = Split::Alternative.find(params[:alternative], params[:experiment])
|
30
|
+
@experiment.winner = @alternative.name
|
31
|
+
@experiment.save
|
32
|
+
redirect url('/')
|
33
|
+
end
|
34
|
+
end
|
35
|
+
end
|
@@ -0,0 +1,48 @@
|
|
1
|
+
html, body, div, span, applet, object, iframe,
|
2
|
+
h1, h2, h3, h4, h5, h6, p, blockquote, pre,
|
3
|
+
a, abbr, acronym, address, big, cite, code,
|
4
|
+
del, dfn, em, font, img, ins, kbd, q, s, samp,
|
5
|
+
small, strike, strong, sub, sup, tt, var,
|
6
|
+
dl, dt, dd, ul, li,
|
7
|
+
form, label, legend,
|
8
|
+
table, caption, tbody, tfoot, thead, tr, th, td {
|
9
|
+
margin: 0;
|
10
|
+
padding: 0;
|
11
|
+
border: 0;
|
12
|
+
outline: 0;
|
13
|
+
font-weight: inherit;
|
14
|
+
font-style: normal;
|
15
|
+
font-size: 100%;
|
16
|
+
font-family: inherit;
|
17
|
+
}
|
18
|
+
|
19
|
+
:focus {
|
20
|
+
outline: 0;
|
21
|
+
}
|
22
|
+
|
23
|
+
body {
|
24
|
+
line-height: 1;
|
25
|
+
}
|
26
|
+
|
27
|
+
ul {
|
28
|
+
list-style: none;
|
29
|
+
}
|
30
|
+
|
31
|
+
table {
|
32
|
+
border-collapse: collapse;
|
33
|
+
border-spacing: 0;
|
34
|
+
}
|
35
|
+
|
36
|
+
caption, th, td {
|
37
|
+
text-align: left;
|
38
|
+
font-weight: normal;
|
39
|
+
}
|
40
|
+
|
41
|
+
blockquote:before, blockquote:after,
|
42
|
+
q:before, q:after {
|
43
|
+
content: "";
|
44
|
+
}
|
45
|
+
|
46
|
+
blockquote, q {
|
47
|
+
quotes: "" "";
|
48
|
+
}
|
@@ -0,0 +1,34 @@
|
|
1
|
+
html { background:#efefef; font-family:Arial, Verdana, sans-serif; font-size:13px; }
|
2
|
+
body { padding:0; margin:0; }
|
3
|
+
|
4
|
+
.header { background:#000; padding:8px 5% 0 5%; border-bottom:1px solid #444;border-bottom:5px solid #0080FF;}
|
5
|
+
.header h1 { color:#333; font-size:90%; font-weight:bold; margin-bottom:6px;}
|
6
|
+
.header ul li { display:inline;}
|
7
|
+
.header ul li a { color:#fff; text-decoration:none; margin-right:10px; display:inline-block; padding:8px; -webkit-border-top-right-radius:6px; -webkit-border-top-left-radius:6px; -moz-border-radius-topleft:6px; -moz-border-radius-topright:6px; }
|
8
|
+
.header ul li a:hover { background:#333;}
|
9
|
+
.header ul li.current a { background:#0080FF; font-weight:bold; color:#fff;}
|
10
|
+
|
11
|
+
#main { padding:10px 5%; background:#fff; overflow:hidden; }
|
12
|
+
#main .logo { float:right; margin:10px;}
|
13
|
+
#main span.hl { background:#efefef; padding:2px;}
|
14
|
+
#main h1 { margin:10px 0; font-size:190%; font-weight:bold; color:#0080FF;}
|
15
|
+
#main h2 { margin:10px 0; font-size:130%;}
|
16
|
+
#main table { width:100%; margin:10px 0;}
|
17
|
+
#main table tr td, #main table tr th { border:1px solid #ccc; padding:6px;}
|
18
|
+
#main table tr th { background:#efefef; color:#888; font-size:80%; font-weight:bold;}
|
19
|
+
#main table tr td.no-data { text-align:center; padding:40px 0; color:#999; font-style:italic; font-size:130%;}
|
20
|
+
#main a { color:#111;}
|
21
|
+
#main p { margin:5px 0;}
|
22
|
+
#main p.intro { margin-bottom:15px; font-size:85%; color:#999; margin-top:0; line-height:1.3;}
|
23
|
+
#main h1.wi { margin-bottom:5px;}
|
24
|
+
#main p.sub { font-size:95%; color:#999;}
|
25
|
+
|
26
|
+
#main table.queues { width:40%;}
|
27
|
+
|
28
|
+
#main table .totals td{ background:#eee; font-weight:bold; }
|
29
|
+
|
30
|
+
#footer { padding:10px 5%; background:#efefef; color:#999; font-size:85%; line-height:1.5; border-top:5px solid #ccc; padding-top:10px;}
|
31
|
+
#footer p a { color:#999;}
|
32
|
+
|
33
|
+
|
34
|
+
|
@@ -0,0 +1,56 @@
|
|
1
|
+
<h1>Split Dashboard</h1>
|
2
|
+
<p class="intro">The list below contains all the registered experiments along with the number of test participants, completed and conversion rate currently in the system.</p>
|
3
|
+
|
4
|
+
<% @experiments.each do |experiment| %>
|
5
|
+
<h2><%= experiment.name %></h2>
|
6
|
+
<table class="queues">
|
7
|
+
<tr>
|
8
|
+
<th>Alternative Name</th>
|
9
|
+
<th>Participants</th>
|
10
|
+
<th>Non-finished</th>
|
11
|
+
<th>Completed</th>
|
12
|
+
<th>Conversion Rate</th>
|
13
|
+
<th>Z-Score</th>
|
14
|
+
<th>Winner</th>
|
15
|
+
</tr>
|
16
|
+
|
17
|
+
<% total_participants = total_completed = 0 %>
|
18
|
+
<% experiment.alternatives.each do |alternative| %>
|
19
|
+
<tr>
|
20
|
+
<td><%= alternative.name %></td>
|
21
|
+
<td><%= alternative.participant_count %></td>
|
22
|
+
<td><%= alternative.participant_count - alternative.completed_count %></td>
|
23
|
+
<td><%= alternative.completed_count %></td>
|
24
|
+
<td><%= (alternative.conversion_rate * 100).round(2) %>%</td>
|
25
|
+
<td><%= alternative.z_score %></td>
|
26
|
+
<td>
|
27
|
+
<% if experiment.winner %>
|
28
|
+
<% if experiment.winner.name == alternative.name %>
|
29
|
+
Winner
|
30
|
+
<% else %>
|
31
|
+
Loser
|
32
|
+
<% end %>
|
33
|
+
<% else %>
|
34
|
+
<form action="<%= url experiment.name %>" method='post'>
|
35
|
+
<input type='hidden' name='alternative' value='<%= alternative.name %>'>
|
36
|
+
<input type="submit" value="Use this">
|
37
|
+
</form>
|
38
|
+
<% end %>
|
39
|
+
</td>
|
40
|
+
</tr>
|
41
|
+
|
42
|
+
<% total_participants += alternative.participant_count %>
|
43
|
+
<% total_completed += alternative.completed_count %>
|
44
|
+
<% end %>
|
45
|
+
|
46
|
+
<tr class="totals">
|
47
|
+
<td>Totals</td>
|
48
|
+
<td><%= total_participants %></td>
|
49
|
+
<td><%= total_participants - total_completed %></td>
|
50
|
+
<td><%= total_completed %></td>
|
51
|
+
<td>N/A</td>
|
52
|
+
<td>N/A</td>
|
53
|
+
<td>N/A</td>
|
54
|
+
</tr>
|
55
|
+
</table>
|
56
|
+
<% end %>
|
@@ -0,0 +1,22 @@
|
|
1
|
+
<!DOCTYPE html>
|
2
|
+
<html>
|
3
|
+
<head>
|
4
|
+
<meta content='text/html; charset=utf-8' http-equiv='Content-Type'>
|
5
|
+
<link href="<%= url 'reset.css' %>" media="screen" rel="stylesheet" type="text/css">
|
6
|
+
<link href="<%= url 'style.css' %>" media="screen" rel="stylesheet" type="text/css">
|
7
|
+
|
8
|
+
<title>Split</title>
|
9
|
+
|
10
|
+
</head>
|
11
|
+
<body>
|
12
|
+
<div class="header"></div>
|
13
|
+
|
14
|
+
<div id="main">
|
15
|
+
<%= yield %>
|
16
|
+
</div>
|
17
|
+
|
18
|
+
<div id="footer">
|
19
|
+
<p>Powered by <a href="http://github.com/andrew/split">Split</a> v<%=Split::VERSION %></p>
|
20
|
+
</div>
|
21
|
+
</body>
|
22
|
+
</html>
|
@@ -0,0 +1,59 @@
|
|
1
|
+
module Split
|
2
|
+
class Experiment
|
3
|
+
attr_accessor :name
|
4
|
+
attr_accessor :alternatives
|
5
|
+
attr_accessor :winner
|
6
|
+
|
7
|
+
def initialize(name, *alternatives)
|
8
|
+
@name = name.to_s
|
9
|
+
@alternatives = alternatives
|
10
|
+
end
|
11
|
+
|
12
|
+
def winner
|
13
|
+
if w = Split.redis.hget(:experiment_winner, name)
|
14
|
+
return Split::Alternative.find(w, name)
|
15
|
+
else
|
16
|
+
nil
|
17
|
+
end
|
18
|
+
end
|
19
|
+
|
20
|
+
def winner=(winner_name)
|
21
|
+
Split.redis.hset(:experiment_winner, name, winner_name.to_s)
|
22
|
+
end
|
23
|
+
|
24
|
+
def alternatives
|
25
|
+
@alternatives.map {|a| Split::Alternative.find_or_create(a, name)}
|
26
|
+
end
|
27
|
+
|
28
|
+
def next_alternative
|
29
|
+
winner || alternatives.sort_by{|a| a.participant_count + rand}.first
|
30
|
+
end
|
31
|
+
|
32
|
+
def save
|
33
|
+
Split.redis.sadd(:experiments, name)
|
34
|
+
@alternatives.each {|a| Split.redis.sadd(name, a) }
|
35
|
+
end
|
36
|
+
|
37
|
+
def self.all
|
38
|
+
Array(Split.redis.smembers(:experiments)).map {|e| find(e)}
|
39
|
+
end
|
40
|
+
|
41
|
+
def self.find(name)
|
42
|
+
if Split.redis.exists(name)
|
43
|
+
self.new(name, *Split.redis.smembers(name))
|
44
|
+
else
|
45
|
+
raise 'Experiment not found'
|
46
|
+
end
|
47
|
+
end
|
48
|
+
|
49
|
+
def self.find_or_create(name, *alternatives)
|
50
|
+
if Split.redis.exists(name)
|
51
|
+
return self.new(name, *Split.redis.smembers(name))
|
52
|
+
else
|
53
|
+
experiment = self.new(name, *alternatives)
|
54
|
+
experiment.save
|
55
|
+
return experiment
|
56
|
+
end
|
57
|
+
end
|
58
|
+
end
|
59
|
+
end
|
data/lib/split/helper.rb
ADDED
@@ -0,0 +1,28 @@
|
|
1
|
+
module Split
|
2
|
+
module Helper
|
3
|
+
def ab_test(experiment_name, *alternatives)
|
4
|
+
experiment = Split::Experiment.find_or_create(experiment_name, *alternatives)
|
5
|
+
return experiment.winner.name if experiment.winner
|
6
|
+
|
7
|
+
if ab_user[experiment_name]
|
8
|
+
return ab_user[experiment_name]
|
9
|
+
else
|
10
|
+
alternative = experiment.next_alternative
|
11
|
+
alternative.increment_participation
|
12
|
+
ab_user[experiment_name] = alternative.name
|
13
|
+
return alternative.name
|
14
|
+
end
|
15
|
+
end
|
16
|
+
|
17
|
+
def finished(experiment_name)
|
18
|
+
alternative_name = ab_user[experiment_name]
|
19
|
+
alternative = Split::Alternative.find(alternative_name, experiment_name)
|
20
|
+
alternative.increment_completion
|
21
|
+
session[:split].delete(experiment_name)
|
22
|
+
end
|
23
|
+
|
24
|
+
def ab_user
|
25
|
+
session[:split] ||= {}
|
26
|
+
end
|
27
|
+
end
|
28
|
+
end
|
@@ -0,0 +1,67 @@
|
|
1
|
+
require 'spec_helper'
|
2
|
+
require 'split/experiment'
|
3
|
+
|
4
|
+
describe Split::Experiment do
|
5
|
+
before(:each) { Split.redis.flushall }
|
6
|
+
|
7
|
+
it "should have a name" do
|
8
|
+
experiment = Split::Experiment.new('basket_text', 'Basket', "Cart")
|
9
|
+
experiment.name.should eql('basket_text')
|
10
|
+
end
|
11
|
+
|
12
|
+
it "should have alternatives" do
|
13
|
+
experiment = Split::Experiment.new('basket_text', 'Basket', "Cart")
|
14
|
+
experiment.alternatives.length.should be 2
|
15
|
+
end
|
16
|
+
|
17
|
+
it "should save to redis" do
|
18
|
+
experiment = Split::Experiment.new('basket_text', 'Basket', "Cart")
|
19
|
+
experiment.save
|
20
|
+
Split.redis.exists('basket_text').should be true
|
21
|
+
end
|
22
|
+
|
23
|
+
it "should return an existing experiment" do
|
24
|
+
experiment = Split::Experiment.new('basket_text', 'Basket', "Cart")
|
25
|
+
experiment.save
|
26
|
+
Split::Experiment.find('basket_text').name.should eql('basket_text')
|
27
|
+
end
|
28
|
+
|
29
|
+
describe 'winner' do
|
30
|
+
it "should have no winner initially" do
|
31
|
+
experiment = Split::Experiment.find_or_create('link_color', 'blue', 'red')
|
32
|
+
experiment.winner.should be_nil
|
33
|
+
end
|
34
|
+
|
35
|
+
it "should allow you to specify a winner" do
|
36
|
+
experiment = Split::Experiment.find_or_create('link_color', 'blue', 'red')
|
37
|
+
experiment.winner = 'red'
|
38
|
+
|
39
|
+
experiment = Split::Experiment.find_or_create('link_color', 'blue', 'red')
|
40
|
+
experiment.winner.name.should == 'red'
|
41
|
+
end
|
42
|
+
end
|
43
|
+
|
44
|
+
describe 'next_alternative' do
|
45
|
+
it "should return a random alternative from those with the least participants" do
|
46
|
+
experiment = Split::Experiment.find_or_create('link_color', 'blue', 'red', 'green')
|
47
|
+
|
48
|
+
Split::Alternative.find('blue', 'link_color').increment_participation
|
49
|
+
Split::Alternative.find('red', 'link_color').increment_participation
|
50
|
+
|
51
|
+
experiment.next_alternative.name.should == 'green'
|
52
|
+
end
|
53
|
+
|
54
|
+
it "should always return the winner if one exists" do
|
55
|
+
experiment = Split::Experiment.find_or_create('link_color', 'blue', 'red', 'green')
|
56
|
+
green = Split::Alternative.find('green', 'link_color')
|
57
|
+
experiment.winner = 'green'
|
58
|
+
|
59
|
+
experiment.next_alternative.name.should == 'green'
|
60
|
+
green.increment_participation
|
61
|
+
|
62
|
+
experiment = Split::Experiment.find_or_create('link_color', 'blue', 'red', 'green')
|
63
|
+
experiment.next_alternative.name.should == 'green'
|
64
|
+
end
|
65
|
+
end
|
66
|
+
end
|
67
|
+
|
data/spec/helper_spec.rb
ADDED
@@ -0,0 +1,86 @@
|
|
1
|
+
require 'spec_helper'
|
2
|
+
|
3
|
+
describe Split::Helper do
|
4
|
+
include Split::Helper
|
5
|
+
|
6
|
+
before(:each) do
|
7
|
+
Split.redis.flushall
|
8
|
+
@session = {}
|
9
|
+
end
|
10
|
+
|
11
|
+
describe "ab_test" do
|
12
|
+
it "should assign a random alternative to a new user when there are an equal number of alternatives assigned" do
|
13
|
+
ab_test('link_color', 'blue', 'red')
|
14
|
+
['red', 'blue'].should include(ab_user['link_color'])
|
15
|
+
end
|
16
|
+
|
17
|
+
it "should increment the participation counter after assignment to a new user" do
|
18
|
+
experiment = Split::Experiment.find_or_create('link_color', 'blue', 'red')
|
19
|
+
|
20
|
+
previous_red_count = Split::Alternative.find('red', 'link_color').participant_count
|
21
|
+
previous_blue_count = Split::Alternative.find('blue', 'link_color').participant_count
|
22
|
+
|
23
|
+
ab_test('link_color', 'blue', 'red')
|
24
|
+
|
25
|
+
new_red_count = Split::Alternative.find('red', 'link_color').participant_count
|
26
|
+
new_blue_count = Split::Alternative.find('blue', 'link_color').participant_count
|
27
|
+
|
28
|
+
(new_red_count + new_blue_count).should eql(previous_red_count + previous_blue_count + 1)
|
29
|
+
end
|
30
|
+
|
31
|
+
it "should return the given alternative for an existing user" do
|
32
|
+
experiment = Split::Experiment.find_or_create('link_color', 'blue', 'red')
|
33
|
+
alternative = ab_test('link_color', 'blue', 'red')
|
34
|
+
repeat_alternative = ab_test('link_color', 'blue', 'red')
|
35
|
+
alternative.should eql repeat_alternative
|
36
|
+
end
|
37
|
+
|
38
|
+
it 'should always return the winner if one is present' do
|
39
|
+
experiment = Split::Experiment.find_or_create('link_color', 'blue', 'red')
|
40
|
+
experiment.winner = "orange"
|
41
|
+
|
42
|
+
ab_test('link_color', 'blue', 'red').should == 'orange'
|
43
|
+
end
|
44
|
+
end
|
45
|
+
|
46
|
+
describe 'finished' do
|
47
|
+
it 'should increment the counter for the completed alternative' do
|
48
|
+
experiment = Split::Experiment.find_or_create('link_color', 'blue', 'red')
|
49
|
+
alternative_name = ab_test('link_color', 'blue', 'red')
|
50
|
+
|
51
|
+
previous_completion_count = Split::Alternative.find(alternative_name, 'link_color').completed_count
|
52
|
+
|
53
|
+
finished('link_color')
|
54
|
+
|
55
|
+
new_completion_count = Split::Alternative.find(alternative_name, 'link_color').completed_count
|
56
|
+
|
57
|
+
new_completion_count.should eql(previous_completion_count + 1)
|
58
|
+
end
|
59
|
+
|
60
|
+
it "should clear out the user's participation from their session" do
|
61
|
+
experiment = Split::Experiment.find_or_create('link_color', 'blue', 'red')
|
62
|
+
alternative_name = ab_test('link_color', 'blue', 'red')
|
63
|
+
|
64
|
+
previous_completion_count = Split::Alternative.find(alternative_name, 'link_color').completed_count
|
65
|
+
|
66
|
+
session[:split].should == {"link_color" => alternative_name}
|
67
|
+
finished('link_color')
|
68
|
+
session[:split].should == {}
|
69
|
+
end
|
70
|
+
end
|
71
|
+
|
72
|
+
describe 'conversions' do
|
73
|
+
it 'should return a conversion rate for an alternative' do
|
74
|
+
experiment = Split::Experiment.find_or_create('link_color', 'blue', 'red')
|
75
|
+
alternative_name = ab_test('link_color', 'blue', 'red')
|
76
|
+
|
77
|
+
previous_convertion_rate = Split::Alternative.find(alternative_name, 'link_color').conversion_rate
|
78
|
+
previous_convertion_rate.should eql(0.0)
|
79
|
+
|
80
|
+
finished('link_color')
|
81
|
+
|
82
|
+
new_convertion_rate = Split::Alternative.find(alternative_name, 'link_color').conversion_rate
|
83
|
+
new_convertion_rate.should eql(1.0)
|
84
|
+
end
|
85
|
+
end
|
86
|
+
end
|
data/spec/spec_helper.rb
ADDED
data/split.gemspec
ADDED
@@ -0,0 +1,27 @@
|
|
1
|
+
# -*- encoding: utf-8 -*-
|
2
|
+
$:.push File.expand_path("../lib", __FILE__)
|
3
|
+
require "split/version"
|
4
|
+
|
5
|
+
Gem::Specification.new do |s|
|
6
|
+
s.name = "split"
|
7
|
+
s.version = Split::VERSION
|
8
|
+
s.platform = Gem::Platform::RUBY
|
9
|
+
s.authors = ["Andrew Nesbitt"]
|
10
|
+
s.email = ["andrewnez@gmail.com"]
|
11
|
+
s.homepage = ""
|
12
|
+
s.summary = %q{Rack based split testing framework}
|
13
|
+
|
14
|
+
s.rubyforge_project = "split"
|
15
|
+
|
16
|
+
s.files = `git ls-files`.split("\n")
|
17
|
+
s.test_files = `git ls-files -- {test,spec,features}/*`.split("\n")
|
18
|
+
s.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
|
19
|
+
s.require_paths = ["lib"]
|
20
|
+
|
21
|
+
s.add_dependency(%q<redis>, ["~> 2.1"])
|
22
|
+
s.add_dependency(%q<redis-namespace>, ["~> 0.10.0"])
|
23
|
+
s.add_dependency(%q<sinatra>, ["~> 1.2.6"])
|
24
|
+
|
25
|
+
# Development Dependencies
|
26
|
+
s.add_development_dependency(%q<rspec>, ["~> 2.6"])
|
27
|
+
end
|
metadata
ADDED
@@ -0,0 +1,141 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: split
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
prerelease: false
|
5
|
+
segments:
|
6
|
+
- 0
|
7
|
+
- 1
|
8
|
+
- 0
|
9
|
+
version: 0.1.0
|
10
|
+
platform: ruby
|
11
|
+
authors:
|
12
|
+
- Andrew Nesbitt
|
13
|
+
autorequire:
|
14
|
+
bindir: bin
|
15
|
+
cert_chain: []
|
16
|
+
|
17
|
+
date: 2011-05-17 00:00:00 -04:00
|
18
|
+
default_executable:
|
19
|
+
dependencies:
|
20
|
+
- !ruby/object:Gem::Dependency
|
21
|
+
name: redis
|
22
|
+
prerelease: false
|
23
|
+
requirement: &id001 !ruby/object:Gem::Requirement
|
24
|
+
none: false
|
25
|
+
requirements:
|
26
|
+
- - ~>
|
27
|
+
- !ruby/object:Gem::Version
|
28
|
+
segments:
|
29
|
+
- 2
|
30
|
+
- 1
|
31
|
+
version: "2.1"
|
32
|
+
type: :runtime
|
33
|
+
version_requirements: *id001
|
34
|
+
- !ruby/object:Gem::Dependency
|
35
|
+
name: redis-namespace
|
36
|
+
prerelease: false
|
37
|
+
requirement: &id002 !ruby/object:Gem::Requirement
|
38
|
+
none: false
|
39
|
+
requirements:
|
40
|
+
- - ~>
|
41
|
+
- !ruby/object:Gem::Version
|
42
|
+
segments:
|
43
|
+
- 0
|
44
|
+
- 10
|
45
|
+
- 0
|
46
|
+
version: 0.10.0
|
47
|
+
type: :runtime
|
48
|
+
version_requirements: *id002
|
49
|
+
- !ruby/object:Gem::Dependency
|
50
|
+
name: sinatra
|
51
|
+
prerelease: false
|
52
|
+
requirement: &id003 !ruby/object:Gem::Requirement
|
53
|
+
none: false
|
54
|
+
requirements:
|
55
|
+
- - ~>
|
56
|
+
- !ruby/object:Gem::Version
|
57
|
+
segments:
|
58
|
+
- 1
|
59
|
+
- 2
|
60
|
+
- 6
|
61
|
+
version: 1.2.6
|
62
|
+
type: :runtime
|
63
|
+
version_requirements: *id003
|
64
|
+
- !ruby/object:Gem::Dependency
|
65
|
+
name: rspec
|
66
|
+
prerelease: false
|
67
|
+
requirement: &id004 !ruby/object:Gem::Requirement
|
68
|
+
none: false
|
69
|
+
requirements:
|
70
|
+
- - ~>
|
71
|
+
- !ruby/object:Gem::Version
|
72
|
+
segments:
|
73
|
+
- 2
|
74
|
+
- 6
|
75
|
+
version: "2.6"
|
76
|
+
type: :development
|
77
|
+
version_requirements: *id004
|
78
|
+
description:
|
79
|
+
email:
|
80
|
+
- andrewnez@gmail.com
|
81
|
+
executables: []
|
82
|
+
|
83
|
+
extensions: []
|
84
|
+
|
85
|
+
extra_rdoc_files: []
|
86
|
+
|
87
|
+
files:
|
88
|
+
- .gitignore
|
89
|
+
- Gemfile
|
90
|
+
- README.mdown
|
91
|
+
- Rakefile
|
92
|
+
- lib/split.rb
|
93
|
+
- lib/split/alternative.rb
|
94
|
+
- lib/split/dashboard.rb
|
95
|
+
- lib/split/dashboard/public/reset.css
|
96
|
+
- lib/split/dashboard/public/style.css
|
97
|
+
- lib/split/dashboard/views/index.erb
|
98
|
+
- lib/split/dashboard/views/layout.erb
|
99
|
+
- lib/split/experiment.rb
|
100
|
+
- lib/split/helper.rb
|
101
|
+
- lib/split/version.rb
|
102
|
+
- spec/experiment_spec.rb
|
103
|
+
- spec/helper_spec.rb
|
104
|
+
- spec/spec_helper.rb
|
105
|
+
- split.gemspec
|
106
|
+
has_rdoc: true
|
107
|
+
homepage: ""
|
108
|
+
licenses: []
|
109
|
+
|
110
|
+
post_install_message:
|
111
|
+
rdoc_options: []
|
112
|
+
|
113
|
+
require_paths:
|
114
|
+
- lib
|
115
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
116
|
+
none: false
|
117
|
+
requirements:
|
118
|
+
- - ">="
|
119
|
+
- !ruby/object:Gem::Version
|
120
|
+
segments:
|
121
|
+
- 0
|
122
|
+
version: "0"
|
123
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
124
|
+
none: false
|
125
|
+
requirements:
|
126
|
+
- - ">="
|
127
|
+
- !ruby/object:Gem::Version
|
128
|
+
segments:
|
129
|
+
- 0
|
130
|
+
version: "0"
|
131
|
+
requirements: []
|
132
|
+
|
133
|
+
rubyforge_project: split
|
134
|
+
rubygems_version: 1.3.7
|
135
|
+
signing_key:
|
136
|
+
specification_version: 3
|
137
|
+
summary: Rack based split testing framework
|
138
|
+
test_files:
|
139
|
+
- spec/experiment_spec.rb
|
140
|
+
- spec/helper_spec.rb
|
141
|
+
- spec/spec_helper.rb
|