bandit 0.0.5 → 0.0.6
Sign up to get free protection for your applications and to get access to all the features.
- data/README.rdoc +6 -1
- data/lib/bandit/extensions/controller_concerns.rb +10 -0
- data/lib/bandit/version.rb +1 -1
- data/whybandit.rdoc +2 -2
- metadata +15 -17
data/README.rdoc
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
= bandit
|
2
2
|
|
3
|
-
Bandit is a multi-armed bandit optimization framework for Rails. It provides an alternative to A/B testing in Rails. For background and a comparison with A/B testing, see the whybandit.rdoc document.
|
3
|
+
Bandit is a multi-armed bandit optimization framework for Rails. It provides an alternative to A/B testing in Rails. For background and a comparison with A/B testing, see the whybandit.rdoc document or the blog post here[http://findingscience.com/rails/vanity/statistics/testing/2011/11/12/bandit:-a-b-testing-alternative-for-rails.html].
|
4
4
|
|
5
5
|
= Installation
|
6
6
|
First, add the following to your Gemfile in your Rails 3 app:
|
@@ -46,6 +46,11 @@ To track a conversion in your controller:
|
|
46
46
|
|
47
47
|
bandit_convert! :click_test
|
48
48
|
|
49
|
+
You can also request a choice in the conroller:
|
50
|
+
|
51
|
+
redirect_to bandit_choose(:some_url_test)
|
52
|
+
|
53
|
+
|
49
54
|
= Dashboard
|
50
55
|
|
51
56
|
rails generate bandit:dashboard
|
@@ -18,6 +18,16 @@ module Bandit
|
|
18
18
|
cookies.delete(cookiename)
|
19
19
|
end
|
20
20
|
end
|
21
|
+
|
22
|
+
def bandit_choose(exp)
|
23
|
+
name = "bandit_#{exp}".intern
|
24
|
+
|
25
|
+
# choose url param with preference
|
26
|
+
value = params[name].nil? ? cookies.signed[name] : params[name]
|
27
|
+
|
28
|
+
# choose with default, and set cookie
|
29
|
+
cookies.signed[name] = Bandit.get_experiment(exp).choose(value)
|
30
|
+
end
|
21
31
|
end
|
22
32
|
|
23
33
|
end
|
data/lib/bandit/version.rb
CHANGED
data/whybandit.rdoc
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
= Bandit vs A/B Testing
|
2
2
|
In a typical A/B test, two alternatives are compared to see which produces the most "conversions" (that is, desired results). For instance, if you have a website with a big "Sign Up" button that you want visitors to click, you may wish to choose different background colors. Under typical A/B testing guildlines, you would pick a number (say, *N*) of users for a test and show half of them one color and half of them another color. After users are shown the button, you record the number of clicks that result from viewing each color. Once *N* users view one of the two alternatives, a statistical test (generally categorical, like a Chi-Square Test or a G-Test) is run to determine whether or not the number of clicks (aka, "conversions") for one color were higher than the number of clicks for the other color. This test determines whether the difference you observed was likely due simply to chance or whether the difference you saw was more likely due to an actual difference in the rate of conversion.
|
3
3
|
|
4
|
-
This method of testing is popular, but is fraught with issues (practical and statistical). Bandit provides an implementation of an alternative method of testing that solves many of these issues.
|
4
|
+
This method of testing is popular, but is fraught with issues (practical and statistical). Bandit provides an implementation of an alternative method of testing for Rails that solves many of these issues.
|
5
5
|
|
6
6
|
== Issues with A/B Testing
|
7
7
|
There are a number of issues with A/B testing (some of which have been described in more detail here[http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-make-out-like-a-bandit]):
|
@@ -11,7 +11,7 @@ There are a number of issues with A/B testing (some of which have been described
|
|
11
11
|
1. With A/B Testing, you need to have a fixed sample size to make the test valid (otherwise, you run the risk of repeated significance testing errors, as described in more detail here[http://www.evanmiller.org/how-not-to-run-an-ab-test.html]).
|
12
12
|
1. Due to the fixed sample size requirement, you may have to wait a while before you get any results from your test (especially if the expected improvement is marginal, in which case your sample size would need to be larger). This problem can be compounded if you don't get much traffic.
|
13
13
|
1. Designers and developers don't want to (and shouldn't have to) understand statistical concepts like power[http://en.wikipedia.org/wiki/Statistical_power], p-values[http://en.wikipedia.org/wiki/P-value], or confidence[http://en.wikipedia.org/wiki/Confidence_intervals] when creating and evaluating tests.
|
14
|
-
1. There are no
|
14
|
+
1. There are no good answers for what you should do when A performs just as well as B. Was the sample size just too small (implying you should try again with a large sample)? Go with A? Go with B? Does it matter? The reality is it may matter - but you won't know.
|
15
15
|
|
16
16
|
== The Bandit Method
|
17
17
|
The ultimate goal of A/B testing is to increase conversions. The problem can be described terms that differ greatly from the multitude of questions A/B testing brings (i.e., "Is A better than B?" followed by "Is B better than C?" followed by "Is C better than D?" _ad_ _infinitum_). Instead, imagine you have a multitude of possible alternatives, and you want to make a decent choice between alternatives you know perform well and alternatives you haven't tried very often each time a user requests a page. With each page load, pick the best alternative most of the time and an alternative that hasn't been displayed much some of the time. After each display, monitor the conversions and update what you consider the "better" alternatives to be. This is the basic method of a solution to what is called the multi-armed bandit problem.
|
metadata
CHANGED
@@ -1,13 +1,13 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: bandit
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
hash:
|
5
|
-
prerelease:
|
4
|
+
hash: 19
|
5
|
+
prerelease:
|
6
6
|
segments:
|
7
7
|
- 0
|
8
8
|
- 0
|
9
|
-
-
|
10
|
-
version: 0.0.
|
9
|
+
- 6
|
10
|
+
version: 0.0.6
|
11
11
|
platform: ruby
|
12
12
|
authors:
|
13
13
|
- Brian Muller
|
@@ -15,12 +15,9 @@ autorequire:
|
|
15
15
|
bindir: bin
|
16
16
|
cert_chain: []
|
17
17
|
|
18
|
-
date:
|
19
|
-
default_executable:
|
18
|
+
date: 2012-04-16 00:00:00 Z
|
20
19
|
dependencies:
|
21
20
|
- !ruby/object:Gem::Dependency
|
22
|
-
name: rails
|
23
|
-
prerelease: false
|
24
21
|
requirement: &id001 !ruby/object:Gem::Requirement
|
25
22
|
none: false
|
26
23
|
requirements:
|
@@ -32,11 +29,11 @@ dependencies:
|
|
32
29
|
- 0
|
33
30
|
- 5
|
34
31
|
version: 3.0.5
|
35
|
-
type: :runtime
|
36
32
|
version_requirements: *id001
|
37
|
-
|
38
|
-
|
33
|
+
name: rails
|
34
|
+
type: :runtime
|
39
35
|
prerelease: false
|
36
|
+
- !ruby/object:Gem::Dependency
|
40
37
|
requirement: &id002 !ruby/object:Gem::Requirement
|
41
38
|
none: false
|
42
39
|
requirements:
|
@@ -48,11 +45,11 @@ dependencies:
|
|
48
45
|
- 2
|
49
46
|
- 2
|
50
47
|
version: 2.2.2
|
51
|
-
type: :runtime
|
52
48
|
version_requirements: *id002
|
53
|
-
|
54
|
-
|
49
|
+
name: redis
|
50
|
+
type: :runtime
|
55
51
|
prerelease: false
|
52
|
+
- !ruby/object:Gem::Dependency
|
56
53
|
requirement: &id003 !ruby/object:Gem::Requirement
|
57
54
|
none: false
|
58
55
|
requirements:
|
@@ -62,8 +59,10 @@ dependencies:
|
|
62
59
|
segments:
|
63
60
|
- 0
|
64
61
|
version: "0"
|
65
|
-
type: :development
|
66
62
|
version_requirements: *id003
|
63
|
+
name: rdoc
|
64
|
+
type: :development
|
65
|
+
prerelease: false
|
67
66
|
description: Bandit provides a way to do multi-armed bandit optimization of alternatives in a rails website
|
68
67
|
email:
|
69
68
|
- brian.muller@livingsocial.com
|
@@ -130,7 +129,6 @@ files:
|
|
130
129
|
- test/redis_storage_test.rb
|
131
130
|
- test/storage_test_base.rb
|
132
131
|
- whybandit.rdoc
|
133
|
-
has_rdoc: true
|
134
132
|
homepage: https://github.com/bmuller/bandit
|
135
133
|
licenses: []
|
136
134
|
|
@@ -160,7 +158,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
160
158
|
requirements: []
|
161
159
|
|
162
160
|
rubyforge_project: bandit
|
163
|
-
rubygems_version: 1.
|
161
|
+
rubygems_version: 1.8.17
|
164
162
|
signing_key:
|
165
163
|
specification_version: 3
|
166
164
|
summary: Multi-armed bandit testing in rails
|