moses-vanity 1.7.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (98) hide show
  1. data/.autotest +22 -0
  2. data/.gitignore +7 -0
  3. data/.rvmrc +3 -0
  4. data/.travis.yml +13 -0
  5. data/CHANGELOG +374 -0
  6. data/Gemfile +28 -0
  7. data/MIT-LICENSE +21 -0
  8. data/README.rdoc +108 -0
  9. data/Rakefile +189 -0
  10. data/bin/vanity +16 -0
  11. data/doc/_config.yml +2 -0
  12. data/doc/_layouts/_header.html +34 -0
  13. data/doc/_layouts/page.html +47 -0
  14. data/doc/_metrics.textile +12 -0
  15. data/doc/ab_testing.textile +210 -0
  16. data/doc/configuring.textile +45 -0
  17. data/doc/contributing.textile +93 -0
  18. data/doc/credits.textile +23 -0
  19. data/doc/css/page.css +83 -0
  20. data/doc/css/print.css +43 -0
  21. data/doc/css/syntax.css +7 -0
  22. data/doc/email.textile +129 -0
  23. data/doc/experimental.textile +31 -0
  24. data/doc/faq.textile +8 -0
  25. data/doc/identity.textile +43 -0
  26. data/doc/images/ab_in_dashboard.png +0 -0
  27. data/doc/images/clear_winner.png +0 -0
  28. data/doc/images/price_options.png +0 -0
  29. data/doc/images/sidebar_test.png +0 -0
  30. data/doc/images/signup_metric.png +0 -0
  31. data/doc/images/vanity.png +0 -0
  32. data/doc/index.textile +91 -0
  33. data/doc/metrics.textile +231 -0
  34. data/doc/rails.textile +89 -0
  35. data/doc/site.js +27 -0
  36. data/generators/templates/vanity_migration.rb +53 -0
  37. data/generators/vanity_generator.rb +8 -0
  38. data/lib/generators/templates/vanity_migration.rb +53 -0
  39. data/lib/generators/vanity_generator.rb +15 -0
  40. data/lib/vanity.rb +36 -0
  41. data/lib/vanity/adapters/abstract_adapter.rb +140 -0
  42. data/lib/vanity/adapters/active_record_adapter.rb +248 -0
  43. data/lib/vanity/adapters/mock_adapter.rb +157 -0
  44. data/lib/vanity/adapters/mongodb_adapter.rb +178 -0
  45. data/lib/vanity/adapters/redis_adapter.rb +160 -0
  46. data/lib/vanity/backport.rb +26 -0
  47. data/lib/vanity/commands/list.rb +21 -0
  48. data/lib/vanity/commands/report.rb +64 -0
  49. data/lib/vanity/commands/upgrade.rb +34 -0
  50. data/lib/vanity/experiment/ab_test.rb +507 -0
  51. data/lib/vanity/experiment/base.rb +214 -0
  52. data/lib/vanity/frameworks.rb +16 -0
  53. data/lib/vanity/frameworks/rails.rb +318 -0
  54. data/lib/vanity/helpers.rb +66 -0
  55. data/lib/vanity/images/x.gif +0 -0
  56. data/lib/vanity/metric/active_record.rb +85 -0
  57. data/lib/vanity/metric/base.rb +244 -0
  58. data/lib/vanity/metric/google_analytics.rb +83 -0
  59. data/lib/vanity/metric/remote.rb +53 -0
  60. data/lib/vanity/playground.rb +396 -0
  61. data/lib/vanity/templates/_ab_test.erb +28 -0
  62. data/lib/vanity/templates/_experiment.erb +5 -0
  63. data/lib/vanity/templates/_experiments.erb +7 -0
  64. data/lib/vanity/templates/_metric.erb +14 -0
  65. data/lib/vanity/templates/_metrics.erb +13 -0
  66. data/lib/vanity/templates/_report.erb +27 -0
  67. data/lib/vanity/templates/_vanity.js.erb +20 -0
  68. data/lib/vanity/templates/flot.min.js +1 -0
  69. data/lib/vanity/templates/jquery.min.js +19 -0
  70. data/lib/vanity/templates/vanity.css +26 -0
  71. data/lib/vanity/templates/vanity.js +82 -0
  72. data/lib/vanity/version.rb +11 -0
  73. data/test/adapters/redis_adapter_test.rb +17 -0
  74. data/test/experiment/ab_test.rb +771 -0
  75. data/test/experiment/base_test.rb +150 -0
  76. data/test/experiments/age_and_zipcode.rb +19 -0
  77. data/test/experiments/metrics/cheers.rb +3 -0
  78. data/test/experiments/metrics/signups.rb +2 -0
  79. data/test/experiments/metrics/yawns.rb +3 -0
  80. data/test/experiments/null_abc.rb +5 -0
  81. data/test/metric/active_record_test.rb +277 -0
  82. data/test/metric/base_test.rb +293 -0
  83. data/test/metric/google_analytics_test.rb +104 -0
  84. data/test/metric/remote_test.rb +109 -0
  85. data/test/myapp/app/controllers/application_controller.rb +2 -0
  86. data/test/myapp/app/controllers/main_controller.rb +7 -0
  87. data/test/myapp/config/boot.rb +110 -0
  88. data/test/myapp/config/environment.rb +10 -0
  89. data/test/myapp/config/environments/production.rb +0 -0
  90. data/test/myapp/config/routes.rb +3 -0
  91. data/test/passenger_test.rb +43 -0
  92. data/test/playground_test.rb +26 -0
  93. data/test/rails_dashboard_test.rb +37 -0
  94. data/test/rails_helper_test.rb +36 -0
  95. data/test/rails_test.rb +389 -0
  96. data/test/test_helper.rb +145 -0
  97. data/vanity.gemspec +26 -0
  98. metadata +202 -0
@@ -0,0 +1,8 @@
1
+ ---
2
+ layout: page
3
+ title: FAQ
4
+ ---
5
+
6
+ *Q:* Is Vanity just for Rails?
7
+
8
+ *A:* In spite of what the name says, no. Vanity can be used with any Ruby project and any Web framework. It also has nice integration for Rails.
@@ -0,0 +1,43 @@
1
+ ---
2
+ layout: page
3
+ title: Managing Identity
4
+ ---
5
+
6
+ For effective A/B tests, you want to:
7
+ # Randomly show different alternatives to different people
8
+ # Consistently show the same alternative to the same person
9
+ # Know which alternative you're showing and tracking
10
+ # When running multiple tests at once, keep them independent
11
+
12
+ Vanity will assign each visitor a unique identifier and store it in a cookie that persists across sessions. That way, each visitor will get to see the same alternatives on repeating visits. Assuming they use the same browser on all visits.
13
+
14
+ If you have a better way of tracking visitors, e.g. using sign in, you'll want to use that instead. That way the same person gets treated to the same experiment, even if they switch between machines and browsers.
15
+
16
+ You can choose either option using the @use_vanity@ method. In the first case, just call @use_vanity@ from within the @ApplicationController@. In the second case, you'll want to pass @use_vanity@ either a block that returns an identity value, or the name of a method that returns an object which provides the identity.
17
+
18
+ Sounds complicated? These two examples are equivalent:
19
+
20
+ <pre>
21
+ class ApplicationController < ActionController::Base
22
+ use_vanity :current_user
23
+ end
24
+ </pre>
25
+
26
+ <pre>
27
+ class ApplicationController < ActionController::Base
28
+ use_vanity { |c| c.current_user && c.current_user.id }
29
+ end
30
+ </pre>
31
+
32
+ If you use either block or method name and they return @nil@, Vanity will fallback on persistent cookie mechanism.
33
+
34
+ An identity can be anything. For example, if you're running an experiment to test a new feature that will be available in some projects but not others, you'll want to slice the audience by project identifier, not user ID.
35
+
36
+ You can also give each experiment a different identity using the @identify@ callback. Here's an example that tells one experiment to use a project identifier:
37
+
38
+ <pre>
39
+ ab_test "New feature" do
40
+ description "New feature only available to some projects"
41
+ identify { |c| c.current_project.id }
42
+ end
43
+ </pre>
Binary file
@@ -0,0 +1,91 @@
1
+ ---
2
+ layout: page
3
+ title: Welcome to Vanity
4
+ ---
5
+
6
+ "Vanity":http://github.com/assaf/vanity is an Experiment Driven Development framework for Rails.
7
+
8
+ !images/sidebar_test.png!
9
+
10
+
11
+ h3. Reading Order
12
+
13
+ * "2 Minute Demo(A/B Testing with Rails in 5 easy steps)":#intro
14
+ * "Metrics":metrics.html
15
+ * "A/B Testing(Everything you need to know)":ab_testing.html
16
+ * "Using with Rails":rails.html
17
+ * "Managing Identity":identity.html
18
+
19
+ Also:
20
+
21
+ * "Experiment Driven Development(Introduction to EDD and Vanity)":http://blog.labnotes.org/2009/11/19/vanity-experiment-driven-development-for-rails/
22
+ * "Get the code(Official Github repository)":http://github.com/assaf/vanity
23
+ * "API reference":api/index.html
24
+ * "Join vanity-talk list":http://groups.google.com/group/vanity-talk
25
+ * "Contributing to Vanity":contributing.html
26
+
27
+
28
+ h3(#intro). A/B Testing with Rails (in 5 easy steps)
29
+
30
+ *Step 1:* Start using Vanity in your Rails application:
31
+
32
+ <pre>
33
+ Rails::Initializer.run do |config|
34
+ gem.config "vanity"
35
+
36
+ config.after_initialize do
37
+ require "vanity"
38
+ end
39
+ end
40
+ </pre>
41
+
42
+ And:
43
+
44
+ <pre>
45
+ class ApplicationController < ActionController::Base
46
+ use_vanity :current_user
47
+ end
48
+ </pre>
49
+
50
+ *Step 2:* Define your first A/B test. This experiment goes in the file <code>experiments/price_options.rb</code>:
51
+
52
+ <pre>
53
+ ab_test "Price options" do
54
+ description "Mirror, mirror on the wall, who's the better price of all?"
55
+ alternatives 19, 25, 29
56
+ metrics :signup
57
+ end
58
+ </pre>
59
+
60
+ *Step 3:* Present the different options to your users:
61
+
62
+ <pre>
63
+ <h2>Get started for only $<%= ab_test :price_options %> a month!</h2>
64
+ </pre>
65
+
66
+ *Step 4:* Measure conversion:
67
+
68
+ <pre>
69
+ class SignupController < ApplicationController
70
+ def signup
71
+ @account = Account.new(params[:account])
72
+ if @account.save
73
+ track! :signup
74
+ redirect_to @acccount
75
+ else
76
+ render action: :offer
77
+ end
78
+ end
79
+ end
80
+ </pre>
81
+
82
+ *Step 5:* Check the report:
83
+
84
+ <pre>
85
+ vanity report --output vanity.html
86
+ </pre>
87
+
88
+ !images/price_options.png!
89
+
90
+ Read more about "A/B Testing ...":ab_testing.html
91
+
@@ -0,0 +1,231 @@
1
+ ---
2
+ layout: page
3
+ title: Metrics
4
+ ---
5
+
6
+ <div id="toc">
7
+ # "Defining a Metric":#define
8
+ # "Metrics From Your Database":#ar
9
+ # "Google Analytics":#ga
10
+ # "Creating Your Own Metric":#custom
11
+ # "Digging Deeper":#deeper
12
+ </div>
13
+
14
+
15
+ A good starting point for improving -- on anything -- is measuring. Vanity allows you to measure multiple metrics, best way to tell how well your experiments are doing.
16
+
17
+ <blockquote>
18
+ "Startup metrics for pirates: AARRR!":http://500hats.typepad.com/500blogs/2007/09/startup-metrics.html
19
+ # Acquisition
20
+ # Activation
21
+ # Retention
22
+ # Referral
23
+ # Revenue
24
+ </blockquote>
25
+
26
+
27
+
28
+ h3(#define). Defining a Metric
29
+
30
+ Vanity always loads metrics defined in the @experiments/metrics@ directory. A metric definition is a Ruby file that looks like this:
31
+
32
+ <pre>
33
+ metric "Signup (Activation)" do
34
+ description "Measures how many people signed up for our awesome service."
35
+ end
36
+ </pre>
37
+
38
+ That's a basic metric and you feed it data by calling the @track!@ method. For example:
39
+
40
+ <pre>
41
+ class AccountsController < ApplicationController
42
+
43
+ def create
44
+ @person = Person.new(params[:person])
45
+ if @person.save
46
+ track! :signup # track successful sign up
47
+ UserSession.create person
48
+ redirect_to root_url
49
+ else
50
+ render :action=>:new
51
+ end
52
+ end
53
+
54
+ end
55
+ </pre>
56
+
57
+ The metric identifier is the same as the file name. The above example defines the metric @:signup@ in the file @experiments/metrics/signup.rb@.
58
+
59
+ You can call @track!@ with a value to track. This example tracks how many items were bought during the day:
60
+
61
+ <pre>
62
+ def checkout
63
+ track! :items, @cart.items.count
64
+ . . .
65
+ end
66
+ </pre>
67
+
68
+ Calling @track!@ with no value is the same as calling with one, and for convenience you can pass zero and negative numbers, both will be ignored.
69
+
70
+ !images/signup_metric.png!
71
+
72
+ Define, track, and you're ready to roll.
73
+
74
+
75
+ h3(#ar). Metrics From Your Database
76
+
77
+ If you already have the data, why not use it?
78
+
79
+ This example defines a metric for signups, based on the number of @Account@ records created each day:
80
+
81
+ <pre>
82
+ metric "Signup (Activation)" do
83
+ description "Measures how many people signed up for our awesome service."
84
+ model Account
85
+ end
86
+ </pre>
87
+
88
+ You don't need to call @track!@ with this metric, all the data already exists. It's a simple query to count the number of records created, grouped by their timestamp (@created_at@). And since it's querying the database, you'll immediately see historical data for the last 90 days.
89
+
90
+ Even though the metric itself doesn't store any information, it needs to update experiments whenever new records are created. To do that, it registers itself as an @after_create@ callback on the model.
91
+
92
+ Some metrics measure values, not occurrences. For example, this metric measures user satisfaction by calculating average value from the column @rating@:
93
+
94
+ <pre>
95
+ metric "Satisfaction Survey" do
96
+ description "Measures how many people signed up for our awesome service."
97
+ model Survey, :average=>:rating
98
+ end
99
+ </pre>
100
+
101
+ The aggregates you can use this way are: @:average@, @:minimum@, @:maximum@ and @:sum@.
102
+
103
+ You can use a condition when the metric only applies to some records. Here's a metric that only measures unlimited accounts:
104
+
105
+ <pre>
106
+ metric "Signups to Unlimited" do
107
+ description "Signups to our All You Can Eat and Burp Unlimited plan."
108
+ model Account, :conditions=>{ :plan_type=>'unlimited' }
109
+ end
110
+ </pre>
111
+
112
+ If you have named scopes, you'll want to use them instead:
113
+
114
+ <pre>
115
+ metric "Signups to Unlimited" do
116
+ description "Signups to our All You Can Eat and Burp Unlimited plan."
117
+ model Account.unlimited
118
+ end
119
+ </pre>
120
+
121
+ When you view this metric, it calculates the number of accounts created on any given day that are currently unlimited plans. So, if ten accounts were created over the past week, and today five of them upgraded to unlimited plan, the metric will show five unlimited accounts (current state) but spread over the past week (their @created_at@ timestamp).
122
+
123
+ If your metric uses aggregates or conditions, and the aggregate/conditional attributes change over time, and you need to know when the change took place, consider tracking the event.
124
+
125
+ This example tracks when accounts were created or upgraded to unlimited plan:
126
+
127
+ <pre>
128
+ metric "Signups (Unlimited)" do
129
+ description "Signups to our All You Can Eat and Burp Unlimited plan (including upgrades)."
130
+ Account.after_save do |account|
131
+ track! if account.plan_type_changed? && account.plan_type == 'unlimited'
132
+ end
133
+ end
134
+ </pre>
135
+
136
+
137
+ h3(#ga). Google Analytics
138
+
139
+ You can easily include Google Analytics metrics in your Vanity dashboard. You'll need, in addition to Vanity, to use "Garb":http://github.com/vigetlabs/garb, a Ruby wrapper for the Google Analytics API.
140
+
141
+ Login to Google Analytics using either username and password, or OAuth authentication token. Here's a sample @config/environment@ snippet:
142
+
143
+ <pre>
144
+ Rails::Initializer.run do |config|
145
+ gems.config "vanity"
146
+ gems.config "garb"
147
+
148
+ . . .
149
+ config.after_initialize do
150
+ require "garb"
151
+ ga = YAML.load_file(Rails.root + "config/ga.yml")
152
+ Garb::Session.login(ga['email'], ga['password'], account_type: "GOOGLE")
153
+ end
154
+ end
155
+ </pre>
156
+
157
+ To define a metric that corresponds to the Google Analytics daily visitors:
158
+
159
+ <pre>
160
+ metric "Acquisition: Visitors" do
161
+ description "Unique visitors on any given page, as tracked by Google Analytics"
162
+ google_analytics "UA-1828623-6", :visitors
163
+ end
164
+ </pre>
165
+
166
+ The first argument is the GA profile, the second argument the GA metric name (defaults to @pageviews@).
167
+
168
+ You can use the full Garb API by accessing the report directly, for example:
169
+
170
+ <pre>
171
+ metric "Activation: Signups" do
172
+ google_analytics "UA-1828623-6"
173
+ report.filters do
174
+ eql(:page_path, 'welcome')
175
+ end
176
+ end
177
+ </pre>
178
+
179
+ See "the Garb documentation":http://rdoc.info/projects/vigetlabs/garb and "Google Analytics API":http://code.google.com/apis/analytics/docs/gdata/gdataReferenceDimensionsMetrics.html#bounceRate for more details.
180
+
181
+
182
+ h3(#custom). Creating Your Own Metric
183
+
184
+ Got other ideas for metrics? Writing your own metric is fairly simple.
185
+
186
+ The easiest way to create your own metric is by adding your own @values@ method, for example:
187
+
188
+ <pre>
189
+ metric "Hours in a day" do
190
+ description "Measures how many hours in each day."
191
+ def values(from, to)
192
+ (from..to).map { |i| 24 }
193
+ end
194
+ end
195
+ </pre>
196
+
197
+ This example is based on @Vanity::Metric@. You can, of course, base your metric on any other class.
198
+
199
+ For simplicity, a metric is any object that implements these two methods:
200
+
201
+ * @name@ -- Returns the metric's name, which will show up in the dashboard/report.
202
+ * @values@ -- Receives a start date and end date and returns an array of values for all dates in that range (inclusive).
203
+
204
+ A metric may also implement these methods:
205
+
206
+ * @description@ -- Returns human readable description.
207
+ * @bounds@ -- Returns acceptable upper and lower bounds (@nil@ if unknown).
208
+ * @hook@ -- "A/B tests":ab_testing.html use this to manage their own book keeping.
209
+
210
+ If you wrote your own metric implementation, please consider "contributing it to Vanity":contributing.html so we can all benefit from it. Thanks.
211
+
212
+
213
+ h3(#deeper). Digging Deeper
214
+
215
+ All metrics are listed in @Vanity.playground.metrics@, a hash that maps metric identifier to metric object. Methods like @track!@ and @metrics@ (see "A/B tests":ab_testing.html) reference metrics using their identifier.
216
+
217
+ On startup, Vanity loads all the metrics it finds in the @experiments/metrics@ directory. The metric identifier is the same as the file name, so @experiments/metrics/coolness.rb@ becomes @:coolness@.
218
+
219
+ You can always populate the hash with your own metrics.
220
+
221
+ When Vanity loads a metric, it evaluates the metric definition in a context that has two methods: @metric@ and @playground@. The @metric@ method creates a new @Vanity::Metric@ object, and evaluates the block in the context of that object, so when you see the metric definition using methods like @description@ or @model@, these are all invoked on the metric object itself.
222
+
223
+ A @Vanity::Metric@ object responds to @track!@ and increments a record in the database (an _O(1)_ operation). It creates one record for each day, accumulating that day's count. When generating reports, the @values@ method fetches the values of all these keys (also _O(1)_).
224
+
225
+ You can call @track!@ with a value higher than one, and it will increment the day's count by that value.
226
+
227
+ Any time you track a metric, the metric passes its identifier, timestamp and count (if more than zero) to all its hooks. "A/B tests":ab_testing.html use hooks to manage their own book keeping. When you define an experiment and tell it which metric(s) to use, the experiment registers itself by calling the @hook@ method.
228
+
229
+ When you call @model@ on a metric, this method changes the metric definition by rewriting the @values@ method to perform a query, rewriting the @track!@ method to update hooks but not the database, and register an @after_create@ callback that updates the hooks.
230
+
231
+ How about some tips & tricks for getting the most out of metrics (you might call them "best practices")? Got any to share?
@@ -0,0 +1,89 @@
1
+ ---
2
+ layout: page
3
+ title: Using with Rails
4
+ ---
5
+
6
+ <div id="toc">
7
+ # "Configuring Vanity":#config
8
+ # "Test Environment":#test
9
+ # "Dashboard":#dashboard
10
+ # "Unicorn and Forking Servers":#fork
11
+ </div>
12
+
13
+ This guide is written for Rails 2.3.5. If you have any tips for Rails 3.0, please share.
14
+
15
+
16
+ h3(#config). Configuring Vanity
17
+
18
+ Start by telling Rails to use the Vanity gem, either using @config.gem "vanity"@ or by adding @gem "vanity"@ to your Gemfile.
19
+
20
+ You will most likely need to @require "vanity"@ from within @after_initialize@ in order to use it everywhere in your app:
21
+
22
+ <pre>
23
+ Rails::Initializer.run do |config|
24
+ . . .
25
+ config.after_initialize do
26
+ require "vanity"
27
+ end
28
+ end
29
+ </pre>
30
+
31
+ If you have a @config/vanity.yml@ file, Vanity will read the configuration for the current environment. For example:
32
+
33
+ <pre>
34
+ staging:
35
+ adapter: redis
36
+ host: staging.internal
37
+ production:
38
+ adapter: mongo
39
+ host: live.internal
40
+ database: vanity
41
+ </pre>
42
+
43
+ If you want to use Google Analytics, you must also tell Rails to include the @garb@ gem, and login for a new session. You'll want to do that for production, not for development if you like developing offline:
44
+
45
+ <pre>
46
+ config.after_initialize do
47
+ require "garb"
48
+ Garb::Session.login('..ga email..', '..ga pwd..', account_type: "GOOGLE")
49
+ end
50
+ </pre>
51
+
52
+ There's generally no need to collect metric and experiment data outside production environment. Under Rails, Vanity turns collection on only if the environment name is "production". You can control this from @config/environments@ by setting @Vanity.playground.collecting@ to true/false. When collection is off, Vanity doesn't connect to the database server, so there's no need to set a database configuration for these environments.
53
+
54
+
55
+ h3(#dashboard). Dashboard
56
+
57
+ Start by adding a new resource in @config/routes.rb@:
58
+
59
+ <pre>
60
+ map.vanity "/vanity/:action/:id", :controller=>:vanity
61
+ </pre>
62
+
63
+ Create a new controller for Vanity:
64
+
65
+ <pre>
66
+ class VanityController < ApplicationController
67
+ include Vanity::Rails::Dashboard
68
+ end
69
+ </pre>
70
+
71
+ Now open your browser to "http://localhost:3000/vanity":http://localhost:3000/vanity.
72
+
73
+ The Dashboard renders complete HTML pages with CSS and all necessary JavaScript libraries. Thankfully, HTML is forgiving enough that it will render correctly even with your existing application layout. You can decide to keep your layout, or tell the controller to set @layout false@.
74
+
75
+
76
+ h3(#fork). Unicorn and Forking Servers
77
+
78
+ Unicorn forks the master process to create worker processes efficiently. Since the master processes opens a connection to the database, all workers end up sharing that connection, resulting in ugly contention issues.
79
+
80
+ The cure is simple, use the @after_fork@ hook to reconnect each worker process. Here's the relevant part from my @config/unicorn.rb@:
81
+
82
+ <pre>
83
+ after_fork do |server, worker|
84
+ ActiveRecord::Base.establish_connection
85
+ Vanity.playground.establish_connection
86
+ end
87
+ </pre>
88
+
89
+ You'll run into this issue with other forking servers. Vanity can detect when it runs under Passenger and automatically reconnect each forked process.