vanity 1.0.0 → 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (57) hide show
  1. data/CHANGELOG +35 -0
  2. data/README.rdoc +33 -6
  3. data/lib/vanity.rb +13 -7
  4. data/lib/vanity/backport.rb +43 -0
  5. data/lib/vanity/commands/report.rb +13 -3
  6. data/lib/vanity/experiment/ab_test.rb +98 -66
  7. data/lib/vanity/experiment/base.rb +51 -5
  8. data/lib/vanity/metric.rb +213 -0
  9. data/lib/vanity/mock_redis.rb +76 -0
  10. data/lib/vanity/playground.rb +78 -61
  11. data/lib/vanity/rails/dashboard.rb +11 -2
  12. data/lib/vanity/rails/helpers.rb +3 -3
  13. data/lib/vanity/templates/_ab_test.erb +3 -4
  14. data/lib/vanity/templates/_experiment.erb +4 -4
  15. data/lib/vanity/templates/_experiments.erb +2 -2
  16. data/lib/vanity/templates/_metric.erb +9 -0
  17. data/lib/vanity/templates/_metrics.erb +13 -0
  18. data/lib/vanity/templates/_report.erb +14 -3
  19. data/lib/vanity/templates/flot.min.js +1 -0
  20. data/lib/vanity/templates/jquery.min.js +19 -0
  21. data/lib/vanity/templates/vanity.css +16 -4
  22. data/lib/vanity/templates/vanity.js +96 -0
  23. data/test/ab_test_test.rb +159 -96
  24. data/test/experiment_test.rb +99 -18
  25. data/test/experiments/age_and_zipcode.rb +1 -0
  26. data/test/experiments/metrics/cheers.rb +3 -0
  27. data/test/experiments/metrics/signups.rb +2 -0
  28. data/test/experiments/metrics/yawns.rb +3 -0
  29. data/test/experiments/null_abc.rb +1 -0
  30. data/test/metric_test.rb +287 -0
  31. data/test/playground_test.rb +1 -80
  32. data/test/rails_test.rb +9 -6
  33. data/test/test_helper.rb +37 -6
  34. data/vanity.gemspec +1 -1
  35. data/vendor/{redis-0.1 → redis-rb}/LICENSE +0 -0
  36. data/vendor/{redis-0.1 → redis-rb}/README.markdown +0 -0
  37. data/vendor/{redis-0.1 → redis-rb}/Rakefile +0 -0
  38. data/vendor/redis-rb/bench.rb +44 -0
  39. data/vendor/redis-rb/benchmarking/suite.rb +24 -0
  40. data/vendor/redis-rb/benchmarking/worker.rb +71 -0
  41. data/vendor/redis-rb/bin/distredis +33 -0
  42. data/vendor/redis-rb/examples/basic.rb +16 -0
  43. data/vendor/redis-rb/examples/incr-decr.rb +18 -0
  44. data/vendor/redis-rb/examples/list.rb +26 -0
  45. data/vendor/redis-rb/examples/sets.rb +36 -0
  46. data/vendor/{redis-0.1 → redis-rb}/lib/dist_redis.rb +0 -0
  47. data/vendor/{redis-0.1 → redis-rb}/lib/hash_ring.rb +0 -0
  48. data/vendor/{redis-0.1 → redis-rb}/lib/pipeline.rb +0 -2
  49. data/vendor/{redis-0.1 → redis-rb}/lib/redis.rb +25 -7
  50. data/vendor/{redis-0.1 → redis-rb}/lib/redis/raketasks.rb +0 -0
  51. data/vendor/redis-rb/profile.rb +22 -0
  52. data/vendor/redis-rb/redis-rb.gemspec +30 -0
  53. data/vendor/{redis-0.1 → redis-rb}/spec/redis_spec.rb +113 -0
  54. data/vendor/{redis-0.1 → redis-rb}/spec/spec_helper.rb +0 -0
  55. data/vendor/redis-rb/speed.rb +16 -0
  56. data/vendor/{redis-0.1 → redis-rb}/tasks/redis.tasks.rb +5 -1
  57. metadata +37 -14
data/CHANGELOG CHANGED
@@ -1,3 +1,38 @@
1
+ == 1.1.0 (2009-12-4)
2
+ This release introduces metrics. Metrics are the gateway drug to better software.
3
+
4
+ It’s as simple as defining a metric:
5
+
6
+ metric "Cheers" do
7
+ description "They love us, don't they?"
8
+ end
9
+
10
+ Tracking it from your code:
11
+
12
+ track! :cheers
13
+
14
+ And watching the graph from the Dashboard.
15
+
16
+ You can (should) also use metrics with your A/B tests, for example:
17
+
18
+ ab_test "Pricing options" do
19
+ metrics :signup
20
+ alternatives 15, 25, 29
21
+ end
22
+
23
+ This new usage may become requirement in a future release.
24
+
25
+ Much thanks to Ian Sefferman for fixing issues with Ruby 1.8.7 and Rails support.
26
+
27
+ * Added: Metrics.
28
+ * Added: Use Vanity.playground.mock! when running tests and you'd rather not access a live Redis server.
29
+ * Changed: A/B tests now using metrics for tracking.
30
+ * Changed: Now throwing NameError instead of LoadError when failing to load experiment/metric. NameError can be rescued on same line.
31
+ * Changed: New, easier URL mapping for Dashboard: map.vanity "/vanity", :controller=>:vanity.
32
+ * Changed: All tests are green on Ruby 1.8.6, 1.8.7 and 1.9.1.
33
+ * Changed: Switched to redis-rb from http://github.com/ezmobius/redis-rb.
34
+ * Deprecated: Please call experiment method with experiment identifier (a symbol) and not experiment name.
35
+
1
36
  == 1.0.0 (2009-11-19)
2
37
  This release changes the way you define a new experiment. You can use a method suitable for the type of experiment you want to define, or call the generic define method (previously: experiment method). For example:
3
38
 
data/README.rdoc CHANGED
@@ -1,13 +1,13 @@
1
1
  Vanity is an Experiment Driven Development framework for Rails.
2
2
 
3
- * Documentation: http://vanity.labnotes.org
3
+ * All about Vanity: http://vanity.labnotes.org
4
4
  * On github: http://github.com/assaf/vanity
5
- * Vanity requires Ruby 1.9.1 or later, Redis 1.0 or later.
5
+ * Vanity requires Redis 1.0 or later.
6
6
 
7
7
  http://farm3.static.flickr.com/2540/4099665871_497f274f68_o.jpg
8
8
 
9
9
 
10
- == A/B Testing with Rails (in 5 easy steps)
10
+ == A/B Testing With Rails (In 5 Easy Steps)
11
11
 
12
12
  <b>Step 1:</b> Start using Vanity in your Rails application:
13
13
 
@@ -24,6 +24,7 @@ And:
24
24
  ab_test "Price options" do
25
25
  description "Mirror, mirror on the wall, who's the better price of all?"
26
26
  alternatives 19, 25, 29
27
+ metrics :signups
27
28
  end
28
29
 
29
30
  <b>Step 3:</b> Present the different options to your users:
@@ -36,7 +37,7 @@ And:
36
37
  def signup
37
38
  @account = Account.new(params[:account])
38
39
  if @account.save
39
- track! :pricing_options # <- here be conversion!
40
+ track! :signups
40
41
  redirect_to @acccount
41
42
  else
42
43
  render action: :offer
@@ -49,8 +50,34 @@ And:
49
50
  vanity --output vanity.html
50
51
 
51
52
 
53
+ == Building From Source
54
+
55
+ To run the test suite for the first time:
56
+
57
+ $ gem install rails mocha timecop
58
+ $ rake
59
+
60
+ You can also +rake test+ if you insist on being explicit.
61
+
62
+ To build the documentation:
63
+
64
+ $ gem install yardoc jekyll
65
+ $ rake docs
66
+ $ open html/index.html
67
+
68
+ To clean up after yourself:
69
+
70
+ $ rake clobber
71
+
72
+ To package Vanity as a gem and install on your machine:
73
+
74
+ $ rake install
75
+
76
+
52
77
  == Credits/License
53
78
 
54
- Idea behind Experiment Driven Development: Nathaniel Talbott (http://blog.talbott.ws).
79
+ Original code, copyright of Assaf Arkin, released under the MIT license.
80
+
81
+ Documentation available under the Creative Commons Attribution license.
55
82
 
56
- Copyright (C) 2009 Assaf Arkin, released under the MIT license.
83
+ For full list of credits and licenses: http://vanity.labnotes.org/credits.html.
data/lib/vanity.rb CHANGED
@@ -1,11 +1,14 @@
1
- $LOAD_PATH.unshift File.join(File.dirname(__FILE__), "../vendor/redis-0.1/lib")
1
+ $LOAD_PATH.unshift File.join(File.dirname(__FILE__), "../vendor/redis-rb/lib")
2
2
  require "redis"
3
3
  require "openssl"
4
+ require "date"
5
+ require "logger"
4
6
 
5
- # All the cool stuff happens in other places:
6
- # - Vanity::Helpers
7
- # - Vanity::Playground
8
- # - Experiment::AbTest
7
+ # All the cool stuff happens in other places.
8
+ # @see Vanity::Rails
9
+ # @see Vanity::Playground
10
+ # @see Vanity::Metric
11
+ # @see Vanity::Experiment
9
12
  module Vanity
10
13
  # Version number.
11
14
  module Version
@@ -17,8 +20,11 @@ module Vanity
17
20
  end
18
21
  end
19
22
 
20
- require "vanity/playground"
23
+ require "vanity/backport" if RUBY_VERSION < "1.9"
24
+ require "vanity/metric"
21
25
  require "vanity/experiment/base"
22
26
  require "vanity/experiment/ab_test"
23
- require "vanity/rails" if defined?(Rails)
27
+ require "vanity/playground"
28
+ Vanity.autoload :MockRedis, "vanity/mock_redis"
24
29
  Vanity.autoload :Commands, "vanity/commands"
30
+ require "vanity/rails" if defined?(Rails)
@@ -0,0 +1,43 @@
1
+ class Time
2
+ unless method_defined?(:to_date)
3
+ # Backported from Ruby 1.9.
4
+ def to_date
5
+ jd = Date.__send__(:civil_to_jd, year, mon, mday, Date::ITALY)
6
+ Date.new!(Date.__send__(:jd_to_ajd, jd, 0, 0), 0, Date::ITALY)
7
+ end
8
+ end
9
+ end
10
+
11
+ class Date
12
+ unless method_defined?(:to_date)
13
+ # Backported from Ruby 1.9.
14
+ def to_date
15
+ self
16
+ end
17
+ end
18
+
19
+ unless method_defined?(:to_time)
20
+ # Backported from Ruby 1.9.
21
+ def to_time
22
+ Time.local(year, mon, mday)
23
+ end
24
+ end
25
+ end
26
+
27
+ class Symbol
28
+ unless method_defined?(:to_proc)
29
+ # Backported from Ruby 1.9.
30
+ def to_proc
31
+ Proc.new { |*args| args.shift.__send__(self, *args) }
32
+ end
33
+ end
34
+ end
35
+
36
+ class Array
37
+ unless method_defined?(:minmax)
38
+ # Backported from Ruby 1.9.
39
+ def minmax
40
+ [min, max]
41
+ end
42
+ end
43
+ end
@@ -15,15 +15,25 @@ module Vanity
15
15
  struct.send :include, Render
16
16
  locals = struct.new(*locals.values_at(*keys))
17
17
  dir, base = File.split(path)
18
- path = File.read(File.join(dir, "_#{base}"))
19
- ERB.new(path, nil, '<').result(locals.instance_eval { binding })
18
+ path = File.join(dir, "_#{base}")
19
+ erb = ERB.new(File.read(path), nil, '<>')
20
+ erb.filename = path
21
+ erb.result(locals.instance_eval { binding })
20
22
  end
21
23
 
22
24
  # Escape HTML.
23
25
  def h(html)
24
- CGI.escape_html(html)
26
+ CGI.escapeHTML(html)
25
27
  end
26
28
 
29
+ # Dumbed down from Rails' simple_format.
30
+ def simple_format(text, options={})
31
+ open = "<p #{options.map { |k,v| "#{k}=\"#{CGI.escapeHTML v}\"" }.join(" ")}>"
32
+ text = open + text.gsub(/\r\n?/, "\n"). # \r\n and \r -> \n
33
+ gsub(/\n\n+/, "</p>\n\n#{open}"). # 2+ newline -> paragraph
34
+ gsub(/([^\n]\n)(?=[^\n])/, '\1<br />') + # 1 newline -> br
35
+ "</p>"
36
+ end
27
37
  end
28
38
 
29
39
  # Commands available when running Vanity from the command line (see bin/vanity).
@@ -44,7 +44,7 @@ module Vanity
44
44
 
45
45
  # Conversion rate calculated as converted/participants, rounded to 3 places.
46
46
  def conversion_rate
47
- @conversion_rate ||= (participants > 0 ? (converted.to_f/participants.to_f).round(3) : 0.0)
47
+ @conversion_rate ||= (participants > 0 ? (converted.to_f/participants.to_f * 1000).round / 1000.0 : 0.0)
48
48
  end
49
49
 
50
50
  # The measure we use to order (sort) alternatives and decide which one is better (by calculating z-score).
@@ -94,19 +94,39 @@ module Vanity
94
94
  @alternatives = [false, true]
95
95
  end
96
96
 
97
+
98
+ # -- Metric --
99
+
100
+ # Tells A/B test which metric we're measuring, or returns metric in use.
101
+ #
102
+ # @example Define A/B test against coolness metric
103
+ # ab_test "Background color" do
104
+ # metrics :coolness
105
+ # alternatives "red", "blue", "orange"
106
+ # end
107
+ # @example Find metric for A/B test
108
+ # puts "Measures: " + experiment(:background_color).metrics.map(&:name)
109
+ def metrics(*args)
110
+ @metrics = args.map { |id| @playground.metric(id) } unless args.empty?
111
+ @metrics
112
+ end
113
+
114
+
97
115
  # -- Alternatives --
98
116
 
99
- # Call this method once to set alternative values for this experiment.
100
- # Require at least two values. For example:
117
+ # Call this method once to set alternative values for this experiment
118
+ # (requires at least two values). Call without arguments to obtain
119
+ # current list of alternatives.
120
+ #
121
+ # @example Define A/B test with three alternatives
101
122
  # ab_test "Background color" do
123
+ # metrics :coolness
102
124
  # alternatives "red", "blue", "orange"
103
125
  # end
104
126
  #
105
- # Call without arguments to obtain current list of alternatives. For example:
127
+ # @example Find out which alternatives this test uses
106
128
  # alts = experiment(:background_color).alternatives
107
129
  # puts "#{alts.count} alternatives, with the colors: #{alts.map(&:value).join(", ")}"
108
- #
109
- # If you want to know how well each alternative is doing, use #score.
110
130
  def alternatives(*args)
111
131
  unless args.empty?
112
132
  @alternatives = args.clone
@@ -129,24 +149,25 @@ module Vanity
129
149
  end
130
150
  private :_alternatives
131
151
 
132
- # Returns an Alternative with the specified value. For example, given:
133
- # ab_test "Which color" do
134
- # alternatives :red, :green, :blue
135
- # end
136
- # Then:
152
+ # Returns an Alternative with the specified value.
153
+ #
154
+ # @example
137
155
  # alternative(:red) == alternatives[0]
138
156
  # alternative(:blue) == alternatives[2]
139
157
  def alternative(value)
140
158
  alternatives.find { |alt| alt.value == value }
141
159
  end
142
160
 
143
- # Defines an A/B test with two alternatives: false and true. For example:
161
+ # Defines an A/B test with two alternatives: false and true. This is the
162
+ # default pair of alternatives, so just syntactic sugar for those who love
163
+ # being explicit.
164
+ #
165
+ # @example
144
166
  # ab_test "More bacon" do
167
+ # metrics :yummyness
145
168
  # false_true
146
169
  # end
147
170
  #
148
- # This is the default pair of alternatives, so just syntactic sugar for
149
- # those who love being explicit.
150
171
  def false_true
151
172
  alternatives false, true
152
173
  end
@@ -160,7 +181,7 @@ module Vanity
160
181
  # alternative for the same identity, and randomly split alternatives
161
182
  # between different identities.
162
183
  #
163
- # For example:
184
+ # @example
164
185
  # color = experiment(:which_blue).choose
165
186
  def choose
166
187
  if active?
@@ -177,38 +198,22 @@ module Vanity
177
198
  @alternatives[index.to_i]
178
199
  end
179
200
 
180
- # Tracks a conversion. You probably want to use the Rails helper method
181
- # track! instead.
182
- #
183
- # For example:
184
- # experiment(:which_blue).track!
185
- def track!
186
- return unless active?
187
- identity = identity()
188
- return if redis[key("participants:#{identity}:show")]
189
- index = alternative_for(identity)
190
- if redis.sismember(key("alts:#{index}:participants"), identity)
191
- redis.sadd key("alts:#{index}:converted"), identity
192
- redis.incr key("alts:#{index}:conversions")
193
- end
194
- check_completion!
195
- end
196
-
197
201
 
198
202
  # -- Testing --
199
203
 
200
204
  # Forces this experiment to use a particular alternative. You'll want to
201
205
  # use this from your test cases to test for the different alternatives.
202
- # For example:
206
+ #
207
+ # @example Setup test to red button
203
208
  # setup do
204
- # experiment(:green_button).select(true)
209
+ # experiment(:button_color).select(:red)
205
210
  # end
206
211
  #
207
- # def test_shows_green_button
212
+ # def test_shows_red_button
208
213
  # . . .
209
214
  # end
210
215
  #
211
- # Use nil to clear out selection:
216
+ # @example Use nil to clear selection
212
217
  # teardown do
213
218
  # experiment(:green_button).select(nil)
214
219
  # end
@@ -324,7 +329,7 @@ module Vanity
324
329
  # -- Completion --
325
330
 
326
331
  # Defines how the experiment can choose the optimal outcome on completion.
327
-
332
+ #
328
333
  # By default, Vanity will take the best alternative (highest conversion
329
334
  # rate) and use that as the outcome. You experiment may have different
330
335
  # needs, maybe you want the least performing alternative, or factor cost
@@ -370,7 +375,7 @@ module Vanity
370
375
  # -- Store/validate --
371
376
 
372
377
  def destroy
373
- @alternatives.count.times do |i|
378
+ @alternatives.size.times do |i|
374
379
  redis.del key("alts:#{i}:participants")
375
380
  redis.del key("alts:#{i}:converted")
376
381
  redis.del key("alts:#{i}:conversions")
@@ -380,55 +385,82 @@ module Vanity
380
385
  end
381
386
 
382
387
  def save
383
- fail "Experiment #{name} needs at least two alternatives" unless alternatives.count >= 2
388
+ fail "Experiment #{name} needs at least two alternatives" unless alternatives.size >= 2
384
389
  super
390
+ if @metrics.nil? || @metrics.empty?
391
+ warn "Please use metrics method to explicitly state which metric you are measuring against."
392
+ metric = @playground.metrics[id] ||= Vanity::Metric.new(@playground, name)
393
+ @metrics = [metric]
394
+ end
395
+ @metrics.each do |metric|
396
+ metric.hook &method(:track!)
397
+ end
398
+ end
399
+
400
+ # Called when tracking associated metric.
401
+ def track!(metric_id, timestamp, count, *args)
402
+ return unless active?
403
+ identity = identity()
404
+ return if redis[key("participants:#{identity}:show")]
405
+ index = alternative_for(identity)
406
+ redis.sadd key("alts:#{index}:converted"), identity if redis.sismember(key("alts:#{index}:participants"), identity)
407
+ redis.incrby key("alts:#{index}:conversions"), count
408
+ check_completion!
385
409
  end
386
410
 
411
+ # If you are not embarrassed by the first version of your product, you’ve
412
+ # launched too late.
413
+ # -- Reid Hoffman, founder of LinkedIn
414
+
387
415
  protected
388
416
 
417
+ # Used for testing.
418
+ def fake(values)
419
+ values.each do |value, (participants, conversions)|
420
+ conversions ||= participants
421
+ participants.times do |identity|
422
+ index = @alternatives.index(value)
423
+ raise ArgumentError, "No alternative #{value.inspect} for #{name}" unless index
424
+ redis.sadd key("alts:#{index}:participants"), identity
425
+ end
426
+ conversions.times do |identity|
427
+ index = @alternatives.index(value)
428
+ raise ArgumentError, "No alternative #{value.inspect} for #{name}" unless index
429
+ redis.sadd key("alts:#{index}:converted"), identity
430
+ redis.incr key("alts:#{index}:conversions")
431
+ end
432
+ end
433
+ end
434
+
389
435
  # Chooses an alternative for the identity and returns its index. This
390
436
  # method always returns the same alternative for a given experiment and
391
437
  # identity, and randomly distributed alternatives for each identity (in the
392
438
  # same experiment).
393
439
  def alternative_for(identity)
394
- Digest::MD5.hexdigest("#{name}/#{identity}").to_i(17) % @alternatives.count
395
- end
396
-
397
- # Used for testing Vanity.
398
- def count_participant(identity, value)
399
- index = @alternatives.index(value)
400
- raise ArgumentError, "No alternative #{value.inspect} for #{name}" unless index
401
- redis.sadd key("alts:#{index}:participants"), identity
402
- self
403
- end
404
-
405
- # Used for testing Vanity.
406
- def count_conversion(identity, value)
407
- index = @alternatives.index(value)
408
- raise ArgumentError, "No alternative #{value.inspect} for #{name}" unless index
409
- redis.sadd key("alts:#{index}:converted"), identity
410
- redis.incr key("alts:#{index}:conversions")
411
- self
440
+ Digest::MD5.hexdigest("#{name}/#{identity}").to_i(17) % @alternatives.size
412
441
  end
413
442
 
414
443
  begin
415
444
  a = 50.0
416
445
  # Returns array of [z-score, percentage]
417
- norm_dist = (0.0..3.1).step(0.01).map { |x| [x, a += 1 / Math.sqrt(2 * Math::PI) * Math::E ** (-x ** 2 / 2)] }
446
+ norm_dist = []
447
+ (0.0..3.1).step(0.01) { |x| norm_dist << [x, a += 1 / Math.sqrt(2 * Math::PI) * Math::E ** (-x ** 2 / 2)] }
418
448
  # We're really only interested in 90%, 95%, 99% and 99.9%.
419
449
  Z_TO_PROBABILITY = [90, 95, 99, 99.9].map { |pct| [norm_dist.find { |x,a| a >= pct }.first, pct] }.reverse
420
450
  end
421
451
 
422
452
  end
423
- end
424
453
 
425
- module Definition
426
- # Define an A/B test with the given name. For example:
427
- # ab_test "New Banner" do
428
- # alternatives :red, :green, :blue
429
- # end
430
- def ab_test(name, &block)
431
- define name, :ab_test, &block
454
+
455
+ module Definition
456
+ # Define an A/B test with the given name. For example:
457
+ # ab_test "New Banner" do
458
+ # alternatives :red, :green, :blue
459
+ # end
460
+ def ab_test(name, &block)
461
+ define name, :ab_test, &block
462
+ end
432
463
  end
464
+
433
465
  end
434
466
  end