split 0.5.0 → 0.6.0

Sign up to get free protection for your applications and to get access to all the features.
@@ -10,8 +10,6 @@ rvm:
10
10
  - jruby-head
11
11
  - rbx-18mode
12
12
  - rbx-19mode
13
- - ree
14
13
  matrix:
15
14
  allow_failures:
16
- - rvm: ruby-head
17
- - rvm: 2.0.0
15
+ - rvm: ruby-head
@@ -1,4 +1,28 @@
1
- ## 0.5.0 (January 28, 2012)
1
+ ## 0.6.0 (April 4, 2013)
2
+
3
+ Features:
4
+
5
+ - Support for Ruby 2.0.0 (@phoet, #142)
6
+ - Multiple Goals (@liujin, #109)
7
+ - Ignoring IPs using Regular Expressions (@waynemoore, #119)
8
+ - Added ability to add more bots to the default list (@themgt, #140)
9
+ - Allow custom configuration of user blocking logic (@phoet , #148)
10
+
11
+ Bugfixes:
12
+
13
+ - Fixed regression in handling of config files (@iangreenleaf, #115)
14
+ - Fixed completion rate increases for experiments users aren't participating in (@philnash, #67)
15
+ - Handle exceptions from invalid JSON in cookies (@iangreenleaf, #126)
16
+
17
+ Misc:
18
+
19
+ - updated minimum json version requirement
20
+ - Refactor Yaml Configuration (@rtwomey, #124)
21
+ - Refactoring of Experiments (@iangreenleaf @tamird, #117 #118)
22
+ - Added more known Bots, including Pingdom, Bing, YandexBot (@julesie, @zinkkrysty)
23
+ - Improved Readme (@iangreenleaf @phoet)
24
+
25
+ ## 0.5.0 (January 28, 2013)
2
26
 
3
27
  Features:
4
28
 
@@ -11,11 +35,6 @@ Bugfixes:
11
35
  - Fixed negative number of non-finished rates (@philnash, #83)
12
36
  - Fixed behaviour of finished(:reset => false) (@philnash, #88)
13
37
  - Only take into consideration positive z-scores (@thomasmaas, #96)
14
-
15
- ## 0.4.7 (November 1, 2012)
16
-
17
- Bugfixes:
18
-
19
38
  - Amended ab_test method to raise ArgumentError if passed integers or symbols as
20
39
  alternatives (@buddhamagnet, #81)
21
40
 
data/Gemfile CHANGED
@@ -1,3 +1,3 @@
1
- source "http://rubygems.org"
1
+ source "https://rubygems.org"
2
2
 
3
3
  gemspec
@@ -82,7 +82,7 @@ end
82
82
 
83
83
  ## Usage
84
84
 
85
- To begin your ab test use the `ab_test` method, naming your experiment with the first argument and then the different variants which you wish to test on as the other arguments.
85
+ To begin your ab test use the `ab_test` method, naming your experiment with the first argument and then the different alternatives which you wish to test on as the other arguments.
86
86
 
87
87
  `ab_test` returns one of the alternatives, if a user has already seen that test they will get the same alternative as before, which you can use to split your code on.
88
88
 
@@ -92,7 +92,7 @@ It can be used to render different templates, show different text or any other c
92
92
 
93
93
  Example: View
94
94
 
95
- ```ruby
95
+ ```erb
96
96
  <% ab_test("login_button", "/images/button1.jpg", "/images/button2.jpg") do |button_file| %>
97
97
  <%= img_tag(button_file, :alt => "Login!") %>
98
98
  <% end %>
@@ -118,8 +118,8 @@ end
118
118
 
119
119
  Example: Conversion tracking (in a view)
120
120
 
121
- ```ruby
122
- Thanks for signing up, dude! <% finished("signup_page_redesign") >
121
+ ```erb
122
+ Thanks for signing up, dude! <% finished("signup_page_redesign") %>
123
123
  ```
124
124
 
125
125
  You can find more examples, tutorials and guides on the [wiki](https://github.com/andrew/split/wiki).
@@ -234,7 +234,7 @@ Then adding this to config/routes.rb
234
234
  mount Split::Dashboard, :at => 'split'
235
235
  ```
236
236
 
237
- You may want to password protect that page, you can do so with `Rack::Auth::Basic`
237
+ You may want to password protect that page, you can do so with `Rack::Auth::Basic` (in your split initializer file)
238
238
 
239
239
  ```ruby
240
240
  Split::Dashboard.use Rack::Auth::Basic do |username, password|
@@ -242,14 +242,16 @@ Split::Dashboard.use Rack::Auth::Basic do |username, password|
242
242
  end
243
243
  ```
244
244
 
245
+ ### Screenshot
246
+
247
+ ![split_screenshot](https://f.cloud.github.com/assets/78887/306152/99c64650-9670-11e2-93f8-197f49495d02.png)
248
+
245
249
  ## Configuration
246
250
 
247
251
  You can override the default configuration options of Split like so:
248
252
 
249
253
  ```ruby
250
254
  Split.configure do |config|
251
- config.robot_regex = /my_custom_robot_regex/
252
- config.ignore_ip_addresses << '81.19.48.130'
253
255
  config.db_failover = true # handle redis errors gracefully
254
256
  config.db_failover_on_db_error = proc{|error| Rails.logger.error(error.message) }
255
257
  config.allow_multiple_experiments = true
@@ -258,41 +260,84 @@ Split.configure do |config|
258
260
  end
259
261
  ```
260
262
 
263
+ ### Filtering
264
+
265
+ In most scenarios you don't want to have AB-Testing enabled for web spiders, robots or special groups of users.
266
+ Split provides functionality to filter this based on a predefined, extensible list of bots, IP-lists or custom exclude logic.
267
+
268
+ ```ruby
269
+ Split.configure do |config|
270
+ # bot config
271
+ config.robot_regex = /my_custom_robot_regex/ # or
272
+ config.bots['newbot'] = "Description for bot with 'newbot' user agent, which will be added to config.robot_regex for exclusion"
273
+
274
+ # IP config
275
+ config.ignore_ip_addresses << '81.19.48.130' # or regex: /81\.19\.48\.[0-9]+/
276
+
277
+ # or provide your own filter functionality, the default is proc{ |request| is_robot? || is_ignored_ip_address? }
278
+ config.ignore_filter = proc{ |request| CustomExcludeLogic.excludes?(request) }
279
+ end
280
+ ```
281
+
261
282
  ### Experiment configuration
262
283
 
263
284
  Instead of providing the experiment options inline, you can store them
264
- in a hash or a configuration file:
265
-
266
- Split.configure do |config|
267
- config.experiments = YAML.load_file "config/experiments.yml"
268
- end
269
-
270
- This hash can control your experiment's variants, weights, algorithm and if the
271
- experiment resets once finished:
272
-
273
- Split.configure do |config|
274
- config.experiments = {
275
- :my_first_experiment => {
276
- :variants => ["a", "b"],
277
- :resettable => false,
278
- },
279
- :my_second_experiment => {
280
- :algorithm => 'Split::Algorithms::Whiplash',
281
- :variants => [
282
- { :name => "a", :percent => 67 },
283
- { :name => "b", :percent => 33 },
284
- ]
285
- }
286
- }
287
- end
285
+ in a hash. This hash can control your experiment's alternatives, weights,
286
+ algorithm and if the experiment resets once finished:
287
+
288
+ ```ruby
289
+ Split.configure do |config|
290
+ config.experiments = {
291
+ "my_first_experiment" => {
292
+ :alternatives => ["a", "b"],
293
+ :resettable => false
294
+ },
295
+ "my_second_experiment" => {
296
+ :algorithm => 'Split::Algorithms::Whiplash',
297
+ :alternatives => [
298
+ { :name => "a", :percent => 67 },
299
+ { :name => "b", :percent => 33 }
300
+ ]
301
+ }
302
+ }
303
+ end
304
+ ```
305
+
306
+ You can also store your experiments in a YAML file:
307
+
308
+ ```ruby
309
+ Split.configure do |config|
310
+ config.experiments = YAML.load_file "config/experiments.yml"
311
+ end
312
+ ```
313
+
314
+ You can then define the YAML file like:
315
+
316
+ ```yaml
317
+ my_first_experiment:
318
+ alternatives:
319
+ - a
320
+ - b
321
+ my_second_experiment:
322
+ alternatives:
323
+ - name: a
324
+ percent: 67
325
+ - name: b
326
+ percent: 33
327
+ resettable: false
328
+ ```
288
329
 
289
330
  This simplifies the calls from your code:
290
331
 
291
- ab_test(:my_first_experiment)
332
+ ```ruby
333
+ ab_test("my_first_experiment")
334
+ ```
292
335
 
293
336
  and:
294
337
 
295
- finished(:my_first_experiment)
338
+ ```ruby
339
+ finished("my_first_experiment")
340
+ ```
296
341
 
297
342
  #### Metrics
298
343
 
@@ -301,28 +346,62 @@ those to complete multiple different experiments without adding more to
301
346
  your code. You can use the configuration hash to do this, thanks to
302
347
  the `:metric` option.
303
348
 
304
- Split.configure do |config|
305
- config.experiments = {
306
- :my_first_experiment => {
307
- :variants => ["a", "b"],
308
- :metric => :conversion,
309
- }
310
- }
311
- end
349
+ ```ruby
350
+ Split.configure do |config|
351
+ config.experiments = {
352
+ "my_first_experiment" => {
353
+ :alternatives => ["a", "b"],
354
+ :metric => :my_metric,
355
+ }
356
+ }
357
+ end
358
+ ```
312
359
 
313
360
  Your code may then track a completion using the metric instead of
314
361
  the experiment name:
315
362
 
316
- finished(:conversion)
317
-
318
- You can also create a new metric by instantiating and saving a new Metric object.
363
+ ```ruby
364
+ finished(:my_metric)
365
+ ```
366
+
367
+ You can also create a new metric by instantiating and saving a new Metric object.
368
+
369
+ ```ruby
370
+ Split::Metric.new(:my_metric)
371
+ Split::Metric.save
372
+ ```
373
+
374
+ #### Goals
319
375
 
320
- Split::Metric.new(:conversion)
321
- Split::Metric.save
376
+ You might wish to allow an experiment to have multiple, distinguishable goals.
377
+ The API to define goals for an experiment is this:
378
+
379
+ ```ruby
380
+ ab_test({"link_color" => ["purchase", "refund"]}, "red", "blue")
381
+ ```
382
+
383
+ or you can you can define them in a configuration file:
384
+
385
+ ```ruby
386
+ Split.configure do |config|
387
+ config.experiments = {
388
+ "link_color" => {
389
+ :alternatives => ["red", "blue"],
390
+ :goals => ["purchase", "refund"]
391
+ }
392
+ }
393
+ end
394
+ ```
395
+
396
+ To complete a goal conversion, you do it like:
397
+
398
+ ```ruby
399
+ finished("link_color" => "purchase")
400
+ ```
322
401
 
323
402
  ### DB failover solution
324
403
 
325
- Due to the fact that Redis has no autom. failover mechanism, it's
404
+ Due to the fact that Redis has no automatic failover mechanism, it's
326
405
  possible to switch on the `db_failover` config option, so that `ab_test`
327
406
  and `finished` will not crash in case of a db failure. `ab_test` always
328
407
  delivers alternative A (the first one) in that case.
@@ -373,7 +452,7 @@ If you're running multiple, separate instances of Split you may want
373
452
  to namespace the keyspaces so they do not overlap. This is not unlike
374
453
  the approach taken by many memcached clients.
375
454
 
376
- This feature is provided by the [redis-namespace][rs] library, which
455
+ This feature is provided by the [redis-namespace](https://github.com/defunkt/redis-namespace) library, which
377
456
  Split uses by default to separate the keys it manages from other keys
378
457
  in your Redis server.
379
458
 
@@ -388,7 +467,7 @@ is configured.
388
467
 
389
468
  ## Outside of a Web Session
390
469
 
391
- Split provides the Helper module to facilitate running experiments inside web sessions.
470
+ Split provides the Helper module to facilitate running experiments inside web sessions.
392
471
 
393
472
  Alternatively, you can access the underlying Metric, Trial, Experiment and Alternative objects to
394
473
  conduct experiments that are not tied to a web session.
@@ -412,16 +491,17 @@ end
412
491
 
413
492
  ## Algorithms
414
493
 
415
- By default, Split ships with an algorithm that randomly selects from possible alternatives for a traditional a/b test.
494
+ By default, Split ships with an algorithm that randomly selects from possible alternatives for a traditional a/b test.
416
495
 
417
- An implementation of a bandit algorithm is also provided.
496
+ An implementation of a bandit algorithm is also provided.
418
497
 
419
- Users may also write their own algorithms. The default algorithm may be specified globally in the configuration file, or on a per experiment basis using the experiments hash of the configuration file.
498
+ Users may also write their own algorithms. The default algorithm may be specified globally in the configuration file, or on a per experiment basis using the experiments hash of the configuration file.
420
499
 
421
500
  ## Extensions
422
501
 
423
502
  - [Split::Export](http://github.com/andrew/split-export) - easily export ab test data out of Split
424
503
  - [Split::Analytics](http://github.com/andrew/split-analytics) - push test data to google analytics
504
+ - [Split::Mongoid](https://github.com/MongoHQ/split-mongoid) - store data in mongoid instead of redis
425
505
 
426
506
  ## Screencast
427
507
 
@@ -436,6 +516,11 @@ Special thanks to the following people for submitting patches:
436
516
  * Andrew Appleton
437
517
  * Phil Nash
438
518
  * Dave Goodchild
519
+ * Ian Young
520
+ * Nathan Woodhull
521
+ * Ville Lautanala
522
+ * Liu Jin
523
+ * Peter Schröder
439
524
 
440
525
  ## Development
441
526
 
@@ -48,7 +48,7 @@ module Split
48
48
  #
49
49
  # @example
50
50
  # Split.configure do |config|
51
- # config.ignore_ips = '192.168.2.1'
51
+ # config.ignore_ip_addresses = '192.168.2.1'
52
52
  # end
53
53
  def configure
54
54
  self.configuration ||= Configuration.new
@@ -19,50 +19,71 @@ module Split
19
19
  name
20
20
  end
21
21
 
22
+ def goals
23
+ self.experiment.goals
24
+ end
25
+
22
26
  def participant_count
23
- @participant_count ||= Split.redis.hget(key, 'participant_count').to_i
27
+ Split.redis.hget(key, 'participant_count').to_i
24
28
  end
25
29
 
26
30
  def participant_count=(count)
27
- @participant_count = count
28
31
  Split.redis.hset(key, 'participant_count', count.to_i)
29
32
  end
30
33
 
31
- def completed_count
32
- @completed_count ||= Split.redis.hget(key, 'completed_count').to_i
34
+ def completed_count(goal = nil)
35
+ field = set_field(goal)
36
+ Split.redis.hget(key, field).to_i
33
37
  end
34
-
38
+
39
+ def all_completed_count
40
+ if goals.empty?
41
+ completed_count
42
+ else
43
+ goals.inject(completed_count) do |sum, g|
44
+ sum + completed_count(g)
45
+ end
46
+ end
47
+ end
48
+
35
49
  def unfinished_count
36
- participant_count - completed_count
50
+ participant_count - all_completed_count
51
+ end
52
+
53
+ def set_field(goal)
54
+ field = "completed_count"
55
+ field += ":" + goal unless goal.nil?
56
+ return field
37
57
  end
38
58
 
39
- def completed_count=(count)
40
- @completed_count = count
41
- Split.redis.hset(key, 'completed_count', count.to_i)
59
+ def set_completed_count (count, goal = nil)
60
+ field = set_field(goal)
61
+ Split.redis.hset(key, field, count.to_i)
42
62
  end
43
63
 
44
64
  def increment_participation
45
- @participant_count = Split.redis.hincrby key, 'participant_count', 1
65
+ Split.redis.hincrby key, 'participant_count', 1
46
66
  end
47
67
 
48
- def increment_completion
49
- @completed_count = Split.redis.hincrby key, 'completed_count', 1
68
+ def increment_completion(goal = nil)
69
+ field = set_field(goal)
70
+ Split.redis.hincrby(key, field, 1)
50
71
  end
51
72
 
52
73
  def control?
53
74
  experiment.control.name == self.name
54
75
  end
55
76
 
56
- def conversion_rate
77
+ def conversion_rate(goal = nil)
57
78
  return 0 if participant_count.zero?
58
- (completed_count.to_f/participant_count.to_f)
79
+ (completed_count(goal).to_f)/participant_count.to_f
59
80
  end
60
81
 
61
82
  def experiment
62
83
  Split::Experiment.find(experiment_name)
63
84
  end
64
85
 
65
- def z_score
86
+ def z_score(goal = nil)
66
87
  # CTR_E = the CTR within the experiment split
67
88
  # CTR_C = the CTR within the control split
68
89
  # E = the number of impressions within the experiment split
@@ -74,8 +95,9 @@ module Split
74
95
 
75
96
  return 'N/A' if control.name == alternative.name
76
97
 
77
- ctr_e = alternative.conversion_rate
78
- ctr_c = control.conversion_rate
98
+ ctr_e = alternative.conversion_rate(goal)
99
+ ctr_c = control.conversion_rate(goal)
100
+
79
101
 
80
102
  e = alternative.participant_count
81
103
  c = control.participant_count
@@ -92,28 +114,34 @@ module Split
92
114
  Split.redis.hsetnx key, 'completed_count', 0
93
115
  end
94
116
 
117
+ def validate!
118
+ unless String === @name || hash_with_correct_values?(@name)
119
+ raise ArgumentError, 'Alternative must be a string'
120
+ end
121
+ end
122
+
95
123
  def reset
96
- @participant_count = nil
97
- @completed_count = nil
98
124
  Split.redis.hmset key, 'participant_count', 0, 'completed_count', 0
125
+ unless goals.empty?
126
+ goals.each do |g|
127
+ field = "completed_count:#{g}"
128
+ Split.redis.hset key, field, 0
129
+ end
130
+ end
99
131
  end
100
132
 
101
133
  def delete
102
134
  Split.redis.del(key)
103
135
  end
104
136
 
105
- def self.valid?(name)
106
- String === name || hash_with_correct_values?(name)
107
- end
137
+ private
108
138
 
109
- def self.hash_with_correct_values?(name)
139
+ def hash_with_correct_values?(name)
110
140
  Hash === name && String === name.keys.first && Float(name.values.first) rescue false
111
141
  end
112
142
 
113
- private
114
-
115
143
  def key
116
144
  "#{experiment_name}:#{name}"
117
145
  end
118
146
  end
119
- end
147
+ end