split 1.1.0 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ea348c09640ba8869242588929d2d337856fc834
-  data.tar.gz: 0bb555acfabcaf683051058097da01b803c56a50
+  metadata.gz: acc9cc61f2fbaacd23c4101ce3533baad6fc2f5e
+  data.tar.gz: 0f1608e29221a58215ee240ac01ac921256e8466
 SHA512:
-  metadata.gz: f411a5636ee2fcc0c296f2e3e1e24515af7524be73eec1fe8bd86998564eb61fcc60419d52dc02588a92ac85ecb28c6525e9a007cc5607efe1942a56c1fa3262
-  data.tar.gz: 6ff0d6a3be6295653be81d34e0c2479c2d38bd8e1ac6a2fb0aa20e5796329add32cf465b884d5ede25ba9121343ae888de62e78f9aff5e468604f635c2efabb6
+  metadata.gz: 5e4cd8b3e649929c5c3a9f406a8a587c7cc23121146f0c0002ff05d6c86b7c61e508a05acab5d70172864d73f4c544263e72d1476a8d5afe134deb34b6012172
+  data.tar.gz: f9be6dc468f3fe7941e07b3734d1fae28a4a05209501cb093769d8f14001ec596d5b3627f808444523b303fe4d44de0946a328e04a1b9afe64434ba5c54c0493

data/CHANGELOG.md

@@ -1,3 +1,14 @@
+## 1.2.0 (January 24th, 2015)
+
+Features
+
+  - Configure redis using environment variables if available (@saratovsource , #293)
+  - Store metadata on experiment configuration (@dekz, #291)
+
+Bugfixes:
+
+ - Revert the Trial#complete! public API to support noargs (@dekz, #292)
+
 ## 1.1.0 (January 9th, 2015)
 
 Features:

data/README.md

@@ -119,7 +119,7 @@ You can find more examples, tutorials and guides on the [wiki](https://github.co
 
 ## Statistical Validity
 
-Split has two options for you to use to determine which alternative is the best. 
+Split has two options for you to use to determine which alternative is the best.
 
 The first option (default on the dashboard) uses a z test (n>30) for the difference between your control and alternative conversion rates to calculate statistical significance. This test will tell you whether an alternative is better or worse than your control, but it will not distinguish between which alternative is the best in an experiment with multiple alternatives. Split will only tell you if your experiment is 90%, 95%, or 99% significant, and this test only works if you have more than 30 participants and 5 conversions for each branch.
 
@@ -364,6 +364,8 @@ Split.configure do |config|
 end
 ```
 
+You can set different Redis host via environment variable ```REDIS_URL```.
+
 ### Filtering
 
 In most scenarios you don't want to have AB-Testing enabled for web spiders, robots or special groups of users.
@@ -443,6 +445,28 @@ and:
 finished("my_first_experiment")
 ```
 
+You can also add meta data for each experiment, very useful when you need more than an alternative name to change behaviour:
+
+```yaml
+my_first_experiment:
+  alternatives:
+    - a
+    - b
+    meta:
+      a:
+        text: "Have a fantastic day"
+        b:
+          text: "Don't get hit by a bus"
+```
+
+This allows for some advanced experiment configuration using methods like:
+
+```ruby
+trial.alternative.name # => "a"
+
+trial.metadata['text'] # => "Have a fantastic day"
+```
+
 #### Metrics
 
 You might wish to track generic metrics, such as conversions, and use
@@ -503,7 +527,7 @@ To complete a goal conversion, you do it like:
 finished("link_color" => "purchase")
 ```
 
-**NOTE:** This does not mean that a single experiment can have/complete progressive goals. 
+**NOTE:** This does not mean that a single experiment can have/complete progressive goals.
 
 **Good Example**: Test if listing Plan A first result in more conversions to Plan A (goal: "plana_conversion") or Plan B (goal: "planb_conversion").
 
@@ -549,11 +573,8 @@ production: redis1.example.com:6379
 And our initializer:
 
 ```ruby
-rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
-rails_env = ENV['RAILS_ENV'] || 'development'
-
-split_config = YAML.load_file(rails_root + '/config/split.yml')
-Split.redis = split_config[rails_env]
+split_config = YAML.load_file(Rails.root.join('config', 'split.yml'))
+Split.redis = split_config[Rails.env]
 ```
 
 ## Namespaces
@@ -601,11 +622,11 @@ end
 
 ## Algorithms
 
-By default, Split ships with `Split::Algorithms::WeightedSample` that randomly selects from possible alternatives for a traditional a/b test. 
+By default, Split ships with `Split::Algorithms::WeightedSample` that randomly selects from possible alternatives for a traditional a/b test.
 It is possible to specify static weights to favor certain alternatives.
 
-`Split::Algorithms::Whiplash` is an implementation of a [multi-armed bandit algorithm](http://stevehanov.ca/blog/index.php?id=132). 
-This algorithm will automatically weight the alternatives based on their relative performance, 
+`Split::Algorithms::Whiplash` is an implementation of a [multi-armed bandit algorithm](http://stevehanov.ca/blog/index.php?id=132).
+This algorithm will automatically weight the alternatives based on their relative performance,
 choosing the better-performing ones more often as trials are completed.
 
 Users may also write their own algorithms. The default algorithm may be specified globally in the configuration file, or on a per experiment basis using the experiments hash of the configuration file.

data/lib/split.rb

@@ -53,7 +53,7 @@ module Split
   # create a new one.
   def redis
     return @redis if @redis
-    self.redis = 'localhost:6379'
+    self.redis = ENV.fetch('REDIS_URL', 'localhost:6379')
     self.redis
   end
 

data/lib/split/configuration.rb

@@ -141,6 +141,10 @@ module Split
             experiment_config[experiment_name.to_sym][:goals] = goals
           end
 
+          if metadata = value_for(settings, :metadata)
+            experiment_config[experiment_name.to_sym][:metadata] = metadata
+          end
+
           if (resettable = value_for(settings, :resettable)) != nil
             experiment_config[experiment_name.to_sym][:resettable] = resettable
           end

data/lib/split/experiment.rb

@@ -6,6 +6,7 @@ module Split
     attr_accessor :goals
     attr_accessor :alternatives
     attr_accessor :alternative_probabilities
+    attr_accessor :metadata
 
     DEFAULT_OPTIONS = {
       :resettable => true
@@ -22,6 +23,7 @@ module Split
         set_alternatives_and_options(
           alternatives: load_alternatives_from_configuration,
           goals: load_goals_from_configuration,
+          metadata: load_metadata_from_configuration,
           resettable: exp_config[:resettable],
           algorithm: exp_config[:algorithm]
         )
@@ -29,6 +31,7 @@ module Split
         set_alternatives_and_options(
           alternatives: alternatives,
           goals: options[:goals],
+          metadata: options[:metadata],
           resettable: options[:resettable],
           algorithm: options[:algorithm]
         )
@@ -40,6 +43,7 @@ module Split
       self.goals = options[:goals]
       self.resettable = options[:resettable]
       self.algorithm = options[:algorithm]
+      self.metadata = options[:metadata]
     end
 
     def extract_alternatives_from_options(options)
@@ -56,6 +60,7 @@ module Split
         if exp_config
           alts = load_alternatives_from_configuration
           options[:goals] = load_goals_from_configuration
+          options[:metadata] = load_metadata_from_configuration
           options[:resettable] = exp_config[:resettable]
           options[:algorithm] = exp_config[:algorithm]
         end
@@ -79,16 +84,20 @@ module Split
         start unless Split.configuration.start_manually
         @alternatives.reverse.each {|a| Split.redis.lpush(name, a.name)}
         @goals.reverse.each {|a| Split.redis.lpush(goals_key, a)} unless @goals.nil?
+        Split.redis.set(metadata_key, @metadata.to_json) unless @metadata.nil?
       else
         existing_alternatives = load_alternatives_from_redis
         existing_goals = load_goals_from_redis
-        unless existing_alternatives == @alternatives.map(&:name) && existing_goals == @goals
+        existing_metadata = load_metadata_from_redis
+        unless existing_alternatives == @alternatives.map(&:name) && existing_goals == @goals && existing_metadata == @metadata
           reset
           @alternatives.each(&:delete)
           delete_goals
+          delete_metadata
           Split.redis.del(@name)
           @alternatives.reverse.each {|a| Split.redis.lpush(name, a.name)}
           @goals.reverse.each {|a| Split.redis.lpush(goals_key, a)} unless @goals.nil?
+          Split.redis.set(metadata_key, @metadata.to_json) unless @metadata.nil?
         end
       end
 
@@ -221,6 +230,10 @@ module Split
       "#{key}:finished"
     end
 
+    def metadata_key
+      "#{name}:metadata"
+    end
+
     def resettable?
       resettable
     end
@@ -238,6 +251,7 @@ module Split
       Split.redis.srem(:experiments, name)
       Split.redis.del(name)
       delete_goals
+      delete_metadata
       Split.configuration.on_experiment_delete.call(self)
       increment_version
     end
@@ -246,12 +260,17 @@ module Split
       Split.redis.del(goals_key)
     end
 
+    def delete_metadata
+      Split.redis.del(metadata_key)
+    end
+
     def load_from_redis
       exp_config = Split.redis.hgetall(experiment_config_key)
       self.resettable = exp_config['resettable']
       self.algorithm = exp_config['algorithm']
       self.alternatives = load_alternatives_from_redis
       self.goals = load_goals_from_redis
+      self.metadata = load_metadata_from_redis
     end
 
     def calc_winning_alternatives
@@ -388,6 +407,10 @@ module Split
       "experiment_configurations/#{@name}"
     end
 
+    def load_metadata_from_configuration
+      metadata = Split.configuration.experiment_for(@name)[:metadata]
+    end
+
     def load_goals_from_configuration
       goals = Split.configuration.experiment_for(@name)[:goals]
       if goals.nil?
@@ -401,6 +424,11 @@ module Split
       Split.redis.lrange(goals_key, 0, -1)
     end
 
+    def load_metadata_from_redis
+      meta = Split.redis.get(metadata_key)
+      JSON.parse(meta) unless meta.nil?
+    end
+
     def load_alternatives_from_configuration
       alts = Split.configuration.experiment_for(@name)[:alternatives]
       raise ArgumentError, "Experiment configuration is missing :alternatives array" unless alts

data/lib/split/trial.rb

@@ -1,10 +1,12 @@
 module Split
   class Trial
     attr_accessor :experiment
+    attr_accessor :metadata
 
     def initialize(attrs = {})
       self.experiment   = attrs.delete(:experiment)
       self.alternative  = attrs.delete(:alternative)
+      self.metadata  = attrs.delete(:metadata)
 
       @user             = attrs.delete(:user)
       @options          = attrs
@@ -12,6 +14,10 @@ module Split
       @alternative_choosen = false
     end
 
+    def metadata
+      @metadata ||= experiment.metadata[alternative.name]
+    end
+
     def alternative
       @alternative ||=  if @experiment.has_winner?
                           @experiment.winner
@@ -26,14 +32,12 @@ module Split
       end
     end
 
-    def complete!(goals, context = nil)
-      goals = goals || []
-
+    def complete!(goals=[], context = nil)
       if alternative
-        if goals.empty?
+        if Array(goals).empty?
           alternative.increment_completion
         else
-          goals.each {|g| alternative.increment_completion(g) }
+          Array(goals).each {|g| alternative.increment_completion(g) }
         end
 
         context.send(Split.configuration.on_trial_complete, self) \

data/lib/split/version.rb

@@ -1,6 +1,6 @@
 module Split
   MAJOR   = 1
-  MINOR   = 1
+  MINOR   = 2
   PATCH   = 0
   VERSION = [MAJOR, MINOR, PATCH].join('.')
 end

data/spec/configuration_spec.rb

@@ -90,6 +90,36 @@ describe Split::Configuration do
         end
       end
 
+      context "in a configuration with metadata" do
+        before do
+          experiments_yaml = <<-eos
+            my_experiment:
+              alternatives:
+                - name: Control Opt
+                  percent: 67
+                - name: Alt One
+                  percent: 10
+                - name: Alt Two
+                  percent: 23
+              metadata:
+                Control Opt:
+                  text: 'Control Option'
+                Alt One:
+                  text: 'Alternative One'
+                Alt Two:
+                  text: 'Alternative Two'
+              resettable: false
+            eos
+          @config.experiments = YAML.load(experiments_yaml)
+        end
+
+        it 'should have metadata on the experiment' do
+          meta = @config.normalized_experiments[:my_experiment][:metadata]
+          expect(meta).to_not be nil
+          expect(meta['Control Opt']['text']).to eq('Control Option')
+        end
+      end
+
       context "in a complex configuration" do
         before do
           experiments_yaml = <<-eos
@@ -120,6 +150,7 @@ describe Split::Configuration do
           expect(@config.metrics).not_to be_nil
           expect(@config.metrics.keys).to eq([:my_metric])
         end
+
       end
     end
 

data/spec/experiment_spec.rb

@@ -147,6 +147,29 @@ describe Split::Experiment do
 
     end
 
+    describe '#metadata' do
+      let(:experiment) { Split::Experiment.new('basket_text', :alternatives => ['Basket', "Cart"], :algorithm => Split::Algorithms::Whiplash, :metadata => meta) }
+      context 'simple hash' do
+        let(:meta) {  { 'basket' => 'a', 'cart' => 'b' } }
+        it "should persist metadata in redis" do
+          experiment.save
+          e = Split::ExperimentCatalog.find('basket_text')
+          expect(e).to eq(experiment)
+          expect(e.metadata).to eq(meta)
+        end
+      end
+
+      context 'nested hash' do
+        let(:meta) {  { 'basket' => { 'one' => 'two' }, 'cart' => 'b' } }
+        it "should persist metadata in redis" do
+          experiment.save
+          e = Split::ExperimentCatalog.find('basket_text')
+          expect(e).to eq(experiment)
+          expect(e.metadata).to eq(meta)
+        end
+      end
+    end
+
     it "should persist algorithm in redis" do
       experiment = Split::Experiment.new('basket_text', :alternatives => ['Basket', "Cart"], :algorithm => Split::Algorithms::Whiplash)
       experiment.save

data/spec/trial_spec.rb

@@ -33,6 +33,27 @@ describe Split::Trial do
     end
   end
 
+  describe "metadata" do
+    let(:alternatives) { ['basket', 'cart'] }
+    let(:metadata) { Hash[alternatives.map { |k| [k, "Metadata for #{k}"] }] }
+    let(:experiment) do
+      Split::Experiment.new('basket_text', :alternatives => alternatives, :metadata => metadata).save
+    end
+
+    it 'has metadata on each trial' do
+      trial = Split::Trial.new(:experiment => experiment, :user => user, :metadata => metadata['cart'],
+                               :override => 'cart')
+      expect(trial.metadata).to eq(metadata['cart'])
+    end
+
+    it 'has metadata on each trial from the experiment' do
+      trial = Split::Trial.new(:experiment => experiment, :user => user)
+      trial.choose!
+      expect(trial.metadata).to eq(metadata[trial.alternative.name])
+      expect(trial.metadata).to match /#{trial.alternative.name}/
+    end
+  end
+
   describe "#choose!" do
     def expect_alternative(trial, alternative_name)
       3.times do
@@ -109,6 +130,42 @@ describe Split::Trial do
       end
     end
 
+    describe "#complete!" do
+      let(:trial) { Split::Trial.new(:user => user, :experiment => experiment) }
+      context 'when there are no goals' do
+        it 'should complete the trial' do
+          trial.choose!
+          old_completed_count = trial.alternative.completed_count
+          trial.complete!
+          expect(trial.alternative.completed_count).to be(old_completed_count+1)
+        end
+      end
+
+      context 'when there are many goals' do
+        let(:goals) { ['first', 'second'] }
+        let(:trial) { Split::Trial.new(:user => user, :experiment => experiment, :goals => goals) }
+        shared_examples_for "goal completion" do
+          it 'should not complete the trial' do
+            trial.choose!
+            old_completed_count = trial.alternative.completed_count
+            trial.complete!(goal)
+            expect(trial.alternative.completed_count).to_not be(old_completed_count+1)
+          end
+        end
+
+        describe 'Array of Goals' do
+          let(:goal) { [goals.first] }
+          it_behaves_like 'goal completion'
+        end
+
+        describe 'String of Goal' do
+          let(:goal) { goals.first }
+          it_behaves_like 'goal completion'
+        end
+
+      end
+    end
+
     describe "alternative recording" do
       before(:each) { Split.configuration.store_override = false }
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: split
 version: !ruby/object:Gem::Version
-  version: 1.1.0
+  version: 1.2.0
 platform: ruby
 authors:
 - Andrew Nesbitt
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-01-09 00:00:00.000000000 Z
+date: 2015-01-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: redis