mountain-goat 0.1.8 → 1.0.0

data/README.md

@@ -1,25 +1,33 @@
 # Mountain-Goat
 
-Embed a high-quality analytics platform in your application in minutes.  Gain important A/B test business insights, and view the results and analytics in real time. 
+Embed a professional analytics and reporting platform into your application in minutes.  Gain important business insights through bandit (automated a/b) testing, view the results and analytics in real time, and get delivered daily or weekly usage reports. 
+
+Note: For updating from version < 1.0.0, please read the **Upgrade** section below.
 
 Add simple hooks in your code to display a/b metrics
 
-     <%= metric_variant(:homescreen_text, :user_signup, "Welcome here") %>
+     <%= bd(:homescreen_text, "Welcome here") %>
   
-This creates a database entry to a/b test "homescreen_text" against a goal "user signup".  Visit "http://yourdomain.com/mg" and you can add / adjust options (variants) for this text.  When a user converts on this goal, you run the following code.
+This creates a database entry for your a/b test "homescreen_text".  Visit "http://yourdomain.com/mg" and you can add / adjust options (variants) for this text.  When a user converts on a goal, you run the following code.
   
+  	 #e.g. users_controller.rb
      def create
        record_conversion(:user_signup)
        ...
      end
 
-This will track a conversion not only for the goal, but for the variant of "homescreen_text" (and any other associated metrics) that the user was served when he came to the home-page.
+This will track a conversion not only for the goal, but for the variant of "homescreen_text" that the user was served when your user came to the home-page.
+
+Bandit testing (see [Wikipedia - Multi-armed Bandit](http://en.wikipedia.org/wiki/Multi-armed_bandit) automatically converges on the variant that achieves the highest success through exploration each variant.  You essentially get a Bayesian solution with no hassle! 
 
-The best part?  The mountain-goat admin console is located on your server and you can view and analyze your data in real time.
+The best part?  The Mountain-Goat Administrative console is located on your server and you can view and analyze your data in real time (or through setting up usage report emails).
 
  - See which metric variants are working and not working ("Cowabunga!" did 120% better than "Enter here", but 10% worse than "Do it rockapella!")
- - A/B testing?  How about A/B/C/D/E testing?
- - Visually analyze the your metric variants; change them on the fly, adding new ones.
+ - Get daily / weekly / whenever emails delivered to your inbox, full of the data you crave (e.g. Signs up by day / comments by referring domain )
+ - Similar to Bayesian learning, Multi-armed bandit solutions will automatically deliver the highest performing variants
+ - This is done while still achieving minimal (logarithmically decreasing) regret (showing of poorer performing variants)
+ - A/B testing?  How about A/B/C/D/E testing?  Add as many variants as you like.
+ - Visually analyze the your metric variants; change them on the fly, adding new ones or kicking out poor performers.
  - Watch goal conversions in real-time with live-action console (grab the popcorn and watch how your users "sign up" and "view items" and ...)
  - You can do more than change text, "switch variants" let you enter arbitrary ruby code, change the control of your site ("Do my users have to sign-in before commenting?  Let's test!")
  - Track goals with meta data (record_conversion(:user_signup, :referrer => request.env['HTTP_REFERER']))
@@ -30,13 +38,23 @@ The best part?  The mountain-goat admin console is located on your server and yo
 
 For more information, read my blog post on how mountain goat quickly accomplishes a/b testing: http://blog.drawn.to/how-we-do-ab-testing
 
+## Upgrade from < 1.0.0
+
+If you are upgrading from Mountain Goat < 1.0.0, please run the following command (please overwrite mountain-goat.yml when prompted):
+
+     ./script/generate mg --update=1.0.0
+     
+     rake db:migrate
+     
+This will install new migrations necessary for version 1.0.0.
+ 
 ## Install
 
 ### Mountain Goat gem
 
      gem install mountain-goat
 
-### Mountain Goat configration
+### Mountain Goat configuration
     
 Next run generator to create config file and necessary database migration (optionally pass in --password=my_mg_password)
      
@@ -45,7 +63,8 @@ Next run generator to create config file and necessary database migration (optio
 This will generate
 
      /config/mountain-goat.yml (for storing a password to access mountain-goat)
-     /db/migrate/xxx_create_mountain_goat_tables.rb (necessary databae migrations to store mg data)
+     /db/migrate/xxx_create_mountain_goat_tables.rb (necessary database migrations to store mg data)
+     /lib/tasks/mountain_goat_reports.rake (add tasks for report delivery)
 
 Modify /config/mountain-goat.yml to setup a password for each environment of your product
 
@@ -66,27 +85,51 @@ Mountain Goat hinges around three core concepts:
 	
 After you set up your database with some mountain-goat tables, the code will handle populating these tables for you.  In your code, you can start A/B testing immediately.
 	
-     <h2><%= metric_variant(:banner_on_store_front, :user_purchases_coffee, "Now arsenic and gluten-free") %></h2>
+     <h2><%= bd(:banner_on_store_front, "Now arsenic and gluten-free") %></h2>
 	
-The metric_variant function (or, mv for short) takes three parameters:
+The bandit (bd) function takes two parameters:
 	
-     mv(metric_name, convert_name, default)
+     bd(metric_name, default)
 	
-This will automatically create a metric and conversion and populate a metric variant with the default value.  Easy, eh?
-	
-From here, you can go into the mountain goat admin center and add new metric variants to fit your need.  It's all built-in to *your* application, in house.  
+This will automatically create a metric and populate a metric variant with the default value.  Easy, eh?
+
+From here, you can go into the mountain goat admin center and add new metric variants to fit your need.  It's all built into *your* application, in house. 
 
      http://{your_rails_app}/mg  (e.g. if you're at railsrocks.com, then visit http://railsrocks.com/mg)
 
 The other important code you'll need to implement is to tell the system when a goal is achieved.
 	
      def purchase #in coffees_controller.rb
-       record_conversion(:user_purchases_coffee)
+       rw(:user_purchases_coffee, 10) #10 "points"
        ...
      end
 	
-This will go in and record a conversion (a "rally") for a user purchasing coffee.  Further, it will track a hit for any metric-variants served to that user that relate to this goal.  For example "Chuck Norris works here" might get one point.  You will see which metrics lead to a conversion; this is the core of A/B testing.
-	
+This will go in and record a conversion (a "rally") for a user purchasing coffee.  Further, it will track a hit for any metric-variants served to that user.  For example "Chuck Norris works here" might get reward points.  You will see which metrics lead to a conversion; this is the core of A/B and bandit testing.
+
+## Bandit Testing
+
+Why points?  Bandit testing is about maximizing a return.  The original idea is if I have a slot machine with 30 handles each with different payouts, how do I maximize my return?  You pull the arm that has given you the best pay out so far, but occasionally pull other arms to make sure there's not something better.  Mountain-Goat's Bandit system will do the same for you.
+
+The most common solution to the multi-armed bandit solution is epsilon-greedy (documented [here](http://www.cs.nyu.edu/~mohri/pub/bandit.pdf)).
+
+     Given epsilon in [0, 1] #commonly 0.05 - 0.15
+     
+     If any variant has never been pulled
+       Pull that variant
+     Otherwise
+     
+       If random(0..1) < epsilon
+         Pull random arm
+       Else
+         Pull winning arm #highest "reward"
+	   End
+
+This way, we most commonly explore the best arm (e.g. your best slogan) while occasionally trying other arms.
+
+A small variant on this is called epsilon-greedy-decreasing, which reduces epsilon each pull by 1/total_pulls.  Thus, after 100 pulls, epsilon = epsilon_0 / 100.  This is shown to have an optimal minimal expectation of regret.
+
+You can configure these options in `mountain-goat.yml`.
+
 ## Mountain Goat admin suite
 
 Navigate to /mg in your rails application (on your actual server instance) to reach the mountain-goat admin center.  Here, you can analyze / adjust your A/B tests.
@@ -110,7 +153,21 @@ Go to "Metrics" and visit a specific metric.  You'll see which metric variants a
 ###Rallies
 
 Rallies shows you what's going on, in real time.  You will see conversions (goals) being hit by your clients in real time.  Grab a bag of pop-corn and watch users struggle (or glide) across your site.  Add meta data to get further information.   This page automatically updates as new rallies come in.
-    
+
+###Reports
+
+In the Mountain Goat Administrative suite, you can add reports.  Reports will be delivered as emails with an attached pdf showing statistics about your product.  You'll need the following installed to use Mountain Goat Reports.
+
+     gem install pdfkit
+     gem install svg-graph
+
+You'll need [wkhtmltopdf](http://code.google.com/p/wkhtmltopdf/) installed.  Please configure the location of the executable in `mountain-goat.yml`.
+
+Finally, simply set up a cron task on your system when you would like your reports delivered.  E.g.
+
+     crontab -l | { cat; echo "45 6 * * * cd /path/to/project && /usr/bin/rake RAILS_ENV=production mg:deliver[daily] >> /path/to/project/log/cron.log 2>&1"; } | crontab -
+     crontab -l | { cat; echo "10 6 1 * * cd /path/to/project && /usr/bin/rake RAILS_ENV=production mg:deliver[monthly]  >> $app_root/current/log/cron.log 2>&1"; } | crontab -     
+     
 ## Advanced Features
 
 ### Meta data
@@ -135,19 +192,13 @@ Instead of just serving text, you can also serve flow control in Mountain Goat,
           discount = 0.90
         end
       
-        variant.whomp_whomp(0.5) do #reduce priority to 0.5 (default 1.0)
+        variant.whomp_whomp do
           discount = 0.0
         end
       end
 
-Mountain goat will automatically break those down into three cases (:ten_percent, :big_winner, :whomp_whomp) and serve them out at random to the user (with whomp_whomp half as likely to be served as the others).
-    
-### Priorities
+Mountain goat will automatically break those down into three cases (:ten_percent, :big_winner, :whomp_whomp) and serve them out at random to the user.
     
-You may want to test certain items with a lower serve rate (bold new slogans).  You can assign priorities to any metric variant.  The change of a given metric variant being shown is
-    
-     my priority / sum(all priorities for this metric)    
-
 ### Meta Options
 
 There is certain meta data that you may wish to collect for a number of different conversions.  For example, you may want to track ip-address so you can later pivot this column to find new / returning users.  To do this, add an initializer that calls MountainGoat.add_meta_option().
@@ -167,30 +218,50 @@ Then, when we track the conversion, you'll get meta-data for the user's ip-addre
 As mountain goat is a suite that is added into your project dynamically, the following routes and tables are added during setup:
 
 - Tables
-  * ci_metas (indexes: ci_metas_cmt_data_index, ci_metas_cmt_index, ci_metas_rally_index)
-  * convert_meta_types
-  * converts
-  * cs_metas (indexes: cs_metas_cmt_data_index, cs_metas_cmt_index, cs_metas_rally_index)
-  * metric_variants
-  * metrics
-  * rallies
-
-- Routes
-  * map.mg '/mg', :controller => :mountain_goat_converts, :action => :index
-  * map.mg_login '/mg/login', :controller => :mountain_goat, :action => :login
-  * map.mg_login_create '/mg/login/create', :controller => :mountain_goat, :action => :login_create
-  * map.resources :mountain_goat_metric_variants
-  * map.resources :mountain_goat_converts, :has_many => [ :mountain_goat_metrics, :mountain_goat_rallies ]
-  * map.resources :mountain_goat_metrics, :has_many => :mountain_goat_metric_variants
-  * map.resources :mountain_goat_rallies
-  * map.new_rallies '/mg/rallies/new', :controller => :mountain_goat_rallies, :action => :new_rallies 
-  * map.fresh_metrics '/fresh-metrics', :controller => :mountain_goat_metrics, :action => :fresh_metrics
-  * map.connect '/mg/public/:file', :controller => :mountain_goat, :action => :fetch
+  * mg_ci_metas (indexes: ci_metas_cmt_data_index, ci_metas_cmt_index, ci_metas_rally_index)
+  * mg_convert_meta_types
+  * mg_converts
+  * mg_cs_metas (indexes: cs_metas_cmt_data_index, cs_metas_cmt_index, cs_metas_rally_index)
+  * mg_metric_variants
+  * mg_metrics
+  * mg_rallies
+  * mg_report_items
+  * mg_reports
+
+- Routes (all namespaced mg)
+  * mg.mg '/mg', :controller => :converts, :action => :index, :path_prefix => ""
+  * mg.login '/login', :controller => :mountain_goat, :action => :login
+  * mg.login_create '/login/create', :controller => :mountain_goat, :action => :login_create
+  * mg.resources :metric_variants
+  * mg.resources :converts, :has_many => [ :rallies ]
+  * mg.resources :metrics, :has_many => :metric_variants
+  * mg.resources :rallies, :collection => { :new_rallies => :get }
+  * mg.resources :reports, :has_many => :report_items, :member => { :show_svg => :get }
+  * mg.resources :report_items, :member => { :destroy => :get, :update => :post }, :collection => { :get_extra => :get }
+  * mg.resources :playground, :collection => { :test => :get }
+  * mg.new_rallies '/rallies/new', :controller => :rallies, :action => :new_rallies 
+  * mg.fresh_metrics '/fresh-metrics', :controller => :metrics, :action => :fresh_metrics
+  * mg.connect '/public/:file', :controller => :mountain_goat, :action => :fetch
     
+- Models
+  * Mg::CiMeta - Integer-typed meta data for Rallies (e.g. 'Click Count')
+  * Mg::ConvertMetaType - Meta-types for Rallies
+  * Mg::Convert - Goals (e.g. 'Page View', 'User Sign-up')
+  * Mg::CsMeta - String-typed meta data for Rallies (e.g. 'Referring domain')
+  * Mg::MetricVariant - Variant for a/b testing (e.g. 'Come see our store!')
+  * Mg::Metric - Type to vary for a/b testing (e.g. 'Homescreen Text')
+  * Mg::Rally - Instance of a goal conversion (e.g. when a user clicks sign up)
+  * Mg::ReportItem - Item to show in a report (e.g. Sign ups by day)
+  * Mg::Report - Report to deliver (e.g. collection of user report items)
+  
+## Change log
+  1.0.0 - Changed from a/b testing to multi-armed bandit
+        - Added Mountain Goat Reporting
+        - Added extensive test cases for stability
+        
 ## TODO
  - Better documentation (rdocs)
- - Add namespacing to avoid conflicts
 
 ## Copyright
 
-Copyright (c) 2011 Geoffrey Hayes, drawn.to. Contact me, Geoff, <geoff@drawn.to> with any questions / ideas / enhacements.  See LICENSE for details.
+Copyright (c) 2011 Geoffrey Hayes, meloncard.com. Contact me, Geoff, <geoff@meloncard.com> with any questions / ideas / enhancements.  See LICENSE for details.

data/generators/mg/mg_generator.rb

@@ -1,9 +1,11 @@
 class MgGenerator < Rails::Generator::Base
   def add_options!(opt)
     opt.on('-p', '--password=password', String, "Your password to access Mountain Goat") { |v| options[:password] = v}
+    opt.on('-w', '--wkhtmltopdf=/path/to/dir', String, "Path to installation of wkhtmltopdf (optional)") { |v| options[:wkhtmltopdf] = v}
+    opt.on('-u', '--update=1.0.0', String, "If you have previously installed Mountain Goat, use to generate *update* tables.") { |v| options[:update] = v}
     puts <<-HELPFUL_INSTRUCTIONS
 
-      Mountain Goat is your home for A/B testing in-house.
+      Mountain Goat is your home for in-house bandit testing.
       
       We have installed your configuration file to config/mountain-goat.yml
       Please choose a password for each environment in this file.  Make
@@ -18,20 +20,26 @@ class MgGenerator < Rails::Generator::Base
       1) Choose an admin-password in config/mountain-goat.yml
       2) Run `rake db:migrate`
       3) Start your server and navigate to 'http://mydomain.com/mg'
-      4) Play around, read the docs, A/B test, enjoy. 
+      4) Play around, read the docs, bandit, enjoy. 
       
       HELPFUL_INSTRUCTIONS
   end
 
   def manifest
-    if options[:password].blank?
-      password = ""
-    else
-      password = options[:password]
-    end
+    password = options[:password] || ""
+    wkhtmltopdf = options[:wkhtmltopdf] || ""
+    update = options[:update]
+    
     record do |m|
-      #m.template 'mountain-goat.yml', 'config/mountain-goat.yml', :assigns => {:password => password}
-      m.migration_template 'create_mountain_goat_tables.rb', 'db/migrate', { :migration_file_name => "create_mountain_goat_tables" }
+      m.template 'mountain_goat_reports.rake', 'lib/tasks/mountain_goat_reports.rake'
+      m.template 'mountain-goat.yml', 'config/mountain-goat.yml', :assigns => { :password => password, :wkhtmltopdf => wkhtmltopdf }
+      
+      if !update.blank?
+        m.migration_template 'create_mountain_goat_tables.rb', 'db/migrate', { :migration_file_name => "create_mountain_goat_tables" }
+      elsif update == "1.0.0"
+        m.migration_template 'update_mountain_goat_tables.rb', 'db/migrate', { :migration_file_name => "update_mountain_goat_tables" }
+      end
+      
     end
   end
   

data/generators/mg/templates/create_mountain_goat_tables.rb

@@ -1,7 +1,7 @@
 class CreateMountainGoatTables < ActiveRecord::Migration
 
   def self.up
-    create_table "ci_metas", :force => true do |t|
+    create_table "mg_ci_metas", :force => true do |t|
       t.integer  "rally_id"
       t.integer  "convert_meta_type_id"
       t.integer  "data"
@@ -9,11 +9,11 @@ class CreateMountainGoatTables < ActiveRecord::Migration
       t.datetime "updated_at"
     end
   
-    add_index "ci_metas", ["convert_meta_type_id", "data"], :name => "ci_metas_cmt_data_index"
-    add_index "ci_metas", ["convert_meta_type_id"], :name => "ci_metas_cmt_index"
-    add_index "ci_metas", ["rally_id"], :name => "ci_metas_rally_index"
-    
-    create_table "convert_meta_types", :force => true do |t|
+    add_index "mg_ci_metas", ["convert_meta_type_id", "data"], :name => "ci_metas_cmt_data_index"
+    add_index "mg_ci_metas", ["convert_meta_type_id"], :name => "ci_metas_cmt_index"
+    add_index "mg_ci_metas", ["rally_id"], :name => "ci_metas_rally_index"
+  
+    create_table "mg_convert_meta_types", :force => true do |t|
       t.integer  "convert_id"
       t.string   "name"
       t.string   "var"
@@ -22,14 +22,16 @@ class CreateMountainGoatTables < ActiveRecord::Migration
       t.datetime "updated_at"
     end
   
-    create_table "converts", :force => true do |t|
+    create_table "mg_converts", :force => true do |t|
       t.string   "convert_type"
       t.datetime "created_at"
       t.datetime "updated_at"
       t.string   "name"
+      t.float    "reward",       :default => 1.0
+      t.datetime "deleted_at"
     end
   
-    create_table "cs_metas", :force => true do |t|
+    create_table "mg_cs_metas", :force => true do |t|
       t.integer  "rally_id"
       t.integer  "convert_meta_type_id"
       t.string   "data"
@@ -37,11 +39,11 @@ class CreateMountainGoatTables < ActiveRecord::Migration
       t.datetime "updated_at"
     end
   
-    add_index "cs_metas", ["convert_meta_type_id", "data"], :name => "cs_metas_cmt_data_index"
-    add_index "cs_metas", ["convert_meta_type_id"], :name => "cs_metas_cmt_index"
-    add_index "cs_metas", ["rally_id"], :name => "cs_metas_rally_index"
-    
-    create_table "metric_variants", :force => true do |t|
+    add_index "mg_cs_metas", ["convert_meta_type_id", "data"], :name => "cs_metas_cmt_data_index"
+    add_index "mg_cs_metas", ["convert_meta_type_id"], :name => "cs_metas_cmt_index"
+    add_index "mg_cs_metas", ["rally_id"], :name => "cs_metas_rally_index"
+  
+    create_table "mg_metric_variants", :force => true do |t|
       t.integer  "metric_id"
       t.text     "value"
       t.text     "opt1"
@@ -51,35 +53,69 @@ class CreateMountainGoatTables < ActiveRecord::Migration
       t.datetime "created_at"
       t.datetime "updated_at"
       t.string   "name"
-      t.float    "priority",    :default => 1.0, :null => false
       t.string   "switch_type"
+      t.float    "reward"
+      t.datetime "deleted_at"
     end
   
-    create_table "metrics", :force => true do |t|
+    create_table "mg_metrics", :force => true do |t|
       t.string   "metric_type"
       t.string   "title"
       t.datetime "created_at"
       t.datetime "updated_at"
-      t.integer  "convert_id"
       t.boolean  "tally_each_serve", :default => true
       t.boolean  "is_switch",        :default => false
+      t.datetime "deleted_at"
     end
-    
-    create_table "rallies", :force => true do |t|
+  
+    create_table "mg_rallies", :force => true do |t|
       t.integer  "convert_id"
       t.datetime "created_at"
       t.datetime "updated_at"
     end
+  
+    create_table "mg_report_items", :force => true do |t|
+      t.integer  "report_id"
+      t.integer  "reportable_id"
+      t.string   "reportable_type"
+      t.integer  "order"
+      t.datetime "created_at"
+      t.datetime "updated_at"
+      t.string   "pivot_type"
+      t.integer  "pivot_id"
+      t.string   "filter"
+      t.string   "meta_type"
+      t.integer  "meta_id"
+      t.string   "extra"
+      t.integer  "variant"
+      t.integer  "subvariant"
+      t.datetime "start"
+      t.datetime "end"
+      t.integer  "size"
+    end
+  
+    create_table "mg_reports", :force => true do |t|
+      t.string   "title"
+      t.text     "description"
+      t.datetime "created_at"
+      t.datetime "updated_at"
+      t.string   "delivery_set"
+      t.string   "recipients"
+      t.datetime "deleted_at"
+      t.string   "theme"
+    end
   end
 
   def self.down
-    drop_table :ci_metas
-    drop_table :convert_meta_types
-    drop_table :converts
-    drop_table :cs_metas
-    drop_table :metric_variants
-    drop_table :metrics
-    drop_table :rallies
+    drop_table :mg_ci_metas
+    drop_table :mg_convert_meta_types
+    drop_table :mg_converts
+    drop_table :mg_cs_metas
+    drop_table :mg_metric_variants
+    drop_table :mg_metrics
+    drop_table :mg_rallies
+    drop_table :mg_report_items
+    drop_table :mg_reports
   end
 
 end

data/generators/mg/templates/mg.rb

@@ -0,0 +1 @@
+#Dir["#{Gem.searcher.find('facebooker').full_gem_path}/lib/tasks/*.rake"].each { |ext| load ext }

data/generators/mg/templates/mountain-goat.yml

@@ -4,12 +4,31 @@
 #################
 
 # Add your access passwords for each environment
-# 
+#
+# Settings: epsilon for e-greedy strategy
+#           strategy: one of [ 'e-greedy', 'e-greedy-decreasing', 'a/b' ] 
+#           storage: ['cookies','session','none']
+
+# wkhtmltopdf must be installed to use Mountain Goat Reporting
+  # Please specify the executable for wkhtmltopdf below
+  # E.g. (Linux) /usr/local/bin/wkhtmltopdf  
+  #      (Windows) C:\Program Files (x86)\wkhtmltopdf\wkhtmltopdf.exe
+  # wkhtmltopdf can be installed from http://code.google.com/p/wkhtmltopdf/
+  
+settings:
+  epsilon: 0.1
+  strategy: e-greedy
+  storage: cookies
+   
 test:
-  password: <%= password %> 
+  password: <%= password %>
+  wkhtmltopdf: <%= wkhtmltopdf %> 
 development:
   password: <%= password %>
+  wkhtmltopdf: <%= wkhtmltopdf %>
 staging:
   password: <%= password %>
+  wkhtmltopdf: <%= wkhtmltopdf %>
 production:
-  password: <%= password %>
+  password: <%= password %>
+  wkhtmltopdf: <%= wkhtmltopdf %>