trailguide 0.1.31 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e756ddcc8e8a84fc537bbc83d97ee29ffac4fe3252c7d0a292b387cade57b611
-  data.tar.gz: 18a31337e3a0a93ab5f72b547f6c2e74d2994bc882ca8bc7b03a8e52a78a9790
+  metadata.gz: 251168ab3ed863718c22e548576f02345799e64af8be5bc956b726cb568e7435
+  data.tar.gz: ff02dc4669e1979cedecf2d465552ffab671ac3c3fb288cc6fa38adc8afd6bb0
 SHA512:
-  metadata.gz: 617d86a39bac52c3ab6faa4ca1052f4d968b2209bb2e9ba60a524cc5ad5484293986cb756bec6b98838e8a8e33c50581a3e599dd2a344e0b79194c45dfb327b7
-  data.tar.gz: 924d10df7b49b1ad9d666ceadcb85e20ce831cc23ddbfb480d26e3066ff0d7da13e3911f136cdbeec4bea6c95b41661b15a3a38175692e513a5133d06ffbee86
+  metadata.gz: 4ba3e7a1c0555990b50db9b580ba65430de92ca317228ebba2580b467288659d1e503db21b2a9d3f5da2efb4de0c3b6f6039bc6e7472b5b655421793c05d669b
+  data.tar.gz: b0be0281fd6d0ca3deab6b7690e171493cdca59c7128d26e278ff5e5bba642a7a3d78a6aeffb3b7f6a5af55a08f988e6661d3afd2ca99c0267300bd23d39d3b8

data/README.md

@@ -188,6 +188,12 @@ You can put these classes anywhere rails will autoload them (or require them you
 # app/experiments/my_complex_experiment.rb
 
 class MyComplexExperiment < TrailGuide::Experiment
+  # if you want to actually use this class as an experiment (like we do in this
+  # example), you must call `register!` in order to register it in the catalog.
+  #
+  # if you want to use your class as a base class, so other experiments can
+  # inherit from it, you should _not_ call `register!`
+  register!
 
   # all standard experiment config goes in the `configure` block
   configure do |config|
@@ -764,24 +770,24 @@ When you define one or more named goals for an experiment, you must pass one of
 trailguide.convert(:button_color, :signed_up)
 ```
 
-## Metrics
+## Groups
 
-If you have multiple experiments that share a relevant conversion point, you can configure them with a shared metric. This allows you to reference and convert multiple experiments at once using that shared metric, and only experiments in which participants have been enrolled will be converted.
+If you have multiple experiments that share a relevant conversion point, you can configure them with a shared group. This allows you to reference and convert multiple experiments at once using that shared group, and only experiments in which participants have been enrolled will be converted.
 
-Shared metrics can only be used for conversion, not for enrollment, since experiments don't share assignments.
+Shared groups can only be used for conversion, not for enrollment, since experiments don't share assignments.
 
-For example if you have multiple experiments where performing a search is considered to be a successful conversion, you can configure them all with the same shared metric then use that metric in your calls to `trailguide.convert`.
+For example if you have multiple experiments where performing a search is considered to be a successful conversion, you can configure them all with the same shared group then use that group in your calls to `trailguide.convert`.
 
 ```ruby
 experiment :first_search_experiment do |config|
-  config.metric = :perform_search
+  config.group = :perform_search
 
   variant :a
   variant :b
 end
 
 experiment :second_search_experiment do |config|
-  config.metric = :perform_search
+  config.groups = [:perform_search, :other_group]
 
   variant :one
   variant :two
@@ -789,7 +795,9 @@ experiment :second_search_experiment do |config|
 end
 
 experiment :third_search_experiment do |config|
-  config.metric = :perform_search
+  group :other_group
+  group :perform_search
+  groups :yet_another_group, :one_more
 
   variant :red
   variant :blue
@@ -803,7 +811,64 @@ class SearchController < ApplicationController
 end
 ```
 
-Since experiments with defined goals require a goal to be passed in when converting, any experiments that are sharing a metric must define the same goals.
+### Orphaned Groups
+
+Sometimes in the real world, you might accidentally remove all the experiments that were sharing a given group, but miss one of the conversion calls that used one of it's groups. Maybe you forgot to search through your code for references to the group, or maybe you just didn't know you were removing the last experiment in that group. Ideally you'd be testing your code thoroughly, and you'd catch the problem before hitting production, but trailguide has a built-in safe guard just in case.
+
+Instead of raising a `TrailGuide::NoExperimentsError` when no experiments match your arguments like `trailguide.choose` and related methods do, the `trailguide.convert` method will log a warning and return `false` as if no conversion happened.
+
+After a failed conversion for an orphaned group, the next time you visit the trailguide admin dashboard you'll see an alert with the details of any logged orphaned groups. If you wish to ignore orphaned groups entirely, perhaps so you can leave conversion calls in your application while you regularly rotate experiments into and out of those groups, you can set the `TrailGuide.configuration.ignore_orphaned_groups = true` config option in your initializer.
+
+### Groups with Goals
+
+Since grouping is only useful when converting, and experiments with defined goals require a goal to be passed in when converting, **any experiments that are sharing a group must define the same goals in order to be converted together.** Not all goals need to overlap, but you will only be able to convert goals that are shared when referencing a group.
+
+If you're grouping your experiments, that probably means you have multiple experiments that are all being used in the same area of your app and therefore are likely sharing the same (or similar) conversion goals. You can assign your groups and goals the same names to make converting easier by referencing a single key:
+
+```ruby
+experiment :first_search_experiment do |config|
+  variant :alpha
+  variant :bravo
+
+  config.groups = [:click_search, :click_banner, :search_experiments]
+  config.goals  = [:click_search, :click_banner, :custom_goal]
+end
+
+experiment :second_search_experiment do |config|
+  variant :one
+  variant :two
+  variant :three
+
+  config.groups = [:click_search, :click_banner, :search_experiments]
+  config.goals  = [:click_search, :click_banner, :some_other_goal]
+end
+
+experiment :third_search_experiment do |config|
+  variant :red
+  variant :blue
+
+  config.groups = [:click_search, :click_banner, :search_experiments]
+  config.goals  = [:click_search, :click_banner]
+end
+
+# then to convert all three experiments for the click_search group, against the
+# click_search goal
+trailguide.convert(:click_search)
+
+# the above is the equivalent of calling their group name (in this case not
+# matching the goal name) and the goal name
+trailguide.convert(:search_experiments, :click_search)
+
+# or the equivalent of converting each of the three experiments individually
+# for that goal
+trailguide.convert(:first_search_experiment, :click_search)
+trailguide.convert(:second_search_experiment, :click_search)
+trailguide.convert(:third_search_experiment, :click_search)
+
+# and you can still convert the individual experiments with goals that are not
+# shared by their group
+trailguide.convert(:first_search_experiment, :custom_goal)
+```
 
 ## Combined Experiments
 

data/app/assets/javascripts/trail_guide/admin/application.js

@@ -13,5 +13,24 @@
 //= require_tree .
 
 $(function () {
-  $('[data-toggle="tooltip"]').tooltip()
+  // using data-tooltip allows placing custom tooltips on objects already using
+  // data-toggle="modal"
+  $('[data-toggle="tooltip"], [data-tooltip="tooltip"]').tooltip()
+
+  // date/time pickers for scheduling experiments
+  $('.datepicker').datetimepicker({
+    format: 'MM/DD/YYYY hh:mm A Z',
+    icons: { time: 'fas fa-clock' },
+    keepOpen: true,
+    allowInputToggle: true,
+  })
+
+  // bootstrap custom file picker
+  bsCustomFileInput.init()
+
+  $('button[href]').on('click', function(evt) {
+    window.location.href = $(evt.currentTarget).attr('href')
+  })
+
+  $('.toast').toast('show')
 })

data/app/assets/stylesheets/trail_guide/admin/application.css

@@ -19,10 +19,32 @@
   height: 30px;
 }
 
+body {
+  padding-top: 60px;
+  padding-bottom: 40px;
+}
+
 main {
   padding-top: 40px;
 }
 
+.toasts {
+  position: fixed;
+  top: 65px;
+  right: 10px;
+  z-index: 900;
+  max-width: 250px;
+}
+
+.alert p:last-child {
+  padding-bottom: 0;
+  margin-bottom: 0;
+}
+
 .footer {
+  position: fixed;
+  bottom: 0;
+  left: 0;
+  width: 100%;
   padding: 10px 15px;
 }

data/app/assets/stylesheets/trail_guide/admin/experiments.css

@@ -1,12 +1,45 @@
-.experiment {
+.experiments button.experiment {
+  display: block;
+  width: 100%;
+  padding: 20px;
+  margin-bottom: 20px;
+  border: none;
+}
+.experiments button.experiment:not(:hover):not(:active) {
+  background-color: #f8f9fa;
+}
+
+.experiments button.experiment .experiment-text {
+  color: #343a40;
+}
+.experiments button.experiment:hover .experiment-text,
+.experiments button.experiment:active .experiment-text {
+  color: inherit !important;
+}
+
+div.experiment {
   margin-bottom: 40px;
 }
 
-.experiment table th, .experiment table td {
+.experiments button.experiment .experiment-text h4,
+.experiments button.experiment .experiment-text p {
+  white-space: nowrap;
+  overflow: hidden;
+  text-overflow: ellipsis;
+}
+
+div.experiment table th, .experiment table td {
   line-height: 31px;
+  vertical-align: middle;
+  text-align: center;
+}
+
+div.experiment table th.btn-col, .experiment table td.btn-col {
+  width: 36px;
+  text-align: center;
 }
 
-.experiment table th h5 {
+div.experiment table th h5 {
   margin: 0;
   padding: 0;
   line-height: 31px;

data/app/controllers/trail_guide/admin/application_controller.rb

@@ -18,35 +18,86 @@ module TrailGuide
 
       def experiment_peekable?(experiment)
         return false unless TrailGuide::Admin.configuration.peek_parameter
-        return false unless experiment.started? && !experiment.stopped?
+        return false unless experiment.started? && !experiment.stopped? && !experiment.winner?
+        return true  if experiment.combined? && !experiment.combined_experiments.all?(&:target_sample_size_reached?)
         return false if experiment.target_sample_size_reached?
         return true
       end
       helper_method :experiment_peekable?
 
+      def peek_param
+        params[TrailGuide::Admin.configuration.peek_parameter]
+      end
+
       def experiment_peeking?(experiment)
-        params[TrailGuide::Admin.configuration.peek_parameter] == experiment.experiment_name.to_s
+        # TODO deprecate the argument/param??
+        return params.key?(TrailGuide::Admin.configuration.peek_parameter)
+        peek_param == experiment.experiment_name.to_s ||
+          experiment.is_combined? && peek_param == experiment.parent.experiment_name.to_s
       end
       helper_method :experiment_peeking?
 
       def experiment_metrics_visible?(experiment)
-        return true unless experiment.started? && !experiment.stopped?
-        return true if params[TrailGuide::Admin.configuration.peek_parameter] == experiment.experiment_name.to_s
+        return true unless experiment.started? && !experiment.stopped? && !experiment.winner?
+        return true if experiment_peeking?(experiment)
+        return false if experiment.combined? && !experiment.combined_experiments.all?(&:target_sample_size_reached?)
         return true if experiment.target_sample_size_reached?
         return false
       end
       helper_method :experiment_metrics_visible?
 
-      def experiment_metric(experiment, metric)
-        return helpers.number_with_delimiter(metric) if experiment_metrics_visible?(experiment)
-        return helpers.content_tag('span', nil, class: 'fas fa-eye-slash', data: {toggle: 'tooltip'}, title: "metrics are hidden until this experiment reaches it's target sample size")
+      def experiment_metric(experiment, metric=nil, &block)
+        if experiment_metrics_visible?(experiment)
+          yield and return if block_given?
+          return helpers.number_with_delimiter(metric.to_i)
+        end
+
+        return helpers.content_tag('span', nil, class: 'fas fa-eye-slash text-muted', data: {toggle: 'tooltip'}, title: "metrics are hidden until this experiment reaches it's target sample size")
       end
       helper_method :experiment_metric
 
       def peek_url(experiment, *args, **opts)
-        trail_guide_admin.experiments_url(*args, opts.merge({TrailGuide::Admin.configuration.peek_parameter => experiment.experiment_name, anchor: experiment.experiment_name}))
+        trail_guide_admin.experiment_url(experiment.experiment_name, *args, opts.merge({TrailGuide::Admin.configuration.peek_parameter => experiment.experiment_name}))
       end
       helper_method :peek_url
+
+      def experiment_icon(experiment)
+        if experiment.winner?
+          'fa-flag-checkered'
+        elsif experiment.started?
+          if experiment.stopped?
+            'fa-stop'
+          elsif experiment.paused?
+            'fa-pause'
+          else
+            'fa-play'
+          end
+        elsif experiment.scheduled?
+          'fa-clock'
+        else
+          'fa-flask'
+        end
+      end
+      helper_method :experiment_icon
+
+      def experiment_color(experiment)
+        if experiment.winner?
+          'primary'
+        elsif experiment.started?
+          if experiment.stopped?
+            'danger'
+          elsif experiment.paused?
+            'warning'
+          else
+            'success'
+          end
+        elsif experiment.scheduled?
+          'info'
+        else
+          'secondary'
+        end
+      end
+      helper_method :experiment_color
     end
   end
 end

data/app/controllers/trail_guide/admin/experiments_controller.rb

@@ -1,69 +1,150 @@
+require 'json'
+
 module TrailGuide
   module Admin
     class ExperimentsController < ::TrailGuide::Admin::ApplicationController
-      before_action except: [:index] do
+      before_action except: [:index, :import] do
         (redirect_to :back rescue redirect_to trail_guide_admin.experiments_path) and return unless experiment.present?
       end
 
       before_action :experiments, only: [:index]
+      before_action :experiment,  except: [:index, :import]
 
       def index
+        respond_to do |format|
+          format.html { render }
+          format.json {
+            send_data JSON.pretty_generate(TrailGuide.catalog.export),
+            filename: "trailguide-#{Rails.env}-#{Time.now.to_i}.json"
+          }
+        end
+      end
+
+      def import
+        import_file = params[:file]
+
+        if import_file
+          if import_file.respond_to?(:read)
+            state_json = JSON.load(import_file.read)
+          elsif import_file.respond_to?(:path)
+            state_json= JSON.load(File.read(import_file.path))
+          end
+          TrailGuide.catalog.import(state_json)
+          flash[:success] = "Experiment state imported successfully"
+          redirect_to trail_guide_admin.experiments_path
+        else
+          raise "Please provide an import file"
+        end
+      rescue => e
+        flash[:error] = "There was a problem importing this file: #{e.message}"
+        redirect_to trail_guide_admin.experiments_path
+      end
+
+      def show
+        @analyzing = params.key?(:analyze)
+        @analyze_goal = params[:goal].present? ?
+          params[:goal].underscore.to_sym :
+          experiment.goals.first.try(:name)
+        @analyze_method = params[:method].present? ?
+          params[:method].underscore.to_sym :
+          :score
       end
 
       def start
+        experiment.reset!(self) if experiment.enable_calibration?
         experiment.start!(self)
-        redirect_to trail_guide_admin.scoped_experiments_path(scope: :running, anchor: experiment.experiment_name)
+
+        flash[:success] = "Experiment started"
+        redirect_to_experiment experiment
+      end
+
+      def schedule
+        experiment.schedule!(schedule_params[:start_at], schedule_params[:stop_at], self)
+
+        flash[:success] = "Experiment scheduled for <strong>#{experiment.started_at.strftime(TrailGuide::Admin::DISPLAY_DATE_FORMAT)}</strong>"
+        redirect_to_experiment experiment
+      rescue => e
+        flash[:danger] = e.message
+        redirect_to_experiment experiment
       end
 
       def pause
         experiment.pause!(self)
-        redirect_to trail_guide_admin.scoped_experiments_path(scope: :paused, anchor: experiment.experiment_name)
+
+        flash[:success] = "Experiment paused"
+        redirect_to_experiment experiment
       end
 
       def stop
         experiment.stop!(self)
-        redirect_to trail_guide_admin.scoped_experiments_path(scope: :stopped, anchor: experiment.experiment_name)
+
+        flash[:success] = "Experiment stopped"
+        redirect_to_experiment experiment
       end
 
       def reset
         experiment.stop!(self)
         experiment.reset!(self)
-        redirect_to trail_guide_admin.experiments_path(anchor: experiment.experiment_name)
+
+        flash[:success] = "Experiment reset"
+        redirect_to_experiment experiment
       end
 
       def resume
         experiment.resume!(self)
-        redirect_to trail_guide_admin.scoped_experiments_path(scope: :running, anchor: experiment.experiment_name)
+
+        flash[:success] = "Experiment resumed"
+        redirect_to_experiment experiment
       end
 
       def restart
         experiment.stop!(self)
         experiment.reset!(self)
         experiment.start!(self)
-        redirect_to trail_guide_admin.scoped_experiments_path(scope: :running, anchor: experiment.experiment_name)
+
+        flash[:success] = "Experiment restarted"
+        redirect_to_experiment experiment
+      end
+
+      def enroll
+        variant = enroll_experiment(experiment)
+        flash[:info] = "You were enrolled in the <strong>#{variant.to_s.humanize.titleize}</strong> variant"
+        redirect_to_experiment experiment
+      end
+
+      def convert
+        params[:goal] = nil if params[:goal] == 'converted'
+        variant = convert_experiment(experiment, params[:goal])
+        if variant
+          flash[:info] = "You successfully converted the goal <strong>#{params[:goal].to_s.humanize.titleize}</strong> in the <strong>#{variant.to_s.humanize.titleize}</strong> variant"
+        else
+          flash[:info] = "You did not convert the goal <strong>#{params[:goal].to_s.humanize.titleize}</strong>"
+        end
+        redirect_to_experiment experiment
       end
 
       def join
-        participant.exit!(experiment)
-        variant = experiment.variants.find { |var| var == params[:variant] }
-        variant.increment_participation!
-        participant.participating!(variant)
-        redirect_to trail_guide_admin.experiments_path(anchor: experiment.experiment_name)
+        join_experiment(experiment)
+        flash[:success] = "You joined the <strong>#{params[:variant].to_s.humanize.titleize}</strong> cohort"
+        redirect_to_experiment experiment
       end
 
       def leave
-        participant.exit!(experiment)
-        redirect_to trail_guide_admin.experiments_path(anchor: experiment.experiment_name)
+        leave_experiment(experiment)
+        flash[:success] = "You left the experiment"
+        redirect_to_experiment experiment
       end
 
       def winner
         experiment.declare_winner!(params[:variant], self)
-        redirect_to trail_guide_admin.experiments_path(anchor: experiment.experiment_name)
+        flash[:success] = "Declared <strong>#{params[:variant].to_s.humanize.titleize}</strong> as the winner"
+        redirect_to_experiment experiment
       end
 
       def clear
         experiment.clear_winner!
-        redirect_to trail_guide_admin.experiments_path(anchor: experiment.experiment_name)
+        flash[:success] = "Removed the winner"
+        redirect_to_experiment experiment
       end
 
       private
@@ -78,10 +159,122 @@ module TrailGuide
         @experiment ||= TrailGuide.catalog.find(params[:id])
       end
 
+      def schedule_params
+        @schedule_params ||= begin
+          exp_params = params.require(:experiment).permit(:start_at, :stop_at)
+          start_at = exp_params[:start_at]
+          stop_at = exp_params[:stop_at]
+
+          exp_params[:start_at] = DateTime.strptime(exp_params[:start_at], TrailGuide::SCHEDULE_DATE_FORMAT) rescue raise(ArgumentError, "Invalid start date")
+          raise ArgumentError, "Invalid start date" unless exp_params[:start_at].strftime(TrailGuide::SCHEDULE_DATE_FORMAT) == start_at
+
+          exp_params[:stop_at] = (DateTime.strptime(exp_params[:stop_at], TrailGuide::SCHEDULE_DATE_FORMAT) rescue nil)
+          if exp_params[:stop_at]
+            raise ArgumentError, "Invalid stop date" unless exp_params[:stop_at].strftime(TrailGuide::SCHEDULE_DATE_FORMAT) == stop_at
+
+            raise ArgumentError, "Experiments cannot be scheduled to stop before they start" if exp_params[:stop_at] <= exp_params[:start_at]
+          end
+
+          exp_params
+        end
+      end
+
+      def enroll_experiment(experiment)
+        if experiment.is_combined?
+          experiment.new(participant).choose!
+        else
+          leave_experiment(experiment)
+          variant = experiment.new(participant).choose!
+          if experiment.combined?
+            experiment.combined_experiments.each do |expmt|
+              expmt.new(participant).choose!
+            end
+          end
+          variant
+        end
+      end
+
+      def convert_experiment(experiment, checkpoint=nil)
+        return false if experiment.combined?
+        experiment.new(participant).convert!(checkpoint)
+      end
+
+      def join_experiment(experiment)
+        # TODO handle explicitly joining combined experiments
+        leave_experiment(experiment)
+        if experiment.is_combined?
+          variant = experiment.parent.variants.find { |var| var == params[:variant] }
+          variant.increment_participation!
+          participant.participating!(variant)
+          experiment.parent.combined_experiments.each do |expmt|
+            variant = expmt.variants.find { |var| var == params[:variant] }
+            variant.increment_participation!
+            participant.participating!(variant)
+          end
+        else
+          variant = experiment.variants.find { |var| var == params[:variant] }
+          variant.increment_participation!
+          participant.participating!(variant)
+          if experiment.combined?
+            experiment.combined_experiments.each do |expmt|
+              variant = expmt.variants.find { |var| var == params[:variant] }
+              variant.increment_participation!
+              participant.participating!(variant)
+            end
+          end
+        end
+      end
+
+      def leave_experiment(experiment)
+        participant.exit!(experiment)
+        if experiment.combined?
+          experiment.combined_experiments.each do |expmt|
+            participant.exit!(expmt)
+          end
+        end
+      end
+
       def participant
         @participant ||= TrailGuide::Participant.new(self)
       end
       helper_method :participant
+
+      def experiment_calculator(experiment, **opts)
+        klass = "TrailGuide::Calculators::#{@analyze_method.to_s.classify}".constantize
+        calculator = klass.new(experiment, **{base: :control, goal: @analyze_goal}.merge(opts))
+        calculator.calculate! if @analyzing
+        calculator
+      end
+      helper_method :experiment_calculator
+
+      def variant_analysis_color(variant, calculator)
+        if !@analyzing || !experiment_metrics_visible?(calculator.experiment)
+          'dark'
+        elsif variant.measure > 0
+          if variant == calculator.base
+            'dark'
+          elsif variant.measure == calculator.best.measure
+            'success'
+          elsif variant.measure == calculator.worst.measure
+            'danger'
+          elsif variant.measure > calculator.base.measure
+            'info'
+          elsif variant.measure < calculator.base.measure
+            'warning'
+          end
+        else
+          'muted'
+        end
+      end
+      helper_method :variant_analysis_color
+
+      def redirect_to_experiment(experiment)
+        if experiment <= TrailGuide::CombinedExperiment
+          redirect_back fallback_location: trail_guide_admin.experiment_path(experiment.parent.experiment_name)
+        else
+          redirect_to trail_guide_admin.experiment_path(experiment.experiment_name)
+        end
+      end
     end
   end
 end

data/app/controllers/trail_guide/admin/groups_controller.rb

@@ -0,0 +1,34 @@
+module TrailGuide
+  module Admin
+    class GroupsController < ::TrailGuide::Admin::ApplicationController
+      before_action except: [:index] do
+        (redirect_to :back rescue redirect_to trail_guide_admin.experiments_path) and return unless group.present?
+      end
+
+      before_action :groups,      only: [:index]
+      before_action :group,       only: [:show]
+      before_action :experiments, only: [:show]
+
+      def index
+      end
+
+      def show
+      end
+
+      private
+
+      def groups
+        @groups = TrailGuide.catalog.groups
+      end
+
+      def group
+        @group ||= TrailGuide.catalog.groups.find { |group| group == params[:id].to_s.underscore.to_sym }
+      end
+
+      def experiments
+        @experiments = TrailGuide.catalog.select(params[:id])
+        @experiments = @experiments.by_started
+      end
+    end
+  end
+end

data/app/controllers/trail_guide/experiments_controller.rb

@@ -22,7 +22,7 @@ module TrailGuide
 
     def convert
       # we use the param here because convert can trigger multiple experiements
-      # based on the passed key via shared metrics
+      # based on the passed key via shared groups
       trailguide.convert!(params[:experiment_name], checkpoint, metadata: metadata)
       render json: {
         experiment: experiment.experiment_name,