abprof 0.2.1 → 0.2.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 7d20589cbf53709f04f0c72d9c172cde437f0338
-  data.tar.gz: 7e4324a28723597c74202b156736abeada7894c1
+  metadata.gz: ae747acd5bc96fca3eafd6e0a1a8b0ba2d4ef62b
+  data.tar.gz: ae5cfee49428221a97f8c45935ef967e3072473a
 SHA512:
-  metadata.gz: 568bce937feeedeccefbf7f4c922d2acf4a746551009310f64974e1c8f98148efd9f9bcf6d29bfb552818339df93b79255e3c747af49142d5c38d239958d49b5
-  data.tar.gz: 4e1bb9d16b3c6ab2fda879ebe986ae78f7f4a8564b40e277156dc9a1e64400e8aa38507e3102e627cb773fe79e8eaec024e3a0395fe8cd02e80c16147da23fd6
+  metadata.gz: 201b109cc93df7c18fe8d219ef6e3966abd8236bd950ca6a080a11c1c54d590506de62bb012e1b70cb540847ed50c7779ce7c3ac075328e3459a2b35501e049c
+  data.tar.gz: e79d72f6a0e8c5d439f250bd90755ca770b55c979229b983d9f3c64ce3c43500aaf724d4362ad3cc0107642c1775d8699b40cc4ab802a442c721bb24b5cdbc98

data/README.md

@@ -234,23 +234,57 @@ Of course, if your test is *really* slow, or you're trying to detect a
 very small difference, it can just take a really long time. Like A/B
 testing, this method has its pitfalls.
 
-### More Control
+### More Control of Sampling
 
 Would you like to explicitly return the value(s) to compare? You can
-replace the "iteration" block above with "iteration\_with\_return\_value"
-or "n\_iterations\_with\_return\_value". In the former case, return a
-single number at then end of the block, which is the measured value
-specifically for that time through the loop. In the latter case, your
-block will take a single parameter N for the number of iterations -
-run the code that many times and return either a single measured speed
-or time, or an array of speeds or times, which will be your samples.
-
-This can be useful when running N iterations doesn't necessarily
-generate exactly N results, or when the time the whole chunk of code
-takes to run isn't the most representative number for performance. The
-statistical test will help filter out random test-setup noise
-somewhat, but sometimes it's best to not count the noise in your
-measurement at all, for many good reasons.
+replace the "iteration" block above with
+"iteration\_with\_return\_value" to return a measurement of your
+choice. That allows you to do setup or teardown inside the block that
+isn't necessarily counted in the total time. You can also use a custom
+counter or timer rather than Ruby's Time.now, which is the default for
+ABProf.
+
+If you return a higher-is-better value like a counter rather than a
+lower-is-better value like time, you'll find that ABProf keeps telling
+you the *lower*-valued process, which may be slower rather than
+faster. ABProf can tell which one gets lower numbers, but it doesn't
+know whether that means better or worse.
+
+That's why the console output shows the word "faster?" with a question
+mark. It knows it's giving you lower. It hopes that means faster.
+
+### More Samples Per Trial
+
+Would you like to control how the N iterations (default 10) per trial
+get run? Want to do setup or teardown before or after them as a group,
+not individuall?
+
+Replace the "iteration" block above with
+"n\_iterations\_with\_return\_value". Your block will take a single
+parameter N for the number of iterations - run the code that many
+times and return either a single measured speed or time, or an array
+of speeds or times, which will be your samples.
+
+Note: this technique has some subtleties -- you're better off *not*
+doing this to rapidly collect many, many samples of very small
+performance differences. If you do, transient conditions like
+background processes can skew the results a *lot* when many T-test
+samples are collected in a short time. You're much better off running
+the same operation many times and returning the cumulative value in
+those cases, or otherwise controlling for transient conditions that
+drift over time.
+
+In those cases, either set the iters-per-trial very low (likely to 1)
+so that both processes are getting the benefit/penalty from transient
+background conditions, or set the number of iterations per trial very
+high so that each trial takes several seconds or longer, to allow
+transient conditions to pass.
+
+ABProf also runs the two processes' iterations in a random order by
+default, starting from one process or the other based on a per-trial
+random number. This helps a little, but only a little. If you *don't*
+want ABProf to do that for some reason, turn on the static_order
+option to get simple "process1 then process2" order for every trial.
 
 ## Development
 

data/examples/measured_sleep.rb

@@ -0,0 +1,13 @@
+#!/usr/bin/env ruby
+
+require "abprof"
+
+puts "ABProf example: sleep 0.1 seconds, manual return value"
+
+ABProf::ABWorker.iteration_with_return_value do
+  t1 = Time.now
+  sleep 0.01
+  (Time.now - t1)
+end
+
+ABProf::ABWorker.start

data/examples/multi_for_loop.rb

@@ -0,0 +1,15 @@
+#!/usr/bin/env ruby
+
+require "abprof"
+
+puts "ABProf example: sleep 0.1 seconds (multiple measurements per trial)"
+
+ABProf::ABWorker.n_iterations_with_return_value do |n|
+  (1..n).map do
+    t1 = Time.now
+    100_000.times {}
+    (Time.now - t1)  # Return array of measurements
+  end
+end
+
+ABProf::ABWorker.start

data/examples/multi_sleep.rb

@@ -0,0 +1,15 @@
+#!/usr/bin/env ruby
+
+require "abprof"
+
+puts "ABProf example: sleep 0.1 seconds (multiple measurements per trial)"
+
+ABProf::ABWorker.n_iterations_with_return_value do |n|
+  (1..n).map do
+    t1 = Time.now
+    sleep 0.01
+    (Time.now - t1)  # Return array of measurements
+  end
+end
+
+ABProf::ABWorker.start

data/exe/abprof

@@ -37,8 +37,7 @@ once.
 A p value is often interpreted as the probability we got a wrong answer.
 That's an oversimplification, but not (usually) a terrible one.
 BANNER
-  opt :debug1,      "Print first-process output to console"
-  opt :debug2,      "Print second-process output to console"
+  opt :debug,       "Print more output to console"
   opt :bare,        "Use bare command-line commands, no Ruby harness", :default => ($0["compare"])
   opt :pvalue,      "P value (certainty) for Welch's T test", :default => 0.05
   opt :burnin,      "'Burn in' repetitions before real trials",  :default => 10
@@ -47,6 +46,7 @@ BANNER
   opt :iters_per_trial, "Iterations per sample set", :default => 10
   opt :print_samples, "Print all sample values for later analysis.", :default => false
   opt :fail_on_divergence, "Return a non-zero code if pvalue is greater than specified."
+  opt :static_order, "Don't randomize the order of sampled processes per trial."
 end
 
 if ARGV.length != 2
@@ -65,56 +65,22 @@ bm_inst = ABProf.compare(:no_at_exit => true) do
   max_trials OPTS[:max_trials]
   iters_per_trial OPTS[:iters_per_trial]
   bare OPTS[:bare]
+  debug OPTS[:debug]
+  static_order OPTS[:static_order]
   # No fail_on_divergence - we do this manually for the CLI utilities
 
   report_command command1
   report_command command2
 end
 
-state = bm_inst.run_sampling
+state = bm_inst.run_sampling(:print_output => true)
 p_val = state[:p_tests][-1]
-diverged = false
-if p_val < bm_inst.p_value
-  puts "Based on measured P value #{p_val}, we believe there is a speed difference."
-  puts "As of end of run, p value is #{p_val}. Now run more times to check, or with lower p."
-
-  summary11 = ABProf.summarize("mean", state[:samples][0])
-  summary12 = ABProf.summarize("median", state[:samples][0])
-  summary21 = ABProf.summarize("mean", state[:samples][1])
-  summary22 = ABProf.summarize("median", state[:samples][1])
-
-  fastest = "1"
-  command = bm_inst.reports[0]
-  mean_times = summary21 / summary11
-  median_times = summary22 / summary12
-  if summary11 > summary21
-    fastest = "2"
-    command = bm_inst.reports[1]
-    mean_times = summary11 / summary21
-    median_times = summary12 / summary22
-  end
-
-  puts "Lower (faster?) process is #{fastest}, command line: #{command.inspect}"
-  puts "Lower command is (very) roughly #{median_times} times lower (faster?) -- assuming linear sampling."
-
-  print "\n"
-  puts "Process 1 mean result: #{summary11}"
-  puts "Process 1 median result: #{summary12}"
-  puts "Process 2 mean result: #{summary21}"
-  puts "Process 2 median result: #{summary22}"
-else
-  puts "Based on measured P value #{p_val} and threshold #{bm_inst.pvalue}, we believe there is"
-  puts "no significant difference detectable with this set of trials."
-  puts "If you believe there is a small difference that wasn't detected, try raising the number"
-  puts "of iterations per trial, or the maximum number of trials."
-  diverged = true
-end
 
 if OPTS[:print_samples]
   puts "Samples for P1: #{state[:samples][0].inspect}"
   puts "Samples for P2: #{state[:samples][1].inspect}"
 end
 
-exit 2 if diverged && OPTS[:fail_on_divergence]
+exit 2 if (p_val >= bm_inst.p_value) && OPTS[:fail_on_divergence]
 
 # Otherwise, return success

data/lib/abprof.rb

@@ -11,15 +11,8 @@ require "multi_json"
 #     QUIT requires no response.
 
 module ABProf
-  def self.debug
-    @debug
-  end
-  def self.debug=(new_val)
-    @debug = new_val
-  end
-
   # These are primarily for DSL use.
-  PROPERTIES = [ :debug, :pvalue, :iters_per_trial, :min_trials, :max_trials, :burnin, :bare, :fail_on_divergence ]
+  PROPERTIES = [ :debug, :pvalue, :iters_per_trial, :min_trials, :max_trials, :burnin, :bare, :fail_on_divergence, :static_order ]
 
   # This class is used by programs that are *being* profiled.
   # It's necessarily a singleton since it needs to control STDIN.
@@ -27,10 +20,10 @@ module ABProf
   # processes.
   class ABWorker
     def debug string
-      STDERR.puts(string) if ABProf.debug
+      STDERR.puts(string) if ENV['ABDEBUG'] == "true"
     end
     def self.debug string
-      STDERR.puts(string) if ABProf.debug
+      STDERR.puts(string) if ENV['ABDEBUG'] == "true"
     end
 
     def self.iteration(&block)
@@ -43,13 +36,13 @@ module ABProf
       @return = :per_iteration
     end
 
-    def self.n_interations_with_return_value(&block)
+    def self.n_iterations_with_return_value(&block)
       @iter_block = block
       @return = :per_n_iterations
     end
 
     def self.run_n(n)
-      debug "WORKER #{Process.pid}: running #{n} times"
+      debug "WORKER #{Process.pid}: running #{n} times [#{@return.inspect}]"
 
       case @return
       when :none
@@ -59,15 +52,17 @@ module ABProf
         STDOUT.write "OK\n"
       when :per_iteration
         values = (0..(n-1)).map { |i| @iter_block.call.to_f }
-        STDOUT.write "VALUES #{values.inspect}"
+        STDOUT.write "VALUES #{values.inspect}\n"
       when :per_n_iterations
         value = @iter_block.call(n)
         if value.respond_to?(:each)
           # Return array of numbers
-          STDOUT.write "VALUES #{value.to_a.inspect}"
+          debug "WORKER #{Process.pid}: Sent to controller: VALUES #{value.to_a.inspect}"
+          STDOUT.write "VALUES #{value.to_a.inspect}\n"
         else
           # Return single number
-          STDOUT.write "VALUE #{value.to_f}"
+          debug "WORKER #{Process.pid}: Sent to controller: VALUE #{value.to_f}"
+          STDOUT.write "VALUE #{value.to_f}\n"
         end
       else
         raise "Unknown @return value #{@return.inspect} inside abprof!"
@@ -131,7 +126,7 @@ module ABProf
     attr_reader :last_iters
 
     def debug string
-      STDERR.puts(string) if @debug && ABProf.debug
+      STDERR.puts(string) if @debug
     end
 
     def initialize command_line, opts = {}
@@ -182,7 +177,7 @@ module ABProf
     attr_reader :last_iters
 
     def debug string
-      STDERR.puts(string) if @debug && ABProf.debug
+      STDERR.puts(string) if @debug
     end
 
     def initialize command_line, opts = {}
@@ -191,14 +186,18 @@ module ABProf
       @out_reader, @out_writer = IO.pipe
       @in_writer.sync = true
       @out_writer.sync = true
+      @debug = opts[:debug]
 
       @pid = fork do
         STDOUT.reopen(@out_writer)
         STDIN.reopen(@in_reader)
         @out_reader.close
         @in_writer.close
+
+        ENV['ABDEBUG'] = @debug.inspect
+
         if command_line.respond_to?(:call)
-          puts "Caution! An ABProf Harness process (non-bare) is being used with a block. This is almost never what you want!"
+          STDERR.puts "Caution! An ABProf Harness process (non-bare) is being used with a block. This is almost never what you want!"
           command_line.call
         elsif command_line.respond_to?(:to_s)
           exec command_line.to_s
@@ -210,7 +209,6 @@ module ABProf
       @out_writer.close
       @in_reader.close
 
-      @debug = opts[:debug]
       debug "Controller spawned #{@pid} (debug: #{@debug.inspect})"
     end
 
@@ -236,7 +234,6 @@ module ABProf
         # Read and block
         output = @out_reader.gets
         ignored_out += output.length
-        puts "Controller of #{@pid} out: #{output.inspect}" if @debug
         debug "Controller of #{@pid} out: #{output.inspect}"
         if output =~ /^VALUES/ # These anchors match newlines, too
           state = :succeeded
@@ -244,25 +241,26 @@ module ABProf
           raise "Must return an array value from iterations!" unless vals.is_a?(Array)
           raise "Must return an array of numbers from iterations!" unless vals[0].is_a?(Numeric)
           @last_run = vals
+          break
         elsif output =~ /^VALUE/ # These anchors match newlines, too
           state = :succeeded
           val = output[6..-1].to_f
           raise "Must return a number from iterations!" unless val.is_a?(Numeric)
           @last_run = [ val ]
+          break
         elsif output =~ /^OK$/   # These anchors match newlines, too
           state = :succeeded_get_time
           break
-        end
-        if output =~ /^NOT OK$/ # These anchors match newlines, too
+        elsif output =~ /^NOT OK$/ # These anchors match newlines, too
           # Failed, break
           state = :explicit_not_ok
           break
-        end
-        if ignored_out > 10_000
+        elsif ignored_out > 10_000
           # 10k of output and no OK? Bail with failed state.
           state = :too_much_output_without_status
           break
         end
+        # None of these? Loop again.
       end
       t_end = Time.now
       unless [:succeeded, :succeeded_get_time].include?(state)

data/lib/abprof/benchmark_dsl.rb

@@ -20,6 +20,7 @@ module ABProf
       @max_trials = 20
       @iters_per_trial = 10
       @bare = false
+      @static_order = false
 
       @state = {
         :samples => [[], []],
@@ -49,9 +50,16 @@ module ABProf
       @process2.run_iters @burnin
     end
 
-    def run_one_iteration(pts = {})
-      @state[:samples][0] += @process1.run_iters @iters_per_trial
-      @state[:samples][1] += @process2.run_iters @iters_per_trial
+    def run_one_trial(pts = {})
+      order_rand = (rand() * 2.0).to_i
+      if @static_order || order_rand == 0
+        @state[:samples][0] += @process1.run_iters @iters_per_trial
+        @state[:samples][1] += @process2.run_iters @iters_per_trial
+      else
+        # Same thing, but do process2 first
+        @state[:samples][1] += @process2.run_iters @iters_per_trial
+        @state[:samples][0] += @process1.run_iters @iters_per_trial
+      end
       @state[:iter] += 1
     end
 
@@ -63,7 +71,6 @@ module ABProf
       @process1 = process_type.new command1, :debug => @debug
       @process2 = process_type.new command2, :debug => @debug
 
-      puts "Beginning #{@burnin} iterations of burn-in for each process." if opts[:print_output]
       run_burnin opts
 
       puts "Beginning sampling from processes." if opts[:print_output]
@@ -71,7 +78,7 @@ module ABProf
       # Sampling
       p_val = 1.0
       @max_trials.times do
-        run_one_iteration opts
+        run_one_trial opts
 
         # No t-test without 3+ samples
         if @state[:samples][0].size > 2
@@ -79,7 +86,11 @@ module ABProf
           t = Statsample::Test.t_two_samples_independent(@state[:samples][0].to_vector, @state[:samples][1].to_vector)
           p_val = t.probability_not_equal_variance
           @state[:p_tests].push p_val
-          puts "Trial #{@state[:iter]}, Welch's T-test p value: #{p_val.inspect}" if opts[:print_output]
+          avg_1 = @state[:samples][0].inject(0.0, &:+) / @state[:samples][0].length
+          avg_2 = @state[:samples][1].inject(0.0, &:+) / @state[:samples][1].length
+          smaller = "1"
+          smaller = "2" if avg_1 > avg_2
+          puts "Trial #{@state[:iter]}, Welch's T-test p value: #{p_val.inspect}   (Guessed smaller: #{smaller})" if opts[:print_output]
         end
 
         # Just finished trial number i+1. So we can exit only if i+1 was at least
@@ -118,7 +129,7 @@ module ABProf
         command = @reports[0]
         mean_times = summary21 / summary11
         median_times = summary22 / summary12
-        if summary11 > summary21
+        if summary12 > summary22
           fastest = "2"
           command = @reports[1]
           mean_times = summary11 / summary21
@@ -126,7 +137,8 @@ module ABProf
         end
 
         puts "Lower (faster?) process is #{fastest}, command line: #{command.inspect}"
-        puts "Lower command is (very) roughly #{median_times} times lower (faster?) -- assuming linear sampling."
+        puts "Lower command is (very) roughly #{median_times} times lower (faster?) -- assuming linear sampling, checking at median."
+        puts "         Checking at mean, it would be #{mean_times} lower (faster?)."
 
         print "\n"
         puts "Process 1 mean result: #{summary11}"

data/lib/abprof/version.rb

@@ -1,3 +1,3 @@
 module Abprof
-  VERSION = "0.2.1"
+  VERSION = "0.2.2"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: abprof
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.2.2
 platform: ruby
 authors:
 - Noah Gibbs
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-08-05 00:00:00.000000000 Z
+date: 2016-08-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -138,6 +138,9 @@ files:
 - examples/inline_ruby_1800.rb
 - examples/inline_ruby_2500.rb
 - examples/inlined_ruby.rb
+- examples/measured_sleep.rb
+- examples/multi_for_loop.rb
+- examples/multi_sleep.rb
 - examples/profiling_ruby.rb
 - examples/simple_dsl.rb
 - examples/sleep.rb