abprof 0.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: d03d3d085332cd91d2e398789ec570640e2e3d3e
+  data.tar.gz: 9326d7c1967628febb8d45838227665d2be11d3f
+SHA512:
+  metadata.gz: 9839e0901b40964238330cf89021ebc9e84305672705c0c66472a4f0adb26aed709031e34b3b9e14880a966f2818443605aa6bd989e5be78dbd201a4c7611807
+  data.tar.gz: 69ad1b67b2786b7d24359c9219760bd322d9b148c073d4b3673620c268168b6259fffd10b4c320afc83183c71e6200ef9de236b2b984417debed4299637782e1

data/.gitignore

@@ -0,0 +1,10 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+abprof-*.gem

data/.travis.yml

@@ -0,0 +1,5 @@
+sudo: false
+language: ruby
+rvm:
+  - 2.3.1
+before_install: gem install bundler -v 1.12.5

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,49 @@
+# Contributor Code of Conduct
+
+As contributors and maintainers of this project, and in the interest of
+fostering an open and welcoming community, we pledge to respect all people who
+contribute through reporting issues, posting feature requests, updating
+documentation, submitting pull requests or patches, and other activities.
+
+We are committed to making participation in this project a harassment-free
+experience for everyone, regardless of level of experience, gender, gender
+identity and expression, sexual orientation, disability, personal appearance,
+body size, race, ethnicity, age, religion, or nationality.
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery
+* Personal attacks
+* Trolling or insulting/derogatory comments
+* Public or private harassment
+* Publishing other's private information, such as physical or electronic
+  addresses, without explicit permission
+* Other unethical or unprofessional conduct
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+By adopting this Code of Conduct, project maintainers commit themselves to
+fairly and consistently applying these principles to every aspect of managing
+this project. Project maintainers who do not follow or enforce the Code of
+Conduct may be permanently removed from the project team.
+
+This code of conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community.
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting a project maintainer at the.codefolio.guy@gmail.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. Maintainers are
+obligated to maintain confidentiality with regard to the reporter of an
+incident.
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 1.3.0, available at
+[http://contributor-covenant.org/version/1/3/0/][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/3/0/

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in abprof.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 AppFolio, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,291 @@
+# ABProf
+
+ABProf attempts to use simple A/B test statistical logic and apply it
+to the question, "which of these two programs is faster?"
+
+Most commonly, you profile by running a program a certain number of
+times ("okay, burn it into cache for 100 iterations, then run it 5000
+times and divide the total time by 5000"). Then, you make changes to
+your program and do the same thing again to compare.
+
+Real statisticians inform us that there are a few problems with that
+approach :-)
+
+We use a [Welch's T Test](https://en.wikipedia.org/wiki/Welch%27s_t-test) on a
+set of measured runtimes to determine how likely the two programs are
+to be different from each other, and after the P value is low enough,
+we give our current estimate of which is faster and by how much.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'abprof'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install abprof
+
+## Usage
+
+### Quick Start - Run Two Programs
+
+The simplest way to use ABProf is the "abcompare" command. Give it two
+commands, let it run them for you and measure the results. If your
+command contains spaces, put it in quotes - standard shell
+programming.
+
+    $ abcompare "cd ../vanilla_ruby && ./tool/runruby.rb ../optcarrot/bin/optcarrot --benchmark ../optcarrot/examples/Lan_Master.nes >> /dev/null" \
+      "cd ../alt_ruby && ./tool/runruby.rb ../optcarrot/bin/optcarrot --benchmark ../optcarrot/examples/Lan_Master.nes >> /dev/null"
+
+This defaults to basic settings (10 iterations of burn-in before
+measuring, P value of 0.05, etc.)  You can change them on the command
+line. Running this way is simple, straightforward, and will take
+a little longer to converge since it's paying the start-a-process tax every
+time it takes a measurement.
+
+Run "abcompare --help" if you want to see what command-line options
+you can supply. For more control in the results, see below.
+
+The abcompare command is identical to abprof except that it uses a raw
+command, not harness code. See below for details.
+
+### Quick Start - Test Harness
+
+Loading and running a program is slow, and it adds a lot of variable
+overhead. That can make it hard to sample the specific operations that
+you want to measure. ABProf prefers to just do the operations you want
+without restarting the worker processes constantly. That takes a bit
+of harness code to do well.
+
+In Ruby, there's an ABProf library you can use which will take care of
+that interface. That's the easiest way to use it, especially since
+you're running a benchmark anyway and would need some structure around
+your code.
+
+For a Ruby snippet to be profiled very simply, do this:
+
+    require "abprof"
+
+    ABProf::ABWorker.iteration do
+      # Code to measure goes here
+	  sleep 0.1
+    end
+
+    ABProf::ABWorker.start
+
+With two such files, you can compare their speed.
+
+Under the hood, ABProf's harness uses a simple communication protocol
+over STDIN and STDOUT to allow the controlling process to tell the
+workers to run iterations. Mostly that's great, but it means you'll
+need to make sure your worker processes aren't using STDIN for
+anything else.
+
+See the examples directory for more. For instance:
+
+    abprof examples/sleep.rb examples/sleep.rb
+
+If abprof is just in the source directory and not installed as a gem,
+you should add RUBYLIB="lib" before "abprof" above to get it to run.
+
+### Quick Start - Benchmark DSL
+
+Want to make a benchmark reproducible? Want better accuracy? ABProf
+has a DSL (Domain-Specific Language) that can help here.
+
+Here's a simple example:
+
+    require "abprof/benchmark_dsl"
+
+    ABProf.compare do
+      warmup 10
+      max_trials 5
+      min_trials 3
+      p_value 0.01
+      iters_per_trial 2
+      bare true
+
+      report do
+        10_000.times {}
+      end
+
+      report do
+        sleep 0.1
+      end
+
+    end
+
+Note that "warmup" is a synonym for "burnin" here -- iterations done
+before ABProf starts measuring and comparing. The "report" blocks are
+run for the sample. You can also have a "report_command", which takes
+a string as an argument and uses that to take a measurement.
+
+### A Digression - Bare and Harness
+
+"Harness" refers to ABProf's internal testing protocol, used to allow
+multiple processes to communicate. A "harness process" or "harness
+worker" means a second process that is used to take measurements, and
+can do so repeatedly without having to restart the process.
+
+A "bare process" means one where the work is run directly. Either a
+new process is spawned for each measurement (slow, inaccurate) or a
+block is run in the same Ruby process (potential for inadvertent
+cross-talk.)
+
+In general, for a "harness" process you'll need to put together a .rb
+file similar to examples/sleep.rb or examples/for\_loop_10k.rb.
+
+You can use the DSL above for either bare or harness processes ("bare
+true" or "bare false") without a problem. But if you tell it to use a
+harness, the process in question should be reading ABProf commands
+from STDIN and writing responses to STDOUT in ABProf protocol,
+normally by using the Ruby Test Harness library.
+
+### Don't Cross the Streams
+
+Harness-enabled tests expect to run forever, fielding requests for
+work.
+
+Non-harness-enabled tests don't know how to do harness stuff.
+
+If you run the wrong way (abcompare with a harness, abprof with no
+harness,) you'll get either an immediate crash or running forever
+without ever finishing burn-in, depending which way you did it.
+
+Normally you'll handle this by just passing your command line directly
+to abcompare rather than packaging it up into a separate Ruby script.
+
+### Comparing Rubies
+
+I'm AppFolio's Ruby fellow, so I'm writing this to compare two
+different locally-built Ruby implementations for speed. The easiest
+way to do that is to build them in multiple directories, then build a
+wrapper that uses that directory to run the program in question.
+
+You can see examples such as examples/alt\_ruby.rb and
+examples/vanilla\_ruby.rb and so on in the examples directory of this
+gem.
+
+Those examples use a benchmark called "optcarrot" which can be quite
+slow. So you'll need to decide whether to do a quick, rough check with
+a few iterations or a more in-depth check which runs many times for
+high certainty.
+
+Here's a slow, very conservative check:
+
+    abprof --burnin=10 --max-trials=50 --min-trials=50 --iters-per-trial=5 examples/vanilla_ruby.rb examples/inline_ruby_1800.rb
+
+Note that since the minimum and maximum trials are both 50, it won't
+stop at a particular certainty (P value.) It will just run for 50
+trials of 5 iterations each. It takes awhile, but gives a pretty good
+estimate of how fast one is compared to the other.
+
+Here's a quicker, rougher check:
+
+    abprof --burnin=5 --max-trials=10 --iters-per-trial=1 examples/vanilla_ruby.rb examples/inline_ruby_1800.rb
+
+It may stop after only a few trials if the difference in speed is big
+enough. By default, it uses a P value of 0.05, which is (very roughly)
+a one in twenty chance of a false result.
+
+If you want a very low chance of a false positive, consider adjusting
+the P value downward, to more like 0.001 (0.1% chance) or 0.00001
+(0.001% chance.) This may require a lot of time to run, especially if
+the two programs are of very similar speed, or have a lot of
+variability in the test results.
+
+    abprof --burnin=5 --max-trials=50 --pvalue 0.001 --iters-per-trial=1 examples/sleep.rb examples/for_loop_10k.rb
+
+### How Many Times Faster?
+
+ABProf will try to give you an estimate of how much faster one option
+is than the other. Be careful taking it at face value -- if you do a
+series of trials and coincidentally get a really different-looking
+run, that may give you an unexpected P value *and* an unexpected
+number of times faster.
+
+In other words, those false positives will tend to happen *together*,
+not independently. If you want to actually check how much faster one
+is than the other in a less-biased way, set the number of trials
+and/or iterations very high, or manually run both yourself some large
+number of times, rather than letting it converge to a P value and then
+taking the result from the output.
+
+See the first example under "Comparing Rubies" for one way to do
+this. Setting the min and max trials equal is good practice for this
+to reduce bias.
+
+### Does This Just Take Forever?
+
+It's easy to accidentally specify a very large number of iterations
+per trial, or total trials, or otherwise make testing a slow program
+take *forever*. Right now, you'll pretty much just need to notice that
+it's happening and drop the iters-per-trial, the min-trials, or the P
+value. When in doubt, try to start with just a very quick, rough test.
+
+Of course, if your test is *really* slow, or you're trying to detect a
+very small difference, it can just take a really long time. Like A/B
+testing, this method has its pitfalls.
+
+### More Control
+
+Would you like to explicitly return the value(s) to compare? You can
+replace the "iteration" block above with "iteration\_with\_return\_value"
+or "n\_iterations\_with\_return\_value". In the former case, return a
+single number at then end of the block, which is the measured value
+specifically for that time through the loop. In the latter case, your
+block will take a single parameter N for the number of iterations -
+run the code that many times and return either a single measured speed
+or time, or an array of speeds or times, which will be your samples.
+
+This can be useful when running N iterations doesn't necessarily
+generate exactly N results, or when the time the whole chunk of code
+takes to run isn't the most representative number for performance. The
+statistical test will help filter out random test-setup noise
+somewhat, but sometimes it's best to not count the noise in your
+measurement at all, for many good reasons.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install
+dependencies. Then, run `rake test` to run the tests. You can also run
+`bin/console` for an interactive prompt that will allow you to
+experiment.
+
+To install this gem onto your local machine, run `bundle exec rake
+install`. To release a new version, update the version number in
+`version.rb`, and then run `bundle exec rake release`, which will
+create a git tag for the version, push git commits and tags, and push
+the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Credit Where Credit Is Due
+
+I feel like I maybe saw this idea (use A/B test math for a profiler)
+somewhere else before, but I can't tell if I really did or if I
+misunderstood or hallucinated it. Either way, why isn't this a
+standard approach that's built into most profiling tools?
+
+After I started implementation I found out that optcarrot, used by the
+Ruby core team for profiling, is already using this technique (!) -- I
+am using it slightly differently, but I'm clearly not the first to
+think of using a statistics test to verify which of two programs is faster.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at
+https://github.com/appfolio/abprof. This project is intended to be a
+safe, welcoming space for collaboration, and contributors are expected
+to adhere to the
+[Contributor Covenant](http://contributor-covenant.org) code of
+conduct.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,10 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList['test/**/*_test.rb']
+end
+
+task :default => :test

data/abprof.gemspec

@@ -0,0 +1,36 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'abprof/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "abprof"
+  spec.version       = Abprof::VERSION
+  spec.authors       = ["Noah Gibbs"]
+  spec.email         = ["noah.gibbs@appfolio.com"]
+
+  spec.summary       = %q{Determine which of two programs is faster, statistically.}
+  spec.description   = %q{Determine which of two program variants is faster, using A/B-Testing-style statistical techniques.}
+  spec.homepage      = "https://github.com/appfolio/abprof"
+  spec.license       = "MIT"
+
+  # Prevent pushing this gem to RubyGems.org. To allow pushes either set the 'allowed_push_host'
+  # to allow pushing to a single host or delete this section to allow pushing to any host.
+  #if spec.respond_to?(:metadata)
+  #  spec.metadata['allowed_push_host'] = "TODO: Set to 'http://mygemserver.com'"
+  #else
+  #  raise "RubyGems 2.0 or newer is required to protect against public gem pushes."
+  #end
+
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.12"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "minitest", "~> 5.0"
+  spec.add_runtime_dependency "trollop", "~>2.1", ">=2.1.0"
+  spec.add_runtime_dependency "statsample", "~>2.0", ">=2.0.0"
+  spec.add_runtime_dependency "multi_json", "~>1.12", ">=1.12.0"
+end