benchmark-memory 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: ea91842c5ad96cfe3dc3c101bf3d7179b6b1871f
+  data.tar.gz: 75947cf78b65b8286bceec66bf4ace28403cc5ac
+SHA512:
+  metadata.gz: 23a9674497bab9bf07ecaaf07c0f51cdf880223a924de621806afa8f9f62fdca4cf0db59913b9f2372d77a8d849b73f297d106a0a80f9c9c0cd0c15878e5503b
+  data.tar.gz: 8eab918622dbb1e0dfbea807fa39c4404489510a90f3d37632dad8bbb6b0f8fad00fea34ce30a684f3f3b4ec74e4a6d00cadd30b742bf826a65631a5907f5c5d

data/CHANGELOG.md

@@ -0,0 +1,16 @@
+# Change Log
+
+All notable changes to this project will be documented in this file. This project adheres to [Semantic Versioning 2.0.0][semver]. Any violations of this scheme are considered to be bugs.
+
+[semver]: http://semver.org/spec/v2.0.0.html
+
+## [0.1.0] - 2016-05-18
+
+### Added
+
+- Main `Benchmark.memory` method.
+- Holding results between invocations for measuring implementations on different versions of Ruby or different versions of libraries.
+- Quiet mode, with no command line output.
+
+[0.1.0]: https://github.com/michaelherold/benchmark-memory/tree/v0.1.0
+[unreleased]: https://github.com/michaelherold/benchmark-memory/compare/v0.1.0...HEAD

data/CONTRIBUTING.md

@@ -0,0 +1,50 @@
+# Contributing
+
+In the spirit of [free software](http://www.fsf.org/licensing/essays/free-sw.html), **everyone** is encouraged to help improve this project. Here are some ways *you* can contribute:
+
+* Use alpha, beta, and pre-release versions.
+* Report bugs.
+* Suggest new features.
+* Write or edit documentation.
+* Write specifications.
+* Write code (**no patch is too small**: fix typos, add comments, clean up inconsistent whitespace).
+* Refactor code.
+* Fix [issues][].
+* Review patches.
+
+[issues]: https://github.com/michaelherold/benchmark-memory/issues
+
+## Submitting an Issue
+
+We use the [GitHub issue tracker][issues] to track bugs and features. Before submitting a bug report or feature request, check to make sure it hasn't already been submitted.
+
+When submitting a bug report, please include a [Gist](https://gist.github.com) that includes a stack trace and any details that may be necessary to reproduce the bug, including your gem version, Ruby version, and operating system.
+
+Ideally, a bug report should include a pull request with failing specs.
+
+## Submitting a Pull Request
+
+1. [Fork the repository](http://help.github.com/fork-a-repo/).
+2. [Create a topic branch](http://learn.github.com/p/branching.html).
+3. Add specs for your unimplemented feature or bug fix.
+4. Run `bundle exec rake spec`. If your specs pass, return to step 3.
+5. Implement your feature or bug fix.
+6. Run `bundle exec rake`. If your specs or any of the linters fail, return to step 5.
+7. Open `coverage/index.html`. If your changes are not completely covered by your tests, return to step 3.
+8. Add documentation for your feature or bug fix.
+9. Run `bundle exec inch`. If your changes are below a B in documentation, go back to step 8.
+10. Commit and push your changes.
+11. [Submit a pull request](http://help.github.com/send-pull-requests/).
+
+## Tools to help you succeed
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+When writing code, you can use the helper application [Guard][guard] to automatically run tests and coverage tools whenever you modify and save a file. This helps to eliminate the tedium of running tests manually and reduces the chance that you will accidentally forget to run the tests. To use Guard, run `bundle exec guard`.
+
+Before committing code, run `rake` to check that the code conforms to the style guidelines of the project, that all of the tests are green (if you're writing a feature; if you're only submitting a failing test, then it does not have to pass!), and that the changes are sufficiently documented.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org][rubygems].
+
+[guard]: http://guardgem.org
+[rubygems]: https://rubygems.org

data/LICENSE.md

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 Michael Herold
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,183 @@
+# benchmark-memory
+
+[![Build Status](https://travis-ci.org/michaelherold/benchmark-memory.svg)][travis]
+[![Code Climate](https://codeclimate.com/github/michaelherold/benchmark-memory/badges/gpa.svg)][codeclimate]
+[![Inline docs](http://inch-ci.org/github/michaelherold/benchmark-memory.svg?branch=master)][inch]
+
+[codeclimate]: https://codeclimate.com/github/michaelherold/benchmark-memory
+[inch]: http://inch-ci.org/github/michaelherold/benchmark-memory
+[travis]: https://travis-ci.org/michaelherold/benchmark-memory
+
+benchmark-memory is a tool that helps you to benchmark the memory usage of different pieces of code. It leverages the power of [memory_profiler] to give you a metric of the total amount of memory allocated and retained by a block, as well as the number of objects and strings allocated and retained.
+
+[memory_profiler]: https://github.com/SamSaffron/memory_profiler
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem "benchmark-memory"
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install benchmark-memory
+
+## Usage
+
+Following the examples of the built-in Benchmark and Evan Phoenix's [benchmark-ips], the most common way of using benchmark-memory is through the `Benchmark.memory` wrapper. An example might look like this:
+
+```ruby
+require "benchmark/memory"
+
+# First method under test
+def allocate_string
+  "this string was dynamically allocated"
+end
+
+# Second method under test
+def give_frozen_string
+  "this string is frozen".freeze
+end
+
+Benchmark.memory do |x|
+  x.report("dynamic allocation") { allocate_string }
+  x.report("frozen string") { give_frozen_string }
+
+  x.compare!
+end
+```
+
+This example tests two methods that are defined inline. Note that you don't have to define them inline; you can just as easily use a method that you require before the benchmark or anything else that you can place in a block.
+
+When you run this example, you see the difference between the two reports:
+
+```txt
+Calculating -------------------------------------
+  dynamic allocation    40.000  memsize (     0.000  retained)
+                         1.000  objects (     0.000  retained)
+                         1.000  strings (     0.000  retained)
+       frozen string     0.000  memsize (     0.000  retained)
+                         0.000  objects (     0.000  retained)
+                         0.000  strings (     0.000  retained)
+
+Comparison:
+       frozen string:          0 allocated
+  dynamic allocation:         40 allocated - Infx more
+```
+
+Reading this output shows that the "dynamic allocation" example allocates one string that is not retained outside the scope of the block. The "frozen string" example, however, does not allocate anything because it reuses the frozen string that was created during the method definition.
+
+[benchmark-ips]: https://github.com/evanphx/benchmark-ips
+
+## Options
+
+There are several options available when running a memory benchmark.
+
+### Suppress all output (Quiet Mode)
+
+```ruby
+Benchmark.memory(:quiet => true)
+```
+
+Passing a `:quiet` flag to the `Benchmark.memory` method suppresses the output of the benchmark. You might find this useful if you want to run a benchmark as part of your test suite, where outputting to `STDOUT` would be disruptive.
+
+### Enable comparison
+
+```ruby
+Benchmark.memory do |x|
+  x.compare!
+end
+```
+
+Calling `#compare!` on the job within the setup block of `Benchmark.memory` enables the output of the comparison section of the benchmark. Without it, this section is suppressed and you only get the raw numbers output during calculation.
+
+### Hold results between invocations
+
+```ruby
+Benchmark.memory do |x|
+  x.hold!("benchmark_results.json")
+end
+```
+
+Often when you want to benchmark something, you compare two implementations of the same method. This is cumbersome because you have to keep two implementations side-by-side and call them in the same manner. Alternatively, you may want to compare how a method performs on two different versions of Ruby. To make both of these scenarios easier, you can enable "holding" on the benchmark.
+
+By calling `#hold!` on the benchmark, you enable the benchmark to write to the given file to store its results in a file that can be read in between invocations of your benchmark.
+
+For example, imagine that you have a library that exposes a method called `Statistics.calculate_monthly_recurring_revenue` that you want to optimize for memory usage because it keeps causing your worker server to run out of memory. You make some changes to the method and commit them to an `optimize-memory` branch in Git.
+
+To test the two implementations, you could then write this benchmark:
+
+```ruby
+require "benchmark/memory"
+
+require "stats" # require your library file here
+data = []       # set up the data that it will call here
+
+Benchmark.memory do |x|
+  x.report("original")  { Stats.monthly_recurring_revenue(data) }
+  x.report("optimized") { Stats.monthly_recurring_revenue(data) }
+
+  x.compare!
+  x.hold("bm_recurring_revenue.json")
+end
+```
+
+Note that the method calls are the same for both tests and that we have enabled result holding in the "bm_recurring_revenue.json" file.
+
+You could then run the following (assuming you saved your benchmark as `benchmark_mrr.rb`:
+
+```sh
+$ git checkout master
+$ ruby benchmark_mrr.rb
+$ git checkout optimize-memory
+$ ruby benchmark_mrr.rb
+```
+
+The first invocation of `ruby benchmark_mrr.rb` runs the benchmark in the "original" entry using your code in your `master` Git branch. The second invocation runs the benchmark in the "optimized" entry using the code in your `optimize-memory` Git branch. These two results are collated and then compared to show you the difference between the two.
+
+When enabling holding, the benchmark writes to the file passed into the `#hold!` method. After you run all of the entries in the benchmark, the benchmark automatically cleans up its log by deleting the file.
+
+## Supported Ruby Versions
+
+This library aims to support and is [tested against][travis] the following Ruby versions:
+
+* Ruby 2.1
+* Ruby 2.2
+* Ruby 2.3
+
+If something doesn't work on one of these versions, it's a bug.
+
+This library may inadvertently work (or seem to work) on other Ruby versions, however, support will only be provided for the versions listed above.
+
+If you would like this library to support another Ruby version or implementation, you may volunteer to be a maintainer. Being a maintainer entails making sure all tests run and pass on that implementation. When something breaks on your implementation, you will be responsible for providing patches in a timely fashion. If critical issues for a particular implementation exist at the time of a major release, support for that Ruby version may be dropped.
+
+## Versioning
+
+This library aims to adhere to [Semantic Versioning 2.0.0][semver]. Violations of this scheme should be reported as bugs. Specifically, if a minor or patch version is released that breaks backward compatibility, that version should be immediately yanked and/or a new version should be immediately released that restores compatibility. Breaking changes to the public API will only be introduced with new major versions. As a result of this policy, you can (and should) specify a dependency on this gem using the [Pessimistic Version Constraint][pessimistic] with two digits of precision. For example:
+
+    spec.add_dependency "benchmark-memory", "~> 0.1"
+
+[pessimistic]: http://guides.rubygems.org/patterns/#pessimistic-version-constraint
+[semver]: http://semver.org/spec/v2.0.0.html
+
+## Acknowledgments
+
+This library wouldn't be possible without two projects and the people behind them:
+
+* Sam Saffron's [memory_profiler] does all of the measurement of the memory allocation and retention in the benchmarks.
+* I based much of the code around Evan Phoenix's [benchmark-ips] project, since it has a clean base from which to work and a logical organization. I also wanted to go for feature- and DSL-parity with it because I really like the way it works.
+
+[benchmark-ips]: https://github.com/evanphx/benchmark-ips
+[memory_profiler]: https://github.com/SamSaffron/memory_profiler
+
+## License
+
+The gem is available as open source under the terms of the [MIT License][license].
+
+[license]: http://opensource.org/licenses/MIT.

data/Rakefile

@@ -0,0 +1,29 @@
+require "bundler"
+Bundler.setup
+Bundler::GemHelper.install_tasks
+
+require "inch/rake"
+Inch::Rake::Suggest.new(:inch)
+
+require "rspec/core/rake_task"
+RSpec::Core::RakeTask.new(:spec)
+
+require "rubocop/rake_task"
+RuboCop::RakeTask.new(:rubocop)
+
+require "yard/rake/yardoc_task"
+YARD::Rake::YardocTask.new(:yard)
+
+task :mutant do
+  command = [
+    "bundle exec mutant",
+    "--include lib",
+    "--require interactor-contracts",
+    "--use rspec",
+    "Interactor::Contracts*",
+  ].join(" ")
+
+  system command
+end
+
+task :default => [:spec, :rubocop, :yard, :inch]

data/benchmark-memory.gemspec

@@ -0,0 +1,24 @@
+# coding: utf-8
+
+require File.expand_path("../lib/benchmark/memory/version", __FILE__)
+
+Gem::Specification.new do |spec|
+  spec.name          = "benchmark-memory"
+  spec.version       = Benchmark::Memory::VERSION
+  spec.authors       = ["Michael Herold"]
+  spec.email         = ["michael.j.herold@gmail.com"]
+
+  spec.summary       = "Benchmark-style memory profiling for Ruby 2.1+"
+  spec.description   = spec.summary
+  spec.homepage      = "https://github.com/michaelherold/benchmark-memory"
+  spec.license       = "MIT"
+
+  spec.files = %w(CHANGELOG.md CONTRIBUTING.md LICENSE.md README.md Rakefile)
+  spec.files += %w(benchmark-memory.gemspec)
+  spec.files += Dir["lib/**/*.rb"]
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "memory_profiler", "~> 0.9"
+
+  spec.add_development_dependency "bundler", "~> 1.12"
+end

data/lib/benchmark-memory.rb

@@ -0,0 +1,2 @@
+# rubocop:disable Style/FileName
+require "benchmark/memory"

data/lib/benchmark/memory.rb

@@ -0,0 +1,34 @@
+require "benchmark/memory/errors"
+require "benchmark/memory/job"
+require "benchmark/memory/version"
+
+# Performance benchmarking library
+module Benchmark
+  # Benchmark memory usage in code to benchmark different approaches.
+  # @see https://github.com/michaelherold/benchmark-memory
+  module Memory
+    # Measure memory usage in report blocks.
+    #
+    # @param quiet [Boolean] A flag to toggle benchmark output.
+    #
+    # @return [Report]
+    def memory(quiet: false)
+      unless block_given?
+        fail(
+          ConfigurationError,
+          "You did not give a test block to your call to `Benchmark.memory`"
+        )
+      end
+
+      job = Job.new(:quiet => quiet)
+
+      yield job
+
+      job.run
+      job.run_comparison
+      job.full_report
+    end
+  end
+
+  extend Benchmark::Memory
+end

data/lib/benchmark/memory/errors.rb

@@ -0,0 +1,7 @@
+module Benchmark
+  module Memory
+    Error = Class.new(StandardError)
+
+    ConfigurationError = Class.new(Error)
+  end
+end

data/lib/benchmark/memory/held_results.rb

@@ -0,0 +1,110 @@
+require "forwardable"
+require "benchmark/memory/held_results/entry_serializer"
+
+module Benchmark
+  module Memory
+    # Collate results that should be held until the next run.
+    class HeldResults
+      extend Forwardable
+
+      # Instantiate a new set of held results on a path.
+      #
+      # @param path [String, IO] The path to write held results to.
+      def initialize(path = nil)
+        @path = path
+        @results = {}
+      end
+
+      # @return [String, IO] The path to write held results to.
+      attr_accessor :path
+
+      # @return [Hash{String => Measurement}] Held results from previous runs.
+      attr_reader :results
+
+      # Allow Hash-like access to the results without asking for them.
+      def_delegator :@results, :[]
+
+      # Add a result to the held results.
+      #
+      # @param entry [Report::Entry] The entry to hold.
+      #
+      # @return [void]
+      def add_result(entry)
+        with_hold_file("a") do |file|
+          file.write EntrySerializer.new(entry)
+          file.write "\n"
+        end
+      end
+
+      # Check whether any results have been stored.
+      #
+      # @return [Boolean]
+      def any?
+        if @path.is_a?(String)
+          File.exist?(@path)
+        else
+          @path.size > 0 # rubocop:disable Style/ZeroLengthPredicate
+        end
+      end
+
+      # Clean up the results after all results have been collated.
+      #
+      # @return [void]
+      def cleanup
+        if @path.is_a?(String) && File.exist?(@path)
+          File.delete(@path)
+        end
+      end
+
+      # Check whether to hold results.
+      #
+      # @return [Boolean]
+      def holding?
+        !!@path
+      end
+
+      # Check whether an entry has been added to the results.
+      #
+      # @param entry [#label] The entry to check.
+      #
+      # @return [Boolean]
+      def include?(entry)
+        holding? && any? && results.key?(entry.label)
+      end
+
+      # Load results from the serialized output.
+      #
+      # @return [void]
+      def load
+        return unless holding? && any?
+
+        results = with_hold_file do |file|
+          file.map { |line| EntrySerializer.load(line) }
+        end
+        @results = Hash[results.map do |result|
+          [result.label, result.measurement]
+        end]
+      end
+
+      private
+
+      # Execute a block on the hold file.
+      #
+      # @param access_mode [String] The mode to use when opening the file.
+      # @param _block [Proc] The block to execute on each line of the file.
+      #
+      # @return [void]
+      def with_hold_file(access_mode = "r", &_block)
+        return unless @path
+
+        if @path.is_a?(String)
+          File.open(@path, access_mode) do |f|
+            yield f
+          end
+        else
+          yield @path
+        end
+      end
+    end
+  end
+end