coverband 1.0.1 → 1.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ff5daaf6442d4b70be9c7f3888150f0a12a70661
-  data.tar.gz: b0c48c0017f95a99b2a2215bbee1109808e8c5a8
+  metadata.gz: 053a4656762684a3293fdaf19c267660d8f1a495
+  data.tar.gz: 372ce2923b56e30cfe94ac2345e62b3db1974518
 SHA512:
-  metadata.gz: 5a273640660fe4a87f2c7bc325865ad90f5ae0ce22d9f730d9e9002f9282e6421beccab62f5a973b6c6472c4ddf41745400815c133f1094309d70a54464b2772
-  data.tar.gz: cac85909606d5fd2f45b93dd795dc3be8375d69e3651909d58c9982de0e1f4ff120615997e6af8031d03f8b90f889d3fc8cc2c7b6f9775433be6d8502c5860ba
+  metadata.gz: 7673df791c9d6f443d7b04436656d0e6800fd414e9c6363be6a6bfe89dc39b39437cd91954639cbc1f519001f66d5ba52af0ffa1120d217583c66985322c8256
+  data.tar.gz: c41882f81a929e15aa9068ddc03774a6fa6af3040dd3405f9b2ca334f8c4f742a9ec54df4dc1e637ed4120383ec1582990965e73334b720e8fbc383dba883bcc

data/README.md

@@ -6,12 +6,14 @@ A gem to measure production code coverage. Coverband allows easy configuration t
 * Ignore directories to avoid overhead data collection on vendor, lib, etc.
 * Take a baseline to get initial app loading coverage.
 
-At the moment, Coverband relies on Ruby's `set_trace_func` hook. I attempted to use the standard lib's `Coverage` support but it proved buggy when sampling or stoping and starting collection. When [Coverage is patched](https://bugs.ruby-lang.org/issues/9572) in future Ruby versions it would likely be better. Using `set_trace_func` has some limitations where it doesn't collect covered lines, but I have been impressed with the coverage it shows for both Sinatra and Rails applications.
+At the moment, Coverband relies on Ruby's `set_trace_func` hook. I attempted to use the standard lib's `Coverage` support but it proved buggy when sampling or stopping and starting collection. When [Coverage is patched](https://bugs.ruby-lang.org/issues/9572) in future Ruby versions it would likely be better. Using `set_trace_func` has some limitations where it doesn't collect covered lines, but I have been impressed with the coverage it shows for both Sinatra and Rails applications.
 
 ###### Success:
 After running in production for 30 minutes, we were able very easily delete 2000 LOC after looking through the data. We expect to be able to clean up much more after it has collected more data. 
 
-This has now been running in production on many applications for months. I will clean up configurations, documentation, and strive to get a 1.0 release out soon.
+###### Performance Impact
+
+At the moment the performance impact of standard Ruby runtime coverage can be pretty large. Once getting things working. I highly recommend adding [coverband_ext](https://github.com/danmayer/coverband_ext) to the project which should shave the performance overhead down to something very reasonable. The two ways to deal with performance right now are lowering the sample rate and using the C extension. Often for smaller projects using the C extension makes 100% coverage possible. 
 
 ## Installation
 
@@ -45,17 +47,20 @@ Details on a example Sinatra app
 
 ## Notes
 
-* Using Redis 2.x gem, while supported, is extremely slow and not recommended. It will have a much larger impact on overhead performance.
-* This has been tested in Ruby 1.9.3, 2 and is running in production on Sinatra, Rails 2.3.x, and Rails 3.2.x
-* No 1.8.7 support
-* There is a performance impact which is why the gem supports sampling. On low traffic sites I am running a sample rake of 20% and on very high traffic sites I am sampling at 1%, which still gives excellent data
-* I believe there are possible ways to get even better data using the new [Ruby2 TracePoint API](http://www.ruby-doc.org/core/TracePoint.html)
-* Make sure to ignore any folders like `vendor` and possibly `lib` as it can help reduce performance overhead
+* Coverband has been running in production on Ruby 1.9.3, 2.x, 2.1.x on Sinatra, Rails 2.3.x, Rails 3.0.x, Rails 3.1.x, and Rails 3.2.x
+* No 1.8.7 support, Coverband requires Ruby 1.9.3+
+* There is a performance impact which is why the gem supports sampling. On low traffic sites I am running a sample rate of 20% and on very high traffic sites I am sampling at 1%, which still gives useful data
+    * The impact with the pure Ruby coverband can't be rather significant on sampled requests
+    * Most of the overhead is in the Ruby coverage collector, you can now use [coverband_ext](https://github.com/danmayer/coverband_ext) to run a C extension collector which is MUCH faster.
+    * Using Redis 2.x gem, while supported, is slow and not recommended. It will have a larger impact on overhead performance. Although the Ruby collection dwarfs the redis time, so it likely doesn't matter much.
+    * Make sure to ignore any folders like `vendor` and possibly `lib` as it can help reduce performance overhead. Or ignore specific frequently hit in app files for better perf.
 
 ## Usage
 
 After installing the gem. There are a few steps to gather data, view reports, and for cleaning up the data.
 
+See an [example Sinatra app](https://github.com/danmayer/churn-site) and example [non rack ruby app](https://github.com/danmayer/coverband_examples) configured with coverband.
+
 1. First configure cover band options using the config file, See the section below
 2. Then configure Rake, with helpful tasks. Make sure things are working by recording your Coverband baseline. See the section below
 3. Setup the rack middleware, the middleware is what makes Coverband gather metrics when your app runs. See below for details
@@ -67,9 +72,10 @@ After installing the gem. There are a few steps to gather data, view reports, an
 
 #### Configure Coverband Options
 
-You need to configure cover band you can either do that passing in all configuration options to `Coverband.configure` in block format, or a much simpler style is to call `Coverband.configure` with nothing while will load `config/coverband.rb` expecting it to configure the app correctly. Below is an example config file for a Sinatra app:
+You need to configure cover band you can either do that passing in all configuration options to `Coverband.configure` in block format, or a simpler style is to call `Coverband.configure` with nothing while will load `config/coverband.rb` expecting it to configure the app correctly. Below is an example config file for a Sinatra app:
 
 ```ruby
+#config/coverband.rb
 require 'json'
 
 baseline = Coverband.parse_baseline
@@ -77,36 +83,32 @@ baseline = Coverband.parse_baseline
 Coverband.configure do |config|
   config.root              = Dir.pwd
   if defined? Redis
-    config.redis             = Redis.new(:host => 'redis.host.com', :port => 49182, :db => 1)
+    config.redis           = Redis.new(:host => 'redis.host.com', :port => 49182, :db => 1)
   end
   config.coverage_baseline = baseline
-  config.root_paths        = ['/app/']
-  config.ignore            = ['vendor']
+  config.root_paths        = ['/app/'] # /app/ is needed for heroku deployments
+  # regex paths can help if you are seeing files duplicated for each capistrano deployment release
+  #config.root_paths       = ['/server/apps/my_app/releases/\d+/'] 
+  config.ignore            = ['vendor','lib/scrazy_i18n_patch_thats_hit_all_the_time.rb']
   # Since rails and other frameworks lazy load code. I have found it is bad to allow
   # initial requests to record with coverband. This ignores first 15 requests
-  config.startup_delay     = 15
-  config.percentage        = 60.0
+  config.startup_delay     = Rails.env.production? ? 15 : 2
+  config.percentage        = Rails.env.production? ? 30.0 : 100.0
+  
+  config.logger            = Rails.logger
+  
+  #stats help you collect how often you are sampling requests and other info
   if defined? Statsd
-    config.stats             = Statsd.new('statsd.host.com', 8125)
+    config.stats           = Statsd.new('statsd.host.com', 8125)
   end
+  # config options false, true, or 'debug'. Always use false in production
+  # true and debug can give helpful and interesting code usage information
+  # they both increase the performance overhead of the gem a little.
+  # they can also help with initially debugging the installation.
   config.verbose           = Rails.env.production? ? false : true
 end
 ```
 
-Here is a alternative configuration example, allowing for production and development settings:
-
-```ruby
-Coverband.configure do |config|
-  config.root              = Dir.pwd
-  config.redis             = Redis.new
-  config.coverage_baseline = JSON.parse(File.read('./tmp/coverband_baseline.json'))
-  config.root_paths        = ['/app/']
-  config.ignore            = ['vendor']
-  config.startup_delay     = Rails.env.production? ? 10 : 1
-  config.percentage        = Rails.env.production? ? 15.0 : 100.0
-end
-```
-
 #### Configuring Rake
 
 Either add the below to your `Rakefile` or to a file included in your Rakefile such as `lib/tasks/coverband` if you want to break it up that way.
@@ -125,7 +127,7 @@ rake coverband:clear         # reset coverband coverage data
 rake coverband:coverage      # report runtime coverband code coverage
 ```
 
-The default Coverband baseline task will try to load the Rails environment. For a non Rails application you can make your own baseline. Below for example is how I take a baseline on a Sinatra app.
+The default Coverband baseline task will try to detect your app as either Rack or Rails environment. It will load the app to take a baseline reading. If the baseline task doesn't load your app well you can override the default baseline to create a better baseline yourself. Below for example is how I take a baseline on a pretty simple Sinatra app.
 
 ```ruby
 namespace :coverband do
@@ -139,7 +141,8 @@ namespace :coverband do
 end
 ```
 
-To verify that rake is working run `rake coverband:baseline` and then run `rake coverband:coverage` to view what your baseline coverage looks like before any runtime traffic has been recorded.
+To verify that rake is working run `rake coverband:baseline`
+then run `rake coverband:coverage` to view what your baseline coverage looks like before any runtime traffic has been recorded.
     
 #### Configure rack middleware
 
@@ -162,11 +165,13 @@ It is easy to use Coverband outside of a Rack environment. Make sure you configu
 For example if you had a base Resque class, you could use the `before_perform` and `after_perform` hooks to add Coverband
 
 ```ruby
+require 'coverband'
+Coverband.configure
+
 def before_perform(*args)
   if (rand * 100.0) <= Coverband.configuration.percentage
-    @@coverband ||= Coverband::Base.new
     @recording_samples = true
-    @@coverband.start
+    Coverband::Base.instance.start
   else
     @recording_samples = false
   end
@@ -174,23 +179,20 @@ end
       
 def after_perform(*args)
   if @recording_samples
-    @@coverband.stop
-    @@coverband.save
+    Coverband::Base.instance.stop
+    Coverband::Base.instance.save
   end
 end
 ```
 
-In general you can run Coverband anywhere by using the lines below
+In general you can run Coverband anywhere by using the lines below. This can be useful to wrap all cron jobs, background jobs, or other code run outside of web requests. I recommend trying to run both background and cron jobs at 100% coverage as the performance impact is less important and often old code hides around those jobs.
+
 
 ```ruby
-require 'coverband'
-	
-Coverband.configure do |config|
-  config.redis             = Redis.new
-  config.percentage        = 50.0
-end
+require "coverband"
+Coverband.configure
   
-coverband = Coverband::Base.new
+coverband = Coverband::Base.instance
     
 #manual
 coverband.start
@@ -203,7 +205,7 @@ coverband.sample {
 }
 ```
  
-## Clearing Line Coverage Data
+#### Clearing Line Coverage Data
 
 After a deploy where code has changed. 
 The line numbers previously recorded in Redis may no longer match the current state of the files. 
@@ -221,7 +223,7 @@ Coverband::Reporter.clear_coverage(Redis.new(:host => 'target.com', :port => 678
 You can also do this with the included rake tasks.
 
 
-## Verbose debug mode for development
+### Verbose debug mode for development
 
 If you are trying to debug locally wondering what code is being run during a request. The verbose modes `config.verbose = true` and `config.verbose = 'debug'` can be useful. With true set it will output the number of lines executed per file, to the passed in log. The files are sorted from least used file to most active file. I have even run that mode in production without much of a problem. The debug verbose mode outputs both file usage and provides the number of calls per line of code. For example if you see something like below which indicates that the `application_helper` has 43150 lines executed. That might seem odd. Then looking at the breakdown of `application_helper` we can see that line `516` was executed 38,577 times. That seems bad, and is likely worth investigating perhaps memoizing or cacheing is required. 
 
@@ -240,11 +242,29 @@ If you are trying to debug locally wondering what code is being run during a req
       [[448, 1], [202, 1],
       ...
      [517, 1617], [516, 38577]]
+     
+### Merge coverage data over time     
+
+If you are clearing data on every deploy. You might want to write the data out to a file first. Then you could merge the data into the final results later.
 
+```ruby
+data = JSON.generate Coverband::Reporter.get_current_scov_data
+File.write("blah.json", data)  
+# Then later on, pass it in to the html reporter:
+data = JSON.parse(File.read("blah.json"))
+Coverband::Reporter.report :additional_scov_data => [data]   
+```
+
+#### Known issues
 
-## TODO
+* `set_trace_func` isn't perfect in sending each line of code executed and can be a bit wonky in a few places. Such as missing the `end` lines in code blocks. If you notice examples of this send them to me.
+* If you don't have a baseline recorded your coverage can look odd like you are missing a bunch of data. It would be good if coverband gave a more actionable warning in this situation.
+* If you have simplecov filters, you need to clear them prior to generating your coverage report. As the filters will be applied to coverband as well and can often filter out everything we are recording.
+* the line numbers reported for `ERB` files are often off and aren't considered useful. I recommend filtering out .erb using the `config.ignore` option.
 
-* Fix network performance by logging to files that purge later (far more time lost in set_trace_func than sending files, hence not a high priority)
+### TODO
+
+* Fix network performance by logging to files that purge later (like NR) (far more time lost in set_trace_func than sending files, hence not a high priority, but would be cool)
 * Add support for [zadd](http://redis.io/topics/data-types-intro) so one could determine single call versus multiple calls on a line, letting us determine the most executed code in production.
 * Possibly add ability to record code run for a given route
 * Improve client code api, around manual usage of sampling (like event usage)
@@ -252,17 +272,27 @@ If you are trying to debug locally wondering what code is being run during a req
   * blank rails app
   * blank Sinatra app
 * report on Coverband files that haven't recorded any coverage (find things like events and crons that aren't recording, or dead files)
-* ability to change the cover band config at runtime by changing the config pushed to the Redis hash. In memory cache around the changes to only make that call periodically.
+* ability to change the coverband config at runtime by changing the config pushed to the Redis hash. In memory cache around the changes to only make that call periodically.
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
 
 ## Resources
 
+These notes of kind of for myself, but if anyone is seriously interested in contributing to the project, these resorces might be helpfu. I learned a lot looking at various existing projects and open source code.
+
 ##### Ruby Std-lib Coverage
 
 * [Fixed bug causing segfaults on 1.9.X](https://www.ruby-forum.com/topic/1811306)
 * [Current Coverage Bug causing issues on 2.1.1](https://bugs.ruby-lang.org/issues/9572)
 * [Ruby Coverage docs](http://www.ruby-doc.org/stdlib-1.9.3/libdoc/coverage/rdoc/Coverage.html)
 
-##### other
+##### Other
 
 * [erb code coverage](http://stackoverflow.com/questions/13030909/how-to-test-code-coverage-for-rails-erb-templates)
 * [more erb code coverage](https://github.com/colszowka/simplecov/issues/38)
@@ -273,14 +303,7 @@ If you are trying to debug locally wondering what code is being run during a req
 * [Jruby coverage bug](http://jira.codehaus.org/browse/JRUBY-6106?page=com.atlassian.jira.plugin.system.issuetabpanels:changehistory-tabpanel)
 * [learn from oboe ruby code](https://github.com/appneta/oboe-ruby#writing-custom-instrumentation)
 * [learn from stackprof](https://github.com/tmm1/stackprof#readme)
-
-## Contributing
-
-1. Fork it
-2. Create your feature branch (`git checkout -b my-new-feature`)
-3. Commit your changes (`git commit -am 'Add some feature'`)
-4. Push to the branch (`git push origin my-new-feature`)
-5. Create new Pull Request
+* I believe there are possible ways to get even better data using the new [Ruby2 TracePoint API](http://www.ruby-doc.org/core/TracePoint.html)
 
 ## MIT License
 See the file license.txt for copying permission.

data/lib/coverband.rb

@@ -10,6 +10,8 @@ require 'coverband/middleware'
 
 module Coverband
 
+  CONFIG_FILE = './config/coverband.rb'
+  
   class << self
     attr_accessor :configuration_data
   end
@@ -23,12 +25,16 @@ module Coverband
   end
 
   def self.configure(file = nil)
-    self.configuration_data ||= Configuration.new
+    configuration
     if block_given?
       yield(configuration)
     else
-      file ||= './config/coverband.rb'
-      require file
+      if File.exists?(CONFIG_FILE)
+        file ||= CONFIG_FILE
+        require file
+      else
+        raise ArgumentError, "configure requires a block or the existance of a #{CONFIG_FILE} in your project"
+      end
     end
   end
 

data/lib/coverband/base.rb

@@ -89,7 +89,7 @@ module Coverband
       if @reporter
         if @reporter.class.name.match(/redis/i)
           before_time = Time.now
-          @stats.increment "coverband.files.recorded_files", @files.length if @stats
+          @stats.count "coverband.files.recorded_files", @files.length if @stats
           @reporter.store_report(@files.dup)
           time_spent = Time.now - before_time
           @stats.timing "coverband.files.recorded_time", time_spent if @stats
@@ -132,17 +132,17 @@ module Coverband
       end
     end
 
+    # file from ruby coverband at this method call is a full path
+    # file from native coverband is also a full path
+    #
+    # at the moment the full paths don't merge so the 'last' one wins and each deploy
+    # with a normal capistrano setup resets coverage.
+    #
+    # should we make it relative in this method (slows down collection)
+    # -- OR --
+    # we could have the reporter MERGE the results after normalizing the filenames
+    # (went with this route see report_scov previous_line_hash)
     def add_file_without_checks(file, line)
-      # file from ruby coverband at this method call is a full path
-      # file from native coverband is also a full path
-      #
-      # at the moment the full paths don't merge so the 'last' one wins and each deploy
-      # with a normal capistrano setup resets coverage.
-      #
-      # should we make it relative in this method (slows down collection)
-      # -- OR --
-      # we could have the reporter MERGE the results after normalizing the filenames
-      # (went with this route see report_scov previous_line_hash)
       if @verbose
         @file_usage[file] += 1
         @file_line_usage[file] = Hash.new(0) unless @file_line_usage.include?(file)

data/lib/coverband/reporter.rb

@@ -12,7 +12,9 @@ module Coverband
       if Coverband.configuration.verbose
         Coverband.configuration.logger.info results.inspect
       end
-      
+
+      config_dir = File.dirname(Coverband.configuration.baseline_file)
+      Dir.mkdir config_dir unless File.exists?(config_dir)
       File.open(Coverband.configuration.baseline_file, 'w') {|f| f.write(results.to_json) }
     end
 
@@ -24,18 +26,20 @@ module Coverband
         return
       end
       redis = Coverband.configuration.redis
-      roots = Coverband.configuration.root_paths
+      roots = get_roots
       existing_coverage = Coverband.configuration.coverage_baseline
       open_report = options.fetch(:open_report){ true }
 
-      roots << "#{current_root}/"
-
       if Coverband.configuration.verbose
         Coverband.configuration.logger.info "fixing root: #{roots.join(', ')}"
       end
 
       if  Coverband.configuration.reporter=='scov'
-        report_scov(redis, existing_coverage, roots, open_report)
+        additional_scov_data = options.fetch(:additional_scov_data){ [] }
+        if Coverband.configuration.verbose
+          print additional_scov_data
+        end
+        report_scov(redis, existing_coverage, additional_scov_data, roots, open_report)
       else
         lines = redis.smembers('coverband').map{|key| report_line(redis, key) }
         Coverband.configuration.logger.info lines.join("\n")
@@ -48,6 +52,12 @@ module Coverband
       redis.del("coverband")
     end
 
+    def self.get_roots
+      roots = Coverband.configuration.root_paths
+      roots << "#{current_root}/"
+      roots
+    end
+
     def self.current_root
       File.expand_path(Coverband.configuration.root)
     end
@@ -84,7 +94,12 @@ module Coverband
      merged
     end
 
-    def self.report_scov(redis, existing_coverage, roots, open_report)
+    
+    def self.get_current_scov_data
+      get_current_scov_data_imp(Coverband.configuration.redis, get_roots)
+    end
+
+    def self.get_current_scov_data_imp(redis, roots)
       scov_style_report = {}
       redis.smembers('coverband').each do |key|
                                    next if Coverband.configuration.ignore.any?{ |i| key.match(i)}
@@ -101,8 +116,17 @@ module Coverband
                                    end
                                  end
       scov_style_report = fix_file_names(scov_style_report, roots)
+      scov_style_report
+    end
+      
+    def self.report_scov(redis, existing_coverage, additional_scov_data, roots, open_report)
+      scov_style_report = get_current_scov_data_imp redis, roots
       existing_coverage = fix_file_names(existing_coverage, roots)
       scov_style_report = merge_existing_coverage(scov_style_report, existing_coverage)
+
+      additional_scov_data.each do |data|
+        scov_style_report = merge_existing_coverage(scov_style_report, data)
+      end
       
       if Coverband.configuration.verbose
         Coverband.configuration.logger.info "report: "
@@ -139,9 +163,13 @@ module Coverband
     # /Users/danmayer/projects/cover_band_server/app/rb: ["54", "55"]
     # /Users/danmayer/projects/cover_band_server/views/layout/erb: ["0", "33", "36", "37", "38", "39", "40", "62", "63", "66", "65532", "65533"]
     def self.report_line(redis, key)
-      "#{key}: #{redis.smembers("coverband.#{key}").inspect}" #" fix line styles
+      "#{key}: #{line_members(redis, key)}"
     end
 
+    def self.line_members(redis, key)
+      redis.smembers("coverband.#{key}").inspect
+    end
+                                                            
     def self.filename_from_key(key, roots)
       filename = key
       roots.each do |root|

data/lib/coverband/tasks.rb

@@ -25,24 +25,26 @@ namespace :coverband do
               rescue Exception
                 #ignore
               end }
-        end
-        if File.exists?("#{Rails.root}/lib")
-          Dir.glob("#{Rails.root}/lib/**/*.rb").sort.each { |file|
+          if File.exists?("#{Rails.root}/lib")
+            Dir.glob("#{Rails.root}/lib/**/*.rb").sort.each { |file|
               begin
                 require_dependency file
               rescue Exception
                 #ignoring file
               end}
+          end
         end
       }
   end
 
   ###
-  # note: If have set a ton of simplecov filters you might want to override them and clear the filters or run the task below.
+  # note: If your project has set many simplecov filters.
+  # You might want to override them and clear the filters.
+  # Or run the task `coverage_no_filters` below.
   ###
   desc "report runtime coverband code coverage"
   task :coverage => :environment do
-                   Coverband::Reporter.report
+    Coverband::Reporter.report
   end
 
   def clear_simplecov_filters
@@ -51,12 +53,17 @@ namespace :coverband do
     end
   end
 
-  desc "report runtime coverband code coverage"
+  desc "report runtime coverband code coverage after disabling simplecov filters"
   task :coverage_no_filters => :environment do
     clear_simplecov_filters
     Coverband::Reporter.report
   end
 
+  ###
+  # You likely want to clear coverage after significant code changes.
+  # You may want to have a hook that saves current coverband data on deploy
+  # and then resets the redis data.
+  ###
   desc "reset coverband coverage data"
   task :clear  => :environment do
     Coverband::Reporter.clear_coverage

data/lib/coverband/version.rb

@@ -1,3 +1,3 @@
 module Coverband
-  VERSION = "1.0.1"
+  VERSION = "1.0.2"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: coverband
 version: !ruby/object:Gem::Version
-  version: 1.0.1
+  version: 1.0.2
 platform: ruby
 authors:
 - Dan Mayer
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-07-31 00:00:00.000000000 Z
+date: 2015-03-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -158,7 +158,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.2.2
+rubygems_version: 2.4.5
 signing_key: 
 specification_version: 4
 summary: Rack middleware to help measure production code coverage