coverband 4.2.0 → 4.2.4

data/changes.md

@@ -16,6 +16,8 @@ Will be the fully modern release that drops maintenance legacy support in favor
   - add built-in support for easy loading via Railties
   - built in support for activejob, sidekiq, and other common frameworks
   - reduced configuration options
+- support oneshot
+- drop middleware figure out a way to kick off background without middelware
 - options on reporting
   - background reporting
   - or middleware reporting
@@ -47,6 +49,7 @@ Will be the fully modern release that drops maintenance legacy support in favor
 - move all code to work with relative paths leaving only stdlib Coverage working on full paths
 - add gem_safe_lists to track only some gems
 - add gem_details_safe list to report on details on some gems
+- - display gems that are in loaded with 0 coverage, thanks @kbaum
 
 ### Coverband_jam_session
 
@@ -54,32 +57,104 @@ This is a possible gem to host experimental or more complex features, which woul
 
 Feature Ideas:
 
+- per request coverage implemented via Ruby 2.6.0 coverage.clear https://bugs.ruby-lang.org/issues/15022
 - statsd adapters (it would allow passing in date ranges on usage)
-- move to SimpleCov console out, or make similar console tabular output
 - Possibly add ability to record code run for a given route
 - integrate recording with deploy tag or deploy timestamp
   - diff code usage across deployed versions
   - what methods increased usage or decreased
-- Improve client code api, around manual usage of sampling (like event usage)
 - ability to change the Coverband config at runtime by changing the config pushed to the Redis hash. In memory cache around the changes to only make that call periodically.
 - Opposed to just showing code usage on a route allow 'tagging' events which would record line coverage for that tag (this would allow tagging all code that modified an ActiveRecord model for example
 - additional adapters (tracepoint, ruby-profiler, etc)
-- code route tracing (entry point to all code executed for example /some_path -> code coverage of that path)
-- tagging of reported Coverage
-- allow only to collect coverage based on route (limiting or scoped coverage)
-- coverage over some other variable like a set of alpha users
+- tagged coverage reports
+  - code route tracing (entry point to all code executed for example /some_path -> code coverage of that path)
+  - deploy git hash tagging of reported Coverage
+  - allow only to collect coverage based on route (limiting or scoped coverage)
+  - coverage over some other variable like a set of alpha users
 - document how to use this on staging / selenium test suite runs
   - possible add API to pull report at end of run
 
-# Alpha
+# Alpha / Beta / Release Candidates
 
-### Coverband 4.3.0?
+### Coverband 4.2.5
 
-- further improvements on eager_loading detection
 - ?
 
 # Released
 
+### Coverband 4.2.4
+
+- fixes related to startup without Redis, skipping Coverband on common rake tasks (assets:precompile), etc
+- avoid calls to redis if coverage not updated
+- split out lua work and added testing around lua scripts use for the hash redis store
+- improvements to hash redis store resulting in 25% faster perf
+- avoid duplicate hash calculations
+- debug data now includes md5_hashes
+- added support to download coverage and view data in JSON format
+- documentation about working with environment variables
+- add cache wiggle to avoid Redis CPU spikes (cache stampede on Redis server)
+- make the nocov consistant on the data download and html view
+- small performance improvements
+
+### Coverband 4.2.3
+
+- readme fixes
+- improved defaults for the next redis hash adapter
+- add tooltips
+- improved messaging around non-loaded files
+- fix on last updated nil issue
+- view tracker improvements
+   - clear all
+   - reset individual file
+   - timestamps on last seen activity
+
+
+### Coverband 4.2.2
+
+- new experimental hash redis store for high volume collection
+(hundreds of clients), thanks @kbaum
+- view_tracker supports tracking view layer files like `.html.erb`
+- documentation improvements, thanks @brossetti1, @jjb, @kbaum
+- we now have discordapp for discussions, thanks @kbaum,
+https://discordapp.com/channels/609509533999562753/609509533999562756
+- perf fix on Rails initialization, thanks @skangg
+- simplified logging
+
+### Coverband 4.2.1
+
+- larger changes
+  - reduce memory usage
+  - fix issue where reports didn't include files with 0 activity
+  - updated runtime relavent lines and runtime percentages
+  - add Oneshot coverage support for Ruby 2.6.0 thanks @zwalker
+  - I would consider this our test oneshot release, please report any issues
+  - improved control over memory vs functionality
+    - oneshot support is the best memory and speed if you are on Ruby 2.6.\*
+    - simulated_oneshot works for Ruby prior to 2.6.\*, keeps lower runtime memory, at a trade of only being able to detect eager_load or runtime hits not both
+    - full hit tracking (The previous and current default), this uses more memory than other options, but provides full robust data
+
+* small changes fixes
+  - further improvements on eager_loading detection, thanks @kbaum
+  - fix on ignore configuration options
+  - fix broken static server
+  - fix issue where clear and coverage trigger coverage reports
+  - improved logging
+  - fix on gem runtime code coverage support, thanks @kbaum
+  - fix sorting of runtime coverage
+  - add runtime coverage to gem summary pages
+  - documented redis TTL, thanks @jjb
+  - resolve regression on coverband rake tasks recording coverage
+  - improved runtime / eager loading tracking
+  - fix on small memory leaks
+  - added warnings that gem tracking isn't production ready
+  - built in support for basic auth on the web rack app, thanks @jjb
+  - don't clobber rake environment method definition, thanks @shioyama
+  - readme improvements, thanks @yuriyalekseyev
+  - fix duplicate requires thanks @yuriyalekseyev
+  - various cleanups and improvements, thanks @kbaum
+  - additional benchmarks to ensure proper memory handling, thanks @kbaum
+  - default redis TTL, thanks @jjb
+
 ### Coverband 4.2.0
 
 **NOTE:** This release introduces load time and runtime Coverage. To get the full benifit of this feature you need to reset you coverage data (`rake coverband:clear` or clear via the web UI). Until you clear the combined result is still correct, but your runtime data will be mixed with previous data. This is due to an additive change in how we store coverage. We didn't reset by default as the coverage history for some folks may be more important than runtime breakdown.
@@ -96,7 +171,7 @@ Feature Ideas:
 - move from thread based reporter instance to process based instance
 - thanks Karl Baum for much of the above mentioned work
 - clear coverage on individual files
-- we have a logo, thanks Dave Woodall
+- we have a logo, thanks Dave Woodall, http://davewoodall.com
 
 ### Coverband 4.1.1
 

data/coverband.gemspec

@@ -28,17 +28,15 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency 'm'
   spec.add_development_dependency 'memory_profiler'
   spec.add_development_dependency 'minitest'
+  spec.add_development_dependency 'minitest-fork_executor'
   spec.add_development_dependency 'mocha', '~> 1.7.0'
   spec.add_development_dependency 'rack'
   spec.add_development_dependency 'rack-test'
   spec.add_development_dependency 'rake'
   spec.add_development_dependency 'resque'
-  # temporarily needed to fun tests for classifier-reborn as part of benchmarks
-  spec.add_development_dependency 'minitest-reporters'
+  spec.add_development_dependency 'rubocop'
+  spec.add_development_dependency 'rubocop-performance'
 
-  # TODO: used for benchmarking and tests I think we have other better benchmarking
-  # perhaps time to drop this and refactor.
-  spec.add_development_dependency 'classifier-reborn'
   spec.add_development_dependency 'coveralls'
   # add when debugging
   # require 'byebug'; byebug

data/lib/coverband/adapters/base.rb

@@ -3,45 +3,48 @@
 module Coverband
   module Adapters
     class Base
-      include Coverband::Utils::FilePathHelper
+      DATA_KEY = 'data'
+      FIRST_UPDATED_KEY = 'first_updated_at'
+      LAST_UPDATED_KEY = 'last_updated_at'
+      FILE_HASH = 'file_hash'
+      ABSTRACT_KEY = 'abstract'
+
       attr_accessor :type
 
       def initialize
-        @file_hash_cache = {}
+        @type = Coverband::RUNTIME_TYPE
       end
 
       def clear!
-        raise 'abstract'
+        raise ABSTRACT_KEY
       end
 
-      def clear_file!(_file)
-        raise 'abstract'
+      def clear_file!(_file = nil)
+        raise ABSTRACT_KEY
       end
 
       def migrate!
-        raise 'abstract'
+        raise ABSTRACT_KEY
       end
 
       def size
-        raise 'abstract'
+        raise ABSTRACT_KEY
       end
 
-      def size_in_mib
-        format('%.2f', (size.to_f / 2**20))
+      def save_coverage
+        raise ABSTRACT_KEY
+      end
+
+      def coverage(_local_type = nil)
+        raise ABSTRACT_KEY
       end
 
-      # Note: This could lead to slight race on redis
-      # where multiple processes pull the old coverage and add to it then push
-      # the Coverband 2 had the same issue,
-      # and the tradeoff has always been acceptable
-      def save_report(report)
-        data = report.dup
-        data = merge_reports(data, get_report)
-        save_coverage(data)
+      def size_in_mib
+        format('%<size>.2f', size: (size.to_f / 2**20))
       end
 
-      def coverage
-        get_report
+      def save_report(_report)
+        raise 'abstract'
       end
 
       def get_coverage_report
@@ -53,60 +56,69 @@ module Coverband
         coverage.keys || []
       end
 
-      # TODO: deprecate / remove?
-      def covered_lines_for_file(file)
-        Array(coverage.dig(file, 'data'))
+      def raw_store
+        raise ABSTRACT_KEY
       end
 
       protected
 
       def split_coverage(types)
         types.reduce({}) do |data, type|
-          data.update(type => get_report(type))
+          if type == Coverband::RUNTIME_TYPE && Coverband.configuration.simulate_oneshot_lines_coverage
+            data.update(type => simulated_runtime_coverage)
+          else
+            data.update(type => coverage(type))
+          end
         end
       end
 
-      def merged_coverage(types)
-        types.reduce({}) do |data, type|
-          merge_reports(data, get_report(type), skip_expansion: true)
+      def simulated_runtime_coverage
+        runtime_data = coverage(Coverband::RUNTIME_TYPE)
+        eager_data = coverage(Coverband::EAGER_TYPE)
+        eager_data.values do |vals|
+          vals['data'].map! { |line_coverage| line_coverage ? (0 - line_coverage) : line_coverage }
         end
+        merge_reports(runtime_data, eager_data, skip_expansion: true)
       end
 
-      def save_coverage
-        raise 'abstract'
-      end
-
-      def get_report
-        raise 'abstract'
+      def merged_coverage(types)
+        types.reduce({}) do |data, type|
+          merge_reports(data, coverage(type), skip_expansion: true)
+        end
       end
 
       def file_hash(file)
-        @file_hash_cache[file] ||= Digest::MD5.file(file).hexdigest
+        Coverband::Utils::FileHasher.hash(file)
       end
 
+      # TODO: modify to extend report inline?
       def expand_report(report)
         expanded = {}
         report_time = Time.now.to_i
-        updated_time = self.type == Coverband::EAGER_TYPE ? nil : report_time
+        updated_time = type == Coverband::EAGER_TYPE ? nil : report_time
         report.each_pair do |key, line_data|
           extended_data = {
-            'first_updated_at' => report_time,
-            'last_updated_at' => updated_time,
-            'file_hash' => file_hash(key),
-            'data' => line_data
+            FIRST_UPDATED_KEY => report_time,
+            LAST_UPDATED_KEY => updated_time,
+            FILE_HASH => file_hash(key),
+            DATA_KEY => line_data
           }
-          expanded[full_path_to_relative(key)] = extended_data
+          expanded[Utils::RelativeFileConverter.convert(key)] = extended_data
         end
         expanded
       end
 
       def merge_reports(new_report, old_report, options = {})
+        # transparently update from RUNTIME_TYPE = nil to RUNTIME_TYPE = :runtime
+        # transparent update for format coveband_3_2
+        old_report = coverage(nil, override_type: nil) if old_report.nil? && type == Coverband::RUNTIME_TYPE
+
         new_report = expand_report(new_report) unless options[:skip_expansion]
         keys = (new_report.keys + old_report.keys).uniq
         keys.each do |file|
           new_report[file] = if new_report[file] &&
                                 old_report[file] &&
-                                new_report[file]['file_hash'] == old_report[file]['file_hash']
+                                new_report[file][FILE_HASH] == old_report[file][FILE_HASH]
                                merge_expanded_data(new_report[file], old_report[file])
                              elsif new_report[file]
                                new_report[file]
@@ -119,15 +131,22 @@ module Coverband
 
       def merge_expanded_data(new_expanded, old_expanded)
         {
-          'first_updated_at' => old_expanded['first_updated_at'],
-          'last_updated_at' => new_expanded['last_updated_at'],
-          'file_hash' => new_expanded['file_hash'],
-          'data' => array_add(new_expanded['data'], old_expanded['data'])
+          FIRST_UPDATED_KEY => old_expanded[FIRST_UPDATED_KEY],
+          LAST_UPDATED_KEY => new_expanded[LAST_UPDATED_KEY],
+          FILE_HASH => new_expanded[FILE_HASH],
+          DATA_KEY => array_add(new_expanded[DATA_KEY], old_expanded[DATA_KEY])
         }
       end
 
+      # TODO: This should only be 2 cases get our dup / not dups aligned
       def array_add(latest, original)
-        latest.map.with_index { |v, i| (v && original[i]) ? v + original[i] : nil }
+        if Coverband.configuration.use_oneshot_lines_coverage
+          latest.map!.with_index { |v, i| (v + original[i] >= 1 ? 1 : 0) if v && original[i] }
+        elsif Coverband.configuration.simulate_oneshot_lines_coverage
+          latest.map.with_index { |v, i| (v + original[i] >= 1 ? 1 : 0) if v && original[i] }
+        else
+          latest.map.with_index { |v, i| (v && original[i]) ? v + original[i] : nil }
+        end
       end
     end
   end

data/lib/coverband/adapters/file_store.rb

@@ -28,21 +28,27 @@ module Coverband
         raise NotImplementedError, "FileStore doesn't support migrations"
       end
 
-      private
-
-      attr_accessor :path
-
-      def save_coverage(report)
-        File.open(path, 'w') { |f| f.write(report.to_json) }
-      end
-
-      def get_report
+      def coverage(_local_type = nil)
         if File.exist?(path)
           JSON.parse(File.read(path))
         else
           {}
         end
       end
+
+      def save_report(report)
+        data = report.dup
+        data = merge_reports(data, coverage)
+        File.open(path, 'w') { |f| f.write(data.to_json) }
+      end
+
+      def raw_store
+        raise NotImplementedError, "FileStore doesn't support raw_store"
+      end
+
+      private
+
+      attr_accessor :path
     end
   end
 end

data/lib/coverband/adapters/hash_redis_store.rb

@@ -0,0 +1,190 @@
+# frozen_string_literal: true
+
+require 'securerandom'
+
+module Coverband
+  module Adapters
+    class HashRedisStore < Base
+      FILE_KEY = 'file'
+      FILE_LENGTH_KEY = 'file_length'
+      META_DATA_KEYS = [DATA_KEY, FIRST_UPDATED_KEY, LAST_UPDATED_KEY, FILE_HASH].freeze
+      ###
+      # This key isn't related to the coverband version, but to the interal format
+      # used to store data to redis. It is changed only when breaking changes to our
+      # redis format are required.
+      ###
+      REDIS_STORAGE_FORMAT_VERSION = 'coverband_hash_3_3'
+
+      JSON_PAYLOAD_EXPIRATION = 5 * 60
+
+      attr_reader :redis_namespace
+
+      def initialize(redis, opts = {})
+        super()
+        @redis_namespace = opts[:redis_namespace]
+        @save_report_batch_size = opts[:save_report_batch_size] || 100
+        @format_version = REDIS_STORAGE_FORMAT_VERSION
+        @redis = redis
+        raise 'HashRedisStore requires redis >= 2.6.0' unless supported?
+
+        @ttl = opts[:ttl]
+        @relative_file_converter = opts[:relative_file_converter] || Utils::RelativeFileConverter
+      end
+
+      def supported?
+        Gem::Version.new(@redis.info['redis_version']) >= Gem::Version.new('2.6.0')
+      end
+
+      def clear!
+        old_type = type
+        Coverband::TYPES.each do |type|
+          self.type = type
+          file_keys = files_set
+          @redis.del(*file_keys) if file_keys.any?
+          @redis.del(files_key)
+        end
+        self.type = old_type
+      end
+
+      def clear_file!(file)
+        file_hash = file_hash(file)
+        relative_path_file = @relative_file_converter.convert(file)
+        Coverband::TYPES.each do |type|
+          @redis.del(key(relative_path_file, type, file_hash: file_hash))
+        end
+        @redis.srem(files_key, relative_path_file)
+      end
+
+      def save_report(report)
+        report_time = Time.now.to_i
+        updated_time = type == Coverband::EAGER_TYPE ? nil : report_time
+        keys = []
+        report.each_slice(@save_report_batch_size) do |slice|
+          files_data = slice.map do |(file, data)|
+            relative_file = @relative_file_converter.convert(file)
+            file_hash = file_hash(relative_file)
+            key = key(relative_file, file_hash: file_hash)
+            keys << key
+            script_input(
+              key: key,
+              file: relative_file,
+              file_hash: file_hash,
+              data: data,
+              report_time: report_time,
+              updated_time: updated_time
+            )
+          end
+          next unless files_data.any?
+
+          arguments_key = [@redis_namespace, SecureRandom.uuid].compact.join('.')
+          @redis.set(arguments_key, { ttl: @ttl, files_data: files_data }.to_json, ex: JSON_PAYLOAD_EXPIRATION)
+          @redis.evalsha(hash_incr_script, [arguments_key])
+        end
+        @redis.sadd(files_key, keys) if keys.any?
+      end
+
+      def coverage(local_type = nil)
+        files_set = files_set(local_type)
+        @redis.pipelined do
+          files_set.map do |key|
+            @redis.hgetall(key)
+          end
+        end.each_with_object({}) do |data_from_redis, hash|
+          add_coverage_for_file(data_from_redis, hash)
+        end
+      end
+
+      def raw_store
+        @redis
+      end
+
+      def size
+        'not available'
+      end
+
+      def size_in_mib
+        'not available'
+      end
+
+      private
+
+      def add_coverage_for_file(data_from_redis, hash)
+        return if data_from_redis.empty?
+
+        file = data_from_redis[FILE_KEY]
+        return unless file_hash(file) == data_from_redis[FILE_HASH]
+
+        data = coverage_data_from_redis(data_from_redis)
+        hash[file] = data_from_redis.select { |meta_data_key, _value| META_DATA_KEYS.include?(meta_data_key) }.merge!('data' => data)
+        hash[file][LAST_UPDATED_KEY] = hash[file][LAST_UPDATED_KEY].blank? ? nil : hash[file][LAST_UPDATED_KEY].to_i
+        hash[file].merge!(LAST_UPDATED_KEY => hash[file][LAST_UPDATED_KEY], FIRST_UPDATED_KEY => hash[file][FIRST_UPDATED_KEY].to_i)
+      end
+
+      def coverage_data_from_redis(data_from_redis)
+        max = data_from_redis[FILE_LENGTH_KEY].to_i - 1
+        Array.new(max + 1) do |index|
+          line_coverage = data_from_redis[index.to_s]
+          line_coverage.nil? ? nil : line_coverage.to_i
+        end
+      end
+
+      def script_input(key:, file:, file_hash:, data:, report_time:, updated_time:)
+        coverage_data = data.each_with_index.each_with_object({}) do |(coverage, index), hash|
+          hash[index] = coverage if coverage
+        end
+        meta = {
+          first_updated_at: report_time,
+          file: file,
+          file_hash: file_hash,
+          file_length: data.length,
+          hash_key: key
+        }
+        meta[:last_updated_at] = updated_time if updated_time
+        {
+          hash_key: key,
+          meta: meta,
+          coverage: coverage_data
+        }
+      end
+
+      def hash_incr_script
+        @hash_incr_script ||= @redis.script(:load, lua_script_content)
+      end
+
+      def lua_script_content
+        File.read(File.join(
+                    File.dirname(__FILE__), '../../../lua/lib/persist-coverage.lua'
+                  ))
+      end
+
+      def values_from_redis(local_type, files)
+        return files if files.empty?
+
+        @redis.mget(*files.map { |file| key(file, local_type) }).map do |value|
+          value.nil? ? {} : JSON.parse(value)
+        end
+      end
+
+      def relative_paths(files)
+        files&.map! { |file| full_path_to_relative(file) }
+      end
+
+      def files_set(local_type = nil)
+        @redis.smembers(files_key(local_type))
+      end
+
+      def files_key(local_type = nil)
+        "#{key_prefix(local_type)}.files"
+      end
+
+      def key(file, local_type = nil, file_hash:)
+        [key_prefix(local_type), file, file_hash].join('.')
+      end
+
+      def key_prefix(local_type = nil)
+        local_type ||= type
+        [@format_version, @redis_namespace, local_type].compact.join('.')
+      end
+    end
+  end
+end

data/lib/coverband/adapters/redis_store.rb

@@ -13,23 +13,31 @@ module Coverband
       ###
       REDIS_STORAGE_FORMAT_VERSION = 'coverband_3_2'
 
+      attr_reader :redis_namespace
+
       def initialize(redis, opts = {})
         super()
         @redis           = redis
         @ttl             = opts[:ttl]
         @redis_namespace = opts[:redis_namespace]
         @format_version  = REDIS_STORAGE_FORMAT_VERSION
+        @keys            = {}
+        Coverband::TYPES.each do |type|
+          @keys[type] = [@format_version, @redis_namespace, type].compact.join('.')
+        end
       end
 
       def clear!
         Coverband::TYPES.each do |type|
           @redis.del(type_base_key(type))
         end
+        # temporarily clear the old namespace of coverband_3_2
+        @redis.del(type_base_key(nil))
       end
 
       def clear_file!(filename)
         Coverband::TYPES.each do |type|
-          data = get_report(type)
+          data = coverage(type)
           data.delete(filename)
           save_coverage(data, type)
         end
@@ -47,18 +55,18 @@ module Coverband
       def migrate!
         reset_base_key
         @format_version = 'coverband3_1'
-        previous_data = get_report
+        previous_data = coverage
         if previous_data.empty?
           puts 'no previous data to migrate found'
           exit 0
         end
         relative_path_report = previous_data.each_with_object({}) do |(key, vals), fixed_report|
-          fixed_report[full_path_to_relative(key)] = vals
+          fixed_report[Utils::RelativeFileConverter.convert(key)] = vals
         end
         clear!
         reset_base_key
         @format_version = REDIS_STORAGE_FORMAT_VERSION
-        save_coverage(merge_reports(get_report, relative_path_report, skip_expansion: true))
+        save_coverage(merge_reports(coverage, relative_path_report, skip_expansion: true))
       end
 
       def type=(type)
@@ -66,6 +74,28 @@ module Coverband
         reset_base_key
       end
 
+      def coverage(local_type = nil, opts = {})
+        local_type ||= opts.key?(:override_type) ? opts[:override_type] : type
+        data = redis.get type_base_key(local_type)
+        data = data ? JSON.parse(data) : {}
+        data.delete_if { |file_path, file_data| file_hash(file_path) != file_data['file_hash'] } unless opts[:skip_hash_check]
+        data
+      end
+
+      # Note: This could lead to slight race on redis
+      # where multiple processes pull the old coverage and add to it then push
+      # the Coverband 2 had the same issue,
+      # and the tradeoff has always been acceptable
+      def save_report(report)
+        data = report.dup
+        data = merge_reports(data, coverage(nil, skip_hash_check: true))
+        save_coverage(data)
+      end
+
+      def raw_store
+        @redis
+      end
+
       private
 
       attr_reader :redis
@@ -79,7 +109,7 @@ module Coverband
       end
 
       def type_base_key(local_type)
-        [@format_version, @redis_namespace, local_type].compact.join('.')
+        @keys[local_type]
       end
 
       def save_coverage(data, local_type = nil)
@@ -87,12 +117,6 @@ module Coverband
         redis.set type_base_key(local_type), data.to_json
         redis.expire(type_base_key(local_type), @ttl) if @ttl
       end
-
-      def get_report(local_type = nil)
-        local_type ||= type
-        data = redis.get type_base_key(local_type)
-        data ? JSON.parse(data) : {}
-      end
     end
   end
 end