churn_vs_complexity 1.2.0 → 1.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d748bfb7d21087a5fbcb360dbc05dcd46b5ec05bf2c2fcc542e2e25368084d78
-  data.tar.gz: 0d160c582aee2696823061dd7623cf17e9b537d798ae31502eb30b065a61391d
+  metadata.gz: 7484ff3a1c015738808226a78087017f6b7aff5ce42d15879023f32df5648717
+  data.tar.gz: ad3bdeff5ba32e9d7f414b45173d8928b1c463313557202b96aaf5fdcf059109
 SHA512:
-  metadata.gz: f521d39fddad8e8adf05ab0f21d8b8adcc2460f9a57c553b90c73eb60451aafed19ab11977768d494987ff72838c856dd04c3910781ec0da34f53d82d18d9d4e
-  data.tar.gz: 3ab17db013540128bd428edb54cf0615f47ce50caa19d1ad418261c20efde494bb36ea8e84408e0ca330f7721013a9e8958f3cd5a16565636186508e59bf7360
+  metadata.gz: a56e26296acfff22e755c414cab9ee36923e4ccd7181d67691f55feb8f99b5aa738f5d2b7645006f25c632efa346fb535d7fe04923443dedf607772ce1a21323
+  data.tar.gz: f406ee696facf7708e792b67ed8b463f1554365b3939deb4bde40c9fab88dfe891997f611727559a283103c5b7234f71d78d5fe108731628cb3fd8a782a73ce5

data/CHANGELOG.md

@@ -12,3 +12,14 @@
 - Fix bug in CLI where new flags and `--since` would not be recognized
 - Improve selection of observations included in the output
 - Fixed calculation of churn that would never be zero
+
+## [1.3.0] - 2024-09-26
+
+- Add support for javascript and typescript complexity calculation using eslint
+- Fixed behavior when --since or short-hand flags were not provided
+
+## [1.4.0] - 2024-10-10
+
+- Add timetravel mode to visualise code quality over time
+- Add alpha, beta, and gamma scores to summaries
+- Fixed broken Ruby complexity calculation

data/README.md

@@ -4,7 +4,7 @@ A tool to visualise code complexity in a project and help direct refactoring eff
 
 Inspired by [Michael Feathers' article "Getting Empirical about Refactoring"](https://www.agileconnection.com/article/getting-empirical-about-refactoring) and the gem [turbulence](https://rubygems.org/gems/turbulence) by Chad Fowler and others.
 
-This gem was built primarily to support analysis of Java and Ruby repositories, but it can easily be extended.
+This gem currently supports analysis of Java, Ruby, JavaScript, and TypeScript repositories, but it can easily be extended.
 
 ## Installation
 
@@ -34,17 +34,33 @@ Execute the `churn_vs_complexity` with the applicable arguments. Output in the r
 Usage: churn_vs_complexity [options] folder
         --java                       Check complexity of java classes
         --ruby                       Check complexity of ruby files
+        --js, --ts, --javascript, --typescript
+                                     Check complexity of javascript and typescript files
         --csv                        Format output as CSV
         --graph                      Format output as HTML page with Churn vs Complexity graph
         --summary                    Output summary statistics (mean and median) for churn and complexity
         --excluded PATTERN           Exclude file paths including this string. Can be used multiple times.
-        --since YYYY-MM-DD           Calculate churn after this date
+        --since YYYY-MM-DD           Normal mode: Calculate churn after this date. Timetravel mode: calculate summaries from this date
     -m, --month                      Calculate churn for the month leading up to the most recent commit
     -q, --quarter                    Calculate churn for the quarter leading up to the most recent commit
     -y, --year                       Calculate churn for the year leading up to the most recent commit
+        --timetravel N               Calculate summary for all commits at intervals of N days throughout project history or from the date specified with --since
+        --dry-run                    Echo the chosen options from the CLI
     -h, --help                       Display help
+
+
 ```
 
+Note that when using the `--timetravel` mode, the semantics of some flags are subtly different from normal mode:
+
+* `--since YYYY-MM-DD`: Calculate summaries from this date
+* `--month`, `--quarter`, `--year`: Calculate churn for the period leading up to each commit being summarised
+
+Timetravel analysis can take many minutes for old and large repositories.
+
+Summaries in normal mode include a gamma score, which is an unnormalised harmonic mean of churn and complexity. This allows for comparison of summaries across different projects with the same language, or over time for a single project.
+
+Summary points in timetravel mode instead include an alpha score, which is the same harmonic mean of churn and complexity, where churn and complexity values are normalised to a 0-1 range to avoid either churn or complexity dominating the score. The summary points also include a beta score, which is the geometric mean of the normalised churn and complexity values.
 ## Examples
 
 `churn_vs_complexity --ruby --csv my_ruby_project > ~/Desktop/ruby-demo.csv`
@@ -53,6 +69,7 @@ Usage: churn_vs_complexity [options] folder
 
 `churn_vs_complexity --ruby --summary -m my_ruby_project >> ~/Desktop/monthly-report.txt`
 
+`churn_vs_complexity --java -m --since 2019-03-01 --timetravel 30 --graph my_java_project > ~/Desktop/timetravel-after-1st-march-2019.html`
 
 ## Development
 

data/TODO

@@ -0,0 +1,11 @@
+TODO:
+
+- Move Timetravel calculations away from serializer
+
+- Database, where we can prepopulate the state of every file and every commit
+  - Populate incrementally from each commit in log. 
+  - Only need to care about deltas between commits, and copy everything else from previous commit.
+  - processed_commit table with sha and version of processing logic
+- Unit tests for simpler new classes
+- Integration test for Timetravel
+- Candlebars on mean dots in graph

data/bin/churn_vs_complexity

@@ -6,6 +6,11 @@ require 'churn_vs_complexity'
 
 begin
   ChurnVsComplexity::CLI.run!
+rescue ChurnVsComplexity::ValidationError => e
+  warn e.message
+  exit 1
 rescue StandardError => e
+  warn e.backtrace
   warn e.message
+  exit 2
 end

data/lib/churn_vs_complexity/churn.rb

@@ -8,8 +8,9 @@ module ChurnVsComplexity
       class << self
         def calculate(folder:, file:, since:)
           git_dir = File.join(folder, '.git')
-          formatted_date = since.strftime('%Y-%m-%d')
-          cmd = %Q(git --git-dir #{git_dir} --work-tree #{folder} log --format="%H" --follow --since="#{formatted_date}" -- #{file} | wc -l)
+          earliest_date = [date_of_first_commit(folder:), since].max
+          formatted_date = earliest_date.strftime('%Y-%m-%d')
+          cmd = %(git --git-dir #{git_dir} --work-tree #{folder} log --format="%H" --follow --since="#{formatted_date}" -- #{file} | wc -l)
           `#{cmd}`.to_i
         end
 
@@ -19,6 +20,10 @@ module ChurnVsComplexity
 
         private
 
+        def date_of_first_commit(folder:)
+          repo(folder).log.last&.date&.to_date || Time.at(0).to_date
+        end
+
         def repo(folder)
           repos[folder] ||= Git.open(folder)
         end

data/lib/churn_vs_complexity/cli.rb

@@ -22,6 +22,11 @@ module ChurnVsComplexity
           options[:language] = :ruby
         end
 
+        opts.on('--js', '--ts', '--javascript', '--typescript',
+                'Check complexity of javascript and typescript files',) do
+          options[:language] = :javascript
+        end
+
         opts.on('--csv', 'Format output as CSV') do
           options[:serializer] = :csv
         end
@@ -39,20 +44,27 @@ module ChurnVsComplexity
           options[:excluded] << value
         end
 
-        opts.on('--since YYYY-MM-DD', 'Calculate churn after this date') do |value|
+        opts.on('--since YYYY-MM-DD',
+                'Normal mode: Calculate churn after this date. Timetravel mode: calculate summaries from this date',) do |value|
           options[:since] = value
         end
 
         opts.on('-m', '--month', 'Calculate churn for the month leading up to the most recent commit') do
-          options[:since] = :month
+          options[:relative_period] = :month
         end
 
         opts.on('-q', '--quarter', 'Calculate churn for the quarter leading up to the most recent commit') do
-          options[:since] = :quarter
+          options[:relative_period] = :quarter
         end
 
         opts.on('-y', '--year', 'Calculate churn for the year leading up to the most recent commit') do
-          options[:since] = :year
+          options[:relative_period] = :year
+        end
+
+        opts.on('--timetravel N',
+                'Calculate summary for all commits at intervals of N days throughout project history or from the date specified with --since',) do |value|
+          options[:mode] = :timetravel
+          options[:jump_days] = value.to_i
         end
 
         opts.on('--dry-run', 'Echo the chosen options from the CLI') do
@@ -80,7 +92,11 @@ module ChurnVsComplexity
 
       config.validate!
 
-      puts config.to_engine.check(folder:)
+      if options[:mode] == :timetravel
+        puts config.timetravel.go(folder:)
+      else
+        puts config.to_engine.check(folder:)
+      end
     end
   end
 end

data/lib/churn_vs_complexity/complexity/eslint_calculator.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+module ChurnVsComplexity
+  module Complexity
+    module ESLintCalculator
+      class << self
+        def folder_based? = false
+
+        def calculate(files:)
+          dir_path = File.join(gem_root, 'tmp', 'eslint-support')
+          script_path = File.join(dir_path, 'complexity-calculator.js')
+          install_command = "npm install --prefix '#{dir_path}'"
+          `#{install_command}`
+
+          command = "node #{script_path} '#{files.to_json}'"
+          complexity = `#{command}`
+
+          raise Error, 'Failed to calculate complexity' if complexity.empty?
+
+          all = JSON.parse(complexity)
+          all.to_h do |abc|
+            [abc['file'], abc['complexity']]
+          end
+        end
+
+        private
+
+        def gem_root
+          File.expand_path('../../..', __dir__)
+        end
+      end
+    end
+  end
+end

data/lib/churn_vs_complexity/complexity/flog_calculator.rb

@@ -5,15 +5,16 @@ require 'flog'
 module ChurnVsComplexity
   module Complexity
     module FlogCalculator
-      CONCURRENCY = Etc.nprocessors
-
       class << self
         def folder_based? = false
 
-        def calculate(file:)
-          flog = Flog.new
-          flog.flog(file)
-          { file => flog.total_score }
+        def calculate(files:)
+          #  TODO: Run this concurrently
+          files.to_h do |file|
+            flog = Flog.new
+            flog.flog(file)
+            [file, flog.total_score]
+          end
         end
       end
     end

data/lib/churn_vs_complexity/complexity/pmd_calculator.rb

@@ -9,7 +9,10 @@ module ChurnVsComplexity
         def folder_based? = true
 
         def calculate(folder:)
-          output = `pmd check -d #{folder} -R #{resolve_ruleset_path} -f json -t #{CONCURRENCY}  --cache #{resolve_cache_path}`
+          cache_path = resolve_cache_path
+          output = `pmd check -d #{folder} -R #{resolve_ruleset_path} -f json -t #{CONCURRENCY}  --cache #{cache_path}`
+          File.delete(cache_path)
+
           Parser.new.parse(output)
         end
 
@@ -29,7 +32,7 @@ module ChurnVsComplexity
         end
 
         def resolve_cache_path
-          File.join(gem_root, 'tmp', 'pmd-support', 'pmd-cache')
+          File.join(gem_root, 'tmp', 'pmd-support', "pmd-cache-#{Process.pid}")
         end
 
         def gem_root

data/lib/churn_vs_complexity/complexity.rb

@@ -2,6 +2,7 @@
 
 require_relative 'complexity/pmd_calculator'
 require_relative 'complexity/flog_calculator'
+require_relative 'complexity/eslint_calculator'
 
 module ChurnVsComplexity
   module Complexity

data/lib/churn_vs_complexity/concurrent_calculator.rb

@@ -30,9 +30,7 @@ module ChurnVsComplexity
           files[:explicitly_excluded].each { |file| result.delete(file) }
           result
         else
-          files[:included].each_with_object({}) do |file, acc|
-            acc.merge!(@complexity.calculate(file:))
-          end
+          @complexity.calculate(files: files[:included])
         end
     end
 
@@ -59,7 +57,7 @@ module ChurnVsComplexity
     def combine_results
       result = {}
       result[:values_by_file] = @complexity_results.keys.each_with_object({}) do |file, acc|
-        # File with complexity score might not have churned in queried period, 
+        # File with complexity score might not have churned in queried period,
         # set zero churn on miss
         acc[file] = [@churn_results[file] || 0, @complexity_results[file]]
       end

data/lib/churn_vs_complexity/config.rb

@@ -7,25 +7,42 @@ module ChurnVsComplexity
       serializer:,
       excluded: [],
       since: nil,
+      relative_period: nil,
       complexity_validator: ComplexityValidator,
-      since_validator: SinceValidator
+      since_validator: SinceValidator,
+      **options
     )
       @language = language
       @serializer = serializer
       @excluded = excluded
       @since = since
+      @relative_period = relative_period
       @complexity_validator = complexity_validator
       @since_validator = since_validator
+      @options = options
     end
 
     def validate!
-      raise Error, "Unsupported language: #{@language}" unless %i[java ruby].include?(@language)
-      raise Error, "Unsupported serializer: #{@serializer}" unless %i[none csv graph summary].include?(@serializer)
+      raise ValidationError, "Unsupported language: #{@language}" unless %i[java ruby javascript].include?(@language)
 
-      @since_validator.validate!(@since)
+      SerializerValidator.validate!(serializer: @serializer, mode: @options[:mode])
+
+      @since_validator.validate!(since: @since, relative_period: @relative_period, mode: @options[:mode])
+      RelativePeriodValidator.validate!(relative_period: @relative_period, mode: @options[:mode])
       @complexity_validator.validate!(@language)
     end
 
+    def timetravel
+      engine = timetravel_engine_config.to_engine
+      Timetravel::Traveller.new(
+        since: @since,
+        relative_period: @relative_period,
+        engine:,
+        jump_days: @options[:jump_days],
+        serializer: @serializer,
+      )
+    end
+
     def to_engine
       case @language
       when :java
@@ -34,7 +51,7 @@ module ChurnVsComplexity
           churn:,
           file_selector: FileSelector::Java.excluding(@excluded),
           serializer:,
-          since: @since,
+          since: @since || @relative_period,
         )
       when :ruby
         Engine.concurrent(
@@ -42,13 +59,34 @@ module ChurnVsComplexity
           churn:,
           file_selector: FileSelector::Ruby.excluding(@excluded),
           serializer:,
-          since: @since,
+          since: @since || @relative_period,
+        )
+      when :javascript
+        Engine.concurrent(
+          complexity: Complexity::ESLintCalculator,
+          churn:,
+          file_selector: FileSelector::JavaScript.excluding(@excluded),
+          serializer:,
+          since: @since || @relative_period,
         )
       end
     end
 
     private
 
+    def timetravel_engine_config
+      Config.new(
+        language: @language,
+        serializer: :pass_through,
+        excluded: @excluded,
+        since: nil, # since has a different meaning in timetravel mode
+        relative_period: @relative_period,
+        complexity_validator: @complexity_validator,
+        since_validator: @since_validator,
+        **@options,
+      )
+    end
+
     def churn = Churn::GitCalculator
 
     def serializer
@@ -61,6 +99,8 @@ module ChurnVsComplexity
         Serializer::Graph.new
       when :summary
         Serializer::Summary
+      when :pass_through
+        Serializer::PassThrough
       end
     end
 
@@ -73,21 +113,45 @@ module ChurnVsComplexity
       end
     end
 
+    # TODO: unit test
+    module SerializerValidator
+      def self.validate!(serializer:, mode:)
+        raise ValidationError, "Unsupported serializer: #{serializer}" \
+          unless %i[none csv graph summary].include?(serializer)
+        raise ValidationError, 'Does not support --summary in --timetravel mode' \
+         if serializer == :summary && mode == :timetravel
+      end
+    end
+
+    # TODO: unit test
+    module RelativePeriodValidator
+      def self.validate!(relative_period:, mode:)
+        if mode == :timetravel && relative_period.nil?
+          raise ValidationError,
+                'Relative period is required in timetravel mode'
+        end
+        return if relative_period.nil? || %i[month quarter year].include?(relative_period)
+
+        raise ValidationError, "Invalid relative period #{relative_period}"
+      end
+    end
+
     module SinceValidator
-      def self.validate!(since)
+      def self.validate!(since:, relative_period:, mode:)
         # since can be nil, a date string or a keyword (:month, :quarter, :year)
         return if since.nil?
 
-        if since.is_a?(Symbol)
-          raise Error, "Invalid since value #{since}" unless %i[month quarter year].include?(since)
-        elsif since.is_a?(String)
-          begin
-            Date.strptime(since, '%Y-%m-%d')
-          rescue StandardError
-            raise Error, "Invalid date #{since}, please use correct format, YYYY-MM-DD"
-          end
-        else
-          raise Error, "Invalid since value #{since}"
+        unless mode == :timetravel || since.nil? || relative_period.nil?
+          raise ValidationError,
+                '--since and relative period (--month, --quarter, --year) can only be used together in --timetravel mode'
+        end
+
+        raise ValidationError, "Invalid since value #{since}" unless since.is_a?(String)
+
+        begin
+          Date.strptime(since, '%Y-%m-%d')
+        rescue Date::Error
+          raise ValidationError, "Invalid date #{since}, please use correct format, YYYY-MM-DD"
         end
       end
     end

data/lib/churn_vs_complexity/file_selector.rb

@@ -10,9 +10,10 @@ module ChurnVsComplexity
     end
 
     class Excluding
-      def initialize(extensions, excluded)
+      def initialize(extensions, excluded, convert_to_absolute_path = false)
         @extensions = extensions
         @excluded = excluded
+        @convert_to_absolute_path = convert_to_absolute_path
       end
 
       def select_files(folder)
@@ -25,6 +26,10 @@ module ChurnVsComplexity
             were_included << f
           end
         end
+        if @convert_to_absolute_path
+          were_excluded.map! { |f| File.absolute_path(f) }
+          were_included.map! { |f| File.absolute_path(f) }
+        end
         { explicitly_excluded: were_excluded, included: were_included }
       end
 
@@ -50,5 +55,11 @@ module ChurnVsComplexity
         Excluding.new(['.rb'], excluded)
       end
     end
+
+    module JavaScript
+      def self.excluding(excluded)
+        Excluding.new(['.js', '.jsx', '.ts', '.tsx'], excluded, true)
+      end
+    end
   end
 end

data/lib/churn_vs_complexity/git_date.rb

@@ -2,6 +2,13 @@
 
 module ChurnVsComplexity
   module GitDate
+    def self.select_dates_with_at_least_interval(dates, interval)
+      ds = dates.sort
+      ds.each_with_object([]) do |date, acc|
+        acc << date if acc.empty? || date - acc.last >= interval
+      end
+    end
+
     def self.git_period(cli_arg_since, latest_commit_date)
       latest_commit_date = latest_commit_date.to_date
       if cli_arg_since.nil?
@@ -22,7 +29,7 @@ module ChurnVsComplexity
         @end_date = end_date
       end
 
-      def effective_start_date = Time.at(0)
+      def effective_start_date = Time.at(0).to_date
 
       def requested_start_date = nil
     end

data/lib/churn_vs_complexity/serializer/csv.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+module ChurnVsComplexity
+  module Serializer
+    module CSV
+      def self.serialize(result)
+        values_by_file = result[:values_by_file]
+        values_by_file.map do |file, values|
+          "#{file},#{values[0]},#{values[1]}\n"
+        end.join
+      end
+    end
+  end
+end

data/lib/churn_vs_complexity/serializer/graph.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module ChurnVsComplexity
+  module Serializer
+    class Graph
+      def initialize(template: Graph.load_template_file)
+        @template = template
+      end
+
+      def serialize(result)
+        data = result[:values_by_file].map do |file, values|
+          "{ file_path: '#{file}', churn: #{values[0]}, complexity: #{values[1]} }"
+        end.join(",\n") + "\n"
+        title = Serializer.title(result)
+        @template.gsub("// INSERT DATA\n", data).gsub('INSERT TITLE', title)
+      end
+
+      def self.load_template_file
+        file_path = File.expand_path('../../../tmp/template/graph.html', __dir__)
+        File.read(file_path)
+      end
+    end
+  end
+end

data/lib/churn_vs_complexity/serializer/pass_through.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+module ChurnVsComplexity
+  module Serializer
+    module PassThrough
+      class << self
+        def serialize(result)
+          values_by_file = result[:values_by_file]
+          end_date = result[:git_period].end_date
+          values = values_by_file.map do |_, values|
+            [values[0].to_f, values[1].to_f]
+          end
+          {
+            end_date:,
+            values:,
+          }
+        end
+      end
+    end
+  end
+end

data/lib/churn_vs_complexity/serializer/summary.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+module ChurnVsComplexity
+  module Serializer
+    module Summary
+      def self.serialize(result)
+        values_by_file = result[:values_by_file]
+        summary = SummaryHash.serialize(result)
+
+        <<~SUMMARY
+          #{Serializer.title(result)}
+
+          Number of observations: #{values_by_file.size}
+
+          Churn:
+          Mean #{summary[:mean_churn]}, Median #{summary[:median_churn]}
+
+          Complexity:
+          Mean #{summary[:mean_complexity]}, Median #{summary[:median_complexity]}
+
+          Gamma score:
+          Mean #{summary[:mean_gamma_score]}, Median #{summary[:median_gamma_score]}
+        SUMMARY
+      end
+    end
+  end
+end

data/lib/churn_vs_complexity/serializer/summary_hash.rb

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+module ChurnVsComplexity
+  module Serializer
+    module SummaryHash
+      class << self
+        def serialize(result)
+          values_by_file = result[:values_by_file]
+          churn_values = values_by_file.map { |_, values| values[0].to_f }
+          complexity_values = values_by_file.map { |_, values| values[1].to_f }
+
+          mean_churn = churn_values.sum / churn_values.size
+          median_churn = churn_values.sort[churn_values.size / 2]
+          mean_complexity = complexity_values.sum / complexity_values.size
+          median_complexity = complexity_values.sort[complexity_values.size / 2]
+
+          max_churn = churn_values.max
+          min_churn = churn_values.min
+          max_complexity = complexity_values.max
+          min_complexity = complexity_values.min
+
+          epsilon = 0.0001
+          gamma_score = values_by_file.map do |_, values|
+            # unnormalised harmonic mean of churn and complexity,
+            # since the summary needs to be comparable over time
+            churn = values[0].to_f + epsilon
+            complexity = values[1].to_f + epsilon
+
+            (2 * churn * complexity) / (churn + complexity)
+          end
+
+          mean_gamma_score = gamma_score.sum / gamma_score.size
+          median_gamma_score = gamma_score.sort[gamma_score.size / 2]
+
+          end_date = result[:git_period].end_date
+
+          {
+            mean_churn:,
+            median_churn:,
+            max_churn:,
+            min_churn:,
+            mean_complexity:,
+            median_complexity:,
+            max_complexity:,
+            min_complexity:,
+            mean_gamma_score:,
+            median_gamma_score:,
+            end_date:,
+          }
+        end
+      end
+    end
+  end
+end