rspec-watchdog 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: f70de9ae9f5063183759f08a354cd314415f1f361853dd8ce5c2cac863c319b7
+  data.tar.gz: 537a8463433e30ad9fd332d98ed5dfab2e8712ed06dd9ce52fec65f694b20b1e
+SHA512:
+  metadata.gz: 43b4fb491cd5e7a08812275a254ee61bc114f35d4dc6aee91cd76af828288d3a309ec2f789c8ce4e96eec138d7ed290e086916b4c253c2332978217d6e7397e2
+  data.tar.gz: e7c13ed73adb19d2d6f8c6029e2db4fbe02f675eb48667350af302c371ae74940d3201354a32c1937d9b6248f3c0e3548df18437127944065af03b683ddd2d0e

data/README.md

@@ -0,0 +1,140 @@
+# RspecWatchdog
+
+RspecWatchdog is a gem designed to track the performance and reliability of your RSpec tests. It provides useful metrics like execution times, test failures, and flaky tests. With seamless integration into your existing RSpec setup, you can monitor test performance and diagnose flaky tests, improving both test suite efficiency and reliability.
+
+## Motivation
+
+Testing is a crucial part of the development process, but it can be challenging to track and maintain an efficient test suite. RspecWatchdog offers a simple way to monitor test performance, identify slow tests, and spot flaky tests—helping you improve test reliability and speed. By integrating with rspec-rebound, this gem gives you insights into tests that frequently fail, allowing you to address instability in your suite.
+
+## Features
+
+- **Performance tracking**: Measure the execution time of individual tests to identify slow or inefficient tests
+- **Test statistics**: View summary metrics, such as total runs, failures, and average test times
+- **Flaky test detection**: Integrated with rspec-rebound to help you spot flaky tests that fail intermittently
+- **Minimal dependencies**: The gem only requires RSpec, making it easy to integrate into any project that uses RSpec for testing
+- **Optional dashboard integration**: For Rails users, RspecWatchdog can send data to [Watchdog::Dashboard](https://github.com/windmotion-io/watchdog-dashboard) for visualization
+
+## Installation
+
+### Add the Gem to Your Project
+
+Add rspec_watchdog to your Gemfile:
+
+```ruby
+gem 'rspec-watchdog'
+```
+
+Then, run:
+
+```bash
+bundle install
+```
+
+## Configuration
+
+Configure rspec_watchdog in your spec_helper.rb or rails_helper.rb.
+
+### Basic Configuration (RSpec-only Projects)
+
+In your spec/spec_helper.rb, add the following:
+
+```ruby
+require "rspec/watchdog"
+
+Rspec::Watchdog.configure do |config|
+  config.show_logs = true
+
+  # Set these only if using the dashboard integration
+  # config.watchdog_api_url = "http://your-app.com/watchdog/analytics"
+  # config.watchdog_api_token = "your_secret_token"
+end
+
+RSpec.configure do |config|
+  config.add_formatter(:progress)
+  config.add_formatter(SlowSpecFormatter)
+
+  # To enable flaky test detection, add the following:
+  config.flaky_spec_detection = true
+  config.flaky_test_callback = proc do |example|
+    example.metadata[:flaky] = true
+  end
+end
+```
+
+### Configuration Options
+
+#### `show_logs`
+
+When set to `true`, this option enables additional logging for RSpec tests:
+
+- These logs provide insights into test execution, including test runtimes and other relevant debugging information
+- This can be useful for diagnosing slow tests or identifying issues during test runs
+
+#### `watchdog_api_url` (Optional)
+
+This is the endpoint where test execution data will be sent after each RSpec test finishes:
+
+- Only needed if you're using the dashboard integration
+- Can point to your own server or a hosted instance of Watchdog Dashboard
+
+#### `watchdog_api_token` (Optional)
+
+This token is used to validate that the request being sent to the API is legitimate:
+
+- If you're running tests in a CI/CD environment (e.g., GitHub Actions or CircleCI)
+- Should match the token configured in your dashboard instance
+
+## Usage
+
+After installation, RspecWatchdog automatically hooks into your RSpec test suite. You can start tracking your tests immediately without any additional configuration.
+
+### Running Your Tests
+
+Simply run your tests as usual with RSpec:
+
+```bash
+bundle exec rspec
+```
+
+RspecWatchdog will capture metrics about your test runs and display them according to your configuration.
+
+### Output Example
+
+RspecWatchdog will output information about your test runs directly to the console:
+
+```
+SlowTest: "MyClass#method_1" - 2.31 seconds
+SlowTest: "MyClass#method_2" - 1.45 seconds
+...
+```
+
+### Integration with Watchdog Dashboard (Optional)
+
+![Watchdog Dashboard](dashboard_1.png "Watchdog Dashboard")
+
+For a more comprehensive visualization of your test metrics, RspecWatchdog can send data to [Watchdog::Dashboard](https://github.com/windmotion-io/watchdog-dashboard), a separate Rails engine that provides a visual interface.
+
+To use this integration:
+
+1. Set up Watchdog Dashboard in your Rails application (see the [Watchdog Dashboard README](https://github.com/windmotion-io/watchdog-dashboard))
+2. Configure RspecWatchdog with the dashboard URL and API token:
+
+```ruby
+RspecWatchdog.configure do |config|
+  config.show_logs = true
+  config.watchdog_api_url = "http://localhost:3000/watchdog/analytics"
+  config.watchdog_api_token = "your_secret_token" # Must match the dashboard token
+end
+```
+
+## Integration with RSpec-Rebound
+
+RspecWatchdog integrates with rspec-rebound(see the [rspec-rebound README](https://github.com/windmotion-io/rspec-rebound)) to track flaky tests. By enabling both gems in your project, you can easily spot tests that fail inconsistently, making it easier to identify root causes and improve the stability of your test suite.
+
+## Contributing
+
+We welcome contributions to RspecWatchdog! If you have ideas, suggestions, or find a bug, please open an issue or submit a pull request on GitHub.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,4 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+task default: %i[]

data/lib/rspec/watchdog/slow_spec_formatter.rb

@@ -0,0 +1,245 @@
+require 'rspec/core/formatters/base_text_formatter'
+require 'net/http'
+require 'json'
+
+class SlowSpecFormatter
+  RSpec::Core::Formatters.register self, :dump_summary
+
+  def initialize(output)
+    @output = output
+    @show_logs = Rspec::Watchdog.config.show_logs
+    @flaky_spec_detection = RSpec.configuration.flaky_spec_detection
+    @watchdog_api_url = Rspec::Watchdog.config.watchdog_api_url
+    @watchdog_api_token = Rspec::Watchdog.config.watchdog_api_token
+  end
+
+  def dump_summary(summary)
+    return unless @show_logs
+
+    puts "\nAll examples sorted by run time (most durable to fastest):"
+
+    all_examples = summary.examples.map do |example|
+      {
+        description: example.full_description,
+        file_path: example.metadata[:file_path],
+        location: example.metadata[:location],
+        run_time: example.execution_result.run_time,
+        status: example.execution_result.status.to_s,
+        error_message: example.execution_result.exception ? example.execution_result.exception.message : nil,
+        flaky: true
+        # flaky: example.metadata[:flaky].presence || false
+      }
+    end
+
+    sorted_examples = all_examples.sort_by { |ex| -ex[:run_time] }
+
+    sorted_examples.each do |ex|
+      puts "#{ex[:description]} (#{ex[:file_path]}) - #{ex[:run_time]} seconds - #{ex[:location]}"
+    end
+
+    calculate_average_time(summary)
+    fastest_test(sorted_examples)
+    slowest_test(sorted_examples)
+    percentiles(sorted_examples)
+    failed_tests(summary)
+    tests_grouped_by_file(sorted_examples)
+    tests_that_tooked_longer_than(sorted_examples, 2.0)
+    time_distribution_analysis(sorted_examples)
+    test_stability_analysis(summary)
+    execution_time_variance(sorted_examples)
+    temporal_complexity_analysis(sorted_examples)
+    test_dependency_analysis(sorted_examples)
+
+    return unless @watchdog_api_url && @watchdog_api_token
+
+    send_to_api(sorted_examples)
+  end
+
+  private
+
+  def calculate_average_time(summary)
+    average_time = summary.duration / summary.example_count
+    puts "\n🕒 \e[34mAverage time per example:\e[0m #{format('%.4f', average_time)} seconds"
+  end
+
+  def fastest_test(sorted_examples)
+    fastest = sorted_examples.last
+    puts "\n🚀 \e[32mFastest test:\e[0m #{fastest[:description]} (#{fastest[:file_path]}) - #{format('%.4f',
+                                                                                                     fastest[:run_time])} seconds"
+  end
+
+  def slowest_test(sorted_examples)
+    slowest = sorted_examples.first
+    puts "\n🐢 \e[31mSlowest test:\e[0m #{slowest[:description]} (#{slowest[:file_path]}) - #{format('%.4f',
+                                                                                                     slowest[:run_time])} seconds"
+  end
+
+  def percentiles(sorted_examples)
+    percentiles = [0.25, 0.5, 0.75].map do |p|
+      index = (sorted_examples.size * p).round - 1
+      example = sorted_examples[index]
+      {
+        percentile: (p * 100).to_i,
+        description: example[:description],
+        file_path: example[:file_path],
+        run_time: example[:run_time]
+      }
+    end
+    percentiles.each do |p|
+      puts "\n📊 \e[35m#{p[:percentile]}th percentile:\e[0m #{p[:description]} (#{p[:file_path]}) - #{format('%.4f',
+                                                                                                             p[:run_time])} seconds"
+    end
+  end
+
+  def failed_tests(summary)
+    failed = summary.examples.select { |example| example.execution_result.status == :failed }
+    puts "\n❌ \e[31mFailed tests:\e[0m"
+    failed.each do |example|
+      puts "\e[31m#{example.full_description} (#{example.metadata[:file_path]}) - #{example.execution_result.run_time} seconds\e[0m"
+      puts "  \e[33mLocation:\e[0m #{example.metadata[:location]}"
+      puts "  \e[31mFailure message:\e[0m #{example.execution_result.exception.message}"
+    end
+  end
+
+  def tests_grouped_by_file(sorted_examples)
+    grouped_by_file = sorted_examples.group_by { |ex| ex[:file_path] }
+    puts "\n📁 \e[36mTests grouped by file:\e[0m"
+    grouped_by_file.each do |file_path, examples|
+      puts "\n\e[36mFile:\e[0m #{file_path}"
+      examples.each do |ex|
+        puts "  🧪 #{ex[:description]} - #{format('%.4f', ex[:run_time])} seconds"
+        puts "  📍 Location: #{ex[:location]}"
+        puts '  '
+      end
+    end
+  end
+
+  def tests_that_tooked_longer_than(sorted_examples, threshold)
+    long_tests = sorted_examples.select { |ex| ex[:run_time] > threshold }
+    puts "\n⏳ \e[33mTests that took longer than #{threshold} seconds:\e[0m"
+    long_tests.each do |ex|
+      puts "\e[33m#{ex[:description]} (#{ex[:file_path]}) - #{format('%.4f', ex[:run_time])} seconds\e[0m"
+      puts "  📍 Location: #{ex[:location]}"
+    end
+  end
+
+  def time_distribution_analysis(sorted_examples)
+    total_tests = sorted_examples.size
+
+    categories = {
+      '⚡ Ultra Fast (< 0.01s)' => 0,
+      '🚀 Fast (0.01s - 0.1s)' => 0,
+      '🏃 Normal (0.1s - 0.5s)' => 0,
+      '🚶 Slow (0.5s - 1s)' => 0,
+      '🐢 Very Slow (> 1s)' => 0
+    }
+
+    sorted_examples.each do |ex|
+      case ex[:run_time]
+      when 0...0.01
+        categories['⚡ Ultra Fast (< 0.01s)'] += 1
+      when 0.01...0.1
+        categories['🚀 Fast (0.01s - 0.1s)'] += 1
+      when 0.1...0.5
+        categories['🏃 Normal (0.1s - 0.5s)'] += 1
+      when 0.5...1.0
+        categories['🚶 Slow (0.5s - 1s)'] += 1
+      else
+        categories['🐢 Very Slow (> 1s)'] += 1
+      end
+    end
+
+    puts "\n📊 \e[36mTime Distribution Analysis:\e[0m"
+    categories.each do |category, count|
+      percentage = (count.to_f / total_tests * 100).round(2)
+      puts "#{category}: #{count} tests (#{percentage}%)"
+    end
+  end
+
+  def test_stability_analysis(summary)
+    total_tests = summary.example_count
+    passed = summary.examples.select { |e| e.execution_result.status == :passed }.count
+    failed = summary.examples.select { |e| e.execution_result.status == :failed }.count
+    pending = summary.examples.select { |e| e.execution_result.status == :pending }.count
+
+    puts "\n🛡️ \e[34mTest Suite Stability:\e[0m"
+    puts "Total Tests: #{total_tests}"
+    puts "\e[32m✅ Passed: #{passed} (#{(passed.to_f / total_tests * 100).round(2)}%)\e[0m"
+    puts "\e[31m❌ Failed: #{failed} (#{(failed.to_f / total_tests * 100).round(2)}%)\e[0m"
+    puts "\e[33m⏳ Pending: #{pending} (#{(pending.to_f / total_tests * 100).round(2)}%)\e[0m"
+  end
+
+  def execution_time_variance(sorted_examples)
+    run_times = sorted_examples.map { |ex| ex[:run_time] }
+    mean = run_times.sum / run_times.size
+    variance = run_times.map { |time| (time - mean)**2 }.sum / run_times.size
+    std_dev = Math.sqrt(variance)
+
+    puts "\n📈 \e[35mExecution Time Variance:\e[0m"
+    puts "Mean Execution Time: #{format('%.4f', mean)} seconds"
+    puts "Variance: #{format('%.4f', variance)} seconds²"
+    puts "Standard Deviation: #{format('%.4f', std_dev)} seconds"
+  end
+
+  def temporal_complexity_analysis(sorted_examples)
+    sorted_by_complexity = sorted_examples.sort_by { |ex| ex[:run_time] }
+
+    puts "\n🧩 \e[32mTemporal Complexity Analysis:\e[0m"
+    puts 'Top 3 Most Complex Tests:'
+    sorted_by_complexity.first(3).each_with_index do |ex, index|
+      puts "#{index + 1}. #{ex[:description]}"
+      puts "   File: #{ex[:file_path]}"
+      puts "   Execution Time: #{format('%.4f', ex[:run_time])} seconds"
+    end
+  end
+
+  def test_dependency_analysis(sorted_examples)
+    file_dependencies = sorted_examples.group_by { |ex| ex[:file_path] }
+
+    puts "\n🔗 \e[33mTest Dependency Analysis:\e[0m"
+    file_dependencies.each do |file, tests|
+      next if tests.size < 2
+
+      puts "Potential Dependency Group: #{file}"
+      puts "Number of Tests: #{tests.size}"
+      puts "Average Execution Time: #{format('%.4f', tests.map { |t| t[:run_time] }.sum / tests.size)} seconds"
+    end
+  end
+
+  def send_to_api(sorted_examples)
+    uri = URI.parse(@watchdog_api_url)
+    http = Net::HTTP.new(uri.host, uri.port)
+    http.use_ssl = (uri.scheme == 'https')
+
+    batch_size = 30
+    sorted_examples.each_slice(batch_size) do |batch|
+      puts "🌐 Sending batch of #{batch.size} to Analytics API"
+      timestamp = Time.now.to_i
+      batch_payload = batch.map do |example|
+        {
+          description: example[:description],
+          file_path: example[:file_path],
+          location: example[:location],
+          run_time: example[:run_time],
+          status: example[:status],
+          error_message: example[:error_message],
+          timestamp: timestamp,
+          flaky: example[:flaky]
+        }.merge(@flaky_spec_detection ? { flaky: example[:flaky] } : {})
+      end
+
+      request = Net::HTTP::Post.new(uri.path, {
+                                      'Content-Type' => 'application/json',
+                                      'Authorization' => @watchdog_api_token
+                                    })
+      request.body = { metrics: batch_payload }.to_json
+
+      begin
+        response = http.request(request)
+        puts "✅ Batch sent successfully: #{response.code} #{response.message}"
+      rescue StandardError => e
+        puts "❌ Error sending batch: #{e.message}"
+      end
+    end
+  end
+end

data/lib/rspec/watchdog/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Rspec
+  module Watchdog
+    VERSION = "0.1.0"
+  end
+end

data/lib/rspec/watchdog.rb

@@ -0,0 +1,23 @@
+require_relative 'watchdog/version'
+require 'rspec/core' # Ensure RSpec is loaded first
+require_relative 'watchdog/slow_spec_formatter'
+require 'rspec/rebound'
+
+module Rspec
+  module Watchdog
+    class Error < StandardError; end
+
+    class << self
+      attr_accessor :config
+
+      def configure
+        self.config ||= Configuration.new
+        yield(config) if block_given?
+      end
+    end
+
+    class Configuration
+      attr_accessor :show_logs, :watchdog_api_url, :watchdog_api_token
+    end
+  end
+end

data/sig/rspec/watchdog.rbs

@@ -0,0 +1,6 @@
+module Rspec
+  module Watchdog
+    VERSION: String
+    # See the writing guide of rbs: https://github.com/ruby/rbs#guides
+  end
+end

metadata

@@ -0,0 +1,82 @@
+--- !ruby/object:Gem::Specification
+name: rspec-watchdog
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Federico Aldunate
+- Agustin Fornio
+autorequire:
+bindir: exe
+cert_chain: []
+date: 2025-04-08 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3'
+- !ruby/object:Gem::Dependency
+  name: rspec-rebound
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.2.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.2.1
+description: Track RSpec test performance, identify slow tests, and generate metrics
+email:
+- tech@windmotion.io
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- Rakefile
+- dashboard_1.png
+- lib/rspec/watchdog.rb
+- lib/rspec/watchdog/slow_spec_formatter.rb
+- lib/rspec/watchdog/version.rb
+- sig/rspec/watchdog.rbs
+homepage: https://github.com/windmotion-io/rspec-watchdog
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/windmotion-io/rspec-watchdog
+  source_code_uri: https://github.com/windmotion-io/rspec-watchdog
+  changelog_uri: https://github.com/windmotion-io/rspec-watchdog
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.0.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.3.7
+signing_key:
+specification_version: 4
+summary: RSpec performance tracking and metrics
+test_files: []