gitlab_quality-test_tooling 3.9.0 → 3.10.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: acd13e22b4fd23cb52365b1137833a0348af808f009a644e36a667c8f8fc0cb9
-  data.tar.gz: eb62c1b5bcd4e763ed35d7425e04b7e19743f40fcaa1e3493b186806443ce517
+  metadata.gz: 6c49f987999c6124651b5f1f21de7d588d1940749b91f8527e21b9fcbbb1921b
+  data.tar.gz: 384d0765fb0d891e373777004ad20a5cecbc2df9b07cc74ca1b70ae30a080641
 SHA512:
-  metadata.gz: 11c99a5294a20167082412d9aa89804b417c982cdf470a58bf9749cac9d1edc264bdfb6d4a580a3b0589a2ec7f97d38959c072504473bc495ef6418e9e1e0185
-  data.tar.gz: f5c10bf74809af017c60b0533ab6f42afafa7c06f328b34856796faa02b65974865de40fb1597edb27af64a6eabd4846c34147d70061b42a13f3232a8a72c295
+  metadata.gz: 4ff2cc394058616512c5c00779772d244d08986e12762a49c17ca9e31065a4af6b79beaeb8ceb7ac35c290efb10410c5888aff363b183d4190bb9b863278a418
+  data.tar.gz: 8cc8717e64a9c260ca10474897aae319bae9565a1257815e3a83e931049d8d0c61c32fb38d9b66f13976f0f0e1ec4e7d3666f5c3311c5e64cae0c25940ff5da6

data/.rubocop_todo.yml

@@ -59,9 +59,3 @@ Style/RedundantReturn:
   Exclude:
     - 'lib/gitlab_quality/test_tooling/runtime/env.rb'
 
-# Offense count: 1
-# This cop supports unsafe autocorrection (--autocorrect-all).
-# Configuration parameters: AllowedMethods, AllowedPatterns.
-Style/ReturnNilInPredicateMethodDefinition:
-  Exclude:
-    - 'lib/gitlab_quality/test_tooling/knapsack_reports/spec_run_time.rb'

data/AGENTS.md

@@ -0,0 +1,95 @@
+# gitlab_quality-test_tooling
+
+Ruby gem providing test tooling for GitLab CI — metrics export, test quarantine, failure reporting, flaky test tracking, and code coverage analysis.
+
+## Tech Stack
+
+- Ruby (see `.ruby-version` and gemspec for version), gem published to RubyGems
+- RSpec for testing, RuboCop (`gitlab-styles` base) for linting
+- Lefthook for pre-push hooks (rspec, rubocop, changelog trailer check)
+
+## Testing
+
+```bash
+bundle exec rspec
+bundle exec rubocop
+```
+
+## Commit Conventions
+
+Every branch must have a `Changelog:` git trailer on its **first commit** (relative to `origin/main`).
+
+Valid values: `added`, `fixed`, `changed`, `deprecated`, `removed`, `security`, `performance`, `other`
+
+Example:
+
+```
+Fix failure_exception for MultipleExceptionError
+
+Description of the change.
+
+Changelog: fixed
+```
+
+### Enforcement
+
+- **lefthook pre-push hook** checks the first commit on the branch
+- **gitlab-dangerfiles** checks in CI (Danger bot comments on MRs)
+- `--no-verify` bypasses lefthook locally, but CI will still fail without the trailer
+
+### Choosing the right value
+
+| Value | When to use |
+|-------|-------------|
+| `added` | New feature or capability |
+| `fixed` | Bug fix |
+| `changed` | Non-breaking change to existing behavior |
+| `deprecated` | Feature marked for removal |
+| `removed` | Feature removed |
+| `security` | Security fix |
+| `performance` | Performance improvement |
+| `other` | Refactoring, docs, CI changes |
+
+Reference: https://docs.gitlab.com/ee/development/changelog.html
+
+### Commit Message Format
+
+- Present tense ("Fix bug" not "Fixed bug")
+- First line under 72 characters
+- Blank line between subject and body
+- `Changelog:` trailer at the end of the first commit's message, after a blank line
+
+## Project Structure
+
+| Directory | Purpose |
+|-----------|---------|
+| `lib/gitlab_quality/test_tooling/` | Main gem code |
+| `exe/` | CLI executables |
+| `spec/` | RSpec tests |
+| `.gitlab/changelog_config.yml` | Changelog category mapping |
+
+## Key Modules
+
+- **TestMetricsExporter** — Serializes RSpec results to ClickHouse (exception classes, failure messages, CI metadata)
+- **Report** — Creates/updates GitLab issues for test failures, flaky tests, slow tests
+- **TestQuarantine** — RSpec formatter that skips quarantined tests
+- **CodeCoverage** — Coverage analysis and responsibility patterns
+- **ClickHouse::Client** — Batch HTTP client for pushing metrics
+
+## Release Process
+
+The [automated gem release process](https://gitlab.com/gitlab-org/quality/pipeline-common#release-process) publishes new versions via CI. It will:
+
+- Publish the gem to https://rubygems.org/gems/gitlab_quality-test_tooling
+- Create a release at https://gitlab.com/gitlab-org/ruby/gems/gitlab_quality-test_tooling/-/releases
+- Populate release notes from changelog entries
+
+### Steps to release
+
+1. Create a branch (e.g., `bump-version-X-Y-Z`)
+2. Update `lib/gitlab_quality/test_tooling/version.rb` with the new version
+3. Run `bundle install` to update `Gemfile.lock`
+4. Commit with message like `Update gem version to X.Y.Z` and trailer `Changelog: changed`
+5. Create an MR using the **Release** MR template (`.gitlab/merge_request_templates/Release.md`)
+6. Fill in the template: update the diff link, check release notes, verify SemVer
+7. Get the MR merged by a maintainer — CI will publish the gem automatically

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    gitlab_quality-test_tooling (3.9.0)
+    gitlab_quality-test_tooling (3.10.1)
       activesupport (>= 7.0)
       amatch (~> 0.4.1)
       fog-google (~> 1.24, >= 1.24.1)

data/README.md

@@ -141,21 +141,6 @@ Usage: exe/slow-test-issues [options]
     -h, --help                       Show the usage
 ```
 
-### `exe/knapsack-report-issues`
-
-```shell
-Purpose: Create spec run time issue when a spec file almost caused job timeout because it ran significantly longer than what Knapsack expected.
-Usage: exe/knapsack-report-issues [options]
-    -i, --input-file INPUT_FILE      Knapsack actual run time report file path glob
-    -e EXPECTED_REPORT,              Knapsack expected report file path
-        --expected-report
-    -p, --project PROJECT            Can be an integer or a group/project string
-    -t, --token TOKEN                A valid access token with `api` scope and Maintainer permission in PROJECT
-        --dry-run                    Perform a dry-run (don't create issues)
-    -v, --version                    Show the version
-    -h, --help                       Show the usage
-```
-
 ### `exe/failed-test-issues`
 
 ```shell

data/lib/gitlab_quality/test_tooling/test_metrics_exporter/test_metrics.rb

@@ -161,14 +161,23 @@ module GitlabQuality
           exception.all_exceptions.flatten
         end
 
-        # Truncated exception stacktrace
+        # Truncated exception message
+        #
+        # For MultipleExceptionError, returns the first wrapped exception's message
+        # instead of the unhelpful wrapper class name.
         #
         # @return [String]
         def failure_exception
           exception = example.execution_result.exception
           return unless exception
 
-          exception.to_s.tr("\n", " ").slice(0, 1000)
+          source = if exception.respond_to?(:all_exceptions)
+                     exception.all_exceptions.flatten.first || exception
+                   else
+                     exception
+                   end
+
+          source.to_s.tr("\n", " ").slice(0, 1000)
         end
 
         # Test run type | suite name

data/lib/gitlab_quality/test_tooling/version.rb

@@ -2,6 +2,6 @@
 
 module GitlabQuality
   module TestTooling
-    VERSION = "3.9.0"
+    VERSION = "3.10.1"
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: gitlab_quality-test_tooling
 version: !ruby/object:Gem::Version
-  version: 3.9.0
+  version: 3.10.1
 platform: ruby
 authors:
 - GitLab Quality
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2026-03-04 00:00:00.000000000 Z
+date: 2026-03-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: climate_control
@@ -420,7 +420,6 @@ executables:
 - feature-readiness-checklist
 - feature-readiness-evaluation
 - flaky-test-issues
-- knapsack-report-issues
 - post-to-slack
 - prepare-stage-reports
 - relate-failure-issue
@@ -439,6 +438,7 @@ files:
 - ".rubocop_todo.yml"
 - ".ruby-version"
 - ".tool-versions"
+- AGENTS.md
 - CODE_OF_CONDUCT.md
 - CONTRIBUTING.md
 - Dangerfile
@@ -454,7 +454,6 @@ files:
 - exe/feature-readiness-checklist
 - exe/feature-readiness-evaluation
 - exe/flaky-test-issues
-- exe/knapsack-report-issues
 - exe/post-to-slack
 - exe/prepare-stage-reports
 - exe/relate-failure-issue
@@ -514,8 +513,6 @@ files:
 - lib/gitlab_quality/test_tooling/gitlab_client/work_items_client.rb
 - lib/gitlab_quality/test_tooling/gitlab_client/work_items_dry_client.rb
 - lib/gitlab_quality/test_tooling/job_trace_analyzer.rb
-- lib/gitlab_quality/test_tooling/knapsack_reports/spec_run_time.rb
-- lib/gitlab_quality/test_tooling/knapsack_reports/spec_run_time_report.rb
 - lib/gitlab_quality/test_tooling/labels_inference.rb
 - lib/gitlab_quality/test_tooling/report/concerns/group_and_category_labels.rb
 - lib/gitlab_quality/test_tooling/report/concerns/issue_reports.rb
@@ -537,7 +534,6 @@ files:
 - lib/gitlab_quality/test_tooling/report/group_issues/issue_updater.rb
 - lib/gitlab_quality/test_tooling/report/health_problem_reporter.rb
 - lib/gitlab_quality/test_tooling/report/issue_logger.rb
-- lib/gitlab_quality/test_tooling/report/knapsack_report_issue.rb
 - lib/gitlab_quality/test_tooling/report/merge_request_slow_tests_report.rb
 - lib/gitlab_quality/test_tooling/report/prepare_stage_reports.rb
 - lib/gitlab_quality/test_tooling/report/relate_failure_issue.rb

data/exe/knapsack-report-issues

@@ -1,54 +0,0 @@
-#!/usr/bin/env ruby
-# frozen_string_literal: true
-
-require "bundler/setup"
-require "optparse"
-
-require_relative "../lib/gitlab_quality/test_tooling"
-
-params = {}
-
-options = OptionParser.new do |opts|
-  opts.banner = "Usage: #{$PROGRAM_NAME} [options]"
-
-  opts.on("-i", "--input-file INPUT_FILE", String, "Knapsack actual run time report file path glob") do |input_file|
-    params[:input_files] = input_file
-  end
-
-  opts.on("-e", "--expected-report EXPECTED_REPORT", String, "Knapsack expected report file path") do |report_path|
-    params[:expected_report] = report_path.strip
-  end
-
-  opts.on('-p', '--project PROJECT', String, 'Can be an integer or a group/project string') do |project|
-    params[:project] = project
-  end
-
-  opts.on('-t', '--token TOKEN', String, 'A valid access token with `api` scope and Maintainer permission in PROJECT') do |token|
-    params[:token] = token
-  end
-
-  opts.on('--dry-run', "Perform a dry-run (don't create issues)") do
-    params[:dry_run] = true
-  end
-
-  opts.on_tail('-v', '--version', 'Show the version') do
-    require_relative "../lib/gitlab_quality/test_tooling/version"
-    puts "#{$PROGRAM_NAME} : #{GitlabQuality::TestTooling::VERSION}"
-    exit
-  end
-
-  opts.on_tail('-h', '--help', 'Show the usage') do
-    puts "Purpose: Create spec run time issue when a spec file almost caused job timeout because it ran significantly longer than what Knapsack expected."
-    puts opts
-    exit
-  end
-
-  opts.parse(ARGV)
-end
-
-if params.any?
-  GitlabQuality::TestTooling::Report::KnapsackReportIssue.new(**params).invoke!
-else
-  puts options
-  exit 1
-end

data/lib/gitlab_quality/test_tooling/knapsack_reports/spec_run_time.rb

@@ -1,120 +0,0 @@
-# frozen_string_literal: true
-
-module GitlabQuality
-  module TestTooling
-    module KnapsackReports
-      class SpecRunTime
-        attr_reader :file, :expected, :actual, :expected_suite_duration, :actual_suite_duration, :project, :ref
-
-        ACTUAL_TO_EXPECTED_SPEC_RUN_TIME_RATIO_THRESHOLD = 1.5 # actual run time is longer than expected by 50% +
-        SPEC_WEIGHT_PERCENTAGE_TRESHOLD = 15 # a spec file takes 15%+ of the total test suite run time
-        SUITE_DURATION_THRESHOLD = 70 * 60 # if test suite takes more than 70 minutes, job risks timing out
-        FEATURE_CATEGORY_METADATA_REGEX = /(?<=feature_category: :)(?<feature_category>\w+)/
-
-        def initialize(
-          file:,
-          expected:,
-          actual:,
-          expected_suite_duration:,
-          actual_suite_duration:,
-          token: '',
-          project: Runtime::Env.ci_project_path,
-          ref: Runtime::Env.ci_commit_ref_name)
-          @file = file
-          @expected = expected.to_f
-          @actual = actual.to_f
-          @expected_suite_duration = expected_suite_duration.to_f
-          @actual_suite_duration = actual_suite_duration.to_f
-          @token = token
-          @project = project
-          @ref = ref
-        end
-
-        def feature_category
-          file_lines.each do |line|
-            match = FEATURE_CATEGORY_METADATA_REGEX.match(line)
-            next unless match
-
-            break match[:feature_category]
-          end
-        end
-
-        def should_report?
-          # guideline proposed in https://gitlab.com/gitlab-org/quality/engineering-productivity/team/-/issues/354
-          exceed_actual_to_expected_ratio_threshold? && test_suite_bottleneck?
-        end
-
-        def ci_pipeline_url_markdown
-          "[#{ci_pipeline_id}](#{ci_pipeline_url})"
-        end
-
-        def ci_pipeline_created_at
-          ENV.fetch('CI_PIPELINE_CREATED_AT', nil)
-        end
-
-        def ci_job_link_markdown
-          "[#{ci_job_name}](#{ci_job_url})"
-        end
-
-        def file_link_markdown
-          "[#{file}](#{file_link})"
-        end
-
-        def actual_percentage
-          (actual / actual_suite_duration * 100).round(2)
-        end
-
-        def name
-          nil
-        end
-
-        private
-
-        attr_reader :token
-
-        def exceed_actual_to_expected_ratio_threshold?
-          actual / expected >= ACTUAL_TO_EXPECTED_SPEC_RUN_TIME_RATIO_THRESHOLD
-        end
-
-        def test_suite_bottleneck?
-          # now we only report bottlenecks when they risk causing job timeouts
-          return unless actual_suite_duration > SUITE_DURATION_THRESHOLD
-
-          actual_percentage > SPEC_WEIGHT_PERCENTAGE_TRESHOLD
-        end
-
-        def ci_job_url
-          ENV.fetch('CI_JOB_URL', nil)
-        end
-
-        def ci_job_name
-          ENV.fetch('CI_JOB_NAME_SLUG', nil)
-        end
-
-        def ci_pipeline_id
-          ENV.fetch('CI_PIPELINE_IID', nil)
-        end
-
-        def ci_pipeline_url
-          ENV.fetch('CI_PIPELINE_URL', nil)
-        end
-
-        def file_link
-          "https://gitlab.com/#{project}/-/blob/#{Runtime::Env.ci_commit_ref_name}/#{file}"
-        end
-
-        def file_lines
-          files_client.file_contents.lines(chomp: true)
-        end
-
-        def files_client
-          @files_client ||= GitlabClient::RepositoryFilesClient.new(
-            token: token,
-            project: project,
-            file_path: file,
-            ref: ref)
-        end
-      end
-    end
-  end
-end

data/lib/gitlab_quality/test_tooling/knapsack_reports/spec_run_time_report.rb

@@ -1,66 +0,0 @@
-# frozen_string_literal: true
-
-require 'json'
-
-module GitlabQuality
-  module TestTooling
-    module KnapsackReports
-      class SpecRunTimeReport
-        attr_reader :project, :expected_report, :actual_report
-
-        def initialize(project:, expected_report_path:, actual_report_path:, token: '')
-          @project = project
-          @expected_report = parse(expected_report_path)
-          @actual_report = parse(actual_report_path)
-          @token = token
-        end
-
-        def filtered_report
-          @filtered_report = actual_report.keys.filter_map do |spec_file|
-            expected_run_time = expected_report[spec_file]
-            actual_run_time = actual_report[spec_file]
-
-            if expected_run_time.nil?
-              puts "#{spec_file} missing from the expected Knapsack report, skipping."
-              next
-            end
-
-            spec_run_time = SpecRunTime.new(
-              token: token,
-              project: project,
-              file: spec_file,
-              expected: expected_run_time,
-              actual: actual_run_time,
-              expected_suite_duration: expected_test_suite_run_time_total,
-              actual_suite_duration: actual_test_suite_run_time_total
-            )
-
-            spec_run_time if spec_run_time.should_report?
-          end
-        end
-
-        private
-
-        attr_reader :token
-
-        def parse(report_path)
-          JSON.parse(File.read(report_path))
-        end
-
-        def expected_test_suite_run_time_total
-          @expected_test_suite_run_time_total ||=
-            expected_report.reduce(0) do |total_run_time, (_spec_file, run_time)|
-              total_run_time + run_time
-            end
-        end
-
-        def actual_test_suite_run_time_total
-          @actual_test_suite_run_time_total ||=
-            actual_report.reduce(0) do |total_run_time, (_spec_file, run_time)|
-              total_run_time + run_time
-            end
-        end
-      end
-    end
-  end
-end

data/lib/gitlab_quality/test_tooling/report/knapsack_report_issue.rb

@@ -1,150 +0,0 @@
-# frozen_string_literal: true
-
-module GitlabQuality
-  module TestTooling
-    module Report
-      # Uses the API to create GitLab issues for spec run time exceeding Knapsack expectation
-      #
-      # - Takes the expected and actual Knapsack JSON reports from the knapsack output
-      # - Takes a project where issues should be created
-      # - For every test file reported with unexpectedly long run time:
-      #   - Find issue by test file name, and if found:
-      #     - Reopen issue if it already exists, but is closed
-      #     - Update the issue with the new run time data
-      #   - If not found:
-      #     - Create a new issue with the run time data
-      class KnapsackReportIssue < ReportAsIssue
-        include Concerns::GroupAndCategoryLabels
-
-        NEW_ISSUE_LABELS = Set.new([
-          'test', 'automation:bot-authored', 'type::maintenance', 'maintenance::performance',
-          'priority::3', 'severity::3', 'knapsack_report', 'suppress-contributor-links'
-        ]).freeze
-        SEARCH_LABELS = %w[test maintenance::performance knapsack_report].freeze
-        JOB_TIMEOUT_EPIC_URL = 'https://gitlab.com/groups/gitlab-org/quality/engineering-productivity/-/epics/19'
-
-        def initialize(token:, input_files:, expected_report:, project: nil, dry_run: false)
-          super
-
-          @expected_report = expected_report
-        end
-
-        private
-
-        attr_reader :expected_report
-
-        def run!
-          puts "Reporting spec file exceeding Knapsack expectaton issues in project `#{project}` via the API at `#{Runtime::Env.gitlab_api_base}`."
-
-          search_and_create_issue
-        end
-
-        def new_issue_labels(test)
-          up_to_date_labels(test: test, new_labels: NEW_ISSUE_LABELS + group_and_category_labels_for_test(test))
-        end
-
-        def search_and_create_issue
-          filtered_report = KnapsackReports::SpecRunTimeReport.new(
-            token: token,
-            project: project,
-            expected_report_path: expected_report_path,
-            actual_report_path: actual_report_path
-          ).filtered_report
-
-          puts "=> Reporting #{filtered_report.count} spec files exceeding Knapsack expectation."
-
-          filtered_report.each do |spec_with_run_time|
-            existing_issues = find_issues_for_test(spec_with_run_time, labels: SEARCH_LABELS)
-
-            if existing_issues.empty?
-              puts "Creating issue for #{spec_with_run_time.file}"
-              create_issue(spec_with_run_time)
-            else
-              update_issue(issue: existing_issues.last, spec_run_time: spec_with_run_time)
-            end
-          end
-        end
-
-        def expected_report_path
-          return if expected_report.nil? || !File.exist?(expected_report)
-
-          expected_report
-        end
-
-        def actual_report_path
-          return if files.nil? || !File.exist?(files.first)
-
-          files.first
-        end
-
-        def new_issue_title(spec_run_time)
-          "Job timeout risk: #{spec_run_time.file} ran much longer than expected"
-        end
-
-        def new_issue_description(spec_run_time)
-          <<~MARKDOWN.chomp
-          /epic #{JOB_TIMEOUT_EPIC_URL}
-
-          ### Why was this issue created?
-
-          #{spec_run_time.file_link_markdown} was reported to have:
-
-          1. exceeded Knapsack's expected runtime by at least 50%, and
-          2. been identified as a notable pipeline bottleneck and a job timeout risk
-
-          ### Suggested steps for investigation
-
-          1. To reproduce in CI by running test files in the same order, you can follow the steps listed [here](https://docs.gitlab.com/ee/development/testing_guide/flaky_tests.html#recreate-job-failure-in-ci-by-forcing-the-job-to-run-the-same-set-of-test-files).
-          1. Identify if a specific test case is stalling the run time. Hint: You can search the job's log for `Starting example group #{spec_run_time.file}` and view the elapsed time after each test case in the proceeding lines starting with `[RSpecRunTime]`.
-          1. If the test file is large, consider refactoring it into multiple files to allow better test parallelization across runners.
-          1. If the run time cannot be fixed in time, consider quarantine the spec(s) to restore performance.
-
-          ### Run time details
-
-          #{run_time_detail(spec_run_time)}
-          MARKDOWN
-        end
-
-        def update_issue(issue:, spec_run_time:)
-          updated_description = <<~MARKDOWN.chomp
-          #{issue.description}
-
-          #{run_time_detail(spec_run_time)}
-          MARKDOWN
-
-          issue_attrs = {
-            description: updated_description
-          }
-
-          # We reopen closed issues to not lose any history
-          state_event = issue.state == 'closed' ? 'reopen' : nil
-          issue_attrs[:state_event] = state_event if state_event
-
-          gitlab.edit_issue(iid: issue.iid, options: issue_attrs)
-          puts "  => Added a report in #{issue.web_url}!"
-        end
-
-        def run_time_detail(spec_run_time)
-          <<~MARKDOWN.chomp
-            - Reported from pipeline #{spec_run_time.ci_pipeline_url_markdown} created at `#{spec_run_time.ci_pipeline_created_at}`
-
-            | Field | Value |
-            | ------ | ------ |
-            | Job URL| #{spec_run_time.ci_job_link_markdown} |
-            | Job total RSpec suite run time | expected: `#{readable_duration(spec_run_time.expected_suite_duration)}`, actual: `#{readable_duration(spec_run_time.actual_suite_duration)}` |
-            | Spec file run time | expected: `#{readable_duration(spec_run_time.expected)}`, actual: `#{readable_duration(spec_run_time.actual)}` |
-            | Spec file weight | `#{spec_run_time.actual_percentage}%` of total suite run time |
-          MARKDOWN
-        end
-
-        def assert_input_files!(_files)
-          missing_expected_report_msg = "Missing a valid expected Knapsack report."
-          missing_actual_report_msg = "Missing a valid actual Knapsack report."
-
-          abort missing_expected_report_msg if expected_report_path.nil?
-          abort missing_actual_report_msg if actual_report_path.nil?
-        end
-      end
-    end
-  end
-end