datadog-ci 1.8.0 → 1.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cd2dfb59d1dd3d17b2e0ccea1f03308fe38b65ce6daeb3f903d158fc0b4ed47b
-  data.tar.gz: cc1a87dcd6fdab402556df3dd0f8a042a4dc96438c4aa7796091ff71a49435b0
+  metadata.gz: 7641cbd9b01a787badb9bab7d3aa74933c99b750b46609c9a625ed3f7daaa11c
+  data.tar.gz: 5489c728c4b6c91ff68b515796533801b02dfb623f506da23849e60e343063a3
 SHA512:
-  metadata.gz: d7eacb805c74e1c627c3f0494fab65c45b5bf940cbd124b9c027edd0eb4c21fcaaadd052aa5e7a1e60f2bc9d38b6e6787676a9f64e2fe93e234af4e8d3370117
-  data.tar.gz: 4d10fc9811a302f609fa36be74b0cbd81d4ae690cdeeb1f333025b7ffbdd398ef92bc6eb8ee455f2d34a9bc83ff76b06cbe091c3d4fe697ded4ce8f9ebf5f75d
+  metadata.gz: 3d2f41e5eb8f58c3eecb72200658add1799109c7778fad032c6d58ec2fd462a37df7e1a47edfb6cf7390e037c4e30ada85dc95c71983c30629de38b2bf62fd35
+  data.tar.gz: 408151cc04150bcc8ec2a0933c0f6d38cd1362b5842e8b145960ba276431681893988cfd0c49ce47f519ae9e7451549b45503a6ea79f123cfd4cc3750cb2a8ea

data/CHANGELOG.md

@@ -1,5 +1,17 @@
 ## [Unreleased]
 
+## [1.9.0] - 2024-11-26
+
+### Added
+
+* Auto instrumentation ([#259][])
+
+## [1.8.1] - 2024-10-18
+
+
+### Fixed
+* Make --spec-path option available to skipped-tests-estimate cli command ([#250][])
+
 ## [1.8.0] - 2024-10-17
 
 ### Added
@@ -350,7 +362,9 @@ Currently test suite level visibility is not used by our instrumentation: it wil
 
 - Ruby versions < 2.7 no longer supported ([#8][])
 
-[Unreleased]: https://github.com/DataDog/datadog-ci-rb/compare/v1.8.0...main
+[Unreleased]: https://github.com/DataDog/datadog-ci-rb/compare/v1.9.0...main
+[1.9.0]: https://github.com/DataDog/datadog-ci-rb/compare/v1.8.1...v1.9.0
+[1.8.1]: https://github.com/DataDog/datadog-ci-rb/compare/v1.8.0...v1.8.1
 [1.8.0]: https://github.com/DataDog/datadog-ci-rb/compare/v1.7.0...v1.8.0
 [1.7.0]: https://github.com/DataDog/datadog-ci-rb/compare/v1.6.0...v1.7.0
 [1.6.0]: https://github.com/DataDog/datadog-ci-rb/compare/v1.5.0...v1.6.0
@@ -505,4 +519,6 @@ Currently test suite level visibility is not used by our instrumentation: it wil
 [#242]: https://github.com/DataDog/datadog-ci-rb/issues/242
 [#243]: https://github.com/DataDog/datadog-ci-rb/issues/243
 [#244]: https://github.com/DataDog/datadog-ci-rb/issues/244
-[#248]: https://github.com/DataDog/datadog-ci-rb/issues/248
+[#248]: https://github.com/DataDog/datadog-ci-rb/issues/248
+[#250]: https://github.com/DataDog/datadog-ci-rb/issues/250
+[#259]: https://github.com/DataDog/datadog-ci-rb/issues/259

data/README.md

@@ -1,9 +1,7 @@
-# Datadog Test Visibility for Ruby
+# Datadog Test Optimization for Ruby
 
 [![Gem Version](https://badge.fury.io/rb/datadog-ci.svg)](https://badge.fury.io/rb/datadog-ci)
 [![YARD documentation](https://img.shields.io/badge/YARD-documentation-blue)](https://datadoghq.dev/datadog-ci-rb/)
-[![codecov](https://codecov.io/gh/DataDog/datadog-ci-rb/branch/main/graph/badge.svg)](https://app.codecov.io/gh/DataDog/datadog-ci-rb/branch/main)
-[![CircleCI](https://dl.circleci.com/status-badge/img/gh/DataDog/datadog-ci-rb/tree/main.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/DataDog/datadog-ci-rb/tree/main)
 
 Datadog's Ruby Library for instrumenting your tests.
 Learn more on our [official website](https://docs.datadoghq.com/tests/) and check out our [documentation for this library](https://docs.datadoghq.com/tests/setup/ruby/?tab=cloudciprovideragentless).
@@ -11,12 +9,12 @@ Learn more on our [official website](https://docs.datadoghq.com/tests/) and chec
 ## Features
 
 - [Test Visibility](https://docs.datadoghq.com/tests/) - collect metrics and results for your tests
-- [Intelligent test runner](https://docs.datadoghq.com/intelligent_test_runner/) - save time by selectively running only tests affected by code changes
-- [Auto test retries](https://docs.datadoghq.com/tests/auto_test_retries/?tab=ruby) - retrying failing tests up to N times to avoid failing your build due to flaky tests
-- [Early flake detection](https://docs.datadoghq.com/tests/early_flake_detection?tab=ruby) - Datadog’s test flakiness solution that identifies flakes early by running newly added tests multiple times
+- [Test impact analysis](https://docs.datadoghq.com/tests/test_impact_analysis/) - save time by selectively running only tests affected by code changes
+- [Flaky test management](https://docs.datadoghq.com/tests/flaky_test_management/) - track, alert, search your flaky tests in Datadog UI
+- [Auto test retries](https://docs.datadoghq.com/tests/flaky_test_management/auto_test_retries/?tab=ruby) - retrying failing tests up to N times to avoid failing your build due to flaky tests
+- [Early flake detection](https://docs.datadoghq.com/tests/flaky_test_management/early_flake_detection/?tab=ruby) - Datadog’s test flakiness solution that identifies flakes early by running newly added tests multiple times
 - [Search and manage CI tests](https://docs.datadoghq.com/tests/search/)
 - [Enhance developer workflows](https://docs.datadoghq.com/tests/developer_workflows)
-- [Flaky test management](https://docs.datadoghq.com/tests/guides/flaky_test_management/)
 - [Add custom measures to your tests](https://docs.datadoghq.com/tests/guides/add_custom_measures/?tab=ruby)
 - [Browser tests integration with Datadog RUM](https://docs.datadoghq.com/tests/browser_tests)
 
@@ -37,7 +35,7 @@ If you used [test visibility for Ruby](https://docs.datadoghq.com/tests/setup/ru
 ## Setup
 
 - [Test visibility setup](https://docs.datadoghq.com/tests/setup/ruby/?tab=cloudciprovideragentless)
-- [Intelligent test runner setup](https://docs.datadoghq.com/intelligent_test_runner/setup/ruby) (test visibility setup is required before setting up intelligent test runner)
+- [Test impact analysis setup](https://docs.datadoghq.com/tests/test_impact_analysis/setup/ruby/?tab=cloudciprovideragentless) (test visibility setup is required before setting up test impact analysis)
 
 ## Contributing
 

data/exe/ddcirb

@@ -2,4 +2,6 @@
 
 require "datadog/ci/cli/cli"
 
-Datadog::CI::CLI.exec(ARGV.first)
+command = ARGV.shift
+
+Datadog::CI::CLI.exec(command, ARGV)

data/lib/datadog/ci/auto_instrument.rb

@@ -0,0 +1,8 @@
+require "datadog/ci"
+
+if RUBY_ENGINE == "jruby"
+  Datadog.logger.error("Auto instrumentation is not supported on JRuby. Please use manual instrumentation instead.")
+  return
+end
+
+Datadog::CI::Contrib::Instrumentation.auto_instrument

data/lib/datadog/ci/cli/cli.rb

@@ -1,14 +1,17 @@
 require "datadog"
 require "datadog/ci"
 
+require_relative "command/exec"
 require_relative "command/skippable_tests_percentage"
 require_relative "command/skippable_tests_percentage_estimate"
 
 module Datadog
   module CI
     module CLI
-      def self.exec(action)
+      def self.exec(action, args = [])
         case action
+        when "exec"
+          Command::Exec.new(args).exec
         when "skipped-tests", "skippable-tests"
           Command::SkippableTestsPercentage.new.exec
         when "skipped-tests-estimate", "skippable-tests-estimate"
@@ -17,6 +20,7 @@ module Datadog
           puts("Usage: bundle exec ddcirb [command] [options]. Available commands:")
           puts("  skippable-tests - calculates the exact percentage of skipped tests and prints it to stdout or file")
           puts("  skippable-tests-estimate - estimates the percentage of skipped tests and prints it to stdout or file")
+          puts("  exec YOUR_TEST_COMMAND - automatically instruments your test command with Datadog and executes it")
         end
       end
     end

data/lib/datadog/ci/cli/command/base.rb

@@ -27,6 +27,7 @@ module Datadog
 
               opts.on("-f", "--file FILENAME", "Output result to file FILENAME")
               opts.on("--verbose", "Verbose output to stdout")
+              opts.on("--spec-path=[SPEC_PATH]", "Relative path to the spec directory, example: spec")
 
               command_options(opts)
             end.parse!(into: ddcirb_options)

data/lib/datadog/ci/cli/command/exec.rb

@@ -0,0 +1,29 @@
+require_relative "base"
+require_relative "../../test_optimisation/skippable_percentage/estimator"
+
+module Datadog
+  module CI
+    module CLI
+      module Command
+        class Exec < Base
+          def initialize(args)
+            super()
+
+            @args = args
+          end
+
+          def exec
+            rubyopts = [
+              "-rdatadog/ci/auto_instrument"
+            ]
+
+            existing_rubyopt = ENV["RUBYOPT"]
+            ENV["RUBYOPT"] = existing_rubyopt ? "#{existing_rubyopt} #{rubyopts.join(" ")}" : rubyopts.join(" ")
+
+            Kernel.exec(*@args)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/datadog/ci/cli/command/skippable_tests_percentage.rb

@@ -18,7 +18,6 @@ module Datadog
 
           def command_options(opts)
             opts.on("--rspec-opts=[OPTIONS]", "Command line options to pass to RSpec")
-            opts.on("--spec-path=[SPEC_PATH]", "Relative path to the spec directory, example: spec")
           end
         end
       end

data/lib/datadog/ci/configuration/settings.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require_relative "../contrib/instrumentation"
 require_relative "../ext/settings"
 require_relative "../utils/bundle"
 
@@ -8,8 +9,6 @@ module Datadog
     module Configuration
       # Adds CI behavior to ddtrace settings
       module Settings
-        InvalidIntegrationError = Class.new(StandardError)
-
         def self.extended(base)
           base = base.singleton_class unless base.is_a?(Class)
           add_settings!(base)
@@ -126,23 +125,11 @@ module Datadog
               define_method(:instrument) do |integration_name, options = {}, &block|
                 return unless enabled
 
-                integration = fetch_integration(integration_name)
-                integration.configure(options, &block)
-
-                return unless integration.enabled
-
-                patch_results = integration.patch
-                next if patch_results == true
-
-                error_message = <<-ERROR
-                  Available?: #{patch_results[:available]}, Loaded?: #{patch_results[:loaded]},
-                  Compatible?: #{patch_results[:compatible]}, Patchable?: #{patch_results[:patchable]}"
-                ERROR
-                Datadog.logger.warn("Unable to patch #{integration_name} (#{error_message})")
+                Contrib::Instrumentation.instrument(integration_name, options, &block)
               end
 
               define_method(:[]) do |integration_name|
-                fetch_integration(integration_name).configuration
+                Contrib::Instrumentation.fetch_integration(integration_name).configuration
               end
 
               option :trace_flush
@@ -151,11 +138,6 @@ module Datadog
                 o.type :hash
                 o.default({})
               end
-
-              define_method(:fetch_integration) do |name|
-                Datadog::CI::Contrib::Integration.registry[name] ||
-                  raise(InvalidIntegrationError, "'#{name}' is not a valid integration.")
-              end
             end
           end
         end

data/lib/datadog/ci/contrib/ciqueue/integration.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+require_relative "../integration"
+require_relative "patcher"
+
+module Datadog
+  module CI
+    module Contrib
+      module Ciqueue
+        # ci-queue test runner instrumentation
+        # https://github.com/Shopify/ci-queue
+        class Integration < Contrib::Integration
+          MINIMUM_VERSION = Gem::Version.new("0.9.0")
+
+          def version
+            Gem.loaded_specs["ci-queue"]&.version
+          end
+
+          def loaded?
+            !defined?(::RSpec::Queue::Runner).nil?
+          end
+
+          def compatible?
+            super && version >= MINIMUM_VERSION
+          end
+
+          def patcher
+            Patcher
+          end
+        end
+      end
+    end
+  end
+end

data/lib/datadog/ci/contrib/ciqueue/patcher.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+require_relative "../patcher"
+require_relative "../rspec/runner"
+
+module Datadog
+  module CI
+    module Contrib
+      module Ciqueue
+        # Patcher enables patching of 'rspec' module.
+        module Patcher
+          include Datadog::CI::Contrib::Patcher
+
+          module_function
+
+          def patch
+            ::RSpec::Queue::Runner.include(Contrib::RSpec::Runner)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/datadog/ci/contrib/cucumber/formatter.rb

@@ -3,6 +3,7 @@
 require_relative "../../ext/test"
 require_relative "../../git/local_repository"
 require_relative "../../utils/test_run"
+require_relative "../instrumentation"
 require_relative "ext"
 
 module Datadog
@@ -38,9 +39,9 @@ module Datadog
             test_visibility_component.start_test_session(
               tags: {
                 CI::Ext::Test::TAG_FRAMEWORK => Ext::FRAMEWORK,
-                CI::Ext::Test::TAG_FRAMEWORK_VERSION => CI::Contrib::Cucumber::Integration.version.to_s
+                CI::Ext::Test::TAG_FRAMEWORK_VERSION => datadog_integration.version.to_s
               },
-              service: configuration[:service_name]
+              service: datadog_configuration[:service_name]
             )
             test_visibility_component.start_test_module(Ext::FRAMEWORK)
           end
@@ -61,7 +62,7 @@ module Datadog
             # @type var tags: Hash[String, String]
             tags = {
               CI::Ext::Test::TAG_FRAMEWORK => Ext::FRAMEWORK,
-              CI::Ext::Test::TAG_FRAMEWORK_VERSION => CI::Contrib::Cucumber::Integration.version.to_s,
+              CI::Ext::Test::TAG_FRAMEWORK_VERSION => datadog_integration.version.to_s,
               CI::Ext::Test::TAG_SOURCE_FILE => Git::LocalRepository.relative_to_root(event.test_case.location.file),
               CI::Ext::Test::TAG_SOURCE_START => event.test_case.location.line.to_s
             }
@@ -81,7 +82,7 @@ module Datadog
               event.test_case.name,
               test_suite_name,
               tags: tags,
-              service: configuration[:service_name]
+              service: datadog_configuration[:service_name]
             )
             if event.test_case.match_tags?("@#{CI::Ext::Test::ITR_UNSKIPPABLE_OPTION}")
               test_span&.itr_unskippable!
@@ -199,7 +200,11 @@ module Datadog
             end
           end
 
-          def configuration
+          def datadog_integration
+            CI::Contrib::Instrumentation.fetch_integration(:cucumber)
+          end
+
+          def datadog_configuration
             Datadog.configuration.ci[:cucumber]
           end
 

data/lib/datadog/ci/contrib/cucumber/integration.rb

@@ -9,30 +9,21 @@ module Datadog
     module Contrib
       module Cucumber
         # Description of Cucumber integration
-        class Integration
-          include Datadog::CI::Contrib::Integration
-
+        class Integration < Contrib::Integration
           MINIMUM_VERSION = Gem::Version.new("3.0.0")
 
-          register_as :cucumber
-
-          def self.version
+          def version
             Gem.loaded_specs["cucumber"]&.version
           end
 
-          def self.loaded?
-            !defined?(::Cucumber).nil? && !defined?(::Cucumber::Runtime).nil?
+          def loaded?
+            !defined?(::Cucumber).nil? && !defined?(::Cucumber::Runtime).nil? && !defined?(::Cucumber::Configuration).nil?
           end
 
-          def self.compatible?
+          def compatible?
             super && version >= MINIMUM_VERSION
           end
 
-          # test environments should not auto instrument test libraries
-          def auto_instrument?
-            false
-          end
-
           def new_configuration
             Configuration::Settings.new
           end

data/lib/datadog/ci/contrib/cucumber/patcher.rb

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 
-require "datadog/tracing/contrib/patcher"
+require_relative "../patcher"
 
 require_relative "instrumentation"
 
@@ -10,14 +10,10 @@ module Datadog
       module Cucumber
         # Patches 'cucumber' gem.
         module Patcher
-          include Datadog::Tracing::Contrib::Patcher
+          include Datadog::CI::Contrib::Patcher
 
           module_function
 
-          def target_version
-            Integration.version
-          end
-
           def patch
             ::Cucumber::Runtime.include(Instrumentation)
           end

data/lib/datadog/ci/contrib/instrumentation.rb

@@ -0,0 +1,173 @@
+# frozen_string_literal: true
+
+require "datadog/core/utils/only_once"
+
+module Datadog
+  module CI
+    module Contrib
+      module Instrumentation
+        class InvalidIntegrationError < StandardError; end
+
+        @registry = {}
+        @auto_instrumented = false
+
+        def self.registry
+          @registry
+        end
+
+        def self.register_integration(integration_class)
+          @registry[integration_name(integration_class)] = integration_class.new
+        end
+
+        def self.auto_instrumented?
+          @auto_instrumented
+        end
+
+        # Auto instrumentation of all integrations.
+        #
+        # Registers a :script_compiled tracepoint to watch for new Ruby files being loaded.
+        # On every file load it checks if any of the integrations are patchable now.
+        # Only the integrations that are available in the environment are checked.
+        def self.auto_instrument
+          Datadog.logger.debug("Auto instrumenting all integrations...")
+          @auto_instrumented = true
+
+          auto_instrumented_integrations = fetch_auto_instrumented_integrations
+          if auto_instrumented_integrations.empty?
+            Datadog.logger.warn(
+              "Auto instrumentation was requested, but no available integrations were found. " \
+              "Tests will be run without Datadog instrumentation."
+            )
+            return
+          end
+
+          # note that `Kernel.require` might be called from a different thread, so
+          # there is a possibility of concurrent execution of this tracepoint
+          mutex = Mutex.new
+          script_compiled_tracepoint = TracePoint.new(:script_compiled) do |tp|
+            all_patched = true
+
+            mutex.synchronize do
+              auto_instrumented_integrations.each do |integration|
+                next if integration.patched?
+
+                all_patched = false
+                next unless integration.loaded?
+
+                auto_configure_datadog
+
+                Datadog.logger.debug("#{integration.class} is loaded")
+                patch_integration(integration)
+              end
+
+              if all_patched
+                Datadog.logger.debug("All expected integrations are patched, disabling the script_compiled tracepoint")
+
+                tp.disable
+              end
+            end
+          end
+          script_compiled_tracepoint.enable
+        end
+
+        # Manual instrumentation of a specific integration.
+        #
+        # This method is called when user has `c.ci.instrument :integration_name` in their code.
+        def self.instrument(integration_name, options = {}, &block)
+          integration = fetch_integration(integration_name)
+          # when manually instrumented, it might be configured via code
+          integration.configure(options, &block)
+
+          return unless integration.enabled
+
+          patch_integration(integration, with_dependencies: true)
+        end
+
+        # This method instruments all additional test libraries (ex: selenium-webdriver) that need to be instrumented
+        # later in the test suite run.
+        #
+        # It is intended to be called when test session starts to add additional capabilities to test visibility.
+        #
+        # This method does not automatically instrument test frameworks (ex: RSpec, Cucumber, etc), it requires
+        # test framework to be already instrumented.
+        def self.instrument_on_session_start
+          Datadog.logger.debug("Instrumenting all late instrumented integrations...")
+
+          @registry.each do |name, integration|
+            next unless integration.late_instrument?
+            next unless integration.enabled
+
+            Datadog.logger.debug "#{name} is allowed to be late instrumented"
+
+            patch_integration(integration)
+          end
+        end
+
+        def self.fetch_integration(name)
+          @registry[name] ||
+            raise(InvalidIntegrationError, "'#{name}' is not a valid integration.")
+        end
+
+        # take the parent module name and downcase it
+        # for example for Datadog::CI::Contrib::RSpec::Integration it will be :rspec
+        def self.integration_name(subclass)
+          result = subclass.name&.split("::")&.[](-2)&.downcase&.to_sym
+          raise "Integration name could not be derived for #{subclass}" if result.nil?
+          result
+        end
+
+        def self.patch_integration(integration, with_dependencies: false)
+          patch_results = integration.patch
+
+          if patch_results[:ok]
+            Datadog.logger.debug("#{integration.class} is patched")
+
+            return unless with_dependencies
+
+            # try to patch dependant integrations (for example knapsack that depends on rspec)
+            dependants = integration.dependants
+              .map { |name| fetch_integration(name) }
+              .filter { |integration| integration.patchable? }
+
+            Datadog.logger.debug("Found dependent integrations for #{integration.class}: #{dependants}")
+
+            dependants.each do |dependent_integration|
+              patch_integration(dependent_integration, with_dependencies: true)
+            end
+
+          else
+            Datadog.logger.debug("Attention: #{integration.class} is not patched (#{patch_results})")
+          end
+        end
+
+        def self.fetch_auto_instrumented_integrations
+          @registry.filter_map do |name, integration|
+            # ignore integrations that are not in the Gemfile or have incompatible versions
+            next unless integration.compatible?
+
+            # late instrumented integrations will be patched when the test session starts
+            next if integration.late_instrument?
+
+            Datadog.logger.debug("#{name} should be auto instrumented")
+            integration
+          end
+        end
+
+        def self.auto_configure_datadog
+          configure_once.run do
+            Datadog.logger.debug("Applying Datadog configuration in CI mode...")
+            Datadog.configure do |c|
+              c.ci.enabled = true
+              c.tracing.enabled = true
+            end
+          end
+        end
+
+        # This is not thread safe, it is synchronized by the caller in the tracepoint
+        def self.configure_once
+          @configure_once ||= Datadog::Core::Utils::OnlyOnce.new
+        end
+      end
+    end
+  end
+end