datadog-ci 1.0.0.beta2 → 1.0.0.beta3

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 6b3e03ab226d98aac3dd518b1c554452a4c3812cb67e9a1d64762e2fe8767633
4
- data.tar.gz: eac352d2b2488d16f979004f617097efe109364069f1c7b78cecfceef5365efa
3
+ metadata.gz: 335897204d48926d8c0e3ab1ef8e7c6247a88b12627e27e021614f1b368ab788
4
+ data.tar.gz: db5b4ac716d7e0f28712cb7fef86c1f438f2a8b5fa28af0bb67ef91d3758d1c0
5
5
  SHA512:
6
- metadata.gz: c56ab91025f2a224e86bd732a5c891e961af184b29618241f8853756780ccba894c11955b00020a70139d72e47f57d22dfdf1f2ed6437f1d4e4a9d998453d7a4
7
- data.tar.gz: e7c1aa73096d8d8b49800bbed168f288b7676d8fa328011c3e9c4f7024c86c17cf2c210a94a64dcd80f94121a80b36a824f0ca4e3aa6ccb1c35f682a1fb9e18b
6
+ metadata.gz: c2ebd289964298131bed28406316a48992aaec8e54a6fb21f58ffb6c3d2c523f49fbc5aedf83c9f72a5b8ba45c54cce6df46b8d83eae6ce42804a8901954b61c
7
+ data.tar.gz: dd1fcd1c56179c0ab933d166801f69cb843efe67449ad69e7491ee674ff3325fcb9b1adb814881a80137d24de6fa264103ad02d71b73c814acf84a56b706a707
data/CHANGELOG.md CHANGED
@@ -1,117 +1,129 @@
1
1
  ## [Unreleased]
2
2
 
3
+ ## [1.0.0.beta3] - 2024-04-30
4
+
5
+ ### Added
6
+
7
+ - "why this test was skipped" feature ([#165])
8
+ - custom configurations tags support for ITR ([#166])
9
+ - unskippable tests for ITR ([#167])
10
+
11
+ ### Changed
12
+
13
+ - additional debug logging, do not skip tests when running in forked processes ([#168])
14
+
3
15
  ## [1.0.0.beta2] - 2024-04-23
4
16
 
5
17
  ### Added
6
18
 
7
- * Code coverage events writer ([#150])
8
- * Git tree upload - git command line integration ([#151])
9
- * Add Git::SearchCommits api client ([#152])
10
- * Upload packfiles API client ([#153])
11
- * Git tree uploader ([#154])
12
- * Git repository unshallowing logic ([#155])
13
- * Git upload async worker ([#156])
14
- * Reduce ITR-induced code coverage overhead for default branch ([#157])
15
- * Skippable tests api client ([#158])
16
- * Request skippable tests when configuring ITR ([#159])
17
- * Test skipping implementation ([#160])
19
+ - Code coverage events writer ([#150])
20
+ - Git tree upload - git command line integration ([#151])
21
+ - Add Git::SearchCommits api client ([#152])
22
+ - Upload packfiles API client ([#153])
23
+ - Git tree uploader ([#154])
24
+ - Git repository unshallowing logic ([#155])
25
+ - Git upload async worker ([#156])
26
+ - Reduce ITR-induced code coverage overhead for default branch ([#157])
27
+ - Skippable tests api client ([#158])
28
+ - Request skippable tests when configuring ITR ([#159])
29
+ - Test skipping implementation ([#160])
18
30
 
19
31
  ## [1.0.0.beta1] - 2024-03-25
20
32
 
21
33
  ### Added
22
34
 
23
- * datadog-cov native extension for per test code coverage ([#137])
24
- * citestcov transport to serialize and send code coverage events ([#148])
35
+ - datadog-cov native extension for per test code coverage ([#137])
36
+ - citestcov transport to serialize and send code coverage events ([#148])
25
37
 
26
38
  ### Removed
27
39
 
28
- * Ruby 2.1-2.6 support is dropped
40
+ - Ruby 2.1-2.6 support is dropped
29
41
 
30
42
  ## [0.8.3] - 2024-03-20
31
43
 
32
44
  ### Fixed
33
45
 
34
- * fix: cucumber-ruby 9.2 includes breaking change for Cucumber::Core::Test::Result ([#145][])
46
+ - fix: cucumber-ruby 9.2 includes breaking change for Cucumber::Core::Test::Result ([#145][])
35
47
 
36
48
  ### Changed
37
49
 
38
- * remove temporary hack and use Core::Remote::Negotiation's new constructor param ([#142][])
39
- * use filter_basic_auth method from Datadog::Core ([#141][])
50
+ - remove temporary hack and use Core::Remote::Negotiation's new constructor param ([#142][])
51
+ - use filter_basic_auth method from Datadog::Core ([#141][])
40
52
 
41
53
  ## [0.8.2] - 2024-03-19
42
54
 
43
55
  ### Fixed
44
56
 
45
- * assign the single running test suite for a test if none found by test suite name ([#139][])
57
+ - assign the single running test suite for a test if none found by test suite name ([#139][])
46
58
 
47
59
  ## [0.8.1] - 2024-03-12
48
60
 
49
61
  ### Fixed
50
62
 
51
- * fix minitest instrumentation with mixins ([#134][])
63
+ - fix minitest instrumentation with mixins ([#134][])
52
64
 
53
65
  ## [0.8.0] - 2024-03-08
54
66
 
55
67
  ### Added
56
68
 
57
- * gzip agent payloads support via evp_proxy/v4 ([#123][])
69
+ - gzip agent payloads support via evp_proxy/v4 ([#123][])
58
70
 
59
71
  ### Changed
60
72
 
61
- * Add note to README on using VCR ([#122][])
73
+ - Add note to README on using VCR ([#122][])
62
74
 
63
75
  ### Fixed
64
76
 
65
- * use framework name as test module name to make test fingerprints stable ([#131][])
77
+ - use framework name as test module name to make test fingerprints stable ([#131][])
66
78
 
67
79
  ## [0.7.0] - 2024-01-26
68
80
 
69
81
  ### Added
70
82
 
71
- * Source code integration ([#95][])
72
- * CODEOWNERS support ([#98][])
73
- * Cucumber scenarios with examples are treated as parametrized tests ([#100][])
74
- * Deduplicate dynamically generated RSpec examples using test.parameters ([#101][])
75
- * Repository name is used as default test service name ([#104][])
76
- * Cucumber v9 support ([#99][])
77
- * ci-queue runner support for minitest ([#110][])
78
- * ci-queue support for rspec ([#112][])
83
+ - Source code integration ([#95][])
84
+ - CODEOWNERS support ([#98][])
85
+ - Cucumber scenarios with examples are treated as parametrized tests ([#100][])
86
+ - Deduplicate dynamically generated RSpec examples using test.parameters ([#101][])
87
+ - Repository name is used as default test service name ([#104][])
88
+ - Cucumber v9 support ([#99][])
89
+ - ci-queue runner support for minitest ([#110][])
90
+ - ci-queue support for rspec ([#112][])
79
91
 
80
92
  ### Fixed
81
93
 
82
- * do not publish sig folder when publishing this gem to prevent steep errors in client applications ([#114][])
83
- * minitest: fix rails parallel test runner ([#115][])
84
- * Test suites and tests skipped by frameworks are correctly reported as skipped to Datadog ([#113][])
94
+ - do not publish sig folder when publishing this gem to prevent steep errors in client applications ([#114][])
95
+ - minitest: fix rails parallel test runner ([#115][])
96
+ - Test suites and tests skipped by frameworks are correctly reported as skipped to Datadog ([#113][])
85
97
 
86
98
  ### Changed
87
99
 
88
- * Enable test suite level visibility by default (with killswitch) ([#109][])
89
- * Test suite names are more human-readable now ([#105][])
90
- * Remove span_type method in tracer-related models ([#107][])
91
- * Manual tracing API: convert type parameter to keyword in Datadog::CI.trace, remove internal-only methods from public API ([#108][])
100
+ - Enable test suite level visibility by default (with killswitch) ([#109][])
101
+ - Test suite names are more human-readable now ([#105][])
102
+ - Remove span_type method in tracer-related models ([#107][])
103
+ - Manual tracing API: convert type parameter to keyword in Datadog::CI.trace, remove internal-only methods from public API ([#108][])
92
104
 
93
105
  ## [0.6.0] - 2024-01-03
94
106
 
95
107
  ### Added
96
108
 
97
- * Test suite level visibility instrumentation for RSpec ([#86][])
98
- * Test suite level visibility instrumentation for Cucumber ([#90][])
99
- * Test suite level visibility instrumentation for Minitest framework ([#92][])
109
+ - Test suite level visibility instrumentation for RSpec ([#86][])
110
+ - Test suite level visibility instrumentation for Cucumber ([#90][])
111
+ - Test suite level visibility instrumentation for Minitest framework ([#92][])
100
112
 
101
113
  ### Fixed
102
114
 
103
- * Do not instantiate TestVisibility::Recorder unless CI visibility is enabled ([#89][])
115
+ - Do not instantiate TestVisibility::Recorder unless CI visibility is enabled ([#89][])
104
116
 
105
117
  ## [0.5.1] - 2023-12-11
106
118
 
107
119
  ### Fixed
108
120
 
109
- * do not collect environment tags when CI is not enabled ([#87][])
121
+ - do not collect environment tags when CI is not enabled ([#87][])
110
122
 
111
123
  ### Changed
112
124
 
113
- * Move private classes and modules deeper in module hierarchy ([#85][])
114
- * update appraisal dependencies ([#84][])
125
+ - Move private classes and modules deeper in module hierarchy ([#85][])
126
+ - update appraisal dependencies ([#84][])
115
127
 
116
128
  ## [0.5.0] - 2023-12-06
117
129
 
@@ -123,98 +135,99 @@ Currently test suite level visibility is not used by our instrumentation: it wil
123
135
 
124
136
  ### Added
125
137
 
126
- * Test suite level visibility: add test session public API ([#72][])
127
- * Test suite level visibility: test module support ([#76][])
128
- * Test suite level visibility: test suites support ([#77][])
129
- * add YARD documentation ([#82][])
130
- * support validation errors for CI spans ([#78][])
138
+ - Test suite level visibility: add test session public API ([#72][])
139
+ - Test suite level visibility: test module support ([#76][])
140
+ - Test suite level visibility: test suites support ([#77][])
141
+ - add YARD documentation ([#82][])
142
+ - support validation errors for CI spans ([#78][])
131
143
 
132
144
  ### Changed
133
145
 
134
- * Validate DD_SITE variable ([#79][])
135
- * Document how to use WebMock with datadog-ci ([#80][])
146
+ - Validate DD_SITE variable ([#79][])
147
+ - Document how to use WebMock with datadog-ci ([#80][])
136
148
 
137
149
  ### Fixed
138
150
 
139
- * Datadog::CI.trace_test always starts a new trace ([#74][])
140
- * Skip tracing when CI mode disabled and manual API is used ([#75][])
151
+ - Datadog::CI.trace_test always starts a new trace ([#74][])
152
+ - Skip tracing when CI mode disabled and manual API is used ([#75][])
141
153
 
142
154
  ### Removed
143
155
 
144
- * Deprecate operation name setting, change service_name to service in public API ([#81][])
156
+ - Deprecate operation name setting, change service_name to service in public API ([#81][])
145
157
 
146
158
  ## [0.4.1] - 2023-11-22
147
159
 
148
160
  ### Fixed
149
161
 
150
- * disable 128-bit trace id generation in CI mode ([#70][])
162
+ - disable 128-bit trace id generation in CI mode ([#70][])
151
163
 
152
164
  ## [0.4.0] - 2023-11-21
153
165
 
154
166
  ### Added
155
167
 
156
- * Public API for manual test instrumentation ([#64][]) ([#61][])
168
+ - Public API for manual test instrumentation ([#64][]) ([#61][])
157
169
 
158
170
  ### Changed
159
171
 
160
- * fix tracing instrumentation example in readme ([#60][])
172
+ - fix tracing instrumentation example in readme ([#60][])
161
173
 
162
174
  ### Fixed
163
175
 
164
- * Remove user credentials from ssh URLs and from GITHUB_REPO_URL environment variable ([#66][])
176
+ - Remove user credentials from ssh URLs and from GITHUB_REPO_URL environment variable ([#66][])
165
177
 
166
178
  ### Removed
167
179
 
168
- * Remove _dd.measured tag from spans ([#65][])
180
+ - Remove _dd.measured tag from spans ([#65][])
169
181
 
170
182
  ## [0.3.0] - 2023-10-25
171
183
 
172
184
  ### Added
173
185
 
174
- * Add AWS CodePipeline support for automatic CI tags extraction ([#54][])
175
- * Support test visibility protocol via Datadog Agent with EVP proxy ([#51][])
186
+ - Add AWS CodePipeline support for automatic CI tags extraction ([#54][])
187
+ - Support test visibility protocol via Datadog Agent with EVP proxy ([#51][])
176
188
 
177
189
  ### Changed
178
190
 
179
- * Migrate to Net::HTTP adapter from Core module of ddtrace gem ([#49][])
191
+ - Migrate to Net::HTTP adapter from Core module of ddtrace gem ([#49][])
180
192
 
181
193
  ## [0.2.0] - 2023-10-05
182
194
 
183
195
  ### Added
184
196
 
185
- * [CIAPP-2959] Agentless mode ([#33][])
197
+ - [CIAPP-2959] Agentless mode ([#33][])
186
198
 
187
199
  ### Fixed
188
200
 
189
- * [CIAPP-4278] Fix an issue with emojis in commit message breaking LocalGit tags provider ([#40][])
201
+ - [CIAPP-4278] Fix an issue with emojis in commit message breaking LocalGit tags provider ([#40][])
190
202
 
191
203
  ## [0.1.1] - 2023-09-14
192
204
 
193
205
  ### Fixed
194
206
 
195
- * Fix circular dependencies warnings ([#31][])
207
+ - Fix circular dependencies warnings ([#31][])
196
208
 
197
209
  ## 0.1.0 - 2023-09-12
198
210
 
199
211
  ### Added
200
212
 
201
- * Add cucumber 8.0.0 support ([#7][])
202
- * Docs: contribution documentation ([#14][], [#28][])
203
- * Dev process: issue templates ([#20][])
213
+ - Add cucumber 8.0.0 support ([#7][])
214
+ - Docs: contribution documentation ([#14][], [#28][])
215
+ - Dev process: issue templates ([#20][])
204
216
 
205
217
  ### Changed
206
218
 
207
- * Validate customer-supplied git tags ([#15][])
219
+ - Validate customer-supplied git tags ([#15][])
208
220
 
209
221
  ### Fixed
210
222
 
211
- * Fix Datadog::CI::Environment to support the new CI specs ([#11][])
223
+ - Fix Datadog::CI::Environment to support the new CI specs ([#11][])
212
224
 
213
225
  ### Removed
214
226
 
215
- * Ruby versions < 2.7 no longer supported ([#8][])
227
+ - Ruby versions < 2.7 no longer supported ([#8][])
216
228
 
217
229
  [Unreleased]: https://github.com/DataDog/datadog-ci-rb/compare/v0.8.3...main
230
+ [1.0.0.beta3]: https://github.com/DataDog/datadog-ci-rb/compare/v1.0.0.beta2...v1.0.0.beta3
218
231
  [1.0.0.beta2]: https://github.com/DataDog/datadog-ci-rb/compare/v1.0.0.beta1...v1.0.0.beta2
219
232
  [1.0.0.beta1]: https://github.com/DataDog/datadog-ci-rb/compare/v0.8.3...v1.0.0.beta1
220
233
  [0.8.3]: https://github.com/DataDog/datadog-ci-rb/compare/v0.8.2...v0.8.3
@@ -304,3 +317,7 @@ Currently test suite level visibility is not used by our instrumentation: it wil
304
317
  [#158]: https://github.com/DataDog/datadog-ci-rb/issues/158
305
318
  [#159]: https://github.com/DataDog/datadog-ci-rb/issues/159
306
319
  [#160]: https://github.com/DataDog/datadog-ci-rb/issues/160
320
+ [#165]: https://github.com/DataDog/datadog-ci-rb/issues/165
321
+ [#166]: https://github.com/DataDog/datadog-ci-rb/issues/166
322
+ [#167]: https://github.com/DataDog/datadog-ci-rb/issues/167
323
+ [#168]: https://github.com/DataDog/datadog-ci-rb/issues/168
@@ -13,6 +13,7 @@ require_relative "../test_visibility/serializers/factories/test_suite_level"
13
13
  require_relative "../test_visibility/transport"
14
14
  require_relative "../transport/api/builder"
15
15
  require_relative "../transport/remote_settings_api"
16
+ require_relative "../utils/test_run"
16
17
  require_relative "../worker"
17
18
 
18
19
  module Datadog
@@ -59,6 +60,9 @@ module Datadog
59
60
  # Choose user defined TraceFlush or default to CI TraceFlush
60
61
  settings.tracing.test_mode.trace_flush = settings.ci.trace_flush || CI::TestVisibility::Flush::Partial.new
61
62
 
63
+ # startup logs are useless for CI visibility and create noise
64
+ settings.diagnostics.startup_logs.enabled = false
65
+
62
66
  # transport creation
63
67
  writer_options = settings.ci.writer_options
64
68
  coverage_writer = nil
@@ -90,14 +94,18 @@ module Datadog
90
94
 
91
95
  settings.tracing.test_mode.writer_options = writer_options
92
96
 
97
+ custom_configuration_tags = Utils::TestRun.custom_configuration(settings.tags)
98
+
93
99
  remote_settings_api = Transport::RemoteSettingsApi.new(
94
100
  api: test_visibility_api,
95
- dd_env: settings.env
101
+ dd_env: settings.env,
102
+ config_tags: custom_configuration_tags
96
103
  )
97
104
 
98
105
  itr = ITR::Runner.new(
99
106
  api: test_visibility_api,
100
107
  dd_env: settings.env,
108
+ config_tags: custom_configuration_tags,
101
109
  coverage_writer: coverage_writer,
102
110
  enabled: settings.ci.enabled && settings.ci.itr_enabled
103
111
  )
@@ -70,12 +70,15 @@ module Datadog
70
70
 
71
71
  start_test_suite(test_suite_name) unless same_test_suite_as_current?(test_suite_name)
72
72
 
73
- CI.start_test(
73
+ test_span = CI.start_test(
74
74
  event.test_case.name,
75
75
  test_suite_name,
76
76
  tags: tags,
77
77
  service: configuration[:service_name]
78
78
  )
79
+ if event.test_case.match_tags?("@#{CI::Ext::Test::ITR_UNSKIPPABLE_OPTION}")
80
+ test_span&.itr_unskippable!
81
+ end
79
82
  end
80
83
 
81
84
  def on_test_case_finished(event)
@@ -2,7 +2,7 @@
2
2
 
3
3
  require_relative "runner"
4
4
  require_relative "reporter"
5
- require_relative "hooks"
5
+ require_relative "test"
6
6
  require_relative "runnable"
7
7
 
8
8
  module Datadog
@@ -25,7 +25,7 @@ module Datadog
25
25
  # test suites (when not executed concurrently)
26
26
  ::Minitest::Runnable.include(Runnable)
27
27
  # tests; test suites (when executed concurrently)
28
- ::Minitest::Test.include(Hooks)
28
+ ::Minitest::Test.include(Test)
29
29
  # test session finish
30
30
  ::Minitest::CompositeReporter.include(Reporter)
31
31
  end
@@ -0,0 +1,105 @@
1
+ # frozen_string_literal: true
2
+
3
+ require_relative "../../ext/test"
4
+ require_relative "../../git/local_repository"
5
+ require_relative "ext"
6
+ require_relative "helpers"
7
+
8
+ module Datadog
9
+ module CI
10
+ module Contrib
11
+ module Minitest
12
+ # Lifecycle hooks to instrument Minitest::Test
13
+ module Test
14
+ def self.included(base)
15
+ base.prepend(InstanceMethods)
16
+ base.singleton_class.prepend(ClassMethods)
17
+ end
18
+
19
+ module InstanceMethods
20
+ def before_setup
21
+ super
22
+ return unless datadog_configuration[:enabled]
23
+
24
+ test_suite_name = Helpers.test_suite_name(self.class, name)
25
+ if Helpers.parallel?(self.class)
26
+ test_suite_name = "#{test_suite_name} (#{name} concurrently)"
27
+
28
+ # for parallel execution we need to start a new test suite for each test
29
+ CI.start_test_suite(test_suite_name)
30
+ end
31
+
32
+ source_file, line_number = method(name).source_location
33
+
34
+ test_span = CI.start_test(
35
+ name,
36
+ test_suite_name,
37
+ tags: {
38
+ CI::Ext::Test::TAG_FRAMEWORK => Ext::FRAMEWORK,
39
+ CI::Ext::Test::TAG_FRAMEWORK_VERSION => CI::Contrib::Minitest::Integration.version.to_s,
40
+ CI::Ext::Test::TAG_SOURCE_FILE => Git::LocalRepository.relative_to_root(source_file),
41
+ CI::Ext::Test::TAG_SOURCE_START => line_number.to_s
42
+ },
43
+ service: datadog_configuration[:service_name]
44
+ )
45
+ test_span&.itr_unskippable! if self.class.dd_suite_unskippable? || self.class.dd_test_unskippable?(name)
46
+ skip(CI::Ext::Test::ITR_TEST_SKIP_REASON) if test_span&.skipped_by_itr?
47
+ end
48
+
49
+ def after_teardown
50
+ test_span = CI.active_test
51
+ return super unless test_span
52
+
53
+ finish_with_result(test_span, result_code)
54
+ if Helpers.parallel?(self.class)
55
+ finish_with_result(test_span.test_suite, result_code)
56
+ end
57
+
58
+ super
59
+ end
60
+
61
+ private
62
+
63
+ def finish_with_result(span, result_code)
64
+ return unless span
65
+
66
+ case result_code
67
+ when "."
68
+ span.passed!
69
+ when "E", "F"
70
+ span.failed!(exception: failure)
71
+ when "S"
72
+ span.skipped!(reason: failure.message)
73
+ end
74
+ span.finish
75
+ end
76
+
77
+ def datadog_configuration
78
+ Datadog.configuration.ci[:minitest]
79
+ end
80
+ end
81
+
82
+ module ClassMethods
83
+ def datadog_itr_unskippable(*args)
84
+ if args.nil? || args.empty?
85
+ @datadog_itr_unskippable_suite = true
86
+ else
87
+ @datadog_itr_unskippable_tests = args
88
+ end
89
+ end
90
+
91
+ def dd_suite_unskippable?
92
+ @datadog_itr_unskippable_suite
93
+ end
94
+
95
+ def dd_test_unskippable?(test_name)
96
+ return false unless @datadog_itr_unskippable_tests
97
+
98
+ @datadog_itr_unskippable_tests.include?(test_name)
99
+ end
100
+ end
101
+ end
102
+ end
103
+ end
104
+ end
105
+ end
@@ -50,6 +50,8 @@ module Datadog
50
50
  },
51
51
  service: datadog_configuration[:service_name]
52
52
  ) do |test_span|
53
+ test_span&.itr_unskippable! if metadata[CI::Ext::Test::ITR_UNSKIPPABLE_OPTION]
54
+
53
55
  metadata[:skip] = CI::Ext::Test::ITR_TEST_SKIP_REASON if test_span&.skipped_by_itr?
54
56
 
55
57
  result = super
@@ -8,7 +8,6 @@ module Datadog
8
8
  module Test
9
9
  CONTEXT_ORIGIN = "ciapp-test"
10
10
 
11
- TAG_ARGUMENTS = "test.arguments"
12
11
  TAG_FRAMEWORK = "test.framework"
13
12
  TAG_FRAMEWORK_VERSION = "test.framework_version"
14
13
  TAG_NAME = "test.name"
@@ -29,6 +28,8 @@ module Datadog
29
28
  TAG_ITR_TEST_SKIPPING_COUNT = "test.itr.tests_skipping.count"
30
29
  TAG_ITR_SKIPPED_BY_ITR = "test.skipped_by_itr"
31
30
  TAG_ITR_TESTS_SKIPPED = "_dd.ci.itr.tests_skipped"
31
+ TAG_ITR_UNSKIPPABLE = "test.itr.unskippable"
32
+ TAG_ITR_FORCED_RUN = "test.itr.forced_run"
32
33
 
33
34
  # Code coverage tags
34
35
  TAG_CODE_COVERAGE_ENABLED = "test.code_coverage.enabled"
@@ -58,6 +59,7 @@ module Datadog
58
59
  # we use test skipping for Ruby
59
60
  ITR_TEST_SKIPPING_MODE = "test"
60
61
  ITR_TEST_SKIP_REASON = "Skipped by Datadog's intelligent test runner"
62
+ ITR_UNSKIPPABLE_OPTION = :datadog_itr_unskippable
61
63
 
62
64
  # test status as recognized by Datadog
63
65
  module Status
@@ -58,9 +58,15 @@ module Datadog
58
58
  def perform(*events)
59
59
  responses = transport.send_events(events)
60
60
 
61
- loop_back_off! if responses.find(&:server_error?)
61
+ if responses.find(&:server_error?)
62
+ loop_back_off!
63
+ Datadog.logger.warn { "Encountered server error while sending coverage events" }
64
+ end
62
65
 
63
66
  nil
67
+ rescue => e
68
+ Datadog.logger.warn { "Error while sending coverage events: #{e}" }
69
+ loop_back_off!
64
70
  end
65
71
 
66
72
  def stop(force_stop = false, timeout = @shutdown_timeout)
@@ -27,6 +27,7 @@ module Datadog
27
27
 
28
28
  def initialize(
29
29
  dd_env:,
30
+ config_tags: {},
30
31
  api: nil,
31
32
  coverage_writer: nil,
32
33
  enabled: false
@@ -34,6 +35,7 @@ module Datadog
34
35
  @enabled = enabled
35
36
  @api = api
36
37
  @dd_env = dd_env
38
+ @config_tags = config_tags || {}
37
39
 
38
40
  @test_skipping_enabled = false
39
41
  @code_coverage_enabled = false
@@ -127,6 +129,11 @@ module Datadog
127
129
 
128
130
  skippable_test_id = Utils::TestRun.skippable_test_id(test.name, test.test_suite_name, test.parameters)
129
131
  if @skippable_tests.include?(skippable_test_id)
132
+ if forked?
133
+ Datadog.logger.warn { "ITR is not supported for forking test runners yet" }
134
+ return
135
+ end
136
+
130
137
  test.set_tag(Ext::Test::TAG_ITR_SKIPPED_BY_ITR, "true")
131
138
 
132
139
  Datadog.logger.debug { "Marked test as skippable: #{skippable_test_id}" }
@@ -136,13 +143,13 @@ module Datadog
136
143
  end
137
144
 
138
145
  def count_skipped_test(test)
146
+ return if !test.skipped? || !test.skipped_by_itr?
147
+
139
148
  if forked?
140
149
  Datadog.logger.warn { "ITR is not supported for forking test runners yet" }
141
150
  return
142
151
  end
143
152
 
144
- return if !test.skipped? || !test.skipped_by_itr?
145
-
146
153
  @mutex.synchronize do
147
154
  @skipped_tests_count += 1
148
155
  end
@@ -151,6 +158,9 @@ module Datadog
151
158
  def write_test_session_tags(test_session)
152
159
  return if !enabled?
153
160
 
161
+ Datadog.logger.debug { "Finished ITR session with test skipping enabled: #{@test_skipping_enabled}" }
162
+ Datadog.logger.debug { "#{@skipped_tests_count} tests were skipped" }
163
+
154
164
  test_session.set_tag(Ext::Test::TAG_ITR_TESTS_SKIPPED, @skipped_tests_count.positive?.to_s)
155
165
  test_session.set_tag(Ext::Test::TAG_ITR_TEST_SKIPPING_COUNT, @skipped_tests_count)
156
166
  end
@@ -191,11 +201,15 @@ module Datadog
191
201
  # we can only request skippable tests if git metadata is already uploaded
192
202
  git_tree_upload_worker.wait_until_done
193
203
 
194
- skippable_response = Skippable.new(api: @api, dd_env: @dd_env).fetch_skippable_tests(test_session)
204
+ skippable_response =
205
+ Skippable.new(api: @api, dd_env: @dd_env, config_tags: @config_tags)
206
+ .fetch_skippable_tests(test_session)
207
+
195
208
  @correlation_id = skippable_response.correlation_id
196
209
  @skippable_tests = skippable_response.tests
197
210
 
198
211
  Datadog.logger.debug { "Fetched skippable tests: \n #{@skippable_tests}" }
212
+ Datadog.logger.debug { "Found #{@skippable_tests.count} skippable tests." }
199
213
  Datadog.logger.debug { "ITR correlation ID: #{@correlation_id}" }
200
214
  end
201
215
  end
@@ -57,9 +57,10 @@ module Datadog
57
57
  end
58
58
  end
59
59
 
60
- def initialize(dd_env:, api: nil)
60
+ def initialize(dd_env:, api: nil, config_tags: {})
61
61
  @api = api
62
62
  @dd_env = dd_env
63
+ @config_tags = config_tags
63
64
  end
64
65
 
65
66
  def fetch_skippable_tests(test_session)
@@ -94,7 +95,8 @@ module Datadog
94
95
  Ext::Test::TAG_OS_ARCHITECTURE => test_session.os_architecture,
95
96
  Ext::Test::TAG_OS_VERSION => test_session.os_version,
96
97
  Ext::Test::TAG_RUNTIME_NAME => test_session.runtime_name,
97
- Ext::Test::TAG_RUNTIME_VERSION => test_session.runtime_version
98
+ Ext::Test::TAG_RUNTIME_VERSION => test_session.runtime_version,
99
+ "custom" => @config_tags
98
100
  }
99
101
  }
100
102
  }
@@ -102,6 +102,13 @@ module Datadog
102
102
  tracer_span.set_tag(key, value)
103
103
  end
104
104
 
105
+ # Removes tag by key.
106
+ # @param [String] key the key of the tag.
107
+ # @return [void]
108
+ def clear_tag(key)
109
+ tracer_span.clear_tag(key)
110
+ end
111
+
105
112
  # Sets metric value by key.
106
113
  # @param [String] key the key of the metric.
107
114
  # @param [Numeric] value the value of the metric.
@@ -69,6 +69,24 @@ module Datadog
69
69
  get_tag(Ext::Test::TAG_ITR_SKIPPED_BY_ITR) == "true"
70
70
  end
71
71
 
72
+ # Marks this test as unskippable by the intelligent test runner.
73
+ # This must be done before the test execution starts.
74
+ #
75
+ # Examples of tests that should be unskippable:
76
+ # - tests that read files from disk
77
+ # - tests that make network requests
78
+ # - tests that call external processes
79
+ # - tests that use forking or threading
80
+ #
81
+ # @return [void]
82
+ def itr_unskippable!
83
+ set_tag(Ext::Test::TAG_ITR_UNSKIPPABLE, "true")
84
+ if skipped_by_itr?
85
+ clear_tag(Ext::Test::TAG_ITR_SKIPPED_BY_ITR)
86
+ set_tag(Ext::Test::TAG_ITR_FORCED_RUN, "true")
87
+ end
88
+ end
89
+
72
90
  # Sets the status of the span to "pass".
73
91
  # @return [void]
74
92
  def passed!
@@ -28,11 +28,12 @@ module Datadog
28
28
  "duration"
29
29
  ].freeze
30
30
 
31
- attr_reader :trace, :span, :meta
31
+ attr_reader :trace, :span, :meta, :options
32
32
 
33
- def initialize(trace, span)
33
+ def initialize(trace, span, options: {})
34
34
  @trace = trace
35
35
  @span = span
36
+ @options = options
36
37
 
37
38
  @meta = @span.meta.reject { |key, _| Ext::Test::TRANSIENT_TAGS.include?(key) }
38
39
 
@@ -14,12 +14,12 @@ module Datadog
14
14
  module TestLevel
15
15
  module_function
16
16
 
17
- def serializer(trace, span)
17
+ def serializer(trace, span, options: {})
18
18
  case span.type
19
19
  when Datadog::CI::Ext::AppTypes::TYPE_TEST
20
- Serializers::TestV1.new(trace, span)
20
+ Serializers::TestV1.new(trace, span, options: options)
21
21
  else
22
- Serializers::Span.new(trace, span)
22
+ Serializers::Span.new(trace, span, options: options)
23
23
  end
24
24
  end
25
25
  end
@@ -15,18 +15,18 @@ module Datadog
15
15
  module TestSuiteLevel
16
16
  module_function
17
17
 
18
- def serializer(trace, span)
18
+ def serializer(trace, span, options: {})
19
19
  case span.type
20
20
  when Datadog::CI::Ext::AppTypes::TYPE_TEST
21
- Serializers::TestV2.new(trace, span)
21
+ Serializers::TestV2.new(trace, span, options: options)
22
22
  when Datadog::CI::Ext::AppTypes::TYPE_TEST_SESSION
23
- Serializers::TestSession.new(trace, span)
23
+ Serializers::TestSession.new(trace, span, options: options)
24
24
  when Datadog::CI::Ext::AppTypes::TYPE_TEST_MODULE
25
- Serializers::TestModule.new(trace, span)
25
+ Serializers::TestModule.new(trace, span, options: options)
26
26
  when Datadog::CI::Ext::AppTypes::TYPE_TEST_SUITE
27
- Serializers::TestSuite.new(trace, span)
27
+ Serializers::TestSuite.new(trace, span, options: options)
28
28
  else
29
- Serializers::Span.new(trace, span)
29
+ Serializers::Span.new(trace, span, options: options)
30
30
  end
31
31
  end
32
32
  end
@@ -10,22 +10,34 @@ module Datadog
10
10
  class TestV2 < TestV1
11
11
  CONTENT_FIELDS = (["test_session_id", "test_module_id", "test_suite_id"] + TestV1::CONTENT_FIELDS).freeze
12
12
 
13
+ CONTENT_FIELDS_WITH_ITR_CORRELATION_ID = (CONTENT_FIELDS + ["itr_correlation_id"]).freeze
14
+
13
15
  CONTENT_MAP_SIZE = calculate_content_map_size(CONTENT_FIELDS)
14
16
 
17
+ CONTENT_MAP_SIZE_WITH_ITR_CORRELATION_ID = calculate_content_map_size(CONTENT_FIELDS_WITH_ITR_CORRELATION_ID)
18
+
15
19
  REQUIRED_FIELDS = (["test_session_id", "test_module_id", "test_suite_id"] + TestV1::REQUIRED_FIELDS).freeze
16
20
 
17
21
  def content_fields
18
- CONTENT_FIELDS
22
+ return CONTENT_FIELDS if itr_correlation_id.nil?
23
+
24
+ CONTENT_FIELDS_WITH_ITR_CORRELATION_ID
19
25
  end
20
26
 
21
27
  def content_map_size
22
- CONTENT_MAP_SIZE
28
+ return CONTENT_MAP_SIZE if itr_correlation_id.nil?
29
+
30
+ CONTENT_MAP_SIZE_WITH_ITR_CORRELATION_ID
23
31
  end
24
32
 
25
33
  def version
26
34
  2
27
35
  end
28
36
 
37
+ def itr_correlation_id
38
+ options[:itr_correlation_id]
39
+ end
40
+
29
41
  private
30
42
 
31
43
  def required_fields
@@ -45,7 +45,7 @@ module Datadog
45
45
  end
46
46
 
47
47
  def encode_span(trace, span)
48
- serializer = serializers_factory.serializer(trace, span)
48
+ serializer = serializers_factory.serializer(trace, span, options: {itr_correlation_id: itr&.correlation_id})
49
49
 
50
50
  if serializer.valid?
51
51
  encoded = encoder.encode(serializer)
@@ -98,6 +98,10 @@ module Datadog
98
98
 
99
99
  packer.write("events")
100
100
  end
101
+
102
+ def itr
103
+ @itr ||= Datadog::CI.send(:itr_runner)
104
+ end
101
105
  end
102
106
  end
103
107
  end
@@ -51,9 +51,10 @@ module Datadog
51
51
  end
52
52
  end
53
53
 
54
- def initialize(dd_env:, api: nil)
54
+ def initialize(dd_env:, api: nil, config_tags: {})
55
55
  @api = api
56
56
  @dd_env = dd_env
57
+ @config_tags = config_tags || {}
57
58
  end
58
59
 
59
60
  def fetch_library_settings(test_session)
@@ -90,7 +91,8 @@ module Datadog
90
91
  Ext::Test::TAG_OS_ARCHITECTURE => test_session.os_architecture,
91
92
  Ext::Test::TAG_OS_VERSION => test_session.os_version,
92
93
  Ext::Test::TAG_RUNTIME_NAME => test_session.runtime_name,
93
- Ext::Test::TAG_RUNTIME_VERSION => test_session.runtime_version
94
+ Ext::Test::TAG_RUNTIME_VERSION => test_session.runtime_version,
95
+ "custom" => @config_tags
94
96
  }
95
97
  }
96
98
  }
@@ -22,6 +22,18 @@ module Datadog
22
22
  }
23
23
  )
24
24
  end
25
+
26
+ def self.custom_configuration(env_tags)
27
+ return {} if env_tags.nil?
28
+
29
+ res = {}
30
+ env_tags.each do |tag, value|
31
+ next unless tag.start_with?("test.configuration.")
32
+
33
+ res[tag.sub("test.configuration.", "")] = value
34
+ end
35
+ res
36
+ end
25
37
  end
26
38
  end
27
39
  end
@@ -6,7 +6,7 @@ module Datadog
6
6
  MAJOR = "1"
7
7
  MINOR = "0"
8
8
  PATCH = "0"
9
- PRE = "beta2"
9
+ PRE = "beta3"
10
10
  BUILD = nil
11
11
  # PRE and BUILD above are modified for dev gems during gem build GHA workflow
12
12
 
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: datadog-ci
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.0.beta2
4
+ version: 1.0.0.beta3
5
5
  platform: ruby
6
6
  authors:
7
7
  - Datadog, Inc.
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2024-04-24 00:00:00.000000000 Z
11
+ date: 2024-04-30 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: datadog
@@ -77,12 +77,12 @@ files:
77
77
  - lib/datadog/ci/contrib/minitest/configuration/settings.rb
78
78
  - lib/datadog/ci/contrib/minitest/ext.rb
79
79
  - lib/datadog/ci/contrib/minitest/helpers.rb
80
- - lib/datadog/ci/contrib/minitest/hooks.rb
81
80
  - lib/datadog/ci/contrib/minitest/integration.rb
82
81
  - lib/datadog/ci/contrib/minitest/patcher.rb
83
82
  - lib/datadog/ci/contrib/minitest/reporter.rb
84
83
  - lib/datadog/ci/contrib/minitest/runnable.rb
85
84
  - lib/datadog/ci/contrib/minitest/runner.rb
85
+ - lib/datadog/ci/contrib/minitest/test.rb
86
86
  - lib/datadog/ci/contrib/rspec/configuration/settings.rb
87
87
  - lib/datadog/ci/contrib/rspec/example.rb
88
88
  - lib/datadog/ci/contrib/rspec/example_group.rb
@@ -188,7 +188,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
188
188
  - !ruby/object:Gem::Version
189
189
  version: 2.0.0
190
190
  requirements: []
191
- rubygems_version: 3.5.6
191
+ rubygems_version: 3.5.9
192
192
  signing_key:
193
193
  specification_version: 4
194
194
  summary: Datadog CI visibility for your ruby application
@@ -1,77 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- require_relative "../../ext/test"
4
- require_relative "../../git/local_repository"
5
- require_relative "ext"
6
- require_relative "helpers"
7
-
8
- module Datadog
9
- module CI
10
- module Contrib
11
- module Minitest
12
- # Lifecycle hooks to instrument Minitest::Test
13
- module Hooks
14
- def before_setup
15
- super
16
- return unless datadog_configuration[:enabled]
17
-
18
- test_suite_name = Helpers.test_suite_name(self.class, name)
19
- if Helpers.parallel?(self.class)
20
- test_suite_name = "#{test_suite_name} (#{name} concurrently)"
21
-
22
- # for parallel execution we need to start a new test suite for each test
23
- CI.start_test_suite(test_suite_name)
24
- end
25
-
26
- source_file, line_number = method(name).source_location
27
-
28
- test_span = CI.start_test(
29
- name,
30
- test_suite_name,
31
- tags: {
32
- CI::Ext::Test::TAG_FRAMEWORK => Ext::FRAMEWORK,
33
- CI::Ext::Test::TAG_FRAMEWORK_VERSION => CI::Contrib::Minitest::Integration.version.to_s,
34
- CI::Ext::Test::TAG_SOURCE_FILE => Git::LocalRepository.relative_to_root(source_file),
35
- CI::Ext::Test::TAG_SOURCE_START => line_number.to_s
36
- },
37
- service: datadog_configuration[:service_name]
38
- )
39
- skip(CI::Ext::Test::ITR_TEST_SKIP_REASON) if test_span&.skipped_by_itr?
40
- end
41
-
42
- def after_teardown
43
- test_span = CI.active_test
44
- return super unless test_span
45
-
46
- finish_with_result(test_span, result_code)
47
- if Helpers.parallel?(self.class)
48
- finish_with_result(test_span.test_suite, result_code)
49
- end
50
-
51
- super
52
- end
53
-
54
- private
55
-
56
- def finish_with_result(span, result_code)
57
- return unless span
58
-
59
- case result_code
60
- when "."
61
- span.passed!
62
- when "E", "F"
63
- span.failed!(exception: failure)
64
- when "S"
65
- span.skipped!(reason: failure.message)
66
- end
67
- span.finish
68
- end
69
-
70
- def datadog_configuration
71
- Datadog.configuration.ci[:minitest]
72
- end
73
- end
74
- end
75
- end
76
- end
77
- end