ddtrace 0.52.0 → 0.54.2

data/docs/ProfilingDevelopment.md

@@ -97,9 +97,9 @@ The profiler backend links a trace covering a given time interval to the profile
 whenever they share the same `runtime-id`.
 
 To further enable filtering of a profile to show only samples related to a given trace/span, each sample taken by the
-profiler is tagged with the trace_id and span_id for the given trace/span.
+profiler is tagged with the `local root span id` and `span id` for the given trace/span.
 
-This is done using the `Datadog::Profiling::TraceIdentifiers::Helper` that retrieves a trace_id and span_id, if
+This is done using the `Datadog::Profiling::TraceIdentifiers::Helper` that retrieves a `root_span_id` and `span_id`, if
 available, from the supported tracers. This helper is called by the `Collectors::Stack` during sampling.
 
 Note that if a given trace executes too fast, it's possible that the profiler will not contain any samples for that

data/ext/ddtrace_profiling_native_extension/NativeExtensionDesign.md

@@ -0,0 +1,86 @@
+# Profiling Native Extension Design
+
+The profiling native extension is used to implement features which are expensive (in terms of resources) or otherwise
+impossible to implement using Ruby code.
+
+This extension is quite coupled with MRI Ruby ("C Ruby") internals, and is not intended to support other rubies such as
+JRuby or TruffleRuby. When below we say "Ruby", read it as "MRI Ruby".
+
+## Disabling
+
+The profiling native extension can be disabled by setting `DD_PROFILING_NO_EXTENSION=true` when installing or running
+the gem. Setting `DD_PROFILING_NO_EXTENSION` at installation time skips compilation of the extension entirely.
+
+(If you're a customer and needed to use this, please tell us why on <https://github.com/DataDog/dd-trace-rb/issues/new>.)
+
+Currently the profiler can still "limp along" when the native extension is disabled, but the plan is to require it
+in future releases -- e.g. disabling the extension will disable profiling entirely.
+
+## Safety
+
+The profiling native extension is (and must always be) designed to **not cause failures** during gem installation, even
+if some features, Ruby versions, or operating systems are not supported.
+
+E.g. the extension must cleanly build on Ruby 2.1 (or the oldest Ruby version we support at the time) on Windows,
+even if at run time it will effectively do nothing for such a setup.
+
+We have a CI setup to help validate this, but this is really important to keep in mind when adding to or changing the
+existing codebase.
+
+## Usage of private VM headers
+
+To implement some of the features below, we sometimes require access to private Ruby header files (that describe VM
+internal types, structures and functions).
+
+Because these private header files are not included in regular Ruby installations, we have two different workarounds:
+
+1. for Ruby versions >= 2.6 we make use use the Ruby private MJIT header
+2. for Ruby versions < 2.6 (legacy Rubies) we make use of the `debase-ruby_core_source` gem
+
+Functions which make use of these headers are defined in the <private_vm_api_acccess.c> file.
+
+**Important Note**: Our medium/long-term plan is to stop relying on all private Ruby headers, and instead request and
+contribute upstream changes so that they become official public VM APIs.
+
+### Approach 1: Using the Ruby private MJIT header
+
+Ruby versions >= 2.6 introduced a JIT compiler called MJIT. This compiler does not directly generate machine code;
+instead it generates C code and uses the system C compiler to turn it into machine code.
+
+The generated C code `#include`s a private header -- which we reference as "the MJIT header" everywhere.
+The MJIT header gets shipped with all MJIT-enabled Rubies and includes the layout of many internal VM structures;
+and of course the intention is that it is only used by the Ruby MJIT compiler.
+
+This header is placed inside the `include/` directory in a Ruby installation, and is named for that specific Ruby
+version. e.g. `rb_mjit_min_header-2.7.4.h`.
+
+### Approach 2: Using the `debase-ruby_core_source` gem
+
+The [`debase-ruby_core_source`](https://github.com/ruby-debug/debase-ruby_core_source) contains almost no code;
+instead, it just contains per-Ruby-version folders with the private VM headers (`.h`) files for that version.
+
+Thus, even though a regular Ruby installation does not include these files, we can access the copy inside this gem.
+
+## Feature: Getting thread CPU-time clock_ids
+
+* **OS support**: Linux
+* **Ruby support**: 2.6+
+
+To enable CPU-time profiling, we use the `pthread_getcpuclockid(pthread_t thread, clockid_t *clockid)` C function to
+obtain a `clockid_t` that can then be used with the `Process.clock_gettime` method (or directly with the
+`clock_gettime()` C function).
+
+The challenge with using `pthread_getcpuclockid()` is that we need to get the `pthread_t` for a given Ruby `Thread`
+object. We previously did this with a weird combination of monkey patching and `pthread_self()` (effectively patching
+every `Thread` to run `pthread_self()` at initialization time and stash that value somewhere), but this had a number
+of downsides.
+
+The approach we use in the profiling native extension is to reach inside the internal structure of the `Thread` object,
+and extract the `pthread_t` that Ruby itself keeps, but does not expose. This is implemented in the `pthread_id_for()`
+function in `private_vm_api_acccess.c`. Thus, using this trick we can at any point in execution go from a `Thread`
+object into the `clockid_t` that we need.
+
+Note that `pthread_getcpuclockid()` is not available on macOS (nor, obviously, on Windows), and hence this feature
+is currently Linux-specific. Thus, in the <clock_id_from_pthread.c> file we implement the feature for supported Ruby
+setups but if something is missing we instead compile in <clock_id_noop.c> that includes a no-op implementation of the
+feature.

data/ext/ddtrace_profiling_native_extension/clock_id.h

@@ -0,0 +1,4 @@
+#pragma once
+
+void self_test_clock_id();
+VALUE clock_id_for(VALUE self, VALUE thread);

data/ext/ddtrace_profiling_native_extension/clock_id_from_pthread.c

@@ -0,0 +1,52 @@
+#include "extconf.h"
+
+// This file is only compiled on systems where pthread_getcpuclockid() is available;
+// Otherwise we compile clock_id_noop.c
+#ifdef HAVE_PTHREAD_GETCPUCLOCKID
+
+#include <pthread.h>
+#include <time.h>
+#include <errno.h>
+
+#include <ruby.h>
+
+#ifdef RUBY_2_1_WORKAROUND
+#include <thread_native.h>
+#else
+#include <ruby/thread_native.h>
+#endif
+
+#include "private_vm_api_access.h"
+
+#include "clock_id.h"
+
+// Validate that our home-cooked pthread_id_for() matches pthread_self() for the current thread
+void self_test_clock_id() {
+  rb_nativethread_id_t expected_pthread_id = pthread_self();
+  rb_nativethread_id_t actual_pthread_id = pthread_id_for(rb_thread_current());
+
+  if (expected_pthread_id != actual_pthread_id) rb_raise(rb_eRuntimeError, "pthread_id_for() self-test failed");
+}
+
+VALUE clock_id_for(VALUE self, VALUE thread) {
+  rb_nativethread_id_t thread_id = pthread_id_for(thread);
+
+  clockid_t clock_id;
+  int error = pthread_getcpuclockid(thread_id, &clock_id);
+
+  if (error == 0) {
+    return CLOCKID2NUM(clock_id);
+  } else {
+    switch(error) {
+      // The more specific error messages are based on the pthread_getcpuclockid(3) man page
+      case ENOENT:
+        rb_exc_raise(rb_syserr_new(error, "Failed to get clock_id for given thread: Per-thread CPU time clocks are not supported by the system."));
+      case ESRCH:
+        rb_exc_raise(rb_syserr_new(error, "Failed to get clock_id for given thread: No thread could be found."));
+      default:
+        rb_exc_raise(rb_syserr_new(error, "Failed to get clock_id for given thread"));
+    }
+  }
+}
+
+#endif

data/ext/ddtrace_profiling_native_extension/clock_id_noop.c

@@ -0,0 +1,14 @@
+#include "extconf.h"
+
+// This file is the dual of clock_id_from_pthread.c for systems where that info
+// is not available.
+#ifndef HAVE_PTHREAD_GETCPUCLOCKID
+
+#include <ruby.h>
+
+#include "clock_id.h"
+
+void self_test_clock_id() { } // Nothing to check
+VALUE clock_id_for(VALUE self, VALUE thread) { return Qnil; } // Nothing to return
+
+#endif

data/ext/ddtrace_profiling_native_extension/extconf.rb

@@ -1,28 +1,197 @@
-# typed: false
-def skip_building_extension?
+# typed: ignore
+# rubocop:disable Style/StderrPuts
+
+# Older Rubies don't have the MJIT header, used by the JIT compiler, so we need to use a different approach
+CAN_USE_MJIT_HEADER = RUBY_VERSION >= '2.6'
+
+def on_jruby?
   # We don't support JRuby for profiling, and JRuby doesn't support native extensions, so let's just skip this entire
   # thing so that JRuby users of dd-trace-rb aren't impacted.
-  on_jruby = RUBY_ENGINE == 'jruby'
+  RUBY_ENGINE == 'jruby'
+end
+
+def on_truffleruby?
+  # We don't officially support TruffleRuby for dd-trace-rb at all BUT let's not break adventurous customers that
+  # want to give it a try.
+  RUBY_ENGINE == 'truffleruby'
+end
+
+def on_windows?
+  # Microsoft Windows is unsupported, so let's not build the extension there.
+  Gem.win_platform?
+end
+
+def expected_to_use_mjit_but_mjit_is_disabled?
+  # On some Rubies, we require the mjit header to be present. If Ruby was installed without MJIT support, we also skip
+  # building the extension.
+  mjit_disabled = CAN_USE_MJIT_HEADER && RbConfig::CONFIG['MJIT_SUPPORT'] != 'yes'
+
+  if mjit_disabled
+    $stderr.puts(%(
++------------------------------------------------------------------------------+
+| Your Ruby has been compiled without JIT support (--disable-jit-support).     |
+| The profiling native extension requires a Ruby compiled with JIT support,    |
+| even if the JIT is not in use by the application itself.                     |
+|                                                                              |
+| WARNING: Without the profiling native extension, some profiling features     |
+| will not be available.                                                       |
++------------------------------------------------------------------------------+
+
+))
+  end
 
+  mjit_disabled
+end
+
+def disabled_via_env?
   # Experimental toggle to disable building the extension.
   # Disabling the extension will lead to the profiler not working in future releases.
   # If you needed to use this, please tell us why on <https://github.com/DataDog/dd-trace-rb/issues/new>.
-  disabled_via_env = ENV['DD_PROFILING_NO_EXTENSION'].to_s.downcase == 'true'
+  ENV['DD_PROFILING_NO_EXTENSION'].to_s.downcase == 'true'
+end
 
-  on_jruby || disabled_via_env
+def skip_building_extension?
+  disabled_via_env? || on_jruby? || on_truffleruby? || on_windows? || expected_to_use_mjit_but_mjit_is_disabled?
 end
 
-if skip_building_extension?
+# IMPORTANT: When adding flags, remember that our customers compile with a wide range of gcc/clang versions, so
+# doublecheck that what you're adding can be reasonably expected to work on their systems.
+def add_compiler_flag(flag)
+  $CFLAGS << ' ' << flag
+end
+
+def skip_building_extension!
   File.write('Makefile', 'all install clean: # dummy makefile that does nothing')
-  return
+  exit
+end
+
+if skip_building_extension?
+  $stderr.puts(%(
++------------------------------------------------------------------------------+
+| Skipping build of profiling native extension and replacing it with a no-op   |
+| Makefile                                                                     |
++------------------------------------------------------------------------------+
+
+))
+  skip_building_extension!
 end
 
+$stderr.puts(%(
++------------------------------------------------------------------------------+
+| ** Preparing to build the ddtrace native extension... **                     |
+|                                                                              |
+| If you run into any failures during this step, you can set the               |
+| `DD_PROFILING_NO_EXTENSION` environment variable to `true` e.g.              |
+| `$ DD_PROFILING_NO_EXTENSION=true bundle install` to skip this step.         |
+|                                                                              |
+| Disabling the extension will lead to the ddtrace profiling features not      |
+| working in future releases.                                                  |
+| If you needed to use this, please tell us why on                             |
+| <https://github.com/DataDog/dd-trace-rb/issues/new> so we can fix it :\)      |
+|                                                                              |
+| Thanks for using ddtrace! You rock!                                          |
++------------------------------------------------------------------------------+
+
+))
+
 # NOTE: we MUST NOT require 'mkmf' before we check the #skip_building_extension? because the require triggers checks
 # that may fail on an environment not properly setup for building Ruby extensions.
 require 'mkmf'
 
+# Gets really noisy when we include the MJIT header, let's omit it
+add_compiler_flag '-Wno-unused-function'
+
+# Allow defining variables at any point in a function
+add_compiler_flag '-Wno-declaration-after-statement'
+
+# If we forget to include a Ruby header, the function call may still appear to work, but then
+# cause a segfault later. Let's ensure that never happens.
+add_compiler_flag '-Werror-implicit-function-declaration'
+
+if RUBY_PLATFORM.include?('linux')
+  # Supposedly, the correct way to do this is
+  # ```
+  # have_library 'pthread'
+  # have_func 'pthread_getcpuclockid'
+  # ```
+  # but it broke the build on Windows and on older Ruby versions (2.1 and 2.2)
+  # so instead we just assume that we have the function we need on Linux, and nowhere else
+  $defs << '-DHAVE_PTHREAD_GETCPUCLOCKID'
+end
+
 # Tag the native extension library with the Ruby version and Ruby platform.
 # This makes it easier for development (avoids "oops I forgot to rebuild when I switched my Ruby") and ensures that
 # the wrong library is never loaded.
 # When requiring, we need to use the exact same string, including the version and the platform.
-create_makefile "ddtrace_profiling_native_extension.#{RUBY_VERSION}_#{RUBY_PLATFORM}"
+EXTENSION_NAME = "ddtrace_profiling_native_extension.#{RUBY_VERSION}_#{RUBY_PLATFORM}".freeze
+
+if CAN_USE_MJIT_HEADER
+  mjit_header_file_name = "rb_mjit_min_header-#{RUBY_VERSION}.h"
+
+  # Validate that the mjit header can actually be compiled on this system. We learned via
+  # https://github.com/DataDog/dd-trace-rb/issues/1799 and https://github.com/DataDog/dd-trace-rb/issues/1792
+  # that even if the header seems to exist, it may not even compile.
+  # `have_macro` actually tries to compile a file that mentions the given macro, so if this passes, we should be good to
+  # use the MJIT header.
+  # Finally, the `COMMON_HEADERS` conflict with the MJIT header so we need to temporarily disable them for this check.
+  original_common_headers = MakeMakefile::COMMON_HEADERS
+  MakeMakefile::COMMON_HEADERS = ''.freeze
+  unless have_macro('RUBY_MJIT_H', mjit_header_file_name)
+    $stderr.puts(%(
++------------------------------------------------------------------------------+
+| WARNING: Unable to compile a needed component for ddtrace native extension.  |
+| Your C compiler or Ruby VM just-in-time compiler seems to be broken.         |
+|                                                                              |
+| You will be NOT be able to use ddtrace profiling features,                   |
+| but all other features will work fine!                                       |
+|                                                                              |
+| For help solving this issue, please contact Datadog support at               |
+| <https://docs.datadoghq.com/help/>.                                          |
++------------------------------------------------------------------------------+
+
+))
+    skip_building_extension!
+  end
+  MakeMakefile::COMMON_HEADERS = original_common_headers
+
+  $defs << '-DUSE_MJIT_HEADER'
+
+  # NOTE: This needs to come after all changes to $defs
+  create_header
+
+  # The MJIT header is always (afaik?) suffixed with the exact Ruby VM version,
+  # including patch (e.g. 2.7.2). Thus, we add to the header file a definition
+  # containing the exact file, so that it can be used in a #include in the C code.
+  header_contents =
+    File.read($extconf_h)
+        .sub('#endif',
+             <<-EXTCONF_H.strip
+#define RUBY_MJIT_HEADER "#{mjit_header_file_name}"
+
+#endif
+             EXTCONF_H
+            )
+  File.open($extconf_h, 'w') { |file| file.puts(header_contents) }
+
+  create_makefile EXTENSION_NAME
+else
+  # On older Rubies, we use the debase-ruby_core_source gem to get access to private VM headers.
+  # This gem ships source code copies of these VM headers for the different Ruby VM versions;
+  # see https://github.com/ruby-debug/debase-ruby_core_source for details
+
+  thread_native_for_ruby_2_1 = proc { true }
+  if RUBY_VERSION < '2.2'
+    # This header became public in Ruby 2.2, but we need to pull it from the private headers folder for 2.1
+    thread_native_for_ruby_2_1 = proc { have_header('thread_native.h') }
+    $defs << '-DRUBY_2_1_WORKAROUND'
+  end
+
+  create_header
+
+  require 'debase/ruby_core_source'
+  dir_config('ruby') # allow user to pass in non-standard core include directory
+
+  Debase::RubyCoreSource
+    .create_makefile_with_core(proc { have_header('vm_core.h') && thread_native_for_ruby_2_1.call }, EXTENSION_NAME)
+end
+# rubocop:enable Style/StderrPuts

data/ext/ddtrace_profiling_native_extension/private_vm_api_access.c

@@ -0,0 +1,35 @@
+#include "extconf.h"
+
+// This file exports functions used to access private Ruby VM APIs and internals.
+// To do this, it imports a few VM internal (private) headers.
+//
+// **Important Note**: Our medium/long-term plan is to stop relying on all private Ruby headers, and instead request and
+// contribute upstream changes so that they become official public VM APIs.
+//
+// In the meanwhile, be very careful when changing things here :)
+
+#ifdef USE_MJIT_HEADER
+// Pick up internal structures from the private Ruby MJIT header file
+#include RUBY_MJIT_HEADER
+#else
+// On older Rubies, use a copy of the VM internal headers shipped in the debase-ruby_core_source gem
+#include <vm_core.h>
+#endif
+
+// MRI has a similar rb_thread_ptr() function which we can't call it directly
+// because Ruby does not expose the thread_data_type publicly.
+// Instead, we have our own version of that function, and we lazily initialize the thread_data_type pointer
+// from a known-correct object: the current thread.
+//
+// Note that beyond returning the rb_thread_struct*, rb_check_typeddata() raises an exception
+// if the argument passed in is not actually a `Thread` instance.
+static inline struct rb_thread_struct *thread_struct_from_object(VALUE thread) {
+  static const rb_data_type_t *thread_data_type = NULL;
+  if (thread_data_type == NULL) thread_data_type = RTYPEDDATA_TYPE(rb_thread_current());
+
+  return (struct rb_thread_struct *) rb_check_typeddata(thread, thread_data_type);
+}
+
+rb_nativethread_id_t pthread_id_for(VALUE thread) {
+  return thread_struct_from_object(thread)->thread_id;
+}

data/ext/ddtrace_profiling_native_extension/private_vm_api_access.h

@@ -0,0 +1,3 @@
+#pragma once
+
+rb_nativethread_id_t pthread_id_for(VALUE thread);

data/ext/ddtrace_profiling_native_extension/profiling.c

@@ -1,5 +1,6 @@
 #include <ruby.h>
-#include <stdio.h>
+
+#include "clock_id.h"
 
 static VALUE native_working_p(VALUE self);
 
@@ -10,8 +11,12 @@ void Init_ddtrace_profiling_native_extension(void) {
 
   rb_define_singleton_method(native_extension_module, "native_working?", native_working_p, 0);
   rb_funcall(native_extension_module, rb_intern("private_class_method"), 1, ID2SYM(rb_intern("native_working?")));
+
+  rb_define_singleton_method(native_extension_module, "clock_id_for", clock_id_for, 1); // from clock_id.h
 }
 
 static VALUE native_working_p(VALUE self) {
+  self_test_clock_id();
+
   return Qtrue;
 }

data/lib/datadog/ci/contrib/cucumber/formatter.rb

@@ -39,6 +39,7 @@ module Datadog
                   service: configuration[:service_name]
                 },
                 framework: Ext::FRAMEWORK,
+                framework_version: Datadog::CI::Contrib::Cucumber::Integration.version.to_s,
                 test_name: event.test_case.name,
                 test_suite: event.test_case.location.file,
                 test_type: Ext::TEST_TYPE

data/lib/datadog/ci/contrib/rspec/example.rb

@@ -37,6 +37,7 @@ module Datadog
                     service: configuration[:service_name]
                   },
                   framework: Ext::FRAMEWORK,
+                  framework_version: Datadog::CI::Contrib::RSpec::Integration.version.to_s,
                   test_name: test_name,
                   test_suite: file_path,
                   test_type: Ext::TEST_TYPE

data/lib/datadog/ci/contrib/rspec/integration.rb

@@ -17,8 +17,8 @@ module Datadog
           register_as :rspec, auto_patch: true
 
           def self.version
-            Gem.loaded_specs['rspec'] \
-              && Gem.loaded_specs['rspec'].version
+            Gem.loaded_specs['rspec-core'] \
+              && Gem.loaded_specs['rspec-core'].version
           end
 
           def self.loaded?

data/lib/datadog/ci/ext/environment.rb

@@ -40,22 +40,27 @@ module Datadog
         module_function
 
         def tags(env)
+          # Extract metadata from CI provider environment variables
           _, extractor = PROVIDERS.find { |provider_env_var, _| env.key?(provider_env_var) }
-          if extractor
-            tags = public_send(extractor, env)
-
-            tags[Datadog::Ext::Git::TAG_TAG] = normalize_ref(tags[Datadog::Ext::Git::TAG_TAG])
-            tags.delete(Datadog::Ext::Git::TAG_BRANCH) unless tags[Datadog::Ext::Git::TAG_TAG].nil?
-            tags[Datadog::Ext::Git::TAG_BRANCH] = normalize_ref(tags[Datadog::Ext::Git::TAG_BRANCH])
-            tags[Datadog::Ext::Git::TAG_REPOSITORY_URL] = filter_sensitive_info(tags[Datadog::Ext::Git::TAG_REPOSITORY_URL])
-
-            # Expand ~
-            workspace_path = tags[TAG_WORKSPACE_PATH]
-            if !workspace_path.nil? && (workspace_path == '~' || workspace_path.start_with?('~/'))
-              tags[TAG_WORKSPACE_PATH] = File.expand_path(workspace_path)
-            end
-          else
-            tags = {}
+          tags = extractor ? public_send(extractor, env).reject { |_, v| v.nil? || v.strip.empty? } : {}
+          tags.delete(Datadog::Ext::Git::TAG_BRANCH) unless tags[Datadog::Ext::Git::TAG_TAG].nil?
+
+          # If user defined metadata is defined, overwrite
+          tags.merge!(extract_user_defined_git(env))
+          if !tags[Datadog::Ext::Git::TAG_BRANCH].nil? && tags[Datadog::Ext::Git::TAG_BRANCH].include?('tags/')
+            tags[Datadog::Ext::Git::TAG_TAG] = tags[Datadog::Ext::Git::TAG_BRANCH]
+            tags.delete(Datadog::Ext::Git::TAG_BRANCH)
+          end
+
+          # Normalize Git references
+          tags[Datadog::Ext::Git::TAG_TAG] = normalize_ref(tags[Datadog::Ext::Git::TAG_TAG])
+          tags[Datadog::Ext::Git::TAG_BRANCH] = normalize_ref(tags[Datadog::Ext::Git::TAG_BRANCH])
+          tags[Datadog::Ext::Git::TAG_REPOSITORY_URL] = filter_sensitive_info(tags[Datadog::Ext::Git::TAG_REPOSITORY_URL])
+
+          # Expand ~
+          workspace_path = tags[TAG_WORKSPACE_PATH]
+          if !workspace_path.nil? && (workspace_path == '~' || workspace_path.start_with?('~/'))
+            tags[TAG_WORKSPACE_PATH] = File.expand_path(workspace_path)
           end
 
           # Fill out tags from local git as fallback
@@ -131,6 +136,8 @@ module Datadog
             TAG_PIPELINE_NUMBER => build_id,
             TAG_PIPELINE_URL => pipeline_url,
             TAG_JOB_URL => job_url,
+            TAG_STAGE_NAME => env['SYSTEM_STAGEDISPLAYNAME'],
+            TAG_JOB_NAME => env['SYSTEM_JOBDISPLAYNAME'],
             Datadog::Ext::Git::TAG_REPOSITORY_URL =>
               env['SYSTEM_PULLREQUEST_SOURCEREPOSITORYURI'] || env['BUILD_REPOSITORY_URI'],
             Datadog::Ext::Git::TAG_COMMIT_SHA => env['SYSTEM_PULLREQUEST_SOURCECOMMITID'] || env['BUILD_SOURCEVERSION'],
@@ -206,16 +213,19 @@ module Datadog
           ref = env['GITHUB_REF'] if ref.nil? || ref.empty?
           branch, tag = branch_or_tag(ref)
 
+          pipeline_url = "#{env['GITHUB_SERVER_URL']}/#{env['GITHUB_REPOSITORY']}/actions/runs/#{env['GITHUB_RUN_ID']}"
+          pipeline_url = "#{pipeline_url}/attempts/#{env['GITHUB_RUN_ATTEMPT']}" if env['GITHUB_RUN_ATTEMPT']
+
           {
             Datadog::Ext::Git::TAG_BRANCH => branch,
             Datadog::Ext::Git::TAG_COMMIT_SHA => env['GITHUB_SHA'],
-            Datadog::Ext::Git::TAG_REPOSITORY_URL => "https://github.com/#{env['GITHUB_REPOSITORY']}.git",
+            Datadog::Ext::Git::TAG_REPOSITORY_URL => "#{env['GITHUB_SERVER_URL']}/#{env['GITHUB_REPOSITORY']}.git",
             Datadog::Ext::Git::TAG_TAG => tag,
-            TAG_JOB_URL => "https://github.com/#{env['GITHUB_REPOSITORY']}/commit/#{env['GITHUB_SHA']}/checks",
+            TAG_JOB_URL => "#{env['GITHUB_SERVER_URL']}/#{env['GITHUB_REPOSITORY']}/commit/#{env['GITHUB_SHA']}/checks",
             TAG_PIPELINE_ID => env['GITHUB_RUN_ID'],
             TAG_PIPELINE_NAME => env['GITHUB_WORKFLOW'],
             TAG_PIPELINE_NUMBER => env['GITHUB_RUN_NUMBER'],
-            TAG_PIPELINE_URL => "https://github.com/#{env['GITHUB_REPOSITORY']}/commit/#{env['GITHUB_SHA']}/checks",
+            TAG_PIPELINE_URL => pipeline_url,
             TAG_PROVIDER_NAME => 'github',
             TAG_WORKSPACE_PATH => env['GITHUB_WORKSPACE'],
             Datadog::Ext::Git::TAG_COMMIT_AUTHOR_NAME => env['BUILD_REQUESTEDFORID'],
@@ -225,12 +235,17 @@ module Datadog
         end
 
         def extract_gitlab(env)
+          commit_author_name, commit_author_email = extract_name_email(env['CI_COMMIT_AUTHOR'])
+
           url = env['CI_PIPELINE_URL']
           {
-            Datadog::Ext::Git::TAG_BRANCH => env['CI_COMMIT_BRANCH'],
+            Datadog::Ext::Git::TAG_BRANCH => env['CI_COMMIT_REF_NAME'],
             Datadog::Ext::Git::TAG_COMMIT_SHA => env['CI_COMMIT_SHA'],
             Datadog::Ext::Git::TAG_REPOSITORY_URL => env['CI_REPOSITORY_URL'],
             Datadog::Ext::Git::TAG_TAG => env['CI_COMMIT_TAG'],
+            Datadog::Ext::Git::TAG_COMMIT_AUTHOR_NAME => commit_author_name,
+            Datadog::Ext::Git::TAG_COMMIT_AUTHOR_EMAIL => commit_author_email,
+            Datadog::Ext::Git::TAG_COMMIT_AUTHOR_DATE => env['CI_COMMIT_TIMESTAMP'],
             TAG_STAGE_NAME => env['CI_JOB_STAGE'],
             TAG_JOB_NAME => env['CI_JOB_NAME'],
             TAG_JOB_URL => env['CI_JOB_URL'],
@@ -254,7 +269,7 @@ module Datadog
           {
             Datadog::Ext::Git::TAG_BRANCH => branch,
             Datadog::Ext::Git::TAG_COMMIT_SHA => env['GIT_COMMIT'],
-            Datadog::Ext::Git::TAG_REPOSITORY_URL => env['GIT_URL'],
+            Datadog::Ext::Git::TAG_REPOSITORY_URL => env['GIT_URL'] || env['GIT_URL_1'],
             Datadog::Ext::Git::TAG_TAG => tag,
             TAG_PIPELINE_ID => env['BUILD_TAG'],
             TAG_PIPELINE_NAME => name,
@@ -312,7 +327,7 @@ module Datadog
           {
             TAG_PROVIDER_NAME => 'bitrise',
             TAG_PIPELINE_ID => env['BITRISE_BUILD_SLUG'],
-            TAG_PIPELINE_NAME => env['BITRISE_APP_TITLE'],
+            TAG_PIPELINE_NAME => env['BITRISE_TRIGGERED_WORKFLOW_ID'],
             TAG_PIPELINE_NUMBER => env['BITRISE_BUILD_NUMBER'],
             TAG_PIPELINE_URL => env['BITRISE_BUILD_URL'],
             TAG_WORKSPACE_PATH => env['BITRISE_SOURCE_DIR'],
@@ -324,6 +339,22 @@ module Datadog
           }
         end
 
+        def extract_user_defined_git(env)
+          {
+            Datadog::Ext::Git::TAG_REPOSITORY_URL => env[Datadog::Ext::Git::ENV_REPOSITORY_URL],
+            Datadog::Ext::Git::TAG_COMMIT_SHA => env[Datadog::Ext::Git::ENV_COMMIT_SHA],
+            Datadog::Ext::Git::TAG_BRANCH => env[Datadog::Ext::Git::ENV_BRANCH],
+            Datadog::Ext::Git::TAG_TAG => env[Datadog::Ext::Git::ENV_TAG],
+            Datadog::Ext::Git::TAG_COMMIT_MESSAGE => env[Datadog::Ext::Git::ENV_COMMIT_MESSAGE],
+            Datadog::Ext::Git::TAG_COMMIT_AUTHOR_NAME => env[Datadog::Ext::Git::ENV_COMMIT_AUTHOR_NAME],
+            Datadog::Ext::Git::TAG_COMMIT_AUTHOR_EMAIL => env[Datadog::Ext::Git::ENV_COMMIT_AUTHOR_EMAIL],
+            Datadog::Ext::Git::TAG_COMMIT_AUTHOR_DATE => env[Datadog::Ext::Git::ENV_COMMIT_AUTHOR_DATE],
+            Datadog::Ext::Git::TAG_COMMIT_COMMITTER_NAME => env[Datadog::Ext::Git::ENV_COMMIT_COMMITTER_NAME],
+            Datadog::Ext::Git::TAG_COMMIT_COMMITTER_EMAIL => env[Datadog::Ext::Git::ENV_COMMIT_COMMITTER_EMAIL],
+            Datadog::Ext::Git::TAG_COMMIT_COMMITTER_DATE => env[Datadog::Ext::Git::ENV_COMMIT_COMMITTER_DATE]
+          }.reject { |_, v| v.nil? || v.strip.empty? }
+        end
+
         def git_commit_users
           # Get committer and author information in one command.
           output = exec_git_command("git show -s --format='%an\t%ae\t%at\t%cn\t%ce\t%ct'")
@@ -428,7 +459,7 @@ module Datadog
 
         def branch_or_tag(branch_or_tag)
           branch = tag = nil
-          if branch_or_tag.include?('tags/')
+          if branch_or_tag && branch_or_tag.include?('tags/')
             tag = branch_or_tag
           else
             branch = branch_or_tag
@@ -436,6 +467,17 @@ module Datadog
 
           [branch, tag]
         end
+
+        def extract_name_email(name_and_email)
+          if name_and_email.include?('<') && (match = /^([^<]*)<([^>]*)>$/.match(name_and_email))
+            name = match[1]
+            name = name.strip if name
+            email = match[2]
+            return [name, email] if name && email
+          end
+
+          [nil, name_and_email]
+        end
       end
       # rubocop:enable Metrics/ModuleLength:
     end

data/lib/datadog/ci/ext/test.rb

@@ -8,6 +8,7 @@ module Datadog
 
         TAG_ARGUMENTS = 'test.arguments'.freeze
         TAG_FRAMEWORK = 'test.framework'.freeze
+        TAG_FRAMEWORK_VERSION = 'test.framework_version'.freeze
         TAG_NAME = 'test.name'.freeze
         TAG_SKIP_REASON = 'test.skip_reason'.freeze # DEV: Not populated yet
         TAG_STATUS = 'test.status'.freeze

data/lib/datadog/ci/test.rb

@@ -38,10 +38,14 @@ module Datadog
         span.context.origin = Ext::Test::CONTEXT_ORIGIN if span.context
         Datadog::Contrib::Analytics.set_measured(span)
         span.set_tag(Ext::Test::TAG_SPAN_KIND, Ext::AppTypes::TEST)
-        Ext::Environment.tags(ENV).each { |k, v| span.set_tag(k, v) }
+
+        # Set environment tags
+        @environment_tags ||= Ext::Environment.tags(ENV)
+        @environment_tags.each { |k, v| span.set_tag(k, v) }
 
         # Set contextual tags
         span.set_tag(Ext::Test::TAG_FRAMEWORK, tags[:framework]) if tags[:framework]
+        span.set_tag(Ext::Test::TAG_FRAMEWORK_VERSION, tags[:framework_version]) if tags[:framework_version]
         span.set_tag(Ext::Test::TAG_NAME, tags[:test_name]) if tags[:test_name]
         span.set_tag(Ext::Test::TAG_SUITE, tags[:test_suite]) if tags[:test_suite]
         span.set_tag(Ext::Test::TAG_TYPE, tags[:test_type]) if tags[:test_type]