ddtrace 0.47.0 → 0.48.0

data/docs/ProfilingDevelopment.md

@@ -0,0 +1,88 @@
+# Profiling Development
+
+This file contains development notes specific to the profiling feature.
+
+For a more practical view of getting started with development of `ddtrace`, see <DevelopmentGuide.md>.
+
+## Profiling components high-level view
+
+Components below live inside <../lib/ddtrace/profiling>:
+
+* `Collectors::Stack`: Collects stack trace samples from Ruby threads for both CPU-time (if available) and wall-clock.
+  Runs on its own background thread.
+* `Encoding::Profile`: Encodes gathered data into the pprof format.
+* `Events::Stack`, `Events::StackSample`, `Events::StackExceptionSample`: Entity classes used to represent stacks.
+* `Ext::CPU`: Monkey patches Ruby's `Thread` with our `Ext::CThread` to enable CPU-time profiling.
+* `Ext::CThread`: Extension used to enable CPU-time profiling via use of Pthread's `getcpuclockid`.
+* `Ext::Forking`: Monkey patches `Kernel#fork`, adding a `Kernel#at_fork` callback mechanism which is used to restore
+  profiling abilities after the VM forks (such as re-instrumenting the main thread, and restarting profiler threads).
+* `Pprof::*` (in <../lib/ddtrace/profiling/pprof>): Converts samples captured in the `Recorder` into the pprof format.
+* `Tasks::Setup`: Takes care of loading our extensions/monkey patches to handle fork() and CPU profiling.
+* `Transport::*` (in <../lib/ddtrace/profiling/transport>): Implements transmission of profiling payloads to the Datadog agent
+  or backend.
+* `BacktraceLocation`: Entity class used to represent an entry in a stack trace.
+* `Buffer`: Bounded buffer used to store profiling events.
+* `Exporter`: Writes profiling data to a given transport.
+* `Flush`: Entity class used to represent metadata for a given profile.
+* `Profiler`: Profiling entry point, which coordinates collectors and a scheduler.
+* `Recorder`: Stores profiling events gathered by `Collector`s.
+* `Scheduler`: Periodically (every 1 minute) takes data from the `Recorder` and pushes them to all configured
+  `Exporter`s. Runs on its own background thread.
+
+## Initialization
+
+When started via `ddtracerb exec` (together with `DD_PROFILING_ENABLED=true`), initialization goes through the following
+flow:
+
+1. <../lib/ddtrace/profiling/preload.rb> triggers the creation of the `Datadog.profiler` instance by calling the method
+2. `Datadog.profiler` is handled by `Datadog::Configuration`, which triggers the configuration of `ddtrace` components
+   in `#build_components`
+3. Inside `Datadog::Components`, the `build_profiler` method triggers the execution of the `Tasks::Setup`
+4. The `Setup` task activates our extensions
+    * `Datadog::Profiling::Ext::Forking`
+    * `Datadog::Profiling::Ext::CPU`
+5. Still inside `Datadog::Components`, the `build_profiler` method then creates and wires up the Profiler:
+    ```ruby
+          recorder = build_profiler_recorder(settings)
+          collectors = build_profiler_collectors(settings, recorder)
+          exporters = build_profiler_exporters(settings)
+          scheduler = build_profiler_scheduler(settings, recorder, exporters)
+
+          Datadog::Profiler.new(collectors, scheduler)
+    ```
+    ```asciiflow
+            +------------+
+            |  Profiler  |
+            +-+--------+-+
+              |        |
+              v        v
+    +---------+--+  +--+--------+
+    | Collectors |  | Scheduler |
+    +---------+--+  +-+-------+-+
+              |       |       |
+              v       |       v
+        +-----+-+     |  +----+------+
+        | Stack |     |  | Exporters |
+        +-----+-+     |  +-----------+
+              |       |
+              v       v
+            +-+-------+-+
+            | Recorder  |
+            +-----------+
+    ```
+6. The profiler gets started when `startup!` is called by `Datadog::Configuration` after component creation.
+
+## Run-time execution
+
+During run-time, the `Scheduler` and the `Collectors::Stack` each execute on their own background thread.
+
+The `Collectors::Stack` samples stack traces of threads, capturing both CPU-time (if available) and wall-clock, storing
+them in the `Recorder`.
+
+The `Scheduler` wakes up every 1 minute to flush the results of the `Recorder` into one or more `exporter`s.
+Usually only one exporter is in use. By default, the `Exporter` delegates to the default `Transport::HTTP` transport, which
+takes care of encoding the data and reporting it to the datadog agent (or to the API, when running without an agent).
+
+## How CPU-time profiling works
+
+**TODO**: Document our pthread-based approach to getting CPU-time for threads.

data/integration/README.md

@@ -6,8 +6,7 @@ Integration tests for `ddtrace` that use a variety of real applications.
 
 1. Build Docker base images:
 
-    ```sh
-    #!/bin/bash
+    ```bash
     ./script/build-images
     ```
 

data/integration/apps/rack/Dockerfile

@@ -15,6 +15,9 @@ ENV DD_DEMO_ENV_GEM_REF_DDTRACE ${ddtrace_ref}
 
 # Install dependencies
 COPY Gemfile /app/Gemfile
+# This forces gems with native extensions to be compiled, rather than using pre-compiled binaries; it's needed because
+# some google-protobuf versions ship with missing binaries for older rubies.
+ENV BUNDLE_FORCE_RUBY_PLATFORM true
 RUN bundle install
 
 # Add files

data/integration/apps/rack/script/build-images

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 set -euo pipefail
 
 APP_SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

data/integration/apps/rack/script/ci

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 set -euo pipefail
 
 APP_SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

data/integration/apps/rails-five/script/build-images

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 set -euo pipefail
 
 APP_SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

data/integration/apps/rails-five/script/ci

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 set -euo pipefail
 
 APP_SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

data/integration/apps/ruby/script/build-images

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 set -euo pipefail
 
 APP_SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

data/integration/apps/ruby/script/ci

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 set -euo pipefail
 
 APP_SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

data/integration/images/include/http-health-check

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 set -euo pipefail
 
 # If health check URL provided, wait till it passes.

data/integration/images/wrk/scripts/entrypoint.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 bash /vendor/dd-demo/http-health-check
 
 # Start the load test

data/integration/script/build-images

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 set -euo pipefail
 
 INTEGRATION_SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

data/lib/ddtrace.rb

@@ -15,6 +15,7 @@ require 'ddtrace/configuration'
 require 'ddtrace/patcher'
 require 'ddtrace/metrics'
 require 'ddtrace/auto_instrument_base'
+require 'ddtrace/profiling'
 
 # \Datadog global namespace that includes all tracing functionality for Tracer and Span classes.
 module Datadog

data/lib/ddtrace/configuration.rb

@@ -2,10 +2,11 @@ require 'forwardable'
 require 'ddtrace/configuration/pin_setup'
 require 'ddtrace/configuration/settings'
 require 'ddtrace/configuration/components'
+require 'ddtrace/utils/only_once'
 
 module Datadog
   # Configuration provides a unique access point for configurations
-  module Configuration
+  module Configuration # rubocop:disable Metrics/ModuleLength
     extend Forwardable
 
     # Used to ensure that @components initialization/reconfiguration is performed one-at-a-time, by a single thread.
@@ -42,6 +43,8 @@ module Datadog
     end
 
     def configure(target = configuration, opts = {})
+      ruby_version_deprecation_warning
+
       if target.is_a?(Settings)
         yield(target) if block_given?
 
@@ -64,6 +67,7 @@ module Datadog
     def_delegators \
       :components,
       :health_metrics,
+      :profiler,
       :runtime_metrics,
       :tracer
 
@@ -94,18 +98,6 @@ module Datadog
       end
     end
 
-    # Gracefully shuts down the tracer and disposes of component references,
-    # allowing execution to start anew.
-    #
-    # In contrast with +#shutdown!+, components will be automatically
-    # reinitialized after a reset.
-    def reset!
-      safely_synchronize do |write_components|
-        @components.shutdown! if components?
-        write_components.call(nil)
-      end
-    end
-
     protected
 
     def components(allow_initialization: true)
@@ -119,6 +111,21 @@ module Datadog
 
     private
 
+    # Gracefully shuts down the tracer and disposes of component references,
+    # allowing execution to start anew.
+    #
+    # In contrast with +#shutdown!+, components will be automatically
+    # reinitialized after a reset.
+    #
+    # Used internally to ensure a clean environment between test runs.
+    def reset!
+      safely_synchronize do |write_components|
+        @components.shutdown! if components?
+        write_components.call(nil)
+        configuration.reset!
+      end
+    end
+
     def safely_synchronize
       # Writes to @components should only happen through this proc. Because this proc is only accessible to callers of
       # safely_synchronize, this forces all writers to go through this method.
@@ -168,5 +175,24 @@ module Datadog
         logger
       end
     end
+
+    # Perform version check only once
+    DEPRECATED_RUBY_VERSION = Gem::Version.new(RUBY_VERSION) < Gem::Version.new('2.1')
+    private_constant :DEPRECATED_RUBY_VERSION
+
+    RUBY_VERSION_DEPRECATION_ONLY_ONCE = Datadog::Utils::OnlyOnce.new
+    private_constant :RUBY_VERSION_DEPRECATION_ONLY_ONCE
+
+    def ruby_version_deprecation_warning
+      return unless DEPRECATED_RUBY_VERSION
+
+      RUBY_VERSION_DEPRECATION_ONLY_ONCE.run do
+        Datadog.logger.warn(
+          "Support for Ruby versions < 2.1 in dd-trace-rb is DEPRECATED.\n" \
+          "Last version to support Ruby < 2.1 will be 0.49.x, which will only receive critical bugfixes.\n" \
+          'Support for Ruby versions < 2.1 will be REMOVED in version 0.50.0.'
+        )
+      end
+    end
   end
 end

data/lib/ddtrace/configuration/components.rb

@@ -1,5 +1,6 @@
 require 'ddtrace/diagnostics/health'
 require 'ddtrace/logger'
+require 'ddtrace/profiling'
 require 'ddtrace/runtime/metrics'
 require 'ddtrace/tracer'
 require 'ddtrace/workers/runtime_metrics'
@@ -8,6 +9,7 @@ module Datadog
   module Configuration
     # Global components for the trace library.
     # rubocop:disable Layout/LineLength
+    # rubocop:disable Metrics/ClassLength
     class Components
       class << self
         def build_health_metrics(settings)
@@ -66,6 +68,22 @@ module Datadog
           tracer
         end
 
+        def build_profiler(settings)
+          return unless Datadog::Profiling.supported? && settings.profiling.enabled
+
+          # Load extensions needed to support some of the Profiling features
+          Datadog::Profiling::Tasks::Setup.new.run
+
+          # NOTE: Please update the Initialization section of ProfilingDevelopment.md with any changes to this method
+
+          recorder = build_profiler_recorder(settings)
+          collectors = build_profiler_collectors(settings, recorder)
+          exporters = build_profiler_exporters(settings)
+          scheduler = build_profiler_scheduler(settings, recorder, exporters)
+
+          Datadog::Profiler.new(collectors, scheduler)
+        end
+
         private
 
         def build_tracer_tags(settings)
@@ -90,11 +108,51 @@ module Datadog
             opts[:writer_options] = settings.writer_options if settings.writer.nil?
           end
         end
+
+        def build_profiler_recorder(settings)
+          event_classes = [Datadog::Profiling::Events::StackSample]
+
+          Datadog::Profiling::Recorder.new(event_classes, settings.profiling.max_events)
+        end
+
+        def build_profiler_collectors(settings, recorder)
+          [
+            Datadog::Profiling::Collectors::Stack.new(
+              recorder,
+              max_frames: settings.profiling.max_frames
+              # TODO: Provide proc that identifies Datadog worker threads?
+              # ignore_thread: settings.profiling.ignore_profiler
+            )
+          ]
+        end
+
+        def build_profiler_exporters(settings)
+          if settings.profiling.exporter.instances.is_a?(Array)
+            settings.profiling.exporter.instances
+          else
+            transport = if settings.profiling.exporter.transport
+                          settings.profiling.exporter.transport
+                        else
+                          transport_options = settings.profiling.exporter.transport_options.dup
+                          transport_options[:site] ||= settings.site if settings.site
+                          transport_options[:api_key] ||= settings.api_key if settings.api_key
+                          transport_options[:timeout] ||= settings.profiling.upload.timeout
+                          Datadog::Profiling::Transport::HTTP.default(transport_options)
+                        end
+
+            [Datadog::Profiling::Exporter.new(transport)]
+          end
+        end
+
+        def build_profiler_scheduler(settings, recorder, exporters)
+          Datadog::Profiling::Scheduler.new(recorder, exporters)
+        end
       end
 
       attr_reader \
         :health_metrics,
         :logger,
+        :profiler,
         :runtime_metrics,
         :tracer
 
@@ -105,6 +163,9 @@ module Datadog
         # Tracer
         @tracer = self.class.build_tracer(settings)
 
+        # Profiler
+        @profiler = self.class.build_profiler(settings)
+
         # Runtime metrics
         @runtime_metrics = self.class.build_runtime_metrics_worker(settings)
 
@@ -113,7 +174,20 @@ module Datadog
       end
 
       # Starts up components
-      def startup!(settings); end
+      def startup!(settings)
+        if settings.profiling.enabled
+          if profiler
+            @logger.debug('Profiling started')
+            profiler.start
+          else
+            # Display a warning for users who expected profiling to autostart
+            protobuf = Datadog::Profiling.google_protobuf_supported?
+            logger.warn("Profiling was enabled but is not supported; profiling disabled. (google-protobuf?: #{protobuf})")
+          end
+        else
+          @logger.debug('Profiling is disabled')
+        end
+      end
 
       # Shuts down all the components in use.
       # If it has another instance to compare to, it will compare
@@ -123,12 +197,20 @@ module Datadog
         # (e.g. a custom tracer instance passed in.)
         tracer.shutdown! unless replacement && tracer == replacement.tracer
 
+        # Shutdown old profiler
+        profiler.shutdown! unless profiler.nil?
+
         # Shutdown workers
-        runtime_metrics.enabled = false
-        runtime_metrics.stop(true)
+        runtime_metrics.stop(true, close_metrics: false)
 
         # Shutdown the old metrics, unless they are still being used.
         # (e.g. custom Statsd instances.)
+        #
+        # TODO: This violates the encapsulation created by Runtime::Metrics and
+        # Health::Metrics, by directly manipulating `statsd` and changing
+        # it's lifecycle management.
+        # If we need to directly have ownership of `statsd` lifecycle, we should
+        # have direct ownership of it.
         old_statsd = [
           runtime_metrics.metrics.statsd,
           health_metrics.statsd

data/lib/ddtrace/configuration/settings.rb

@@ -3,6 +3,7 @@ require 'ddtrace/configuration/base'
 
 require 'ddtrace/ext/analytics'
 require 'ddtrace/ext/distributed'
+require 'ddtrace/ext/profiling'
 require 'ddtrace/ext/runtime'
 require 'ddtrace/ext/sampling'
 
@@ -108,6 +109,36 @@ module Datadog
         get_option(:logger).instance = logger
       end
 
+      settings :profiling do
+        option :enabled do |o|
+          o.default { env_to_bool(Ext::Profiling::ENV_ENABLED, false) }
+          o.lazy
+        end
+
+        settings :exporter do
+          option :instances
+          option :transport
+          option :transport_options, default: ->(_o) { {} }, lazy: true
+        end
+
+        option :max_events, default: 32768
+
+        # Controls the maximum number of frames for each thread sampled. Can be tuned to avoid omitted frames in the
+        # produced profiles. Increasing this may increase the overhead of profiling.
+        option :max_frames do |o|
+          o.default { env_to_int(Ext::Profiling::ENV_MAX_FRAMES, 400) }
+          o.lazy
+        end
+
+        settings :upload do
+          option :timeout do |o|
+            o.setter { |value| value.nil? ? 30.0 : value.to_f }
+            o.default { env_to_float(Ext::Profiling::ENV_UPLOAD_TIMEOUT, 30.0) }
+            o.lazy
+          end
+        end
+      end
+
       option :report_hostname do |o|
         o.default { env_to_bool(Ext::NET::ENV_REPORT_HOSTNAME, false) }
         o.lazy