ddtrace 0.54.2 → 1.4.0

data/docs/ProfilingDevelopment.md

@@ -1,107 +0,0 @@
-# Profiling Development
-
-This file contains development notes specific to the profiling feature.
-
-For a more practical view of getting started with development of `ddtrace`, see <DevelopmentGuide.md>.
-
-## Profiling components high-level view
-
-Components below live inside <../lib/ddtrace/profiling>:
-
-* `Collectors::Stack`: Collects stack trace samples from Ruby threads for both CPU-time (if available) and wall-clock.
-  Runs on its own background thread.
-* `Encoding::Profile`: Encodes gathered data into the pprof format.
-* `Events::Stack`, `Events::StackSample`: Entity classes used to represent stacks.
-* `Ext::CPU`: Monkey patches Ruby's `Thread` with our `Ext::CThread` to enable CPU-time profiling.
-* `Ext::CThread`: Extension used to enable CPU-time profiling via use of Pthread's `getcpuclockid`.
-* `Ext::Forking`: Monkey patches `Kernel#fork`, adding a `Kernel#at_fork` callback mechanism which is used to restore
-  profiling abilities after the VM forks (such as re-instrumenting the main thread, and restarting profiler threads).
-* `Pprof::*` (in <../lib/ddtrace/profiling/pprof>): Converts samples captured in the `Recorder` into the pprof format.
-* `Tasks::Setup`: Takes care of loading our extensions/monkey patches to handle fork() and CPU profiling.
-* `Transport::*` (in <../lib/ddtrace/profiling/transport>): Implements transmission of profiling payloads to the Datadog agent
-  or backend.
-* `TraceIdentifiers::*`: Used to retrieve trace id and span id from tracers, to be used to connect traces to profiles.
-* `BacktraceLocation`: Entity class used to represent an entry in a stack trace.
-* `Buffer`: Bounded buffer used to store profiling events.
-* `Exporter`: Writes profiling data to a given transport.
-* `Flush`: Entity class used to represent metadata for a given profile.
-* `Profiler`: Profiling entry point, which coordinates collectors and a scheduler.
-* `Recorder`: Stores profiling events gathered by `Collector`s.
-* `Scheduler`: Periodically (every 1 minute) takes data from the `Recorder` and pushes them to all configured
-  `Exporter`s. Runs on its own background thread.
-
-## Initialization
-
-When started via `ddtracerb exec` (together with `DD_PROFILING_ENABLED=true`), initialization goes through the following
-flow:
-
-1. <../lib/ddtrace/profiling/preload.rb> triggers the creation of the `Datadog.profiler` instance by calling the method
-2. `Datadog.profiler` is handled by `Datadog::Configuration`, which triggers the configuration of `ddtrace` components
-   in `#build_components`
-3. Inside `Datadog::Components`, the `build_profiler` method triggers the execution of the `Tasks::Setup`
-4. The `Setup` task activates our extensions
-    * `Datadog::Profiling::Ext::Forking`
-    * `Datadog::Profiling::Ext::CPU`
-5. Still inside `Datadog::Components`, the `build_profiler` method then creates and wires up the Profiler as such:
-    ```asciiflow
-            +------------+
-            |  Profiler  |
-            +-+-------+--+
-              |       |
-              v       v
-    +---------+--+  +-+---------+
-    | Collectors |  | Scheduler |
-    +---------+--+  +-+-------+-+
-              |       |       |
-              v       |       v
-        +-----+-+     |  +----+------+
-        | Stack |     |  | Exporters |
-        +-----+-+     |  +-----------+
-              |       |
-              v       v
-            +-+-------+-+
-            | Recorder  |
-            +-----------+
-    ```
-6. The profiler gets started when `startup!` is called by `Datadog::Configuration` after component creation.
-
-## Run-time execution
-
-During run-time, the `Scheduler` and the `Collectors::Stack` each execute on their own background thread.
-
-The `Collectors::Stack` samples stack traces of threads, capturing both CPU-time (if available) and wall-clock, storing
-them in the `Recorder`.
-
-The `Scheduler` wakes up every 1 minute to flush the results of the `Recorder` into one or more `exporter`s.
-Usually only one exporter is in use. By default, the `Exporter` delegates to the default `Transport::HTTP` transport, which
-takes care of encoding the data and reporting it to the datadog agent (or to the API, when running without an agent).
-
-## How CPU-time profiling works
-
-**TODO**: Document our pthread-based approach to getting CPU-time for threads.
-
-## How linking of traces to profiles works
-
-The [code hotspots feature](https://docs.datadoghq.com/tracing/profiler/connect_traces_and_profiles) allows users to start
-from a trace and then to investigate the profile that corresponds to that trace.
-
-This works in two steps:
-1. Linking a trace to the profile that was gathered while it executed
-2. Enabling the filtering of a profile to contain only the samples relating to a given trace/span
-
-To link a trace to a profile, we must ensure that both have the same `runtime-id` tag.
-This tag is in `Datadog::Runtime::Identity.id` and is automatically added by both the tracer and the profiler to reported
-traces/profiles.
-
-The profiler backend links a trace covering a given time interval to the profiles covering the same time interval,
-whenever they share the same `runtime-id`.
-
-To further enable filtering of a profile to show only samples related to a given trace/span, each sample taken by the
-profiler is tagged with the `local root span id` and `span id` for the given trace/span.
-
-This is done using the `Datadog::Profiling::TraceIdentifiers::Helper` that retrieves a `root_span_id` and `span_id`, if
-available, from the supported tracers. This helper is called by the `Collectors::Stack` during sampling.
-
-Note that if a given trace executes too fast, it's possible that the profiler will not contain any samples for that
-specific trace. Nevertheless, the linking still works and is useful, as it allows users to explore what was going on their
-profile at that time, even if they can't filter down to the specific request.

data/lib/datadog/ci/context_flush.rb

@@ -1,29 +0,0 @@
-# typed: true
-require 'ddtrace/context_flush'
-
-module Datadog
-  module CI
-    module ContextFlush
-      # Common behavior for CI flushing
-      module Tagging
-        # Decorate a trace with CI tags
-        def get_trace(context)
-          context.get do |trace|
-            # Origin tag is required on every span
-            trace.each { |span| context.attach_origin(span) } if trace
-          end
-        end
-      end
-
-      # Consumes only completed traces (where all spans have finished)
-      class Finished < Datadog::ContextFlush::Finished
-        prepend Tagging
-      end
-
-      # Performs partial trace flushing to avoid large traces residing in memory for too long
-      class Partial < Datadog::ContextFlush::Partial
-        prepend Tagging
-      end
-    end
-  end
-end

data/lib/datadog/contrib.rb

@@ -1,71 +0,0 @@
-# typed: true
-require 'ddtrace/contrib/registry'
-
-module Datadog
-  module Contrib
-    # Registry is a global, declarative repository of all available integrations.
-    #
-    # Integrations should register themselves with the registry as early as
-    # possible as the initial tracer configuration can only activate integrations
-    # if they have already been registered.
-    #
-    # Despite that, integrations *can* be appended to the registry at any point
-    # of the program execution. Newly appended integrations can then be
-    # activated during tracer reconfiguration.
-    #
-    # The registry does not depend on runtime configuration and registered integrations
-    # must execute correctly after successive configuration changes.
-    # The registered integrations themselves can depend on the stateful configuration
-    # of the tracer.
-    #
-    # `Datadog.registry` is a helper accessor to this constant, but it's only available
-    # after the tracer has complete initialization. Use `Datadog::Contrib::REGISTRY` instead
-    # of `Datadog.registry` when you code might be called during tracer initialization.
-    REGISTRY = Registry.new
-  end
-end
-
-require 'ddtrace/contrib/action_cable/integration'
-require 'ddtrace/contrib/action_mailer/integration'
-require 'ddtrace/contrib/action_pack/integration'
-require 'ddtrace/contrib/action_view/integration'
-require 'ddtrace/contrib/active_model_serializers/integration'
-require 'ddtrace/contrib/active_job/integration'
-require 'ddtrace/contrib/active_record/integration'
-require 'ddtrace/contrib/active_support/integration'
-require 'ddtrace/contrib/aws/integration'
-require 'ddtrace/contrib/concurrent_ruby/integration'
-require 'ddtrace/contrib/dalli/integration'
-require 'ddtrace/contrib/delayed_job/integration'
-require 'ddtrace/contrib/elasticsearch/integration'
-require 'ddtrace/contrib/ethon/integration'
-require 'ddtrace/contrib/excon/integration'
-require 'ddtrace/contrib/faraday/integration'
-require 'ddtrace/contrib/grape/integration'
-require 'ddtrace/contrib/graphql/integration'
-require 'ddtrace/contrib/grpc/integration'
-require 'ddtrace/contrib/http/integration'
-require 'ddtrace/contrib/httpclient/integration'
-require 'ddtrace/contrib/httprb/integration'
-require 'ddtrace/contrib/integration'
-require 'ddtrace/contrib/kafka/integration'
-require 'ddtrace/contrib/lograge/integration'
-require 'ddtrace/contrib/mongodb/integration'
-require 'ddtrace/contrib/mysql2/integration'
-require 'ddtrace/contrib/presto/integration'
-require 'ddtrace/contrib/qless/integration'
-require 'ddtrace/contrib/que/integration'
-require 'ddtrace/contrib/racecar/integration'
-require 'ddtrace/contrib/rack/integration'
-require 'ddtrace/contrib/rails/integration'
-require 'ddtrace/contrib/rake/integration'
-require 'ddtrace/contrib/redis/integration'
-require 'ddtrace/contrib/resque/integration'
-require 'ddtrace/contrib/rest_client/integration'
-require 'ddtrace/contrib/semantic_logger/integration'
-require 'ddtrace/contrib/sequel/integration'
-require 'ddtrace/contrib/shoryuken/integration'
-require 'ddtrace/contrib/sidekiq/integration'
-require 'ddtrace/contrib/sinatra/integration'
-require 'ddtrace/contrib/sneakers/integration'
-require 'ddtrace/contrib/sucker_punch/integration'

data/lib/ddtrace/analytics.rb

@@ -1,39 +0,0 @@
-# typed: true
-require 'ddtrace/ext/analytics'
-
-module Datadog
-  # Defines analytics behavior
-  module Analytics
-    class << self
-      def set_sample_rate(span, sample_rate)
-        return if span.nil? || !sample_rate.is_a?(Numeric)
-
-        span.set_metric(Datadog::Ext::Analytics::TAG_SAMPLE_RATE, sample_rate)
-      end
-
-      def set_measured(span, value = true)
-        return if span.nil?
-
-        # rubocop:disable Style/MultipleComparison
-        value = value == true || value == 1 ? 1 : 0
-        span.set_metric(Datadog::Ext::Analytics::TAG_MEASURED, value)
-      end
-    end
-
-    # Extension for Datadog::Span
-    module Span
-      def set_tag(key, value)
-        case key
-        when Ext::Analytics::TAG_ENABLED
-          # If true, set rate to 1.0, otherwise set 0.0.
-          value = value == true ? Ext::Analytics::DEFAULT_SAMPLE_RATE : 0.0
-          Analytics.set_sample_rate(self, value)
-        when Ext::Analytics::TAG_SAMPLE_RATE
-          Analytics.set_sample_rate(self, value)
-        else
-          super if defined?(super)
-        end
-      end
-    end
-  end
-end

data/lib/ddtrace/buffer.rb

@@ -1,340 +0,0 @@
-# typed: true
-require 'ddtrace/diagnostics/health'
-
-# Trace buffer that accumulates traces for a consumer.
-# Consumption can happen from a different thread.
-module Datadog
-  # Buffer that stores objects. The buffer has a maximum size and when
-  # the buffer is full, a random object is discarded.
-  class Buffer
-    def initialize(max_size)
-      @max_size = max_size
-      @items = []
-      @closed = false
-    end
-
-    # Add a new ``item`` in the local queue. This method doesn't block the execution
-    # even if the buffer is full.
-    #
-    # When the buffer is full, we try to ensure that we are fairly sampling newly
-    # pushed traces by randomly inserting them into the buffer slots. This discards
-    # old traces randomly while trying to ensure that recent traces are still captured.
-    def push(item)
-      return if closed?
-
-      full? ? replace!(item) : add!(item)
-      item
-    end
-
-    # A bulk push alternative to +#push+. Use this method if
-    # pushing more than one item for efficiency.
-    def concat(items)
-      return if closed?
-
-      # Segment items into underflow and overflow
-      underflow, overflow = overflow_segments(items)
-
-      # Concatenate items do not exceed capacity.
-      add_all!(underflow) unless underflow.nil?
-
-      # Iteratively replace items, to ensure pseudo-random replacement.
-      overflow.each { |item| replace!(item) } unless overflow.nil?
-    end
-
-    # Stored items are returned and the local buffer is reset.
-    def pop
-      drain!
-    end
-
-    # Return the current number of stored traces.
-    def length
-      @items.length
-    end
-
-    # Return if the buffer is empty.
-    def empty?
-      @items.empty?
-    end
-
-    # Closes this buffer, preventing further pushing.
-    # Draining is still allowed.
-    def close
-      @closed = true
-    end
-
-    def closed?
-      @closed
-    end
-
-    protected
-
-    # Segment items into two segments: underflow and overflow.
-    # Underflow are items that will fit into buffer.
-    # Overflow are items that will exceed capacity, after underflow is added.
-    # Returns each array, and nil if there is no underflow/overflow.
-    def overflow_segments(items)
-      underflow = nil
-      overflow = nil
-
-      overflow_size = @max_size > 0 ? (@items.length + items.length) - @max_size : 0
-
-      if overflow_size > 0
-        # Items will overflow
-        if overflow_size < items.length
-          # Partial overflow
-          underflow_end_index = items.length - overflow_size - 1
-          underflow = items[0..underflow_end_index]
-          overflow = items[(underflow_end_index + 1)..-1]
-        else
-          # Total overflow
-          overflow = items
-        end
-      else
-        # Items do not exceed capacity.
-        underflow = items
-      end
-
-      [underflow, overflow]
-    end
-
-    def full?
-      @max_size > 0 && @items.length >= @max_size
-    end
-
-    def add_all!(items)
-      @items.concat(items)
-    end
-
-    def add!(item)
-      @items << item
-    end
-
-    def replace!(item)
-      # Choose random item to be replaced
-      replace_index = rand(@items.length)
-
-      # Replace random item
-      discarded_item = @items[replace_index]
-      @items[replace_index] = item
-
-      # Return discarded item
-      discarded_item
-    end
-
-    def drain!
-      items = @items
-      @items = []
-      items
-    end
-  end
-
-  # Buffer that stores objects, has a maximum size, and
-  # can be safely used concurrently on any environment.
-  #
-  # This implementation uses a {Mutex} around public methods, incurring
-  # overhead in order to ensure thread-safety.
-  #
-  # This is implementation is recommended for non-CRuby environments.
-  # If using CRuby, {Datadog::CRubyBuffer} is a faster implementation with minimal compromise.
-  class ThreadSafeBuffer < Buffer
-    def initialize(max_size)
-      super
-
-      @mutex = Mutex.new
-    end
-
-    # Add a new ``item`` in the local queue. This method doesn't block the execution
-    # even if the buffer is full. In that case, a random item is discarded.
-    def push(item)
-      synchronize { super }
-    end
-
-    def concat(items)
-      synchronize { super }
-    end
-
-    # Return the current number of stored traces.
-    def length
-      synchronize { super }
-    end
-
-    # Return if the buffer is empty.
-    def empty?
-      synchronize { super }
-    end
-
-    # Stored traces are returned and the local buffer is reset.
-    def pop
-      synchronize { super }
-    end
-
-    def close
-      synchronize { super }
-    end
-
-    def synchronize(&block)
-      @mutex.synchronize(&block)
-    end
-  end
-
-  # Buffer that stores objects, has a maximum size, and
-  # can be safely used concurrently with CRuby.
-  #
-  # Because singular +Array+ operations are thread-safe in CRuby,
-  # we can implement the trace buffer without an explicit lock,
-  # while making the compromise of allowing the buffer to go
-  # over its maximum limit under extreme circumstances.
-  #
-  # On the following scenario:
-  # * 4.5 million spans/second.
-  # * Pushed into a single CRubyTraceBuffer from 1000 threads.
-  #
-  # This implementation allocates less memory and is faster
-  # than {Datadog::ThreadSafeBuffer}.
-  #
-  # @see spec/ddtrace/benchmark/buffer_benchmark_spec.rb Buffer benchmarks
-  # @see https://github.com/ruby-concurrency/concurrent-ruby/blob/c1114a0c6891d9634f019f1f9fe58dcae8658964/lib/concurrent-ruby/concurrent/array.rb#L23-L27
-  class CRubyBuffer < Buffer
-    # A very large number to allow us to effectively
-    # drop all items when invoking `slice!(i, FIXNUM_MAX)`.
-    FIXNUM_MAX = (1 << 62) - 1
-
-    # Add a new ``trace`` in the local queue. This method doesn't block the execution
-    # even if the buffer is full. In that case, a random trace is discarded.
-    def replace!(item)
-      # Ensure buffer stays within +max_size+ items.
-      # This can happen when there's concurrent modification
-      # between a call the check in `full?` and the `add!` call in
-      # `full? ? replace!(item) : add!(item)`.
-      #
-      # We can still have `@items.size > @max_size` for a short period of
-      # time, but we will always try to correct it here.
-      #
-      # `slice!` is performed before `delete_at` & `<<` to avoid always
-      # removing the item that was just inserted.
-      #
-      # DEV: `slice!` with two integer arguments is ~10% faster than
-      # `slice!` with a {Range} argument.
-      @items.slice!(@max_size, FIXNUM_MAX)
-
-      # We should replace a random trace with the new one
-      replace_index = rand(@max_size)
-      @items[replace_index] = item
-    end
-  end
-
-  # Health metrics for trace buffers.
-  module MeasuredBuffer
-    include Kernel # Ensure that kernel methods are always available (https://sorbet.org/docs/error-reference#7003)
-
-    def initialize(*_)
-      super
-
-      @buffer_accepted = 0
-      @buffer_accepted_lengths = 0
-      @buffer_dropped = 0
-      @buffer_spans = 0
-    end
-
-    def add!(trace)
-      super
-
-      # Emit health metrics
-      measure_accept(trace)
-    end
-
-    def add_all!(traces)
-      super
-
-      # Emit health metrics
-      traces.each { |trace| measure_accept(trace) }
-    end
-
-    def replace!(trace)
-      discarded_trace = super
-
-      # Emit health metrics
-      measure_accept(trace)
-      measure_drop(discarded_trace) if discarded_trace
-
-      discarded_trace
-    end
-
-    # Stored traces are returned and the local buffer is reset.
-    def drain!
-      traces = super
-      measure_pop(traces)
-      traces
-    end
-
-    def measure_accept(trace)
-      @buffer_accepted += 1
-      @buffer_accepted_lengths += trace.length
-
-      @buffer_spans += trace.length
-    rescue StandardError => e
-      Datadog.logger.debug("Failed to measure queue accept. Cause: #{e.message} Source: #{Array(e.backtrace).first}")
-    end
-
-    def measure_drop(trace)
-      @buffer_dropped += 1
-
-      @buffer_spans -= trace.length
-    rescue StandardError => e
-      Datadog.logger.debug("Failed to measure queue drop. Cause: #{e.message} Source: #{Array(e.backtrace).first}")
-    end
-
-    def measure_pop(traces)
-      # Accepted, cumulative totals
-      Datadog.health_metrics.queue_accepted(@buffer_accepted)
-      Datadog.health_metrics.queue_accepted_lengths(@buffer_accepted_lengths)
-
-      # Dropped, cumulative totals
-      Datadog.health_metrics.queue_dropped(@buffer_dropped)
-      # TODO: are we missing a +queue_dropped_lengths+ metric?
-
-      # Queue gauges, current values
-      Datadog.health_metrics.queue_max_length(@max_size)
-      Datadog.health_metrics.queue_spans(@buffer_spans)
-      Datadog.health_metrics.queue_length(traces.length)
-
-      # Reset aggregated metrics
-      @buffer_accepted = 0
-      @buffer_accepted_lengths = 0
-      @buffer_dropped = 0
-      @buffer_spans = 0
-    rescue StandardError => e
-      Datadog.logger.debug("Failed to measure queue. Cause: #{e.message} Source: #{Array(e.backtrace).first}")
-    end
-  end
-
-  # Trace buffer that stores application traces, has a maximum size, and
-  # can be safely used concurrently on any environment.
-  #
-  # @see {Datadog::ThreadSafeBuffer}
-  class ThreadSafeTraceBuffer < ThreadSafeBuffer
-    prepend MeasuredBuffer
-  end
-
-  # Trace buffer that stores application traces, has a maximum size, and
-  # can be safely used concurrently with CRuby.
-  #
-  # @see {Datadog::CRubyBuffer}
-  class CRubyTraceBuffer < CRubyBuffer
-    prepend MeasuredBuffer
-  end
-
-  # Trace buffer that stores application traces. The buffer has a maximum size and when
-  # the buffer is full, a random trace is discarded. This class is thread-safe and is used
-  # automatically by the ``Tracer`` instance when a ``Span`` is finished.
-  #
-  # We choose the default TraceBuffer implementation for current platform dynamically here.
-  #
-  # TODO We should restructure this module, so that classes are not declared at top-level ::Datadog.
-  # TODO Making such a change is potentially breaking for users manually configuring the tracer.
-  TraceBuffer = if Datadog::Core::Environment::Ext::RUBY_ENGINE == 'ruby'
-                  CRubyTraceBuffer
-                else
-                  ThreadSafeTraceBuffer
-                end
-end

data/lib/ddtrace/chunker.rb

@@ -1,35 +0,0 @@
-# typed: false
-require 'json'
-require 'msgpack'
-
-module Datadog
-  # Chunks list of elements into batches
-  module Chunker
-    module_function
-
-    # Chunks a list into batches of at most +max_chunk_size+ elements each.
-    #
-    # An exception can occur if a single element is too large. That single
-    # element will be returned in its own chunk. You have to verify by yourself
-    # when such elements are returned.
-    #
-    # @param list [Enumerable] list of elements
-    # @param max_chunk_size [Numeric] maximum acceptable chunk size
-    # @return [Enumerable] lazy list of chunks
-    def chunk_by_size(list, max_chunk_size)
-      chunk_agg = 0
-      list.slice_before do |elem|
-        size = elem.size
-        chunk_agg += size
-        if chunk_agg > max_chunk_size
-          # Can't fit element in current chunk, start a new one.
-          chunk_agg = size
-          true
-        else
-          # Add to current chunk
-          false
-        end
-      end
-    end
-  end
-end