ddtrace 0.43.0 → 0.48.0

data/docs/ProfilingDevelopment.md

@@ -0,0 +1,88 @@
+# Profiling Development
+
+This file contains development notes specific to the profiling feature.
+
+For a more practical view of getting started with development of `ddtrace`, see <DevelopmentGuide.md>.
+
+## Profiling components high-level view
+
+Components below live inside <../lib/ddtrace/profiling>:
+
+* `Collectors::Stack`: Collects stack trace samples from Ruby threads for both CPU-time (if available) and wall-clock.
+  Runs on its own background thread.
+* `Encoding::Profile`: Encodes gathered data into the pprof format.
+* `Events::Stack`, `Events::StackSample`, `Events::StackExceptionSample`: Entity classes used to represent stacks.
+* `Ext::CPU`: Monkey patches Ruby's `Thread` with our `Ext::CThread` to enable CPU-time profiling.
+* `Ext::CThread`: Extension used to enable CPU-time profiling via use of Pthread's `getcpuclockid`.
+* `Ext::Forking`: Monkey patches `Kernel#fork`, adding a `Kernel#at_fork` callback mechanism which is used to restore
+  profiling abilities after the VM forks (such as re-instrumenting the main thread, and restarting profiler threads).
+* `Pprof::*` (in <../lib/ddtrace/profiling/pprof>): Converts samples captured in the `Recorder` into the pprof format.
+* `Tasks::Setup`: Takes care of loading our extensions/monkey patches to handle fork() and CPU profiling.
+* `Transport::*` (in <../lib/ddtrace/profiling/transport>): Implements transmission of profiling payloads to the Datadog agent
+  or backend.
+* `BacktraceLocation`: Entity class used to represent an entry in a stack trace.
+* `Buffer`: Bounded buffer used to store profiling events.
+* `Exporter`: Writes profiling data to a given transport.
+* `Flush`: Entity class used to represent metadata for a given profile.
+* `Profiler`: Profiling entry point, which coordinates collectors and a scheduler.
+* `Recorder`: Stores profiling events gathered by `Collector`s.
+* `Scheduler`: Periodically (every 1 minute) takes data from the `Recorder` and pushes them to all configured
+  `Exporter`s. Runs on its own background thread.
+
+## Initialization
+
+When started via `ddtracerb exec` (together with `DD_PROFILING_ENABLED=true`), initialization goes through the following
+flow:
+
+1. <../lib/ddtrace/profiling/preload.rb> triggers the creation of the `Datadog.profiler` instance by calling the method
+2. `Datadog.profiler` is handled by `Datadog::Configuration`, which triggers the configuration of `ddtrace` components
+   in `#build_components`
+3. Inside `Datadog::Components`, the `build_profiler` method triggers the execution of the `Tasks::Setup`
+4. The `Setup` task activates our extensions
+    * `Datadog::Profiling::Ext::Forking`
+    * `Datadog::Profiling::Ext::CPU`
+5. Still inside `Datadog::Components`, the `build_profiler` method then creates and wires up the Profiler:
+    ```ruby
+          recorder = build_profiler_recorder(settings)
+          collectors = build_profiler_collectors(settings, recorder)
+          exporters = build_profiler_exporters(settings)
+          scheduler = build_profiler_scheduler(settings, recorder, exporters)
+
+          Datadog::Profiler.new(collectors, scheduler)
+    ```
+    ```asciiflow
+            +------------+
+            |  Profiler  |
+            +-+--------+-+
+              |        |
+              v        v
+    +---------+--+  +--+--------+
+    | Collectors |  | Scheduler |
+    +---------+--+  +-+-------+-+
+              |       |       |
+              v       |       v
+        +-----+-+     |  +----+------+
+        | Stack |     |  | Exporters |
+        +-----+-+     |  +-----------+
+              |       |
+              v       v
+            +-+-------+-+
+            | Recorder  |
+            +-----------+
+    ```
+6. The profiler gets started when `startup!` is called by `Datadog::Configuration` after component creation.
+
+## Run-time execution
+
+During run-time, the `Scheduler` and the `Collectors::Stack` each execute on their own background thread.
+
+The `Collectors::Stack` samples stack traces of threads, capturing both CPU-time (if available) and wall-clock, storing
+them in the `Recorder`.
+
+The `Scheduler` wakes up every 1 minute to flush the results of the `Recorder` into one or more `exporter`s.
+Usually only one exporter is in use. By default, the `Exporter` delegates to the default `Transport::HTTP` transport, which
+takes care of encoding the data and reporting it to the datadog agent (or to the API, when running without an agent).
+
+## How CPU-time profiling works
+
+**TODO**: Document our pthread-based approach to getting CPU-time for threads.

data/integration/README.md

@@ -0,0 +1,67 @@
+# Datadog APM Ruby integration test suite
+
+Integration tests for `ddtrace` that use a variety of real applications.
+
+## Quickstart
+
+1. Build Docker base images:
+
+    ```bash
+    ./script/build-images
+    ```
+
+2. Choose an application and follow instructions (in corresponding `README.md`.)
+
+## Demo applications
+
+Ruby demo applications are configured with Datadog APM, which can be used to generate sample traces/profiles. These are used to drive tests in the integration suite.
+
+### Applications
+
+See `README.md` in each directory for more information:
+
+- `apps/rack`: Rack application
+- `apps/rails-five`: Rails 5 application
+
+### Base images
+
+The `images/` folders hosts some images for Ruby applications.
+
+- `datadog/dd-apm-demo:wrk` / `images/wrk/Dockerfile`: `wrk` load testing application (for generating load)
+- `datadog/dd-apm-demo:agent` / `images/agent/Dockerfile`: Datadog agent (with default configuration)
+- `datadog/dd-apm-demo:rb-<RUBY_VERSION>` / `images/<RUBY_VERSION>/Dockerfile`: MRI Ruby & `Datadog::DemoEnv` (where `<RUBY_VERSION>` is minor version e.g. `2.7`)
+
+Ruby base images include `Datadog::DemoEnv` and other helpers.
+
+### Debugging
+
+#### Profiling memory
+
+Create a memory heap dump via:
+
+```sh
+# Profile for 5 minutes, dump heap to /data/app/ruby-heap.dump
+# Where PID = process ID
+bundle exec rbtrace -p PID -e 'Thread.new{GC.start; require "objspace"; ObjectSpace.trace_object_allocations_start; sleep(300); io=File.open("/data/app/ruby-heap.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'
+```
+
+Then analyze it using `analyzer.rb` (built into the Ruby base images) with:
+
+```sh
+# Group objects by generation
+ruby /vendor/dd-demo/datadog/analyzer.rb /data/app/ruby-heap.dump
+
+# List objects in GEN_NUM, group by source location
+ruby /vendor/dd-demo/datadog/analyzer.rb /data/app/ruby-heap.dump GEN_NUM
+
+# List objects in all generations, group by source location, descending.
+ruby /vendor/dd-demo/datadog/analyzer.rb /data/app/ruby-heap.dump objects
+
+# List objects in all generations, group by source location, descending.
+# Up to generation END_GEN.
+ruby /vendor/dd-demo/datadog/analyzer.rb /data/app/ruby-heap.dump objects END_GEN
+
+# List objects in all generations, group by source location, descending.
+# Between generations START_GEN to END_GEN inclusive.
+ruby /vendor/dd-demo/datadog/analyzer.rb /data/app/ruby-heap.dump objects END_GEN START_GEN
+```

data/integration/apps/rack/.dockerignore

@@ -0,0 +1 @@
+Gemfile.lock

data/integration/apps/rack/.envrc.sample

@@ -0,0 +1 @@
+export DD_API_KEY=<Your Datadog API key here>

data/integration/apps/rack/.gitignore

@@ -0,0 +1,4 @@
+# Ignore local variables
+.envrc
+.byebug_history
+data

data/integration/apps/rack/.rspec

@@ -0,0 +1 @@
+--require spec_helper

data/integration/apps/rack/Dockerfile

@@ -0,0 +1,28 @@
+# Select base image
+ARG BASE_IMAGE
+FROM ${BASE_IMAGE}
+
+# Setup directory
+RUN mkdir /app
+WORKDIR /app
+
+# Setup specific version of ddtrace, if specified.
+ARG ddtrace_git
+ENV DD_DEMO_ENV_GEM_GIT_DDTRACE ${ddtrace_git}
+
+ARG ddtrace_ref
+ENV DD_DEMO_ENV_GEM_REF_DDTRACE ${ddtrace_ref}
+
+# Install dependencies
+COPY Gemfile /app/Gemfile
+# This forces gems with native extensions to be compiled, rather than using pre-compiled binaries; it's needed because
+# some google-protobuf versions ship with missing binaries for older rubies.
+ENV BUNDLE_FORCE_RUBY_PLATFORM true
+RUN bundle install
+
+# Add files
+COPY . /app
+
+# Set entrypoint
+ENTRYPOINT ["/bin/bash", "-c"]
+CMD ["bin/setup && bin/run"]

data/integration/apps/rack/Dockerfile-ci

@@ -0,0 +1,11 @@
+# Select base image
+ARG BASE_IMAGE
+FROM ${BASE_IMAGE}
+
+# Add gem
+COPY . /vendor/dd-trace-rb
+
+# Install dependencies
+# Setup specific version of ddtrace, if specified.
+ENV DD_DEMO_ENV_GEM_LOCAL_DDTRACE /vendor/dd-trace-rb
+RUN bundle install

data/integration/apps/rack/Gemfile

@@ -0,0 +1,24 @@
+require 'datadog/demo_env'
+
+source 'https://rubygems.org' do
+  gem 'puma'
+  gem 'unicorn'
+
+  gem 'dogstatsd-ruby'
+  gem 'rack'
+
+  gem 'ffi'
+  gem 'google-protobuf'
+
+  # Choose correct specs for 'ddtrace' demo environment
+  gem 'ddtrace', *Datadog::DemoEnv.gem_spec('ddtrace')
+
+  # Development
+  gem 'byebug'
+  # gem 'pry-stack_explorer', platform: :ruby
+  # gem 'rbtrace'
+  # gem 'ruby-prof'
+
+  # Testing/CI
+  gem 'rspec'
+end

data/integration/apps/rack/README.md

@@ -0,0 +1,93 @@
+# Rack: Demo application for Datadog APM
+
+A generic Rack web application with some common use scenarios.
+
+For generating Datadog APM traces and profiles.
+
+## Installation
+
+Install [direnv](https://github.com/direnv/direnv) for applying local settings.
+
+1. `cp .envrc.sample .envrc` and add your Datadog API key.
+2. `direnv allow` to load the env var.
+4. `docker-compose run --rm app bin/setup`
+
+## Running the application
+
+### To monitor performance of Docker containers with Datadog
+
+```sh
+docker run --rm --name dd-agent  -v /var/run/docker.sock:/var/run/docker.sock:ro -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e API_KEY=$DD_API_KEY datadog/docker-dd-agent:latest
+```
+
+### Starting the web server
+
+Run `docker-compose up` to auto-start the webserver. It should bind to `localhost:80`.
+
+Alternatively, you can run it manually with:
+
+```sh
+docker-compose run --rm -p 80:80 app bin/run <process>
+```
+
+The `<process>` argument is optional, and will default to `DD_DEMO_ENV_PROCESS` if not provided. See [Processes](#processes) for more details.
+
+##### Processes
+
+Within the container, run `bin/run <process>` where `<process>` is one of the following values:
+
+ - `puma`: Puma web server
+ - `unicorn`: Unicorn web server
+ - `irb`: IRB session
+
+ Alternatively, set `DD_DEMO_ENV_PROCESS` to run a particular process by default when `bin/run` is run.
+
+##### Features
+
+Set `DD_DEMO_ENV_PROCESS` to a comma-delimited list of any of the following values to activate the feature:
+
+ - `tracing`: Tracing instrumentation
+ - `profiling`: Profiling (NOTE: Must also set `DD_PROFILING_ENABLED` to match.)
+ - `debug`: Enable diagnostic debug mode
+ - `analytics`: Enable trace analytics
+ - `runtime_metrics`: Enable runtime metrics
+ - `pprof_to_file`: Dump profiling pprof to file instead of agent.
+
+e.g. `DD_DEMO_ENV_PROCESS=tracing,profiling`
+
+##### Routes
+
+```sh
+# Health check
+curl -v localhost/health
+
+# Basic demo scenarios
+curl -v localhost/basic/fibonacci
+curl -v -XPOST localhost/basic/default
+
+# Job demo scenarios
+curl -v -XPOST localhost/jobs
+```
+
+### Load tester
+
+Docker configuration automatically creates and runs [Wrk](https://github.com/wg/wrk) load testing containers. By default it runs the `basic/default` scenario described in the `wrk` image to give a baseload.
+
+You can modify the `load-tester` container in `docker-compose.yml` to change the load type or scenario run. Set the container's `command` to any set of arguments `wrk` accepts.
+
+You can also define your own custom scenario by creating a LUA file, mounting it into the container, and passing it as an argument via `command`.
+
+### Running integration tests
+
+You can run integration tests using the following and substituting for the Ruby major and minor version (e.g. `2.7`)
+
+```sh
+./script/build-images -v <RUBY_VERSION>
+./script/ci -v <RUBY_VERSION>
+```
+
+Or inside a running container:
+
+```sh
+./bin/test
+```

data/integration/apps/rack/app/acme.rb

@@ -0,0 +1,80 @@
+require 'rack'
+
+module Acme
+  class Application
+    def call(env)
+      request = Rack::Request.new(env)
+      router.route!(request)
+    end
+
+    def router
+      Router.new(
+        '/' => { controller: controllers[:health], action: :check },
+        '/health' => { controller: controllers[:health], action: :check },
+        '/basic/fibonacci' => { controller: controllers[:basic], action: :fibonacci },
+        '/basic/default' => { controller: controllers[:basic], action: :default }
+      )
+    end
+
+    def controllers
+      {
+        basic: Controllers::Basic.new,
+        health: Controllers::Health.new
+      }
+    end
+  end
+
+  class Router
+    attr_reader :routes
+
+    def initialize(routes)
+      @routes = routes
+    end
+
+    def route!(request)
+      begin
+        if route = routes[request.path]
+          route[:controller].send(route[:action], request)
+        else
+          not_found(request)
+        end
+      rescue StandardError => e
+        application_error(request, e)
+      end
+    end
+
+    def not_found(request)
+      [404, { 'Content-Type' => 'text/plain' }, ["404 Not Found: #{request.path}"]]
+    end
+
+    def application_error(request, error)
+      [500, { 'Content-Type' => 'text/plain' }, ["500 Application Error: #{error.message} Location: #{error.backtrace.first(3)}"]]
+    end
+  end
+
+  module Controllers
+    class Basic
+      def fibonacci(request)
+        n = rand(25..35)
+        result = fib(n)
+        ['200', { 'Content-Type' => 'text/plain' }, ["Basic: Fibonacci(#{n}): #{result}"]]
+      end
+
+      def default(request)
+        ['200', { 'Content-Type' => 'text/plain' }, ['Basic: Default']]
+      end
+
+      private
+
+      def fib(n)
+        n <= 1 ? n : fib(n-1) + fib(n-2)
+      end
+    end
+
+    class Health
+      def check(request)
+        ['204', {}, []]
+      end
+    end
+  end
+end

data/integration/apps/rack/app/datadog.rb

@@ -0,0 +1,17 @@
+require 'datadog/demo_env'
+require 'ddtrace'
+
+Datadog.configure do |c|
+  c.diagnostics.debug = true if Datadog::DemoEnv.feature?('debug')
+  c.analytics_enabled = true if Datadog::DemoEnv.feature?('analytics')
+  c.runtime_metrics.enabled = true if Datadog::DemoEnv.feature?('runtime_metrics')
+
+  if Datadog::DemoEnv.feature?('tracing')
+    c.use :rack, service_name: 'acme-rack'
+  end
+
+  if Datadog::DemoEnv.feature?('pprof_to_file')
+    # Reconfigure transport to write pprof to file
+    c.profiling.exporter.transport = Datadog::DemoEnv.profiler_file_transport
+  end
+end

data/integration/apps/rack/bin/run

@@ -0,0 +1,22 @@
+#!/usr/bin/env ruby
+
+# Start application process
+puts "\n== Starting application process =="
+
+profiling = Datadog::DemoEnv.feature?('profiling') ? 'DD_PROFILING_ENABLED=true ddtracerb exec ' : ''
+process = (ARGV[0] || Datadog::DemoEnv.process)
+command = case process
+          when 'puma'
+            "bundle exec #{profiling}puma -C /app/config/puma.rb /app/config.ru"
+          when 'unicorn'
+            "bundle exec #{profiling}unicorn -c /app/config/unicorn.rb /app/config.ru"
+          when 'irb'
+            "bundle exec #{profiling}irb"
+          when nil, ''
+            abort("\n== ERROR: Must specify a application process! ==")
+          else
+            abort("\n== ERROR: Unknown application process '#{process}' ==")
+          end
+
+puts "Run: #{command}"
+Kernel.exec(command)