ddtrace 0.45.0 → 0.47.0

data/docs/DevelopmentGuide.md

@@ -114,6 +114,34 @@ Because you are likely not running all tests locally, your report will contain p
 You *must* check the CI step `coverage` for the complete test coverage report, ensuring coverage is not
 decreased.
 
+**Ensuring tests don't leak resources**
+
+Tests execution can create resources that are hard to track: threads, sockets, files, etc. Because these resources can come
+from the both the test setup as well as the code under test, making sure all resources are properly disposed is important
+to prevent the application from inadvertently creating cumulative resources during its execution.
+
+When running tests that utilize threads, you might see an error message similar to this one:
+
+```
+Test leaked 1 thread: "Datadog::Workers::AsyncTransport integration tests"
+Ensure all threads are terminated when test finishes:
+1: #<Thread:0x00007fcbc99863d0 /Users/marco.costa/work/dd-trace-rb/spec/spec_helper.rb:145 sleep> (Thread)
+Thread Creation Site:
+        ./dd-trace-rb/spec/ddtrace/workers_integration_spec.rb:245:in 'new'
+        ./dd-trace-rb/spec/ddtrace/workers_integration_spec.rb:245:in 'block (4 levels) in <top (required)>'
+Thread Backtrace:
+        ./dd-trace-rb/spec/ddtrace/workers_integration_spec.rb:262:in 'sleep'
+        .dd-trace-rb/spec/ddtrace/workers_integration_spec.rb:262:in 'block (5 levels) in <top (required)>'
+        ./dd-trace-rb/spec/spec_helper.rb:147:in 'block in initialize'
+```
+
+This means that this test did not finish all threads by the time the test had finished. In this case, the thread
+creation can be traced to `workers_integration_spec.rb:245:in 'new'`. The thread itself is sleeping at `workers_integration_spec.rb:262:in 'sleep'`.
+
+The actionable in this case would be to ensure that the thread created in `workers_integration_spec.rb:245` is properly terminated by invoking `Thread#join` during the test tear down, which will wait for the thread to finish before returning.
+
+Depending on the situation, the thread in question might need to be forced to terminate. It's recommended to have a mechanism in place to terminate it (a shared variable that changes value when the thread should exit), but as a last resort, `Thread#terminate` forces the thread to finish. Keep in mind that regardless of the termination method, `Thread#join` must be called to ensure that the thread has completely finished its shutdown process.
+
 ### Checking code quality
 
 **Linting**

data/docs/GettingStarted.md

@@ -46,6 +46,7 @@ To contribute, check out the [contribution guidelines][contribution docs] and [d
      - [gRPC](#grpc)
      - [http.rb](#http-rb)
      - [httpclient](#httpclient)
+     - [httpx](#httpx)
      - [MongoDB](#mongodb)
      - [MySQL2](#mysql2)
      - [Net/HTTP](#net-http)
@@ -145,7 +146,7 @@ Install and configure the Datadog Agent to receive traces from your now instrume
 
 ### Quickstart for Rails applications
 
-#### Rails Auto Instrument all Integrations
+#### Automatic instrumentation
 
 1. Add the `ddtrace` gem to your Gemfile:
 
@@ -158,7 +159,7 @@ Install and configure the Datadog Agent to receive traces from your now instrume
 
 3. You can configure, override, or disable any specific integration settings by also adding a [Rails Manual Configuration](#rails-manual-configuration) file.
 
-#### Rails Manual Configuration
+#### Manual instrumentation
 
 1. Add the `ddtrace` gem to your Gemfile:
 
@@ -181,10 +182,10 @@ Install and configure the Datadog Agent to receive traces from your now instrume
 
 ### Quickstart for Ruby applications
 
-#### Ruby Auto Instrument all Integrations
+#### Automatic instrumentation
 
 1. Install the gem with `gem install ddtrace`
-2. Requiring any [supported libraries or frameworks](#integration-instrumentation) that should be instrumented. 
+2. Requiring any [supported libraries or frameworks](#integration-instrumentation) that should be instrumented.
 3. Add `require 'ddtrace/auto_instrument'` to your application. _Note:_ This must be done _after_ requiring any supported libraries or frameworks.
 
     ```ruby
@@ -195,10 +196,10 @@ Install and configure the Datadog Agent to receive traces from your now instrume
 
     require 'ddtrace/auto_instrument'
     ```
- 
+
     You can configure, override, or disable any specific integration settings by also adding a [Ruby Manual Configuration Block](#ruby-manual-configuration).
 
-#### Ruby Manual Configuration
+#### Manual instrumentation
 
 1. Install the gem with `gem install ddtrace`
 2. Add a configuration block to your Ruby application:
@@ -398,6 +399,7 @@ For a list of available integrations, and their configuration options, please re
 | gRPC                     | `grpc`                     | `>= 1.7`                 | *gem not available*       | *[Link](#grpc)*                     | *[Link](https://github.com/grpc/grpc/tree/master/src/rubyc)*                   |
 | http.rb                  | `httprb`                   | `>= 2.0`                 | `>= 2.0`                  | *[Link](#http-rb)*                  | *[Link](https://github.com/httprb/http)*                                       |
 | httpclient                | `httpclient`              | `>= 2.2`                 | `>= 2.2`                  | *[Link](#httpclient)*               | *[Link](https://github.com/nahi/httpclient)*                                     |
+| httpx                     | `httpx`                   | `>= 0.11`                | `>= 0.11`                 | *[Link](#httpx)*                    | *[Link](https://gitlab.com/honeyryderchuck/httpx)*                             |
 | Kafka                    | `ruby-kafka`               | `>= 0.7.10`              | `>= 0.7.10`               | *[Link](#kafka)*                    | *[Link](https://github.com/zendesk/ruby-kafka)*                                |
 | MongoDB                  | `mongo`                    | `>= 2.1`                 | `>= 2.1`                  | *[Link](#mongodb)*                  | *[Link](https://github.com/mongodb/mongo-ruby-driver)*                         |
 | MySQL2                   | `mysql2`                   | `>= 0.3.21`              | *gem not available*       | *[Link](#mysql2)*                   | *[Link](https://github.com/brianmario/mysql2)*                                 |
@@ -546,17 +548,20 @@ Datadog.configure do |c|
   # Symbol matching your database connection in config/database.yml
   # Only available if you are using Rails with ActiveRecord.
   c.use :active_record, describes: :secondary_database, service_name: 'secondary-db'
-
+  
+  # Block configuration pattern.
   c.use :active_record, describes: :secondary_database do |second_db|
     second_db.service_name = 'secondary-db'
   end
 
   # Connection string with the following connection settings:
-  # Adapter, user, host, port, database
+  # adapter, username, host, port, database
+  # Other fields are ignored.
   c.use :active_record, describes: 'mysql2://root@127.0.0.1:3306/mysql', service_name: 'secondary-db'
 
-  # Hash with following connection settings
-  # Adapter, user, host, port, database
+  # Hash with following connection settings:
+  # adapter, username, host, port, database
+  # Other fields are ignored.
   c.use :active_record, describes: {
       adapter:  'mysql2',
       host:     '127.0.0.1',
@@ -568,6 +573,27 @@ Datadog.configure do |c|
 end
 ```
 
+You can also create configurations based on partial matching of database connection fields:
+
+```ruby
+Datadog.configure do |c|
+  # Matches any connection on host `127.0.0.1`.
+  c.use :active_record, describes: { host:  '127.0.0.1' }, service_name: 'local-db'
+
+  # Matches any `mysql2` connection.
+  c.use :active_record, describes: { adapter: 'mysql2'}, service_name: 'mysql-db'
+  
+  # Matches any `mysql2` connection to the `reports` database.
+  #
+  # In case of multiple matching `describe` configurations, the latest one applies.
+  # In this case a connection with both adapter `mysql` and database `reports`
+  # will be configured `service_name: 'reports-db'`, not `service_name: 'mysql-db'`.
+  c.use :active_record, describes: { adapter: 'mysql2', database:  'reports'}, service_name: 'reports-db'
+end
+```
+
+When multiple `describes` configurations match a connection, the latest configured rule that matches will be applied.
+
 If ActiveRecord traces an event that uses a connection that matches a key defined by `describes`, it will use the trace settings assigned to that connection. If the connection does not match any of the described connections, it will use default settings defined by `c.use :active_record` instead.
 
 ### Active Support
@@ -673,7 +699,6 @@ Where `options` is an optional `Hash` that accepts the following parameters:
 
 | Key | Description | Default |
 | --- | ----------- | ------- |
-| `analytics_enabled` | Enable analytics for spans produced by this integration. `true` for on, `nil` to defer to global setting, `false` for off. | `true` |
 | `enabled` | Defines whether Cucumber tests should be traced. Useful for temporarily disabling tracing. `true` or `false` | `true` |
 | `service_name` | Service name used for `cucumber` instrumentation. | `'cucumber'` |
 | `operation_name` | Operation name used for `cucumber` instrumentation. Useful if you want rename automatic trace metrics e.g. `trace.#{operation_name}.errors`. | `'cucumber.test'` |
@@ -1058,7 +1083,7 @@ Where `options` is an optional `Hash` that accepts the following parameters:
 The httpclient integration will trace any HTTP call using the httpclient gem.
 
 ```ruby
-require 'http'
+require 'httpclient'
 require 'ddtrace'
 Datadog.configure do |c|
   c.use :httpclient, options
@@ -1079,6 +1104,25 @@ Where `options` is an optional `Hash` that accepts the following parameters:
 | `service_name` | Service name for `httpclient` instrumentation. | `'httpclient'` |
 | `split_by_domain` | Uses the request domain as the service name when set to `true`. | `false` |
 
+### httpx
+
+`httpx` maintains its [own integration with `ddtrace`](https://honeyryderchuck.gitlab.io/httpx/wiki/Datadog-Adapter):
+
+```ruby
+require "ddtrace"
+require "httpx/adapters/datadog"
+
+Datadog.configure do |c|
+  c.use :httpx
+
+  # optionally, specify a different service name for hostnames matching a regex
+  c.use :httpx, describes: /user-[^.]+\.example\.com/ do |http|
+    http.service_name = 'user.example.com'
+    http.split_by_domain = false # Only necessary if split_by_domain is true by default
+  end
+end
+```
+
 ### Kafka
 
 The Kafka integration provides tracing of the `ruby-kafka` gem:
@@ -1538,12 +1582,17 @@ Datadog.configure do |c|
   # The default configuration for any redis client
   c.use :redis, service_name: 'redis-default'
 
-  # The configuration matching a given unix socket
+  # The configuration matching a given unix socket.
   c.use :redis, describes: { url: 'unix://path/to/file' }, service_name: 'redis-unix'
 
-  # Connection string
-  c.use :redis, describes: { url: 'redis://127.0.0.1:6379/0' }, service_name: 'redis-connection-string'
-  # Client host, port, db, scheme
+  # For network connections, only these fields are considered during matching:
+  # scheme, host, port, db
+  # Other fields are ignored.
+  
+  # Network connection string
+  c.use :redis, describes: 'redis://127.0.0.1:6379/0', service_name: 'redis-connection-string'
+  c.use :redis, describes: { url: 'redis://127.0.0.1:6379/1' }, service_name: 'redis-connection-url'
+  # Network client hash
   c.use :redis, describes: { host: 'my-host.com', port: 6379, db: 1, scheme: 'redis' }, service_name: 'redis-connection-hash'
   # Only a subset of the connection hash
   c.use :redis, describes: { host: ENV['APP_CACHE_HOST'], port: ENV['APP_CACHE_PORT'] }, service_name: 'redis-cache'
@@ -1551,6 +1600,8 @@ Datadog.configure do |c|
 end
 ```
 
+When multiple `describes` configurations match a connection, the latest configured rule that matches will be applied.
+
 ### Resque
 
 The Resque integration uses Resque hooks that wraps the `perform` method.
@@ -1621,7 +1672,6 @@ Where `options` is an optional `Hash` that accepts the following parameters:
 
 | Key | Description | Default |
 | --- | ----------- | ------- |
-| `analytics_enabled` | Enable analytics for spans produced by this integration. `true` for on, `nil` to defer to global setting, `false` for off. | `true` |
 | `enabled` | Defines whether RSpec tests should be traced. Useful for temporarily disabling tracing. `true` or `false` | `true` |
 | `service_name` | Service name used for `rspec` instrumentation. | `'rspec'` |
 | `operation_name` | Operation name used for `rspec` instrumentation. Useful if you want rename automatic trace metrics e.g. `trace.#{operation_name}.errors`. | `'rspec.example'` |
@@ -1864,6 +1914,7 @@ Available options are:
  - `sampler`: set to a custom `Datadog::Sampler` instance. If provided, the tracer will use this sampler to determine sampling behavior.
  - `diagnostics.startup_logs.enabled`: Startup configuration and diagnostic log. Defaults to `true`. Can be configured through the `DD_TRACE_STARTUP_LOGS` environment variable.
  - `diagnostics.debug`: set to true to enable debug logging. Can be configured through the `DD_TRACE_DEBUG` environment variable. Defaults to `false`.
+ - `time_now_provider`: when testing, it might be helpful to use a different time provider. For Timecop, for example, `->{ Time.now_without_mock_time }` allows the tracer to use the real wall time. Span duration calculation will still use the system monotonic clock when available, thus not being affected by this setting. Defaults to `->{ Time.now }`.
 
 #### Custom logging
 
@@ -2088,6 +2139,7 @@ For more details on how to activate distributed tracing for integrations, see th
 - [Sinatra](#sinatra)
 - [http.rb](#http-rb)
 - [httpclient](#httpclient)
+- [httpx](#httpx)
 
 **Using the HTTP propagator**
 

data/integration/README.md

@@ -0,0 +1,68 @@
+# Datadog APM Ruby integration test suite
+
+Integration tests for `ddtrace` that use a variety of real applications.
+
+## Quickstart
+
+1. Build Docker base images:
+
+    ```sh
+    #!/bin/bash
+    ./script/build-images
+    ```
+
+2. Choose an application and follow instructions (in corresponding `README.md`.)
+
+## Demo applications
+
+Ruby demo applications are configured with Datadog APM, which can be used to generate sample traces/profiles. These are used to drive tests in the integration suite.
+
+### Applications
+
+See `README.md` in each directory for more information:
+
+- `apps/rack`: Rack application
+- `apps/rails-five`: Rails 5 application
+
+### Base images
+
+The `images/` folders hosts some images for Ruby applications.
+
+- `datadog/dd-apm-demo:wrk` / `images/wrk/Dockerfile`: `wrk` load testing application (for generating load)
+- `datadog/dd-apm-demo:agent` / `images/agent/Dockerfile`: Datadog agent (with default configuration)
+- `datadog/dd-apm-demo:rb-<RUBY_VERSION>` / `images/<RUBY_VERSION>/Dockerfile`: MRI Ruby & `Datadog::DemoEnv` (where `<RUBY_VERSION>` is minor version e.g. `2.7`)
+
+Ruby base images include `Datadog::DemoEnv` and other helpers.
+
+### Debugging
+
+#### Profiling memory
+
+Create a memory heap dump via:
+
+```sh
+# Profile for 5 minutes, dump heap to /data/app/ruby-heap.dump
+# Where PID = process ID
+bundle exec rbtrace -p PID -e 'Thread.new{GC.start; require "objspace"; ObjectSpace.trace_object_allocations_start; sleep(300); io=File.open("/data/app/ruby-heap.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'
+```
+
+Then analyze it using `analyzer.rb` (built into the Ruby base images) with:
+
+```sh
+# Group objects by generation
+ruby /vendor/dd-demo/datadog/analyzer.rb /data/app/ruby-heap.dump
+
+# List objects in GEN_NUM, group by source location
+ruby /vendor/dd-demo/datadog/analyzer.rb /data/app/ruby-heap.dump GEN_NUM
+
+# List objects in all generations, group by source location, descending.
+ruby /vendor/dd-demo/datadog/analyzer.rb /data/app/ruby-heap.dump objects
+
+# List objects in all generations, group by source location, descending.
+# Up to generation END_GEN.
+ruby /vendor/dd-demo/datadog/analyzer.rb /data/app/ruby-heap.dump objects END_GEN
+
+# List objects in all generations, group by source location, descending.
+# Between generations START_GEN to END_GEN inclusive.
+ruby /vendor/dd-demo/datadog/analyzer.rb /data/app/ruby-heap.dump objects END_GEN START_GEN
+```

data/integration/apps/rack/.dockerignore

@@ -0,0 +1 @@
+Gemfile.lock

data/integration/apps/rack/.envrc.sample

@@ -0,0 +1 @@
+export DD_API_KEY=<Your Datadog API key here>

data/integration/apps/rack/.gitignore

@@ -0,0 +1,4 @@
+# Ignore local variables
+.envrc
+.byebug_history
+data

data/integration/apps/rack/.rspec

@@ -0,0 +1 @@
+--require spec_helper

data/integration/apps/rack/Dockerfile

@@ -0,0 +1,25 @@
+# Select base image
+ARG BASE_IMAGE
+FROM ${BASE_IMAGE}
+
+# Setup directory
+RUN mkdir /app
+WORKDIR /app
+
+# Setup specific version of ddtrace, if specified.
+ARG ddtrace_git
+ENV DD_DEMO_ENV_GEM_GIT_DDTRACE ${ddtrace_git}
+
+ARG ddtrace_ref
+ENV DD_DEMO_ENV_GEM_REF_DDTRACE ${ddtrace_ref}
+
+# Install dependencies
+COPY Gemfile /app/Gemfile
+RUN bundle install
+
+# Add files
+COPY . /app
+
+# Set entrypoint
+ENTRYPOINT ["/bin/bash", "-c"]
+CMD ["bin/setup && bin/run"]

data/integration/apps/rack/Dockerfile-ci

@@ -0,0 +1,11 @@
+# Select base image
+ARG BASE_IMAGE
+FROM ${BASE_IMAGE}
+
+# Add gem
+COPY . /vendor/dd-trace-rb
+
+# Install dependencies
+# Setup specific version of ddtrace, if specified.
+ENV DD_DEMO_ENV_GEM_LOCAL_DDTRACE /vendor/dd-trace-rb
+RUN bundle install

data/integration/apps/rack/Gemfile

@@ -0,0 +1,24 @@
+require 'datadog/demo_env'
+
+source 'https://rubygems.org' do
+  gem 'puma'
+  gem 'unicorn'
+
+  gem 'dogstatsd-ruby'
+  gem 'rack'
+
+  gem 'ffi'
+  gem 'google-protobuf'
+
+  # Choose correct specs for 'ddtrace' demo environment
+  gem 'ddtrace', *Datadog::DemoEnv.gem_spec('ddtrace')
+
+  # Development
+  gem 'byebug'
+  # gem 'pry-stack_explorer', platform: :ruby
+  # gem 'rbtrace'
+  # gem 'ruby-prof'
+
+  # Testing/CI
+  gem 'rspec'
+end

data/integration/apps/rack/README.md

@@ -0,0 +1,93 @@
+# Rack: Demo application for Datadog APM
+
+A generic Rack web application with some common use scenarios.
+
+For generating Datadog APM traces and profiles.
+
+## Installation
+
+Install [direnv](https://github.com/direnv/direnv) for applying local settings.
+
+1. `cp .envrc.sample .envrc` and add your Datadog API key.
+2. `direnv allow` to load the env var.
+4. `docker-compose run --rm app bin/setup`
+
+## Running the application
+
+### To monitor performance of Docker containers with Datadog
+
+```sh
+docker run --rm --name dd-agent  -v /var/run/docker.sock:/var/run/docker.sock:ro -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e API_KEY=$DD_API_KEY datadog/docker-dd-agent:latest
+```
+
+### Starting the web server
+
+Run `docker-compose up` to auto-start the webserver. It should bind to `localhost:80`.
+
+Alternatively, you can run it manually with:
+
+```sh
+docker-compose run --rm -p 80:80 app bin/run <process>
+```
+
+The `<process>` argument is optional, and will default to `DD_DEMO_ENV_PROCESS` if not provided. See [Processes](#processes) for more details.
+
+##### Processes
+
+Within the container, run `bin/run <process>` where `<process>` is one of the following values:
+
+ - `puma`: Puma web server
+ - `unicorn`: Unicorn web server
+ - `irb`: IRB session
+
+ Alternatively, set `DD_DEMO_ENV_PROCESS` to run a particular process by default when `bin/run` is run.
+
+##### Features
+
+Set `DD_DEMO_ENV_PROCESS` to a comma-delimited list of any of the following values to activate the feature:
+
+ - `tracing`: Tracing instrumentation
+ - `profiling`: Profiling (NOTE: Must also set `DD_PROFILING_ENABLED` to match.)
+ - `debug`: Enable diagnostic debug mode
+ - `analytics`: Enable trace analytics
+ - `runtime_metrics`: Enable runtime metrics
+ - `pprof_to_file`: Dump profiling pprof to file instead of agent.
+
+e.g. `DD_DEMO_ENV_PROCESS=tracing,profiling`
+
+##### Routes
+
+```sh
+# Health check
+curl -v localhost/health
+
+# Basic demo scenarios
+curl -v localhost/basic/fibonacci
+curl -v -XPOST localhost/basic/default
+
+# Job demo scenarios
+curl -v -XPOST localhost/jobs
+```
+
+### Load tester
+
+Docker configuration automatically creates and runs [Wrk](https://github.com/wg/wrk) load testing containers. By default it runs the `basic/default` scenario described in the `wrk` image to give a baseload.
+
+You can modify the `load-tester` container in `docker-compose.yml` to change the load type or scenario run. Set the container's `command` to any set of arguments `wrk` accepts.
+
+You can also define your own custom scenario by creating a LUA file, mounting it into the container, and passing it as an argument via `command`.
+
+### Running integration tests
+
+You can run integration tests using the following and substituting for the Ruby major and minor version (e.g. `2.7`)
+
+```sh
+./script/build-images -v <RUBY_VERSION>
+./script/ci -v <RUBY_VERSION>
+```
+
+Or inside a running container:
+
+```sh
+./bin/test
+```