elastic-apm 3.6.0 → 3.7.0

data/docs/getting-started-rails.asciidoc

@@ -25,4 +25,6 @@ server_url: http://localhost:8200
 secret_token: ''
 ----
 
+Or if you prefer environment variables, skip the file and set `ELASTIC_APM_SERVER_URL` and `ELASTIC_APM_SECRET_TOKEN` in your local or server environment.
+
 This automatically sets up error logging and performance tracking but of course there are knobs to turn if you'd like to. See <<configuration>>.

data/docs/images/dynamic-config.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="59" height="20"><linearGradient id="b" x2="0" y2="100%"><stop offset="0" stop-color="#bbb" stop-opacity=".1"/><stop offset="1" stop-opacity=".1"/></linearGradient><clipPath id="a"><rect width="59" height="20" rx="3" fill="#fff"/></clipPath><g clip-path="url(#a)"><path fill="#9f9f9f" d="M0 0h0v20H0z"/><path fill="#9f9f9f" d="M0 0h59v20H0z"/><path fill="url(#b)" d="M0 0h59v20H0z"/></g><g fill="#fff" text-anchor="middle" font-family="DejaVu Sans,Verdana,Geneva,sans-serif" font-size="110"> <text x="295" y="150" fill="#010101" fill-opacity=".3" transform="scale(.1)" textLength="490">Dynamic</text><text x="295" y="140" transform="scale(.1)" textLength="490">Dynamic</text></g> </svg>

data/docs/index.asciidoc

@@ -29,9 +29,10 @@ include::./graphql.asciidoc[]
 
 include::./log-correlation.asciidoc[]
 
+include::./performance-tuning.asciidoc[]
+
 include::./debugging.asciidoc[]
 
 include::./upgrading.asciidoc[]
 
 include::./release-notes.asciidoc[]
-

data/docs/introduction.asciidoc

@@ -32,4 +32,5 @@ You can then use the APM app in Kibana to gain insight into latency issues and e
 === Additional Components
 
 APM Agents work in conjunction with the {apm-server-ref-v}/index.html[APM Server], {ref}/index.html[Elasticsearch], and {kibana-ref}/index.html[Kibana].
-Please view the {apm-overview-ref-v}/index.html[APM Overview] for details on how these components work together. 
+The {apm-overview-ref-v}/index.html[APM Overview] provides details on how these components work together,
+and provides a matrix outlining {apm-overview-ref-v}/agent-server-compatibility.html[Agent and Server compatibility].

data/docs/performance-tuning.asciidoc

@@ -0,0 +1,106 @@
+[[tuning-and-overhead]]
+== Performance tuning
+
+Using any APM solution comes with trade-offs, and the Elastic APM Ruby Agent is no different.
+Instrumenting your code, using timers, recording context data, etc., uses resources—for example:
+
+* CPU time
+* Memory
+* Bandwidth
+* Elasticsearch storage
+
+We invest a lot of effort to ensure that the Ruby Agent is suitable for production code
+and that its overhead remains as low as possible.
+However, because every application is different, there are some knobs you can turn and tweak to adapt the Agent to your specific needs.
+
+[float]
+[[tuning-sample-rate]]
+=== Transaction sample rate
+
+By default, the Agent samples every transaction.
+The easiest way to reduce both the overhead of the agent and storage requirements,
+is to tell it to do less, i.e., sample fewer transactions.
+To do this, set the <<config-transaction-sample-rate,`transaction_sample_rate`>>
+to a value between `0.0` and `1.0`—the percentage of transactions you'd like to randomly sample.
+The Agent will still record the overall time and result of unsampled transactions,
+but not context information, tags, or spans.
+
+[float]
+[[tuning-frame-context]]
+=== Collecting frame context
+
+The Agent automatically captures several lines of source code around each frame location in the stack trace.
+This enables the APM app to provide greater insight into exactly where an error or span is occurring in your code.
+This insight does come at a cost—in terms of performance, stack trace collection is the most expensive thing the Agent does.
+
+There are settings you can modify to control this behavior:
+
+1. Disable stack trace frame collection for short-duration spans by setting
+<<config-span-frames-min-duration-ms,`span_frames_min_duration`>> to `0`.
+
+2. Modify the number of source code lines collected.
+These settings are divided between app frames, which represent your application code,
+and library frames, which represent the code of your dependencies.
+Each of these categories are further split into separate error and span settings.
++
+* <<config-source-lines-error-app-frames, `source_lines_error_app_frames`>>
+* <<config-source-lines-error-library-frames,`source_lines_error_library_frames`>>
+* <<config-source-lines-span-app-frames,`source_lines_span_app_frames`>>
+* <<config-source-lines-span-library-frames,`source_lines_span_library_frames`>>
+
+3. If you're using the API to create a custom span, you can disable stack trace collection with the
+<<api-agent-start_span,`include_stacktrace` argument>>.
+
+Reading source files inside a running application can cause a lot of disk I/O,
+and sending up source lines for each frame will have a network and storage cost that is quite high.
+Turning these limits down will prevent storing excessive amounts of data in Elasticsearch.
+
+[float]
+[[tuning-queue]]
+=== Transaction queue
+
+The Agent does not send every transaction as it happens.
+Instead, to reduce load on the APM Server, the Agent uses a queue.
+The queue is flushed periodically, or when it reaches a maximum size.
+
+While this reduces the load on the APM Server, holding on to transaction data in a queue uses memory.
+If you notice a large increase in memory use, try adjusting these settings:
+
+ * <<config-api-request-time,`api_request_time`>> to reduce the duration of a single streaming request.
+ This setting is helpful if you have a sustained high number of transactions.
+ * <<config-api-request-size,`api_request_size`>> to reduce the maximum size of one request.
+ This setting can help if you experience transaction peaks (a large number in a short period of time).
+
+Keep in mind that reducing the value of either setting will cause the agent to send more HTTP requests to the APM Server,
+potentially causing a higher load.
+
+[float]
+[[tuning-max-spans]]
+=== Spans per transaction
+
+The number of spans per transaction will influence both how much time the agent spends in each transaction collecting contextual data,
+and how much storage space is needed in Elasticsearch.
+In our experience, most _usual_ transactions should have well below 100 spans.
+In some cases, however, the number of spans can explode—for example:
+
+* Long-running transactions
+* Unoptimized code, e.g., doing hundreds of SQL queries in a loop
+
+To avoid these edge cases which overload both the Agent and the APM Server,
+the Agent will stop recording spans when a specified limit is reached.
+This limit is configurable with <<config-transaction-max-spans,`transaction_max_spans`>>.
+
+[float]
+[[tuning-body-headers]]
+=== Capturing headers and request body
+
+You can configure the Agent to capture headers and request bodies with
+<<config-capture-headers,`capture_headers`>> and <<config-capture-body,`capture_body`>>.
+By default, headers are captured and request bodies are not.
+
+Depending on the nature of your POST requests, capturing request bodies for transactions may introduce noticeable overhead,
+as well as increased storage use.
+In most scenarios, we advise against enabling request body capturing for transactions, and only enabling it if necessary for errors.
+
+Capturing request/response headers has less overhead on the agent than capturing request bodies,
+but can have an impact on storage use.

data/docs/set-up.asciidoc

@@ -9,8 +9,11 @@ endif::[]
 
 To get you off the ground, we’ve prepared guides for setting up the Agent with different frameworks:
 
+ * <<getting-started-rails,Rails>>
+ * <<getting-started-rack,Rack>>
+
+For custom instrumentation, see the <<api>>.
+
 include::./getting-started-rails.asciidoc[]
 
 include::./getting-started-rack.asciidoc[]
-
-For custom instrumentation, see the <<api>>.

data/docs/supported-technologies.asciidoc

@@ -87,6 +87,7 @@ We automatically instrument background processing using:
 - Shoryuken
 - Sneakers (v2.12.0+) (Experimental, see https://github.com/elastic/apm-agent-ruby/pull/676[#676])
 - Resque (v2.0.0+)
+- SuckerPunch (v2.0.0+)
 
 [float]
 [[supported-technologies-resque]]
@@ -113,6 +114,21 @@ I, [XXX #81227]  INFO -- : Running before_first_fork hooks
 D, [XXX #81227] DEBUG -- : Starting ElasticAPM agent
 ----
 
+Also be sure to set the Resque environment variable `RUN_AT_EXIT_HOOKS` to `true`. Otherwise, the fork may be
+terminated before the agent has a chance to send all the fork's events to the APM server.
+
+[float]
+[[supported-technologies-sucker-punch]]
+=== SuckerPunch
+
+Asynchronously executed jobs in SuckerPunch are automatically instrumented.
+
+Note that errors raised in the user-defined `JobClass#perform` method will be first handled by the SuckerPunch exception
+handler before being handled by the agent. The handler is accessed/set via `SuckerPunch.exception_handler` in version
+2.0. The agent transaction will be marked as successful unless you re-raise the error in the exception handler.
+You can also explicitly report the error via <<api-agent-report,`ElasticAPM.report`>> in a custom SuckerPunch exception
+handler.
+
 [float]
 [[supported-technologies-grpc]]
 === gRPC

data/elastic-apm.gemspec

@@ -1,3 +1,20 @@
+# Licensed to Elasticsearch B.V. under one or more contributor
+# license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright
+# ownership. Elasticsearch B.V. licenses this file to you under
+# the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
 lib = File.expand_path('../lib', __FILE__)
 $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 require 'elastic_apm/version'

data/lib/elastic-apm.rb

@@ -1,3 +1,20 @@
+# Licensed to Elasticsearch B.V. under one or more contributor
+# license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright
+# ownership. Elasticsearch B.V. licenses this file to you under
+# the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
 # frozen_string_literal: true
 
 # Make bundler auto-require work

data/lib/elastic_apm.rb

@@ -1,3 +1,20 @@
+# Licensed to Elasticsearch B.V. under one or more contributor
+# license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright
+# ownership. Elasticsearch B.V. licenses this file to you under
+# the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
 # frozen_string_literal: true
 
 require 'erb'
@@ -51,7 +68,7 @@ module ElasticAPM
     # one, if it is provided.
     # Starts the agent if it is not running.
     # Stops and starts the agent if it is running.
-    def restart(config = {})
+    def restart(config = nil)
       config ||= agent&.config
       stop if running?
       start(config)

data/lib/elastic_apm/agent.rb

@@ -1,3 +1,20 @@
+# Licensed to Elasticsearch B.V. under one or more contributor
+# license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright
+# ownership. Elasticsearch B.V. licenses this file to you under
+# the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
 # frozen_string_literal: true
 
 require 'elastic_apm/error'
@@ -34,9 +51,9 @@ module ElasticAPM
       LOCK.synchronize do
         return @instance if @instance
 
-        unless config.active?
+        unless config.enabled?
           config.logger.debug format(
-            "%sAgent disabled with `active: false'",
+            "%sAgent disabled with `enabled: false'",
             Logging::PREFIX
           )
           return
@@ -72,6 +89,7 @@ module ElasticAPM
         metrics: metrics,
         stacktrace_builder: stacktrace_builder
       ) { |event| enqueue event }
+      @pid = Process.pid
     end
 
     attr_reader(
@@ -145,6 +163,9 @@ module ElasticAPM
       context: nil,
       trace_context: nil
     )
+      return unless config.recording?
+      detect_forking!
+
       instrumenter.start_transaction(
         name,
         type,
@@ -170,6 +191,12 @@ module ElasticAPM
       parent: nil,
       sync: nil
     )
+      detect_forking!
+
+      # We don't check config.recording? because the span
+      # will not be created if there's no transaction.
+      # We want to use the recording value from the config
+      # that existed when start_transaction was called. ~estolfo
       instrumenter.start_span(
         name,
         type,
@@ -207,6 +234,8 @@ module ElasticAPM
     # errors
 
     def report(exception, context: nil, handled: true)
+      return unless config.recording?
+      detect_forking!
       return if config.filter_exception_types.include?(exception.class.to_s)
 
       error = @error_builder.build_exception(
@@ -219,6 +248,9 @@ module ElasticAPM
     end
 
     def report_message(message, context: nil, backtrace: nil, **attrs)
+      return unless config.recording?
+      detect_forking!
+
       error = @error_builder.build_log(
         message,
         context: context,
@@ -240,5 +272,19 @@ module ElasticAPM
     def inspect
       super.split.first + '>'
     end
+
+    def detect_forking!
+      return if @pid == Process.pid
+
+      config.logger.debug "Detected forking,
+        restarting threads in process [PID:#{Process.pid}]"
+
+      central_config.handle_forking!
+      transport.handle_forking!
+      instrumenter.handle_forking!
+      metrics.handle_forking!
+
+      @pid = Process.pid
+    end
   end
 end

data/lib/elastic_apm/central_config.rb

@@ -1,3 +1,20 @@
+# Licensed to Elasticsearch B.V. under one or more contributor
+# license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright
+# ownership. Elasticsearch B.V. licenses this file to you under
+# the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
 # frozen_string_literal: true
 
 require 'elastic_apm/central_config/cache_control'
@@ -81,15 +98,16 @@ module ElasticAPM
         update[key] = @modified_options.delete(key)
       end
 
-      update_config(update)
+      @config.replace_options(update)
     end
 
-    private
-
-    def update_config(new_options)
-      @config = config.dup.tap { |new_config| new_config.assign(new_options) }
+    def handle_forking!
+      stop
+      start
     end
 
+    private
+
     # rubocop:disable Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
     def handle_success(resp)
       if (etag = resp.headers['Etag'])
@@ -104,9 +122,10 @@ module ElasticAPM
           assign(update)
         end
 
-        if @modified_options.any?
+        if update && update.any?
           info 'Updated config from Kibana'
-          debug 'Modified: %s', @modified_options.inspect
+          debug 'Modified: %s', update.inspect
+          debug 'Modified original options: %s', @modified_options.inspect
         end
       end
 
@@ -151,7 +170,7 @@ module ElasticAPM
     def schedule_next_fetch(resp = nil)
       headers = resp&.headers
       seconds =
-        if headers['Cache-Control']
+        if headers && headers['Cache-Control']
           CacheControl.new(headers['Cache-Control']).max_age
         else
           DEFAULT_MAX_AGE

data/lib/elastic_apm/central_config/cache_control.rb

@@ -1,3 +1,20 @@
+# Licensed to Elasticsearch B.V. under one or more contributor
+# license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright
+# ownership. Elasticsearch B.V. licenses this file to you under
+# the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
 # frozen_string_literal: true
 
 module ElasticAPM

data/lib/elastic_apm/child_durations.rb

@@ -1,3 +1,20 @@
+# Licensed to Elasticsearch B.V. under one or more contributor
+# license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright
+# ownership. Elasticsearch B.V. licenses this file to you under
+# the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
 # frozen_string_literal: true
 
 module ElasticAPM