elastic-apm 2.8.1 → 2.11.0

data/docs/api.asciidoc

@@ -30,9 +30,10 @@ ElasticAPM.start(server_url: 'http://localhost:8200')
   * `config`: An optional hash or `ElasticAPM::Config` instance with configuration
   options.  See <<configuration,Configuration>>.
 
-If you are using <<getting-started-rails,Ruby on Rails>> this is done
-automatically for you.
-If not see <<getting-started-rack,Getting started with Rack>>.
+If you are using <<getting-started-rails,Ruby on Rails>> this is done automatically for you.
+If you choose not to require the `elastic_apm` gem and instead want to start the
+agent and hook into Rails manually, see <<rails-start,hooking into Rails explicitly>>.
+If you're not using Rails, see <<getting-started-rack,Getting started with Rack>>.
 
 [float]
 [[api-agent-stop]]
@@ -140,7 +141,9 @@ end
 Arguments:
 
   * `name`: A name for your span. **Required**.
-  * `type`: The type of work eg. `db.postgresql.query`.
+  * `type`: The type of span eg. `db`.
+  * `subtype`: The subtype of span eg. `postgresql`.
+  * `action`: The action type of span eg. `connect` or `query`.
   * `context`: An instance of `Span::Context`.
   * `include_stacktrace`: Whether or not to collect a Stacktrace.
   * `&block`: An optional block to wrap with the span.
@@ -163,7 +166,9 @@ Wraps a block in a Span.
 Arguments:
 
   * `name`: A name for your span. **Required**.
-  * `type`: The type of work eg. `db.postgresql.query`.
+  * `type`: The type of span eg. `db`.
+  * `subtype`: The subtype of span eg. `postgresql`.
+  * `action`: The action type of span eg. `connect` or `query`.
   * `context`: An instance of `Span::Context`.
   * `include_stacktrace`: Whether or not to collect a Stacktrace.
   * `&block`: An optional block to wrap with the span.
@@ -186,6 +191,18 @@ Arguments:
 
 Returns the built context.
 
+[float]
+[[rails-start]]
+=== Manually hooking into Rails
+
+Start the agent and hook into Rails explicitly. This is useful if you skip requiring
+the gem and using the `Railtie`.
+
+[source,ruby]
+----
+ElasticAPM::Rails.start(server_url: 'http://localhost:8200')
+----
+
 [float]
 === Errors
 
@@ -212,7 +229,7 @@ Arguments:
   * `handled`: Whether the error was _handled_ eg. wasn't rescued and was represented
   to the user. Default: `true`.
 
-Returns `[ElasticAPM::Error]`.
+Returns `[String]` ID of the generated `[ElasticAPM::Error]` object.
 
 [float]
 [[api-agent-report-message]]
@@ -231,7 +248,7 @@ Arguments:
 
   * `message`: A custom error string. **Required**.
 
-Returns `[ElasticAPM::Error]`.
+Returns `[String]` ID of the generated `[ElasticAPM::Error]` object.
 
 [float]
 [[api-context]]

data/docs/configuration.asciidoc

@@ -62,7 +62,7 @@ Some options can be set with `ENV` variables and all of them may be set in
 your source code.
 
 When setting values for lists using `ENV` variables, strings are split by comma
-eg `ELASTIC_APM_ENABLED_ENVIRONMENTS=development,production`.
+eg `ELASTIC_APM_CUSTOM_KEY_FILTERS="a,b" # => [/a/, /b/]`.
 
 [float]
 [[config-config-file]]
@@ -213,6 +213,21 @@ Whether or not to attach the request headers to transactions and errors.
 
 Whether or not to attach `ENV` from Rack to transactions and errors.
 
+[float]
+[[config-central-config]]
+==== `central_config`
+|============
+| Environment                  | `Config` key     | Default
+| `ELASTIC_APM_CENTRAL_CONFIG` | `central_config` | `true`
+|============
+
+Enable {kibana-ref}/agent-configuration.html[APM Agent Configuration via Kibana].
+If set to `true`, the client will poll the APM Server regularly for new agent configuration.
+
+Usually APM Server determines how often to poll, but if not the default interval is 5 minutes.
+
+NOTE: This feature requires APM Server v7.3 or later and that the APM Server is configured with `kibana.enabled: true`.
+
 [float]
 [[config-custom-key-filters]]
 ==== `custom_key_filters`
@@ -244,6 +259,10 @@ Add default tags to add to every transaction.
 
 WARNING: Be aware that tags are indexed in Elasticsearch. Using too many unique keys will result in *https://www.elastic.co/blog/found-crash-elasticsearch#mapping-explosion[Mapping explosion]*.
 
+NOTE: `global_labels` are supported as of APM server version 7.2. `default_tags` will eventually be
+deprecated so please transition to using `global_labels` instead. In the meantime, any `default_tags`
+that are set will override `global_labels`.
+
 [float]
 [[config-disable-send]]
 ==== `disable_send`
@@ -292,11 +311,14 @@ Get an array of enabled spies with `ElasticAPM.agent.config.enabled_spies`.
 The name of the environment this service is deployed in,
 e.g. "production" or "staging".
 
+Environments allow you to easily filter data on a global level in the APM UI.
+It's important to be consistent when naming environments across agents.
+See {kibana-ref}/filters.html#environment-selector[environment selector] in the Kibana UI for more information.
+
 Defaults to `ENV['RAILS_ENV'] || ENV['RACK_ENV']`.
 
-NOTE: Kibana will not differentiate between different environments. To avoid
-mixing data from multiple environments, consider appending the environment to
-`service_name`.
+NOTE: This feature is fully supported in the APM UI in Kibana versions >= 7.2.
+You must use the query bar to filter for a specific environment in versions prior to 7.2.
 
 [float]
 [[config-filter-exception-types]]
@@ -334,6 +356,23 @@ Version number of the used framework.
 For Ruby on Rails and Sinatra, this defaults to the used version of the framework,
 otherwise, the default is `nil`.
 
+[float]
+[[config-global-labels]]
+==== `global_labels`
+
+[options="header"]
+|============
+| Environment                 | `Config` key    | Default  | Example
+| `ELASTIC_APM_GLOBAL_LABELS` | `global_labels` | `nil`    | `dept=engineering,rack=number8`
+|============
+
+Labels added to all events, with the format key=value[,key=value[,...]].
+
+NOTE: This option requires APM Server 7.2 or greater, and will have no effect when using older
+server versions. `default_tags` will eventually be deprecated but in the meantime, their value
+will override any `global_labels`. Please transition to using `global_labels` instead of
+`default_tags` in light of this deprecation.
+
 [float]
 [[config-hostname]]
 ==== `hostname`

data/docs/index.asciidoc

@@ -18,6 +18,8 @@ include::./getting-started-rack.asciidoc[]
 
 include::./configuration.asciidoc[]
 
+include::./log-correlation.asciidoc[]
+
 include::./metrics.asciidoc[]
 
 include::./advanced.asciidoc[]

data/docs/log-correlation.asciidoc

@@ -0,0 +1,96 @@
+ifdef::env-github[]
+NOTE: For the best reading experience,
+please view this documentation at https://www.elastic.co/guide/en/apm/agent/ruby[elastic.co]
+endif::[]
+
+[[log-correlation]]
+== Log correlation
+
+Trace/log correlation can be set up in three different ways.
+
+[float]
+[[rails-tagged-logging]]
+==== Rails TaggedLogging
+
+Rails applications configured with an `ActiveSupport::TaggedLogging` logger can append the correlation IDs to log output.
+For example in your `config/environments/production.rb` file, add the following:
+
+[source,ruby]
+----
+config.log_tags = [ :request_id, proc { ElasticAPM.log_ids } ]
+
+# Logs will then include the correlation IDs:
+#
+# [transaction.id=c1ae84c8642891eb trace.id=b899fc7915e801b7558e336e4952bafe] Started GET "/" for 127.0.0.1 at 2019-09-16 11:28:46 +0200
+# [transaction.id=c1ae84c8642891eb trace.id=b899fc7915e801b7558e336e4952bafe] Processing by ApplicationController#index as HTML
+# [transaction.id=c1ae84c8642891eb trace.id=b899fc7915e801b7558e336e4952bafe]   Rendering text template
+# [transaction.id=c1ae84c8642891eb trace.id=b899fc7915e801b7558e336e4952bafe]   Rendered text template (Duration: 0.1ms | Allocations: 17)
+# [transaction.id=c1ae84c8642891eb trace.id=b899fc7915e801b7558e336e4952bafe] Completed 200 OK in 1ms (Views: 0.4ms | Allocations: 171)
+----
+**Note:** Because of the order in which Rails computes the tags for logs and executes the request, the span id might not be included.
+Consider using `Lograge` instead, as the timing of its hooks allow the span id to be captured in logs.
+
+[float]
+[[lograge]]
+==== Lograge
+
+With `lograge` enabled and set up in your Rails application, modify the `custom_options` block in the Rails environment
+configuration file. The returned `Hash` will be included in the structured Lograge logs.
+
+[source,ruby]
+----
+config.lograge.custom_options = lambda do |event|
+  ElasticAPM.log_ids do |transaction_id, span_id, trace_id|
+    { :'transaction.id' => transaction_id,
+      :'span.id' => span_id,
+      :'trace.id' => trace_id }
+  end
+end
+
+# Logs will then include the correlation IDs:
+#
+# I, [2019-09-16T11:59:05.439602 #8674]  INFO -- : method=GET path=/ format=html controller=ApplicationController action=index status=200 duration=0.36 view=0.20 transaction.id=56a9186a9257aa08 span.id=8e84a786ab0abbb2 trace.id=1bbab8ac4c7c9584f53eb882ff0dfdd8
+----
+
+You can also nest the ids in a separate document as in the following example:
+
+[source,ruby]
+----
+config.lograge.custom_options = lambda do |event|
+  ElasticAPM.log_ids do |transaction_id, span_id, trace_id|
+    { elastic_apm: { :'transaction.id' => transaction_id,
+                     :'span.id' => span_id,
+                     :'trace.id' => trace_id } }
+  end
+end
+
+# Logs will then include the correlation IDs in a separate document:
+#
+# I, [2019-09-16T13:39:35.962603 #9327]  INFO -- : method=GET path=/ format=html controller=ApplicationController action=index status=200 duration=0.37 view=0.20 elastic_apm={:transaction_id=>"2fb84f5d0c48a296", :span_id=>"2e5c5a7c85f83be7", :trace_id=>"43e1941c4a6fff343a4e018ff7b92000"}
+----
+
+[float]
+[[manually-formatting-logs]]
+==== Manually formatting logs
+
+You can access the correlation ids directly and add them through the log formatter.
+
+[source,ruby]
+----
+require 'elastic_apm'
+require 'logger'
+
+logger = Logger.new(STDOUT)
+logger.progname = 'TestRubyApp'
+logger.formatter  = proc do |severity, datetime, progname, msg|
+  "[#{datetime}][#{progname}][#{severity}][#{ElasticAPM.log_ids}] #{msg}\n"
+end
+
+# Logs will then include the correlation IDs:
+#
+# [2019-09-16 11:54:59 +0200][RailsTestApp][INFO][transaction.id=3b92edcccc0a6d1e trace.id=1275686e35de91f776557637e799651e] Started GET "/" for 127.0.0.1 at 2019-09-16 11:54:59 +0200
+# [2019-09-16 11:54:59 +0200][RailsTestApp][INFO][transaction.id=3b92edcccc0a6d1e trace.id=1275686e35de91f776557637e799651e] Processing by ApplicationController#index as HTML
+# [2019-09-16 11:54:59 +0200][RailsTestApp][INFO][transaction.id=3b92edcccc0a6d1e span.id=3bde4e9c85ab359c trace.id=1275686e35de91f776557637e799651e]   Rendering text template
+# [2019-09-16 11:54:59 +0200][RailsTestApp][INFO][transaction.id=3b92edcccc0a6d1e span.id=f3d7e32f176d4c93 trace.id=1275686e35de91f776557637e799651e]   Rendered text template (Duration: 0.1ms | Allocations: 17)
+# [2019-09-16 11:54:59 +0200][RailsTestApp][INFO][transaction.id=3b92edcccc0a6d1e span.id=3bde4e9c85ab359c trace.id=1275686e35de91f776557637e799651e] Completed 200 OK in 1ms (Views: 0.3ms | Allocations: 187)
+----

data/docs/metrics.asciidoc

@@ -12,11 +12,15 @@ You can adjust the interval by setting <<config-metrics-interval,`metrics_interv
 
 The metrics will be stored in the `apm-*` index and have the `processor.event` property set to `metric`.
 
+[float]
+[[metrics-system]]
+=== System metrics
+
 **Note:** Metrics from the Ruby agent are Linux only for now.
 
 [float]
 [[metric-system.cpu.total.norm.pct]]
-=== `system.cpu.total.norm.pct`
+==== `system.cpu.total.norm.pct`
 
 * *Type:* Float
 * *Format:* Percent
@@ -26,7 +30,7 @@ normalised by the number of cores.
 
 [float]
 [[metric-system.memory.total]]
-=== `system.memory.total`
+==== `system.memory.total`
 
 * *Type:* Long
 * *Format:* Bytes
@@ -35,7 +39,7 @@ The total memory of the system in bytes.
 
 [float]
 [[metric-system.memory.actual.free]]
-=== `system.memory.actual.free`
+==== `system.memory.actual.free`
 
 * *Type:* Long
 * *Format:* Bytes
@@ -44,7 +48,7 @@ Free memory of the system in bytes.
 
 [float]
 [[metric-system.process.cpu.total.norm.pct]]
-=== `system.process.cpu.total.norm.pct`
+==== `system.process.cpu.total.norm.pct`
 
 * *Type:* Float
 * *Format:* Percent
@@ -54,7 +58,7 @@ This value is normalized by the number of CPU cores and it ranges from 0 to 100%
 
 [float]
 [[metric-system.process.memory.size]]
-=== `system.process.memory.size`
+==== `system.process.memory.size`
 
 * *Type:* Long
 * *Format:* Bytes
@@ -63,10 +67,77 @@ The total virtual memory the process has.
 
 [float]
 [[metric-system.process.memory.rss.bytes]]
-=== `system.process.memory.rss.bytes`
+==== `system.process.memory.rss.bytes`
 
 * *Type:* Long
 * *Format:* Bytes
 
 The Resident Set Size,
 the amount of memory the process occupies in main memory (RAM).
+
+[float]
+[[metrics-ruby]]
+=== Ruby Metrics
+
+[float]
+[[metric-ruby.gc.counts]]
+==== `ruby.gc.count`
+
+* *Type:* Integer
+* *Format:* Count
+
+The number of Garbage Collection runs since the process started.
+
+[float]
+[[metric-ruby.threads]]
+==== `ruby.threads`
+
+* *Type:* Integer
+* *Format:* Count
+
+The number of threads belonging to the current process.
+
+[float]
+[[metric-ruby.heap.slots.live]]
+==== `ruby.heap.slots.live`
+
+* *Type:* Integer
+* *Format:* Slots
+
+Current amount of heap slots that are live.
+
+**NB:** Not currently supported on JRuby.
+
+[float]
+[[metric-ruby.heap.slots.free]]
+==== `ruby.heap.slots.free`
+
+* *Type:* Integer
+* *Format:* Slots
+
+Current amount of heap slots that are free.
+
+**NB:** Not currently supported on JRuby.
+
+[float]
+[[metrics-ruby.heap.allocations.total]]
+==== `ruby.heap.allocations.total`
+
+* *Type:* Integer
+* *Format:* Objects
+
+Current amount of allocated objects on the heap.
+
+**NB:** Not currently supported on JRuby.
+
+[float]
+[[metrics-ruby.gc.time]]
+==== `ruby.gc.time`
+
+* *Type:* Float
+* *Format:* Seconds
+
+The total time spent in garbage collection.
+
+**NB:** You need to enable Ruby's GC Profiler for this to get reported.
+You can do this at any time when your application boots by calling `GC::Profiler.enable`.

data/lib/elastic_apm.rb

@@ -62,6 +62,26 @@ module ElasticAPM # rubocop:disable Metrics/ModuleLength
       agent&.current_span
     end
 
+    # rubocop:disable Metrics/AbcSize
+    # Get a formatted string containing transaction, span, and trace ids.
+    # If a block is provided, the ids are yielded.
+    #
+    # @yield [String|nil, String|nil, String|nil] The transaction, span, and trace ids.
+    # @return [String] Unless block given
+    def log_ids
+      trace_id = (current_transaction || current_span)&.trace_id
+      if block_given?
+        return yield(current_transaction&.id, current_span&.id, trace_id)
+      end
+
+      ids = []
+      ids << "transaction.id=#{current_transaction.id}" if current_transaction
+      ids << "span.id=#{current_span.id}" if current_span
+      ids << "trace.id=#{trace_id}" if trace_id
+      ids.join(' ')
+    end
+    # rubocop:enable Metrics/AbcSize
+
     # Start a new transaction or return the currently running
     #
     # @param name [String] A description of the transaction, eg
@@ -207,16 +227,21 @@ module ElasticAPM # rubocop:disable Metrics/ModuleLength
 
     deprecate :span, :with_span
 
+    # rubocop:disable Metrics/MethodLength, Metrics/ParameterLists
     # Start a new span
     #
     # @param name [String] A description of the span, eq `SELECT FROM "users"`
-    # @param type [String] The kind of span, eq `db.mysql2.query`
+    # @param type [String] The span type, eq `db`
+    # @param subtype [String] The span subtype, eq `postgresql`
+    # @param action [String] The span action type, eq `connect` or `query`
     # @param context [Span::Context] Context information about the span
     # @param include_stacktrace [Boolean] Whether or not to capture a stacktrace
     # @return [Span]
     def start_span(
       name,
       type = nil,
+      subtype: nil,
+      action: nil,
       context: nil,
       include_stacktrace: true,
       trace_context: nil
@@ -224,6 +249,8 @@ module ElasticAPM # rubocop:disable Metrics/ModuleLength
       agent&.start_span(
         name,
         type,
+        subtype: subtype,
+        action: action,
         context: context,
         trace_context: trace_context
       ).tap do |span|
@@ -233,6 +260,7 @@ module ElasticAPM # rubocop:disable Metrics/ModuleLength
         span.original_backtrace ||= caller
       end
     end
+    # rubocop:enable Metrics/MethodLength, Metrics/ParameterLists
 
     # Ends the current span
     #
@@ -241,7 +269,7 @@ module ElasticAPM # rubocop:disable Metrics/ModuleLength
       agent&.end_span
     end
 
-    # rubocop:disable Metrics/MethodLength
+    # rubocop:disable Metrics/MethodLength, Metrics/ParameterLists
     # Wrap a block in a Span, ending it after the block
     #
     # @param name [String] A description of the span, eq `SELECT FROM "users"`
@@ -253,6 +281,8 @@ module ElasticAPM # rubocop:disable Metrics/ModuleLength
     def with_span(
       name,
       type = nil,
+      subtype: nil,
+      action: nil,
       context: nil,
       include_stacktrace: true,
       trace_context: nil
@@ -269,6 +299,8 @@ module ElasticAPM # rubocop:disable Metrics/ModuleLength
           start_span(
             name,
             type,
+            subtype: subtype,
+            action: action,
             context: context,
             include_stacktrace: include_stacktrace,
             trace_context: trace_context
@@ -278,7 +310,7 @@ module ElasticAPM # rubocop:disable Metrics/ModuleLength
         end_span
       end
     end
-    # rubocop:enable Metrics/MethodLength
+    # rubocop:enable Metrics/MethodLength, Metrics/ParameterLists
 
     # Build a [Context] from a Rack `env`. The context may include information
     # about the request, response, current user and more
@@ -305,7 +337,7 @@ module ElasticAPM # rubocop:disable Metrics/ModuleLength
     # @param exception [Exception] The exception
     # @param context [Context] An optional [Context]
     # @param handled [Boolean] Whether the exception was rescued
-    # @return [Error] The generated [Error]
+    # @return [String] ID of the generated [Error]
     def report(exception, context: nil, handled: true)
       agent&.report(exception, context: context, handled: handled)
     end
@@ -314,7 +346,7 @@ module ElasticAPM # rubocop:disable Metrics/ModuleLength
     #
     # @param message [String] The message
     # @param context [Context] An optional [Context]
-    # @return [Error] The generated [Error]
+    # @return [String] ID of the generated [Error]
     def report_message(message, context: nil, **attrs)
       agent&.report_message(
         message,