semantic_logger 5.2.0 → 5.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (98) hide show
  1. checksums.yaml +4 -4
  2. data/docs/api.md +160 -9
  3. data/docs/appenders.md +461 -34
  4. data/docs/config.md +543 -36
  5. data/docs/log.md +38 -7
  6. data/docs/metrics.md +139 -28
  7. data/docs/operations.md +293 -24
  8. data/docs/rails.md +319 -5
  9. data/docs/security.md +135 -107
  10. data/docs/testing.md +23 -0
  11. data/docs/upgrading.md +644 -1
  12. data/lib/semantic_logger/appender/async.rb +29 -11
  13. data/lib/semantic_logger/appender/bugsnag.rb +7 -2
  14. data/lib/semantic_logger/appender/cloudwatch_logs.rb +46 -6
  15. data/lib/semantic_logger/appender/elasticsearch_base.rb +197 -29
  16. data/lib/semantic_logger/appender/elasticsearch_http.rb +22 -5
  17. data/lib/semantic_logger/appender/file.rb +29 -14
  18. data/lib/semantic_logger/appender/graylog.rb +81 -7
  19. data/lib/semantic_logger/appender/honeybadger.rb +7 -1
  20. data/lib/semantic_logger/appender/honeybadger_insights.rb +8 -2
  21. data/lib/semantic_logger/appender/http.rb +244 -41
  22. data/lib/semantic_logger/appender/io.rb +11 -6
  23. data/lib/semantic_logger/appender/kafka.rb +192 -46
  24. data/lib/semantic_logger/appender/loki.rb +42 -6
  25. data/lib/semantic_logger/appender/mongodb.rb +94 -9
  26. data/lib/semantic_logger/appender/new_relic.rb +2 -3
  27. data/lib/semantic_logger/appender/new_relic_logs.rb +30 -8
  28. data/lib/semantic_logger/appender/open_telemetry.rb +37 -35
  29. data/lib/semantic_logger/appender/rabbitmq.rb +241 -17
  30. data/lib/semantic_logger/appender/sentry.rb +2 -3
  31. data/lib/semantic_logger/appender/sentry_ruby.rb +82 -15
  32. data/lib/semantic_logger/appender/splunk.rb +4 -4
  33. data/lib/semantic_logger/appender/splunk_http.rb +92 -1
  34. data/lib/semantic_logger/appender/syslog.rb +89 -10
  35. data/lib/semantic_logger/appender/tcp.rb +68 -98
  36. data/lib/semantic_logger/appender/text_output.rb +65 -0
  37. data/lib/semantic_logger/appender/udp.rb +130 -18
  38. data/lib/semantic_logger/appender/wrapper.rb +10 -7
  39. data/lib/semantic_logger/appender.rb +115 -10
  40. data/lib/semantic_logger/appenders.rb +136 -4
  41. data/lib/semantic_logger/backtrace_cleaning.rb +77 -0
  42. data/lib/semantic_logger/base.rb +53 -71
  43. data/lib/semantic_logger/call_site.rb +235 -0
  44. data/lib/semantic_logger/context.rb +222 -0
  45. data/lib/semantic_logger/core_ext/process.rb +20 -5
  46. data/lib/semantic_logger/core_ext/warning.rb +70 -0
  47. data/lib/semantic_logger/deprecation.rb +139 -0
  48. data/lib/semantic_logger/exception_details.rb +4 -3
  49. data/lib/semantic_logger/formatters/base.rb +139 -19
  50. data/lib/semantic_logger/formatters/color.rb +3 -4
  51. data/lib/semantic_logger/formatters/default.rb +35 -7
  52. data/lib/semantic_logger/formatters/ecs.rb +9 -4
  53. data/lib/semantic_logger/formatters/json.rb +9 -0
  54. data/lib/semantic_logger/formatters/logfmt.rb +9 -7
  55. data/lib/semantic_logger/formatters/loki/labels.rb +282 -0
  56. data/lib/semantic_logger/formatters/loki.rb +412 -24
  57. data/lib/semantic_logger/formatters/new_relic_logs.rb +15 -13
  58. data/lib/semantic_logger/formatters/one_line.rb +44 -10
  59. data/lib/semantic_logger/formatters/open_telemetry.rb +186 -29
  60. data/lib/semantic_logger/formatters/pattern.rb +60 -8
  61. data/lib/semantic_logger/formatters/raw.rb +38 -8
  62. data/lib/semantic_logger/formatters/signalfx.rb +55 -15
  63. data/lib/semantic_logger/formatters/syslog.rb +4 -2
  64. data/lib/semantic_logger/formatters/syslog_cee.rb +10 -4
  65. data/lib/semantic_logger/formatters/syslog_header.rb +49 -0
  66. data/lib/semantic_logger/formatters.rb +1 -0
  67. data/lib/semantic_logger/log.rb +236 -80
  68. data/lib/semantic_logger/loggable.rb +55 -95
  69. data/lib/semantic_logger/logger.rb +31 -5
  70. data/lib/semantic_logger/metric/new_relic.rb +32 -17
  71. data/lib/semantic_logger/metric/signalfx.rb +21 -12
  72. data/lib/semantic_logger/metric/statsd.rb +26 -8
  73. data/lib/semantic_logger/processor.rb +194 -5
  74. data/lib/semantic_logger/queue_processor/backoff.rb +140 -0
  75. data/lib/semantic_logger/queue_processor/batch_failure.rb +69 -0
  76. data/lib/semantic_logger/queue_processor/held_write.rb +62 -0
  77. data/lib/semantic_logger/queue_processor/inbox.rb +97 -0
  78. data/lib/semantic_logger/queue_processor/reopener.rb +56 -0
  79. data/lib/semantic_logger/queue_processor/reporter.rb +113 -0
  80. data/lib/semantic_logger/queue_processor/request.rb +48 -10
  81. data/lib/semantic_logger/queue_processor/throttled_counter.rb +60 -17
  82. data/lib/semantic_logger/queue_processor.rb +426 -176
  83. data/lib/semantic_logger/ractor_copy.rb +120 -0
  84. data/lib/semantic_logger/ractor_processor.rb +82 -0
  85. data/lib/semantic_logger/ractor_receiver.rb +282 -0
  86. data/lib/semantic_logger/ractor_support.rb +28 -0
  87. data/lib/semantic_logger/redactor.rb +310 -0
  88. data/lib/semantic_logger/semantic_logger.rb +313 -79
  89. data/lib/semantic_logger/subscriber.rb +169 -15
  90. data/lib/semantic_logger/sync_processor.rb +226 -26
  91. data/lib/semantic_logger/test/capture_log_events.rb +20 -2
  92. data/lib/semantic_logger/test/minitest.rb +2 -1
  93. data/lib/semantic_logger/test/rspec.rb +1 -1
  94. data/lib/semantic_logger/trap_safe.rb +113 -0
  95. data/lib/semantic_logger/utils.rb +266 -79
  96. data/lib/semantic_logger/version.rb +1 -1
  97. data/lib/semantic_logger.rb +35 -2
  98. metadata +22 -2
data/docs/log.md CHANGED
@@ -55,8 +55,9 @@ A handful of attributes are populated for **every** event:
55
55
 
56
56
  The rest are **situational**: they are set only when relevant to that particular call. For example
57
57
  `duration` is present only for `measure_*` calls, `exception` only when an exception was logged,
58
- `metric`/`metric_amount`/`dimensions` only for metrics, and `backtrace` only when backtrace
59
- capture is enabled (`SemanticLogger.backtrace_level`). Always guard against `nil` when reading a
58
+ `metric`/`metric_amount`/`dimensions` only for metrics, `backtrace` only when backtrace
59
+ capture is enabled (`SemanticLogger.backtrace_level`), and `call_site` only when call site recording
60
+ is enabled (`SemanticLogger.call_site_level`). Always guard against `nil` when reading a
60
61
  situational field.
61
62
 
62
63
  ### Attributes
@@ -73,13 +74,39 @@ situational field.
73
74
  |`tags`|`Array<String>`|Tags active on the thread when the call was made.|
74
75
  |`named_tags`|`Hash<String, Object>`|Named tags active on the thread when the call was made.|
75
76
  |`context`|`Hash`|Named contexts captured in-line at the point the event was created. `nil` when none.|
77
+ |`captures`|`Hash`|Objects an appender captured in-line for its own use, such as the OpenTelemetry context. Unlike `context`, formatters do not write them, except one built for that appender, as `:new_relic_logs` writes the linking metadata its appender captures. `nil` when none.|
76
78
  |`duration`|`Float`|Time in milliseconds taken by a `measure_*` call. `nil` for ordinary log calls.|
77
79
  |`exception`|`Exception`|The Ruby exception that was logged. `nil` when none. Use `each_exception` to walk a `cause` chain.|
78
80
  |`backtrace`|`Array<String>`|Backtrace captured at the call site, present only when the level is at or above `SemanticLogger.backtrace_level`.|
81
+ |`call_site`|`SemanticLogger::CallSite`|Where the entry was logged, present only when the level is at or above `SemanticLogger.call_site_level`, and the log call did not pass `call_site: false`. See [The call site](#the-call-site).|
79
82
  |`metric`|`String`|The metric name, present for metric and `measure_*` calls. See [Metrics](metrics.html).|
80
- |`metric_amount`|`Float`|Numeric amount for a counter or gauge metric, for example the quantity purchased.|
83
+ |`metric_amount`|`Numeric`|Amount to add to a counter metric, for example the quantity purchased. A counter adds 1 when it is `nil`. A negative amount with a `metric` is deprecated, see [Negative metric amounts](upgrading.html#negative-metric-amounts).|
81
84
  |`dimensions`|`Hash`|Dimensions (key/value labels) supplied for a metric.|
82
85
 
86
+ ### The call site
87
+
88
+ `call_site` is a frozen value object that records where the entry was logged. See
89
+ [Recording where each entry was logged](config.html#recording-where-each-entry-was-logged) for how
90
+ it is found.
91
+
92
+ |Method|Returns|Description|
93
+ |------|-------|-----------|
94
+ |`path`|`String`|The file, as in a backtrace, for example `"/app/models/order.rb"`.|
95
+ |`line`|`Integer`|The line number in the file.|
96
+ |`label`|`String` or `nil`|The label Ruby gives the frame, usually the method: `"Order#submit"` on Ruby 3.4 and later, `"submit"` on Ruby 3.2, 3.3, and JRuby.|
97
+ |`file_name_and_line(short_name: false)`|`[String, Integer]`|The path and line. With `short_name: true` the file name has no directory, for example `"order.rb"`.|
98
+ |`to_s`|`String`|The call site as a backtrace line, for example `"/app/models/order.rb:42:in 'Order#submit'"`.|
99
+
100
+ ~~~ruby
101
+ class MyFormatter
102
+ def call(log, _logger)
103
+ site = log.call_site
104
+ location = site ? " (#{site.label} at #{File.basename(site.path)}:#{site.line})" : ""
105
+ "#{log.level_to_s} #{log.name} -- #{log.message}#{location}"
106
+ end
107
+ end
108
+ ~~~
109
+
83
110
  ### Helper methods
84
111
 
85
112
  These read-only helpers derive convenient values from the attributes above:
@@ -87,18 +114,22 @@ These read-only helpers derive convenient values from the attributes above:
87
114
  |Method|Returns|Description|
88
115
  |------|-------|-----------|
89
116
  |`payload?`|`true`/`false`|Whether the event carries a non-empty payload.|
117
+ |`dimensions?`|`true`/`false`|Whether the event has a metric with dimensions, a `Hash` that is not empty. The formatters write the dimensions only then.|
90
118
  |`payload_to_s`|`String` or `nil`|The payload rendered with `inspect`, or `nil` when absent.|
91
- |`metric_only?`|`true`/`false`|`true` when the event has a metric but no message and no exception. Text appenders typically skip these; machine-readable (JSON) appenders usually keep them.|
119
+ |`metric_only?`|`true`/`false`|`true` when the event has a metric but no message and no exception. Metric subscribers receive these; log appenders leave them out unless their `metrics` option is true, the default only for `:kafka`, `:rabbitmq`, and `:udp` until v6. See [Metrics in log appenders](metrics.html#metrics-in-log-appenders).|
92
120
  |`duration_to_s`|`String` or `nil`|The duration in milliseconds as a short string, for example `"12ms"`. `nil` when there is no duration.|
93
121
  |`duration_human`|`String` or `nil`|The duration in human readable form, scaling up to seconds, minutes, hours, or days as needed.|
94
122
  |`level_to_s`|`String`|The level as a single upper-case character, for example `"I"` for `:info`.|
95
123
  |`cleansed_message`|`String`|The message with Rails/ANSI color escape codes stripped. Used to keep terminal codes out of structured output.|
96
124
  |`each_exception`|Enumerator|Iterates the exception and its nested `cause` chain, yielding `(exception, depth)`.|
97
- |`backtrace_to_s`|`String`|The exception backtrace as a string, including every exception in the `cause` chain. With a block, each cause's message and each backtrace line is replaced by the value the block returns, for example to escape it.|
125
+ |`backtrace_to_s(cleaner: nil)`|`String`|The exception backtrace as a string, including every exception in the `cause` chain. Lines are separated by a line break, with none before the first or after the last, so put your own line break before it. With a block, each cause's message and each backtrace line is replaced by the value the block returns, for example to escape it. With a `cleaner`, each backtrace is first cleaned as `backtrace_of` cleans it. Each cause is written as `Cause: Class: message`, with the class named as Ruby names it in `Exception#inspect`: the constant it was assigned to, even when the class overrides `self.name`, or for an anonymous class the class as Ruby writes it, such as `#<Class:0x000...>`. A cause's message, as written and as passed to the block, is always a `String`: when its `#message` raises, the message it was raised with is used instead, and a `nil` message is empty. Text in encodings that cannot be joined, such as a binary message and a UTF-8 backtrace line, is joined as UTF-8, and so are binary or US-ASCII backtrace lines that hold bytes outside ASCII, such as Ruby's own where no locale is set in a directory such as `café`, which would otherwise make the whole text binary or invalid US-ASCII.|
126
+ |`backtrace_of(exception, cleaner: nil)`|`Array<String>` or `nil`|The backtrace of `exception` (this event's exception or one of its causes). With a [backtrace cleaner](config.html#cleaning-backtraces) object, such as `Rails.backtrace_cleaner` (not `true`, as the formatter option accepts), the lines it keeps, or the full backtrace when it keeps none; the exception is not changed. A cleaned backtrace, and each of its lines, is frozen and cached on the event.|
98
127
  |`exception_details(exception, source: false)`|`String` or `nil`|The detail Ruby's `Exception#detailed_message` adds to the message of `exception` (this event's exception or one of its causes), such as did_you_mean suggestions, without the class name. `source: true` also includes error_highlight's source snippet. `nil` when there is none. Cached on the event.|
99
- |`file_name_and_line(short_name = false)`|`[String, Integer]` or `nil`|The file name and line number from the event's backtrace or exception. Pass `true` for just the base file name.|
128
+ |`file_name_and_line(short_name = false)`|`[String, Integer]` or `nil`|The file name and line number from the event's call site, or else its backtrace or exception. Pass `true` for just the base file name.|
100
129
  |`to_h`|`Hash`|The event as a plain Hash (via the Raw formatter), including `host`, `application`, and `environment`. Handy for inspecting an event or building a custom formatter.|
101
- |`set_context(key, value)`|`Hash`|Lazily initializes `context` and stores a key/value pair on it.|
130
+ |`set_context(key, value)`|`Object`|Lazily initializes `context`, stores a key/value pair on it, and returns the value.|
131
+ |`capture(key, value)`|`Object`|Lazily initializes `captures`, stores a key/value pair on it, and returns the value, for an appender that needs state of the logging thread, such as the current trace, that is not log data.|
132
+ |`captured(key)`|`Object` or `nil`|The value captured with `key`, or `nil` when there is none.|
102
133
 
103
134
  ### Inspecting an event
104
135
 
data/docs/metrics.md CHANGED
@@ -27,8 +27,10 @@ There are two parts to using metrics:
27
27
  1. **Emit** a metric by adding the `:metric` option to any log or `measure_` call. This is covered in
28
28
  Steps 1 to 3 below.
29
29
  2. **Subscribe** a destination (Statsd, New Relic, SignalFx, and so on) that receives those metrics.
30
- Until a subscriber is registered, the `:metric` option is harmless and the number goes nowhere.
31
- See [Send metrics to a backend](#send-metrics-to-a-backend).
30
+ Until a subscriber is registered, the `:metric` option is harmless: the number reaches no metrics
31
+ backend, though the log appenders that write structured data still write it with the entry. See
32
+ [Send metrics to a backend](#send-metrics-to-a-backend) and
33
+ [Metrics in log appenders](#metrics-in-log-appenders).
32
34
 
33
35
  Metric subscribers are notified asynchronously on the background log thread, so emitting a metric
34
36
  never slows the calling thread. A metric rides along with its log entry, so it obeys the same level
@@ -49,12 +51,18 @@ and the metric is incremented by 1:
49
51
  logger.info(message: "Signed up", metric: "users/signup")
50
52
  ~~~
51
53
 
52
- Supply `:metric_amount` to change the amount. To decrement:
54
+ Supply `:metric_amount` to change the amount. A metric counts upward, so the amount is zero or
55
+ more. To count something that goes down, such as items removed from a cart, give it a metric of its
56
+ own:
53
57
 
54
58
  ~~~ruby
55
- logger.info(message: "Item removed", metric: "cart/items", metric_amount: -1)
59
+ logger.info(message: "Item removed", metric: "cart/items_removed")
56
60
  ~~~
57
61
 
62
+ A negative `:metric_amount` with a `:metric` is deprecated: it is still logged and sent as before,
63
+ but reports a [deprecation warning](upgrading.html#seeing-deprecation-warnings), and v6 drops it. See
64
+ [Negative metric amounts](upgrading.html#negative-metric-amounts).
65
+
58
66
  To add to a running total, such as the dollar amount of purchases per department:
59
67
 
60
68
  ~~~ruby
@@ -85,23 +93,70 @@ logger.info(message: "Called supplier", metric: "supplier/add_user", duration: 1
85
93
  ## Step 3: Break a metric down with dimensions
86
94
 
87
95
  Dimensions are key/value labels attached to a metric, so a backend that supports them (such as
88
- SignalFx) can slice it, for example by user, action, or state. Add them with the `:dimensions`
89
- option:
96
+ SignalFx) can break it down, for example by region, plan, or status. Add them with the
97
+ `:dimensions` option:
90
98
 
91
99
  ~~~ruby
92
- # A counter, broken down by user:
93
- logger.info(metric: "filters.count", dimensions: {user: "jbloggs"})
100
+ # A counter, broken down by plan:
101
+ logger.info(metric: "filters.count", dimensions: {plan: "premium"})
102
+
103
+ # An amount, broken down by region:
104
+ logger.info(metric: "filters.applied", metric_amount: 3, dimensions: {region: "eu-west"})
94
105
 
95
- # A gauge with an amount and dimensions:
96
- logger.info(metric: "filters.average", metric_amount: 1.2, dimensions: {user: "jbloggs"})
106
+ # A duration, broken down by supplier:
107
+ logger.measure_info("Called supplier", metric: "supplier/add_user", dimensions: {supplier: "acme"}) do
108
+ # Code to call the external service ...
109
+ end
97
110
  ~~~
98
111
 
99
- Not every backend supports dimensions. **Statsd** and **New Relic** ignore any metric that carries
100
- dimensions; **SignalFx** is built around them. See each subscriber below.
112
+ Not every subscriber supports dimensions. **Statsd** and **New Relic** record a metric with
113
+ dimensions under its name alone, leaving out the dimensions, which their clients do not accept;
114
+ **SignalFx** is built around them. See each subscriber below. To keep the metrics with dimensions
115
+ out of Statsd or New Relic, for example when they also go to SignalFx, add the subscriber with
116
+ `filter: ->(log) { !log.dimensions? }`. The formatters that write structured data also write the
117
+ dimensions with the entry, see [Metrics in log appenders](#metrics-in-log-appenders).
118
+
119
+ ### Keep the values of each dimension bounded
120
+
121
+ Give each dimension a small, fixed set of values, such as a status, a region, a plan, an HTTP
122
+ method, or the route of a request (`/users/:id`, not `/users/42`). Never use a value that grows
123
+ with your data or your traffic, such as a user id, an email address, an order or request id, an IP
124
+ address, a session, a timestamp, or a URL with ids in it.
125
+
126
+ A metrics backend keeps a separate time series for every combination of dimension values that it
127
+ receives under one metric name, and most count each one as a metric of its own:
128
+
129
+ * [Datadog](https://docs.datadoghq.com/account_management/billing/custom_metrics/) identifies a
130
+ custom metric by its name together with its tag values, so each combination is billed as a
131
+ custom metric.
132
+ * [SignalFx](https://help.splunk.com/en/splunk-observability-cloud/manage-data/metrics-metadata-and-events/metrics-events-and-metadata/get-started-with-metrics)
133
+ (Splunk Observability Cloud) keeps a metric time series (MTS) for each combination of a metric
134
+ and its dimensions. The active ones count toward a
135
+ [limit](https://help.splunk.com/en/splunk-observability-cloud/administer/org-reference-info/per-product-system-limits-in-splunk-observability-cloud/infrastructure-system-limits-details)
136
+ that your subscription sets, beyond which it refuses new time series without an error, so new
137
+ values stop being recorded.
138
+ * [Google Cloud Monitoring](https://docs.cloud.google.com/monitoring/api/v3/metric-model#cardinality)
139
+ keeps a time series for each combination of label values, and asks that the values of every
140
+ label be constrained: no unique identifiers, user ids, IP addresses, timestamps, or URLs that
141
+ are not parameterized.
142
+
143
+ The combinations multiply: a metric with a `status` of 5 values, a `region` of 4, and a `plan` of 3
144
+ can make up to 60 time series. Add a `user_id` dimension for 10,000 users, and the same metric can
145
+ make up to 600,000, each one billed or counted against a limit.
146
+
147
+ The same holds for the metric name: never build it from an id, as in `"users/#{user.id}/fetch"`.
148
+ Statsd and New Relic then record a new metric for every user, and SignalFx turns the parts of the
149
+ name into `class` and `action` dimensions, a new time series for every user.
150
+
151
+ To find what happened to one user, order, or request, log its id in the
152
+ [payload](api.html#step-3-add-structured-data) of the entry instead, where a log appender stores
153
+ it with that entry rather than as a metric of its own. Named tags are written with the entry too,
154
+ except by the `:loki` appender, which makes each named tag a stream label: Loki keeps a stream for
155
+ every combination of label values, so an id there has the same effect as an id in a dimension.
101
156
 
102
157
  ## Send metrics to a backend
103
158
 
104
- A metric goes nowhere until a **metric subscriber** is registered. Add one with
159
+ A metric reaches no metrics backend until a **metric subscriber** is registered. Add one with
105
160
  `SemanticLogger.add_appender(metric: ...)`, usually when your application starts. A subscriber
106
161
  receives every logged entry that has a `:metric`, asynchronously on the background thread.
107
162
 
@@ -115,7 +170,11 @@ forward them to [Graphite](https://graphiteapp.org), MongoDB, and others (requir
115
170
  SemanticLogger.add_appender(metric: :statsd, url: "udp://localhost:8125")
116
171
  ~~~
117
172
 
118
- Counters are integers (float amounts are rounded). Does not support dimensions.
173
+ The `url:` must be `udp://`, and its host must resolve when the subscriber is added, or `add_appender`
174
+ raises. A path in the url, as in `udp://localhost:8125/my_app`, is the namespace of every metric.
175
+
176
+ Counters are integers (float amounts are rounded). A metric is sent under its name alone, leaving out
177
+ its dimensions, since the `statsd-ruby` client has no tags.
119
178
 
120
179
  ### New Relic
121
180
 
@@ -126,7 +185,11 @@ Forward metrics to New Relic so they can be displayed on custom dashboards (requ
126
185
  SemanticLogger.add_appender(metric: :new_relic)
127
186
  ~~~
128
187
 
129
- Does not support dimensions.
188
+ New Relic charts custom metrics only under `Custom/`, so the subscriber records
189
+ `metric: "Orders/placed"` as `Custom/Orders/placed`, and a name that already starts with `Custom/`
190
+ as it is. To group an application's metrics, pass a `prefix:` that also starts with `Custom/`:
191
+ `prefix: "Custom/MyApp"` records `Orders/placed` as `Custom/MyApp/Orders/placed`. A metric is sent
192
+ under its name alone, leaving out its dimensions, since the agent's metric API takes none.
130
193
 
131
194
  ### SignalFx
132
195
 
@@ -137,27 +200,75 @@ which is built around dimensions:
137
200
  SemanticLogger.add_appender(metric: :signalfx, token: "SIGNALFX_ORG_ACCESS_TOKEN")
138
201
  ~~~
139
202
 
140
- `application` and `host` are always sent as dimensions. To also forward specific named tags as
141
- dimensions whenever they are present on a log entry, list them:
203
+ `application` and `host` are always sent as dimensions, and so is `environment` when it is set. To
204
+ also forward specific named tags as dimensions whenever they are present, and not blank, on a log
205
+ entry that has no dimensions of its own, list them:
142
206
 
143
207
  ~~~ruby
144
208
  SemanticLogger.add_appender(
145
209
  metric: :signalfx,
146
210
  token: "SIGNALFX_ORG_ACCESS_TOKEN",
147
- dimensions: [:user_id, :state]
211
+ dimensions: [:region, :plan]
148
212
  )
149
213
  ~~~
150
214
 
151
- When a duration metric has no dimensions, SignalFx receives both a gauge and a counter, so you can
152
- chart both the timing and the number of occurrences. When dimensions are present, the metric is sent
153
- as-is.
154
-
155
- ### Elasticsearch and Splunk
156
-
157
- These are ordinary log [appenders](appenders.html), not metric subscribers, but a metric is part of
158
- the log entry, so `metric`, `metric_amount`, and any dimensions are written into the document
159
- automatically. You can build dashboards on those fields directly, without registering a separate
160
- metric subscriber.
215
+ List only named tags whose values are bounded (see
216
+ [Keep the values of each dimension bounded](#keep-the-values-of-each-dimension-bounded)): a named
217
+ tag such as a request or user id would make a new metric time series for every request or user.
218
+
219
+ When a duration metric has no dimensions, SignalFx receives both its value and a count, so you can
220
+ chart both the timing and the number of occurrences. The value is the duration and the count is 1,
221
+ unless a `metric_amount` is given, which is sent as both. They are sent as `Application.average` and
222
+ `Application.counter`, with `class` and `action` dimensions taken from the metric name, such as
223
+ `class: "supplier"` and `action: "add_user"` for `supplier/add_user`. A count without dimensions is
224
+ sent as `Application.counter` alone, with the same `class` and `action` dimensions.
225
+
226
+ When dimensions are present, the metric is sent as-is: under its own name, without its leading `/`
227
+ characters and with each other `/` replaced by `.`, with its value (the `metric_amount` when given,
228
+ otherwise the duration, or 1 for a count), and, for a duration, without a count. Its dimensions are sent in place of the named tags
229
+ listed in `dimensions:` above, so add any of those values it needs to its own dimensions.
230
+
231
+ A dimension whose value is blank, `nil` or text that is empty or only whitespace, is left out, since
232
+ it carries nothing to break the metric down by. Dimensions that are empty, such as `{}`, or whose
233
+ values are all blank, such as `{state: ""}` or `{supplier: nil}`, count as none, so the metric is
234
+ sent as a metric without dimensions is.
235
+
236
+ ## Metrics in log appenders
237
+
238
+ Metric subscribers are the destinations for metrics. Log [appenders](appenders.html) treat a metric
239
+ as part of the log entry that carries it:
240
+
241
+ * **A metric on an entry with a message or an exception** is written with the entry by the
242
+ formatters that write structured data, so a log store such as Elasticsearch or Splunk can chart
243
+ it without a metric subscriber. The text formatters do not write it; they show only the
244
+ entry's duration.
245
+ * **A metric-only entry**, one with a metric but no message and no exception, such as
246
+ `logger.info(metric: "users/signup")`, is meant for the metric subscribers. There can be many
247
+ such entries, so log appenders leave them out, unless one is added with `metrics: true`:
248
+
249
+ ~~~ruby
250
+ # Also index metric-only entries in Elasticsearch:
251
+ SemanticLogger.add_appender(appender: :elasticsearch, url: "http://localhost:9200", metrics: true)
252
+ ~~~
253
+
254
+ Until v6, the `:kafka`, `:rabbitmq`, and `:udp` appenders send them by default, and report a
255
+ [deprecation warning](upgrading.html#seeing-deprecation-warnings) when they are added without
256
+ `metrics:`. Pass `metrics: false` to leave them out, as v6 does by default, or `metrics: true` to
257
+ keep sending them.
258
+
259
+ What each formatter writes for a metric on an entry with a message:
260
+
261
+ | Formatter | Metric fields written |
262
+ |-----------|-----------------------|
263
+ | `:json`, `:raw`, `:fluentd`, `:syslog_cee`, `:new_relic_logs` | `metric`, `metric_amount`, and `dimensions` |
264
+ | `:ecs` | `metric`, `metric_amount`, and `dimensions`, under its namespace, `semantic_logger` by default, or in `labels` when it has none |
265
+ | `:loki` | `metric`, `metric_value`, and `dimensions` (as JSON), in the structured metadata |
266
+ | `:open_telemetry` | The `semantic_logger.metric` and `semantic_logger.metric_amount` attributes, and a `semantic_logger.dimensions.<key>` attribute for each dimension |
267
+ | `:default`, `:color`, `:one_line`, `:pattern`, `:syslog`, `:logfmt` | None; only the duration is shown |
268
+
269
+ An appender writes what its formatter writes. For example `:elasticsearch`, `:opensearch`,
270
+ `:splunk_http`, `:mongodb`, and `:graylog` use `:raw` by default, and `:http`, `:tcp`, `:udp`,
271
+ `:kafka`, `:rabbitmq`, and `:cloudwatch_logs` use `:json`.
161
272
 
162
273
  ## How metrics behave
163
274