semantic_logger 5.2.0 → 5.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (98) hide show
  1. checksums.yaml +4 -4
  2. data/docs/api.md +160 -9
  3. data/docs/appenders.md +461 -34
  4. data/docs/config.md +543 -36
  5. data/docs/log.md +38 -7
  6. data/docs/metrics.md +139 -28
  7. data/docs/operations.md +293 -24
  8. data/docs/rails.md +319 -5
  9. data/docs/security.md +135 -107
  10. data/docs/testing.md +23 -0
  11. data/docs/upgrading.md +644 -1
  12. data/lib/semantic_logger/appender/async.rb +29 -11
  13. data/lib/semantic_logger/appender/bugsnag.rb +7 -2
  14. data/lib/semantic_logger/appender/cloudwatch_logs.rb +46 -6
  15. data/lib/semantic_logger/appender/elasticsearch_base.rb +197 -29
  16. data/lib/semantic_logger/appender/elasticsearch_http.rb +22 -5
  17. data/lib/semantic_logger/appender/file.rb +29 -14
  18. data/lib/semantic_logger/appender/graylog.rb +81 -7
  19. data/lib/semantic_logger/appender/honeybadger.rb +7 -1
  20. data/lib/semantic_logger/appender/honeybadger_insights.rb +8 -2
  21. data/lib/semantic_logger/appender/http.rb +244 -41
  22. data/lib/semantic_logger/appender/io.rb +11 -6
  23. data/lib/semantic_logger/appender/kafka.rb +192 -46
  24. data/lib/semantic_logger/appender/loki.rb +42 -6
  25. data/lib/semantic_logger/appender/mongodb.rb +94 -9
  26. data/lib/semantic_logger/appender/new_relic.rb +2 -3
  27. data/lib/semantic_logger/appender/new_relic_logs.rb +30 -8
  28. data/lib/semantic_logger/appender/open_telemetry.rb +37 -35
  29. data/lib/semantic_logger/appender/rabbitmq.rb +241 -17
  30. data/lib/semantic_logger/appender/sentry.rb +2 -3
  31. data/lib/semantic_logger/appender/sentry_ruby.rb +82 -15
  32. data/lib/semantic_logger/appender/splunk.rb +4 -4
  33. data/lib/semantic_logger/appender/splunk_http.rb +92 -1
  34. data/lib/semantic_logger/appender/syslog.rb +89 -10
  35. data/lib/semantic_logger/appender/tcp.rb +68 -98
  36. data/lib/semantic_logger/appender/text_output.rb +65 -0
  37. data/lib/semantic_logger/appender/udp.rb +130 -18
  38. data/lib/semantic_logger/appender/wrapper.rb +10 -7
  39. data/lib/semantic_logger/appender.rb +115 -10
  40. data/lib/semantic_logger/appenders.rb +136 -4
  41. data/lib/semantic_logger/backtrace_cleaning.rb +77 -0
  42. data/lib/semantic_logger/base.rb +53 -71
  43. data/lib/semantic_logger/call_site.rb +235 -0
  44. data/lib/semantic_logger/context.rb +222 -0
  45. data/lib/semantic_logger/core_ext/process.rb +20 -5
  46. data/lib/semantic_logger/core_ext/warning.rb +70 -0
  47. data/lib/semantic_logger/deprecation.rb +139 -0
  48. data/lib/semantic_logger/exception_details.rb +4 -3
  49. data/lib/semantic_logger/formatters/base.rb +139 -19
  50. data/lib/semantic_logger/formatters/color.rb +3 -4
  51. data/lib/semantic_logger/formatters/default.rb +35 -7
  52. data/lib/semantic_logger/formatters/ecs.rb +9 -4
  53. data/lib/semantic_logger/formatters/json.rb +9 -0
  54. data/lib/semantic_logger/formatters/logfmt.rb +9 -7
  55. data/lib/semantic_logger/formatters/loki/labels.rb +282 -0
  56. data/lib/semantic_logger/formatters/loki.rb +412 -24
  57. data/lib/semantic_logger/formatters/new_relic_logs.rb +15 -13
  58. data/lib/semantic_logger/formatters/one_line.rb +44 -10
  59. data/lib/semantic_logger/formatters/open_telemetry.rb +186 -29
  60. data/lib/semantic_logger/formatters/pattern.rb +60 -8
  61. data/lib/semantic_logger/formatters/raw.rb +38 -8
  62. data/lib/semantic_logger/formatters/signalfx.rb +55 -15
  63. data/lib/semantic_logger/formatters/syslog.rb +4 -2
  64. data/lib/semantic_logger/formatters/syslog_cee.rb +10 -4
  65. data/lib/semantic_logger/formatters/syslog_header.rb +49 -0
  66. data/lib/semantic_logger/formatters.rb +1 -0
  67. data/lib/semantic_logger/log.rb +236 -80
  68. data/lib/semantic_logger/loggable.rb +55 -95
  69. data/lib/semantic_logger/logger.rb +31 -5
  70. data/lib/semantic_logger/metric/new_relic.rb +32 -17
  71. data/lib/semantic_logger/metric/signalfx.rb +21 -12
  72. data/lib/semantic_logger/metric/statsd.rb +26 -8
  73. data/lib/semantic_logger/processor.rb +194 -5
  74. data/lib/semantic_logger/queue_processor/backoff.rb +140 -0
  75. data/lib/semantic_logger/queue_processor/batch_failure.rb +69 -0
  76. data/lib/semantic_logger/queue_processor/held_write.rb +62 -0
  77. data/lib/semantic_logger/queue_processor/inbox.rb +97 -0
  78. data/lib/semantic_logger/queue_processor/reopener.rb +56 -0
  79. data/lib/semantic_logger/queue_processor/reporter.rb +113 -0
  80. data/lib/semantic_logger/queue_processor/request.rb +48 -10
  81. data/lib/semantic_logger/queue_processor/throttled_counter.rb +60 -17
  82. data/lib/semantic_logger/queue_processor.rb +426 -176
  83. data/lib/semantic_logger/ractor_copy.rb +120 -0
  84. data/lib/semantic_logger/ractor_processor.rb +82 -0
  85. data/lib/semantic_logger/ractor_receiver.rb +282 -0
  86. data/lib/semantic_logger/ractor_support.rb +28 -0
  87. data/lib/semantic_logger/redactor.rb +310 -0
  88. data/lib/semantic_logger/semantic_logger.rb +313 -79
  89. data/lib/semantic_logger/subscriber.rb +169 -15
  90. data/lib/semantic_logger/sync_processor.rb +226 -26
  91. data/lib/semantic_logger/test/capture_log_events.rb +20 -2
  92. data/lib/semantic_logger/test/minitest.rb +2 -1
  93. data/lib/semantic_logger/test/rspec.rb +1 -1
  94. data/lib/semantic_logger/trap_safe.rb +113 -0
  95. data/lib/semantic_logger/utils.rb +266 -79
  96. data/lib/semantic_logger/version.rb +1 -1
  97. data/lib/semantic_logger.rb +35 -2
  98. metadata +22 -2
data/docs/upgrading.md CHANGED
@@ -3,7 +3,7 @@ layout: default
3
3
  title: Upgrading
4
4
  description: >-
5
5
  Version-by-version upgrade notes for Semantic Logger, from v4.0 through the
6
- v5.2 security fixes, remote syslog facility fix, and deprecation of the Splunk SDK appender.
6
+ v5.3 OpenTelemetry log record changes.
7
7
  ---
8
8
 
9
9
 
@@ -12,6 +12,649 @@ description: >-
12
12
  * TOC
13
13
  {:toc}
14
14
 
15
+ ### Seeing deprecation warnings
16
+
17
+ Semantic Logger reports the use of a deprecated appender or option as a Ruby deprecation warning, the
18
+ same way that Ruby reports its own, naming the line of your application that used it, and what to use
19
+ instead:
20
+
21
+ ~~~
22
+ config/initializers/logging.rb:12: warning: SemanticLogger: SemanticLogger::Appender::Sentry (appender: :sentry) is deprecated and will be removed in v6. It depends on the end-of-life sentry-raven gem. Use appender: :sentry_ruby instead.
23
+ ~~~
24
+
25
+ When a gem adds the appender for you, the warning names the line of that gem that added it. A
26
+ warning about a log entry that an appender reports while it writes the entry, on Semantic Logger's
27
+ background thread, names no line, since your code is not on that thread; to see the line, run with
28
+ `SemanticLogger.sync!`, as tests often do.
29
+
30
+ Ruby only shows deprecation warnings while `Warning[:deprecated]` is true, which it is not by default,
31
+ so production stays quiet without any configuration:
32
+
33
+ | Where | Deprecation warnings |
34
+ |-------|----------------------|
35
+ | Tests run with Minitest or test-unit | On, since both turn them on |
36
+ | Tests run with RSpec | Off, unless run with `rspec --warnings`, or `Warning[:deprecated] = true` is set in `spec_helper.rb` |
37
+ | Development | Off, unless `RUBYOPT=-W:deprecated` is set in the environment, or `Warning[:deprecated] = true` in code that only runs in development, such as `config/environments/development.rb` |
38
+ | Production | Off |
39
+
40
+ Rails settings such as `config.active_support.deprecation` only apply to the deprecation warnings of
41
+ Rails, not to those that Ruby reports.
42
+
43
+ Ruby writes each warning to `$stderr`. To send them elsewhere, such as to a file of their own, or to
44
+ raise them in your tests so that each use is found, extend `Warning` with your own `warn`:
45
+
46
+ ~~~ruby
47
+ module SemanticLoggerDeprecations
48
+ DEPRECATIONS = ::Logger.new("log/deprecations.log")
49
+
50
+ def warn(message, *, **)
51
+ return super unless message.include?("SemanticLogger: ")
52
+
53
+ DEPRECATIONS.warn(message.chomp)
54
+ end
55
+ end
56
+ Warning.extend(SemanticLoggerDeprecations)
57
+ ~~~
58
+
59
+ Do not log them through Semantic Logger itself: a warning reported on its background thread would
60
+ then wait on that thread's own queue, which can stop all logging while the queue is full.
61
+
62
+ ### Upgrading to Semantic Logger v5.3
63
+
64
+ #### OpenTelemetry log records
65
+
66
+ The `:open_telemetry` appender now sends the message as the body of each log record, as its
67
+ documentation described, instead of a map of the whole log entry. This matters if you query, alert
68
+ on, or build dashboards from the body of these records.
69
+
70
+ ~~~
71
+ # Before:
72
+ body: {"host" => "web-1", "application" => "my_app", "environment" => "production",
73
+ "pid" => 12345, "thread" => "puma srv tp 001", "named_tags" => {request_id: "abc"},
74
+ "name" => "UserService", "message" => "Created user"}
75
+ attributes: {"user_id" => 42}
76
+
77
+ # After:
78
+ body: "Created user"
79
+ attributes: {"request_id" => "abc", "user_id" => 42, "semantic_logger.name" => "UserService",
80
+ "thread.name" => "puma srv tp 001", "process.pid" => 12345}
81
+ ~~~
82
+
83
+ * The payload attributes keep their names. An Array that mixes types, such as `[1, "two"]`, is now
84
+ sent as Strings, `["1", "two"]`, since OpenTelemetry does not accept it otherwise.
85
+ * Each named tag is an attribute of its own, such as `request_id`, instead of `body.named_tags`. A
86
+ payload key takes precedence over a named tag of the same name.
87
+ * The other fields of the entry are attributes too. A logged exception uses the OpenTelemetry
88
+ semantic conventions, `exception.type`, `exception.message`, and `exception.stacktrace`, so that
89
+ backends recognize it, and so do `code.file.path`, `code.line.number`, `thread.name`, and
90
+ `process.pid`. The logger name, tags, duration, and metric are `semantic_logger.name`,
91
+ `semantic_logger.tags`, `semantic_logger.duration_ms`, `semantic_logger.metric`, and
92
+ `semantic_logger.metric_amount`, and each of a metric's dimensions is a
93
+ `semantic_logger.dimensions.<key>` attribute. A named tag or payload key with one of these names, or
94
+ starting with `semantic_logger.dimensions.`, is prefixed with its source, as in
95
+ `payload.exception.type`.
96
+ * The host, application, and environment are no longer sent. Set them in the OpenTelemetry resource
97
+ instead, which is sent once per export, with `OTEL_SERVICE_NAME=my_app` and
98
+ `OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production,host.name=web-1`.
99
+ * An entry without a message, such as `logger.error(exception)`, has no body.
100
+ * A named tag whose value is a Symbol, Time, or other object was sent as an empty value. It is now
101
+ sent as its `#to_s`, or as JSON for a Hash, Array, `Struct`, or `Set`.
102
+ * `code.file.path` and `code.line.number` are where the entry was logged, and are left out when the
103
+ entry has no call site (see `call_site_level`) and did not capture a backtrace. With a call site,
104
+ `code.function.name` names the method. The old body's `file` and `line` fell back to where a logged
105
+ exception was raised, which is now in `exception.stacktrace`.
106
+
107
+ A custom formatter given to this appender with `formatter:` must now return the keyword arguments for
108
+ the OpenTelemetry `Logger#on_emit` (`severity_text:`, `severity_number:`, `timestamp:`, `body:`, and
109
+ `attributes:`), as `SemanticLogger::Formatters::OpenTelemetry` does, instead of a Hash shaped like the
110
+ `:raw` formatter's.
111
+
112
+ The appender no longer sends metric-only entries, such as `logger.info(metric: "users/signup")`,
113
+ which have a metric but no message and no exception. It sent each one as a log record with no body,
114
+ its metric in the attributes, never as an OpenTelemetry metric. Metric-only entries are meant for the
115
+ [metric subscribers](metrics.html#send-metrics-to-a-backend), and most log appenders leave them out.
116
+ To keep sending them as log records, add the appender with `metrics: true`:
117
+
118
+ ~~~ruby
119
+ SemanticLogger.add_appender(appender: :open_telemetry, metrics: true)
120
+ ~~~
121
+
122
+ #### Choose `metrics:` for the `:kafka`, `:rabbitmq`, and `:udp` appenders
123
+
124
+ Unlike the other log appenders, `:kafka`, `:rabbitmq`, and `:udp` send metric-only entries by
125
+ default: entries such as `logger.info(metric: "users/signup")`, which have a metric but no message and
126
+ no exception, and are meant for the [metric subscribers](metrics.html#send-metrics-to-a-backend). v6
127
+ leaves them out by default, as the other log appenders do. Until then, adding one of these appenders
128
+ without `metrics:` still sends them, and reports a
129
+ [deprecation warning](#seeing-deprecation-warnings). Pass `metrics:` to choose:
130
+
131
+ ~~~ruby
132
+ # Keep sending metric-only entries:
133
+ SemanticLogger.add_appender(appender: :kafka, seed_brokers: ["kafka1:9092"], metrics: true)
134
+
135
+ # Leave them out, as v6 does by default:
136
+ SemanticLogger.add_appender(appender: :kafka, seed_brokers: ["kafka1:9092"], metrics: false)
137
+ ~~~
138
+
139
+ #### Negative metric amounts
140
+
141
+ A metric counts upward. In v6, a counter drops a negative `metric_amount`, and reports it once on
142
+ Semantic Logger's internal logger, rather than subtracting it. Until then, a log call or a
143
+ `measure_*` call with a `metric:` and a negative `metric_amount` is still logged and sent as before,
144
+ and reports a [deprecation warning](#seeing-deprecation-warnings) that names the line of the call.
145
+ It is reported
146
+ once per process, for the first negative amount only, so that a loop does not repeat it; once that
147
+ call is changed, the next one, if any, is reported. A `metric_amount` of zero is not negative, and
148
+ one without a `metric:`, which no metric subscriber receives, is not reported.
149
+
150
+ ~~~ruby
151
+ # Deprecated:
152
+ logger.info(message: "Item removed", metric: "cart/items", metric_amount: -1)
153
+
154
+ # Count what goes down as a metric of its own:
155
+ logger.info(message: "Item removed", metric: "cart/items_removed")
156
+ ~~~
157
+
158
+ To record a value that goes up and down, such as the number of items in a cart, a `gauge` is planned
159
+ for the metrics API of v6.
160
+
161
+ #### Choose where Loki writes `pid` and `thread`
162
+
163
+ The `:loki` formatter writes the process id and the thread name as the stream labels `pid` and
164
+ `thread`, so every process restart, and every thread, adds a Loki stream. Its new
165
+ `structured_metadata:` option writes either or both as structured metadata instead, as Loki advises,
166
+ and v6 writes both there by default. Until then, adding the `:loki` appender, or building its
167
+ formatter, without `structured_metadata:` still writes them as labels, and reports a
168
+ [deprecation warning](#seeing-deprecation-warnings). Pass it to choose:
169
+
170
+ ~~~ruby
171
+ # Write them as structured metadata, as v6 does by default:
172
+ SemanticLogger.add_appender(appender: :loki, url: "https://logs-prod-001.grafana.net",
173
+ formatter: {loki: {structured_metadata: [:pid, :thread]}})
174
+
175
+ # Keep them as labels:
176
+ SemanticLogger.add_appender(appender: :loki, url: "https://logs-prod-001.grafana.net",
177
+ formatter: {loki: {structured_metadata: []}})
178
+ ~~~
179
+
180
+ Structured metadata needs a Loki with schema v13 and `allow_structured_metadata`, which is on by
181
+ default since Loki 3.0. Any other Loki rejects every entry once `pid` is structured metadata, so
182
+ keep them as labels until your Loki accepts it.
183
+
184
+ As structured metadata, a query filters on them after the stream selector, a payload key named `pid`
185
+ or `thread` is written as `payload_pid` or `payload_thread`, and a named tag or context key of that
186
+ name as the label `pid_2` or `thread_2`:
187
+
188
+ ```
189
+ # Labels:
190
+ {application="shop", thread="worker-1"}
191
+ # Structured metadata:
192
+ {application="shop"} | thread="worker-1"
193
+ ```
194
+
195
+ See [pid and thread as structured metadata](appenders.html#pid-and-thread-as-structured-metadata).
196
+
197
+ #### Set the options of an appender instance on the instance
198
+
199
+ `add_appender` adds an appender or metric subscriber that is already created, given as `appender:` or
200
+ `metric:`, as it is, since it was created with its own options. A formatter block, or the options of
201
+ an appender, such as `level:`, `formatter:`, or `filter:`, given with it were ignored without a word.
202
+ They still are, but `add_appender` now reports a [deprecation warning](#seeing-deprecation-warnings)
203
+ that names them, and v6 raises `ArgumentError` for them. Set them on the instance itself, such as
204
+ when creating a [custom appender](config.html#custom-appenders):
205
+
206
+ ~~~ruby
207
+ # Before, level: and formatter: were ignored:
208
+ appender = SimpleAppender.new(url: "https://logs.example.com")
209
+ SemanticLogger.add_appender(appender: appender, level: :warn, formatter: :json)
210
+
211
+ # After:
212
+ appender = SimpleAppender.new(url: "https://logs.example.com", level: :warn, formatter: :json)
213
+ SemanticLogger.add_appender(appender: appender)
214
+ ~~~
215
+
216
+ The options of `add_appender` itself, such as `async:`, `batch:`, and `max_queue_size:`, still apply
217
+ to an instance. An option given as `nil`, such as `level: nil`, sets nothing, so it is not reported.
218
+
219
+ #### Give `add_appender` one destination
220
+
221
+ `add_appender` adds one destination, given as `file_name:`, `io:`, `logger:`, `appender:`, or
222
+ `metric:`. Given more than one of them, it added only the first, in that order, and ignored the
223
+ others without a word, so that `add_appender(file_name: "log/app.log", io: $stdout)` wrote to the
224
+ file, and nothing to `$stdout`. It still does, but now reports a
225
+ [deprecation warning](#seeing-deprecation-warnings) that names the destination it uses and those
226
+ it ignores, and v6 raises `ArgumentError` for them. Call `add_appender` once for each destination:
227
+
228
+ ~~~ruby
229
+ # Before, $stdout was ignored:
230
+ SemanticLogger.add_appender(file_name: "log/app.log", io: $stdout)
231
+
232
+ # After:
233
+ SemanticLogger.add_appender(file_name: "log/app.log")
234
+ SemanticLogger.add_appender(io: $stdout)
235
+ ~~~
236
+
237
+ An option that only the ignored destination accepts, such as the `url:` of `appender: :http`, went
238
+ to the one that was added, which raised `ArgumentError: unknown keyword`. While
239
+ [deprecation warnings are shown](#seeing-deprecation-warnings), the warning now comes before it,
240
+ naming the destination that was ignored.
241
+
242
+ `appender: :wrapper` only worked with `logger:`, which `add_appender` took as the destination
243
+ instead, so it now reports the same warning. Leave out `appender: :wrapper`:
244
+ `add_appender(logger: my_logger)` adds the same appender.
245
+
246
+ A destination given as `nil`, such as `file_name: options[:log_file]` when no log file is set, is not
247
+ one, so it is not reported.
248
+
249
+ #### Batch an appender with a formatter that can format a batch
250
+
251
+ `add_appender` now raises `ArgumentError` when it would batch an appender whose formatter cannot
252
+ format a batch, such as `formatter: :default` or a Proc given to `:http`, `:loki`, or
253
+ `metric: :signalfx` with `batch: true`, or a formatter block, which v5.3 now uses (see
254
+ [Other behaviour changes](#other-behaviour-changes)). Before, it added the appender, but every batch
255
+ failed on the appender's worker thread, which backed off and tried again, so nothing was ever
256
+ written, and once its queue was full, logging blocked, unless it was added with
257
+ `non_blocking: true`. Add the appender without
258
+ `batch: true`, or give it a formatter that implements `#batch(logs, logger)`, such as the one it
259
+ uses by default:
260
+
261
+ ~~~ruby
262
+ # Before, nothing was ever sent:
263
+ SemanticLogger.add_appender(appender: :http, url: "https://logs.example.com", batch: true,
264
+ formatter: ->(log, _logger) { log.message })
265
+
266
+ # After, one request for each entry:
267
+ SemanticLogger.add_appender(appender: :http, url: "https://logs.example.com",
268
+ formatter: ->(log, _logger) { log.message })
269
+
270
+ # Or one request for each batch, formatted by the default :json formatter:
271
+ SemanticLogger.add_appender(appender: :http, url: "https://logs.example.com", batch: true)
272
+ ~~~
273
+
274
+ `:splunk_http` with `batch: true` is not refused: it now sends each batch as the HTTP Event
275
+ Collector's event objects, stacked one after the other. Before, every batch failed. See
276
+ [Splunk HTTP](appenders.html#splunk-http).
277
+
278
+ A custom appender whose `#batch` formats each entry with `format_log` is batched with any formatter,
279
+ as before, including a subclass of `SemanticLogger::Appender::Http` that replaces its `#batch`. A
280
+ custom appender that cannot be batched in some configuration, such as one whose `#batch` formats the
281
+ whole batch with `format_batch`, can now say why from `#batch_refusal`, so that `add_appender`
282
+ raises instead. So can a subclass of `SemanticLogger::Appender::Http` whose `#batch` calls `super`,
283
+ which is not refused on its own. See [Custom appenders](config.html#custom-appenders).
284
+
285
+ #### Batch Elasticsearch with the `:elasticsearch` appender
286
+
287
+ `add_appender(appender: :elasticsearch_http, batch: true)` now raises `ArgumentError`. Batching it
288
+ never wrote anything: it posted each batch as a JSON array to the url's own path, such as `/`, which
289
+ Elasticsearch rejects, since it indexes a batch of entries only through its bulk API. Use the
290
+ `:elasticsearch` appender, which sends its entries in bulk requests, a batch at a time by default,
291
+ or add `:elasticsearch_http` without `batch: true`, to send each entry in its own request:
292
+
293
+ ~~~ruby
294
+ # Before, nothing was ever indexed:
295
+ SemanticLogger.add_appender(appender: :elasticsearch_http, url: "http://localhost:9200", batch: true)
296
+
297
+ # After, in batches, through the bulk API (requires the elasticsearch gem):
298
+ SemanticLogger.add_appender(appender: :elasticsearch, url: "http://localhost:9200")
299
+
300
+ # Or one request for each entry:
301
+ SemanticLogger.add_appender(appender: :elasticsearch_http, url: "http://localhost:9200")
302
+ ~~~
303
+
304
+ #### Loki stack traces
305
+
306
+ The `:loki` formatter now writes the `stack_trace` of a logged exception one frame per line, followed
307
+ by the backtrace of each exception that caused it, as the text formatters do. It wrote an inspected
308
+ Ruby Array and left out the causes. Update any query or alert that matches the old form:
309
+
310
+ ~~~
311
+ # Before:
312
+ ["app/models/order.rb:12:in 'Order#ship'", "app/controllers/orders_controller.rb:7:in 'OrdersController#create'"]
313
+
314
+ # After:
315
+ app/models/order.rb:12:in 'Order#ship'
316
+ app/controllers/orders_controller.rb:7:in 'OrdersController#create'
317
+ Cause: ArgumentError: invalid quantity
318
+ app/models/line_item.rb:3:in 'LineItem#save'
319
+ ~~~
320
+
321
+ #### Custom formatters with a `backtrace_cleaner` of their own
322
+
323
+ Formatters now accept a `backtrace_cleaner` option (see
324
+ [Cleaning backtraces](config.html#cleaning-backtraces)), which the constructor of
325
+ `SemanticLogger::Formatters::Base` sets. A formatter of your own that keeps its own cleaner in
326
+ `@backtrace_cleaner`, and assigns it before calling `super`, has it replaced, by `true` in a
327
+ formatter built on the text or JSON formatters.
328
+
329
+ If your formatter cleans backtraces itself, rename its variable. Otherwise pass the cleaner to
330
+ `super` as the option, and the formatter it is built on cleans the backtraces it writes with it:
331
+
332
+ ~~~ruby
333
+ class MyJson < SemanticLogger::Formatters::Json
334
+ def initialize(**args)
335
+ super(backtrace_cleaner: Rails.backtrace_cleaner, **args)
336
+ end
337
+ end
338
+ ~~~
339
+
340
+ #### `logger_measure_method` has been removed
341
+
342
+ `logger_measure_method`, an experiment that `SemanticLogger::Loggable` added to each class, has been
343
+ removed. A class that still calls it raises `NoMethodError` when it is loaded. To keep logging how
344
+ long a method takes, wrap its body in a `measure_*` block, and pass `metric:` to keep the metric it
345
+ logged:
346
+
347
+ ~~~ruby
348
+ def run
349
+ logger.measure_info("#run", metric: "MyClass/run") do
350
+ # ...
351
+ end
352
+ end
353
+ ~~~
354
+
355
+ To trace the method instead, use OpenTelemetry's `tracer.in_span`, or the method tracer of an APM
356
+ agent.
357
+
358
+ #### Rails: cleaned backtraces, redaction, and call sites by default
359
+
360
+ If you use [rails_semantic_logger](rails.html), its v5.3 release requires Semantic Logger v5.3 and
361
+ turns on three of its new features by default: exception backtraces are cleaned with
362
+ `Rails.backtrace_cleaner`, the keys in `config.filter_parameters` are redacted from every log entry,
363
+ and in development each entry records its call site. Each changes the log output and can be turned
364
+ off. See [Upgrading to v5.3](rails.html#upgrading-to-v53) for details.
365
+
366
+ #### Other behaviour changes
367
+
368
+ * The `:loki` formatter now writes a payload key named like one of its own structured metadata
369
+ fields (`metric`, `metric_value`, `dimensions`, `duration`, `duration_human`, `exception_name`,
370
+ `exception_message`, and `stack_trace`), or that Loki stores under one of those names, such as
371
+ `metric.value`, with a `payload_` prefix, also in an entry without that field, and under the name
372
+ Loki stores it as, such as `payload_metric_value`. Before, Loki kept either the payload's value or
373
+ the formatter's. Update any query or alert on such a payload key:
374
+
375
+ ```
376
+ # Before:
377
+ {application="shop"} | duration="PT5S"
378
+ # After:
379
+ {application="shop"} | payload_duration="PT5S"
380
+ ```
381
+ * The `:loki` formatter no longer lets a named tag, or a value that an `on_log` subscriber adds to
382
+ the context, replace a label of the same name. Its own labels (`pid`, `application`,
383
+ `environment`, `host`, `level`, `thread`, and `tags`) keep their values, a named tag keeps its
384
+ value over a context key, and the later of the two is written with a number added, as in
385
+ `host_2`, unless both have the same value. Before, the later value replaced the earlier one. Update
386
+ any query or alert that selects such a named tag or context value by that name:
387
+
388
+ ```
389
+ # Before, with the named tag host: "db-1":
390
+ {application="shop", host="db-1"}
391
+ # After:
392
+ {application="shop", host_2="db-1"}
393
+ ```
394
+
395
+ A subclass of `SemanticLogger::Formatters::Loki` that reads, replaces, or deletes a label in
396
+ `stream[:stream]` needs to use a String key, such as `stream[:stream]["host"]`. The label names
397
+ are now Strings rather than Symbols, so that a named tag or context key, whether a Symbol or a
398
+ String, is found among them by name. A label that the subclass writes under a Symbol, as v5.2
399
+ did, is written under its name as a String, and with a number added when a label written before
400
+ it has that name, as a named tag is: `stream[:stream][:level] = "INFO"` after `super` adds
401
+ `level_2="INFO"` rather than replacing `level`.
402
+
403
+ A subclass that overrides both `#build_stream` and `#call` (or `#batch`), calling `super` in
404
+ neither, leaves out each named tag and context value whose name another label has, or that Loki
405
+ does not accept, such as `request-id`. The formatter writes those labels last, once the labels
406
+ that keep their names are written: from its own `#build_stream`, or, for a subclass whose
407
+ `#build_stream` does not call `super`, from its own `#call` and `#batch`. Such a subclass calls
408
+ neither. To keep them, call `super` from the subclass's `#build_stream` and add or change its own
409
+ labels in the stream it returns, or remove its `#call` and `#batch` when they repeat the
410
+ formatter's own, as a copy of v5.2's does.
411
+ * The `:loki` formatter now writes a label past Loki's default limits, the 16th and later label of a
412
+ stream, or one whose name is longer than 1024 bytes or whose value is longer than 2048 bytes, to
413
+ the structured metadata instead, since Loki rejected the whole push for it. If your Loki raises
414
+ these limits in its `limits_config`, set the formatter's options of the same names to match, or
415
+ those labels can no longer be selected as labels:
416
+
417
+ ```ruby
418
+ SemanticLogger.add_appender(
419
+ appender: :loki,
420
+ url: "https://logs-prod-001.grafana.net",
421
+ formatter: {loki: {structured_metadata: [:pid, :thread], max_label_names_per_series: 30}}
422
+ )
423
+ ```
424
+
425
+ See [Loki's label limits](appenders.html#lokis-label-limits).
426
+ * The `#logger` of an appender added with `logger:`, such as the one `add_appender(logger: ...)`
427
+ returns, is now its internal logger, the one it reports its own problems to, as for every other
428
+ appender, and `#logger=` replaces that internal logger. Before, they returned and replaced the
429
+ logger the appender writes to, which is now `#wrapped_logger` (the appender returned by
430
+ `add_appender(logger: ..., async: true)` also answers it): `appender.logger = other` sent the
431
+ entries to `other`, and `appender.logger.level = ::Logger::DEBUG` changed the level of the logger
432
+ written to. To write to another logger, add an appender for it with
433
+ `SemanticLogger.add_appender(logger: other)`, and remove the old one with
434
+ `SemanticLogger.remove_appender(appender)`. The `:open_telemetry` appender's `#logger` and
435
+ `#logger=` likewise no longer return or replace the OpenTelemetry logger.
436
+
437
+ A subclass of `SemanticLogger::Appender::Wrapper` that writes through `logger` or `@logger` needs
438
+ to use `wrapped_logger` instead. `logger` is now the internal logger, which by default writes only
439
+ warnings and errors, to standard error, so the subclass's entries would be left out, and `@logger`
440
+ is not set until `logger` is first called.
441
+ * The `:statsd` and `:new_relic` metric subscribers now record a metric that has `dimensions` under
442
+ its name alone, leaving out the dimensions, which their clients do not accept. Before, they left
443
+ such a metric out, so a counter or timer logged with dimensions can now show higher counts there.
444
+ To leave them out again, add either subscriber with a filter:
445
+
446
+ ~~~ruby
447
+ SemanticLogger.add_appender(metric: :statsd, filter: ->(log) { !log.dimensions? })
448
+ SemanticLogger.add_appender(metric: :new_relic, filter: ->(log) { !log.dimensions? })
449
+ ~~~
450
+ * A `measure_*` call now keeps its `dimensions:` option, which it used to drop. So the `:signalfx`
451
+ metric subscriber sends a metric measured with dimensions as it sends any metric with dimensions:
452
+ under its own name, such as `supplier.add_user` for `metric: "supplier/add_user"`, with those
453
+ dimensions, and without a count. Before, it sent it as `Application.average` and
454
+ `Application.counter`, with `class` and `action` dimensions taken from the metric name. It also
455
+ no longer adds the named tags listed in the subscriber's own `dimensions:` option, which it only
456
+ adds to a metric without dimensions, so put any of those values the metric needs in its
457
+ `dimensions:`, such as `dimensions: {supplier: "acme", region: "eu-west"}`. A filter on
458
+ `log.dimensions?`, as above, now applies to it too.
459
+
460
+ Empty dimensions, such as `dimensions: {}` or `{supplier: nil}.compact`, and dimensions whose
461
+ values are all blank (`nil`, or text that is empty or only whitespace), such as `{supplier: nil}`
462
+ or `{state: ""}`, now count as none for `:signalfx`, on a log call too: the metric is sent as it
463
+ would be without them, so a duration is sent as `Application.average` and `Application.counter`,
464
+ and a count as `Application.counter`, each with the `class` and `action` dimensions taken from the
465
+ metric name and the named tags listed in the subscriber's `dimensions:` option. Before, either was
466
+ sent under its own name, without those dimensions, and a duration without a count. A value of only
467
+ whitespace, which was sent before, is now also left out of dimensions that have other values, and
468
+ of the named tags it forwards. `log.dimensions?`, and so a filter on it, as above, still counts
469
+ dimensions whose values are all blank, such as `{state: ""}`, as dimensions; only empty ones, such
470
+ as `{}`, count as none.
471
+ * A `tagged` block now restores the tags and named tags that it found when it exits, instead of
472
+ popping the ones it pushed, as `silence` and `with_level` blocks already did. This only changes code
473
+ that pushes or pops tags with `push_tags` and `pop_tags`, or `push_named_tags` and
474
+ `pop_named_tags`, across the edge of a block:
475
+ * Tags pushed inside a block, and not popped, are removed when the block exits. Before, the block
476
+ popped them in place of its own tags, which then stayed on the thread.
477
+ * Tags pushed before a block, and popped inside it, come back when the block exits.
478
+
479
+ Push and pop tags in matching pairs, or use the block form.
480
+ * `log_exception: :partial`, the default for `measure_*` blocks, now also adds the class of each
481
+ exception that caused the logged one to the message, and its message when the text does not
482
+ already hold it, as `log_exception: :full` already wrote them:
483
+
484
+ ```
485
+ # Before:
486
+ Saving order -- Exception: RuntimeError: Could not save
487
+ # After:
488
+ Saving order -- Exception: RuntimeError: Could not save -- Cause: IOError: disk full
489
+ ```
490
+
491
+ Alerts, searches, or tests that match the whole message of such an entry need updating.
492
+ * An exception whose class overrides `self.name` or `self.to_s` is now named by the constant the
493
+ class was assigned to, as Ruby names it in `Exception#inspect`, wherever Semantic Logger writes the
494
+ class. Before, what `self.name` returned was written as `name` in `:raw`, and so in `:json`,
495
+ `:logfmt`, `:fluentd`, `:ecs`, `:syslog_cee`, and `:new_relic_logs`, as `exception_name` in
496
+ `:loki`, and in each `Cause:` line of a backtrace, while the text formatters (`:default`,
497
+ `:color`, `:one_line`, `:pattern`, and `:syslog`) and `log_exception: :partial` wrote what
498
+ `self.to_s` returned. A class without a constant is named as Ruby writes the class, such as
499
+ `#<Class:0x000...>`, even when it gives itself a name. Update any search, alert, or dashboard that
500
+ selects such an exception by the name its class gave itself:
501
+
502
+ ```ruby
503
+ class DeclinedError < StandardError
504
+ def self.name = "Payments::Declined"
505
+ def self.to_s = name
506
+ end
507
+ # Before, in :json: "exception":{"name":"Payments::Declined",...}
508
+ # in :default: -- Exception: Payments::Declined: Card declined
509
+ # After, in :json: "exception":{"name":"DeclinedError",...}
510
+ # in :default: -- Exception: DeclinedError: Card declined
511
+ ```
512
+ * `:default`, `:color`, `:pattern`, and `:syslog` with `escape_control_chars: false` now always
513
+ write UTF-8, as the JSON based formatters, `:one_line`, and `:syslog` with its default
514
+ `escape_control_chars: true` already did. Text in another encoding, such as ISO-8859-1, used to be
515
+ written in that encoding, and an entry that mixed it with UTF-8 text was lost; it is now converted
516
+ to UTF-8, and so are a pattern or time format in another encoding. Binary text is read as the
517
+ UTF-8 it usually holds, without its bytes that are not valid UTF-8, instead of being written
518
+ byte for byte. An application whose log files, or the tools that read them, expect another
519
+ encoding needs to expect UTF-8 instead.
520
+ * `escape_control_chars: true`, which the `:syslog` formatter and the syslog appender use by
521
+ default, now also escapes the Unicode characters that control how text is shown, as `\uXXXX`: the
522
+ C1 control characters (U+0080 to U+009F), the line and paragraph separators (U+2028, U+2029), and
523
+ the bidirectional controls (U+202A to U+202E, U+2066 to U+2069). Searches or parsers that match
524
+ logged text holding them, such as localized text wrapped in bidirectional isolates, need to match
525
+ the escaped form instead.
526
+ * The `:new_relic` metric subscriber now adds `Custom/` to a metric name that starts with `Custom`
527
+ but not with `Custom/`, recording `Customers/signup` as `Custom/Customers/signup`, and
528
+ `CustomerSignup` as `Custom/CustomerSignup`. Before, it recorded such a name as it was, which New
529
+ Relic does not chart as a custom metric. With another `prefix:`, the same applies to a name that
530
+ starts with that prefix but not with it and a `/`. A prefix given with a trailing `/` no longer
531
+ doubles it: with `prefix: "Custom/"`, `Orders/placed` is recorded as `Custom/Orders/placed`, not
532
+ `Custom//Orders/placed`. Dashboards, alerts, and NRQL queries on an old name stop receiving data,
533
+ so update them to the new one.
534
+ * `SemanticLogger.reopen` within the same process, such as `reopen(force: true)` after a log
535
+ rotation, keeps the log entries that are still queued, and the thread that writes them, rather
536
+ than replacing both and losing the entries. So it no longer stops a thread that is waiting
537
+ between retries after an appender failed, or one that is stuck writing; it only starts the thread
538
+ again once it has stopped. After a fork, the child still starts with an empty queue.
539
+ * The `:http`, `:elasticsearch_http`, `:splunk_http`, and `:loki` appenders, and the `:signalfx`
540
+ metric subscriber, now raise an error from `#log` and `#batch` when the server answers with a status
541
+ other than 2xx, instead of reporting it and carrying on. With `async: true` or `batch: true`, the
542
+ entries the server did not accept are now counted as `failed` in `SemanticLogger.stats`, rather
543
+ than `processed`. On a 408, 429, or 503, the worker thread now backs off (1 second, then 2, ...),
544
+ then sends the same entry or batch again, up to 3 times, before it discards it and carries on with
545
+ the next one. So does a request it could not send because it could not connect, which was
546
+ discarded before, except that such a failure does not count towards the 3 times, since the server
547
+ never answered. On any other 5xx, such as a 500, 502, or 504, or a redirect, it discards the entry
548
+ or batch, then backs off before sending the next one. It stops for good after `async_max_retries`
549
+ (default `100`) failures in a row. During an outage the entries logged meanwhile now wait on the
550
+ queue while the worker thread backs off, rather than being sent, and discarded, one per back-off,
551
+ so with a capped queue (the default) and without `non_blocking`, logging waits once the queue is
552
+ full until the server is back or the thread gives up. Pass `async_max_retries: -1` to keep trying
553
+ through a long outage. On a 400, 413, or 422, a batch is now
554
+ split up and sent again in parts, to find the entries the server rejects, so a server that kept
555
+ some entries of a request it rejected can store them twice. On any other 4xx, such as 401 or 404,
556
+ which the server answers whatever the request holds, the whole batch is discarded at once. See
557
+ [When the server does not accept a request](appenders.html#when-the-server-does-not-accept-a-request).
558
+
559
+ While the worker thread backs off, the entries logged meanwhile wait on the appender's queue, as
560
+ they already did while the server could not be reached. Once that queue, and then the main queue,
561
+ are full, logging calls block until the server accepts requests again. Add an HTTP appender that
562
+ uses `async: true` or `batch: true` with `non_blocking: true` as well, so that its entries are
563
+ dropped once its queue is full instead; see
564
+ [Drop messages instead of blocking](operations.html#drop-messages-instead-of-blocking).
565
+
566
+ Without `async:` or `batch:`, each failure is still reported to the internal logger, now as
567
+ `Failed to log to appender`, with the error and its backtrace. A subclass of
568
+ `SemanticLogger::Appender::Http` that checks the result of `post` or `put` now gets a
569
+ `SemanticLogger::Appender::Http::ResponseError` raised instead of `false`.
570
+ * A flush or close of an appender added with `async: true` or in batches no longer waits out the
571
+ back-offs of its worker thread during an outage, which could keep the flush at exit waiting for
572
+ minutes: 1 second for the first entry queued, 2 for the next, and so on. It ends the back-off
573
+ under way, and until it has run, an entry or batch that the destination fails is discarded,
574
+ counted as `failed`, rather than sent again after a back-off, and so are the entries queued after
575
+ it, ahead of the flush or close, without being sent (see the next note). A custom appender can now
576
+ say that a write that raised wrote nothing, from `#wrote_nothing?(exception)`, so that the worker
577
+ thread writes it again after the back-off, up to 3 times once the destination answered it, and
578
+ that the write never reached the destination, from `#not_sent?(exception)`, which is not counted;
579
+ see
580
+ [Custom appenders](config.html#custom-appenders). A subclass of `SemanticLogger::Appender::Http`
581
+ whose `#log` or `#batch` sends more than one request inherits a `#wrote_nothing?` that answers
582
+ `true` for a 503 or a refused connection from any of them, which sends the entries of the earlier
583
+ requests a second time: override it to return `false` for a failure after the first request.
584
+ * During an outage of the destination of an appender added with `async: true` or in batches, a flush
585
+ or close, including the flush at exit, now discards that appender's entries queued ahead of it once
586
+ a write fails, counted as `failed` in `SemanticLogger.stats`, rather than send each one in turn.
587
+ Before, it waited while each one failed, up to the `open_timeout` of an HTTP based appender
588
+ whose server does not answer, so that a process with 10,000 entries queued could take hours to
589
+ exit. At exit, each such appender is also told that the process is exiting before the flush, so
590
+ that it stops backing off even while the thread that writes to every appender is held up handing
591
+ it an entry. The entries queued after a flush, and every entry to a destination that takes them,
592
+ are written as before. The entries discarded are reported to the internal logger, as for any entry
593
+ that could not be written.
594
+ * During an outage of the destination of an appender added without `async: true`, and not in
595
+ batches, such as a plain `:http`, `:splunk_http`, `:tcp`, or `:udp`, the process now exits within
596
+ seconds too. Once a write to such an appender fails because of its destination while the process
597
+ exits, it is given no more of the entries queued ahead of the flush at exit, or logged meanwhile,
598
+ and the number it was not given is reported to the internal logger, as
599
+ `Discarded ... log message(s) during the flush at exit, after the destination failed`. The flush
600
+ at exit still flushes it, so that it delivers the entries it already accepted. Before, the
601
+ flush at exit waited while each one failed in turn, up to the `read_timeout` or `open_timeout` of
602
+ an HTTP based appender each. The other appenders still get every entry, and while the process
603
+ runs, such an appender is still given every entry. See
604
+ [Retry a failing appender](operations.html#retry-a-failing-appender).
605
+ * The `:elasticsearch` and `:opensearch` appenders now send again the entries of a bulk request that
606
+ the server rejected with status 429, because its write queue was full, up to 3 times, after waiting
607
+ 0.2, 0.4, then 0.8 seconds. Before, they were reported and discarded straight away. That holds up
608
+ the thread that writes the entries for waits of at most 1.4 seconds for a request, plus up to 3
609
+ more bulk requests. It is the appender's own thread when it writes in batches, as it does by
610
+ default, or with `async: true`. With `batch: false`, and without `async: true`, it is the thread
611
+ that writes to every appender, or with `SemanticLogger.sync!`, the thread that logged, and every
612
+ other thread that logs waits for it. The report of the entries still not indexed now comes after
613
+ those waits. See [Elasticsearch](appenders.html#elasticsearch).
614
+ * `SemanticLogger.reopen` now only asks for the appenders to be reopened, each by the thread that
615
+ writes its entries, before it writes the next one. With `SemanticLogger.sync!` on, that is still
616
+ the calling thread for every appender without a thread of its own (added without `async: true` or
617
+ batching). Before, it
618
+ reopened them on the calling thread, often a signal trap after a log rotation, while the thread
619
+ that writes the entries could be writing to them: the file appender could close its file under an
620
+ entry being written, and the HTTP appenders could end a connection under a request, or, with some
621
+ versions of the timeout gem, fail with `ThreadError` in a trap. So `reopen` now returns before the
622
+ appenders have reopened; call `SemanticLogger.flush` after it to wait until they have, for example
623
+ in a test. An entry still queued when `reopen` is called can now be written to the new file. The
624
+ error from an appender that cannot be reopened is reported by that thread, rather than raised from
625
+ the proxy of an appender added with `async: true` or in batches, which now reports it to the
626
+ appender's own internal logger, as `Async: Failed to reopen the appender`.
627
+ * A block given to `add_appender` with an appender or metric subscriber named by a Symbol, such as
628
+ `add_appender(appender: :http, url: ...) { |log| log.message }`, now formats its entries, taking
629
+ precedence over `formatter:`, as it does for `io:`, `file_name:`, and `logger:`. v5.2 ignored it
630
+ and used the appender's `formatter:` or its default formatter, so what such an appender sends
631
+ changes. With `batch: true`, `:http`, `:loki`, and `metric: :signalfx` now raise `ArgumentError`
632
+ for it (see [Batch an appender with a formatter that can format a batch](#batch-an-appender-with-a-formatter-that-can-format-a-batch)),
633
+ where v5.2 sent batches formatted by the default formatter. Remove the block to keep the output
634
+ of v5.2.
635
+ * Some appenders now check their settings as they are added, and raise there, rather than failing
636
+ each entry later:
637
+ * `add_appender(metric: :statsd, url: ...)` connects as it is added, so it raises when its `url:`
638
+ is not `udp://` or its host cannot be resolved, where v5.2 added it and then failed every
639
+ metric. Add it only where the Statsd host resolves.
640
+ * `add_appender(appender: :udp, server: ...)` raises `ArgumentError` for a `server:` that is not
641
+ `"host:port"`, such as one without a port. Write an IPv6 address in brackets, as
642
+ `"[::1]:8088"`.
643
+ * `SemanticLogger::Processor.logger=` (and `SemanticLogger::SyncProcessor.logger=`) now raises
644
+ `ArgumentError` when given a `SemanticLogger::Logger`, whose reports of a failing appender went
645
+ back to that appender, failed again, and were reported again. Give it an appender, such as
646
+ `SemanticLogger::Appender::IO.new($stderr)`, or a Ruby or Rails `Logger`, which it now accepts.
647
+ * `SemanticLogger.remove_appender` now waits, as `SemanticLogger.flush` does, until the messages
648
+ already logged are written and the appenders flushed, before it closes the appender, so it can
649
+ take as long as a flush. Called on the thread of an appender added with `async: true` or in
650
+ batches, it returns `true` at once, and the appender is closed shortly after. See
651
+ [Adding and removing appenders](config.html#adding-and-removing-appenders).
652
+ * The `:open_telemetry` and `:new_relic_logs` appenders now keep the trace context they capture
653
+ with `Log#capture`, which formatters do not write, instead of in `log.context`. A custom formatter
654
+ or appender that read `log.context[:open_telemetry]` or `log.context[:new_relic_metadata]` reads
655
+ `log.captured(:open_telemetry)` or `log.captured(:new_relic_metadata)` instead, and an `on_log`
656
+ subscriber that set either key with `log.set_context` sets it with `log.capture`.
657
+
15
658
  ### Upgrading to Semantic Logger v5.2
16
659
 
17
660
  v5.2 sends the configured facility to remote syslog, deprecates the `:splunk` appender, stops