semantic_logger 5.2.0 → 5.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (98) hide show
  1. checksums.yaml +4 -4
  2. data/docs/api.md +160 -9
  3. data/docs/appenders.md +461 -34
  4. data/docs/config.md +543 -36
  5. data/docs/log.md +38 -7
  6. data/docs/metrics.md +139 -28
  7. data/docs/operations.md +293 -24
  8. data/docs/rails.md +319 -5
  9. data/docs/security.md +135 -107
  10. data/docs/testing.md +23 -0
  11. data/docs/upgrading.md +644 -1
  12. data/lib/semantic_logger/appender/async.rb +29 -11
  13. data/lib/semantic_logger/appender/bugsnag.rb +7 -2
  14. data/lib/semantic_logger/appender/cloudwatch_logs.rb +46 -6
  15. data/lib/semantic_logger/appender/elasticsearch_base.rb +197 -29
  16. data/lib/semantic_logger/appender/elasticsearch_http.rb +22 -5
  17. data/lib/semantic_logger/appender/file.rb +29 -14
  18. data/lib/semantic_logger/appender/graylog.rb +81 -7
  19. data/lib/semantic_logger/appender/honeybadger.rb +7 -1
  20. data/lib/semantic_logger/appender/honeybadger_insights.rb +8 -2
  21. data/lib/semantic_logger/appender/http.rb +244 -41
  22. data/lib/semantic_logger/appender/io.rb +11 -6
  23. data/lib/semantic_logger/appender/kafka.rb +192 -46
  24. data/lib/semantic_logger/appender/loki.rb +42 -6
  25. data/lib/semantic_logger/appender/mongodb.rb +94 -9
  26. data/lib/semantic_logger/appender/new_relic.rb +2 -3
  27. data/lib/semantic_logger/appender/new_relic_logs.rb +30 -8
  28. data/lib/semantic_logger/appender/open_telemetry.rb +37 -35
  29. data/lib/semantic_logger/appender/rabbitmq.rb +241 -17
  30. data/lib/semantic_logger/appender/sentry.rb +2 -3
  31. data/lib/semantic_logger/appender/sentry_ruby.rb +82 -15
  32. data/lib/semantic_logger/appender/splunk.rb +4 -4
  33. data/lib/semantic_logger/appender/splunk_http.rb +92 -1
  34. data/lib/semantic_logger/appender/syslog.rb +89 -10
  35. data/lib/semantic_logger/appender/tcp.rb +68 -98
  36. data/lib/semantic_logger/appender/text_output.rb +65 -0
  37. data/lib/semantic_logger/appender/udp.rb +130 -18
  38. data/lib/semantic_logger/appender/wrapper.rb +10 -7
  39. data/lib/semantic_logger/appender.rb +115 -10
  40. data/lib/semantic_logger/appenders.rb +136 -4
  41. data/lib/semantic_logger/backtrace_cleaning.rb +77 -0
  42. data/lib/semantic_logger/base.rb +53 -71
  43. data/lib/semantic_logger/call_site.rb +235 -0
  44. data/lib/semantic_logger/context.rb +222 -0
  45. data/lib/semantic_logger/core_ext/process.rb +20 -5
  46. data/lib/semantic_logger/core_ext/warning.rb +70 -0
  47. data/lib/semantic_logger/deprecation.rb +139 -0
  48. data/lib/semantic_logger/exception_details.rb +4 -3
  49. data/lib/semantic_logger/formatters/base.rb +139 -19
  50. data/lib/semantic_logger/formatters/color.rb +3 -4
  51. data/lib/semantic_logger/formatters/default.rb +35 -7
  52. data/lib/semantic_logger/formatters/ecs.rb +9 -4
  53. data/lib/semantic_logger/formatters/json.rb +9 -0
  54. data/lib/semantic_logger/formatters/logfmt.rb +9 -7
  55. data/lib/semantic_logger/formatters/loki/labels.rb +282 -0
  56. data/lib/semantic_logger/formatters/loki.rb +412 -24
  57. data/lib/semantic_logger/formatters/new_relic_logs.rb +15 -13
  58. data/lib/semantic_logger/formatters/one_line.rb +44 -10
  59. data/lib/semantic_logger/formatters/open_telemetry.rb +186 -29
  60. data/lib/semantic_logger/formatters/pattern.rb +60 -8
  61. data/lib/semantic_logger/formatters/raw.rb +38 -8
  62. data/lib/semantic_logger/formatters/signalfx.rb +55 -15
  63. data/lib/semantic_logger/formatters/syslog.rb +4 -2
  64. data/lib/semantic_logger/formatters/syslog_cee.rb +10 -4
  65. data/lib/semantic_logger/formatters/syslog_header.rb +49 -0
  66. data/lib/semantic_logger/formatters.rb +1 -0
  67. data/lib/semantic_logger/log.rb +236 -80
  68. data/lib/semantic_logger/loggable.rb +55 -95
  69. data/lib/semantic_logger/logger.rb +31 -5
  70. data/lib/semantic_logger/metric/new_relic.rb +32 -17
  71. data/lib/semantic_logger/metric/signalfx.rb +21 -12
  72. data/lib/semantic_logger/metric/statsd.rb +26 -8
  73. data/lib/semantic_logger/processor.rb +194 -5
  74. data/lib/semantic_logger/queue_processor/backoff.rb +140 -0
  75. data/lib/semantic_logger/queue_processor/batch_failure.rb +69 -0
  76. data/lib/semantic_logger/queue_processor/held_write.rb +62 -0
  77. data/lib/semantic_logger/queue_processor/inbox.rb +97 -0
  78. data/lib/semantic_logger/queue_processor/reopener.rb +56 -0
  79. data/lib/semantic_logger/queue_processor/reporter.rb +113 -0
  80. data/lib/semantic_logger/queue_processor/request.rb +48 -10
  81. data/lib/semantic_logger/queue_processor/throttled_counter.rb +60 -17
  82. data/lib/semantic_logger/queue_processor.rb +426 -176
  83. data/lib/semantic_logger/ractor_copy.rb +120 -0
  84. data/lib/semantic_logger/ractor_processor.rb +82 -0
  85. data/lib/semantic_logger/ractor_receiver.rb +282 -0
  86. data/lib/semantic_logger/ractor_support.rb +28 -0
  87. data/lib/semantic_logger/redactor.rb +310 -0
  88. data/lib/semantic_logger/semantic_logger.rb +313 -79
  89. data/lib/semantic_logger/subscriber.rb +169 -15
  90. data/lib/semantic_logger/sync_processor.rb +226 -26
  91. data/lib/semantic_logger/test/capture_log_events.rb +20 -2
  92. data/lib/semantic_logger/test/minitest.rb +2 -1
  93. data/lib/semantic_logger/test/rspec.rb +1 -1
  94. data/lib/semantic_logger/trap_safe.rb +113 -0
  95. data/lib/semantic_logger/utils.rb +266 -79
  96. data/lib/semantic_logger/version.rb +1 -1
  97. data/lib/semantic_logger.rb +35 -2
  98. metadata +22 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 5b42624348f8dc147462cc1f2d26f7cf0d4ef3df91fe3157a119591e8f6bd23b
4
- data.tar.gz: ca1f00571b5280a9cd5b4bf0f116944ba4850eb1536b50aff53bfba9b6ba540b
3
+ metadata.gz: bfb8a2265081eac50cc50aa7044a5d51b277fba016cb11eb76166d0c9216fe2b
4
+ data.tar.gz: 678e62939127794a93e02280e9e0ed898af39a6015118bdba1f652a65abc4a2f
5
5
  SHA512:
6
- metadata.gz: 31f5b285c287b386979f89b794d71f9084276b13510dc5b7798a510815f512e713bdf44a87743808890a8f4cef1e74a8263a8ca78edba8f7247235bf53aa6fd5
7
- data.tar.gz: 3f114bc7d9352eb40c1a4e8a31ecfa13b20e0bfec61c72fceaba522ffc041c97ffe50e0f788b63415c4265ac68586a185473a7c4320d59e8f708ac8facedbd72
6
+ metadata.gz: a32ccdf8c3801b3e5166c6d3be83d9c8423defbd2b65e50dc11c262f7cab8c1b0b864a19e90ba9415f2e66d0548a45c0979cb42a03e93252f1381c87634289bc
7
+ data.tar.gz: adab42f9b9e25495ea3a9b99c3322a30a95e4e7eb767f85d27a147831f30861053f08fe22d33318b77c805bf70d301e6c2adcff869cf4ea002607140b68c5ea8
data/docs/api.md CHANGED
@@ -67,6 +67,30 @@ Every entry is identified as coming from `Supplier`:
67
67
  2012-08-30 15:37:29.474 I [48308:main] Supplier -- Accessible from instance methods
68
68
  ~~~
69
69
 
70
+ A module can include `SemanticLogger::Loggable` too. The module gets a `logger` of its own, and its
71
+ instance methods get a `logger` in any class that includes the module, whether or not that class
72
+ includes `SemanticLogger::Loggable`:
73
+
74
+ ~~~ruby
75
+ module Pricing
76
+ include SemanticLogger::Loggable
77
+
78
+ def price(amount)
79
+ logger.info("Pricing", amount: amount)
80
+ end
81
+ end
82
+
83
+ class Order
84
+ include Pricing
85
+ end
86
+
87
+ Order.new.price(10)
88
+ ~~~
89
+
90
+ The entry is identified as coming from `Order`, the class that called `price`. When that class has a
91
+ class level `logger`, such as one from `SemanticLogger::Loggable`, the module's methods use it instead.
92
+ When the class itself defines a `logger` instance method, they use that.
93
+
70
94
  By default `SemanticLogger[...]` returns a brand new logger on each call. To share a single logger
71
95
  per class instead, enable [logger caching](config.html#caching-loggers).
72
96
 
@@ -176,6 +200,13 @@ logger.error(message: "Calling Supplier", duration: 100)
176
200
 
177
201
  # Attach a metric (see Step 6)
178
202
  logger.error(message: "Calling Supplier", metric: "Supplier/inquiry", metric_amount: 21)
203
+
204
+ # Attach a backtrace of your own, an Array of Strings, instead of the logger's call stack
205
+ logger.warn(message: "Open redirect detected", backtrace: redirect_stack)
206
+
207
+ # Say where the entry was logged, or that it has no such place (see call_site_level)
208
+ logger.debug(message: "Order Load", duration: 0.4, call_site: query_location)
209
+ logger.info(message: "Started GET /orders", call_site: false)
179
210
  ~~~
180
211
 
181
212
  In both forms, a key such as `duration` or `exception` is only used as an argument when its value is
@@ -183,6 +214,11 @@ valid for it; otherwise it is logged as part of the payload. Any other keys supp
183
214
  `payload:` are merged into it, without overwriting its own values. When the Hash holds untrusted
184
215
  data, see [Logging untrusted hashes](security.html#logging-untrusted-hashes).
185
216
 
217
+ A block can return the same Hash, but it never replaces an argument the entry already has. That
218
+ includes the backtrace that an entry captures when its level is at or above
219
+ `SemanticLogger.backtrace_level`, so to attach a backtrace of your own, pass `backtrace:` with the
220
+ call rather than returning it from the block.
221
+
186
222
  ## Step 6: Measure how long something takes
187
223
 
188
224
  It is good practice to "measure everything" in production, so that when things slow down it is
@@ -236,6 +272,17 @@ logger.measure_info("Called external interface", metric: "Supplier/inquiry") do
236
272
  end
237
273
  ~~~
238
274
 
275
+ Add `dimensions` to break the metric down, such as by supplier. See
276
+ [Break a metric down with dimensions](metrics.html#step-3-break-a-metric-down-with-dimensions):
277
+
278
+ ~~~ruby
279
+ logger.measure_info("Called external interface",
280
+ metric: "Supplier/inquiry",
281
+ dimensions: {supplier: "acme"}) do
282
+ # ...
283
+ end
284
+ ~~~
285
+
239
286
  ### Supply the duration yourself
240
287
 
241
288
  When you already have a duration, log it without a block. This keeps the duration as structured data
@@ -255,16 +302,26 @@ The second argument to a measure call is a Hash of options:
255
302
  - `:min_duration` [Float]: only log if the block takes longer than this many milliseconds. Default
256
303
  `0.0` (always log).
257
304
  - `:metric` [String]: notify metric subscribers with this metric name.
305
+ - `:dimensions` [Hash]: key/value labels that break the metric down, such as `{supplier: "acme"}`.
258
306
  - `:payload` [Hash]: an optional payload to log with the entry.
259
307
  - `:exception` [Exception]: an exception to log along with the duration.
260
308
  - `:duration` [Float]: the duration in ms, used when no block is supplied (then it is mandatory; with
261
309
  a block it is ignored).
262
310
  - `:log_exception` [Symbol]: how to report an exception raised in the block. `:full` logs the class,
263
- message, and backtrace; `:partial` logs the class and message only; `:none` does not log it.
264
- Default `:partial`.
311
+ message, and backtrace of the exception and of each exception that caused it; `:partial` adds their
312
+ classes and messages to the log message, without backtraces; `:none` does not log it. Default
313
+ `:partial`. With `:partial`, a cause's message is left out when it is empty, is its class name, or
314
+ is already in the log message, as when the exception that wraps it copied it into its own:
315
+
316
+ ```
317
+ Rendering -- Exception: ActionDispatch::Http::MimeNegotiation::InvalidType: "foo/bar" is not a valid MIME type -- Cause: Mime::Type::InvalidMimeType
318
+ ```
265
319
  - `:on_exception_level` [Symbol]: if an exception is raised, raise the log level to this level.
266
- - `:silence` [Symbol]: the level to silence other log messages to within the block (current thread
320
+ - `:silence` [Symbol]: the level to silence other log messages to within the block (current fiber
267
321
  only).
322
+ - `:call_site` [Thread::Backtrace::Location|SemanticLogger::CallSite|false]: where the entry was
323
+ logged, or `false` for none. See
324
+ [Recording where each entry was logged](config.html#recording-where-each-entry-was-logged).
268
325
 
269
326
  An option with a value that is not valid for it, such as a non-numeric `:min_duration` or an unknown
270
327
  level, is ignored and its default is used. Other keys in the Hash are not logged; pass the data to
@@ -303,13 +360,60 @@ SemanticLogger.tagged(user: "Jack", zip_code: 12345) do
303
360
  end
304
361
  ~~~
305
362
 
306
- Tags are scoped to the current thread, so a new thread started inside the block does not inherit them.
307
- [Parallel Minion](https://github.com/reidmorrison/parallel_minion) creates threads that copy the tags
308
- across automatically.
363
+ Tags belong to the current fiber: each thread has its own, and so does each fiber of a fiber
364
+ scheduler such as [Async](https://github.com/socketry/async), which runs every request of a
365
+ [Falcon](https://github.com/socketry/falcon) server in its own fiber. So concurrent requests never
366
+ see each other's tags, and a thread, fiber, or Async task started inside the block does not have
367
+ them either. Hand them over explicitly, as shown next.
368
+
369
+ ### Hand tags to another thread or fiber
370
+
371
+ `SemanticLogger.context` returns a frozen snapshot of the current tags, named tags, and
372
+ [silence](#silence-noisy-code) level. `SemanticLogger.with_context` applies it in another thread or
373
+ fiber for the duration of a block:
374
+
375
+ ~~~ruby
376
+ SemanticLogger.tagged(request_id: request.uuid) do
377
+ context = SemanticLogger.context
378
+
379
+ Thread.new do
380
+ SemanticLogger.with_context(context) do
381
+ logger.info("Generating report") # carries request_id
382
+ end
383
+ end
384
+ end
385
+ ~~~
386
+
387
+ The same works for an Async task or a thread pool:
388
+
389
+ ~~~ruby
390
+ context = SemanticLogger.context
391
+ barrier = Async::Barrier.new
392
+ urls.each do |url|
393
+ barrier.async do
394
+ SemanticLogger.with_context(context) { fetch(url) }
395
+ end
396
+ end
397
+ barrier.wait
398
+
399
+ context = SemanticLogger.context
400
+ pool.post do
401
+ SemanticLogger.with_context(context) { process(item) }
402
+ end
403
+ ~~~
404
+
405
+ Within the block, the thread or fiber has exactly the tags, named tags, and silence level of the
406
+ snapshot, in place of its own, which come back when the block exits. Add more tags inside the block
407
+ as usual. The snapshot does not carry levels set with a logger's `with_level`, which, like Ruby's own
408
+ `Logger#with_level`, apply only to the fiber that set them.
409
+
410
+ Tags are not handed over automatically because a long-lived thread, such as a thread pool's worker,
411
+ would otherwise keep the tags of whichever request happened to start it, and log them for every
412
+ later request it serves.
309
413
 
310
414
  ### Bind tags to one logger (child loggers)
311
415
 
312
- The block form above scopes tags to the thread. Sometimes it is more convenient to bind tags to a
416
+ The block form above scopes tags to the fiber. Sometimes it is more convenient to bind tags to a
313
417
  single logger instance, for example when the logger belongs to an object with its own identity (an
314
418
  ActiveRecord model or a background job).
315
419
 
@@ -400,6 +504,52 @@ On JRuby, naming a thread also names the underlying JVM thread, which shows up w
400
504
  JVM with tools such as jconsole or jstack. JRuby names the main thread `main`, and a thread created
401
505
  by Java code, such as a servlet container's request thread, is logged with its JVM thread name.
402
506
 
507
+ ### Log from Ractors (experimental)
508
+
509
+ On Ruby 4.0 and later, code running in a [Ractor](https://docs.ruby-lang.org/en/4.0/Ractor.html)
510
+ logs as usual, and the entries are written by the appenders added in the main Ractor. Like Ruby's
511
+ own Ractor API, this is experimental, and may change in a minor release.
512
+
513
+ ~~~ruby
514
+ SemanticLogger.add_appender(io: $stdout, formatter: :color)
515
+
516
+ context = SemanticLogger.tagged(job_id: 42) { SemanticLogger.context }
517
+ worker = Ractor.new(context, name: "worker") do |job_context|
518
+ SemanticLogger.with_context(job_context) do
519
+ logger = SemanticLogger["Importer"]
520
+ logger.info("Importing", rows: 1_000) # carries job_id: 42
521
+ end
522
+ end
523
+ worker.join
524
+ ~~~
525
+
526
+ Each entry is sent to the main Ractor, where a thread hands it to the appenders, so a Ractor never
527
+ waits on an appender. The thread name of an entry starts with the Ractor's name, or its number, as in
528
+ `worker/12345` or `ractor-3/12345`.
529
+
530
+ Things to know:
531
+
532
+ - A new Ractor does not have the tags of the code that created it. Hand them over with
533
+ [`SemanticLogger.context`](#hand-tags-to-another-thread-or-fiber), as shown above.
534
+ - Add appenders, and change any setting, such as `SemanticLogger.default_level`, in the main Ractor.
535
+ Doing so in another Ractor raises `Ractor::IsolationError`.
536
+ - Each Ractor has its own loggers: a `Loggable` class has a logger of its own in each Ractor, and so
537
+ does a class with `SemanticLogger.cache_loggers` enabled. Setting a level on the main Ractor's logger
538
+ therefore does not change the level of a Ractor's logger; the global default level applies to both.
539
+ - `on_log` subscribers run for an entry logged in a Ractor only when they are shareable, such as a
540
+ module with a `call` method, or a block made with `Ractor.shareable_proc`.
541
+ - An entry is copied to the main Ractor. A value that Ruby cannot copy, such as a Proc in the payload,
542
+ is written as its `inspect`, and so is such a value held by a logged exception. The extra detail
543
+ that Ruby adds to some exceptions' messages, such as did_you_mean's "Did you mean?" suggestions, is
544
+ not written for an exception logged in a Ractor.
545
+ - `SemanticLogger.flush` in a Ractor waits until the main Ractor has written that Ractor's entries.
546
+ - The main Ractor accepts entries from other Ractors as fast as they are logged, so logging never
547
+ slows a Ractor down, however busy the appenders are. `max_queue_size` and `non_blocking` then apply
548
+ to the entries as usual, once the receiving thread hands them to the processor. In sync mode, too,
549
+ the receiving thread writes them.
550
+ - On Ruby 3.4 and earlier, logging from a Ractor other than the main one raises
551
+ `Ractor::IsolationError`, as it always has.
552
+
403
553
  ### Change one class's level at runtime
404
554
 
405
555
  Because each class has its own logger, you can change one class's level on the fly, for example to
@@ -421,7 +571,7 @@ To change the global default level for every logger that has not been set explic
421
571
 
422
572
  ### Silence noisy code
423
573
 
424
- `silence` raises the level within a block, on the current thread only, to quiet a noisy section:
574
+ `silence` raises the level within a block, on the current fiber only, to quiet a noisy section:
425
575
 
426
576
  ~~~ruby
427
577
  # Within this block, log only :error and above
@@ -441,7 +591,8 @@ end
441
591
  ~~~
442
592
 
443
593
  `silence` has no effect on loggers whose level was set explicitly (those that do not follow the global
444
- default), and does not affect threads spawned inside the block.
594
+ default), and does not affect threads or fibers started inside the block, unless they are handed the
595
+ block's [context](#hand-tags-to-another-thread-or-fiber).
445
596
 
446
597
  ### Map a noisy gem's debug logs to trace
447
598