semantic_logger 5.2.0 → 5.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/docs/api.md +160 -9
- data/docs/appenders.md +461 -34
- data/docs/config.md +543 -36
- data/docs/log.md +38 -7
- data/docs/metrics.md +139 -28
- data/docs/operations.md +293 -24
- data/docs/rails.md +319 -5
- data/docs/security.md +135 -107
- data/docs/testing.md +23 -0
- data/docs/upgrading.md +644 -1
- data/lib/semantic_logger/appender/async.rb +29 -11
- data/lib/semantic_logger/appender/bugsnag.rb +7 -2
- data/lib/semantic_logger/appender/cloudwatch_logs.rb +46 -6
- data/lib/semantic_logger/appender/elasticsearch_base.rb +197 -29
- data/lib/semantic_logger/appender/elasticsearch_http.rb +22 -5
- data/lib/semantic_logger/appender/file.rb +29 -14
- data/lib/semantic_logger/appender/graylog.rb +81 -7
- data/lib/semantic_logger/appender/honeybadger.rb +7 -1
- data/lib/semantic_logger/appender/honeybadger_insights.rb +8 -2
- data/lib/semantic_logger/appender/http.rb +244 -41
- data/lib/semantic_logger/appender/io.rb +11 -6
- data/lib/semantic_logger/appender/kafka.rb +192 -46
- data/lib/semantic_logger/appender/loki.rb +42 -6
- data/lib/semantic_logger/appender/mongodb.rb +94 -9
- data/lib/semantic_logger/appender/new_relic.rb +2 -3
- data/lib/semantic_logger/appender/new_relic_logs.rb +30 -8
- data/lib/semantic_logger/appender/open_telemetry.rb +37 -35
- data/lib/semantic_logger/appender/rabbitmq.rb +241 -17
- data/lib/semantic_logger/appender/sentry.rb +2 -3
- data/lib/semantic_logger/appender/sentry_ruby.rb +82 -15
- data/lib/semantic_logger/appender/splunk.rb +4 -4
- data/lib/semantic_logger/appender/splunk_http.rb +92 -1
- data/lib/semantic_logger/appender/syslog.rb +89 -10
- data/lib/semantic_logger/appender/tcp.rb +68 -98
- data/lib/semantic_logger/appender/text_output.rb +65 -0
- data/lib/semantic_logger/appender/udp.rb +130 -18
- data/lib/semantic_logger/appender/wrapper.rb +10 -7
- data/lib/semantic_logger/appender.rb +115 -10
- data/lib/semantic_logger/appenders.rb +136 -4
- data/lib/semantic_logger/backtrace_cleaning.rb +77 -0
- data/lib/semantic_logger/base.rb +53 -71
- data/lib/semantic_logger/call_site.rb +235 -0
- data/lib/semantic_logger/context.rb +222 -0
- data/lib/semantic_logger/core_ext/process.rb +20 -5
- data/lib/semantic_logger/core_ext/warning.rb +70 -0
- data/lib/semantic_logger/deprecation.rb +139 -0
- data/lib/semantic_logger/exception_details.rb +4 -3
- data/lib/semantic_logger/formatters/base.rb +139 -19
- data/lib/semantic_logger/formatters/color.rb +3 -4
- data/lib/semantic_logger/formatters/default.rb +35 -7
- data/lib/semantic_logger/formatters/ecs.rb +9 -4
- data/lib/semantic_logger/formatters/json.rb +9 -0
- data/lib/semantic_logger/formatters/logfmt.rb +9 -7
- data/lib/semantic_logger/formatters/loki/labels.rb +282 -0
- data/lib/semantic_logger/formatters/loki.rb +412 -24
- data/lib/semantic_logger/formatters/new_relic_logs.rb +15 -13
- data/lib/semantic_logger/formatters/one_line.rb +44 -10
- data/lib/semantic_logger/formatters/open_telemetry.rb +186 -29
- data/lib/semantic_logger/formatters/pattern.rb +60 -8
- data/lib/semantic_logger/formatters/raw.rb +38 -8
- data/lib/semantic_logger/formatters/signalfx.rb +55 -15
- data/lib/semantic_logger/formatters/syslog.rb +4 -2
- data/lib/semantic_logger/formatters/syslog_cee.rb +10 -4
- data/lib/semantic_logger/formatters/syslog_header.rb +49 -0
- data/lib/semantic_logger/formatters.rb +1 -0
- data/lib/semantic_logger/log.rb +236 -80
- data/lib/semantic_logger/loggable.rb +55 -95
- data/lib/semantic_logger/logger.rb +31 -5
- data/lib/semantic_logger/metric/new_relic.rb +32 -17
- data/lib/semantic_logger/metric/signalfx.rb +21 -12
- data/lib/semantic_logger/metric/statsd.rb +26 -8
- data/lib/semantic_logger/processor.rb +194 -5
- data/lib/semantic_logger/queue_processor/backoff.rb +140 -0
- data/lib/semantic_logger/queue_processor/batch_failure.rb +69 -0
- data/lib/semantic_logger/queue_processor/held_write.rb +62 -0
- data/lib/semantic_logger/queue_processor/inbox.rb +97 -0
- data/lib/semantic_logger/queue_processor/reopener.rb +56 -0
- data/lib/semantic_logger/queue_processor/reporter.rb +113 -0
- data/lib/semantic_logger/queue_processor/request.rb +48 -10
- data/lib/semantic_logger/queue_processor/throttled_counter.rb +60 -17
- data/lib/semantic_logger/queue_processor.rb +426 -176
- data/lib/semantic_logger/ractor_copy.rb +120 -0
- data/lib/semantic_logger/ractor_processor.rb +82 -0
- data/lib/semantic_logger/ractor_receiver.rb +282 -0
- data/lib/semantic_logger/ractor_support.rb +28 -0
- data/lib/semantic_logger/redactor.rb +310 -0
- data/lib/semantic_logger/semantic_logger.rb +313 -79
- data/lib/semantic_logger/subscriber.rb +169 -15
- data/lib/semantic_logger/sync_processor.rb +226 -26
- data/lib/semantic_logger/test/capture_log_events.rb +20 -2
- data/lib/semantic_logger/test/minitest.rb +2 -1
- data/lib/semantic_logger/test/rspec.rb +1 -1
- data/lib/semantic_logger/trap_safe.rb +113 -0
- data/lib/semantic_logger/utils.rb +266 -79
- data/lib/semantic_logger/version.rb +1 -1
- data/lib/semantic_logger.rb +35 -2
- metadata +22 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: bfb8a2265081eac50cc50aa7044a5d51b277fba016cb11eb76166d0c9216fe2b
|
|
4
|
+
data.tar.gz: 678e62939127794a93e02280e9e0ed898af39a6015118bdba1f652a65abc4a2f
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: a32ccdf8c3801b3e5166c6d3be83d9c8423defbd2b65e50dc11c262f7cab8c1b0b864a19e90ba9415f2e66d0548a45c0979cb42a03e93252f1381c87634289bc
|
|
7
|
+
data.tar.gz: adab42f9b9e25495ea3a9b99c3322a30a95e4e7eb767f85d27a147831f30861053f08fe22d33318b77c805bf70d301e6c2adcff869cf4ea002607140b68c5ea8
|
data/docs/api.md
CHANGED
|
@@ -67,6 +67,30 @@ Every entry is identified as coming from `Supplier`:
|
|
|
67
67
|
2012-08-30 15:37:29.474 I [48308:main] Supplier -- Accessible from instance methods
|
|
68
68
|
~~~
|
|
69
69
|
|
|
70
|
+
A module can include `SemanticLogger::Loggable` too. The module gets a `logger` of its own, and its
|
|
71
|
+
instance methods get a `logger` in any class that includes the module, whether or not that class
|
|
72
|
+
includes `SemanticLogger::Loggable`:
|
|
73
|
+
|
|
74
|
+
~~~ruby
|
|
75
|
+
module Pricing
|
|
76
|
+
include SemanticLogger::Loggable
|
|
77
|
+
|
|
78
|
+
def price(amount)
|
|
79
|
+
logger.info("Pricing", amount: amount)
|
|
80
|
+
end
|
|
81
|
+
end
|
|
82
|
+
|
|
83
|
+
class Order
|
|
84
|
+
include Pricing
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
Order.new.price(10)
|
|
88
|
+
~~~
|
|
89
|
+
|
|
90
|
+
The entry is identified as coming from `Order`, the class that called `price`. When that class has a
|
|
91
|
+
class level `logger`, such as one from `SemanticLogger::Loggable`, the module's methods use it instead.
|
|
92
|
+
When the class itself defines a `logger` instance method, they use that.
|
|
93
|
+
|
|
70
94
|
By default `SemanticLogger[...]` returns a brand new logger on each call. To share a single logger
|
|
71
95
|
per class instead, enable [logger caching](config.html#caching-loggers).
|
|
72
96
|
|
|
@@ -176,6 +200,13 @@ logger.error(message: "Calling Supplier", duration: 100)
|
|
|
176
200
|
|
|
177
201
|
# Attach a metric (see Step 6)
|
|
178
202
|
logger.error(message: "Calling Supplier", metric: "Supplier/inquiry", metric_amount: 21)
|
|
203
|
+
|
|
204
|
+
# Attach a backtrace of your own, an Array of Strings, instead of the logger's call stack
|
|
205
|
+
logger.warn(message: "Open redirect detected", backtrace: redirect_stack)
|
|
206
|
+
|
|
207
|
+
# Say where the entry was logged, or that it has no such place (see call_site_level)
|
|
208
|
+
logger.debug(message: "Order Load", duration: 0.4, call_site: query_location)
|
|
209
|
+
logger.info(message: "Started GET /orders", call_site: false)
|
|
179
210
|
~~~
|
|
180
211
|
|
|
181
212
|
In both forms, a key such as `duration` or `exception` is only used as an argument when its value is
|
|
@@ -183,6 +214,11 @@ valid for it; otherwise it is logged as part of the payload. Any other keys supp
|
|
|
183
214
|
`payload:` are merged into it, without overwriting its own values. When the Hash holds untrusted
|
|
184
215
|
data, see [Logging untrusted hashes](security.html#logging-untrusted-hashes).
|
|
185
216
|
|
|
217
|
+
A block can return the same Hash, but it never replaces an argument the entry already has. That
|
|
218
|
+
includes the backtrace that an entry captures when its level is at or above
|
|
219
|
+
`SemanticLogger.backtrace_level`, so to attach a backtrace of your own, pass `backtrace:` with the
|
|
220
|
+
call rather than returning it from the block.
|
|
221
|
+
|
|
186
222
|
## Step 6: Measure how long something takes
|
|
187
223
|
|
|
188
224
|
It is good practice to "measure everything" in production, so that when things slow down it is
|
|
@@ -236,6 +272,17 @@ logger.measure_info("Called external interface", metric: "Supplier/inquiry") do
|
|
|
236
272
|
end
|
|
237
273
|
~~~
|
|
238
274
|
|
|
275
|
+
Add `dimensions` to break the metric down, such as by supplier. See
|
|
276
|
+
[Break a metric down with dimensions](metrics.html#step-3-break-a-metric-down-with-dimensions):
|
|
277
|
+
|
|
278
|
+
~~~ruby
|
|
279
|
+
logger.measure_info("Called external interface",
|
|
280
|
+
metric: "Supplier/inquiry",
|
|
281
|
+
dimensions: {supplier: "acme"}) do
|
|
282
|
+
# ...
|
|
283
|
+
end
|
|
284
|
+
~~~
|
|
285
|
+
|
|
239
286
|
### Supply the duration yourself
|
|
240
287
|
|
|
241
288
|
When you already have a duration, log it without a block. This keeps the duration as structured data
|
|
@@ -255,16 +302,26 @@ The second argument to a measure call is a Hash of options:
|
|
|
255
302
|
- `:min_duration` [Float]: only log if the block takes longer than this many milliseconds. Default
|
|
256
303
|
`0.0` (always log).
|
|
257
304
|
- `:metric` [String]: notify metric subscribers with this metric name.
|
|
305
|
+
- `:dimensions` [Hash]: key/value labels that break the metric down, such as `{supplier: "acme"}`.
|
|
258
306
|
- `:payload` [Hash]: an optional payload to log with the entry.
|
|
259
307
|
- `:exception` [Exception]: an exception to log along with the duration.
|
|
260
308
|
- `:duration` [Float]: the duration in ms, used when no block is supplied (then it is mandatory; with
|
|
261
309
|
a block it is ignored).
|
|
262
310
|
- `:log_exception` [Symbol]: how to report an exception raised in the block. `:full` logs the class,
|
|
263
|
-
message, and backtrace
|
|
264
|
-
|
|
311
|
+
message, and backtrace of the exception and of each exception that caused it; `:partial` adds their
|
|
312
|
+
classes and messages to the log message, without backtraces; `:none` does not log it. Default
|
|
313
|
+
`:partial`. With `:partial`, a cause's message is left out when it is empty, is its class name, or
|
|
314
|
+
is already in the log message, as when the exception that wraps it copied it into its own:
|
|
315
|
+
|
|
316
|
+
```
|
|
317
|
+
Rendering -- Exception: ActionDispatch::Http::MimeNegotiation::InvalidType: "foo/bar" is not a valid MIME type -- Cause: Mime::Type::InvalidMimeType
|
|
318
|
+
```
|
|
265
319
|
- `:on_exception_level` [Symbol]: if an exception is raised, raise the log level to this level.
|
|
266
|
-
- `:silence` [Symbol]: the level to silence other log messages to within the block (current
|
|
320
|
+
- `:silence` [Symbol]: the level to silence other log messages to within the block (current fiber
|
|
267
321
|
only).
|
|
322
|
+
- `:call_site` [Thread::Backtrace::Location|SemanticLogger::CallSite|false]: where the entry was
|
|
323
|
+
logged, or `false` for none. See
|
|
324
|
+
[Recording where each entry was logged](config.html#recording-where-each-entry-was-logged).
|
|
268
325
|
|
|
269
326
|
An option with a value that is not valid for it, such as a non-numeric `:min_duration` or an unknown
|
|
270
327
|
level, is ignored and its default is used. Other keys in the Hash are not logged; pass the data to
|
|
@@ -303,13 +360,60 @@ SemanticLogger.tagged(user: "Jack", zip_code: 12345) do
|
|
|
303
360
|
end
|
|
304
361
|
~~~
|
|
305
362
|
|
|
306
|
-
Tags
|
|
307
|
-
[
|
|
308
|
-
|
|
363
|
+
Tags belong to the current fiber: each thread has its own, and so does each fiber of a fiber
|
|
364
|
+
scheduler such as [Async](https://github.com/socketry/async), which runs every request of a
|
|
365
|
+
[Falcon](https://github.com/socketry/falcon) server in its own fiber. So concurrent requests never
|
|
366
|
+
see each other's tags, and a thread, fiber, or Async task started inside the block does not have
|
|
367
|
+
them either. Hand them over explicitly, as shown next.
|
|
368
|
+
|
|
369
|
+
### Hand tags to another thread or fiber
|
|
370
|
+
|
|
371
|
+
`SemanticLogger.context` returns a frozen snapshot of the current tags, named tags, and
|
|
372
|
+
[silence](#silence-noisy-code) level. `SemanticLogger.with_context` applies it in another thread or
|
|
373
|
+
fiber for the duration of a block:
|
|
374
|
+
|
|
375
|
+
~~~ruby
|
|
376
|
+
SemanticLogger.tagged(request_id: request.uuid) do
|
|
377
|
+
context = SemanticLogger.context
|
|
378
|
+
|
|
379
|
+
Thread.new do
|
|
380
|
+
SemanticLogger.with_context(context) do
|
|
381
|
+
logger.info("Generating report") # carries request_id
|
|
382
|
+
end
|
|
383
|
+
end
|
|
384
|
+
end
|
|
385
|
+
~~~
|
|
386
|
+
|
|
387
|
+
The same works for an Async task or a thread pool:
|
|
388
|
+
|
|
389
|
+
~~~ruby
|
|
390
|
+
context = SemanticLogger.context
|
|
391
|
+
barrier = Async::Barrier.new
|
|
392
|
+
urls.each do |url|
|
|
393
|
+
barrier.async do
|
|
394
|
+
SemanticLogger.with_context(context) { fetch(url) }
|
|
395
|
+
end
|
|
396
|
+
end
|
|
397
|
+
barrier.wait
|
|
398
|
+
|
|
399
|
+
context = SemanticLogger.context
|
|
400
|
+
pool.post do
|
|
401
|
+
SemanticLogger.with_context(context) { process(item) }
|
|
402
|
+
end
|
|
403
|
+
~~~
|
|
404
|
+
|
|
405
|
+
Within the block, the thread or fiber has exactly the tags, named tags, and silence level of the
|
|
406
|
+
snapshot, in place of its own, which come back when the block exits. Add more tags inside the block
|
|
407
|
+
as usual. The snapshot does not carry levels set with a logger's `with_level`, which, like Ruby's own
|
|
408
|
+
`Logger#with_level`, apply only to the fiber that set them.
|
|
409
|
+
|
|
410
|
+
Tags are not handed over automatically because a long-lived thread, such as a thread pool's worker,
|
|
411
|
+
would otherwise keep the tags of whichever request happened to start it, and log them for every
|
|
412
|
+
later request it serves.
|
|
309
413
|
|
|
310
414
|
### Bind tags to one logger (child loggers)
|
|
311
415
|
|
|
312
|
-
The block form above scopes tags to the
|
|
416
|
+
The block form above scopes tags to the fiber. Sometimes it is more convenient to bind tags to a
|
|
313
417
|
single logger instance, for example when the logger belongs to an object with its own identity (an
|
|
314
418
|
ActiveRecord model or a background job).
|
|
315
419
|
|
|
@@ -400,6 +504,52 @@ On JRuby, naming a thread also names the underlying JVM thread, which shows up w
|
|
|
400
504
|
JVM with tools such as jconsole or jstack. JRuby names the main thread `main`, and a thread created
|
|
401
505
|
by Java code, such as a servlet container's request thread, is logged with its JVM thread name.
|
|
402
506
|
|
|
507
|
+
### Log from Ractors (experimental)
|
|
508
|
+
|
|
509
|
+
On Ruby 4.0 and later, code running in a [Ractor](https://docs.ruby-lang.org/en/4.0/Ractor.html)
|
|
510
|
+
logs as usual, and the entries are written by the appenders added in the main Ractor. Like Ruby's
|
|
511
|
+
own Ractor API, this is experimental, and may change in a minor release.
|
|
512
|
+
|
|
513
|
+
~~~ruby
|
|
514
|
+
SemanticLogger.add_appender(io: $stdout, formatter: :color)
|
|
515
|
+
|
|
516
|
+
context = SemanticLogger.tagged(job_id: 42) { SemanticLogger.context }
|
|
517
|
+
worker = Ractor.new(context, name: "worker") do |job_context|
|
|
518
|
+
SemanticLogger.with_context(job_context) do
|
|
519
|
+
logger = SemanticLogger["Importer"]
|
|
520
|
+
logger.info("Importing", rows: 1_000) # carries job_id: 42
|
|
521
|
+
end
|
|
522
|
+
end
|
|
523
|
+
worker.join
|
|
524
|
+
~~~
|
|
525
|
+
|
|
526
|
+
Each entry is sent to the main Ractor, where a thread hands it to the appenders, so a Ractor never
|
|
527
|
+
waits on an appender. The thread name of an entry starts with the Ractor's name, or its number, as in
|
|
528
|
+
`worker/12345` or `ractor-3/12345`.
|
|
529
|
+
|
|
530
|
+
Things to know:
|
|
531
|
+
|
|
532
|
+
- A new Ractor does not have the tags of the code that created it. Hand them over with
|
|
533
|
+
[`SemanticLogger.context`](#hand-tags-to-another-thread-or-fiber), as shown above.
|
|
534
|
+
- Add appenders, and change any setting, such as `SemanticLogger.default_level`, in the main Ractor.
|
|
535
|
+
Doing so in another Ractor raises `Ractor::IsolationError`.
|
|
536
|
+
- Each Ractor has its own loggers: a `Loggable` class has a logger of its own in each Ractor, and so
|
|
537
|
+
does a class with `SemanticLogger.cache_loggers` enabled. Setting a level on the main Ractor's logger
|
|
538
|
+
therefore does not change the level of a Ractor's logger; the global default level applies to both.
|
|
539
|
+
- `on_log` subscribers run for an entry logged in a Ractor only when they are shareable, such as a
|
|
540
|
+
module with a `call` method, or a block made with `Ractor.shareable_proc`.
|
|
541
|
+
- An entry is copied to the main Ractor. A value that Ruby cannot copy, such as a Proc in the payload,
|
|
542
|
+
is written as its `inspect`, and so is such a value held by a logged exception. The extra detail
|
|
543
|
+
that Ruby adds to some exceptions' messages, such as did_you_mean's "Did you mean?" suggestions, is
|
|
544
|
+
not written for an exception logged in a Ractor.
|
|
545
|
+
- `SemanticLogger.flush` in a Ractor waits until the main Ractor has written that Ractor's entries.
|
|
546
|
+
- The main Ractor accepts entries from other Ractors as fast as they are logged, so logging never
|
|
547
|
+
slows a Ractor down, however busy the appenders are. `max_queue_size` and `non_blocking` then apply
|
|
548
|
+
to the entries as usual, once the receiving thread hands them to the processor. In sync mode, too,
|
|
549
|
+
the receiving thread writes them.
|
|
550
|
+
- On Ruby 3.4 and earlier, logging from a Ractor other than the main one raises
|
|
551
|
+
`Ractor::IsolationError`, as it always has.
|
|
552
|
+
|
|
403
553
|
### Change one class's level at runtime
|
|
404
554
|
|
|
405
555
|
Because each class has its own logger, you can change one class's level on the fly, for example to
|
|
@@ -421,7 +571,7 @@ To change the global default level for every logger that has not been set explic
|
|
|
421
571
|
|
|
422
572
|
### Silence noisy code
|
|
423
573
|
|
|
424
|
-
`silence` raises the level within a block, on the current
|
|
574
|
+
`silence` raises the level within a block, on the current fiber only, to quiet a noisy section:
|
|
425
575
|
|
|
426
576
|
~~~ruby
|
|
427
577
|
# Within this block, log only :error and above
|
|
@@ -441,7 +591,8 @@ end
|
|
|
441
591
|
~~~
|
|
442
592
|
|
|
443
593
|
`silence` has no effect on loggers whose level was set explicitly (those that do not follow the global
|
|
444
|
-
default), and does not affect threads
|
|
594
|
+
default), and does not affect threads or fibers started inside the block, unless they are handed the
|
|
595
|
+
block's [context](#hand-tags-to-another-thread-or-fiber).
|
|
445
596
|
|
|
446
597
|
### Map a noisy gem's debug logs to trace
|
|
447
598
|
|