semantic_logger 5.1.0 → 5.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (71) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +13 -12
  3. data/Rakefile +15 -4
  4. data/docs/api.md +37 -11
  5. data/docs/appenders.md +123 -39
  6. data/docs/config.md +115 -13
  7. data/docs/index.md +31 -5
  8. data/docs/log.md +19 -11
  9. data/docs/metrics.md +5 -2
  10. data/docs/operations.md +63 -17
  11. data/docs/rails.md +77 -3
  12. data/docs/security.md +183 -16
  13. data/docs/testing.md +5 -3
  14. data/docs/upgrading.md +295 -7
  15. data/lib/semantic_logger/appender/async.rb +18 -5
  16. data/lib/semantic_logger/appender/bugsnag.rb +1 -1
  17. data/lib/semantic_logger/appender/cloudwatch_logs.rb +122 -27
  18. data/lib/semantic_logger/appender/elasticsearch_base.rb +31 -6
  19. data/lib/semantic_logger/appender/elasticsearch_http.rb +1 -1
  20. data/lib/semantic_logger/appender/file.rb +138 -30
  21. data/lib/semantic_logger/appender/graylog.rb +22 -6
  22. data/lib/semantic_logger/appender/honeybadger.rb +1 -1
  23. data/lib/semantic_logger/appender/honeybadger_insights.rb +1 -1
  24. data/lib/semantic_logger/appender/http.rb +70 -11
  25. data/lib/semantic_logger/appender/io.rb +1 -1
  26. data/lib/semantic_logger/appender/kafka.rb +26 -12
  27. data/lib/semantic_logger/appender/loki.rb +13 -3
  28. data/lib/semantic_logger/appender/mongodb.rb +20 -1
  29. data/lib/semantic_logger/appender/new_relic.rb +1 -1
  30. data/lib/semantic_logger/appender/new_relic_logs.rb +9 -11
  31. data/lib/semantic_logger/appender/open_telemetry.rb +2 -2
  32. data/lib/semantic_logger/appender/rabbitmq.rb +6 -1
  33. data/lib/semantic_logger/appender/sentry.rb +1 -1
  34. data/lib/semantic_logger/appender/sentry_ruby.rb +91 -17
  35. data/lib/semantic_logger/appender/splunk.rb +10 -1
  36. data/lib/semantic_logger/appender/syslog.rb +64 -26
  37. data/lib/semantic_logger/appender/tcp.rb +6 -1
  38. data/lib/semantic_logger/appender/udp.rb +11 -1
  39. data/lib/semantic_logger/appender/wrapper.rb +1 -1
  40. data/lib/semantic_logger/appenders.rb +9 -1
  41. data/lib/semantic_logger/base.rb +82 -30
  42. data/lib/semantic_logger/exception_details.rb +50 -0
  43. data/lib/semantic_logger/formatters/base.rb +52 -6
  44. data/lib/semantic_logger/formatters/color.rb +18 -2
  45. data/lib/semantic_logger/formatters/default.rb +22 -5
  46. data/lib/semantic_logger/formatters/logfmt.rb +153 -21
  47. data/lib/semantic_logger/formatters/loki.rb +26 -22
  48. data/lib/semantic_logger/formatters/one_line.rb +67 -3
  49. data/lib/semantic_logger/formatters/open_telemetry.rb +16 -17
  50. data/lib/semantic_logger/formatters/pattern.rb +8 -4
  51. data/lib/semantic_logger/formatters/raw.rb +4 -5
  52. data/lib/semantic_logger/formatters/syslog.rb +110 -18
  53. data/lib/semantic_logger/formatters/syslog_cee.rb +29 -16
  54. data/lib/semantic_logger/levels.rb +10 -1
  55. data/lib/semantic_logger/log.rb +183 -37
  56. data/lib/semantic_logger/loggable.rb +26 -3
  57. data/lib/semantic_logger/metric/new_relic.rb +1 -1
  58. data/lib/semantic_logger/metric/signalfx.rb +8 -6
  59. data/lib/semantic_logger/metric/statsd.rb +15 -7
  60. data/lib/semantic_logger/queue_processor/request.rb +55 -0
  61. data/lib/semantic_logger/queue_processor/throttled_counter.rb +46 -0
  62. data/lib/semantic_logger/queue_processor.rb +165 -66
  63. data/lib/semantic_logger/semantic_logger.rb +15 -2
  64. data/lib/semantic_logger/subscriber.rb +96 -10
  65. data/lib/semantic_logger/sync_processor.rb +11 -0
  66. data/lib/semantic_logger/test/capture_log_events.rb +8 -5
  67. data/lib/semantic_logger/utils.rb +244 -16
  68. data/lib/semantic_logger/version.rb +1 -1
  69. data/lib/semantic_logger.rb +1 -1
  70. metadata +21 -5
  71. data/lib/semantic_logger/core_ext/thread.rb +0 -30
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: f18701015c0b556099814aff1b79e251df13f556c103a8366b58357d9d10fd8c
4
- data.tar.gz: cf18f050902946ac09a065a52a6f4cee9841f5c250ef30162d39347977e071d2
3
+ metadata.gz: 5b42624348f8dc147462cc1f2d26f7cf0d4ef3df91fe3157a119591e8f6bd23b
4
+ data.tar.gz: ca1f00571b5280a9cd5b4bf0f116944ba4850eb1536b50aff53bfba9b6ba540b
5
5
  SHA512:
6
- metadata.gz: f62d59fc517122f6da01702c62cf5a33e47d629605f0dd5e096f953ebdcd7d3e6dbb9bf2e25d166d32f171dbe16ba1470726ebe760343e6e04994cef7b5c6dcd
7
- data.tar.gz: 3c741614f72b723e50ed3e3ec2a785de85eb6a026b9ed52d67bb064c1617814d77afeb67e5f8f7f4cb6f20f339affd14f54a82ffe23194e256ba66da9e8aedf0
6
+ metadata.gz: 31f5b285c287b386979f89b794d71f9084276b13510dc5b7798a510815f512e713bdf44a87743808890a8f4cef1e74a8263a8ca78edba8f7247235bf53aa6fd5
7
+ data.tar.gz: 3f114bc7d9352eb40c1a4e8a31ecfa13b20e0bfec61c72fceaba522ffc041c97ffe50e0f788b63415c4265ac68586a185473a7c4320d59e8f708ac8facedbd72
data/README.md CHANGED
@@ -1,5 +1,5 @@
1
1
  # Semantic Logger
2
- [![Gem Version](https://img.shields.io/gem/v/semantic_logger.svg)](https://rubygems.org/gems/semantic_logger) [![Build Status](https://github.com/reidmorrison/semantic_logger/workflows/build/badge.svg)](https://github.com/reidmorrison/semantic_logger/actions?query=workflow%3Abuild) [![Downloads](https://img.shields.io/gem/dt/semantic_logger.svg)](https://rubygems.org/gems/semantic_logger) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](http://opensource.org/licenses/Apache-2.0) ![](https://img.shields.io/badge/status-Production%20Ready-blue.svg)
2
+ [![Gem Version](https://img.shields.io/gem/v/semantic_logger.svg)](https://rubygems.org/gems/semantic_logger) [![Build Status](https://github.com/reidmorrison/semantic_logger/workflows/build/badge.svg)](https://github.com/reidmorrison/semantic_logger/actions?query=workflow%3Abuild) [![Downloads](https://img.shields.io/gem/dt/semantic_logger.svg)](https://rubygems.org/gems/semantic_logger) [![Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/reidmorrison/semantic_logger/badges/coverage.json)](https://github.com/reidmorrison/semantic_logger/actions/workflows/ci.yml) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](http://opensource.org/licenses/Apache-2.0) ![](https://img.shields.io/badge/status-Production%20Ready-blue.svg)
3
3
 
4
4
  Semantic Logger is a high-performance, asynchronous structured logging framework for Ruby
5
5
  and Rails.
@@ -38,12 +38,12 @@ replaces the Rails default logger with Semantic Logger automatically.
38
38
 
39
39
  ## Documentation
40
40
 
41
- Start with the [Introduction](https://logger.rocketjob.io/), then the
42
- [Programmer's Guide](https://logger.rocketjob.io/api.html).
41
+ Start with the [Introduction](https://logger.reidmorrison.com/), then the
42
+ [Programmer's Guide](https://logger.reidmorrison.com/api.html).
43
43
 
44
- * Full guide: [https://logger.rocketjob.io/](https://logger.rocketjob.io/)
45
- * For AI assistants: [llms.txt](https://logger.rocketjob.io/llms.txt) (documentation index) and
46
- [llms-full.txt](https://logger.rocketjob.io/llms-full.txt) (complete documentation in one file).
44
+ * Full guide: [https://logger.reidmorrison.com/](https://logger.reidmorrison.com/)
45
+ * For AI assistants: [llms.txt](https://logger.reidmorrison.com/llms.txt) (documentation index) and
46
+ [llms-full.txt](https://logger.reidmorrison.com/llms-full.txt) (complete documentation in one file).
47
47
  The same pages are also included as markdown in the installed gem, under `docs/`.
48
48
 
49
49
  ## Logging Destinations
@@ -82,7 +82,7 @@ instead of Semantic Logger directly since it will automatically replace the Rail
82
82
 
83
83
  ## Rocket Job
84
84
 
85
- Checkout the sister project [Rocket Job](http://rocketjob.io): Ruby's missing batch system.
85
+ Checkout the sister project [Rocket Job](https://rocketjob.reidmorrison.com): Ruby's missing batch system.
86
86
 
87
87
  Fully supports Semantic Logger when running jobs in the background. Complete support for job metrics
88
88
  sent via Semantic Logger to your favorite dashboards.
@@ -97,9 +97,10 @@ and are therefore not automatically included by this gem:
97
97
  - HoneybadgerInsights Appender: gem 'honeybadger'
98
98
  - NewRelic Appender: gem 'newrelic_rpm'
99
99
  - NewRelicLogs Appender: gem 'newrelic_rpm'
100
- - Syslog Appender: gem 'syslog_protocol' 0.9.2 or above
101
- - Syslog Appender to a remote syslogng server over TCP or UDP: gem 'net_tcp_client'
102
- - Splunk Appender: gem 'splunk-sdk-ruby'
100
+ - Syslog Appender to a local syslog daemon: gem 'syslog' (a bundled gem since Ruby 3.4)
101
+ - Syslog Appender to a remote syslog server over TCP or UDP: gem 'syslog_protocol' 0.9.2 or above
102
+ - Syslog Appender to a remote syslog server over TCP: gem 'net_tcp_client'
103
+ - Splunk Appender (deprecated, use the Splunk HTTP appender): gem 'splunk-sdk-ruby'
103
104
  - Elasticsearch Appender: gem 'elasticsearch'
104
105
  - OpenSearch Appender: gem 'opensearch-ruby'
105
106
  - Kafka Appender: gem 'ruby-kafka'
@@ -109,7 +110,7 @@ and are therefore not automatically included by this gem:
109
110
 
110
111
  ## Upgrading
111
112
 
112
- See the [Upgrading Guide](https://logger.rocketjob.io/upgrading.html) for instructions on
113
+ See the [Upgrading Guide](https://logger.reidmorrison.com/upgrading.html) for instructions on
113
114
  upgrading between major versions.
114
115
 
115
116
  ## Install
@@ -128,7 +129,7 @@ SemanticLogger.default_level = :trace
128
129
  SemanticLogger.add_appender(file_name: 'development.log', formatter: :color)
129
130
  ~~~
130
131
 
131
- If running rails, see: [Semantic Logger Rails](https://logger.rocketjob.io/rails.html)
132
+ If running rails, see: [Semantic Logger Rails](https://logger.reidmorrison.com/rails.html)
132
133
 
133
134
  ## Contributing
134
135
 
data/Rakefile CHANGED
@@ -25,19 +25,30 @@ task :llms_full do
25
25
 
26
26
  > Semantic Logger is a high-performance, asynchronous structured logging framework for Ruby and Rails.
27
27
 
28
- This file concatenates every page of https://logger.rocketjob.io for consumption by AI assistants.
28
+ This file concatenates every page of https://logger.reidmorrison.com for consumption by AI assistants.
29
29
  It is generated from the markdown sources in docs/ by `bundle exec rake llms_full`; do not edit it directly.
30
- A per-page index is available at https://logger.rocketjob.io/llms.txt
30
+ A per-page index is available at https://logger.reidmorrison.com/llms.txt
31
31
  HEADER
32
32
 
33
33
  sections = pages.map do |page|
34
- text = File.read("docs/#{page}.md").
34
+ raw = File.read("docs/#{page}.md")
35
+
36
+ # The page title lives in front matter, which the shared docs theme renders
37
+ # as the h1. This file strips front matter, so lift the title back out and
38
+ # re-emit it as a heading; without this every section here would open with
39
+ # no indication of which page it is.
40
+ front_matter = raw[/\A---\n(.*?)\n---\n/m, 1].to_s
41
+ title = front_matter[/^(?:title|heading):[ \t]*(.+)$/, 1].to_s.strip.delete_prefix('"').delete_suffix('"')
42
+
43
+ text = raw.
35
44
  sub(/\A---\n.*?\n---\n/m, ""). # Jekyll front matter
36
45
  gsub(/^\{:.*\}\n/, ""). # kramdown attribute lines ({:toc}, {:.no_toc}, ...)
37
46
  gsub(/^\* TOC\n/, "").
38
47
  gsub(/^\*\*Contents\*\*\n/, "").
39
48
  gsub(/^!\[.*\n/, "") # images (relative paths, useless in plain text)
40
- "<!-- source: docs/#{page}.md -->\n\n#{text.strip}\n"
49
+
50
+ body = title.empty? ? text.strip : "## #{title}\n\n#{text.strip}"
51
+ "<!-- source: docs/#{page}.md -->\n\n#{body}\n"
41
52
  end
42
53
 
43
54
  File.write("docs/llms-full.txt", ([header] + sections).join("\n\n---\n\n"))
data/docs/api.md CHANGED
@@ -1,9 +1,12 @@
1
1
  ---
2
2
  layout: default
3
+ title: Programming Guide
4
+ description: >-
5
+ How to log with Semantic Logger: getting a logger, log levels, structured
6
+ payloads, exceptions, the block form, measuring durations, and tagging
7
+ related entries.
3
8
  ---
4
9
 
5
- ## Programming Guide
6
- {:.no_toc}
7
10
 
8
11
  **Contents**
9
12
 
@@ -175,6 +178,11 @@ logger.error(message: "Calling Supplier", duration: 100)
175
178
  logger.error(message: "Calling Supplier", metric: "Supplier/inquiry", metric_amount: 21)
176
179
  ~~~
177
180
 
181
+ In both forms, a key such as `duration` or `exception` is only used as an argument when its value is
182
+ valid for it; otherwise it is logged as part of the payload. Any other keys supplied alongside
183
+ `payload:` are merged into it, without overwriting its own values. When the Hash holds untrusted
184
+ data, see [Logging untrusted hashes](security.html#logging-untrusted-hashes).
185
+
178
186
  ## Step 6: Measure how long something takes
179
187
 
180
188
  It is good practice to "measure everything" in production, so that when things slow down it is
@@ -252,12 +260,16 @@ The second argument to a measure call is a Hash of options:
252
260
  - `:duration` [Float]: the duration in ms, used when no block is supplied (then it is mandatory; with
253
261
  a block it is ignored).
254
262
  - `:log_exception` [Symbol]: how to report an exception raised in the block. `:full` logs the class,
255
- message, and backtrace; `:partial` logs the class and message only; `:off` does not log it.
263
+ message, and backtrace; `:partial` logs the class and message only; `:none` does not log it.
256
264
  Default `:partial`.
257
265
  - `:on_exception_level` [Symbol]: if an exception is raised, raise the log level to this level.
258
266
  - `:silence` [Symbol]: the level to silence other log messages to within the block (current thread
259
267
  only).
260
268
 
269
+ An option with a value that is not valid for it, such as a non-numeric `:min_duration` or an unknown
270
+ level, is ignored and its default is used. Other keys in the Hash are not logged; pass the data to
271
+ log as `:payload`.
272
+
261
273
  Putting several together:
262
274
 
263
275
  ~~~ruby
@@ -347,11 +359,11 @@ The features above cover everyday logging. The rest of this guide covers less co
347
359
 
348
360
  ### Name your threads
349
361
 
350
- Semantic Logger includes the thread name (or id) in every entry. On Ruby MRI the name defaults to the
351
- thread's object id:
362
+ Semantic Logger includes the thread name in every entry. A thread that has no name is identified by
363
+ its object id instead:
352
364
 
353
365
  ~~~
354
- 2013-11-07 16:25:14.279 I [35841:70184354571980] (0.0ms) ExternalSupplier -- Calling external interface
366
+ 2013-11-07 16:25:14.279 I [35841:16] (0.0ms) ExternalSupplier -- Calling external interface
355
367
  ~~~
356
368
 
357
369
  Give a thread a readable name so it stands out in the logs:
@@ -364,15 +376,29 @@ Thread.current.name = "User calculation thread 32"
364
376
  2013-11-07 16:26:02.744 I [35841:User calculation thread 32] (0.0ms) ExternalSupplier -- Calling external interface
365
377
  ~~~
366
378
 
367
- Keep the name unique, otherwise concurrent threads are hard to tell apart. Including the object id is
368
- one way to guarantee that:
379
+ Keep the name unique, otherwise concurrent threads are hard to tell apart. For example, number the
380
+ threads in a pool:
369
381
 
370
382
  ~~~ruby
371
- Thread.current.name = "Worker Thread:#{Thread.current.object_id}"
383
+ workers = Array.new(5) do |i|
384
+ Thread.new do
385
+ Thread.current.name = "calc #{i + 1}"
386
+ # ...
387
+ end
388
+ end
372
389
  ~~~
373
390
 
374
- On JRuby this also sets the underlying JVM thread name, which is useful when monitoring the JVM over
375
- JMX with tools such as jconsole.
391
+ This is Ruby's own `Thread#name=`, so the name must be a String, and Ruby raises for one that
392
+ contains a NUL byte or is not in an ASCII-compatible encoding. Ruby also gives the name to the
393
+ operating system, so it shows up in tools that list threads, such as `top -H`, debuggers, and
394
+ profilers. Log entries show the whole name, but Linux keeps only its first 15 bytes, so keep the
395
+ name short, or put the part that tells threads apart first. Semantic Logger keeps its own thread
396
+ names short for the same reason, naming each after its appender, such as `SL Appenders` and
397
+ `SL File`.
398
+
399
+ On JRuby, naming a thread also names the underlying JVM thread, which shows up when monitoring the
400
+ JVM with tools such as jconsole or jstack. JRuby names the main thread `main`, and a thread created
401
+ by Java code, such as a servlet container's request thread, is logged with its JVM thread name.
376
402
 
377
403
  ### Change one class's level at runtime
378
404
 
data/docs/appenders.md CHANGED
@@ -1,9 +1,12 @@
1
1
  ---
2
2
  layout: default
3
+ title: Appenders
4
+ description: >-
5
+ Every destination Semantic Logger writes to and how to configure it: files
6
+ and streams, structured formats, network protocols, log aggregators, and
7
+ error monitors.
3
8
  ---
4
9
 
5
- ## Appenders
6
- {:.no_toc}
7
10
 
8
11
  **Contents**
9
12
 
@@ -83,7 +86,7 @@ time the appender is used, and is never a hard dependency of Semantic Logger its
83
86
  | [HTTP(S)](#https) | `appender: :http` | |
84
87
  | [TCP (+SSL)](#tcp-appender-ssl) | `appender: :tcp` | `net_tcp_client` |
85
88
  | [UDP](#udp-appender) | `appender: :udp` | |
86
- | [Syslog](#syslog) | `appender: :syslog` | `syslog_protocol`, `net_tcp_client` (remote) |
89
+ | [Syslog](#syslog) | `appender: :syslog` | `syslog` (local, Ruby 3.4+), `syslog_protocol` (remote), `net_tcp_client` (TCP) |
87
90
 
88
91
  **Centralized logging and log aggregators**
89
92
 
@@ -93,7 +96,7 @@ time the appender is used, and is never a hard dependency of Semantic Logger its
93
96
  | [OpenSearch](#opensearch) | `appender: :opensearch` | `opensearch-ruby` |
94
97
  | [Graylog](#graylog) | `appender: :graylog` | `gelf` |
95
98
  | [Splunk over HTTP](#splunk-http) | `appender: :splunk_http` | |
96
- | [Splunk SDK](#splunk-sdk) | `appender: :splunk` | `splunk-sdk-ruby` |
99
+ | [Splunk SDK](#splunk-sdk) (deprecated) | `appender: :splunk` | `splunk-sdk-ruby` |
97
100
  | [Grafana Loki](#grafana-loki) | `appender: :loki` | |
98
101
  | [CloudWatch Logs](#cloudwatch-logs) | `appender: :cloudwatch_logs` | `aws-sdk-cloudwatchlogs` |
99
102
  | [OpenTelemetry](#opentelemetry) | `appender: :open_telemetry` | `opentelemetry-logs-sdk` |
@@ -158,6 +161,14 @@ file is created and to an existing log file:
158
161
  SemanticLogger.add_appender(file_name: "production.log", permissions: 0o640)
159
162
  ~~~
160
163
 
164
+ With `permissions:`, the log file must be a regular file owned by the process, with a single hard
165
+ link, that the file name names directly rather than through a symbolic link. Nothing is written to
166
+ any other file, such as a log file linked to `/dev/stdout`, nor when the file cannot be opened:
167
+ `add_appender` raises `IOError` giving the reason, and a file that fails the check when it is
168
+ reopened later, for example on a rotation, is reported and not written to until it is fixed. To log
169
+ to standard output, use `io: $stdout` instead. See
170
+ [Log file permissions](security.html#log-file-permissions).
171
+
161
172
  ### IO Streams
162
173
 
163
174
  Log to any IO stream instance, such as `$stdout` or `$stderr`:
@@ -259,6 +270,9 @@ Notes:
259
270
  * The layout above is formatted for readability. The real output is a single line terminated by one
260
271
  newline, with no embedded newlines.
261
272
  * A field with a `nil` value is excluded from the output.
273
+ * To also log the detail Ruby adds to an exception's message, such as did_you_mean suggestions, as
274
+ `exception.details`, enable `log_exception_details`: `formatter: {json: {log_exception_details: true}}`.
275
+ See [Exception details](config.html#exception-details).
262
276
 
263
277
  ### Fluentd
264
278
 
@@ -351,13 +365,30 @@ timestamp="2024-07-20T08:32:05.375276Z" level="info" name="MyClass" message="Hel
351
365
 
352
366
  Notes:
353
367
 
354
- * The timestamp is ISO 8601. All values are quoted and escaped, so a newline in the data cannot
355
- forge or split a record.
356
- * Named tags and payload fields are merged into the top-level `key=value` pairs. On a name
357
- conflict the payload wins.
368
+ * The timestamp is ISO 8601, under the key `timestamp` unless the formatter's `time_key` is set
369
+ (`formatter: {logfmt: {time_key: :ts}}`). All values are quoted and escaped, so a newline in the
370
+ data cannot forge or split a record.
371
+ * Keys (from tags, named tags, and the payload) cannot be quoted in logfmt, so spaces, control
372
+ characters (including the Unicode line and paragraph separators), invisible format characters
373
+ (such as a zero width space), `=`, and `"` in a key are replaced with `_`, and invalid UTF-8 is
374
+ dropped. For example the tag `second breakfast` is emitted as `second_breakfast=true`.
375
+ * Named tags and payload fields are merged into the top-level `key=value` pairs. A key named like
376
+ one of the entry's own fields (the time key, `level`, `name`, `message`, `duration`,
377
+ `duration_ms`, `tag`, or an exception field) is emitted with its source as a prefix, so that
378
+ logged data cannot replace those fields: `payload.level="fatal"`, `named_tags.message="..."`, or
379
+ `tags.message=true`. A key that starts with `tags.`, `named_tags.`, or `payload.` is prefixed
380
+ too, as in `payload.named_tags.level`, so that it cannot pass for another source's field.
381
+ * Other keys that are the same once encoded, including keys that only differ by type (such as
382
+ `:user` and `"user"`), are emitted once, with the value from the later source: the payload wins
383
+ over named tags, and named tags over tags.
358
384
  * Unnamed tags are emitted as keys with a `true` value.
385
+ * A nested value, such as a Hash or an Array in the payload, is emitted as its JSON text, quoted:
386
+ `user="{\"id\":7}"`.
359
387
  * `tag="success"` is emitted on every entry, becoming `tag="exception"` when the entry carries an
360
- exception (the exception class, message, and backtrace are then included as fields).
388
+ exception, which is then included as the fields `exception_name`, `exception_message`,
389
+ `stack_trace`, and, when the exception has a cause, `cause`. With `log_exception_details: true`
390
+ (off by default), the detail Ruby adds to the exception's message is included as
391
+ `exception_details`; see [Exception details](config.html#exception-details).
361
392
 
362
393
  ## Network protocols
363
394
 
@@ -388,6 +419,31 @@ end
388
419
  SemanticLogger.add_appender(appender: :http, url: "https://localhost:8088/path", formatter: formatter)
389
420
  ~~~
390
421
 
422
+ #### Query parameters
423
+
424
+ Some log servers are configured through the query string of the ingestion url rather than through
425
+ headers or the body. A query string on `url:` is kept and sent with every request that submits log
426
+ data, for example when pointing at
427
+ [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/data-ingestion/):
428
+
429
+ ~~~ruby
430
+ SemanticLogger.add_appender(
431
+ appender: :http,
432
+ url: "http://localhost:9428/insert/jsonline?_msg_field=message&_time_field=timestamp"
433
+ )
434
+ ~~~
435
+
436
+ The query is applied after the request path, so it also works for the appenders built on top of the
437
+ HTTP appender (`:elasticsearch_http`, `:splunk_http`, `:loki`, and the `:signalfx` metrics appender),
438
+ each of which appends its own path to the url:
439
+
440
+ ~~~ruby
441
+ SemanticLogger.add_appender(
442
+ appender: :elasticsearch_http,
443
+ url: "http://localhost:9428/insert/elasticsearch?_msg_field=message&_time_field=timestamp"
444
+ )
445
+ ~~~
446
+
391
447
  #### Batching
392
448
 
393
449
  By default each entry is sent in its own HTTP request. To send multiple entries in one request as a
@@ -424,11 +480,12 @@ SemanticLogger.add_appender(appender: :tcp, server: "localhost:8088")
424
480
  # TCP with SSL:
425
481
  SemanticLogger.add_appender(appender: :tcp, server: "localhost:8088", ssl: true)
426
482
 
427
- # With self-signed certificates, or to disable server certificate verification:
483
+ # With a self-signed certificate, or one from a private certificate authority, trust that
484
+ # certificate instead of disabling verification:
428
485
  SemanticLogger.add_appender(
429
486
  appender: :tcp,
430
487
  server: "localhost:8088",
431
- ssl: {verify_mode: OpenSSL::SSL::VERIFY_NONE}
488
+ ssl: {ca_file: "/path/to/ca.pem"}
432
489
  )
433
490
  ~~~
434
491
 
@@ -449,8 +506,10 @@ options.
449
506
 
450
507
  The TCP and UDP appenders separate records with a newline, so they default to the JSON formatter,
451
508
  which escapes embedded newlines and is safe with untrusted data. If you switch to a text formatter
452
- such as `:default` or `:color`, enable `escape_control_chars` so a newline in the data cannot forge or
453
- split a record:
509
+ such as `:default` or `:color`, enable `escape_control_chars` so a newline in the logged data cannot
510
+ forge a record. Those formatters still write an entry with an exception over several lines (the
511
+ exception, then its backtrace), so a receiver that frames records by newline splits such an entry;
512
+ use the JSON formatter or the `:syslog` formatter when every record must be a single line:
454
513
 
455
514
  ~~~ruby
456
515
  SemanticLogger.add_appender(
@@ -482,7 +541,8 @@ SemanticLogger.add_appender(appender: :udp, server: "localhost:8088", formatter:
482
541
 
483
542
  ### Syslog
484
543
 
485
- Log to a local Syslog daemon:
544
+ Log to a local Syslog daemon. Since Ruby 3.4 `syslog` is a bundled gem, so add `gem "syslog"` to the
545
+ Gemfile:
486
546
 
487
547
  ~~~ruby
488
548
  SemanticLogger.add_appender(appender: :syslog)
@@ -495,12 +555,32 @@ The default packet size is 1024 bytes:
495
555
  SemanticLogger.add_appender(appender: :syslog, url: "tcp://myloghost:514", max_size: 2048)
496
556
  ~~~
497
557
 
558
+ `max_size` also applies when `formatter:` names the `:syslog` or `:syslog_cee` formatter. A
559
+ formatter passed as an instance uses its own `max_size`, which also defaults to 1024 bytes:
560
+
561
+ ~~~ruby
562
+ SemanticLogger.add_appender(appender: :syslog, url: "udp://myloghost:514", formatter: :syslog_cee, max_size: 2048)
563
+ ~~~
564
+
498
565
  Or over UDP (the `syslog_protocol` gem is required):
499
566
 
500
567
  ~~~ruby
501
568
  SemanticLogger.add_appender(appender: :syslog, url: "udp://myloghost:514")
502
569
  ~~~
503
570
 
571
+ Set the facility with one of Ruby's `::Syslog::LOG_*` facility constants (the default is
572
+ `::Syslog::LOG_USER`). For remote syslog without the `syslog` gem, the same constants are available
573
+ from `SemanticLogger::Formatters::Syslog`, for example `SemanticLogger::Formatters::Syslog::LOG_LOCAL0`:
574
+
575
+ ~~~ruby
576
+ SemanticLogger.add_appender(appender: :syslog, url: "udp://myloghost:514", facility: ::Syslog::LOG_LOCAL0)
577
+ ~~~
578
+
579
+ The `facility:`, `level_map:`, and `max_size:` options also apply when `formatter:` names the `:syslog` or
580
+ `:syslog_cee` formatter, for example `formatter: :syslog_cee`. Options given to the formatter itself,
581
+ as in `formatter: {syslog: {facility: ::Syslog::LOG_LOCAL1}}`, take precedence. A formatter passed
582
+ as an instance keeps its own settings.
583
+
504
584
  Add a filter to exclude noisy entries such as health checks:
505
585
 
506
586
  ~~~ruby
@@ -512,8 +592,10 @@ SemanticLogger.add_appender(
512
592
  ~~~
513
593
 
514
594
  Syslog frames each record, so embedded newlines or other control characters in untrusted data could
515
- forge or split records. The syslog formatters therefore escape control characters by default. To pass
516
- them through unchanged:
595
+ forge or split records. The syslog formatters therefore escape control characters by default. The
596
+ `:syslog` formatter also escapes the newlines between an exception and its backtrace lines (written
597
+ as `\n`), so that an entry with an exception is a single line. To pass control characters through
598
+ unchanged, which also restores the multi-line exception layout:
517
599
 
518
600
  ~~~ruby
519
601
  SemanticLogger.add_appender(
@@ -603,24 +685,13 @@ instead, which reached about 1,400 entries per second (1,200 with SSL).
603
685
 
604
686
  ### Splunk SDK
605
687
 
606
- `appender: :splunk` submits entries through the official `splunk-sdk-ruby` gem to the Splunk
607
- management port (8089 by default). Authenticate with either a username and password, or a
608
- pre-authenticated `token:`:
688
+ > **Deprecated:** `appender: :splunk` is deprecated and will be removed in v6. Use the
689
+ > [HTTP Event Collector](#splunk-http) (`appender: :splunk_http`) or the
690
+ > [TCP appender](#tcp-appender-ssl) instead.
609
691
 
610
- ~~~ruby
611
- SemanticLogger.add_appender(
612
- appender: :splunk,
613
- username: "username",
614
- password: "password",
615
- host: "localhost",
616
- port: 8089,
617
- scheme: :https,
618
- index: "main"
619
- )
620
- ~~~
621
-
622
- For most deployments the [HTTP Event Collector](#splunk-http) or the [TCP appender](#tcp-appender-ssl)
623
- is the simpler choice; use the SDK appender when the management API is the only ingress available.
692
+ `appender: :splunk` submits entries through the `splunk-sdk-ruby` gem to the Splunk management
693
+ port. That gem was last released in 2015 and does not verify the server's TLS certificate, so the
694
+ connection, including the login credentials, is open to interception.
624
695
 
625
696
  ### Grafana Loki
626
697
 
@@ -638,7 +709,8 @@ SemanticLogger.add_appender(
638
709
  ~~~
639
710
 
640
711
  Set the URL, username, and password to match your Loki instance. Set `compress: true` to compress the
641
- log messages.
712
+ log messages. The ingestion path is appended to the url, so a query string on the url is kept and sent
713
+ with every push.
642
714
 
643
715
  ### CloudWatch Logs
644
716
 
@@ -804,19 +876,27 @@ Some logging context is forwarded to Sentry:
804
876
 
805
877
  * From named tags, `transaction_name`. See
806
878
  <https://docs.sentry.io/platforms/ruby/enriching-events/transaction-name/>.
807
- * From named tags or payload, `user` is built from the `user_id`, `username`, `user_email`, and
879
+ * From named tags, `user` is built from the `user_id`, `username`, `user_email`, and
808
880
  `ip_address` keys, plus any `user` key that is itself a hash. See
809
881
  <https://docs.sentry.io/platforms/ruby/enriching-events/identify-user/>.
810
- * From the payload, `fingerprint` configures grouping granularity. See
882
+ * From named tags, `fingerprint` (an Array of Strings, Symbols, or numbers) configures grouping
883
+ granularity. See
811
884
  <https://docs.sentry.io/platforms/ruby/usage/sdk-fingerprinting/>.
812
885
  * Named tags are sent as Sentry tags. See
813
886
  <https://docs.sentry.io/platforms/ruby/enriching-events/tags/>.
814
887
  * Unnamed tags are sent as the `:tag` tag, separated by commas.
815
- * Everything else from the payload and context is added to `extras`.
888
+ * Everything else from the payload and context is added to `extras`, including a value of the
889
+ wrong type for one of the keys above.
890
+
891
+ > **Deprecated:** the `user` and `fingerprint` keys are also still taken from the payload, but that
892
+ > is deprecated and will be removed in v6, since the payload often holds untrusted data, such as a
893
+ > request body, that could set the user an error is reported for. Set them with named tags. The
894
+ > named tags win over the payload, and a user field is only taken from the payload when it is a
895
+ > String, Symbol, or Integer.
816
896
 
817
897
  ~~~ruby
818
- SemanticLogger.tagged(transaction_name: "foo", user_id: 42, baz: "quz") do
819
- logger.error("some message", username: "joe", fingerprint: ["bar"])
898
+ SemanticLogger.tagged(transaction_name: "foo", user_id: 42, username: "joe", fingerprint: ["bar"], baz: "quz") do
899
+ logger.error("some message")
820
900
  end
821
901
  ~~~
822
902
 
@@ -868,6 +948,10 @@ SemanticLogger.on_log do |log|
868
948
  end
869
949
  ~~~
870
950
 
951
+ It sends the whole entry, payload included, so register it after any `on_log` subscriber that
952
+ [redacts sensitive data](security.html#redacting-sensitive-data): subscribers run in the order they
953
+ were registered.
954
+
871
955
  ## Databases and message queues
872
956
 
873
957
  ### MongoDB