semantic_logger 5.1.0 → 5.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +13 -12
- data/Rakefile +15 -4
- data/docs/api.md +37 -11
- data/docs/appenders.md +123 -39
- data/docs/config.md +115 -13
- data/docs/index.md +31 -5
- data/docs/log.md +19 -11
- data/docs/metrics.md +5 -2
- data/docs/operations.md +63 -17
- data/docs/rails.md +77 -3
- data/docs/security.md +183 -16
- data/docs/testing.md +5 -3
- data/docs/upgrading.md +295 -7
- data/lib/semantic_logger/appender/async.rb +18 -5
- data/lib/semantic_logger/appender/bugsnag.rb +1 -1
- data/lib/semantic_logger/appender/cloudwatch_logs.rb +122 -27
- data/lib/semantic_logger/appender/elasticsearch_base.rb +31 -6
- data/lib/semantic_logger/appender/elasticsearch_http.rb +1 -1
- data/lib/semantic_logger/appender/file.rb +138 -30
- data/lib/semantic_logger/appender/graylog.rb +22 -6
- data/lib/semantic_logger/appender/honeybadger.rb +1 -1
- data/lib/semantic_logger/appender/honeybadger_insights.rb +1 -1
- data/lib/semantic_logger/appender/http.rb +70 -11
- data/lib/semantic_logger/appender/io.rb +1 -1
- data/lib/semantic_logger/appender/kafka.rb +26 -12
- data/lib/semantic_logger/appender/loki.rb +13 -3
- data/lib/semantic_logger/appender/mongodb.rb +20 -1
- data/lib/semantic_logger/appender/new_relic.rb +1 -1
- data/lib/semantic_logger/appender/new_relic_logs.rb +9 -11
- data/lib/semantic_logger/appender/open_telemetry.rb +2 -2
- data/lib/semantic_logger/appender/rabbitmq.rb +6 -1
- data/lib/semantic_logger/appender/sentry.rb +1 -1
- data/lib/semantic_logger/appender/sentry_ruby.rb +91 -17
- data/lib/semantic_logger/appender/splunk.rb +10 -1
- data/lib/semantic_logger/appender/syslog.rb +64 -26
- data/lib/semantic_logger/appender/tcp.rb +6 -1
- data/lib/semantic_logger/appender/udp.rb +11 -1
- data/lib/semantic_logger/appender/wrapper.rb +1 -1
- data/lib/semantic_logger/appenders.rb +9 -1
- data/lib/semantic_logger/base.rb +82 -30
- data/lib/semantic_logger/exception_details.rb +50 -0
- data/lib/semantic_logger/formatters/base.rb +52 -6
- data/lib/semantic_logger/formatters/color.rb +18 -2
- data/lib/semantic_logger/formatters/default.rb +22 -5
- data/lib/semantic_logger/formatters/logfmt.rb +153 -21
- data/lib/semantic_logger/formatters/loki.rb +26 -22
- data/lib/semantic_logger/formatters/one_line.rb +67 -3
- data/lib/semantic_logger/formatters/open_telemetry.rb +16 -17
- data/lib/semantic_logger/formatters/pattern.rb +8 -4
- data/lib/semantic_logger/formatters/raw.rb +4 -5
- data/lib/semantic_logger/formatters/syslog.rb +110 -18
- data/lib/semantic_logger/formatters/syslog_cee.rb +29 -16
- data/lib/semantic_logger/levels.rb +10 -1
- data/lib/semantic_logger/log.rb +183 -37
- data/lib/semantic_logger/loggable.rb +26 -3
- data/lib/semantic_logger/metric/new_relic.rb +1 -1
- data/lib/semantic_logger/metric/signalfx.rb +8 -6
- data/lib/semantic_logger/metric/statsd.rb +15 -7
- data/lib/semantic_logger/queue_processor/request.rb +55 -0
- data/lib/semantic_logger/queue_processor/throttled_counter.rb +46 -0
- data/lib/semantic_logger/queue_processor.rb +165 -66
- data/lib/semantic_logger/semantic_logger.rb +15 -2
- data/lib/semantic_logger/subscriber.rb +96 -10
- data/lib/semantic_logger/sync_processor.rb +11 -0
- data/lib/semantic_logger/test/capture_log_events.rb +8 -5
- data/lib/semantic_logger/utils.rb +244 -16
- data/lib/semantic_logger/version.rb +1 -1
- data/lib/semantic_logger.rb +1 -1
- metadata +21 -5
- data/lib/semantic_logger/core_ext/thread.rb +0 -30
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 5b42624348f8dc147462cc1f2d26f7cf0d4ef3df91fe3157a119591e8f6bd23b
|
|
4
|
+
data.tar.gz: ca1f00571b5280a9cd5b4bf0f116944ba4850eb1536b50aff53bfba9b6ba540b
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 31f5b285c287b386979f89b794d71f9084276b13510dc5b7798a510815f512e713bdf44a87743808890a8f4cef1e74a8263a8ca78edba8f7247235bf53aa6fd5
|
|
7
|
+
data.tar.gz: 3f114bc7d9352eb40c1a4e8a31ecfa13b20e0bfec61c72fceaba522ffc041c97ffe50e0f788b63415c4265ac68586a185473a7c4320d59e8f708ac8facedbd72
|
data/README.md
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
# Semantic Logger
|
|
2
|
-
[](https://rubygems.org/gems/semantic_logger) [](https://github.com/reidmorrison/semantic_logger/actions?query=workflow%3Abuild) [](https://rubygems.org/gems/semantic_logger) [](http://opensource.org/licenses/Apache-2.0) 
|
|
2
|
+
[](https://rubygems.org/gems/semantic_logger) [](https://github.com/reidmorrison/semantic_logger/actions?query=workflow%3Abuild) [](https://rubygems.org/gems/semantic_logger) [](https://github.com/reidmorrison/semantic_logger/actions/workflows/ci.yml) [](http://opensource.org/licenses/Apache-2.0) 
|
|
3
3
|
|
|
4
4
|
Semantic Logger is a high-performance, asynchronous structured logging framework for Ruby
|
|
5
5
|
and Rails.
|
|
@@ -38,12 +38,12 @@ replaces the Rails default logger with Semantic Logger automatically.
|
|
|
38
38
|
|
|
39
39
|
## Documentation
|
|
40
40
|
|
|
41
|
-
Start with the [Introduction](https://logger.
|
|
42
|
-
[Programmer's Guide](https://logger.
|
|
41
|
+
Start with the [Introduction](https://logger.reidmorrison.com/), then the
|
|
42
|
+
[Programmer's Guide](https://logger.reidmorrison.com/api.html).
|
|
43
43
|
|
|
44
|
-
* Full guide: [https://logger.
|
|
45
|
-
* For AI assistants: [llms.txt](https://logger.
|
|
46
|
-
[llms-full.txt](https://logger.
|
|
44
|
+
* Full guide: [https://logger.reidmorrison.com/](https://logger.reidmorrison.com/)
|
|
45
|
+
* For AI assistants: [llms.txt](https://logger.reidmorrison.com/llms.txt) (documentation index) and
|
|
46
|
+
[llms-full.txt](https://logger.reidmorrison.com/llms-full.txt) (complete documentation in one file).
|
|
47
47
|
The same pages are also included as markdown in the installed gem, under `docs/`.
|
|
48
48
|
|
|
49
49
|
## Logging Destinations
|
|
@@ -82,7 +82,7 @@ instead of Semantic Logger directly since it will automatically replace the Rail
|
|
|
82
82
|
|
|
83
83
|
## Rocket Job
|
|
84
84
|
|
|
85
|
-
Checkout the sister project [Rocket Job](
|
|
85
|
+
Checkout the sister project [Rocket Job](https://rocketjob.reidmorrison.com): Ruby's missing batch system.
|
|
86
86
|
|
|
87
87
|
Fully supports Semantic Logger when running jobs in the background. Complete support for job metrics
|
|
88
88
|
sent via Semantic Logger to your favorite dashboards.
|
|
@@ -97,9 +97,10 @@ and are therefore not automatically included by this gem:
|
|
|
97
97
|
- HoneybadgerInsights Appender: gem 'honeybadger'
|
|
98
98
|
- NewRelic Appender: gem 'newrelic_rpm'
|
|
99
99
|
- NewRelicLogs Appender: gem 'newrelic_rpm'
|
|
100
|
-
- Syslog Appender: gem '
|
|
101
|
-
- Syslog Appender to a remote
|
|
102
|
-
-
|
|
100
|
+
- Syslog Appender to a local syslog daemon: gem 'syslog' (a bundled gem since Ruby 3.4)
|
|
101
|
+
- Syslog Appender to a remote syslog server over TCP or UDP: gem 'syslog_protocol' 0.9.2 or above
|
|
102
|
+
- Syslog Appender to a remote syslog server over TCP: gem 'net_tcp_client'
|
|
103
|
+
- Splunk Appender (deprecated, use the Splunk HTTP appender): gem 'splunk-sdk-ruby'
|
|
103
104
|
- Elasticsearch Appender: gem 'elasticsearch'
|
|
104
105
|
- OpenSearch Appender: gem 'opensearch-ruby'
|
|
105
106
|
- Kafka Appender: gem 'ruby-kafka'
|
|
@@ -109,7 +110,7 @@ and are therefore not automatically included by this gem:
|
|
|
109
110
|
|
|
110
111
|
## Upgrading
|
|
111
112
|
|
|
112
|
-
See the [Upgrading Guide](https://logger.
|
|
113
|
+
See the [Upgrading Guide](https://logger.reidmorrison.com/upgrading.html) for instructions on
|
|
113
114
|
upgrading between major versions.
|
|
114
115
|
|
|
115
116
|
## Install
|
|
@@ -128,7 +129,7 @@ SemanticLogger.default_level = :trace
|
|
|
128
129
|
SemanticLogger.add_appender(file_name: 'development.log', formatter: :color)
|
|
129
130
|
~~~
|
|
130
131
|
|
|
131
|
-
If running rails, see: [Semantic Logger Rails](https://logger.
|
|
132
|
+
If running rails, see: [Semantic Logger Rails](https://logger.reidmorrison.com/rails.html)
|
|
132
133
|
|
|
133
134
|
## Contributing
|
|
134
135
|
|
data/Rakefile
CHANGED
|
@@ -25,19 +25,30 @@ task :llms_full do
|
|
|
25
25
|
|
|
26
26
|
> Semantic Logger is a high-performance, asynchronous structured logging framework for Ruby and Rails.
|
|
27
27
|
|
|
28
|
-
This file concatenates every page of https://logger.
|
|
28
|
+
This file concatenates every page of https://logger.reidmorrison.com for consumption by AI assistants.
|
|
29
29
|
It is generated from the markdown sources in docs/ by `bundle exec rake llms_full`; do not edit it directly.
|
|
30
|
-
A per-page index is available at https://logger.
|
|
30
|
+
A per-page index is available at https://logger.reidmorrison.com/llms.txt
|
|
31
31
|
HEADER
|
|
32
32
|
|
|
33
33
|
sections = pages.map do |page|
|
|
34
|
-
|
|
34
|
+
raw = File.read("docs/#{page}.md")
|
|
35
|
+
|
|
36
|
+
# The page title lives in front matter, which the shared docs theme renders
|
|
37
|
+
# as the h1. This file strips front matter, so lift the title back out and
|
|
38
|
+
# re-emit it as a heading; without this every section here would open with
|
|
39
|
+
# no indication of which page it is.
|
|
40
|
+
front_matter = raw[/\A---\n(.*?)\n---\n/m, 1].to_s
|
|
41
|
+
title = front_matter[/^(?:title|heading):[ \t]*(.+)$/, 1].to_s.strip.delete_prefix('"').delete_suffix('"')
|
|
42
|
+
|
|
43
|
+
text = raw.
|
|
35
44
|
sub(/\A---\n.*?\n---\n/m, ""). # Jekyll front matter
|
|
36
45
|
gsub(/^\{:.*\}\n/, ""). # kramdown attribute lines ({:toc}, {:.no_toc}, ...)
|
|
37
46
|
gsub(/^\* TOC\n/, "").
|
|
38
47
|
gsub(/^\*\*Contents\*\*\n/, "").
|
|
39
48
|
gsub(/^!\[.*\n/, "") # images (relative paths, useless in plain text)
|
|
40
|
-
|
|
49
|
+
|
|
50
|
+
body = title.empty? ? text.strip : "## #{title}\n\n#{text.strip}"
|
|
51
|
+
"<!-- source: docs/#{page}.md -->\n\n#{body}\n"
|
|
41
52
|
end
|
|
42
53
|
|
|
43
54
|
File.write("docs/llms-full.txt", ([header] + sections).join("\n\n---\n\n"))
|
data/docs/api.md
CHANGED
|
@@ -1,9 +1,12 @@
|
|
|
1
1
|
---
|
|
2
2
|
layout: default
|
|
3
|
+
title: Programming Guide
|
|
4
|
+
description: >-
|
|
5
|
+
How to log with Semantic Logger: getting a logger, log levels, structured
|
|
6
|
+
payloads, exceptions, the block form, measuring durations, and tagging
|
|
7
|
+
related entries.
|
|
3
8
|
---
|
|
4
9
|
|
|
5
|
-
## Programming Guide
|
|
6
|
-
{:.no_toc}
|
|
7
10
|
|
|
8
11
|
**Contents**
|
|
9
12
|
|
|
@@ -175,6 +178,11 @@ logger.error(message: "Calling Supplier", duration: 100)
|
|
|
175
178
|
logger.error(message: "Calling Supplier", metric: "Supplier/inquiry", metric_amount: 21)
|
|
176
179
|
~~~
|
|
177
180
|
|
|
181
|
+
In both forms, a key such as `duration` or `exception` is only used as an argument when its value is
|
|
182
|
+
valid for it; otherwise it is logged as part of the payload. Any other keys supplied alongside
|
|
183
|
+
`payload:` are merged into it, without overwriting its own values. When the Hash holds untrusted
|
|
184
|
+
data, see [Logging untrusted hashes](security.html#logging-untrusted-hashes).
|
|
185
|
+
|
|
178
186
|
## Step 6: Measure how long something takes
|
|
179
187
|
|
|
180
188
|
It is good practice to "measure everything" in production, so that when things slow down it is
|
|
@@ -252,12 +260,16 @@ The second argument to a measure call is a Hash of options:
|
|
|
252
260
|
- `:duration` [Float]: the duration in ms, used when no block is supplied (then it is mandatory; with
|
|
253
261
|
a block it is ignored).
|
|
254
262
|
- `:log_exception` [Symbol]: how to report an exception raised in the block. `:full` logs the class,
|
|
255
|
-
message, and backtrace; `:partial` logs the class and message only; `:
|
|
263
|
+
message, and backtrace; `:partial` logs the class and message only; `:none` does not log it.
|
|
256
264
|
Default `:partial`.
|
|
257
265
|
- `:on_exception_level` [Symbol]: if an exception is raised, raise the log level to this level.
|
|
258
266
|
- `:silence` [Symbol]: the level to silence other log messages to within the block (current thread
|
|
259
267
|
only).
|
|
260
268
|
|
|
269
|
+
An option with a value that is not valid for it, such as a non-numeric `:min_duration` or an unknown
|
|
270
|
+
level, is ignored and its default is used. Other keys in the Hash are not logged; pass the data to
|
|
271
|
+
log as `:payload`.
|
|
272
|
+
|
|
261
273
|
Putting several together:
|
|
262
274
|
|
|
263
275
|
~~~ruby
|
|
@@ -347,11 +359,11 @@ The features above cover everyday logging. The rest of this guide covers less co
|
|
|
347
359
|
|
|
348
360
|
### Name your threads
|
|
349
361
|
|
|
350
|
-
Semantic Logger includes the thread name
|
|
351
|
-
|
|
362
|
+
Semantic Logger includes the thread name in every entry. A thread that has no name is identified by
|
|
363
|
+
its object id instead:
|
|
352
364
|
|
|
353
365
|
~~~
|
|
354
|
-
2013-11-07 16:25:14.279 I [35841:
|
|
366
|
+
2013-11-07 16:25:14.279 I [35841:16] (0.0ms) ExternalSupplier -- Calling external interface
|
|
355
367
|
~~~
|
|
356
368
|
|
|
357
369
|
Give a thread a readable name so it stands out in the logs:
|
|
@@ -364,15 +376,29 @@ Thread.current.name = "User calculation thread 32"
|
|
|
364
376
|
2013-11-07 16:26:02.744 I [35841:User calculation thread 32] (0.0ms) ExternalSupplier -- Calling external interface
|
|
365
377
|
~~~
|
|
366
378
|
|
|
367
|
-
Keep the name unique, otherwise concurrent threads are hard to tell apart.
|
|
368
|
-
|
|
379
|
+
Keep the name unique, otherwise concurrent threads are hard to tell apart. For example, number the
|
|
380
|
+
threads in a pool:
|
|
369
381
|
|
|
370
382
|
~~~ruby
|
|
371
|
-
|
|
383
|
+
workers = Array.new(5) do |i|
|
|
384
|
+
Thread.new do
|
|
385
|
+
Thread.current.name = "calc #{i + 1}"
|
|
386
|
+
# ...
|
|
387
|
+
end
|
|
388
|
+
end
|
|
372
389
|
~~~
|
|
373
390
|
|
|
374
|
-
|
|
375
|
-
|
|
391
|
+
This is Ruby's own `Thread#name=`, so the name must be a String, and Ruby raises for one that
|
|
392
|
+
contains a NUL byte or is not in an ASCII-compatible encoding. Ruby also gives the name to the
|
|
393
|
+
operating system, so it shows up in tools that list threads, such as `top -H`, debuggers, and
|
|
394
|
+
profilers. Log entries show the whole name, but Linux keeps only its first 15 bytes, so keep the
|
|
395
|
+
name short, or put the part that tells threads apart first. Semantic Logger keeps its own thread
|
|
396
|
+
names short for the same reason, naming each after its appender, such as `SL Appenders` and
|
|
397
|
+
`SL File`.
|
|
398
|
+
|
|
399
|
+
On JRuby, naming a thread also names the underlying JVM thread, which shows up when monitoring the
|
|
400
|
+
JVM with tools such as jconsole or jstack. JRuby names the main thread `main`, and a thread created
|
|
401
|
+
by Java code, such as a servlet container's request thread, is logged with its JVM thread name.
|
|
376
402
|
|
|
377
403
|
### Change one class's level at runtime
|
|
378
404
|
|
data/docs/appenders.md
CHANGED
|
@@ -1,9 +1,12 @@
|
|
|
1
1
|
---
|
|
2
2
|
layout: default
|
|
3
|
+
title: Appenders
|
|
4
|
+
description: >-
|
|
5
|
+
Every destination Semantic Logger writes to and how to configure it: files
|
|
6
|
+
and streams, structured formats, network protocols, log aggregators, and
|
|
7
|
+
error monitors.
|
|
3
8
|
---
|
|
4
9
|
|
|
5
|
-
## Appenders
|
|
6
|
-
{:.no_toc}
|
|
7
10
|
|
|
8
11
|
**Contents**
|
|
9
12
|
|
|
@@ -83,7 +86,7 @@ time the appender is used, and is never a hard dependency of Semantic Logger its
|
|
|
83
86
|
| [HTTP(S)](#https) | `appender: :http` | |
|
|
84
87
|
| [TCP (+SSL)](#tcp-appender-ssl) | `appender: :tcp` | `net_tcp_client` |
|
|
85
88
|
| [UDP](#udp-appender) | `appender: :udp` | |
|
|
86
|
-
| [Syslog](#syslog) | `appender: :syslog` | `syslog_protocol
|
|
89
|
+
| [Syslog](#syslog) | `appender: :syslog` | `syslog` (local, Ruby 3.4+), `syslog_protocol` (remote), `net_tcp_client` (TCP) |
|
|
87
90
|
|
|
88
91
|
**Centralized logging and log aggregators**
|
|
89
92
|
|
|
@@ -93,7 +96,7 @@ time the appender is used, and is never a hard dependency of Semantic Logger its
|
|
|
93
96
|
| [OpenSearch](#opensearch) | `appender: :opensearch` | `opensearch-ruby` |
|
|
94
97
|
| [Graylog](#graylog) | `appender: :graylog` | `gelf` |
|
|
95
98
|
| [Splunk over HTTP](#splunk-http) | `appender: :splunk_http` | |
|
|
96
|
-
| [Splunk SDK](#splunk-sdk) | `appender: :splunk` | `splunk-sdk-ruby` |
|
|
99
|
+
| [Splunk SDK](#splunk-sdk) (deprecated) | `appender: :splunk` | `splunk-sdk-ruby` |
|
|
97
100
|
| [Grafana Loki](#grafana-loki) | `appender: :loki` | |
|
|
98
101
|
| [CloudWatch Logs](#cloudwatch-logs) | `appender: :cloudwatch_logs` | `aws-sdk-cloudwatchlogs` |
|
|
99
102
|
| [OpenTelemetry](#opentelemetry) | `appender: :open_telemetry` | `opentelemetry-logs-sdk` |
|
|
@@ -158,6 +161,14 @@ file is created and to an existing log file:
|
|
|
158
161
|
SemanticLogger.add_appender(file_name: "production.log", permissions: 0o640)
|
|
159
162
|
~~~
|
|
160
163
|
|
|
164
|
+
With `permissions:`, the log file must be a regular file owned by the process, with a single hard
|
|
165
|
+
link, that the file name names directly rather than through a symbolic link. Nothing is written to
|
|
166
|
+
any other file, such as a log file linked to `/dev/stdout`, nor when the file cannot be opened:
|
|
167
|
+
`add_appender` raises `IOError` giving the reason, and a file that fails the check when it is
|
|
168
|
+
reopened later, for example on a rotation, is reported and not written to until it is fixed. To log
|
|
169
|
+
to standard output, use `io: $stdout` instead. See
|
|
170
|
+
[Log file permissions](security.html#log-file-permissions).
|
|
171
|
+
|
|
161
172
|
### IO Streams
|
|
162
173
|
|
|
163
174
|
Log to any IO stream instance, such as `$stdout` or `$stderr`:
|
|
@@ -259,6 +270,9 @@ Notes:
|
|
|
259
270
|
* The layout above is formatted for readability. The real output is a single line terminated by one
|
|
260
271
|
newline, with no embedded newlines.
|
|
261
272
|
* A field with a `nil` value is excluded from the output.
|
|
273
|
+
* To also log the detail Ruby adds to an exception's message, such as did_you_mean suggestions, as
|
|
274
|
+
`exception.details`, enable `log_exception_details`: `formatter: {json: {log_exception_details: true}}`.
|
|
275
|
+
See [Exception details](config.html#exception-details).
|
|
262
276
|
|
|
263
277
|
### Fluentd
|
|
264
278
|
|
|
@@ -351,13 +365,30 @@ timestamp="2024-07-20T08:32:05.375276Z" level="info" name="MyClass" message="Hel
|
|
|
351
365
|
|
|
352
366
|
Notes:
|
|
353
367
|
|
|
354
|
-
* The timestamp is ISO 8601
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
368
|
+
* The timestamp is ISO 8601, under the key `timestamp` unless the formatter's `time_key` is set
|
|
369
|
+
(`formatter: {logfmt: {time_key: :ts}}`). All values are quoted and escaped, so a newline in the
|
|
370
|
+
data cannot forge or split a record.
|
|
371
|
+
* Keys (from tags, named tags, and the payload) cannot be quoted in logfmt, so spaces, control
|
|
372
|
+
characters (including the Unicode line and paragraph separators), invisible format characters
|
|
373
|
+
(such as a zero width space), `=`, and `"` in a key are replaced with `_`, and invalid UTF-8 is
|
|
374
|
+
dropped. For example the tag `second breakfast` is emitted as `second_breakfast=true`.
|
|
375
|
+
* Named tags and payload fields are merged into the top-level `key=value` pairs. A key named like
|
|
376
|
+
one of the entry's own fields (the time key, `level`, `name`, `message`, `duration`,
|
|
377
|
+
`duration_ms`, `tag`, or an exception field) is emitted with its source as a prefix, so that
|
|
378
|
+
logged data cannot replace those fields: `payload.level="fatal"`, `named_tags.message="..."`, or
|
|
379
|
+
`tags.message=true`. A key that starts with `tags.`, `named_tags.`, or `payload.` is prefixed
|
|
380
|
+
too, as in `payload.named_tags.level`, so that it cannot pass for another source's field.
|
|
381
|
+
* Other keys that are the same once encoded, including keys that only differ by type (such as
|
|
382
|
+
`:user` and `"user"`), are emitted once, with the value from the later source: the payload wins
|
|
383
|
+
over named tags, and named tags over tags.
|
|
358
384
|
* Unnamed tags are emitted as keys with a `true` value.
|
|
385
|
+
* A nested value, such as a Hash or an Array in the payload, is emitted as its JSON text, quoted:
|
|
386
|
+
`user="{\"id\":7}"`.
|
|
359
387
|
* `tag="success"` is emitted on every entry, becoming `tag="exception"` when the entry carries an
|
|
360
|
-
exception
|
|
388
|
+
exception, which is then included as the fields `exception_name`, `exception_message`,
|
|
389
|
+
`stack_trace`, and, when the exception has a cause, `cause`. With `log_exception_details: true`
|
|
390
|
+
(off by default), the detail Ruby adds to the exception's message is included as
|
|
391
|
+
`exception_details`; see [Exception details](config.html#exception-details).
|
|
361
392
|
|
|
362
393
|
## Network protocols
|
|
363
394
|
|
|
@@ -388,6 +419,31 @@ end
|
|
|
388
419
|
SemanticLogger.add_appender(appender: :http, url: "https://localhost:8088/path", formatter: formatter)
|
|
389
420
|
~~~
|
|
390
421
|
|
|
422
|
+
#### Query parameters
|
|
423
|
+
|
|
424
|
+
Some log servers are configured through the query string of the ingestion url rather than through
|
|
425
|
+
headers or the body. A query string on `url:` is kept and sent with every request that submits log
|
|
426
|
+
data, for example when pointing at
|
|
427
|
+
[VictoriaLogs](https://docs.victoriametrics.com/victorialogs/data-ingestion/):
|
|
428
|
+
|
|
429
|
+
~~~ruby
|
|
430
|
+
SemanticLogger.add_appender(
|
|
431
|
+
appender: :http,
|
|
432
|
+
url: "http://localhost:9428/insert/jsonline?_msg_field=message&_time_field=timestamp"
|
|
433
|
+
)
|
|
434
|
+
~~~
|
|
435
|
+
|
|
436
|
+
The query is applied after the request path, so it also works for the appenders built on top of the
|
|
437
|
+
HTTP appender (`:elasticsearch_http`, `:splunk_http`, `:loki`, and the `:signalfx` metrics appender),
|
|
438
|
+
each of which appends its own path to the url:
|
|
439
|
+
|
|
440
|
+
~~~ruby
|
|
441
|
+
SemanticLogger.add_appender(
|
|
442
|
+
appender: :elasticsearch_http,
|
|
443
|
+
url: "http://localhost:9428/insert/elasticsearch?_msg_field=message&_time_field=timestamp"
|
|
444
|
+
)
|
|
445
|
+
~~~
|
|
446
|
+
|
|
391
447
|
#### Batching
|
|
392
448
|
|
|
393
449
|
By default each entry is sent in its own HTTP request. To send multiple entries in one request as a
|
|
@@ -424,11 +480,12 @@ SemanticLogger.add_appender(appender: :tcp, server: "localhost:8088")
|
|
|
424
480
|
# TCP with SSL:
|
|
425
481
|
SemanticLogger.add_appender(appender: :tcp, server: "localhost:8088", ssl: true)
|
|
426
482
|
|
|
427
|
-
# With self-signed
|
|
483
|
+
# With a self-signed certificate, or one from a private certificate authority, trust that
|
|
484
|
+
# certificate instead of disabling verification:
|
|
428
485
|
SemanticLogger.add_appender(
|
|
429
486
|
appender: :tcp,
|
|
430
487
|
server: "localhost:8088",
|
|
431
|
-
ssl: {
|
|
488
|
+
ssl: {ca_file: "/path/to/ca.pem"}
|
|
432
489
|
)
|
|
433
490
|
~~~
|
|
434
491
|
|
|
@@ -449,8 +506,10 @@ options.
|
|
|
449
506
|
|
|
450
507
|
The TCP and UDP appenders separate records with a newline, so they default to the JSON formatter,
|
|
451
508
|
which escapes embedded newlines and is safe with untrusted data. If you switch to a text formatter
|
|
452
|
-
such as `:default` or `:color`, enable `escape_control_chars` so a newline in the data cannot
|
|
453
|
-
|
|
509
|
+
such as `:default` or `:color`, enable `escape_control_chars` so a newline in the logged data cannot
|
|
510
|
+
forge a record. Those formatters still write an entry with an exception over several lines (the
|
|
511
|
+
exception, then its backtrace), so a receiver that frames records by newline splits such an entry;
|
|
512
|
+
use the JSON formatter or the `:syslog` formatter when every record must be a single line:
|
|
454
513
|
|
|
455
514
|
~~~ruby
|
|
456
515
|
SemanticLogger.add_appender(
|
|
@@ -482,7 +541,8 @@ SemanticLogger.add_appender(appender: :udp, server: "localhost:8088", formatter:
|
|
|
482
541
|
|
|
483
542
|
### Syslog
|
|
484
543
|
|
|
485
|
-
Log to a local Syslog daemon
|
|
544
|
+
Log to a local Syslog daemon. Since Ruby 3.4 `syslog` is a bundled gem, so add `gem "syslog"` to the
|
|
545
|
+
Gemfile:
|
|
486
546
|
|
|
487
547
|
~~~ruby
|
|
488
548
|
SemanticLogger.add_appender(appender: :syslog)
|
|
@@ -495,12 +555,32 @@ The default packet size is 1024 bytes:
|
|
|
495
555
|
SemanticLogger.add_appender(appender: :syslog, url: "tcp://myloghost:514", max_size: 2048)
|
|
496
556
|
~~~
|
|
497
557
|
|
|
558
|
+
`max_size` also applies when `formatter:` names the `:syslog` or `:syslog_cee` formatter. A
|
|
559
|
+
formatter passed as an instance uses its own `max_size`, which also defaults to 1024 bytes:
|
|
560
|
+
|
|
561
|
+
~~~ruby
|
|
562
|
+
SemanticLogger.add_appender(appender: :syslog, url: "udp://myloghost:514", formatter: :syslog_cee, max_size: 2048)
|
|
563
|
+
~~~
|
|
564
|
+
|
|
498
565
|
Or over UDP (the `syslog_protocol` gem is required):
|
|
499
566
|
|
|
500
567
|
~~~ruby
|
|
501
568
|
SemanticLogger.add_appender(appender: :syslog, url: "udp://myloghost:514")
|
|
502
569
|
~~~
|
|
503
570
|
|
|
571
|
+
Set the facility with one of Ruby's `::Syslog::LOG_*` facility constants (the default is
|
|
572
|
+
`::Syslog::LOG_USER`). For remote syslog without the `syslog` gem, the same constants are available
|
|
573
|
+
from `SemanticLogger::Formatters::Syslog`, for example `SemanticLogger::Formatters::Syslog::LOG_LOCAL0`:
|
|
574
|
+
|
|
575
|
+
~~~ruby
|
|
576
|
+
SemanticLogger.add_appender(appender: :syslog, url: "udp://myloghost:514", facility: ::Syslog::LOG_LOCAL0)
|
|
577
|
+
~~~
|
|
578
|
+
|
|
579
|
+
The `facility:`, `level_map:`, and `max_size:` options also apply when `formatter:` names the `:syslog` or
|
|
580
|
+
`:syslog_cee` formatter, for example `formatter: :syslog_cee`. Options given to the formatter itself,
|
|
581
|
+
as in `formatter: {syslog: {facility: ::Syslog::LOG_LOCAL1}}`, take precedence. A formatter passed
|
|
582
|
+
as an instance keeps its own settings.
|
|
583
|
+
|
|
504
584
|
Add a filter to exclude noisy entries such as health checks:
|
|
505
585
|
|
|
506
586
|
~~~ruby
|
|
@@ -512,8 +592,10 @@ SemanticLogger.add_appender(
|
|
|
512
592
|
~~~
|
|
513
593
|
|
|
514
594
|
Syslog frames each record, so embedded newlines or other control characters in untrusted data could
|
|
515
|
-
forge or split records. The syslog formatters therefore escape control characters by default.
|
|
516
|
-
|
|
595
|
+
forge or split records. The syslog formatters therefore escape control characters by default. The
|
|
596
|
+
`:syslog` formatter also escapes the newlines between an exception and its backtrace lines (written
|
|
597
|
+
as `\n`), so that an entry with an exception is a single line. To pass control characters through
|
|
598
|
+
unchanged, which also restores the multi-line exception layout:
|
|
517
599
|
|
|
518
600
|
~~~ruby
|
|
519
601
|
SemanticLogger.add_appender(
|
|
@@ -603,24 +685,13 @@ instead, which reached about 1,400 entries per second (1,200 with SSL).
|
|
|
603
685
|
|
|
604
686
|
### Splunk SDK
|
|
605
687
|
|
|
606
|
-
`appender: :splunk`
|
|
607
|
-
|
|
608
|
-
|
|
688
|
+
> **Deprecated:** `appender: :splunk` is deprecated and will be removed in v6. Use the
|
|
689
|
+
> [HTTP Event Collector](#splunk-http) (`appender: :splunk_http`) or the
|
|
690
|
+
> [TCP appender](#tcp-appender-ssl) instead.
|
|
609
691
|
|
|
610
|
-
|
|
611
|
-
|
|
612
|
-
|
|
613
|
-
username: "username",
|
|
614
|
-
password: "password",
|
|
615
|
-
host: "localhost",
|
|
616
|
-
port: 8089,
|
|
617
|
-
scheme: :https,
|
|
618
|
-
index: "main"
|
|
619
|
-
)
|
|
620
|
-
~~~
|
|
621
|
-
|
|
622
|
-
For most deployments the [HTTP Event Collector](#splunk-http) or the [TCP appender](#tcp-appender-ssl)
|
|
623
|
-
is the simpler choice; use the SDK appender when the management API is the only ingress available.
|
|
692
|
+
`appender: :splunk` submits entries through the `splunk-sdk-ruby` gem to the Splunk management
|
|
693
|
+
port. That gem was last released in 2015 and does not verify the server's TLS certificate, so the
|
|
694
|
+
connection, including the login credentials, is open to interception.
|
|
624
695
|
|
|
625
696
|
### Grafana Loki
|
|
626
697
|
|
|
@@ -638,7 +709,8 @@ SemanticLogger.add_appender(
|
|
|
638
709
|
~~~
|
|
639
710
|
|
|
640
711
|
Set the URL, username, and password to match your Loki instance. Set `compress: true` to compress the
|
|
641
|
-
log messages.
|
|
712
|
+
log messages. The ingestion path is appended to the url, so a query string on the url is kept and sent
|
|
713
|
+
with every push.
|
|
642
714
|
|
|
643
715
|
### CloudWatch Logs
|
|
644
716
|
|
|
@@ -804,19 +876,27 @@ Some logging context is forwarded to Sentry:
|
|
|
804
876
|
|
|
805
877
|
* From named tags, `transaction_name`. See
|
|
806
878
|
<https://docs.sentry.io/platforms/ruby/enriching-events/transaction-name/>.
|
|
807
|
-
* From named tags
|
|
879
|
+
* From named tags, `user` is built from the `user_id`, `username`, `user_email`, and
|
|
808
880
|
`ip_address` keys, plus any `user` key that is itself a hash. See
|
|
809
881
|
<https://docs.sentry.io/platforms/ruby/enriching-events/identify-user/>.
|
|
810
|
-
* From
|
|
882
|
+
* From named tags, `fingerprint` (an Array of Strings, Symbols, or numbers) configures grouping
|
|
883
|
+
granularity. See
|
|
811
884
|
<https://docs.sentry.io/platforms/ruby/usage/sdk-fingerprinting/>.
|
|
812
885
|
* Named tags are sent as Sentry tags. See
|
|
813
886
|
<https://docs.sentry.io/platforms/ruby/enriching-events/tags/>.
|
|
814
887
|
* Unnamed tags are sent as the `:tag` tag, separated by commas.
|
|
815
|
-
* Everything else from the payload and context is added to `extras
|
|
888
|
+
* Everything else from the payload and context is added to `extras`, including a value of the
|
|
889
|
+
wrong type for one of the keys above.
|
|
890
|
+
|
|
891
|
+
> **Deprecated:** the `user` and `fingerprint` keys are also still taken from the payload, but that
|
|
892
|
+
> is deprecated and will be removed in v6, since the payload often holds untrusted data, such as a
|
|
893
|
+
> request body, that could set the user an error is reported for. Set them with named tags. The
|
|
894
|
+
> named tags win over the payload, and a user field is only taken from the payload when it is a
|
|
895
|
+
> String, Symbol, or Integer.
|
|
816
896
|
|
|
817
897
|
~~~ruby
|
|
818
|
-
SemanticLogger.tagged(transaction_name: "foo", user_id: 42, baz: "quz") do
|
|
819
|
-
logger.error("some message"
|
|
898
|
+
SemanticLogger.tagged(transaction_name: "foo", user_id: 42, username: "joe", fingerprint: ["bar"], baz: "quz") do
|
|
899
|
+
logger.error("some message")
|
|
820
900
|
end
|
|
821
901
|
~~~
|
|
822
902
|
|
|
@@ -868,6 +948,10 @@ SemanticLogger.on_log do |log|
|
|
|
868
948
|
end
|
|
869
949
|
~~~
|
|
870
950
|
|
|
951
|
+
It sends the whole entry, payload included, so register it after any `on_log` subscriber that
|
|
952
|
+
[redacts sensitive data](security.html#redacting-sensitive-data): subscribers run in the order they
|
|
953
|
+
were registered.
|
|
954
|
+
|
|
871
955
|
## Databases and message queues
|
|
872
956
|
|
|
873
957
|
### MongoDB
|