semantic_logger 5.2.0 → 5.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/docs/api.md +160 -9
- data/docs/appenders.md +461 -34
- data/docs/config.md +543 -36
- data/docs/log.md +38 -7
- data/docs/metrics.md +139 -28
- data/docs/operations.md +293 -24
- data/docs/rails.md +319 -5
- data/docs/security.md +135 -107
- data/docs/testing.md +23 -0
- data/docs/upgrading.md +644 -1
- data/lib/semantic_logger/appender/async.rb +29 -11
- data/lib/semantic_logger/appender/bugsnag.rb +7 -2
- data/lib/semantic_logger/appender/cloudwatch_logs.rb +46 -6
- data/lib/semantic_logger/appender/elasticsearch_base.rb +197 -29
- data/lib/semantic_logger/appender/elasticsearch_http.rb +22 -5
- data/lib/semantic_logger/appender/file.rb +29 -14
- data/lib/semantic_logger/appender/graylog.rb +81 -7
- data/lib/semantic_logger/appender/honeybadger.rb +7 -1
- data/lib/semantic_logger/appender/honeybadger_insights.rb +8 -2
- data/lib/semantic_logger/appender/http.rb +244 -41
- data/lib/semantic_logger/appender/io.rb +11 -6
- data/lib/semantic_logger/appender/kafka.rb +192 -46
- data/lib/semantic_logger/appender/loki.rb +42 -6
- data/lib/semantic_logger/appender/mongodb.rb +94 -9
- data/lib/semantic_logger/appender/new_relic.rb +2 -3
- data/lib/semantic_logger/appender/new_relic_logs.rb +30 -8
- data/lib/semantic_logger/appender/open_telemetry.rb +37 -35
- data/lib/semantic_logger/appender/rabbitmq.rb +241 -17
- data/lib/semantic_logger/appender/sentry.rb +2 -3
- data/lib/semantic_logger/appender/sentry_ruby.rb +82 -15
- data/lib/semantic_logger/appender/splunk.rb +4 -4
- data/lib/semantic_logger/appender/splunk_http.rb +92 -1
- data/lib/semantic_logger/appender/syslog.rb +89 -10
- data/lib/semantic_logger/appender/tcp.rb +68 -98
- data/lib/semantic_logger/appender/text_output.rb +65 -0
- data/lib/semantic_logger/appender/udp.rb +130 -18
- data/lib/semantic_logger/appender/wrapper.rb +10 -7
- data/lib/semantic_logger/appender.rb +115 -10
- data/lib/semantic_logger/appenders.rb +136 -4
- data/lib/semantic_logger/backtrace_cleaning.rb +77 -0
- data/lib/semantic_logger/base.rb +53 -71
- data/lib/semantic_logger/call_site.rb +235 -0
- data/lib/semantic_logger/context.rb +222 -0
- data/lib/semantic_logger/core_ext/process.rb +20 -5
- data/lib/semantic_logger/core_ext/warning.rb +70 -0
- data/lib/semantic_logger/deprecation.rb +139 -0
- data/lib/semantic_logger/exception_details.rb +4 -3
- data/lib/semantic_logger/formatters/base.rb +139 -19
- data/lib/semantic_logger/formatters/color.rb +3 -4
- data/lib/semantic_logger/formatters/default.rb +35 -7
- data/lib/semantic_logger/formatters/ecs.rb +9 -4
- data/lib/semantic_logger/formatters/json.rb +9 -0
- data/lib/semantic_logger/formatters/logfmt.rb +9 -7
- data/lib/semantic_logger/formatters/loki/labels.rb +282 -0
- data/lib/semantic_logger/formatters/loki.rb +412 -24
- data/lib/semantic_logger/formatters/new_relic_logs.rb +15 -13
- data/lib/semantic_logger/formatters/one_line.rb +44 -10
- data/lib/semantic_logger/formatters/open_telemetry.rb +186 -29
- data/lib/semantic_logger/formatters/pattern.rb +60 -8
- data/lib/semantic_logger/formatters/raw.rb +38 -8
- data/lib/semantic_logger/formatters/signalfx.rb +55 -15
- data/lib/semantic_logger/formatters/syslog.rb +4 -2
- data/lib/semantic_logger/formatters/syslog_cee.rb +10 -4
- data/lib/semantic_logger/formatters/syslog_header.rb +49 -0
- data/lib/semantic_logger/formatters.rb +1 -0
- data/lib/semantic_logger/log.rb +236 -80
- data/lib/semantic_logger/loggable.rb +55 -95
- data/lib/semantic_logger/logger.rb +31 -5
- data/lib/semantic_logger/metric/new_relic.rb +32 -17
- data/lib/semantic_logger/metric/signalfx.rb +21 -12
- data/lib/semantic_logger/metric/statsd.rb +26 -8
- data/lib/semantic_logger/processor.rb +194 -5
- data/lib/semantic_logger/queue_processor/backoff.rb +140 -0
- data/lib/semantic_logger/queue_processor/batch_failure.rb +69 -0
- data/lib/semantic_logger/queue_processor/held_write.rb +62 -0
- data/lib/semantic_logger/queue_processor/inbox.rb +97 -0
- data/lib/semantic_logger/queue_processor/reopener.rb +56 -0
- data/lib/semantic_logger/queue_processor/reporter.rb +113 -0
- data/lib/semantic_logger/queue_processor/request.rb +48 -10
- data/lib/semantic_logger/queue_processor/throttled_counter.rb +60 -17
- data/lib/semantic_logger/queue_processor.rb +426 -176
- data/lib/semantic_logger/ractor_copy.rb +120 -0
- data/lib/semantic_logger/ractor_processor.rb +82 -0
- data/lib/semantic_logger/ractor_receiver.rb +282 -0
- data/lib/semantic_logger/ractor_support.rb +28 -0
- data/lib/semantic_logger/redactor.rb +310 -0
- data/lib/semantic_logger/semantic_logger.rb +313 -79
- data/lib/semantic_logger/subscriber.rb +169 -15
- data/lib/semantic_logger/sync_processor.rb +226 -26
- data/lib/semantic_logger/test/capture_log_events.rb +20 -2
- data/lib/semantic_logger/test/minitest.rb +2 -1
- data/lib/semantic_logger/test/rspec.rb +1 -1
- data/lib/semantic_logger/trap_safe.rb +113 -0
- data/lib/semantic_logger/utils.rb +266 -79
- data/lib/semantic_logger/version.rb +1 -1
- data/lib/semantic_logger.rb +35 -2
- metadata +22 -2
data/docs/config.md
CHANGED
|
@@ -62,10 +62,12 @@ To log everything, set the global default to `:trace`:
|
|
|
62
62
|
SemanticLogger.default_level = :trace
|
|
63
63
|
~~~
|
|
64
64
|
|
|
65
|
-
Every logger
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
65
|
+
Every logger uses this global default unless it has been given its own level. Once a logger has an
|
|
66
|
+
explicit level, changing `SemanticLogger.default_level` no longer affects it. An appender does not
|
|
67
|
+
use the global default: without a level of its own it writes every entry that the loggers emit, and
|
|
68
|
+
with one it keeps only those at or above it (see [Per-appender settings](#per-appender-settings)).
|
|
69
|
+
The global default can also be changed at runtime (see [Signals](operations.html#linux-signals) for
|
|
70
|
+
changing it in a running process without a restart):
|
|
69
71
|
|
|
70
72
|
~~~ruby
|
|
71
73
|
SemanticLogger.default_level = :debug
|
|
@@ -93,10 +95,16 @@ When not set explicitly, these default to:
|
|
|
93
95
|
Each value can also be overridden for a single appender by passing `application:`, `environment:`,
|
|
94
96
|
or `host:` to `add_appender` (see [Per-appender settings](#per-appender-settings)).
|
|
95
97
|
|
|
98
|
+
The default and `:color` text formatters do not print these values. To see them in text output, use
|
|
99
|
+
the `:pattern` formatter with `%{application}`, `%{environment}`, or `%{host}`, or a structured
|
|
100
|
+
formatter such as `:json`.
|
|
101
|
+
|
|
96
102
|
### Capturing backtraces
|
|
97
103
|
|
|
98
|
-
Semantic Logger can capture the
|
|
99
|
-
|
|
104
|
+
Semantic Logger can capture the backtrace of the code that created each log entry, include its file
|
|
105
|
+
name and line number in the output, and forward it to error services such as Bugsnag. To record only
|
|
106
|
+
where each entry was logged, at a fraction of the cost, see
|
|
107
|
+
[Recording where each entry was logged](#recording-where-each-entry-was-logged) below.
|
|
100
108
|
|
|
101
109
|
Capturing a backtrace is expensive, so it is controlled by its own level, which defaults to
|
|
102
110
|
`:error`. Only entries at this level or higher capture a backtrace:
|
|
@@ -109,6 +117,216 @@ SemanticLogger.backtrace_level = :error
|
|
|
109
117
|
To capture a backtrace for every entry, set it to `:trace`. To turn backtrace capture off entirely,
|
|
110
118
|
set it to `nil`. It is strongly recommended to leave this at `:error` or higher in production.
|
|
111
119
|
|
|
120
|
+
### Recording where each entry was logged
|
|
121
|
+
|
|
122
|
+
To record the file, line, and method of the code that made each log call, set `call_site_level`.
|
|
123
|
+
Entries at this level or higher record their call site. It is off (`nil`) by default:
|
|
124
|
+
|
|
125
|
+
~~~ruby
|
|
126
|
+
# Record where :info and higher entries were logged
|
|
127
|
+
SemanticLogger.call_site_level = :info
|
|
128
|
+
|
|
129
|
+
# Every entry
|
|
130
|
+
SemanticLogger.call_site_level = :trace
|
|
131
|
+
|
|
132
|
+
# Off (the default)
|
|
133
|
+
SemanticLogger.call_site_level = nil
|
|
134
|
+
~~~
|
|
135
|
+
|
|
136
|
+
In Rails, set it in `config/application.rb` with `config.semantic_logger.call_site_level = :info`.
|
|
137
|
+
|
|
138
|
+
The text formatters then show the method after the file name and line number, and the JSON formatter
|
|
139
|
+
adds a `method` field:
|
|
140
|
+
|
|
141
|
+
~~~
|
|
142
|
+
2026-11-15 13:58:05.906754 I [8462:main order.rb:42 Order#submit] Order -- Submitted
|
|
143
|
+
~~~
|
|
144
|
+
|
|
145
|
+
~~~json
|
|
146
|
+
{"file":"/app/models/order.rb","line":42,"method":"Order#submit","name":"Order","message":"Submitted"}
|
|
147
|
+
~~~
|
|
148
|
+
|
|
149
|
+
Unlike a backtrace, the call site is found by reading only the stack frames up to the code that made
|
|
150
|
+
the log call. A backtrace builds a String for every frame of the stack, so its cost grows with the
|
|
151
|
+
depth of the stack:
|
|
152
|
+
|
|
153
|
+
| | `backtrace_level` | `call_site_level` |
|
|
154
|
+
|-|-------------------|-------------------|
|
|
155
|
+
| Records | The whole stack, as an Array of Strings | The file, line, and method of the log call |
|
|
156
|
+
| Time added to a `logger.info` call, 100 frames deep | About 16 µs (Ruby 3.2) to 26 µs (Ruby 4.0) | About 1.5 µs |
|
|
157
|
+
| On a deeper stack | Grows with the depth | The same |
|
|
158
|
+
| Default | `:error` | `nil` (off) |
|
|
159
|
+
|
|
160
|
+
(Measured on an Apple M5; a `logger.info` call without either takes about 1.3 µs. Reading the stack
|
|
161
|
+
costs much more on JRuby: about 25 µs for the call site, and 180 µs for a backtrace.) The call site is
|
|
162
|
+
cheap enough to record for every entry in most applications, but it still about doubles the time a
|
|
163
|
+
log call spends on the thread that logs, so measure it when an application logs at a very high rate.
|
|
164
|
+
|
|
165
|
+
The method is the label Ruby gives the frame in a backtrace:
|
|
166
|
+
|
|
167
|
+
| Code that logged | Ruby 3.4 and later | Ruby 3.2, 3.3, and JRuby |
|
|
168
|
+
|------------------|--------------------|--------------------------|
|
|
169
|
+
| Instance method | `Order#submit` | `submit` |
|
|
170
|
+
| Class method | `Order.find_open` | `find_open` |
|
|
171
|
+
| Block inside a method | `block in Order#submit` | `block in submit` |
|
|
172
|
+
| Method defined with `define_method` | `block in <class:Order>` | `block in <class:Order>` |
|
|
173
|
+
| Outside any method | `<main>`, `<class:Order>` | `<main>`, `<class:Order>` |
|
|
174
|
+
|
|
175
|
+
JRuby gives a method defined with `define_method` its name, such as `submit`.
|
|
176
|
+
|
|
177
|
+
Where each formatter writes the call site:
|
|
178
|
+
|
|
179
|
+
| Formatter | File and line | Method |
|
|
180
|
+
|-----------|---------------|--------|
|
|
181
|
+
| `:default`, `:color`, `:one_line`, `:syslog` | `order.rb:42`, after the thread name | After the line number |
|
|
182
|
+
| `:pattern` | `%{file_name}`, `%{line}` | `%{method}` |
|
|
183
|
+
| `:json`, `:raw`, `:syslog_cee`, `:fluentd` (with `need_process_info: true`), and the appenders that use them | `file`, `line` | `method` |
|
|
184
|
+
| `:ecs` | `log.origin.file.name`, `log.origin.file.line` | `log.origin.function` |
|
|
185
|
+
| `:new_relic_logs` | `file.name`, `line.number` | `method.name` |
|
|
186
|
+
| `:open_telemetry` | `code.file.path`, `code.line.number` | `code.function.name` |
|
|
187
|
+
|
|
188
|
+
The `:logfmt` and `:loki` formatters write neither. A custom formatter reads the call site from
|
|
189
|
+
[`log.call_site`](log.html#attributes).
|
|
190
|
+
|
|
191
|
+
Which code counts as the call site:
|
|
192
|
+
|
|
193
|
+
* When the log call, or a `measure_*` call, supplies `call_site:`, that call site. Code that logs on
|
|
194
|
+
behalf of other code, such as a log subscriber, can say which code the entry is about with a
|
|
195
|
+
`Thread::Backtrace::Location` (for example from `Thread.each_caller_location`) or a
|
|
196
|
+
`SemanticLogger::CallSite`, or that it has no such code with `false`, which records no call site
|
|
197
|
+
without reading the stack for one. `nil` means none was supplied, so the call site is found as
|
|
198
|
+
usual. A supplied call site is only recorded when the entry is at or above `call_site_level`, so
|
|
199
|
+
that it never adds a call site to the output of an application that has not turned call sites on:
|
|
200
|
+
|
|
201
|
+
~~~ruby
|
|
202
|
+
# The application code that ran the query, or none when no frame is the application's
|
|
203
|
+
logger.debug(message: "Order Load", duration: 0.4,
|
|
204
|
+
call_site: backtrace_cleaner.first_clean_location || false)
|
|
205
|
+
logger.info(message: "Started GET /orders", call_site: false)
|
|
206
|
+
~~~
|
|
207
|
+
|
|
208
|
+
A String, or any other value that parsed data can hold, is not accepted as a call site, nor is a
|
|
209
|
+
`CallSite` without a String path, an Integer line, and a String or `nil` label: a log call logs it
|
|
210
|
+
as part of the payload, and a `measure_*` call ignores it. An entry
|
|
211
|
+
with `call_site: false` that captures a backtrace (see `backtrace_level`), or logs an exception,
|
|
212
|
+
still has a file name and line number from that backtrace, as the last point below says.
|
|
213
|
+
* The first frame outside Semantic Logger, `rails_semantic_logger`, and Ruby's internal files (paths
|
|
214
|
+
that start with `<internal:`). So an entry logged by a `measure_*` call, or by a logging block,
|
|
215
|
+
records the code that called the logger. When the logger is called through `public_send`,
|
|
216
|
+
`Method#call`, or `UnboundMethod#bind_call`, the call site is the code that called that method. (On
|
|
217
|
+
JRuby, `Method#call` inside a block records the right file and line, but the label `block in call`.)
|
|
218
|
+
Code in Ruby's standard library that calls the logger for you, such as a method that `Forwardable`
|
|
219
|
+
delegates, is recorded as the call site: add its path to `call_site_skip_paths` (see below).
|
|
220
|
+
* When the log call supplies a `backtrace:`, the first line of that backtrace. For example
|
|
221
|
+
`logger.backtrace` passes the backtrace of the thread it logs, and `rails_semantic_logger` passes
|
|
222
|
+
the application code that ran a SQL query when it captures a backtrace for the entry (see
|
|
223
|
+
`backtrace_level` above). A backtrace returned by a logging block does not change the call site.
|
|
224
|
+
* Entries that `rails_semantic_logger` logs for Rails' own events, such as `Processing`,
|
|
225
|
+
`Completed`, `Rendered`, and SQL entries without a backtrace, are logged from Active Support's
|
|
226
|
+
notifications, so their call site is an Active Support frame rather than the application's code.
|
|
227
|
+
* With a call site, an entry's file name and line number are the call site's, also for an entry that
|
|
228
|
+
logs an exception. Without one, they still come from a captured backtrace, or else the first line
|
|
229
|
+
of the exception's backtrace.
|
|
230
|
+
|
|
231
|
+
Code that logs on behalf of its caller, such as an application's own logging helper, would always be
|
|
232
|
+
recorded as the call site. Skip its frames with `call_site_skip_paths`, so that entries record the
|
|
233
|
+
code that called it. Each entry matches any file whose path includes it:
|
|
234
|
+
|
|
235
|
+
~~~ruby
|
|
236
|
+
# AuditLog.record(...) logs for its callers
|
|
237
|
+
SemanticLogger.call_site_skip_paths = ["app/lib/audit_log.rb"]
|
|
238
|
+
|
|
239
|
+
# Every file of a gem that wraps the logger
|
|
240
|
+
SemanticLogger.call_site_skip_paths = ["app/lib/audit_log.rb", "/my_logging_gem/"]
|
|
241
|
+
~~~
|
|
242
|
+
|
|
243
|
+
### Cleaning backtraces
|
|
244
|
+
|
|
245
|
+
Most of an exception's backtrace is usually gems and Ruby itself, with the application's own frames
|
|
246
|
+
somewhere in the middle. A backtrace cleaner leaves out or shortens lines before they are logged.
|
|
247
|
+
Set the application's cleaner once:
|
|
248
|
+
|
|
249
|
+
~~~ruby
|
|
250
|
+
SemanticLogger.backtrace_cleaner = Rails.backtrace_cleaner
|
|
251
|
+
~~~
|
|
252
|
+
|
|
253
|
+
A cleaner is any object that responds to `#clean`, as `ActiveSupport::BacktraceCleaner` does, or to
|
|
254
|
+
`#call`, such as a Proc. It is given each backtrace as an Array of Strings, and returns the lines to
|
|
255
|
+
log:
|
|
256
|
+
|
|
257
|
+
~~~ruby
|
|
258
|
+
app_root = "#{Dir.pwd}/"
|
|
259
|
+
SemanticLogger.backtrace_cleaner = lambda do |lines|
|
|
260
|
+
lines.select { |line| line.start_with?(app_root) }.map { |line| line.delete_prefix(app_root) }
|
|
261
|
+
end
|
|
262
|
+
~~~
|
|
263
|
+
|
|
264
|
+
Only the text and JSON output changes: the `:default`, `:color`, `:pattern`, `:one_line`
|
|
265
|
+
(which does not write backtraces), `:syslog`, `:json`, and `:fluentd` formatters clean the backtrace
|
|
266
|
+
of the logged exception and of each exception that caused it. So do the appenders that use `:json`
|
|
267
|
+
unless given another formatter: `:http`, `:elasticsearch_http`, `:tcp`, `:udp`, `:kafka`,
|
|
268
|
+
`:rabbitmq`, and `:cloudwatch_logs`. To send full backtraces to a log service through one of them,
|
|
269
|
+
give it `formatter: {json: {backtrace_cleaner: false}}`.
|
|
270
|
+
|
|
271
|
+
The exception itself is never changed, so appenders that send the exception to an error service,
|
|
272
|
+
such as Sentry, Bugsnag, or Honeybadger, still send its full backtrace. The other formatters,
|
|
273
|
+
including `:raw` and the formatters written for a particular service, such as `:ecs`, `:loki`,
|
|
274
|
+
`:new_relic_logs`, and `:open_telemetry`, write the full backtrace. So do the `:elasticsearch`,
|
|
275
|
+
`:opensearch`, `:mongodb`, `:graylog`, `:splunk_http`, and `:honeybadger_insights` appenders, which
|
|
276
|
+
build their entries with formatters of their own: giving one of them a `formatter` replaces how it
|
|
277
|
+
builds its entries.
|
|
278
|
+
|
|
279
|
+
The `backtrace_cleaner` option of an appender's formatter chooses for that appender:
|
|
280
|
+
|
|
281
|
+
| Value | Cleans backtraces with |
|
|
282
|
+
|-------|------------------------|
|
|
283
|
+
| `true` | The application's cleaner, `SemanticLogger.backtrace_cleaner`, when one is set. The default for the formatters listed above. |
|
|
284
|
+
| `false` | Nothing: the full backtrace is written. The default for every other formatter. |
|
|
285
|
+
| A cleaner | That cleaner, in place of the application's. |
|
|
286
|
+
|
|
287
|
+
~~~ruby
|
|
288
|
+
# Keep full backtraces in the JSON file:
|
|
289
|
+
SemanticLogger.add_appender(file_name: "production.json", formatter: {json: {backtrace_cleaner: false}})
|
|
290
|
+
|
|
291
|
+
# Clean the backtraces in logfmt output too:
|
|
292
|
+
SemanticLogger.add_appender(io: $stdout, formatter: {logfmt: {backtrace_cleaner: true}})
|
|
293
|
+
|
|
294
|
+
# Use a cleaner of its own for one appender:
|
|
295
|
+
SemanticLogger.add_appender(io: $stdout, formatter: {color: {backtrace_cleaner: MyCleaner.new}})
|
|
296
|
+
~~~
|
|
297
|
+
|
|
298
|
+
Notes:
|
|
299
|
+
|
|
300
|
+
* When the cleaner leaves out every line, for example for an exception raised entirely inside gems,
|
|
301
|
+
the full backtrace is logged, so the only clue to where it came from is never lost. The full
|
|
302
|
+
backtrace is also logged when the cleaner does not return an Array, or raises an error, which is
|
|
303
|
+
reported to the internal logger.
|
|
304
|
+
* The cleaner is called on the appenders' threads, not the thread that logged the entry, so it must be
|
|
305
|
+
thread safe. Its result is kept with the entry, so it is normally called once for each exception,
|
|
306
|
+
however many appenders write it. Appenders that write the same entry at the same moment can each
|
|
307
|
+
call it.
|
|
308
|
+
* The application's cleaner is looked up as each entry is written, so it can be set before or after
|
|
309
|
+
the appenders are added.
|
|
310
|
+
* The file name and line number of an entry are not cleaned. They are where the entry was logged when
|
|
311
|
+
it has a call site (see `call_site_level` above) or captured a backtrace (see `backtrace_level`
|
|
312
|
+
above), and otherwise the first line of the exception's backtrace.
|
|
313
|
+
|
|
314
|
+
### Redacting sensitive keys
|
|
315
|
+
|
|
316
|
+
Name the keys whose values must never be logged, such as passwords and access tokens, and their
|
|
317
|
+
values are replaced with `[FILTERED]` in the payload, named tags, context, and metric dimensions of
|
|
318
|
+
every log entry, at any depth, before any appender writes it:
|
|
319
|
+
|
|
320
|
+
~~~ruby
|
|
321
|
+
SemanticLogger.sensitive_keys = %i[passw secret token _key authorization cookie]
|
|
322
|
+
~~~
|
|
323
|
+
|
|
324
|
+
Symbols and Strings match any key whose name contains them, ignoring case, as in Rails'
|
|
325
|
+
`config.filter_parameters`, and an `ActiveSupport::ParameterFilter`, a Regexp, or a Proc can be given
|
|
326
|
+
instead. rails_semantic_logger sets it from `config.filter_parameters`. See
|
|
327
|
+
[Redacting sensitive data](security.html#redacting-sensitive-data) for what is matched and redacted,
|
|
328
|
+
and what is not.
|
|
329
|
+
|
|
112
330
|
### Caching loggers
|
|
113
331
|
|
|
114
332
|
By default `SemanticLogger[...]` returns a brand new logger instance on every call. Enable logger
|
|
@@ -143,23 +361,32 @@ Notes:
|
|
|
143
361
|
## Per-appender settings
|
|
144
362
|
|
|
145
363
|
Each destination is added with `SemanticLogger.add_appender`. The full catalogue of destinations,
|
|
146
|
-
and the options specific to each, is in [Appenders](appenders.html).
|
|
147
|
-
|
|
364
|
+
and the options specific to each, is in [Appenders](appenders.html). Each call adds one destination,
|
|
365
|
+
given as one of `file_name:`, `io:`, `logger:`, `appender:`, or `metric:`; to write to several, call
|
|
366
|
+
it once for each. Given more than one of them, `add_appender` adds only the first, in that order, and
|
|
367
|
+
reports a [deprecation warning](upgrading.html#seeing-deprecation-warnings) that names the others,
|
|
368
|
+
which it ignores; v6 raises an `ArgumentError`. A destination given as `nil` is not one, so that
|
|
369
|
+
settings of which only one is set can be passed on, such as
|
|
370
|
+
`file_name: options[:log_file], io: options[:log_io]`.
|
|
371
|
+
|
|
372
|
+
In addition to its own settings, almost every appender accepts these common options:
|
|
148
373
|
|
|
149
374
|
| Option | Description |
|
|
150
375
|
|--------|-------------|
|
|
151
|
-
| `level` | Only write entries at this level or higher to this appender.
|
|
152
|
-
| `formatter` | How to format the output, for example `:default`, `:color`, or `:json`. See [Custom formatters](#custom-formatters). |
|
|
376
|
+
| `level` | Only write entries at this level or higher to this appender. Without it, an appender writes every entry it receives; those that send errors to a service, such as Bugsnag and Sentry, default to `:error` instead, and Honeybadger Insights to `:info`. Each logger only emits entries at or above its own level, which defaults to `SemanticLogger.default_level`, so set an appender's `level` to keep fewer entries than the loggers emit. |
|
|
377
|
+
| `formatter` | How to format the output, for example `:default`, `:color`, or `:json`. A [block given to `add_appender`](#formatter-as-a-block-or-proc) takes its place. See [Custom formatters](#custom-formatters). |
|
|
153
378
|
| `filter` | A `Regexp` or `Proc` selecting which entries this appender accepts. See [Filtering](#filtering). |
|
|
154
379
|
| `application`, `environment`, `host` | Override the global values for this appender only. |
|
|
155
380
|
|
|
156
381
|
A per-appender `level` lets each destination keep a different subset of the logs. For example, keep
|
|
157
|
-
a full trace log and a separate warnings-only log
|
|
382
|
+
a full trace log and a separate warnings-only log. An appender only receives the entries that the
|
|
383
|
+
loggers emit, so for the trace log to receive trace entries the loggers must emit them too:
|
|
158
384
|
|
|
159
385
|
~~~ruby
|
|
160
386
|
require "semantic_logger"
|
|
161
387
|
|
|
162
|
-
|
|
388
|
+
# Loggers emit every entry:
|
|
389
|
+
SemanticLogger.default_level = :trace
|
|
163
390
|
|
|
164
391
|
# Everything at :trace and above:
|
|
165
392
|
SemanticLogger.add_appender(file_name: "log/trace.log", level: :trace)
|
|
@@ -273,10 +500,11 @@ filter: ->(log) { log.message !~ /\Aheartbeat/ }
|
|
|
273
500
|
### Example: rewriting a message inside a filter
|
|
274
501
|
|
|
275
502
|
A filter can also **modify** the log event before it is written, as long as it still returns `true`
|
|
276
|
-
so the (now edited) entry is kept. This is handy for
|
|
277
|
-
as below, or in `on_log
|
|
278
|
-
Do not redact in an appender's
|
|
279
|
-
|
|
503
|
+
so the (now edited) entry is kept. This is handy for removing sensitive data from a message in a
|
|
504
|
+
logger's filter, as below, or in `on_log`. To redact the values of sensitive keys in every entry, use
|
|
505
|
+
[`SemanticLogger.sensitive_keys`](#redacting-sensitive-keys) instead. Do not redact in an appender's
|
|
506
|
+
filter: every appender writes the same log event, so the appenders added before it have already
|
|
507
|
+
written the data. Resque, for example, logs the entire job payload,
|
|
280
508
|
which may contain private information:
|
|
281
509
|
|
|
282
510
|
```ruby
|
|
@@ -307,7 +535,7 @@ The formatter turns each log event into the text or JSON that an appender writes
|
|
|
307
535
|
`formatter:` when adding an appender. The simplest options are the built-in formatters selected by
|
|
308
536
|
name (`:default`, `:color`, `:json`, `:logfmt`, and others; see [Appenders](appenders.html)). For
|
|
309
537
|
anything beyond those, you have three choices, in increasing order of effort: a pattern string, a
|
|
310
|
-
Proc, or a formatter class.
|
|
538
|
+
block or Proc, or a formatter class.
|
|
311
539
|
|
|
312
540
|
### Pattern formatter
|
|
313
541
|
|
|
@@ -353,6 +581,7 @@ Available directives:
|
|
|
353
581
|
| `%{pid}` | Process id. |
|
|
354
582
|
| `%{file_name}` | Ruby file name that logged the message, e.g. `app.rb`. |
|
|
355
583
|
| `%{line}` | Line number within the Ruby file, e.g. `42`. |
|
|
584
|
+
| `%{method}` | Method that logged the message, e.g. `Order#submit`, when the entry has a call site. See [Recording where each entry was logged](#recording-where-each-entry-was-logged). |
|
|
356
585
|
| `%{tags}` | Tags, comma separated. |
|
|
357
586
|
| `%{named_tags}` | All named tags. One tag with `%{named_tags:request_id}`. |
|
|
358
587
|
| `%{host}` | Host name. |
|
|
@@ -387,11 +616,56 @@ SemanticLogger.add_appender(
|
|
|
387
616
|
)
|
|
388
617
|
~~~
|
|
389
618
|
|
|
390
|
-
### Formatter as a Proc
|
|
619
|
+
### Formatter as a block or Proc
|
|
620
|
+
|
|
621
|
+
For full control with minimal ceremony, give `add_appender` a block. It formats every entry written
|
|
622
|
+
to that appender. It is called with two arguments, the log event and the appender. A block, or a
|
|
623
|
+
Proc made with `proc`, may take just the log event; a lambda or method, passed with `&` or as
|
|
624
|
+
`formatter:`, must accept both arguments. For the structure of the log event, see
|
|
625
|
+
[Log Event](log.html). The appender holds the `host`, `application`, and `environment` to write
|
|
626
|
+
with the entry.
|
|
627
|
+
|
|
628
|
+
~~~ruby
|
|
629
|
+
SemanticLogger.application = "MyApp"
|
|
630
|
+
|
|
631
|
+
SemanticLogger.add_appender(io: $stdout) do |log, appender|
|
|
632
|
+
"#{log.level.upcase} #{appender.application} #{log.name} -- #{log.message}"
|
|
633
|
+
end
|
|
634
|
+
# => INFO MyApp Order -- Order submitted
|
|
635
|
+
~~~
|
|
636
|
+
|
|
637
|
+
The block returns what the appender writes. For `io:`, `file_name:`, and `logger:`, return the entry
|
|
638
|
+
as a String: `io:` and `file_name:` write it followed by a newline, and `logger:` passes it to the
|
|
639
|
+
wrapped logger.
|
|
640
|
+
|
|
641
|
+
A block works the same way for an appender or metric subscriber named by a Symbol. It takes the
|
|
642
|
+
place of the formatter the appender uses by default, so it returns what that formatter returns: a
|
|
643
|
+
String for the text and JSON destinations, such as `:tcp`, `:udp`, `:http`, and `:syslog` with its
|
|
644
|
+
default local url, and a Hash for those that store a document, such as `:elasticsearch` and
|
|
645
|
+
`:mongodb`. For `:loki`, and for `:syslog` with a `tcp://` or `udp://` url, the default formatter
|
|
646
|
+
builds the whole request body or syslog packet, so a block would have to build it too: use a
|
|
647
|
+
formatter class instead, such as a subclass of `SemanticLogger::Formatters::Loki` or
|
|
648
|
+
`SemanticLogger::Formatters::Syslog` (see [Formatter as a class](#formatter-as-a-class)). A metric
|
|
649
|
+
subscriber formats a metric: `metric: :new_relic` expects the metric name, `metric: :signalfx` the
|
|
650
|
+
JSON body it posts, and `metric: :statsd` does not use a formatter, so it never calls the block.
|
|
651
|
+
|
|
652
|
+
~~~ruby
|
|
653
|
+
SemanticLogger.add_appender(file_name: "development.log") { |log| "#{log.name} -- #{log.message}" }
|
|
654
|
+
SemanticLogger.add_appender(appender: :tcp, server: "localhost:514") { |log| "#{log.level} #{log.message}" }
|
|
655
|
+
~~~
|
|
656
|
+
|
|
657
|
+
A block formats one entry at a time, so it cannot format the single request that `:http`, `:loki`,
|
|
658
|
+
and `metric: :signalfx` send for a whole batch when added with `batch: true`: those need a formatter
|
|
659
|
+
that implements `#batch(logs, logger)`, such as the one each uses by default. `add_appender` raises
|
|
660
|
+
an `ArgumentError` for a block, or any other formatter that cannot format a batch, given to one of
|
|
661
|
+
them with `batch: true`. Add it without `batch: true` to send one request for each entry, formatted
|
|
662
|
+
by the block.
|
|
391
663
|
|
|
392
|
-
|
|
393
|
-
|
|
394
|
-
|
|
664
|
+
When both a block and `formatter:` are given, the block takes precedence, and `formatter:` is
|
|
665
|
+
ignored.
|
|
666
|
+
|
|
667
|
+
The same formatter can be given to `formatter:` as a Proc, for example to share it between
|
|
668
|
+
appenders:
|
|
395
669
|
|
|
396
670
|
~~~ruby
|
|
397
671
|
formatter = proc do |log|
|
|
@@ -399,6 +673,17 @@ formatter = proc do |log|
|
|
|
399
673
|
log.inspect
|
|
400
674
|
end
|
|
401
675
|
SemanticLogger.add_appender(io: $stdout, formatter: formatter)
|
|
676
|
+
SemanticLogger.add_appender(file_name: "development.log", formatter: formatter)
|
|
677
|
+
~~~
|
|
678
|
+
|
|
679
|
+
An appender created with `new` and added as `appender:` or `metric:` already has its formatter, so
|
|
680
|
+
`add_appender` ignores a block given with it, as it does `formatter:`, and reports a
|
|
681
|
+
[deprecation warning](upgrading.html#seeing-deprecation-warnings); v6 raises an `ArgumentError`.
|
|
682
|
+
Give the block, or `formatter:`, to `new` instead:
|
|
683
|
+
|
|
684
|
+
~~~ruby
|
|
685
|
+
appender = SemanticLogger::Appender::Tcp.new(server: "localhost:514") { |log| "#{log.level} #{log.message}" }
|
|
686
|
+
SemanticLogger.add_appender(appender: appender)
|
|
402
687
|
~~~
|
|
403
688
|
|
|
404
689
|
### Formatter as a class
|
|
@@ -477,8 +762,12 @@ For the text formatters, enable the `escape_control_chars` option to replace con
|
|
|
477
762
|
untrusted log data (the logger name, thread name, message, payload, tags, named tags, the message,
|
|
478
763
|
details, and backtrace lines of the logged exception, and the message and backtrace lines of each
|
|
479
764
|
exception that caused it) with a printable, escaped form. For example a newline is written as `\n`
|
|
480
|
-
and the ANSI escape as `\e`. The
|
|
481
|
-
|
|
765
|
+
and the ANSI escape as `\e`. The Unicode characters that control how text is shown are escaped too,
|
|
766
|
+
in the `\uXXXX` form: the C1 control characters (U+0080 to U+009F), the line and paragraph
|
|
767
|
+
separators (U+2028, U+2029), at which editors and some log viewers start a new line, and the
|
|
768
|
+
bidirectional controls (U+202A to U+202E, U+2066 to U+2069), such as the right-to-left override that
|
|
769
|
+
can make logged text read backwards. The option is **disabled by default** to preserve the existing
|
|
770
|
+
human readable output:
|
|
482
771
|
|
|
483
772
|
```ruby
|
|
484
773
|
# Text appender that escapes control characters in untrusted data:
|
|
@@ -573,12 +862,12 @@ Basic outline for an appender:
|
|
|
573
862
|
require "semantic_logger"
|
|
574
863
|
|
|
575
864
|
class SimpleAppender < SemanticLogger::Subscriber
|
|
576
|
-
attr_reader :
|
|
865
|
+
attr_reader :url
|
|
577
866
|
|
|
578
867
|
# Add additional arguments to the initializer while supporting all existing ones.
|
|
579
|
-
def initialize(
|
|
580
|
-
@host = host
|
|
868
|
+
def initialize(url:, **args, &block)
|
|
581
869
|
super(**args, &block)
|
|
870
|
+
@url = url
|
|
582
871
|
end
|
|
583
872
|
|
|
584
873
|
# Display the log struct and the text formatted output
|
|
@@ -602,12 +891,21 @@ class SimpleAppender < SemanticLogger::Subscriber
|
|
|
602
891
|
end
|
|
603
892
|
~~~
|
|
604
893
|
|
|
894
|
+
Do not give the appender an option of the same name as one that `super` already accepts, such as
|
|
895
|
+
`host` or `level`: the appender's own keyword would take the value, so `super` would never receive
|
|
896
|
+
it. Call `super` before setting the appender's own instance variables, since `super` sets those of
|
|
897
|
+
every appender, such as `@name` and `@host`.
|
|
898
|
+
|
|
605
899
|
When the appender runs asynchronously (`async: true` or batching), an exception raised by `#log`
|
|
606
|
-
|
|
607
|
-
exception caused by the content of one message
|
|
608
|
-
|
|
609
|
-
|
|
610
|
-
|
|
900
|
+
discards the message, and the worker thread backs off before writing the next one, on the basis
|
|
901
|
+
that the destination may be briefly unavailable. An exception caused by the content of one message
|
|
902
|
+
says nothing about whether the destination is available, and nor does one that writing the
|
|
903
|
+
message again cannot fix, such as a request the destination refuses outright for its credentials,
|
|
904
|
+
so the thread carries on straight away instead. `#message_error?(exception)` decides which is
|
|
905
|
+
which, and these are the message errors. Format each message with `format_log(log)` (or
|
|
906
|
+
a batch with `format_batch(logs)`) rather than `formatter.call`: any error the
|
|
907
|
+
formatter raises, including a `ScriptError` such as `NotImplementedError`, which would otherwise
|
|
908
|
+
stop the worker thread, is then raised as a `SemanticLogger::Subscriber::FormatError`, which
|
|
611
909
|
`#message_error?` recognizes, since a formatter can only fail because of the content of the message.
|
|
612
910
|
By default it also recognizes the errors raised when serializing a value that cannot be written
|
|
613
911
|
(`JSON::GeneratorError`, `JSON::NestingError`, `EncodingError`, and `FloatDomainError`). Override it
|
|
@@ -619,6 +917,39 @@ def message_error?(exception)
|
|
|
619
917
|
end
|
|
620
918
|
~~~
|
|
621
919
|
|
|
920
|
+
Any other error is the destination's, and discards the message (or the batch) by default, since
|
|
921
|
+
the write may have stored some or all of it. When the write that raised it cannot have stored any of
|
|
922
|
+
the messages, and writing them again later may succeed, such as when the appender could not connect,
|
|
923
|
+
or the destination answered that it is unavailable for now, return `true` from
|
|
924
|
+
`#wrote_nothing?(exception)`. The worker thread then writes the same message, or batch, again once
|
|
925
|
+
it has backed off. Once the destination has answered it with a failure after it was written again 3
|
|
926
|
+
times, the worker thread discards it and carries on with the next one. Keep the default, `false`,
|
|
927
|
+
whenever the destination may have stored them, such as when its answer timed out, since writing
|
|
928
|
+
them again would store them twice, and when writing them again is bound to fail the same way. While
|
|
929
|
+
a flush or close is waiting, the message or batch is discarded either way, so that the flush does
|
|
930
|
+
not wait out the back-offs.
|
|
931
|
+
|
|
932
|
+
When the write never reached the destination, such as when the appender could not connect, also
|
|
933
|
+
return `true` from `#not_sent?(exception)`, which is asked only once `#wrote_nothing?` has returned
|
|
934
|
+
`true`. Writing it again after such a failure does not count towards the 3 times, since the
|
|
935
|
+
destination has said nothing about the message, so that an outage loses no messages until the
|
|
936
|
+
worker thread gives up after `async_max_retries`:
|
|
937
|
+
|
|
938
|
+
~~~ruby
|
|
939
|
+
def wrote_nothing?(exception)
|
|
940
|
+
not_sent?(exception) || exception.is_a?(MyClient::ServiceUnavailable)
|
|
941
|
+
end
|
|
942
|
+
|
|
943
|
+
def not_sent?(exception)
|
|
944
|
+
exception.is_a?(MyClient::ConnectionRefused)
|
|
945
|
+
end
|
|
946
|
+
~~~
|
|
947
|
+
|
|
948
|
+
When `#log` or `#batch` writes the messages in more than one request, return `false` for an error
|
|
949
|
+
from any request after the first: the destination stored the messages of the earlier ones, which
|
|
950
|
+
would otherwise be written again. That includes a subclass of `SemanticLogger::Appender::Http`
|
|
951
|
+
whose `#log` or `#batch` calls `post` more than once.
|
|
952
|
+
|
|
622
953
|
A `SystemStackError` is handled separately: the message is discarded, and since a thread must not
|
|
623
954
|
overflow its stack twice (on macOS that crashes Ruby), the worker carries on in a new thread. Do not
|
|
624
955
|
rescue it in `#log` or `#batch`.
|
|
@@ -639,10 +970,110 @@ def batch_all_or_nothing?
|
|
|
639
970
|
end
|
|
640
971
|
~~~
|
|
641
972
|
|
|
973
|
+
A message error that the destination raises for any batch, whatever messages it holds, such as a
|
|
974
|
+
request refused for its credentials or its url, would be raised again for every part. Recognize it in
|
|
975
|
+
`#message_error?`, since writing the batch again cannot fix it, and return `true` from
|
|
976
|
+
`#fails_any_batch?(exception)` for it, so that the whole batch is discarded at once, rather than
|
|
977
|
+
every message sent again on its own to be refused in turn. `#fails_any_batch?` is asked only about
|
|
978
|
+
a message error, and returns `false` by default:
|
|
979
|
+
|
|
980
|
+
~~~ruby
|
|
981
|
+
def message_error?(exception)
|
|
982
|
+
exception.is_a?(MyClient::Unauthorized) || exception.is_a?(MyClient::InvalidDocument) || super
|
|
983
|
+
end
|
|
984
|
+
|
|
985
|
+
def fails_any_batch?(exception)
|
|
986
|
+
exception.is_a?(MyClient::Unauthorized)
|
|
987
|
+
end
|
|
988
|
+
~~~
|
|
989
|
+
|
|
990
|
+
The HTTP appenders answer both from the status of the response: a request rejected with a `400`,
|
|
991
|
+
`413`, or `422` is a message error that only some of the messages cause, and one rejected with any
|
|
992
|
+
other 4xx but `408` and `429`, such as a `401`, `403`, or `404`, is one that fails any batch.
|
|
993
|
+
|
|
994
|
+
A destination that stores the messages of a batch in order, and stops at the first one it rejects,
|
|
995
|
+
may say which one that is. Return its position in the batch, counted from 0, from
|
|
996
|
+
`#rejected_entry(exception)`: the messages before it are then counted as written, it is discarded
|
|
997
|
+
(with every message after it, when `#fails_any_batch?` is true), and the messages after it are
|
|
998
|
+
written again as a batch, so that none is written twice. It returns `nil` by default, when the
|
|
999
|
+
appender cannot tell. The `:splunk_http` appender answers it from the HTTP Event Collector's
|
|
1000
|
+
`invalid-event-number`:
|
|
1001
|
+
|
|
1002
|
+
~~~ruby
|
|
1003
|
+
def rejected_entry(exception)
|
|
1004
|
+
exception.rejected_index if exception.is_a?(MyClient::Rejected)
|
|
1005
|
+
end
|
|
1006
|
+
~~~
|
|
1007
|
+
|
|
1008
|
+
`#batch_all_or_nothing?` says nothing about an error from the destination: a batch formatted before
|
|
1009
|
+
any of it is sent can still be stored by a destination whose answer then times out. Answer that from
|
|
1010
|
+
`#wrote_nothing?`, for `#log` and `#batch` alike.
|
|
1011
|
+
|
|
1012
|
+
Such a `#batch` works with any formatter, including a formatter block. An appender that cannot be
|
|
1013
|
+
batched in some configuration says why from `#batch_refusal`, and what to do instead, so that
|
|
1014
|
+
`add_appender` raises an `ArgumentError` with it when asked to batch the appender, with
|
|
1015
|
+
`batch: true` or by default, rather than add an appender that writes nothing. It returns `nil`, by
|
|
1016
|
+
default, when the appender can be batched. For example, a `#batch` that formats the whole batch with
|
|
1017
|
+
`format_batch(logs)` needs a formatter that implements `#batch(logs, logger)`, so return
|
|
1018
|
+
`format_batch_refusal`, which says whether the appender's formatter can format a batch:
|
|
1019
|
+
|
|
1020
|
+
~~~ruby
|
|
1021
|
+
def batch(logs)
|
|
1022
|
+
client.send_request(format_batch(logs))
|
|
1023
|
+
end
|
|
1024
|
+
|
|
1025
|
+
def batch_refusal
|
|
1026
|
+
format_batch_refusal
|
|
1027
|
+
end
|
|
1028
|
+
~~~
|
|
1029
|
+
|
|
1030
|
+
Like `format_log` and `format_batch`, `format_batch_refusal` is a private helper of
|
|
1031
|
+
`SemanticLogger::Subscriber`. It returns `nil` when the formatter implements
|
|
1032
|
+
`#batch(logs, logger)`, and otherwise why it cannot format a batch, naming the formatter's class,
|
|
1033
|
+
and what to do instead. That includes an appender that is its own formatter, since it implements
|
|
1034
|
+
`#call`, for which the only advice is `batch: false`.
|
|
1035
|
+
|
|
1036
|
+
The error starts with `add_appender cannot batch <class>: `, followed by the reason.
|
|
1037
|
+
`SemanticLogger::Appender::Http`, and the appenders derived from it, such as `:loki`, answer for
|
|
1038
|
+
their own `#batch`, so a subclass that replaces `#batch` is batched with any formatter, unless its
|
|
1039
|
+
own `#batch_refusal` says otherwise. A subclass whose `#batch` calls `super`, or that a prepended
|
|
1040
|
+
module wraps, still formats the whole batch with `format_batch`, yet is not refused on its own, so
|
|
1041
|
+
define its `#batch_refusal` too:
|
|
1042
|
+
|
|
1043
|
+
~~~ruby
|
|
1044
|
+
class FilteredHttp < SemanticLogger::Appender::Http
|
|
1045
|
+
def batch(logs)
|
|
1046
|
+
super(logs.reject { |log| log.name == "HealthCheck" })
|
|
1047
|
+
end
|
|
1048
|
+
|
|
1049
|
+
def batch_refusal
|
|
1050
|
+
format_batch_refusal
|
|
1051
|
+
end
|
|
1052
|
+
end
|
|
1053
|
+
~~~
|
|
1054
|
+
|
|
1055
|
+
An appender that inherits a `#batch` that its destination rejects, whatever the formatter, returns a
|
|
1056
|
+
reason of its own from `#batch_refusal`:
|
|
1057
|
+
|
|
1058
|
+
~~~ruby
|
|
1059
|
+
def batch_refusal
|
|
1060
|
+
"the server takes one entry per request. Add it without batch: true."
|
|
1061
|
+
end
|
|
1062
|
+
~~~
|
|
1063
|
+
|
|
642
1064
|
Register the appender by passing an instance to `add_appender`:
|
|
643
1065
|
|
|
644
1066
|
~~~ruby
|
|
645
|
-
SemanticLogger.add_appender(appender: SimpleAppender.new)
|
|
1067
|
+
SemanticLogger.add_appender(appender: SimpleAppender.new(url: "https://logs.example.com"))
|
|
1068
|
+
~~~
|
|
1069
|
+
|
|
1070
|
+
Give the instance its own options, such as `level:` and `formatter:`, and a formatter block, when
|
|
1071
|
+
creating it, since `add_appender` adds it as it is. Only the options of `add_appender` itself, such as
|
|
1072
|
+
`async:` and `batch:`, apply to an instance:
|
|
1073
|
+
|
|
1074
|
+
~~~ruby
|
|
1075
|
+
appender = SimpleAppender.new(url: "https://logs.example.com", level: :warn, formatter: :json)
|
|
1076
|
+
SemanticLogger.add_appender(appender: appender, async: true)
|
|
646
1077
|
~~~
|
|
647
1078
|
|
|
648
1079
|
Look at the
|
|
@@ -668,6 +1099,31 @@ appender = SemanticLogger.add_appender(file_name: "development.log")
|
|
|
668
1099
|
SemanticLogger.remove_appender(appender)
|
|
669
1100
|
~~~
|
|
670
1101
|
|
|
1102
|
+
`remove_appender` stops sending log messages to the appender, then waits, as `SemanticLogger.flush`
|
|
1103
|
+
does, until the messages already logged are written to the other appenders and those are flushed,
|
|
1104
|
+
so that none is still being written to the appender when it closes it. Closing the appender closes
|
|
1105
|
+
its log file, or its connections to its server, and `:cloudwatch_logs` sends the entries it has
|
|
1106
|
+
buffered. In a forked child, closing an appender leaves the connections it inherited to the parent
|
|
1107
|
+
process, and `remove_appender` first lets the appenders reopen, when they have not yet, so that the
|
|
1108
|
+
appender closes only what the child built (see [Process forking](operations.html#process-forking)).
|
|
1109
|
+
|
|
1110
|
+
An appender added with `async: true` or in batches writes on a thread of its own, which the thread
|
|
1111
|
+
that writes to every appender may be waiting for. So when `remove_appender` is called on that
|
|
1112
|
+
thread, such as by an appender of your own as it writes, removing itself or another appender, it
|
|
1113
|
+
does not wait: it returns `true` at once. The thread that writes to every appender then, once it
|
|
1114
|
+
has written the messages logged before, closes the appender between writes, never while it is
|
|
1115
|
+
writing to it, and removes it from the list. Until then, `SemanticLogger.appenders` still lists the
|
|
1116
|
+
appender, and it is still given the messages logged before. An appender that removes itself is
|
|
1117
|
+
closed once its current write has returned. When the process forks before that, the child still
|
|
1118
|
+
removes the appender, once it has reopened it, so that it closes only what the child built.
|
|
1119
|
+
|
|
1120
|
+
A message logged on that thread, such as by a library that your appender calls as it writes, is
|
|
1121
|
+
dropped at once when it finds the queue of the thread that writes to every appender full while your
|
|
1122
|
+
appender's own queue is full too, or a flush or close of your appender is waiting, since that thread
|
|
1123
|
+
may then be waiting for room on your appender's queue, or for your appender to flush. That holds
|
|
1124
|
+
whoever asked for the flush or close, so such a message can be dropped even when it could safely
|
|
1125
|
+
have waited: a dropped message is the price of never waiting forever.
|
|
1126
|
+
|
|
671
1127
|
Other appender management methods:
|
|
672
1128
|
|
|
673
1129
|
~~~ruby
|
|
@@ -681,16 +1137,63 @@ SemanticLogger.clear_appenders!
|
|
|
681
1137
|
SemanticLogger.close
|
|
682
1138
|
~~~
|
|
683
1139
|
|
|
1140
|
+
### Changing an appender while running
|
|
1141
|
+
|
|
1142
|
+
Keep the appender returned by `add_appender` to change its level or formatter later, without
|
|
1143
|
+
restarting the application. The formatter accepts the same values as the `formatter:` option:
|
|
1144
|
+
|
|
1145
|
+
~~~ruby
|
|
1146
|
+
console = SemanticLogger.add_appender(io: $stdout, formatter: :color)
|
|
1147
|
+
|
|
1148
|
+
# ... later
|
|
1149
|
+
console.level = :debug
|
|
1150
|
+
console.formatter = :json
|
|
1151
|
+
~~~
|
|
1152
|
+
|
|
1153
|
+
An appender that formats its messages as one batch, such as `:http` or `:loki` with `batch: true`,
|
|
1154
|
+
needs a formatter that implements `#batch(logs, logger)`, such as `:json`. `add_appender` checks the
|
|
1155
|
+
formatter it is added with, but not one set later: a batch that the new formatter cannot format is
|
|
1156
|
+
not written.
|
|
1157
|
+
|
|
1158
|
+
Messages are formatted as they are written. When the appender runs asynchronously, messages still
|
|
1159
|
+
on its queue are written with the new formatter. Call `SemanticLogger.flush` just before changing it
|
|
1160
|
+
so that messages logged before the flush keep the old one.
|
|
1161
|
+
|
|
1162
|
+
Only one appender may write to each of `$stdout` and `$stderr`, since a second one would duplicate
|
|
1163
|
+
every line, so `add_appender` ignores (and warns about) a second appender for the same console
|
|
1164
|
+
stream, and returns `nil`. To change how the console output looks, find the existing appender and
|
|
1165
|
+
change its formatter as above:
|
|
1166
|
+
|
|
1167
|
+
~~~ruby
|
|
1168
|
+
console = SemanticLogger.appenders.find { |appender| appender.console_stream == :stdout }
|
|
1169
|
+
console&.formatter = :json
|
|
1170
|
+
~~~
|
|
1171
|
+
|
|
1172
|
+
`find` returns `nil` when no appender writes to that stream.
|
|
1173
|
+
|
|
1174
|
+
Or remove it with `SemanticLogger.remove_appender(console)` and then add the new one.
|
|
1175
|
+
|
|
684
1176
|
### Flushing
|
|
685
1177
|
|
|
686
|
-
Semantic Logger automatically flushes all appenders (log files, etc.) when a process exits.
|
|
687
|
-
|
|
688
|
-
|
|
1178
|
+
Semantic Logger automatically flushes all appenders (log files, etc.) when a process exits. During
|
|
1179
|
+
an outage of the destination of an appender added with `async: true` or in batches, the flush
|
|
1180
|
+
discards that appender's queued messages once a write fails, rather than wait for each one to fail
|
|
1181
|
+
in turn. At exit, any other appender whose destination fails a write is given no more messages for
|
|
1182
|
+
the rest of the flush, which still flushes it; see
|
|
1183
|
+
[Retry a failing appender](operations.html#retry-a-failing-appender). The `flush` method
|
|
1184
|
+
is not defined on individual logger instances, since there may be many of them. To perform a global
|
|
1185
|
+
flush of all appenders and wait for any queued messages to be written:
|
|
689
1186
|
|
|
690
1187
|
~~~ruby
|
|
691
1188
|
SemanticLogger.flush
|
|
692
1189
|
~~~
|
|
693
1190
|
|
|
1191
|
+
Called on the thread of an appender added with `async: true` or in batches, such as by an appender
|
|
1192
|
+
of your own as it writes, `SemanticLogger.flush` returns `false` at once without flushing, since the
|
|
1193
|
+
thread that writes to every appender waits for that thread to flush its appender, and so would wait
|
|
1194
|
+
forever. So do `SemanticLogger.close` and `SemanticLogger.clear_appenders!`. The first such call is
|
|
1195
|
+
reported to the internal logger.
|
|
1196
|
+
|
|
694
1197
|
### Capturing context with `on_log`
|
|
695
1198
|
|
|
696
1199
|
Register a block to be called for every log entry, just before it is placed on the queue. The block
|
|
@@ -704,4 +1207,8 @@ end
|
|
|
704
1207
|
~~~
|
|
705
1208
|
|
|
706
1209
|
Because these callbacks run on the application's own thread, keep them fast. The captured context is
|
|
707
|
-
available to appenders and formatters as `log.context
|
|
1210
|
+
available to appenders and formatters as `log.context`, and formatters that write the context, such
|
|
1211
|
+
as `:loki`, write it too. To hand an appender of your own an object it needs from the logging thread
|
|
1212
|
+
that is not log data, such as a client's current scope, use `log.capture(key, value)` instead, and
|
|
1213
|
+
read it in the appender with `log.captured(key)`: formatters do not write it, unless one is built
|
|
1214
|
+
for that appender and reads its key.
|