semantic_logger 5.2.0 → 5.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/docs/api.md +160 -9
- data/docs/appenders.md +461 -34
- data/docs/config.md +543 -36
- data/docs/log.md +38 -7
- data/docs/metrics.md +139 -28
- data/docs/operations.md +293 -24
- data/docs/rails.md +319 -5
- data/docs/security.md +135 -107
- data/docs/testing.md +23 -0
- data/docs/upgrading.md +644 -1
- data/lib/semantic_logger/appender/async.rb +29 -11
- data/lib/semantic_logger/appender/bugsnag.rb +7 -2
- data/lib/semantic_logger/appender/cloudwatch_logs.rb +46 -6
- data/lib/semantic_logger/appender/elasticsearch_base.rb +197 -29
- data/lib/semantic_logger/appender/elasticsearch_http.rb +22 -5
- data/lib/semantic_logger/appender/file.rb +29 -14
- data/lib/semantic_logger/appender/graylog.rb +81 -7
- data/lib/semantic_logger/appender/honeybadger.rb +7 -1
- data/lib/semantic_logger/appender/honeybadger_insights.rb +8 -2
- data/lib/semantic_logger/appender/http.rb +244 -41
- data/lib/semantic_logger/appender/io.rb +11 -6
- data/lib/semantic_logger/appender/kafka.rb +192 -46
- data/lib/semantic_logger/appender/loki.rb +42 -6
- data/lib/semantic_logger/appender/mongodb.rb +94 -9
- data/lib/semantic_logger/appender/new_relic.rb +2 -3
- data/lib/semantic_logger/appender/new_relic_logs.rb +30 -8
- data/lib/semantic_logger/appender/open_telemetry.rb +37 -35
- data/lib/semantic_logger/appender/rabbitmq.rb +241 -17
- data/lib/semantic_logger/appender/sentry.rb +2 -3
- data/lib/semantic_logger/appender/sentry_ruby.rb +82 -15
- data/lib/semantic_logger/appender/splunk.rb +4 -4
- data/lib/semantic_logger/appender/splunk_http.rb +92 -1
- data/lib/semantic_logger/appender/syslog.rb +89 -10
- data/lib/semantic_logger/appender/tcp.rb +68 -98
- data/lib/semantic_logger/appender/text_output.rb +65 -0
- data/lib/semantic_logger/appender/udp.rb +130 -18
- data/lib/semantic_logger/appender/wrapper.rb +10 -7
- data/lib/semantic_logger/appender.rb +115 -10
- data/lib/semantic_logger/appenders.rb +136 -4
- data/lib/semantic_logger/backtrace_cleaning.rb +77 -0
- data/lib/semantic_logger/base.rb +53 -71
- data/lib/semantic_logger/call_site.rb +235 -0
- data/lib/semantic_logger/context.rb +222 -0
- data/lib/semantic_logger/core_ext/process.rb +20 -5
- data/lib/semantic_logger/core_ext/warning.rb +70 -0
- data/lib/semantic_logger/deprecation.rb +139 -0
- data/lib/semantic_logger/exception_details.rb +4 -3
- data/lib/semantic_logger/formatters/base.rb +139 -19
- data/lib/semantic_logger/formatters/color.rb +3 -4
- data/lib/semantic_logger/formatters/default.rb +35 -7
- data/lib/semantic_logger/formatters/ecs.rb +9 -4
- data/lib/semantic_logger/formatters/json.rb +9 -0
- data/lib/semantic_logger/formatters/logfmt.rb +9 -7
- data/lib/semantic_logger/formatters/loki/labels.rb +282 -0
- data/lib/semantic_logger/formatters/loki.rb +412 -24
- data/lib/semantic_logger/formatters/new_relic_logs.rb +15 -13
- data/lib/semantic_logger/formatters/one_line.rb +44 -10
- data/lib/semantic_logger/formatters/open_telemetry.rb +186 -29
- data/lib/semantic_logger/formatters/pattern.rb +60 -8
- data/lib/semantic_logger/formatters/raw.rb +38 -8
- data/lib/semantic_logger/formatters/signalfx.rb +55 -15
- data/lib/semantic_logger/formatters/syslog.rb +4 -2
- data/lib/semantic_logger/formatters/syslog_cee.rb +10 -4
- data/lib/semantic_logger/formatters/syslog_header.rb +49 -0
- data/lib/semantic_logger/formatters.rb +1 -0
- data/lib/semantic_logger/log.rb +236 -80
- data/lib/semantic_logger/loggable.rb +55 -95
- data/lib/semantic_logger/logger.rb +31 -5
- data/lib/semantic_logger/metric/new_relic.rb +32 -17
- data/lib/semantic_logger/metric/signalfx.rb +21 -12
- data/lib/semantic_logger/metric/statsd.rb +26 -8
- data/lib/semantic_logger/processor.rb +194 -5
- data/lib/semantic_logger/queue_processor/backoff.rb +140 -0
- data/lib/semantic_logger/queue_processor/batch_failure.rb +69 -0
- data/lib/semantic_logger/queue_processor/held_write.rb +62 -0
- data/lib/semantic_logger/queue_processor/inbox.rb +97 -0
- data/lib/semantic_logger/queue_processor/reopener.rb +56 -0
- data/lib/semantic_logger/queue_processor/reporter.rb +113 -0
- data/lib/semantic_logger/queue_processor/request.rb +48 -10
- data/lib/semantic_logger/queue_processor/throttled_counter.rb +60 -17
- data/lib/semantic_logger/queue_processor.rb +426 -176
- data/lib/semantic_logger/ractor_copy.rb +120 -0
- data/lib/semantic_logger/ractor_processor.rb +82 -0
- data/lib/semantic_logger/ractor_receiver.rb +282 -0
- data/lib/semantic_logger/ractor_support.rb +28 -0
- data/lib/semantic_logger/redactor.rb +310 -0
- data/lib/semantic_logger/semantic_logger.rb +313 -79
- data/lib/semantic_logger/subscriber.rb +169 -15
- data/lib/semantic_logger/sync_processor.rb +226 -26
- data/lib/semantic_logger/test/capture_log_events.rb +20 -2
- data/lib/semantic_logger/test/minitest.rb +2 -1
- data/lib/semantic_logger/test/rspec.rb +1 -1
- data/lib/semantic_logger/trap_safe.rb +113 -0
- data/lib/semantic_logger/utils.rb +266 -79
- data/lib/semantic_logger/version.rb +1 -1
- data/lib/semantic_logger.rb +35 -2
- metadata +22 -2
data/docs/upgrading.md
CHANGED
|
@@ -3,7 +3,7 @@ layout: default
|
|
|
3
3
|
title: Upgrading
|
|
4
4
|
description: >-
|
|
5
5
|
Version-by-version upgrade notes for Semantic Logger, from v4.0 through the
|
|
6
|
-
v5.
|
|
6
|
+
v5.3 OpenTelemetry log record changes.
|
|
7
7
|
---
|
|
8
8
|
|
|
9
9
|
|
|
@@ -12,6 +12,649 @@ description: >-
|
|
|
12
12
|
* TOC
|
|
13
13
|
{:toc}
|
|
14
14
|
|
|
15
|
+
### Seeing deprecation warnings
|
|
16
|
+
|
|
17
|
+
Semantic Logger reports the use of a deprecated appender or option as a Ruby deprecation warning, the
|
|
18
|
+
same way that Ruby reports its own, naming the line of your application that used it, and what to use
|
|
19
|
+
instead:
|
|
20
|
+
|
|
21
|
+
~~~
|
|
22
|
+
config/initializers/logging.rb:12: warning: SemanticLogger: SemanticLogger::Appender::Sentry (appender: :sentry) is deprecated and will be removed in v6. It depends on the end-of-life sentry-raven gem. Use appender: :sentry_ruby instead.
|
|
23
|
+
~~~
|
|
24
|
+
|
|
25
|
+
When a gem adds the appender for you, the warning names the line of that gem that added it. A
|
|
26
|
+
warning about a log entry that an appender reports while it writes the entry, on Semantic Logger's
|
|
27
|
+
background thread, names no line, since your code is not on that thread; to see the line, run with
|
|
28
|
+
`SemanticLogger.sync!`, as tests often do.
|
|
29
|
+
|
|
30
|
+
Ruby only shows deprecation warnings while `Warning[:deprecated]` is true, which it is not by default,
|
|
31
|
+
so production stays quiet without any configuration:
|
|
32
|
+
|
|
33
|
+
| Where | Deprecation warnings |
|
|
34
|
+
|-------|----------------------|
|
|
35
|
+
| Tests run with Minitest or test-unit | On, since both turn them on |
|
|
36
|
+
| Tests run with RSpec | Off, unless run with `rspec --warnings`, or `Warning[:deprecated] = true` is set in `spec_helper.rb` |
|
|
37
|
+
| Development | Off, unless `RUBYOPT=-W:deprecated` is set in the environment, or `Warning[:deprecated] = true` in code that only runs in development, such as `config/environments/development.rb` |
|
|
38
|
+
| Production | Off |
|
|
39
|
+
|
|
40
|
+
Rails settings such as `config.active_support.deprecation` only apply to the deprecation warnings of
|
|
41
|
+
Rails, not to those that Ruby reports.
|
|
42
|
+
|
|
43
|
+
Ruby writes each warning to `$stderr`. To send them elsewhere, such as to a file of their own, or to
|
|
44
|
+
raise them in your tests so that each use is found, extend `Warning` with your own `warn`:
|
|
45
|
+
|
|
46
|
+
~~~ruby
|
|
47
|
+
module SemanticLoggerDeprecations
|
|
48
|
+
DEPRECATIONS = ::Logger.new("log/deprecations.log")
|
|
49
|
+
|
|
50
|
+
def warn(message, *, **)
|
|
51
|
+
return super unless message.include?("SemanticLogger: ")
|
|
52
|
+
|
|
53
|
+
DEPRECATIONS.warn(message.chomp)
|
|
54
|
+
end
|
|
55
|
+
end
|
|
56
|
+
Warning.extend(SemanticLoggerDeprecations)
|
|
57
|
+
~~~
|
|
58
|
+
|
|
59
|
+
Do not log them through Semantic Logger itself: a warning reported on its background thread would
|
|
60
|
+
then wait on that thread's own queue, which can stop all logging while the queue is full.
|
|
61
|
+
|
|
62
|
+
### Upgrading to Semantic Logger v5.3
|
|
63
|
+
|
|
64
|
+
#### OpenTelemetry log records
|
|
65
|
+
|
|
66
|
+
The `:open_telemetry` appender now sends the message as the body of each log record, as its
|
|
67
|
+
documentation described, instead of a map of the whole log entry. This matters if you query, alert
|
|
68
|
+
on, or build dashboards from the body of these records.
|
|
69
|
+
|
|
70
|
+
~~~
|
|
71
|
+
# Before:
|
|
72
|
+
body: {"host" => "web-1", "application" => "my_app", "environment" => "production",
|
|
73
|
+
"pid" => 12345, "thread" => "puma srv tp 001", "named_tags" => {request_id: "abc"},
|
|
74
|
+
"name" => "UserService", "message" => "Created user"}
|
|
75
|
+
attributes: {"user_id" => 42}
|
|
76
|
+
|
|
77
|
+
# After:
|
|
78
|
+
body: "Created user"
|
|
79
|
+
attributes: {"request_id" => "abc", "user_id" => 42, "semantic_logger.name" => "UserService",
|
|
80
|
+
"thread.name" => "puma srv tp 001", "process.pid" => 12345}
|
|
81
|
+
~~~
|
|
82
|
+
|
|
83
|
+
* The payload attributes keep their names. An Array that mixes types, such as `[1, "two"]`, is now
|
|
84
|
+
sent as Strings, `["1", "two"]`, since OpenTelemetry does not accept it otherwise.
|
|
85
|
+
* Each named tag is an attribute of its own, such as `request_id`, instead of `body.named_tags`. A
|
|
86
|
+
payload key takes precedence over a named tag of the same name.
|
|
87
|
+
* The other fields of the entry are attributes too. A logged exception uses the OpenTelemetry
|
|
88
|
+
semantic conventions, `exception.type`, `exception.message`, and `exception.stacktrace`, so that
|
|
89
|
+
backends recognize it, and so do `code.file.path`, `code.line.number`, `thread.name`, and
|
|
90
|
+
`process.pid`. The logger name, tags, duration, and metric are `semantic_logger.name`,
|
|
91
|
+
`semantic_logger.tags`, `semantic_logger.duration_ms`, `semantic_logger.metric`, and
|
|
92
|
+
`semantic_logger.metric_amount`, and each of a metric's dimensions is a
|
|
93
|
+
`semantic_logger.dimensions.<key>` attribute. A named tag or payload key with one of these names, or
|
|
94
|
+
starting with `semantic_logger.dimensions.`, is prefixed with its source, as in
|
|
95
|
+
`payload.exception.type`.
|
|
96
|
+
* The host, application, and environment are no longer sent. Set them in the OpenTelemetry resource
|
|
97
|
+
instead, which is sent once per export, with `OTEL_SERVICE_NAME=my_app` and
|
|
98
|
+
`OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production,host.name=web-1`.
|
|
99
|
+
* An entry without a message, such as `logger.error(exception)`, has no body.
|
|
100
|
+
* A named tag whose value is a Symbol, Time, or other object was sent as an empty value. It is now
|
|
101
|
+
sent as its `#to_s`, or as JSON for a Hash, Array, `Struct`, or `Set`.
|
|
102
|
+
* `code.file.path` and `code.line.number` are where the entry was logged, and are left out when the
|
|
103
|
+
entry has no call site (see `call_site_level`) and did not capture a backtrace. With a call site,
|
|
104
|
+
`code.function.name` names the method. The old body's `file` and `line` fell back to where a logged
|
|
105
|
+
exception was raised, which is now in `exception.stacktrace`.
|
|
106
|
+
|
|
107
|
+
A custom formatter given to this appender with `formatter:` must now return the keyword arguments for
|
|
108
|
+
the OpenTelemetry `Logger#on_emit` (`severity_text:`, `severity_number:`, `timestamp:`, `body:`, and
|
|
109
|
+
`attributes:`), as `SemanticLogger::Formatters::OpenTelemetry` does, instead of a Hash shaped like the
|
|
110
|
+
`:raw` formatter's.
|
|
111
|
+
|
|
112
|
+
The appender no longer sends metric-only entries, such as `logger.info(metric: "users/signup")`,
|
|
113
|
+
which have a metric but no message and no exception. It sent each one as a log record with no body,
|
|
114
|
+
its metric in the attributes, never as an OpenTelemetry metric. Metric-only entries are meant for the
|
|
115
|
+
[metric subscribers](metrics.html#send-metrics-to-a-backend), and most log appenders leave them out.
|
|
116
|
+
To keep sending them as log records, add the appender with `metrics: true`:
|
|
117
|
+
|
|
118
|
+
~~~ruby
|
|
119
|
+
SemanticLogger.add_appender(appender: :open_telemetry, metrics: true)
|
|
120
|
+
~~~
|
|
121
|
+
|
|
122
|
+
#### Choose `metrics:` for the `:kafka`, `:rabbitmq`, and `:udp` appenders
|
|
123
|
+
|
|
124
|
+
Unlike the other log appenders, `:kafka`, `:rabbitmq`, and `:udp` send metric-only entries by
|
|
125
|
+
default: entries such as `logger.info(metric: "users/signup")`, which have a metric but no message and
|
|
126
|
+
no exception, and are meant for the [metric subscribers](metrics.html#send-metrics-to-a-backend). v6
|
|
127
|
+
leaves them out by default, as the other log appenders do. Until then, adding one of these appenders
|
|
128
|
+
without `metrics:` still sends them, and reports a
|
|
129
|
+
[deprecation warning](#seeing-deprecation-warnings). Pass `metrics:` to choose:
|
|
130
|
+
|
|
131
|
+
~~~ruby
|
|
132
|
+
# Keep sending metric-only entries:
|
|
133
|
+
SemanticLogger.add_appender(appender: :kafka, seed_brokers: ["kafka1:9092"], metrics: true)
|
|
134
|
+
|
|
135
|
+
# Leave them out, as v6 does by default:
|
|
136
|
+
SemanticLogger.add_appender(appender: :kafka, seed_brokers: ["kafka1:9092"], metrics: false)
|
|
137
|
+
~~~
|
|
138
|
+
|
|
139
|
+
#### Negative metric amounts
|
|
140
|
+
|
|
141
|
+
A metric counts upward. In v6, a counter drops a negative `metric_amount`, and reports it once on
|
|
142
|
+
Semantic Logger's internal logger, rather than subtracting it. Until then, a log call or a
|
|
143
|
+
`measure_*` call with a `metric:` and a negative `metric_amount` is still logged and sent as before,
|
|
144
|
+
and reports a [deprecation warning](#seeing-deprecation-warnings) that names the line of the call.
|
|
145
|
+
It is reported
|
|
146
|
+
once per process, for the first negative amount only, so that a loop does not repeat it; once that
|
|
147
|
+
call is changed, the next one, if any, is reported. A `metric_amount` of zero is not negative, and
|
|
148
|
+
one without a `metric:`, which no metric subscriber receives, is not reported.
|
|
149
|
+
|
|
150
|
+
~~~ruby
|
|
151
|
+
# Deprecated:
|
|
152
|
+
logger.info(message: "Item removed", metric: "cart/items", metric_amount: -1)
|
|
153
|
+
|
|
154
|
+
# Count what goes down as a metric of its own:
|
|
155
|
+
logger.info(message: "Item removed", metric: "cart/items_removed")
|
|
156
|
+
~~~
|
|
157
|
+
|
|
158
|
+
To record a value that goes up and down, such as the number of items in a cart, a `gauge` is planned
|
|
159
|
+
for the metrics API of v6.
|
|
160
|
+
|
|
161
|
+
#### Choose where Loki writes `pid` and `thread`
|
|
162
|
+
|
|
163
|
+
The `:loki` formatter writes the process id and the thread name as the stream labels `pid` and
|
|
164
|
+
`thread`, so every process restart, and every thread, adds a Loki stream. Its new
|
|
165
|
+
`structured_metadata:` option writes either or both as structured metadata instead, as Loki advises,
|
|
166
|
+
and v6 writes both there by default. Until then, adding the `:loki` appender, or building its
|
|
167
|
+
formatter, without `structured_metadata:` still writes them as labels, and reports a
|
|
168
|
+
[deprecation warning](#seeing-deprecation-warnings). Pass it to choose:
|
|
169
|
+
|
|
170
|
+
~~~ruby
|
|
171
|
+
# Write them as structured metadata, as v6 does by default:
|
|
172
|
+
SemanticLogger.add_appender(appender: :loki, url: "https://logs-prod-001.grafana.net",
|
|
173
|
+
formatter: {loki: {structured_metadata: [:pid, :thread]}})
|
|
174
|
+
|
|
175
|
+
# Keep them as labels:
|
|
176
|
+
SemanticLogger.add_appender(appender: :loki, url: "https://logs-prod-001.grafana.net",
|
|
177
|
+
formatter: {loki: {structured_metadata: []}})
|
|
178
|
+
~~~
|
|
179
|
+
|
|
180
|
+
Structured metadata needs a Loki with schema v13 and `allow_structured_metadata`, which is on by
|
|
181
|
+
default since Loki 3.0. Any other Loki rejects every entry once `pid` is structured metadata, so
|
|
182
|
+
keep them as labels until your Loki accepts it.
|
|
183
|
+
|
|
184
|
+
As structured metadata, a query filters on them after the stream selector, a payload key named `pid`
|
|
185
|
+
or `thread` is written as `payload_pid` or `payload_thread`, and a named tag or context key of that
|
|
186
|
+
name as the label `pid_2` or `thread_2`:
|
|
187
|
+
|
|
188
|
+
```
|
|
189
|
+
# Labels:
|
|
190
|
+
{application="shop", thread="worker-1"}
|
|
191
|
+
# Structured metadata:
|
|
192
|
+
{application="shop"} | thread="worker-1"
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
See [pid and thread as structured metadata](appenders.html#pid-and-thread-as-structured-metadata).
|
|
196
|
+
|
|
197
|
+
#### Set the options of an appender instance on the instance
|
|
198
|
+
|
|
199
|
+
`add_appender` adds an appender or metric subscriber that is already created, given as `appender:` or
|
|
200
|
+
`metric:`, as it is, since it was created with its own options. A formatter block, or the options of
|
|
201
|
+
an appender, such as `level:`, `formatter:`, or `filter:`, given with it were ignored without a word.
|
|
202
|
+
They still are, but `add_appender` now reports a [deprecation warning](#seeing-deprecation-warnings)
|
|
203
|
+
that names them, and v6 raises `ArgumentError` for them. Set them on the instance itself, such as
|
|
204
|
+
when creating a [custom appender](config.html#custom-appenders):
|
|
205
|
+
|
|
206
|
+
~~~ruby
|
|
207
|
+
# Before, level: and formatter: were ignored:
|
|
208
|
+
appender = SimpleAppender.new(url: "https://logs.example.com")
|
|
209
|
+
SemanticLogger.add_appender(appender: appender, level: :warn, formatter: :json)
|
|
210
|
+
|
|
211
|
+
# After:
|
|
212
|
+
appender = SimpleAppender.new(url: "https://logs.example.com", level: :warn, formatter: :json)
|
|
213
|
+
SemanticLogger.add_appender(appender: appender)
|
|
214
|
+
~~~
|
|
215
|
+
|
|
216
|
+
The options of `add_appender` itself, such as `async:`, `batch:`, and `max_queue_size:`, still apply
|
|
217
|
+
to an instance. An option given as `nil`, such as `level: nil`, sets nothing, so it is not reported.
|
|
218
|
+
|
|
219
|
+
#### Give `add_appender` one destination
|
|
220
|
+
|
|
221
|
+
`add_appender` adds one destination, given as `file_name:`, `io:`, `logger:`, `appender:`, or
|
|
222
|
+
`metric:`. Given more than one of them, it added only the first, in that order, and ignored the
|
|
223
|
+
others without a word, so that `add_appender(file_name: "log/app.log", io: $stdout)` wrote to the
|
|
224
|
+
file, and nothing to `$stdout`. It still does, but now reports a
|
|
225
|
+
[deprecation warning](#seeing-deprecation-warnings) that names the destination it uses and those
|
|
226
|
+
it ignores, and v6 raises `ArgumentError` for them. Call `add_appender` once for each destination:
|
|
227
|
+
|
|
228
|
+
~~~ruby
|
|
229
|
+
# Before, $stdout was ignored:
|
|
230
|
+
SemanticLogger.add_appender(file_name: "log/app.log", io: $stdout)
|
|
231
|
+
|
|
232
|
+
# After:
|
|
233
|
+
SemanticLogger.add_appender(file_name: "log/app.log")
|
|
234
|
+
SemanticLogger.add_appender(io: $stdout)
|
|
235
|
+
~~~
|
|
236
|
+
|
|
237
|
+
An option that only the ignored destination accepts, such as the `url:` of `appender: :http`, went
|
|
238
|
+
to the one that was added, which raised `ArgumentError: unknown keyword`. While
|
|
239
|
+
[deprecation warnings are shown](#seeing-deprecation-warnings), the warning now comes before it,
|
|
240
|
+
naming the destination that was ignored.
|
|
241
|
+
|
|
242
|
+
`appender: :wrapper` only worked with `logger:`, which `add_appender` took as the destination
|
|
243
|
+
instead, so it now reports the same warning. Leave out `appender: :wrapper`:
|
|
244
|
+
`add_appender(logger: my_logger)` adds the same appender.
|
|
245
|
+
|
|
246
|
+
A destination given as `nil`, such as `file_name: options[:log_file]` when no log file is set, is not
|
|
247
|
+
one, so it is not reported.
|
|
248
|
+
|
|
249
|
+
#### Batch an appender with a formatter that can format a batch
|
|
250
|
+
|
|
251
|
+
`add_appender` now raises `ArgumentError` when it would batch an appender whose formatter cannot
|
|
252
|
+
format a batch, such as `formatter: :default` or a Proc given to `:http`, `:loki`, or
|
|
253
|
+
`metric: :signalfx` with `batch: true`, or a formatter block, which v5.3 now uses (see
|
|
254
|
+
[Other behaviour changes](#other-behaviour-changes)). Before, it added the appender, but every batch
|
|
255
|
+
failed on the appender's worker thread, which backed off and tried again, so nothing was ever
|
|
256
|
+
written, and once its queue was full, logging blocked, unless it was added with
|
|
257
|
+
`non_blocking: true`. Add the appender without
|
|
258
|
+
`batch: true`, or give it a formatter that implements `#batch(logs, logger)`, such as the one it
|
|
259
|
+
uses by default:
|
|
260
|
+
|
|
261
|
+
~~~ruby
|
|
262
|
+
# Before, nothing was ever sent:
|
|
263
|
+
SemanticLogger.add_appender(appender: :http, url: "https://logs.example.com", batch: true,
|
|
264
|
+
formatter: ->(log, _logger) { log.message })
|
|
265
|
+
|
|
266
|
+
# After, one request for each entry:
|
|
267
|
+
SemanticLogger.add_appender(appender: :http, url: "https://logs.example.com",
|
|
268
|
+
formatter: ->(log, _logger) { log.message })
|
|
269
|
+
|
|
270
|
+
# Or one request for each batch, formatted by the default :json formatter:
|
|
271
|
+
SemanticLogger.add_appender(appender: :http, url: "https://logs.example.com", batch: true)
|
|
272
|
+
~~~
|
|
273
|
+
|
|
274
|
+
`:splunk_http` with `batch: true` is not refused: it now sends each batch as the HTTP Event
|
|
275
|
+
Collector's event objects, stacked one after the other. Before, every batch failed. See
|
|
276
|
+
[Splunk HTTP](appenders.html#splunk-http).
|
|
277
|
+
|
|
278
|
+
A custom appender whose `#batch` formats each entry with `format_log` is batched with any formatter,
|
|
279
|
+
as before, including a subclass of `SemanticLogger::Appender::Http` that replaces its `#batch`. A
|
|
280
|
+
custom appender that cannot be batched in some configuration, such as one whose `#batch` formats the
|
|
281
|
+
whole batch with `format_batch`, can now say why from `#batch_refusal`, so that `add_appender`
|
|
282
|
+
raises instead. So can a subclass of `SemanticLogger::Appender::Http` whose `#batch` calls `super`,
|
|
283
|
+
which is not refused on its own. See [Custom appenders](config.html#custom-appenders).
|
|
284
|
+
|
|
285
|
+
#### Batch Elasticsearch with the `:elasticsearch` appender
|
|
286
|
+
|
|
287
|
+
`add_appender(appender: :elasticsearch_http, batch: true)` now raises `ArgumentError`. Batching it
|
|
288
|
+
never wrote anything: it posted each batch as a JSON array to the url's own path, such as `/`, which
|
|
289
|
+
Elasticsearch rejects, since it indexes a batch of entries only through its bulk API. Use the
|
|
290
|
+
`:elasticsearch` appender, which sends its entries in bulk requests, a batch at a time by default,
|
|
291
|
+
or add `:elasticsearch_http` without `batch: true`, to send each entry in its own request:
|
|
292
|
+
|
|
293
|
+
~~~ruby
|
|
294
|
+
# Before, nothing was ever indexed:
|
|
295
|
+
SemanticLogger.add_appender(appender: :elasticsearch_http, url: "http://localhost:9200", batch: true)
|
|
296
|
+
|
|
297
|
+
# After, in batches, through the bulk API (requires the elasticsearch gem):
|
|
298
|
+
SemanticLogger.add_appender(appender: :elasticsearch, url: "http://localhost:9200")
|
|
299
|
+
|
|
300
|
+
# Or one request for each entry:
|
|
301
|
+
SemanticLogger.add_appender(appender: :elasticsearch_http, url: "http://localhost:9200")
|
|
302
|
+
~~~
|
|
303
|
+
|
|
304
|
+
#### Loki stack traces
|
|
305
|
+
|
|
306
|
+
The `:loki` formatter now writes the `stack_trace` of a logged exception one frame per line, followed
|
|
307
|
+
by the backtrace of each exception that caused it, as the text formatters do. It wrote an inspected
|
|
308
|
+
Ruby Array and left out the causes. Update any query or alert that matches the old form:
|
|
309
|
+
|
|
310
|
+
~~~
|
|
311
|
+
# Before:
|
|
312
|
+
["app/models/order.rb:12:in 'Order#ship'", "app/controllers/orders_controller.rb:7:in 'OrdersController#create'"]
|
|
313
|
+
|
|
314
|
+
# After:
|
|
315
|
+
app/models/order.rb:12:in 'Order#ship'
|
|
316
|
+
app/controllers/orders_controller.rb:7:in 'OrdersController#create'
|
|
317
|
+
Cause: ArgumentError: invalid quantity
|
|
318
|
+
app/models/line_item.rb:3:in 'LineItem#save'
|
|
319
|
+
~~~
|
|
320
|
+
|
|
321
|
+
#### Custom formatters with a `backtrace_cleaner` of their own
|
|
322
|
+
|
|
323
|
+
Formatters now accept a `backtrace_cleaner` option (see
|
|
324
|
+
[Cleaning backtraces](config.html#cleaning-backtraces)), which the constructor of
|
|
325
|
+
`SemanticLogger::Formatters::Base` sets. A formatter of your own that keeps its own cleaner in
|
|
326
|
+
`@backtrace_cleaner`, and assigns it before calling `super`, has it replaced, by `true` in a
|
|
327
|
+
formatter built on the text or JSON formatters.
|
|
328
|
+
|
|
329
|
+
If your formatter cleans backtraces itself, rename its variable. Otherwise pass the cleaner to
|
|
330
|
+
`super` as the option, and the formatter it is built on cleans the backtraces it writes with it:
|
|
331
|
+
|
|
332
|
+
~~~ruby
|
|
333
|
+
class MyJson < SemanticLogger::Formatters::Json
|
|
334
|
+
def initialize(**args)
|
|
335
|
+
super(backtrace_cleaner: Rails.backtrace_cleaner, **args)
|
|
336
|
+
end
|
|
337
|
+
end
|
|
338
|
+
~~~
|
|
339
|
+
|
|
340
|
+
#### `logger_measure_method` has been removed
|
|
341
|
+
|
|
342
|
+
`logger_measure_method`, an experiment that `SemanticLogger::Loggable` added to each class, has been
|
|
343
|
+
removed. A class that still calls it raises `NoMethodError` when it is loaded. To keep logging how
|
|
344
|
+
long a method takes, wrap its body in a `measure_*` block, and pass `metric:` to keep the metric it
|
|
345
|
+
logged:
|
|
346
|
+
|
|
347
|
+
~~~ruby
|
|
348
|
+
def run
|
|
349
|
+
logger.measure_info("#run", metric: "MyClass/run") do
|
|
350
|
+
# ...
|
|
351
|
+
end
|
|
352
|
+
end
|
|
353
|
+
~~~
|
|
354
|
+
|
|
355
|
+
To trace the method instead, use OpenTelemetry's `tracer.in_span`, or the method tracer of an APM
|
|
356
|
+
agent.
|
|
357
|
+
|
|
358
|
+
#### Rails: cleaned backtraces, redaction, and call sites by default
|
|
359
|
+
|
|
360
|
+
If you use [rails_semantic_logger](rails.html), its v5.3 release requires Semantic Logger v5.3 and
|
|
361
|
+
turns on three of its new features by default: exception backtraces are cleaned with
|
|
362
|
+
`Rails.backtrace_cleaner`, the keys in `config.filter_parameters` are redacted from every log entry,
|
|
363
|
+
and in development each entry records its call site. Each changes the log output and can be turned
|
|
364
|
+
off. See [Upgrading to v5.3](rails.html#upgrading-to-v53) for details.
|
|
365
|
+
|
|
366
|
+
#### Other behaviour changes
|
|
367
|
+
|
|
368
|
+
* The `:loki` formatter now writes a payload key named like one of its own structured metadata
|
|
369
|
+
fields (`metric`, `metric_value`, `dimensions`, `duration`, `duration_human`, `exception_name`,
|
|
370
|
+
`exception_message`, and `stack_trace`), or that Loki stores under one of those names, such as
|
|
371
|
+
`metric.value`, with a `payload_` prefix, also in an entry without that field, and under the name
|
|
372
|
+
Loki stores it as, such as `payload_metric_value`. Before, Loki kept either the payload's value or
|
|
373
|
+
the formatter's. Update any query or alert on such a payload key:
|
|
374
|
+
|
|
375
|
+
```
|
|
376
|
+
# Before:
|
|
377
|
+
{application="shop"} | duration="PT5S"
|
|
378
|
+
# After:
|
|
379
|
+
{application="shop"} | payload_duration="PT5S"
|
|
380
|
+
```
|
|
381
|
+
* The `:loki` formatter no longer lets a named tag, or a value that an `on_log` subscriber adds to
|
|
382
|
+
the context, replace a label of the same name. Its own labels (`pid`, `application`,
|
|
383
|
+
`environment`, `host`, `level`, `thread`, and `tags`) keep their values, a named tag keeps its
|
|
384
|
+
value over a context key, and the later of the two is written with a number added, as in
|
|
385
|
+
`host_2`, unless both have the same value. Before, the later value replaced the earlier one. Update
|
|
386
|
+
any query or alert that selects such a named tag or context value by that name:
|
|
387
|
+
|
|
388
|
+
```
|
|
389
|
+
# Before, with the named tag host: "db-1":
|
|
390
|
+
{application="shop", host="db-1"}
|
|
391
|
+
# After:
|
|
392
|
+
{application="shop", host_2="db-1"}
|
|
393
|
+
```
|
|
394
|
+
|
|
395
|
+
A subclass of `SemanticLogger::Formatters::Loki` that reads, replaces, or deletes a label in
|
|
396
|
+
`stream[:stream]` needs to use a String key, such as `stream[:stream]["host"]`. The label names
|
|
397
|
+
are now Strings rather than Symbols, so that a named tag or context key, whether a Symbol or a
|
|
398
|
+
String, is found among them by name. A label that the subclass writes under a Symbol, as v5.2
|
|
399
|
+
did, is written under its name as a String, and with a number added when a label written before
|
|
400
|
+
it has that name, as a named tag is: `stream[:stream][:level] = "INFO"` after `super` adds
|
|
401
|
+
`level_2="INFO"` rather than replacing `level`.
|
|
402
|
+
|
|
403
|
+
A subclass that overrides both `#build_stream` and `#call` (or `#batch`), calling `super` in
|
|
404
|
+
neither, leaves out each named tag and context value whose name another label has, or that Loki
|
|
405
|
+
does not accept, such as `request-id`. The formatter writes those labels last, once the labels
|
|
406
|
+
that keep their names are written: from its own `#build_stream`, or, for a subclass whose
|
|
407
|
+
`#build_stream` does not call `super`, from its own `#call` and `#batch`. Such a subclass calls
|
|
408
|
+
neither. To keep them, call `super` from the subclass's `#build_stream` and add or change its own
|
|
409
|
+
labels in the stream it returns, or remove its `#call` and `#batch` when they repeat the
|
|
410
|
+
formatter's own, as a copy of v5.2's does.
|
|
411
|
+
* The `:loki` formatter now writes a label past Loki's default limits, the 16th and later label of a
|
|
412
|
+
stream, or one whose name is longer than 1024 bytes or whose value is longer than 2048 bytes, to
|
|
413
|
+
the structured metadata instead, since Loki rejected the whole push for it. If your Loki raises
|
|
414
|
+
these limits in its `limits_config`, set the formatter's options of the same names to match, or
|
|
415
|
+
those labels can no longer be selected as labels:
|
|
416
|
+
|
|
417
|
+
```ruby
|
|
418
|
+
SemanticLogger.add_appender(
|
|
419
|
+
appender: :loki,
|
|
420
|
+
url: "https://logs-prod-001.grafana.net",
|
|
421
|
+
formatter: {loki: {structured_metadata: [:pid, :thread], max_label_names_per_series: 30}}
|
|
422
|
+
)
|
|
423
|
+
```
|
|
424
|
+
|
|
425
|
+
See [Loki's label limits](appenders.html#lokis-label-limits).
|
|
426
|
+
* The `#logger` of an appender added with `logger:`, such as the one `add_appender(logger: ...)`
|
|
427
|
+
returns, is now its internal logger, the one it reports its own problems to, as for every other
|
|
428
|
+
appender, and `#logger=` replaces that internal logger. Before, they returned and replaced the
|
|
429
|
+
logger the appender writes to, which is now `#wrapped_logger` (the appender returned by
|
|
430
|
+
`add_appender(logger: ..., async: true)` also answers it): `appender.logger = other` sent the
|
|
431
|
+
entries to `other`, and `appender.logger.level = ::Logger::DEBUG` changed the level of the logger
|
|
432
|
+
written to. To write to another logger, add an appender for it with
|
|
433
|
+
`SemanticLogger.add_appender(logger: other)`, and remove the old one with
|
|
434
|
+
`SemanticLogger.remove_appender(appender)`. The `:open_telemetry` appender's `#logger` and
|
|
435
|
+
`#logger=` likewise no longer return or replace the OpenTelemetry logger.
|
|
436
|
+
|
|
437
|
+
A subclass of `SemanticLogger::Appender::Wrapper` that writes through `logger` or `@logger` needs
|
|
438
|
+
to use `wrapped_logger` instead. `logger` is now the internal logger, which by default writes only
|
|
439
|
+
warnings and errors, to standard error, so the subclass's entries would be left out, and `@logger`
|
|
440
|
+
is not set until `logger` is first called.
|
|
441
|
+
* The `:statsd` and `:new_relic` metric subscribers now record a metric that has `dimensions` under
|
|
442
|
+
its name alone, leaving out the dimensions, which their clients do not accept. Before, they left
|
|
443
|
+
such a metric out, so a counter or timer logged with dimensions can now show higher counts there.
|
|
444
|
+
To leave them out again, add either subscriber with a filter:
|
|
445
|
+
|
|
446
|
+
~~~ruby
|
|
447
|
+
SemanticLogger.add_appender(metric: :statsd, filter: ->(log) { !log.dimensions? })
|
|
448
|
+
SemanticLogger.add_appender(metric: :new_relic, filter: ->(log) { !log.dimensions? })
|
|
449
|
+
~~~
|
|
450
|
+
* A `measure_*` call now keeps its `dimensions:` option, which it used to drop. So the `:signalfx`
|
|
451
|
+
metric subscriber sends a metric measured with dimensions as it sends any metric with dimensions:
|
|
452
|
+
under its own name, such as `supplier.add_user` for `metric: "supplier/add_user"`, with those
|
|
453
|
+
dimensions, and without a count. Before, it sent it as `Application.average` and
|
|
454
|
+
`Application.counter`, with `class` and `action` dimensions taken from the metric name. It also
|
|
455
|
+
no longer adds the named tags listed in the subscriber's own `dimensions:` option, which it only
|
|
456
|
+
adds to a metric without dimensions, so put any of those values the metric needs in its
|
|
457
|
+
`dimensions:`, such as `dimensions: {supplier: "acme", region: "eu-west"}`. A filter on
|
|
458
|
+
`log.dimensions?`, as above, now applies to it too.
|
|
459
|
+
|
|
460
|
+
Empty dimensions, such as `dimensions: {}` or `{supplier: nil}.compact`, and dimensions whose
|
|
461
|
+
values are all blank (`nil`, or text that is empty or only whitespace), such as `{supplier: nil}`
|
|
462
|
+
or `{state: ""}`, now count as none for `:signalfx`, on a log call too: the metric is sent as it
|
|
463
|
+
would be without them, so a duration is sent as `Application.average` and `Application.counter`,
|
|
464
|
+
and a count as `Application.counter`, each with the `class` and `action` dimensions taken from the
|
|
465
|
+
metric name and the named tags listed in the subscriber's `dimensions:` option. Before, either was
|
|
466
|
+
sent under its own name, without those dimensions, and a duration without a count. A value of only
|
|
467
|
+
whitespace, which was sent before, is now also left out of dimensions that have other values, and
|
|
468
|
+
of the named tags it forwards. `log.dimensions?`, and so a filter on it, as above, still counts
|
|
469
|
+
dimensions whose values are all blank, such as `{state: ""}`, as dimensions; only empty ones, such
|
|
470
|
+
as `{}`, count as none.
|
|
471
|
+
* A `tagged` block now restores the tags and named tags that it found when it exits, instead of
|
|
472
|
+
popping the ones it pushed, as `silence` and `with_level` blocks already did. This only changes code
|
|
473
|
+
that pushes or pops tags with `push_tags` and `pop_tags`, or `push_named_tags` and
|
|
474
|
+
`pop_named_tags`, across the edge of a block:
|
|
475
|
+
* Tags pushed inside a block, and not popped, are removed when the block exits. Before, the block
|
|
476
|
+
popped them in place of its own tags, which then stayed on the thread.
|
|
477
|
+
* Tags pushed before a block, and popped inside it, come back when the block exits.
|
|
478
|
+
|
|
479
|
+
Push and pop tags in matching pairs, or use the block form.
|
|
480
|
+
* `log_exception: :partial`, the default for `measure_*` blocks, now also adds the class of each
|
|
481
|
+
exception that caused the logged one to the message, and its message when the text does not
|
|
482
|
+
already hold it, as `log_exception: :full` already wrote them:
|
|
483
|
+
|
|
484
|
+
```
|
|
485
|
+
# Before:
|
|
486
|
+
Saving order -- Exception: RuntimeError: Could not save
|
|
487
|
+
# After:
|
|
488
|
+
Saving order -- Exception: RuntimeError: Could not save -- Cause: IOError: disk full
|
|
489
|
+
```
|
|
490
|
+
|
|
491
|
+
Alerts, searches, or tests that match the whole message of such an entry need updating.
|
|
492
|
+
* An exception whose class overrides `self.name` or `self.to_s` is now named by the constant the
|
|
493
|
+
class was assigned to, as Ruby names it in `Exception#inspect`, wherever Semantic Logger writes the
|
|
494
|
+
class. Before, what `self.name` returned was written as `name` in `:raw`, and so in `:json`,
|
|
495
|
+
`:logfmt`, `:fluentd`, `:ecs`, `:syslog_cee`, and `:new_relic_logs`, as `exception_name` in
|
|
496
|
+
`:loki`, and in each `Cause:` line of a backtrace, while the text formatters (`:default`,
|
|
497
|
+
`:color`, `:one_line`, `:pattern`, and `:syslog`) and `log_exception: :partial` wrote what
|
|
498
|
+
`self.to_s` returned. A class without a constant is named as Ruby writes the class, such as
|
|
499
|
+
`#<Class:0x000...>`, even when it gives itself a name. Update any search, alert, or dashboard that
|
|
500
|
+
selects such an exception by the name its class gave itself:
|
|
501
|
+
|
|
502
|
+
```ruby
|
|
503
|
+
class DeclinedError < StandardError
|
|
504
|
+
def self.name = "Payments::Declined"
|
|
505
|
+
def self.to_s = name
|
|
506
|
+
end
|
|
507
|
+
# Before, in :json: "exception":{"name":"Payments::Declined",...}
|
|
508
|
+
# in :default: -- Exception: Payments::Declined: Card declined
|
|
509
|
+
# After, in :json: "exception":{"name":"DeclinedError",...}
|
|
510
|
+
# in :default: -- Exception: DeclinedError: Card declined
|
|
511
|
+
```
|
|
512
|
+
* `:default`, `:color`, `:pattern`, and `:syslog` with `escape_control_chars: false` now always
|
|
513
|
+
write UTF-8, as the JSON based formatters, `:one_line`, and `:syslog` with its default
|
|
514
|
+
`escape_control_chars: true` already did. Text in another encoding, such as ISO-8859-1, used to be
|
|
515
|
+
written in that encoding, and an entry that mixed it with UTF-8 text was lost; it is now converted
|
|
516
|
+
to UTF-8, and so are a pattern or time format in another encoding. Binary text is read as the
|
|
517
|
+
UTF-8 it usually holds, without its bytes that are not valid UTF-8, instead of being written
|
|
518
|
+
byte for byte. An application whose log files, or the tools that read them, expect another
|
|
519
|
+
encoding needs to expect UTF-8 instead.
|
|
520
|
+
* `escape_control_chars: true`, which the `:syslog` formatter and the syslog appender use by
|
|
521
|
+
default, now also escapes the Unicode characters that control how text is shown, as `\uXXXX`: the
|
|
522
|
+
C1 control characters (U+0080 to U+009F), the line and paragraph separators (U+2028, U+2029), and
|
|
523
|
+
the bidirectional controls (U+202A to U+202E, U+2066 to U+2069). Searches or parsers that match
|
|
524
|
+
logged text holding them, such as localized text wrapped in bidirectional isolates, need to match
|
|
525
|
+
the escaped form instead.
|
|
526
|
+
* The `:new_relic` metric subscriber now adds `Custom/` to a metric name that starts with `Custom`
|
|
527
|
+
but not with `Custom/`, recording `Customers/signup` as `Custom/Customers/signup`, and
|
|
528
|
+
`CustomerSignup` as `Custom/CustomerSignup`. Before, it recorded such a name as it was, which New
|
|
529
|
+
Relic does not chart as a custom metric. With another `prefix:`, the same applies to a name that
|
|
530
|
+
starts with that prefix but not with it and a `/`. A prefix given with a trailing `/` no longer
|
|
531
|
+
doubles it: with `prefix: "Custom/"`, `Orders/placed` is recorded as `Custom/Orders/placed`, not
|
|
532
|
+
`Custom//Orders/placed`. Dashboards, alerts, and NRQL queries on an old name stop receiving data,
|
|
533
|
+
so update them to the new one.
|
|
534
|
+
* `SemanticLogger.reopen` within the same process, such as `reopen(force: true)` after a log
|
|
535
|
+
rotation, keeps the log entries that are still queued, and the thread that writes them, rather
|
|
536
|
+
than replacing both and losing the entries. So it no longer stops a thread that is waiting
|
|
537
|
+
between retries after an appender failed, or one that is stuck writing; it only starts the thread
|
|
538
|
+
again once it has stopped. After a fork, the child still starts with an empty queue.
|
|
539
|
+
* The `:http`, `:elasticsearch_http`, `:splunk_http`, and `:loki` appenders, and the `:signalfx`
|
|
540
|
+
metric subscriber, now raise an error from `#log` and `#batch` when the server answers with a status
|
|
541
|
+
other than 2xx, instead of reporting it and carrying on. With `async: true` or `batch: true`, the
|
|
542
|
+
entries the server did not accept are now counted as `failed` in `SemanticLogger.stats`, rather
|
|
543
|
+
than `processed`. On a 408, 429, or 503, the worker thread now backs off (1 second, then 2, ...),
|
|
544
|
+
then sends the same entry or batch again, up to 3 times, before it discards it and carries on with
|
|
545
|
+
the next one. So does a request it could not send because it could not connect, which was
|
|
546
|
+
discarded before, except that such a failure does not count towards the 3 times, since the server
|
|
547
|
+
never answered. On any other 5xx, such as a 500, 502, or 504, or a redirect, it discards the entry
|
|
548
|
+
or batch, then backs off before sending the next one. It stops for good after `async_max_retries`
|
|
549
|
+
(default `100`) failures in a row. During an outage the entries logged meanwhile now wait on the
|
|
550
|
+
queue while the worker thread backs off, rather than being sent, and discarded, one per back-off,
|
|
551
|
+
so with a capped queue (the default) and without `non_blocking`, logging waits once the queue is
|
|
552
|
+
full until the server is back or the thread gives up. Pass `async_max_retries: -1` to keep trying
|
|
553
|
+
through a long outage. On a 400, 413, or 422, a batch is now
|
|
554
|
+
split up and sent again in parts, to find the entries the server rejects, so a server that kept
|
|
555
|
+
some entries of a request it rejected can store them twice. On any other 4xx, such as 401 or 404,
|
|
556
|
+
which the server answers whatever the request holds, the whole batch is discarded at once. See
|
|
557
|
+
[When the server does not accept a request](appenders.html#when-the-server-does-not-accept-a-request).
|
|
558
|
+
|
|
559
|
+
While the worker thread backs off, the entries logged meanwhile wait on the appender's queue, as
|
|
560
|
+
they already did while the server could not be reached. Once that queue, and then the main queue,
|
|
561
|
+
are full, logging calls block until the server accepts requests again. Add an HTTP appender that
|
|
562
|
+
uses `async: true` or `batch: true` with `non_blocking: true` as well, so that its entries are
|
|
563
|
+
dropped once its queue is full instead; see
|
|
564
|
+
[Drop messages instead of blocking](operations.html#drop-messages-instead-of-blocking).
|
|
565
|
+
|
|
566
|
+
Without `async:` or `batch:`, each failure is still reported to the internal logger, now as
|
|
567
|
+
`Failed to log to appender`, with the error and its backtrace. A subclass of
|
|
568
|
+
`SemanticLogger::Appender::Http` that checks the result of `post` or `put` now gets a
|
|
569
|
+
`SemanticLogger::Appender::Http::ResponseError` raised instead of `false`.
|
|
570
|
+
* A flush or close of an appender added with `async: true` or in batches no longer waits out the
|
|
571
|
+
back-offs of its worker thread during an outage, which could keep the flush at exit waiting for
|
|
572
|
+
minutes: 1 second for the first entry queued, 2 for the next, and so on. It ends the back-off
|
|
573
|
+
under way, and until it has run, an entry or batch that the destination fails is discarded,
|
|
574
|
+
counted as `failed`, rather than sent again after a back-off, and so are the entries queued after
|
|
575
|
+
it, ahead of the flush or close, without being sent (see the next note). A custom appender can now
|
|
576
|
+
say that a write that raised wrote nothing, from `#wrote_nothing?(exception)`, so that the worker
|
|
577
|
+
thread writes it again after the back-off, up to 3 times once the destination answered it, and
|
|
578
|
+
that the write never reached the destination, from `#not_sent?(exception)`, which is not counted;
|
|
579
|
+
see
|
|
580
|
+
[Custom appenders](config.html#custom-appenders). A subclass of `SemanticLogger::Appender::Http`
|
|
581
|
+
whose `#log` or `#batch` sends more than one request inherits a `#wrote_nothing?` that answers
|
|
582
|
+
`true` for a 503 or a refused connection from any of them, which sends the entries of the earlier
|
|
583
|
+
requests a second time: override it to return `false` for a failure after the first request.
|
|
584
|
+
* During an outage of the destination of an appender added with `async: true` or in batches, a flush
|
|
585
|
+
or close, including the flush at exit, now discards that appender's entries queued ahead of it once
|
|
586
|
+
a write fails, counted as `failed` in `SemanticLogger.stats`, rather than send each one in turn.
|
|
587
|
+
Before, it waited while each one failed, up to the `open_timeout` of an HTTP based appender
|
|
588
|
+
whose server does not answer, so that a process with 10,000 entries queued could take hours to
|
|
589
|
+
exit. At exit, each such appender is also told that the process is exiting before the flush, so
|
|
590
|
+
that it stops backing off even while the thread that writes to every appender is held up handing
|
|
591
|
+
it an entry. The entries queued after a flush, and every entry to a destination that takes them,
|
|
592
|
+
are written as before. The entries discarded are reported to the internal logger, as for any entry
|
|
593
|
+
that could not be written.
|
|
594
|
+
* During an outage of the destination of an appender added without `async: true`, and not in
|
|
595
|
+
batches, such as a plain `:http`, `:splunk_http`, `:tcp`, or `:udp`, the process now exits within
|
|
596
|
+
seconds too. Once a write to such an appender fails because of its destination while the process
|
|
597
|
+
exits, it is given no more of the entries queued ahead of the flush at exit, or logged meanwhile,
|
|
598
|
+
and the number it was not given is reported to the internal logger, as
|
|
599
|
+
`Discarded ... log message(s) during the flush at exit, after the destination failed`. The flush
|
|
600
|
+
at exit still flushes it, so that it delivers the entries it already accepted. Before, the
|
|
601
|
+
flush at exit waited while each one failed in turn, up to the `read_timeout` or `open_timeout` of
|
|
602
|
+
an HTTP based appender each. The other appenders still get every entry, and while the process
|
|
603
|
+
runs, such an appender is still given every entry. See
|
|
604
|
+
[Retry a failing appender](operations.html#retry-a-failing-appender).
|
|
605
|
+
* The `:elasticsearch` and `:opensearch` appenders now send again the entries of a bulk request that
|
|
606
|
+
the server rejected with status 429, because its write queue was full, up to 3 times, after waiting
|
|
607
|
+
0.2, 0.4, then 0.8 seconds. Before, they were reported and discarded straight away. That holds up
|
|
608
|
+
the thread that writes the entries for waits of at most 1.4 seconds for a request, plus up to 3
|
|
609
|
+
more bulk requests. It is the appender's own thread when it writes in batches, as it does by
|
|
610
|
+
default, or with `async: true`. With `batch: false`, and without `async: true`, it is the thread
|
|
611
|
+
that writes to every appender, or with `SemanticLogger.sync!`, the thread that logged, and every
|
|
612
|
+
other thread that logs waits for it. The report of the entries still not indexed now comes after
|
|
613
|
+
those waits. See [Elasticsearch](appenders.html#elasticsearch).
|
|
614
|
+
* `SemanticLogger.reopen` now only asks for the appenders to be reopened, each by the thread that
|
|
615
|
+
writes its entries, before it writes the next one. With `SemanticLogger.sync!` on, that is still
|
|
616
|
+
the calling thread for every appender without a thread of its own (added without `async: true` or
|
|
617
|
+
batching). Before, it
|
|
618
|
+
reopened them on the calling thread, often a signal trap after a log rotation, while the thread
|
|
619
|
+
that writes the entries could be writing to them: the file appender could close its file under an
|
|
620
|
+
entry being written, and the HTTP appenders could end a connection under a request, or, with some
|
|
621
|
+
versions of the timeout gem, fail with `ThreadError` in a trap. So `reopen` now returns before the
|
|
622
|
+
appenders have reopened; call `SemanticLogger.flush` after it to wait until they have, for example
|
|
623
|
+
in a test. An entry still queued when `reopen` is called can now be written to the new file. The
|
|
624
|
+
error from an appender that cannot be reopened is reported by that thread, rather than raised from
|
|
625
|
+
the proxy of an appender added with `async: true` or in batches, which now reports it to the
|
|
626
|
+
appender's own internal logger, as `Async: Failed to reopen the appender`.
|
|
627
|
+
* A block given to `add_appender` with an appender or metric subscriber named by a Symbol, such as
|
|
628
|
+
`add_appender(appender: :http, url: ...) { |log| log.message }`, now formats its entries, taking
|
|
629
|
+
precedence over `formatter:`, as it does for `io:`, `file_name:`, and `logger:`. v5.2 ignored it
|
|
630
|
+
and used the appender's `formatter:` or its default formatter, so what such an appender sends
|
|
631
|
+
changes. With `batch: true`, `:http`, `:loki`, and `metric: :signalfx` now raise `ArgumentError`
|
|
632
|
+
for it (see [Batch an appender with a formatter that can format a batch](#batch-an-appender-with-a-formatter-that-can-format-a-batch)),
|
|
633
|
+
where v5.2 sent batches formatted by the default formatter. Remove the block to keep the output
|
|
634
|
+
of v5.2.
|
|
635
|
+
* Some appenders now check their settings as they are added, and raise there, rather than failing
|
|
636
|
+
each entry later:
|
|
637
|
+
* `add_appender(metric: :statsd, url: ...)` connects as it is added, so it raises when its `url:`
|
|
638
|
+
is not `udp://` or its host cannot be resolved, where v5.2 added it and then failed every
|
|
639
|
+
metric. Add it only where the Statsd host resolves.
|
|
640
|
+
* `add_appender(appender: :udp, server: ...)` raises `ArgumentError` for a `server:` that is not
|
|
641
|
+
`"host:port"`, such as one without a port. Write an IPv6 address in brackets, as
|
|
642
|
+
`"[::1]:8088"`.
|
|
643
|
+
* `SemanticLogger::Processor.logger=` (and `SemanticLogger::SyncProcessor.logger=`) now raises
|
|
644
|
+
`ArgumentError` when given a `SemanticLogger::Logger`, whose reports of a failing appender went
|
|
645
|
+
back to that appender, failed again, and were reported again. Give it an appender, such as
|
|
646
|
+
`SemanticLogger::Appender::IO.new($stderr)`, or a Ruby or Rails `Logger`, which it now accepts.
|
|
647
|
+
* `SemanticLogger.remove_appender` now waits, as `SemanticLogger.flush` does, until the messages
|
|
648
|
+
already logged are written and the appenders flushed, before it closes the appender, so it can
|
|
649
|
+
take as long as a flush. Called on the thread of an appender added with `async: true` or in
|
|
650
|
+
batches, it returns `true` at once, and the appender is closed shortly after. See
|
|
651
|
+
[Adding and removing appenders](config.html#adding-and-removing-appenders).
|
|
652
|
+
* The `:open_telemetry` and `:new_relic_logs` appenders now keep the trace context they capture
|
|
653
|
+
with `Log#capture`, which formatters do not write, instead of in `log.context`. A custom formatter
|
|
654
|
+
or appender that read `log.context[:open_telemetry]` or `log.context[:new_relic_metadata]` reads
|
|
655
|
+
`log.captured(:open_telemetry)` or `log.captured(:new_relic_metadata)` instead, and an `on_log`
|
|
656
|
+
subscriber that set either key with `log.set_context` sets it with `log.capture`.
|
|
657
|
+
|
|
15
658
|
### Upgrading to Semantic Logger v5.2
|
|
16
659
|
|
|
17
660
|
v5.2 sends the configured facility to remote syslog, deprecates the `:splunk` appender, stops
|