lumberjack 1.2.10 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/ARCHITECTURE.md +244 -0
- data/CHANGELOG.md +84 -0
- data/README.md +168 -63
- data/VERSION +1 -1
- data/lib/lumberjack/context.rb +17 -10
- data/lib/lumberjack/device/rolling_log_file.rb +1 -1
- data/lib/lumberjack/device/writer.rb +12 -8
- data/lib/lumberjack/formatter/date_time_formatter.rb +1 -1
- data/lib/lumberjack/formatter/multiply_formatter.rb +25 -0
- data/lib/lumberjack/formatter/redact_formatter.rb +23 -0
- data/lib/lumberjack/formatter/round_formatter.rb +21 -0
- data/lib/lumberjack/formatter/tagged_message.rb +39 -0
- data/lib/lumberjack/formatter.rb +28 -13
- data/lib/lumberjack/log_entry.rb +76 -17
- data/lib/lumberjack/logger.rb +144 -52
- data/lib/lumberjack/rack/context.rb +20 -1
- data/lib/lumberjack/rack/request_id.rb +6 -2
- data/lib/lumberjack/rack/unit_of_work.rb +5 -1
- data/lib/lumberjack/tag_context.rb +78 -0
- data/lib/lumberjack/tag_formatter.rb +102 -27
- data/lib/lumberjack/tagged_logger_support.rb +25 -10
- data/lib/lumberjack/tags.rb +1 -7
- data/lib/lumberjack/template.rb +1 -1
- data/lib/lumberjack/utils.rb +182 -0
- data/lib/lumberjack.rb +12 -5
- data/lumberjack.gemspec +8 -2
- metadata +17 -7
data/README.md
CHANGED
|
@@ -1,15 +1,14 @@
|
|
|
1
1
|
# Lumberjack
|
|
2
2
|
|
|
3
3
|
[](https://github.com/bdurand/lumberjack/actions/workflows/continuous_integration.yml)
|
|
4
|
-
[](https://github.com/bdurand/lumberjack/actions/workflows/regression_test.yml)
|
|
5
4
|
[](https://github.com/testdouble/standard)
|
|
6
5
|
[](https://badge.fury.io/rb/lumberjack)
|
|
7
6
|
|
|
8
|
-
Lumberjack is a simple, powerful, and fast logging implementation in Ruby. It uses nearly the same API as the Logger class in the Ruby standard library and as ActiveSupport::BufferedLogger in Rails.
|
|
7
|
+
Lumberjack is a simple, powerful, and fast logging implementation in Ruby. It uses nearly the same API as the Logger class in the Ruby standard library and as ActiveSupport::BufferedLogger in Rails. It is designed with structured logging in mind, but can be used for simple text logging as well.
|
|
9
8
|
|
|
10
9
|
## Usage
|
|
11
10
|
|
|
12
|
-
This code aims to be extremely simple to use. The core interface is the Lumberjack::Logger which is used to log messages (which can be any object) with a specified Severity. Each logger has a level associated with it and messages are only written if their severity is greater than or equal to the level.
|
|
11
|
+
This code aims to be extremely simple to use and matches the standard Ruby `Logger` interface. The core interface is the Lumberjack::Logger which is used to log messages (which can be any object) with a specified Severity. Each logger has a level associated with it and messages are only written if their severity is greater than or equal to the level.
|
|
13
12
|
|
|
14
13
|
```ruby
|
|
15
14
|
logger = Lumberjack::Logger.new("logs/application.log") # Open a new log file with INFO level
|
|
@@ -24,27 +23,25 @@ This code aims to be extremely simple to use. The core interface is the Lumberja
|
|
|
24
23
|
logger.info("End request")
|
|
25
24
|
```
|
|
26
25
|
|
|
27
|
-
This is all you need to know to log messages.
|
|
28
|
-
|
|
29
26
|
## Features
|
|
30
27
|
|
|
31
|
-
###
|
|
28
|
+
### Metadata
|
|
32
29
|
|
|
33
30
|
When messages are added to the log, additional data about the message is kept in a Lumberjack::LogEntry. This means you don't need to worry about adding the time or process id to your log messages as they will be automatically recorded.
|
|
34
31
|
|
|
35
32
|
The following information is recorded for each message:
|
|
36
33
|
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
34
|
+
- severity - The severity recorded for the message.
|
|
35
|
+
- time - The time at which the message was recorded.
|
|
36
|
+
- program name - The name of the program logging the message. This can be either set for all messages or customized with each message.
|
|
37
|
+
- process id - The process id (pid) of the process that logged the message.
|
|
38
|
+
- tags - A map of name value pairs for additional information about the log context.
|
|
42
39
|
|
|
43
40
|
### Tags
|
|
44
41
|
|
|
45
|
-
You can use tags to provide additional meta data about a log message or the context that the log message is being made in. Using tags can keep your log messages clean. You can avoid string interpolation to add additional meta data.
|
|
42
|
+
You can use tags to provide additional meta data about a log message or the context that the log message is being made in. Using tags can keep your log messages clean. You can avoid string interpolation to add additional meta data. Tags enable a structured logging approach where you can add additional information to log messages without changing the message format.
|
|
46
43
|
|
|
47
|
-
Each of the logger methods includes an additional argument that can be used to specify tags on a
|
|
44
|
+
Each of the logger methods includes an additional argument that can be used to specify tags on a message:
|
|
48
45
|
|
|
49
46
|
```ruby
|
|
50
47
|
logger.info("request completed", duration: elapsed_time, status: response.status)
|
|
@@ -53,7 +50,7 @@ logger.info("request completed", duration: elapsed_time, status: response.status
|
|
|
53
50
|
You can also specify tags on a logger that will be included with every log message.
|
|
54
51
|
|
|
55
52
|
```ruby
|
|
56
|
-
logger.
|
|
53
|
+
logger.tag_globally(host: Socket.gethostname.force_encoding("UTF-8"))
|
|
57
54
|
```
|
|
58
55
|
|
|
59
56
|
You can specify tags that will only be applied to the logger in a block as well.
|
|
@@ -67,31 +64,92 @@ end
|
|
|
67
64
|
logger.info("there") # Will not include the `thread_id` or `count` tag
|
|
68
65
|
```
|
|
69
66
|
|
|
67
|
+
The block opens up a new tag context. The context applies only within a block and only for the currently executing thread. You can also open up a new context block without specifying any tags and then add tags to the context within the block.
|
|
68
|
+
|
|
69
|
+
```ruby
|
|
70
|
+
logger.context do # When a block is given to `context`, it opens a new empty tag context.
|
|
71
|
+
# `context` can be called without a block, to get an TagContext object for manipulating tags.
|
|
72
|
+
logger.context.tag(user_id: current_user.id, username: current_user.username)
|
|
73
|
+
logger.context["user_role"] = current_user.role # You can also use hash syntax to set tags.
|
|
74
|
+
logger.info("user logged in") # Will include the `user_id`, `username`, and `user_role` tags.
|
|
75
|
+
end
|
|
76
|
+
logger.info("no user") # Will not include the tags
|
|
77
|
+
```
|
|
78
|
+
|
|
70
79
|
You can also set tags to `Proc` objects that will be evaluated when creating a log entry.
|
|
71
80
|
|
|
72
81
|
```ruby
|
|
73
|
-
logger.
|
|
82
|
+
logger.tag_globally(thread_id: lambda { Thread.current.object_id })
|
|
83
|
+
|
|
74
84
|
Thread.new do
|
|
75
85
|
logger.info("inside thread") # Will include the `thread_id` tag with id of the spawned thread
|
|
76
86
|
end
|
|
87
|
+
|
|
77
88
|
logger.info("outside thread") # Will include the `thread_id` tag with id of the main thread
|
|
78
89
|
```
|
|
79
90
|
|
|
80
|
-
Finally, you can specify a logging context
|
|
91
|
+
Finally, you can specify a global logging context that applies to all loggers.
|
|
81
92
|
|
|
82
93
|
```ruby
|
|
83
94
|
Lumberjack.context do
|
|
84
|
-
Lumberjack.tag(request_id: SecureRandom.hex)
|
|
95
|
+
Lumberjack.tag(request_id: SecureRandom.hex) # add a global context tag
|
|
85
96
|
logger.info("begin request") # Will include the `request_id` tag
|
|
97
|
+
event_logger.info("http.request") # Will also include the `request_id` tag
|
|
86
98
|
end
|
|
87
99
|
logger.info("no requests") # Will not include the `request_id` tag
|
|
88
100
|
```
|
|
89
101
|
|
|
90
102
|
Tag keys are always converted to strings. Tags are inherited so that message tags take precedence over block tags which take precedence over global tags.
|
|
91
103
|
|
|
104
|
+
#### Structured Logging with Tags
|
|
105
|
+
|
|
106
|
+
Tags are particularly powerful for structured logging, where you want to capture machine-readable data alongside human-readable log messages. Instead of embedding variable data directly in log messages (which makes parsing difficult), you can use tags to separate the static message from the dynamic data.
|
|
107
|
+
|
|
108
|
+
```ruby
|
|
109
|
+
# Instead of this (harder to parse)
|
|
110
|
+
logger.info("User john_doe logged in from IP 192.168.1.100 in 0.25 seconds")
|
|
111
|
+
|
|
112
|
+
# Do this (structured and parseable)
|
|
113
|
+
logger.info("User logged in", {
|
|
114
|
+
user_id: "john_doe",
|
|
115
|
+
ip_address: "192.168.1.100",
|
|
116
|
+
duration: 0.25,
|
|
117
|
+
action: "login"
|
|
118
|
+
})
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
This approach provides several benefits:
|
|
122
|
+
|
|
123
|
+
- **Consistent message format** - The base message stays the same while data varies
|
|
124
|
+
- **Easy filtering and searching** - You can search by specific tag values
|
|
125
|
+
- **Better analytics** - Aggregate data by tag values (e.g., average login duration)
|
|
126
|
+
- **Machine processing** - Automated systems can easily extract and process tag data
|
|
127
|
+
|
|
128
|
+
You can also use nested structures in tags for complex data:
|
|
129
|
+
|
|
130
|
+
```ruby
|
|
131
|
+
logger.info("API request completed", {
|
|
132
|
+
request: {
|
|
133
|
+
method: "POST",
|
|
134
|
+
path: "/api/users",
|
|
135
|
+
user_agent: request.user_agent
|
|
136
|
+
},
|
|
137
|
+
response: {
|
|
138
|
+
status: 201,
|
|
139
|
+
duration_ms: 150
|
|
140
|
+
},
|
|
141
|
+
user: {
|
|
142
|
+
id: current_user.id,
|
|
143
|
+
role: current_user.role
|
|
144
|
+
}
|
|
145
|
+
})
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
When combined with structured output devices (like [`lumberjack_json_device`](https://github.com/bdurand/lumberjack_json_device)), this creates logs that are both human-readable and machine-processable, making them ideal for log aggregation systems, monitoring, and analytics.
|
|
149
|
+
|
|
92
150
|
#### Compatibility with ActiveSupport::TaggedLogging
|
|
93
151
|
|
|
94
|
-
`Lumberjack::Logger`
|
|
152
|
+
`Lumberjack::Logger` is compatible with `ActiveSupport::TaggedLogging`. This is so that other code that expects to have a logger that responds to the `tagged` method will work. Any tags added with the `tagged` method will be appended to an array in the "tagged" tag.
|
|
95
153
|
|
|
96
154
|
```ruby
|
|
97
155
|
logger.tagged("foo", "bar=1", "other") do
|
|
@@ -103,38 +161,37 @@ end
|
|
|
103
161
|
|
|
104
162
|
The built in `Lumberjack::Device::Writer` class has built in support for including tags in the output using the `Lumberjack::Template` class.
|
|
105
163
|
|
|
106
|
-
You can specify any tag name you want in a template as well as the `:tags` macro for all tags. If a tag name has been used as
|
|
107
|
-
|
|
108
|
-
#### Unit Of Work
|
|
109
|
-
|
|
110
|
-
Lumberjack 1.0 had a concept of a unit of work id that could be used to tie log messages together. This has been replaced by tags. There is still an implementation of `Lumberjack.unit_of_work`, but it is just a wrapper on the tag implementation.
|
|
164
|
+
You can specify any tag name you want in a template as well as the `:tags` macro for all tags. If a tag name has been used as its own macro, it will not be included in the `:tags` macro.
|
|
111
165
|
|
|
112
166
|
### Pluggable Devices
|
|
113
167
|
|
|
114
168
|
When a Logger logs a LogEntry, it sends it to a Lumberjack::Device. Lumberjack comes with a variety of devices for logging to IO streams or files.
|
|
115
169
|
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
170
|
+
- Lumberjack::Device::Writer - Writes log entries to an IO stream.
|
|
171
|
+
- Lumberjack::Device::LogFile - Writes log entries to a file.
|
|
172
|
+
- Lumberjack::Device::DateRollingLogFile - Writes log entries to a file that will automatically roll itself based on date.
|
|
173
|
+
- Lumberjack::Device::SizeRollingLogFile - Writes log entries to a file that will automatically roll itself based on size.
|
|
174
|
+
- Lumberjack::Device::Multi - This device wraps multiple other devices and will write log entries to each of them.
|
|
175
|
+
- Lumberjack::Device::Null - This device produces no output and is intended for testing environments.
|
|
122
176
|
|
|
123
|
-
If you'd like to send
|
|
177
|
+
If you'd like to send your log to a different kind of output, you just need to extend the Device class and implement the `write` method. Or check out these plugins:
|
|
124
178
|
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
179
|
+
- [lumberjack_json_device](https://github.com/bdurand/lumberjack_json_device) - output your log messages as stream of JSON objects for structured logging.
|
|
180
|
+
- [lumberjack_syslog_device](https://github.com/bdurand/lumberjack_syslog_device) - send your log messages to the system wide syslog service
|
|
181
|
+
- [lumberjack_mongo_device](https://github.com/bdurand/lumberjack_mongo_device) - store your log messages to a [MongoDB](http://www.mongodb.org/) NoSQL data store
|
|
182
|
+
- [lumberjack_redis_device](https://github.com/bdurand/lumberjack_redis_device) - store your log messages in a [Redis](https://redis.io/) data store
|
|
183
|
+
- [lumberjack-couchdb-driver](https://github.com/narkisr/lumberjack-couchdb-driver) - store your log messages to a [CouchDB](http://couchdb.apache.org/) NoSQL data store
|
|
184
|
+
- [lumberjack_heroku_device](https://github.com/tonycoco/lumberjack_heroku_device) - log to Heroku's logging system
|
|
185
|
+
- [lumberjack_capture_device](https://github.com/bdurand/lumberjack_capture_device) - capture log messages in memory in test environments so that you can include log output assertions in your tests.
|
|
129
186
|
|
|
130
187
|
### Customize Formatting
|
|
131
188
|
|
|
132
189
|
#### Formatters
|
|
133
190
|
|
|
134
|
-
The message you send to the logger can be any object type and does not need to be a string. You can specify
|
|
191
|
+
The message you send to the logger can be any object type and does not need to be a string. You can specify how to format different object types with a formatter. The formatter is responsible for converting the object into a string representation for logging. You do this by mapping classes or modules to formatter code. This code can be either a block or an object that responds to the `call` method. The formatter will be called with the object logged as the message and the returned value will be what is sent to the device.
|
|
135
192
|
|
|
136
193
|
```ruby
|
|
137
|
-
# Format all floating point
|
|
194
|
+
# Format all floating point numbers with three significant digits.
|
|
138
195
|
logger.formatter.add(Float) { |value| value.round(3) }
|
|
139
196
|
|
|
140
197
|
# Format all enumerable objects as a comma delimited string.
|
|
@@ -148,25 +205,54 @@ There are several built in classes you can add as formatters. You can use a symb
|
|
|
148
205
|
logger.formatter.add(Hash, Lumberjack::Formatter::PrettyPrintFormatter.new) # alternative using a formatter instance
|
|
149
206
|
```
|
|
150
207
|
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
208
|
+
- `:object` - `Lumberjack::Formatter::ObjectFormatter` - no op conversion that returns the object itself.
|
|
209
|
+
- `:string` - `Lumberjack::Formatter::StringFormatter` - calls `to_s` on the object.
|
|
210
|
+
- `:strip` - `Lumberjack::Formatter::StripFormatter` - calls `to_s.strip` on the object.
|
|
211
|
+
- `:inspect` - `Lumberjack::Formatter::InspectFormatter` - calls `inspect` on the object.
|
|
212
|
+
- `:exception` - `Lumberjack::Formatter::ExceptionFormatter` - special formatter for exceptions which logs them as multi-line statements with the message and backtrace.
|
|
213
|
+
- `:date_time` - `Lumberjack::Formatter::DateTimeFormatter` - special formatter for dates and times to format them using `strftime`.
|
|
214
|
+
- `:pretty_print` - `Lumberjack::Formatter::PrettyPrintFormatter` - returns the pretty print format of the object.
|
|
215
|
+
- `:id` - `Lumberjack::Formatter::IdFormatter` - returns a hash of the object with keys for the id attribute and class.
|
|
216
|
+
- `:structured` - `Lumberjack::Formatter::StructuredFormatter` - crawls the object and applies the formatter recursively to Enumerable objects found in it (arrays, hashes, etc.).
|
|
217
|
+
- `:truncate` - `Lumberjack::Formatter::TruncateFormatter` - truncates long strings to a specified length.
|
|
160
218
|
|
|
161
219
|
To define your own formatter, either provide a block or an object that responds to `call` with a single argument.
|
|
162
220
|
|
|
221
|
+
#### Default Formatter
|
|
222
|
+
|
|
223
|
+
The default formatter is applied to all objects being logged. This includes both messages and tags.
|
|
224
|
+
|
|
163
225
|
The default formatter will pass through values for strings, numbers, and booleans, and use the `:inspect` formatter for all objects except for exceptions which will be formatted with the `:exception` formatter.
|
|
164
226
|
|
|
227
|
+
#### Message Formatter
|
|
228
|
+
|
|
229
|
+
You can add a formatter for just the log message with the `message_formatter` method. This formatter will only apply to the message and not to any tags.
|
|
230
|
+
|
|
231
|
+
```ruby
|
|
232
|
+
logger.message_formatter.add(String, :truncate, 1000) # Will truncate all string messages to 1000 characters
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
##### Extracting Tags from Messages
|
|
236
|
+
|
|
237
|
+
If you are using structured logging, you can use a formatter to extract tags from the log message by adding a formatter that returns a `Lumberjack::Formatter::TaggedMessage`. For example, if you want to extract metadata from exceptions and add them as tags, you could do this:
|
|
238
|
+
|
|
239
|
+
```ruby
|
|
240
|
+
logger.message_formatter.add(Exception, ->(e) {
|
|
241
|
+
Lumberjack::Formatter::TaggedMessage.new(e.inspect, {
|
|
242
|
+
"error.message": e.message,
|
|
243
|
+
"error.class": e.class.name,
|
|
244
|
+
"error.trace": e.backtrace
|
|
245
|
+
})
|
|
246
|
+
})
|
|
247
|
+
|
|
248
|
+
logger.error(exception) # Will log the exception and add tags for the message, class, and trace.
|
|
249
|
+
```
|
|
250
|
+
|
|
165
251
|
#### Tag Formatters
|
|
166
252
|
|
|
167
|
-
The `logger.formatter` will only apply to log messages. You can use `logger.tag_formatter` to register formatters for tags. You can register both default formatters that will apply to all tag values, as well as tag
|
|
253
|
+
The `logger.formatter` will only apply to log messages. You can use `logger.tag_formatter` to register formatters for tags. You can register both default formatters that will apply to all tag values, as well as tag specific formatters that will apply only to objects with a specific tag name.
|
|
168
254
|
|
|
169
|
-
The
|
|
255
|
+
The formatter values can be either a `Lumberjack::Formatter` or a block or an object that responds to `call`. If you supply a `Lumberjack::Formatter`, the tag value will be passed through the rules for that formatter. If you supply a block or other object, it will be called with the tag value.
|
|
170
256
|
|
|
171
257
|
```ruby
|
|
172
258
|
# These will all do the same thing formatting all tag values with `inspect`
|
|
@@ -177,11 +263,20 @@ logger.tag_formatter.default { |value| value.inspect }
|
|
|
177
263
|
# This will register formatters only on specific tag names
|
|
178
264
|
logger.tag_formatter.add(:thread) { |thread| "Thread(#{thread.name})" }
|
|
179
265
|
logger.tag_formatter.add(:current_user, Lumberjack::Formatter::IdFormatter.new)
|
|
266
|
+
|
|
267
|
+
# You can also register formatters for tag values by class
|
|
268
|
+
logger.tag_formatter.add(Numeric, &:round)
|
|
269
|
+
|
|
270
|
+
# Tag formatters will be applied to nested hashes and arrays as well.
|
|
271
|
+
|
|
272
|
+
# Name formatters use dot syntax to apply to nested hashes.
|
|
273
|
+
logger.tag_formatter.add("user.username", &:upcase)
|
|
274
|
+
# logger.tag(user: {username: "john_doe"}) # Will log the tag as {"user" => "username" => "JOHN_DOE"}
|
|
180
275
|
```
|
|
181
276
|
|
|
182
277
|
#### Templates
|
|
183
278
|
|
|
184
|
-
If you use the built
|
|
279
|
+
If you use the built-in `Lumberjack::Device::Writer` derived devices, you can also customize the Template used to format the LogEntry.
|
|
185
280
|
|
|
186
281
|
See `Lumberjack::Template` for a complete list of macros you can use in the template. You can also use a block that receives a `Lumberjack::LogEntry` as a template.
|
|
187
282
|
|
|
@@ -203,11 +298,11 @@ See `Lumberjack::Template` for a complete list of macros you can use in the temp
|
|
|
203
298
|
|
|
204
299
|
### Buffered Logging
|
|
205
300
|
|
|
206
|
-
The logger has hooks for devices that support buffering to potentially increase performance by batching physical writes. Log entries are not guaranteed to be written until the Lumberjack::Logger#flush method is called. Buffering can improve performance if I/O is slow or there high overhead writing to the log device.
|
|
301
|
+
The logger has hooks for devices that support buffering to potentially increase performance by batching physical writes. Log entries are not guaranteed to be written until the Lumberjack::Logger#flush method is called. Buffering can improve performance if I/O is slow or there is high overhead writing to the log device.
|
|
207
302
|
|
|
208
303
|
You can use the `:flush_seconds` option on the logger to periodically flush the log. This is usually a good idea so you can more easily debug hung processes. Without periodic flushing, a process that hangs may never write anything to the log because the messages are sitting in a buffer. By turning on periodic flushing, the logged messages will be written which can greatly aid in debugging the problem.
|
|
209
304
|
|
|
210
|
-
The built in stream based logging devices use an internal buffer. The size of the buffer (in bytes) can be set with the `:buffer_size` options when initializing a logger. The default behavior is to not
|
|
305
|
+
The built in stream based logging devices use an internal buffer. The size of the buffer (in bytes) can be set with the `:buffer_size` options when initializing a logger. The default behavior is to not buffer.
|
|
211
306
|
|
|
212
307
|
```ruby
|
|
213
308
|
# Set buffer to flush after 8K has been written to the log.
|
|
@@ -223,28 +318,38 @@ The built in devices include two that can automatically roll log files based eit
|
|
|
223
318
|
|
|
224
319
|
There is a similar feature in the standard library Logger class, but the implementation here is safe to use with multiple processes writing to the same log file.
|
|
225
320
|
|
|
226
|
-
##
|
|
321
|
+
## Integrations
|
|
322
|
+
|
|
323
|
+
Lumberjack has built in support for logging extensions in Rails.
|
|
324
|
+
|
|
325
|
+
You can use the [`lumberjack_sidekiq`](https://github.com/bdurand/lumberjack_sidekiq) gem to replace Sidekiq's default logger with Lumberjack. This allows you to use all of Lumberjack's features, such as structured logging and tag support, in your Sidekiq jobs.
|
|
326
|
+
|
|
327
|
+
If you are using DataDog for logging, you can use the [`lumberjack_data_dog`](https://github.com/bdurand/lumberjack_data_dog) gem to format your logs in DataDog's standard attributes format.
|
|
328
|
+
|
|
329
|
+
## Differences from Standard Library Logger
|
|
227
330
|
|
|
228
|
-
`Lumberjack::Logger` does not extend from the `Logger` class in the standard library, but it does implement a
|
|
331
|
+
`Lumberjack::Logger` does not extend from the `Logger` class in the standard library, but it does implement a compatible API. The main difference is in the flow of how messages are ultimately sent to devices for output.
|
|
229
332
|
|
|
230
333
|
The standard library Logger logic converts the log entries to strings and then sends the string to the device to be written to a stream. Lumberjack, on the other hand, sends structured data in the form of a `Lumberjack::LogEntry` to the device and lets the device worry about how to format it. The reason for this flip is to better support structured data logging. Devices (even ones that write to streams) can format the entire payload including non-string objects and tags however they need to.
|
|
231
334
|
|
|
232
|
-
The logging methods (`debug`,
|
|
335
|
+
The logging methods (`debug`, `info`, `warn`, `error`, `fatal`) are overloaded with an additional argument for setting tags on the log entry.
|
|
233
336
|
|
|
234
337
|
## Examples
|
|
235
338
|
|
|
236
|
-
These
|
|
339
|
+
These examples are for Rails applications, but there is no dependency on Rails for using this gem. Most of the examples are applicable to any Ruby application.
|
|
237
340
|
|
|
238
341
|
In a Rails application you can replace the default production logger by adding this to your config/environments/production.rb file:
|
|
239
342
|
|
|
240
343
|
```ruby
|
|
241
|
-
#
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
344
|
+
# Add the ActionDispatch request id as a global tag on all log entries.
|
|
345
|
+
config.middleware.insert_after(
|
|
346
|
+
ActionDispatch::RequestId,
|
|
347
|
+
Lumberjack::Rack::Context,
|
|
348
|
+
request_id: ->(env) { env["action_dispatch.request_id"] }
|
|
349
|
+
)
|
|
246
350
|
# Change the logger to use Lumberjack
|
|
247
|
-
|
|
351
|
+
log_file = Rails.root + "log" + "#{Rails.env}.log"
|
|
352
|
+
# log_file = $stdout # or write to stdout instead of a file
|
|
248
353
|
config.logger = Lumberjack::Logger.new(log_file, :level => :warn)
|
|
249
354
|
```
|
|
250
355
|
|
|
@@ -272,7 +377,7 @@ To change the log message format to output JSON, you could use this code:
|
|
|
272
377
|
config.logger = Lumberjack::Logger.new(log_file_path, :template => lambda{|e| JSON.dump(time: e.time, level: e.severity_label, message: e.message)})
|
|
273
378
|
```
|
|
274
379
|
|
|
275
|
-
To send log messages to syslog instead of to a file, you could use this (
|
|
380
|
+
To send log messages to syslog instead of to a file, you could use this (requires the lumberjack_syslog_device gem):
|
|
276
381
|
|
|
277
382
|
```ruby
|
|
278
383
|
config.logger = Lumberjack::Logger.new(Lumberjack::SyslogDevice.new)
|
|
@@ -283,12 +388,12 @@ To send log messages to syslog instead of to a file, you could use this (require
|
|
|
283
388
|
Add this line to your application's Gemfile:
|
|
284
389
|
|
|
285
390
|
```ruby
|
|
286
|
-
gem
|
|
391
|
+
gem "lumberjack"
|
|
287
392
|
```
|
|
288
393
|
|
|
289
394
|
And then execute:
|
|
290
395
|
```bash
|
|
291
|
-
$ bundle
|
|
396
|
+
$ bundle install
|
|
292
397
|
```
|
|
293
398
|
|
|
294
399
|
Or install it yourself as:
|
data/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
1.
|
|
1
|
+
1.4.0
|
data/lib/lumberjack/context.rb
CHANGED
|
@@ -5,37 +5,44 @@ module Lumberjack
|
|
|
5
5
|
class Context
|
|
6
6
|
attr_reader :tags
|
|
7
7
|
|
|
8
|
-
# @param [Context]
|
|
8
|
+
# @param parent_context [Context] A parent context to inherit tags from.
|
|
9
9
|
def initialize(parent_context = nil)
|
|
10
10
|
@tags = {}
|
|
11
11
|
@tags.merge!(parent_context.tags) if parent_context
|
|
12
|
+
@tag_context = TagContext.new(@tags)
|
|
12
13
|
end
|
|
13
14
|
|
|
14
15
|
# Set tags on the context.
|
|
15
16
|
#
|
|
16
|
-
# @param [Hash]
|
|
17
|
+
# @param tags [Hash] The tags to set.
|
|
17
18
|
# @return [void]
|
|
18
19
|
def tag(tags)
|
|
19
|
-
tags
|
|
20
|
-
@tags[key.to_s] = value
|
|
21
|
-
end
|
|
20
|
+
@tag_context.tag(tags)
|
|
22
21
|
end
|
|
23
22
|
|
|
24
23
|
# Get a context tag.
|
|
25
24
|
#
|
|
26
|
-
# @param [String, Symbol]
|
|
25
|
+
# @param key [String, Symbol] The tag key.
|
|
27
26
|
# @return [Object] The tag value.
|
|
28
27
|
def [](key)
|
|
29
|
-
@
|
|
28
|
+
@tag_context[key]
|
|
30
29
|
end
|
|
31
30
|
|
|
32
31
|
# Set a context tag.
|
|
33
32
|
#
|
|
34
|
-
# @param [String, Symbol]
|
|
35
|
-
# @param [Object]
|
|
33
|
+
# @param key [String, Symbol] The tag key.
|
|
34
|
+
# @param value [Object] The tag value.
|
|
36
35
|
# @return [void]
|
|
37
36
|
def []=(key, value)
|
|
38
|
-
@
|
|
37
|
+
@tag_context[key] = value
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
# Remove tags from the context.
|
|
41
|
+
#
|
|
42
|
+
# @param keys [Array<String, Symbol>] The tag keys to remove.
|
|
43
|
+
# @return [void]
|
|
44
|
+
def delete(*keys)
|
|
45
|
+
@tag_context.delete(*keys)
|
|
39
46
|
end
|
|
40
47
|
|
|
41
48
|
# Clear all the context data.
|
|
@@ -55,9 +55,8 @@ module Lumberjack
|
|
|
55
55
|
# :additional_lines and :time_format options will be passed through to the
|
|
56
56
|
# Template constuctor.
|
|
57
57
|
#
|
|
58
|
-
# The default template is "[:time :severity :progname(:pid)
|
|
59
|
-
# with additional lines formatted as "\n
|
|
60
|
-
# work id will only appear if it is present.
|
|
58
|
+
# The default template is "[:time :severity :progname(:pid)] :message"
|
|
59
|
+
# with additional lines formatted as "\n :message".
|
|
61
60
|
#
|
|
62
61
|
# The size of the internal buffer in bytes can be set by providing :buffer_size (defaults to 32K).
|
|
63
62
|
#
|
|
@@ -68,12 +67,12 @@ module Lumberjack
|
|
|
68
67
|
@stream = stream
|
|
69
68
|
@stream.sync = true if @stream.respond_to?(:sync=)
|
|
70
69
|
@buffer = Buffer.new
|
|
71
|
-
@buffer_size =
|
|
72
|
-
template =
|
|
70
|
+
@buffer_size = options[:buffer_size] || 0
|
|
71
|
+
template = options[:template] || DEFAULT_FIRST_LINE_TEMPLATE
|
|
73
72
|
if template.respond_to?(:call)
|
|
74
73
|
@template = template
|
|
75
74
|
else
|
|
76
|
-
additional_lines =
|
|
75
|
+
additional_lines = options[:additional_lines] || DEFAULT_ADDITIONAL_LINES_TEMPLATE
|
|
77
76
|
@template = Template.new(template, additional_lines: additional_lines, time_format: options[:time_format])
|
|
78
77
|
end
|
|
79
78
|
end
|
|
@@ -150,6 +149,13 @@ module Lumberjack
|
|
|
150
149
|
end
|
|
151
150
|
end
|
|
152
151
|
|
|
152
|
+
# Return the underlying stream. Provided for API compatibility with Logger devices.
|
|
153
|
+
#
|
|
154
|
+
# @return [IO] The underlying stream.
|
|
155
|
+
def dev
|
|
156
|
+
@stream
|
|
157
|
+
end
|
|
158
|
+
|
|
153
159
|
protected
|
|
154
160
|
|
|
155
161
|
# Set the underlying stream.
|
|
@@ -163,8 +169,6 @@ module Lumberjack
|
|
|
163
169
|
def write_to_stream(lines)
|
|
164
170
|
return if lines.empty?
|
|
165
171
|
lines = lines.first if lines.is_a?(Array) && lines.size == 1
|
|
166
|
-
|
|
167
|
-
out = nil
|
|
168
172
|
out = if lines.is_a?(Array)
|
|
169
173
|
"#{lines.join(Lumberjack::LINE_SEPARATOR)}#{Lumberjack::LINE_SEPARATOR}"
|
|
170
174
|
else
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Lumberjack
|
|
4
|
+
class Formatter
|
|
5
|
+
# This formatter can be used to multiply a numeric value by a specified multiplier and
|
|
6
|
+
# optionally round to a specified number of decimal places.
|
|
7
|
+
class MultiplyFormatter
|
|
8
|
+
# @param multiplier [Numeric] The multiplier to apply to the value.
|
|
9
|
+
# @param decimals [Integer, nil] The number of decimal places to round the result to.
|
|
10
|
+
# If nil, no rounding is applied.
|
|
11
|
+
def initialize(multiplier, decimals = nil)
|
|
12
|
+
@multiplier = multiplier
|
|
13
|
+
@decimals = decimals
|
|
14
|
+
end
|
|
15
|
+
|
|
16
|
+
def call(value)
|
|
17
|
+
return value unless value.is_a?(Numeric)
|
|
18
|
+
|
|
19
|
+
value *= @multiplier
|
|
20
|
+
value = value.round(@decimals) if @decimals
|
|
21
|
+
value
|
|
22
|
+
end
|
|
23
|
+
end
|
|
24
|
+
end
|
|
25
|
+
end
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Lumberjack
|
|
4
|
+
class Formatter
|
|
5
|
+
# Log sensitive information in a redacted format showing the firat and last
|
|
6
|
+
# characters of the value, with the rest replaced by asterisks. The number of
|
|
7
|
+
# characters shown is dependent onthe length of the value; short values will
|
|
8
|
+
# not show any characters in order to avoid revealing too much information.
|
|
9
|
+
class RedactFormatter
|
|
10
|
+
def call(obj)
|
|
11
|
+
return obj unless obj.is_a?(String)
|
|
12
|
+
|
|
13
|
+
if obj.length > 8
|
|
14
|
+
"#{obj[0..1]}#{"*" * (obj.length - 4)}#{obj[-2..-1]}"
|
|
15
|
+
elsif obj.length > 5
|
|
16
|
+
"#{obj[0]}#{"*" * (obj.length - 2)}#{obj[-1]}"
|
|
17
|
+
else
|
|
18
|
+
"*****"
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
end
|
|
22
|
+
end
|
|
23
|
+
end
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Lumberjack
|
|
4
|
+
class Formatter
|
|
5
|
+
# Round numeric values to a set number of decimal places. This is useful when logging
|
|
6
|
+
# floating point numbers to reduce noise and rounding errors in the logs.
|
|
7
|
+
class RoundFormatter
|
|
8
|
+
def initialize(precision = 3)
|
|
9
|
+
@precision = precision
|
|
10
|
+
end
|
|
11
|
+
|
|
12
|
+
def call(obj)
|
|
13
|
+
if obj.is_a?(Numeric)
|
|
14
|
+
obj.round(@precision)
|
|
15
|
+
else
|
|
16
|
+
obj
|
|
17
|
+
end
|
|
18
|
+
end
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
end
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Lumberjack
|
|
4
|
+
class Formatter
|
|
5
|
+
# This class can be used as the return value from a formatter `call` method to
|
|
6
|
+
# extract additional tags from an object being logged. This can be useful when there
|
|
7
|
+
# using structured logging to include important metadata in the log message.
|
|
8
|
+
#
|
|
9
|
+
# @example
|
|
10
|
+
# # Automatically add tags with error details when logging an exception.
|
|
11
|
+
# logger.add_formatter(Exception, ->(e) {
|
|
12
|
+
# Lumberjack::Formatter::TaggedMessage.new(e.message, {
|
|
13
|
+
# error: {
|
|
14
|
+
# message: e.message,
|
|
15
|
+
# class: e.class.name,
|
|
16
|
+
# trace: e.backtrace
|
|
17
|
+
# }
|
|
18
|
+
# })
|
|
19
|
+
# })
|
|
20
|
+
class TaggedMessage
|
|
21
|
+
attr_reader :message, :tags
|
|
22
|
+
|
|
23
|
+
# @param [Formatter] formatter The formatter to apply the transformation to.
|
|
24
|
+
# @param [Proc] transform The transformation function to apply to the formatted string.
|
|
25
|
+
def initialize(message, tags)
|
|
26
|
+
@message = message
|
|
27
|
+
@tags = tags || {}
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def to_s
|
|
31
|
+
inspect
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
def inspect
|
|
35
|
+
{message: @message, tags: @tags}.inspect
|
|
36
|
+
end
|
|
37
|
+
end
|
|
38
|
+
end
|
|
39
|
+
end
|