lumberjack_json_device 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGE_LOG.md +3 -0
- data/MIT_LICENSE +20 -0
- data/README.md +110 -0
- data/VERSION +1 -0
- data/lib/lumberjack_json_device.rb +203 -0
- data/lumberjack_json_device.gemspec +34 -0
- metadata +105 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: '09791e96f24532aeb09e536ed838a28e26d08ed47a868f3f31698b1fefd12c29'
|
4
|
+
data.tar.gz: 4314fc238210e81ffb8743c660bf6cefe2921567e3611dd122b898542fe4578f
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: db055ceb22bfca8ebcd2b04454ef2bc4634a48af53d960e64196ff328f06eecee687bab2f8b776a203b3a7f52d3f131e5c2d4a8ff8525930ebfe06010d52ccef
|
7
|
+
data.tar.gz: bee629eeef6c71b6c19c39ec3d4e863baa796f053879202efb460aec88d6649f9127437bd4672a1dafb7a201e6de36f11eaf1e7cfb3f02768687216e77d808d0
|
data/CHANGE_LOG.md
ADDED
data/MIT_LICENSE
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
Copyright (c) 2020 Brian Durand
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
4
|
+
a copy of this software and associated documentation files (the
|
5
|
+
"Software"), to deal in the Software without restriction, including
|
6
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
7
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
8
|
+
permit persons to whom the Software is furnished to do so, subject to
|
9
|
+
the following conditions:
|
10
|
+
|
11
|
+
The above copyright notice and this permission notice shall be
|
12
|
+
included in all copies or substantial portions of the Software.
|
13
|
+
|
14
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
15
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
16
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
17
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
18
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
19
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
20
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,110 @@
|
|
1
|
+
# Lumberjack JSON Device
|
2
|
+
|
3
|
+
[](https://travis-ci.org/bdurand/lumberjack_json_device)
|
4
|
+
[](https://codeclimate.com/github/bdurand/lumberjack_json_device/maintainability)
|
5
|
+
|
6
|
+
This gem provides a logging device for the [lumberjack](https://github.com/bdurand/lumberjack) gem that will log JSON formatted output to a stream with one JSON document per line. This can be used as part of a log processing pipeline to ship the log to a structured data store or logging service.
|
7
|
+
|
8
|
+
## Destination
|
9
|
+
|
10
|
+
You can send the JSON output either to a stream or to another Lumberjack device.
|
11
|
+
|
12
|
+
```ruby
|
13
|
+
# Send to STDOUT
|
14
|
+
device = Lumberjack::JsonDevice.new(STDOUT)
|
15
|
+
|
16
|
+
# Send to another logging device
|
17
|
+
log_file = Lumberjack::Device::LogFile.new("/var/log/app.log")
|
18
|
+
device = Lumberjack::JsonDevice.new(log_file)
|
19
|
+
```
|
20
|
+
|
21
|
+
## JSON Structure
|
22
|
+
|
23
|
+
By default, the JSON document will map to the `Lumberjack::LogEntry` data structure.
|
24
|
+
|
25
|
+
```json
|
26
|
+
{"timestamp": "2020-01-02T19:47:45.123455", "severity": "INFO", "progname": "web", "pid": 101, "message": "test", "tags": {"foo": "bar"}}
|
27
|
+
```
|
28
|
+
|
29
|
+
You can specify a mapping to the device to customize the JSON document data structure. You can map the standard field names (time, severity, progname, pid, message, and tags) to custom field names.
|
30
|
+
|
31
|
+
If you map a field to an array, it will be mapped into a nested hash in the JSON document.
|
32
|
+
|
33
|
+
Any keys beyond the standard field names will be populated by a tag with the same name. These tags will not be included with the rest of the tags.
|
34
|
+
|
35
|
+
```ruby
|
36
|
+
device = Lumberjack::JsonDevice.new(STDOUT, mapping: {
|
37
|
+
time: "timestamp",
|
38
|
+
severity: "level",
|
39
|
+
progname: ["app", "name"],
|
40
|
+
pid: ["app", "pid"],
|
41
|
+
message: "message",
|
42
|
+
duration: "duration",
|
43
|
+
tags: "tags"
|
44
|
+
})
|
45
|
+
```
|
46
|
+
|
47
|
+
```json
|
48
|
+
{"timestamp": "2020-01-02T19:47:45.123455", "level": "INFO", "app": {"name": "web", "pid": 101}, "message": "test", "duration": 5, "tags": {"foo": "bar"}}
|
49
|
+
```
|
50
|
+
|
51
|
+
If you omit any fields in the mapping, they will not appear in the JSON document.
|
52
|
+
|
53
|
+
```ruby
|
54
|
+
device = Lumberjack::JsonDevice.new(STDOUT, mapping: {
|
55
|
+
time: "timestamp",
|
56
|
+
severity: "level",
|
57
|
+
message: "message",
|
58
|
+
})
|
59
|
+
```
|
60
|
+
|
61
|
+
```json
|
62
|
+
{"timestamp": "2020-01-02T19:47:45.123455", "level": "INFO", "message": "test"}
|
63
|
+
```
|
64
|
+
|
65
|
+
You can also provide a block or any object that responds to `call` in a mapping. The block will be called with the value and should emit a hash that will be merged into the JSON document.
|
66
|
+
|
67
|
+
```ruby
|
68
|
+
device = Lumberjack::JsonDevice.new(STDOUT, mapping: {
|
69
|
+
time: lambda { |val| (timestamp: val.to_f * 1000).round} },
|
70
|
+
severity: "level",
|
71
|
+
message: "test",
|
72
|
+
})
|
73
|
+
```
|
74
|
+
|
75
|
+
```json
|
76
|
+
{"timestamp": 1578125375588, "level": "INFO", "message": "test"}
|
77
|
+
```
|
78
|
+
|
79
|
+
Finally, you can specify `true` in the mapping as a short cut to map the field to the same name. If the field name contains periods, it will be mapped to a nested structure.
|
80
|
+
|
81
|
+
```ruby
|
82
|
+
device = Lumberjack::JsonDevice.new(STDOUT, mapping: {
|
83
|
+
"message" => true,
|
84
|
+
"http.status" => true,
|
85
|
+
"http.method" => true,
|
86
|
+
"http.path" => true
|
87
|
+
})
|
88
|
+
```
|
89
|
+
|
90
|
+
```json
|
91
|
+
{"message": "test", "http": {"status": 200, "method": "GET", "path": "/resource"}}
|
92
|
+
```
|
93
|
+
|
94
|
+
|
95
|
+
## Data Formatting
|
96
|
+
|
97
|
+
The device will have a `Lumberjack::Formatter` that will be used to format objects before serializing them as JSON. You can add additional formatters for classes to the default formatter, or supply a custom one when creating the device.
|
98
|
+
|
99
|
+
```ruby
|
100
|
+
device.formatter.add(Exception, LumberjacK::Formatter::InspectFormatter.new)
|
101
|
+
device.formatter.add(ActiveRecord::Base, LumberjacK::Formatter::IdFormatter.new)
|
102
|
+
```
|
103
|
+
|
104
|
+
You can also specify the `datetime_format` that will be used to serialize Time and DateTime objects.
|
105
|
+
|
106
|
+
```ruby
|
107
|
+
device.datetime_format = "%Y-%m-%dT%H:%M:%S.%3N"
|
108
|
+
```
|
109
|
+
|
110
|
+
Log entries with no message will not be written to the log.
|
data/VERSION
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
1.0.0
|
@@ -0,0 +1,203 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require 'lumberjack'
|
4
|
+
require 'multi_json'
|
5
|
+
require 'thread'
|
6
|
+
|
7
|
+
module Lumberjack
|
8
|
+
# This Lumberjack device logs output to another device as JSON formatted text with one document per line.
|
9
|
+
#
|
10
|
+
# The mapping parameter can be used to define the JSON data structure. To define the structure pass in a
|
11
|
+
# hash with key indicating the log entry field and the value indicating the JSON document key.
|
12
|
+
#
|
13
|
+
# The standard entry fields are mapped with the following keys:
|
14
|
+
#
|
15
|
+
# * :time
|
16
|
+
# * :severity
|
17
|
+
# * :progname
|
18
|
+
# * :pid
|
19
|
+
# * :message
|
20
|
+
# * :tags
|
21
|
+
#
|
22
|
+
# Any additional keys will be pulled from the tags. If any of the standard keys are missing or have a nil
|
23
|
+
# mapping, the entry field will not be included in the JSON output.
|
24
|
+
#
|
25
|
+
# You can create a nested JSON structure by specifying an array as the JSON key.
|
26
|
+
class JsonDevice < Device
|
27
|
+
|
28
|
+
DEFAULT_MAPPING = {
|
29
|
+
time: true,
|
30
|
+
severity: true,
|
31
|
+
progname: true,
|
32
|
+
pid: true,
|
33
|
+
message: true,
|
34
|
+
tags: true
|
35
|
+
}.freeze
|
36
|
+
|
37
|
+
DEFAULT_TIME_FORMAT = "%Y-%m-%dT%H:%M:%S.%6N%z"
|
38
|
+
|
39
|
+
attr_accessor :formatter
|
40
|
+
attr_reader :mapping
|
41
|
+
|
42
|
+
def initialize(stream_or_device, mapping: DEFAULT_MAPPING, formatter: nil, datetime_format: nil)
|
43
|
+
@mutex = Mutex.new
|
44
|
+
|
45
|
+
if stream_or_device.is_a?(Device)
|
46
|
+
@device = stream_or_device
|
47
|
+
else
|
48
|
+
@device = Writer.new(stream_or_device)
|
49
|
+
end
|
50
|
+
|
51
|
+
self.mapping = mapping
|
52
|
+
|
53
|
+
if formatter
|
54
|
+
@formatter = formatter
|
55
|
+
else
|
56
|
+
@formatter = default_formatter
|
57
|
+
datetime_format = DEFAULT_TIME_FORMAT if datetime_format.nil?
|
58
|
+
end
|
59
|
+
add_datetime_formatter!(datetime_format) unless datetime_format.nil?
|
60
|
+
end
|
61
|
+
|
62
|
+
def write(entry)
|
63
|
+
return if entry.message.nil? || entry.message == ""
|
64
|
+
data = entry_as_json(entry)
|
65
|
+
json = MultiJson.dump(data)
|
66
|
+
@device.write(json)
|
67
|
+
end
|
68
|
+
|
69
|
+
def flush
|
70
|
+
@device.flush
|
71
|
+
end
|
72
|
+
|
73
|
+
def datetime_format
|
74
|
+
@datetime_format
|
75
|
+
end
|
76
|
+
|
77
|
+
# Set the datetime format for the log timestamp.
|
78
|
+
def datetime_format=(format)
|
79
|
+
add_datetime_formatter!(format)
|
80
|
+
end
|
81
|
+
|
82
|
+
# Set the mapping for how to map an entry to a JSON object.
|
83
|
+
def mapping=(mapping)
|
84
|
+
@mutex.synchronize do
|
85
|
+
keys = {}
|
86
|
+
mapping.each do |key, value|
|
87
|
+
if value == true
|
88
|
+
value = key.to_s.split(".")
|
89
|
+
value = value.first if value.size == 1
|
90
|
+
end
|
91
|
+
keys[key.to_sym] = value if value
|
92
|
+
end
|
93
|
+
|
94
|
+
@time_key = keys.delete(:time)
|
95
|
+
@severity_key = keys.delete(:severity)
|
96
|
+
@progname_key = keys.delete(:progname)
|
97
|
+
@pid_key = keys.delete(:pid)
|
98
|
+
@message_key = keys.delete(:message)
|
99
|
+
@tags_key = keys.delete(:tags)
|
100
|
+
@custom_keys = keys
|
101
|
+
@mapping = mapping
|
102
|
+
end
|
103
|
+
nil
|
104
|
+
end
|
105
|
+
|
106
|
+
def map(field_mapping)
|
107
|
+
new_mapping = {}
|
108
|
+
field_mapping.each do |key, value|
|
109
|
+
new_mapping[key.to_sym] = value
|
110
|
+
end
|
111
|
+
self.mapping = mapping.merge(new_mapping)
|
112
|
+
end
|
113
|
+
|
114
|
+
# Convert a Lumberjack::LogEntry to a Hash using the specified field mapping.
|
115
|
+
def entry_as_json(entry)
|
116
|
+
data = {}
|
117
|
+
set_attribute(data, @time_key, entry.time) unless @time_key.nil?
|
118
|
+
set_attribute(data, @severity_key, entry.severity_label) unless @severity_key.nil?
|
119
|
+
set_attribute(data, @progname_key, entry.progname) unless @progname_key.nil?
|
120
|
+
set_attribute(data, @pid_key, entry.pid) unless @pid_key.nil?
|
121
|
+
set_attribute(data, @message_key, entry.message) unless @message_key.nil?
|
122
|
+
|
123
|
+
tags = entry.tags
|
124
|
+
if @custom_keys.size > 0
|
125
|
+
tags = (tags.nil? ? {} : tags.dup)
|
126
|
+
@custom_keys.each do |name, key|
|
127
|
+
set_attribute(data, key, tags.delete(name.to_s))
|
128
|
+
end
|
129
|
+
end
|
130
|
+
|
131
|
+
unless @tags_key.nil?
|
132
|
+
tags ||= {}
|
133
|
+
set_attribute(data, @tags_key, tags)
|
134
|
+
end
|
135
|
+
|
136
|
+
data = @formatter.format(data) if @formatter
|
137
|
+
data
|
138
|
+
end
|
139
|
+
|
140
|
+
private
|
141
|
+
|
142
|
+
def set_attribute(data, key, value)
|
143
|
+
return if value.nil?
|
144
|
+
|
145
|
+
if (value.is_a?(Time) || value.is_a?(DateTime)) && @time_formatter
|
146
|
+
value = @time_formatter.call(value)
|
147
|
+
end
|
148
|
+
|
149
|
+
if key.is_a?(Array)
|
150
|
+
unless key.empty?
|
151
|
+
if key.size == 1
|
152
|
+
data[key.first] = value
|
153
|
+
else
|
154
|
+
data[key.first] ||= {}
|
155
|
+
set_attribute(data[key.first], key[1, key.size], value)
|
156
|
+
end
|
157
|
+
end
|
158
|
+
elsif key.respond_to?(:call)
|
159
|
+
hash = key.call(value)
|
160
|
+
if hash.is_a?(Hash)
|
161
|
+
deep_merge!(data, Lumberjack::Tags.stringify_keys(hash))
|
162
|
+
end
|
163
|
+
else
|
164
|
+
data[key] = value unless key.nil?
|
165
|
+
end
|
166
|
+
end
|
167
|
+
|
168
|
+
def default_formatter
|
169
|
+
formatter = Formatter.new.clear
|
170
|
+
object_formatter = Lumberjack::Formatter::ObjectFormatter.new
|
171
|
+
formatter.add(String, object_formatter)
|
172
|
+
formatter.add(Object, object_formatter)
|
173
|
+
formatter.add(Enumerable, Formatter::StructuredFormatter.new(formatter))
|
174
|
+
formatter
|
175
|
+
end
|
176
|
+
|
177
|
+
def add_datetime_formatter!(datetime_format)
|
178
|
+
if datetime_format
|
179
|
+
@datetime_format = datetime_format
|
180
|
+
time_formatter = Lumberjack::Formatter::DateTimeFormatter.new(datetime_format)
|
181
|
+
formatter.add(Time, time_formatter)
|
182
|
+
formatter.add(Date, time_formatter)
|
183
|
+
else
|
184
|
+
@datetime_format = nil
|
185
|
+
formatter.remove(Time)
|
186
|
+
formatter.remove(Date)
|
187
|
+
end
|
188
|
+
end
|
189
|
+
|
190
|
+
def deep_merge!(hash, other_hash, &block)
|
191
|
+
hash.merge!(other_hash) do |key, this_val, other_val|
|
192
|
+
if this_val.is_a?(Hash) && other_val.is_a?(Hash)
|
193
|
+
deep_merge!(this_val, other_val, &block)
|
194
|
+
elsif block_given?
|
195
|
+
block.call(key, this_val, other_val)
|
196
|
+
else
|
197
|
+
other_val
|
198
|
+
end
|
199
|
+
end
|
200
|
+
end
|
201
|
+
|
202
|
+
end
|
203
|
+
end
|
@@ -0,0 +1,34 @@
|
|
1
|
+
Gem::Specification.new do |spec|
|
2
|
+
spec.name = 'lumberjack_json_device'
|
3
|
+
spec.version = File.read(File.expand_path("../VERSION", __FILE__)).strip
|
4
|
+
spec.authors = ['Brian Durand']
|
5
|
+
spec.email = ['bbdurand@gmail.com']
|
6
|
+
|
7
|
+
spec.summary = "A logging device for the lumberjack gem that writes log entries as JSON documentspec."
|
8
|
+
spec.homepage = "https://github.com/bdurand/lumberjack_json_device"
|
9
|
+
spec.license = "MIT"
|
10
|
+
|
11
|
+
# Specify which files should be added to the gem when it is released.
|
12
|
+
# The `git ls-files -z` loads the files in the RubyGem that have been added into git.
|
13
|
+
ignore_files = %w(
|
14
|
+
.gitignore
|
15
|
+
.travis.yml
|
16
|
+
Appraisals
|
17
|
+
Gemfile
|
18
|
+
Gemfile.lock
|
19
|
+
Rakefile
|
20
|
+
gemfiles/
|
21
|
+
spec/
|
22
|
+
)
|
23
|
+
spec.files = Dir.chdir(File.expand_path('..', __FILE__)) do
|
24
|
+
`git ls-files -z`.split("\x0").reject{ |f| ignore_files.any?{ |path| f.start_with?(path) } }
|
25
|
+
end
|
26
|
+
|
27
|
+
spec.require_paths = ['lib']
|
28
|
+
|
29
|
+
spec.add_dependency "lumberjack", ">=1.2"
|
30
|
+
spec.add_dependency "multi_json"
|
31
|
+
|
32
|
+
spec.add_development_dependency("rspec", ["~> 3.0"])
|
33
|
+
spec.add_development_dependency "rake"
|
34
|
+
end
|
metadata
ADDED
@@ -0,0 +1,105 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: lumberjack_json_device
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 1.0.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Brian Durand
|
8
|
+
autorequire:
|
9
|
+
bindir: bin
|
10
|
+
cert_chain: []
|
11
|
+
date: 2020-01-20 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: lumberjack
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - ">="
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '1.2'
|
20
|
+
type: :runtime
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - ">="
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '1.2'
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: multi_json
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - ">="
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '0'
|
34
|
+
type: :runtime
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - ">="
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: '0'
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: rspec
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - "~>"
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '3.0'
|
48
|
+
type: :development
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - "~>"
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: '3.0'
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: rake
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - ">="
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '0'
|
62
|
+
type: :development
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - ">="
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '0'
|
69
|
+
description:
|
70
|
+
email:
|
71
|
+
- bbdurand@gmail.com
|
72
|
+
executables: []
|
73
|
+
extensions: []
|
74
|
+
extra_rdoc_files: []
|
75
|
+
files:
|
76
|
+
- CHANGE_LOG.md
|
77
|
+
- MIT_LICENSE
|
78
|
+
- README.md
|
79
|
+
- VERSION
|
80
|
+
- lib/lumberjack_json_device.rb
|
81
|
+
- lumberjack_json_device.gemspec
|
82
|
+
homepage: https://github.com/bdurand/lumberjack_json_device
|
83
|
+
licenses:
|
84
|
+
- MIT
|
85
|
+
metadata: {}
|
86
|
+
post_install_message:
|
87
|
+
rdoc_options: []
|
88
|
+
require_paths:
|
89
|
+
- lib
|
90
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
91
|
+
requirements:
|
92
|
+
- - ">="
|
93
|
+
- !ruby/object:Gem::Version
|
94
|
+
version: '0'
|
95
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
96
|
+
requirements:
|
97
|
+
- - ">="
|
98
|
+
- !ruby/object:Gem::Version
|
99
|
+
version: '0'
|
100
|
+
requirements: []
|
101
|
+
rubygems_version: 3.0.3
|
102
|
+
signing_key:
|
103
|
+
specification_version: 4
|
104
|
+
summary: A logging device for the lumberjack gem that writes log entries as JSON documentspec.
|
105
|
+
test_files: []
|