lumberjack_json_device 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: '09791e96f24532aeb09e536ed838a28e26d08ed47a868f3f31698b1fefd12c29'
4
+ data.tar.gz: 4314fc238210e81ffb8743c660bf6cefe2921567e3611dd122b898542fe4578f
5
+ SHA512:
6
+ metadata.gz: db055ceb22bfca8ebcd2b04454ef2bc4634a48af53d960e64196ff328f06eecee687bab2f8b776a203b3a7f52d3f131e5c2d4a8ff8525930ebfe06010d52ccef
7
+ data.tar.gz: bee629eeef6c71b6c19c39ec3d4e863baa796f053879202efb460aec88d6649f9127437bd4672a1dafb7a201e6de36f11eaf1e7cfb3f02768687216e77d808d0
@@ -0,0 +1,3 @@
1
+ # 1.0.0
2
+
3
+ * Initial release
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2020 Brian Durand
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,110 @@
1
+ # Lumberjack JSON Device
2
+
3
+ [![Build Status](https://travis-ci.org/bdurand/lumberjack_json_device.svg?branch=master)](https://travis-ci.org/bdurand/lumberjack_json_device)
4
+ [![Maintainability](https://api.codeclimate.com/v1/badges/c62cb886b86381560810/maintainability)](https://codeclimate.com/github/bdurand/lumberjack_json_device/maintainability)
5
+
6
+ This gem provides a logging device for the [lumberjack](https://github.com/bdurand/lumberjack) gem that will log JSON formatted output to a stream with one JSON document per line. This can be used as part of a log processing pipeline to ship the log to a structured data store or logging service.
7
+
8
+ ## Destination
9
+
10
+ You can send the JSON output either to a stream or to another Lumberjack device.
11
+
12
+ ```ruby
13
+ # Send to STDOUT
14
+ device = Lumberjack::JsonDevice.new(STDOUT)
15
+
16
+ # Send to another logging device
17
+ log_file = Lumberjack::Device::LogFile.new("/var/log/app.log")
18
+ device = Lumberjack::JsonDevice.new(log_file)
19
+ ```
20
+
21
+ ## JSON Structure
22
+
23
+ By default, the JSON document will map to the `Lumberjack::LogEntry` data structure.
24
+
25
+ ```json
26
+ {"timestamp": "2020-01-02T19:47:45.123455", "severity": "INFO", "progname": "web", "pid": 101, "message": "test", "tags": {"foo": "bar"}}
27
+ ```
28
+
29
+ You can specify a mapping to the device to customize the JSON document data structure. You can map the standard field names (time, severity, progname, pid, message, and tags) to custom field names.
30
+
31
+ If you map a field to an array, it will be mapped into a nested hash in the JSON document.
32
+
33
+ Any keys beyond the standard field names will be populated by a tag with the same name. These tags will not be included with the rest of the tags.
34
+
35
+ ```ruby
36
+ device = Lumberjack::JsonDevice.new(STDOUT, mapping: {
37
+ time: "timestamp",
38
+ severity: "level",
39
+ progname: ["app", "name"],
40
+ pid: ["app", "pid"],
41
+ message: "message",
42
+ duration: "duration",
43
+ tags: "tags"
44
+ })
45
+ ```
46
+
47
+ ```json
48
+ {"timestamp": "2020-01-02T19:47:45.123455", "level": "INFO", "app": {"name": "web", "pid": 101}, "message": "test", "duration": 5, "tags": {"foo": "bar"}}
49
+ ```
50
+
51
+ If you omit any fields in the mapping, they will not appear in the JSON document.
52
+
53
+ ```ruby
54
+ device = Lumberjack::JsonDevice.new(STDOUT, mapping: {
55
+ time: "timestamp",
56
+ severity: "level",
57
+ message: "message",
58
+ })
59
+ ```
60
+
61
+ ```json
62
+ {"timestamp": "2020-01-02T19:47:45.123455", "level": "INFO", "message": "test"}
63
+ ```
64
+
65
+ You can also provide a block or any object that responds to `call` in a mapping. The block will be called with the value and should emit a hash that will be merged into the JSON document.
66
+
67
+ ```ruby
68
+ device = Lumberjack::JsonDevice.new(STDOUT, mapping: {
69
+ time: lambda { |val| (timestamp: val.to_f * 1000).round} },
70
+ severity: "level",
71
+ message: "test",
72
+ })
73
+ ```
74
+
75
+ ```json
76
+ {"timestamp": 1578125375588, "level": "INFO", "message": "test"}
77
+ ```
78
+
79
+ Finally, you can specify `true` in the mapping as a short cut to map the field to the same name. If the field name contains periods, it will be mapped to a nested structure.
80
+
81
+ ```ruby
82
+ device = Lumberjack::JsonDevice.new(STDOUT, mapping: {
83
+ "message" => true,
84
+ "http.status" => true,
85
+ "http.method" => true,
86
+ "http.path" => true
87
+ })
88
+ ```
89
+
90
+ ```json
91
+ {"message": "test", "http": {"status": 200, "method": "GET", "path": "/resource"}}
92
+ ```
93
+
94
+
95
+ ## Data Formatting
96
+
97
+ The device will have a `Lumberjack::Formatter` that will be used to format objects before serializing them as JSON. You can add additional formatters for classes to the default formatter, or supply a custom one when creating the device.
98
+
99
+ ```ruby
100
+ device.formatter.add(Exception, LumberjacK::Formatter::InspectFormatter.new)
101
+ device.formatter.add(ActiveRecord::Base, LumberjacK::Formatter::IdFormatter.new)
102
+ ```
103
+
104
+ You can also specify the `datetime_format` that will be used to serialize Time and DateTime objects.
105
+
106
+ ```ruby
107
+ device.datetime_format = "%Y-%m-%dT%H:%M:%S.%3N"
108
+ ```
109
+
110
+ Log entries with no message will not be written to the log.
data/VERSION ADDED
@@ -0,0 +1 @@
1
+ 1.0.0
@@ -0,0 +1,203 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'lumberjack'
4
+ require 'multi_json'
5
+ require 'thread'
6
+
7
+ module Lumberjack
8
+ # This Lumberjack device logs output to another device as JSON formatted text with one document per line.
9
+ #
10
+ # The mapping parameter can be used to define the JSON data structure. To define the structure pass in a
11
+ # hash with key indicating the log entry field and the value indicating the JSON document key.
12
+ #
13
+ # The standard entry fields are mapped with the following keys:
14
+ #
15
+ # * :time
16
+ # * :severity
17
+ # * :progname
18
+ # * :pid
19
+ # * :message
20
+ # * :tags
21
+ #
22
+ # Any additional keys will be pulled from the tags. If any of the standard keys are missing or have a nil
23
+ # mapping, the entry field will not be included in the JSON output.
24
+ #
25
+ # You can create a nested JSON structure by specifying an array as the JSON key.
26
+ class JsonDevice < Device
27
+
28
+ DEFAULT_MAPPING = {
29
+ time: true,
30
+ severity: true,
31
+ progname: true,
32
+ pid: true,
33
+ message: true,
34
+ tags: true
35
+ }.freeze
36
+
37
+ DEFAULT_TIME_FORMAT = "%Y-%m-%dT%H:%M:%S.%6N%z"
38
+
39
+ attr_accessor :formatter
40
+ attr_reader :mapping
41
+
42
+ def initialize(stream_or_device, mapping: DEFAULT_MAPPING, formatter: nil, datetime_format: nil)
43
+ @mutex = Mutex.new
44
+
45
+ if stream_or_device.is_a?(Device)
46
+ @device = stream_or_device
47
+ else
48
+ @device = Writer.new(stream_or_device)
49
+ end
50
+
51
+ self.mapping = mapping
52
+
53
+ if formatter
54
+ @formatter = formatter
55
+ else
56
+ @formatter = default_formatter
57
+ datetime_format = DEFAULT_TIME_FORMAT if datetime_format.nil?
58
+ end
59
+ add_datetime_formatter!(datetime_format) unless datetime_format.nil?
60
+ end
61
+
62
+ def write(entry)
63
+ return if entry.message.nil? || entry.message == ""
64
+ data = entry_as_json(entry)
65
+ json = MultiJson.dump(data)
66
+ @device.write(json)
67
+ end
68
+
69
+ def flush
70
+ @device.flush
71
+ end
72
+
73
+ def datetime_format
74
+ @datetime_format
75
+ end
76
+
77
+ # Set the datetime format for the log timestamp.
78
+ def datetime_format=(format)
79
+ add_datetime_formatter!(format)
80
+ end
81
+
82
+ # Set the mapping for how to map an entry to a JSON object.
83
+ def mapping=(mapping)
84
+ @mutex.synchronize do
85
+ keys = {}
86
+ mapping.each do |key, value|
87
+ if value == true
88
+ value = key.to_s.split(".")
89
+ value = value.first if value.size == 1
90
+ end
91
+ keys[key.to_sym] = value if value
92
+ end
93
+
94
+ @time_key = keys.delete(:time)
95
+ @severity_key = keys.delete(:severity)
96
+ @progname_key = keys.delete(:progname)
97
+ @pid_key = keys.delete(:pid)
98
+ @message_key = keys.delete(:message)
99
+ @tags_key = keys.delete(:tags)
100
+ @custom_keys = keys
101
+ @mapping = mapping
102
+ end
103
+ nil
104
+ end
105
+
106
+ def map(field_mapping)
107
+ new_mapping = {}
108
+ field_mapping.each do |key, value|
109
+ new_mapping[key.to_sym] = value
110
+ end
111
+ self.mapping = mapping.merge(new_mapping)
112
+ end
113
+
114
+ # Convert a Lumberjack::LogEntry to a Hash using the specified field mapping.
115
+ def entry_as_json(entry)
116
+ data = {}
117
+ set_attribute(data, @time_key, entry.time) unless @time_key.nil?
118
+ set_attribute(data, @severity_key, entry.severity_label) unless @severity_key.nil?
119
+ set_attribute(data, @progname_key, entry.progname) unless @progname_key.nil?
120
+ set_attribute(data, @pid_key, entry.pid) unless @pid_key.nil?
121
+ set_attribute(data, @message_key, entry.message) unless @message_key.nil?
122
+
123
+ tags = entry.tags
124
+ if @custom_keys.size > 0
125
+ tags = (tags.nil? ? {} : tags.dup)
126
+ @custom_keys.each do |name, key|
127
+ set_attribute(data, key, tags.delete(name.to_s))
128
+ end
129
+ end
130
+
131
+ unless @tags_key.nil?
132
+ tags ||= {}
133
+ set_attribute(data, @tags_key, tags)
134
+ end
135
+
136
+ data = @formatter.format(data) if @formatter
137
+ data
138
+ end
139
+
140
+ private
141
+
142
+ def set_attribute(data, key, value)
143
+ return if value.nil?
144
+
145
+ if (value.is_a?(Time) || value.is_a?(DateTime)) && @time_formatter
146
+ value = @time_formatter.call(value)
147
+ end
148
+
149
+ if key.is_a?(Array)
150
+ unless key.empty?
151
+ if key.size == 1
152
+ data[key.first] = value
153
+ else
154
+ data[key.first] ||= {}
155
+ set_attribute(data[key.first], key[1, key.size], value)
156
+ end
157
+ end
158
+ elsif key.respond_to?(:call)
159
+ hash = key.call(value)
160
+ if hash.is_a?(Hash)
161
+ deep_merge!(data, Lumberjack::Tags.stringify_keys(hash))
162
+ end
163
+ else
164
+ data[key] = value unless key.nil?
165
+ end
166
+ end
167
+
168
+ def default_formatter
169
+ formatter = Formatter.new.clear
170
+ object_formatter = Lumberjack::Formatter::ObjectFormatter.new
171
+ formatter.add(String, object_formatter)
172
+ formatter.add(Object, object_formatter)
173
+ formatter.add(Enumerable, Formatter::StructuredFormatter.new(formatter))
174
+ formatter
175
+ end
176
+
177
+ def add_datetime_formatter!(datetime_format)
178
+ if datetime_format
179
+ @datetime_format = datetime_format
180
+ time_formatter = Lumberjack::Formatter::DateTimeFormatter.new(datetime_format)
181
+ formatter.add(Time, time_formatter)
182
+ formatter.add(Date, time_formatter)
183
+ else
184
+ @datetime_format = nil
185
+ formatter.remove(Time)
186
+ formatter.remove(Date)
187
+ end
188
+ end
189
+
190
+ def deep_merge!(hash, other_hash, &block)
191
+ hash.merge!(other_hash) do |key, this_val, other_val|
192
+ if this_val.is_a?(Hash) && other_val.is_a?(Hash)
193
+ deep_merge!(this_val, other_val, &block)
194
+ elsif block_given?
195
+ block.call(key, this_val, other_val)
196
+ else
197
+ other_val
198
+ end
199
+ end
200
+ end
201
+
202
+ end
203
+ end
@@ -0,0 +1,34 @@
1
+ Gem::Specification.new do |spec|
2
+ spec.name = 'lumberjack_json_device'
3
+ spec.version = File.read(File.expand_path("../VERSION", __FILE__)).strip
4
+ spec.authors = ['Brian Durand']
5
+ spec.email = ['bbdurand@gmail.com']
6
+
7
+ spec.summary = "A logging device for the lumberjack gem that writes log entries as JSON documentspec."
8
+ spec.homepage = "https://github.com/bdurand/lumberjack_json_device"
9
+ spec.license = "MIT"
10
+
11
+ # Specify which files should be added to the gem when it is released.
12
+ # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
13
+ ignore_files = %w(
14
+ .gitignore
15
+ .travis.yml
16
+ Appraisals
17
+ Gemfile
18
+ Gemfile.lock
19
+ Rakefile
20
+ gemfiles/
21
+ spec/
22
+ )
23
+ spec.files = Dir.chdir(File.expand_path('..', __FILE__)) do
24
+ `git ls-files -z`.split("\x0").reject{ |f| ignore_files.any?{ |path| f.start_with?(path) } }
25
+ end
26
+
27
+ spec.require_paths = ['lib']
28
+
29
+ spec.add_dependency "lumberjack", ">=1.2"
30
+ spec.add_dependency "multi_json"
31
+
32
+ spec.add_development_dependency("rspec", ["~> 3.0"])
33
+ spec.add_development_dependency "rake"
34
+ end
metadata ADDED
@@ -0,0 +1,105 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: lumberjack_json_device
3
+ version: !ruby/object:Gem::Version
4
+ version: 1.0.0
5
+ platform: ruby
6
+ authors:
7
+ - Brian Durand
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2020-01-20 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: lumberjack
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ">="
18
+ - !ruby/object:Gem::Version
19
+ version: '1.2'
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - ">="
25
+ - !ruby/object:Gem::Version
26
+ version: '1.2'
27
+ - !ruby/object:Gem::Dependency
28
+ name: multi_json
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - ">="
32
+ - !ruby/object:Gem::Version
33
+ version: '0'
34
+ type: :runtime
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - ">="
39
+ - !ruby/object:Gem::Version
40
+ version: '0'
41
+ - !ruby/object:Gem::Dependency
42
+ name: rspec
43
+ requirement: !ruby/object:Gem::Requirement
44
+ requirements:
45
+ - - "~>"
46
+ - !ruby/object:Gem::Version
47
+ version: '3.0'
48
+ type: :development
49
+ prerelease: false
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - "~>"
53
+ - !ruby/object:Gem::Version
54
+ version: '3.0'
55
+ - !ruby/object:Gem::Dependency
56
+ name: rake
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - ">="
60
+ - !ruby/object:Gem::Version
61
+ version: '0'
62
+ type: :development
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - ">="
67
+ - !ruby/object:Gem::Version
68
+ version: '0'
69
+ description:
70
+ email:
71
+ - bbdurand@gmail.com
72
+ executables: []
73
+ extensions: []
74
+ extra_rdoc_files: []
75
+ files:
76
+ - CHANGE_LOG.md
77
+ - MIT_LICENSE
78
+ - README.md
79
+ - VERSION
80
+ - lib/lumberjack_json_device.rb
81
+ - lumberjack_json_device.gemspec
82
+ homepage: https://github.com/bdurand/lumberjack_json_device
83
+ licenses:
84
+ - MIT
85
+ metadata: {}
86
+ post_install_message:
87
+ rdoc_options: []
88
+ require_paths:
89
+ - lib
90
+ required_ruby_version: !ruby/object:Gem::Requirement
91
+ requirements:
92
+ - - ">="
93
+ - !ruby/object:Gem::Version
94
+ version: '0'
95
+ required_rubygems_version: !ruby/object:Gem::Requirement
96
+ requirements:
97
+ - - ">="
98
+ - !ruby/object:Gem::Version
99
+ version: '0'
100
+ requirements: []
101
+ rubygems_version: 3.0.3
102
+ signing_key:
103
+ specification_version: 4
104
+ summary: A logging device for the lumberjack gem that writes log entries as JSON documentspec.
105
+ test_files: []