fluent-plugin-windows-eventlog 0.3.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 8c05a4b1a785316299232e1cd4c3a5f60e7458b93b5b4d717720a6de5f079cbb
4
- data.tar.gz: aad0c8d51e5b88d96f86704bb72394d6a16253edecd8b975ed6c3334f433ebf4
3
+ metadata.gz: 5ea9a134414a5572bf7d3c9dca355ea53d47d6a8701b6a631f0426bafa0de5c9
4
+ data.tar.gz: 26d9fa562585f26fdbef294aa6f64a13e7749ed11915390d1822c5367656ccb7
5
5
  SHA512:
6
- metadata.gz: 185857a8f3114029de23b8a0b9b262de929e20b6ad98f783a937bb4eb48e8d594a84df33405ff7a5b348c43f0f81d601bdfea72288f4e30637e98035f92ed834
7
- data.tar.gz: 1bb7650803e3852a2a14c701f9e3aeeeab932438b36702719f246c8e6b2be4b6c3ffa67a567e2c2dae1fc0cff7dd82144fcc874acb7045ea9beb5c819f6f28eb
6
+ metadata.gz: e292b037dc58e418aa8b74b1fa8b19db22124c927876e4fbf2e59b457042e08944e912e37104891fdd93f841b306ea8d09d2b1bbf939dad2015f18e26c021a11
7
+ data.tar.gz: 6d82ed5588b14f92bca8bdf42a8c9a085306ed23717e9b0f0fc6cdc994b376a00195469c51dcb9ddeb5a6a786fc7c5b96e24279037bc20bfa96f42477407851d
@@ -1,3 +1,9 @@
1
+ # Release v0.4.0 - 2019/10/10
2
+
3
+ * in_windows_eventlog2: Add new `render_as_xml` parameter to switch rendering as XML or Ruby Hash object
4
+ * in_windows_eventlog2: Support rate limit with `rate_limit` option
5
+ * parser_winevt_xml: Separate `parser_winevt_xml` plugin to other repository and published as Fluentd parser plugin
6
+
1
7
  # Release v0.3.0 - 2019/07/08
2
8
 
3
9
  * Add new `in_windows_eventlog2` plugin. This plugin uses newer windows event logging API.
data/README.md CHANGED
@@ -4,10 +4,10 @@
4
4
 
5
5
  ### fluentd Input plugin for the Windows Event Log
6
6
 
7
- [Fluentd](http://fluentd.org) plugin to read the Windows Event Log.
7
+ [Fluentd](https://www.fluentd.org/) plugin to read the Windows Event Log.
8
8
 
9
9
  ## Installation
10
- gem install fluent-plugin-windows-eventlog
10
+ ridk exec gem install fluent-plugin-windows-eventlog
11
11
 
12
12
  ## Configuration
13
13
 
@@ -15,7 +15,7 @@
15
15
 
16
16
  Check [in_windows_eventlog2](https://github.com/fluent/fluent-plugin-windows-eventlog#in_windows_eventlog2) first. `in_windows_eventlog` will be replaced with `in_windows_eventlog2`.
17
17
 
18
- #### fluentd Input plugin for the Windows Event Log using old Windows Event Logging API
18
+ fluentd Input plugin for the Windows Event Log using old Windows Event Logging API
19
19
 
20
20
  <source>
21
21
  @type windows_eventlog
@@ -130,7 +130,7 @@ If your `description` doesn't follow this format, the parsed result is only `des
130
130
 
131
131
  ### in_windows_eventlog2
132
132
 
133
- #### fluentd Input plugin for the Windows Event Log using newer Windows Event Logging API
133
+ fluentd Input plugin for the Windows Event Log using newer Windows Event Logging API. This is successor to `in_windows_eventlog`. See also [this slide](https://www.slideshare.net/cosmo0920/fluentd-meetup-2019) for the details of `in_windows_eventlog2` plugin.
134
134
 
135
135
  <source>
136
136
  @type windows_eventlog2
@@ -138,6 +138,8 @@ If your `description` doesn't follow this format, the parsed result is only `des
138
138
  channels application,system
139
139
  read_interval 2
140
140
  tag winevt.raw
141
+ render_as_xml false # default is true.
142
+ rate_limit 200 # default is -1(Winevt::EventLog::Subscribe::RATE_INFINITE).
141
143
  <storage>
142
144
  @type local # @type local is the default.
143
145
  persistent true # default is true. Set to false to use in-memory storage.
@@ -153,6 +155,10 @@ If your `description` doesn't follow this format, the parsed result is only `des
153
155
 
154
156
  **NOTE:** When `Description` contains error message such as `The message resource is present but the message was not found in the message table.`, eventlog's resource file (.mui) related to error generating event is something wrong. This issue is also occurred in built-in Windows Event Viewer which is the part of Windows management tool.
155
157
 
158
+ **NOTE:** When `render_as_xml` as `false`, the dependent winevt_c gem renders Windows EventLog as Ruby Hash object directly. This reduces bottleneck to consume EventLog. Specifying `render_as_xml` as `false` should be faster consuming than `render_as_xml` as `true` case.
159
+
160
+ **NOTE:** If you encountered CPU spike due to massively huge EventLog channel, `rate_limit` parameter may help you. Currently, this paramter can handle the multiples of 10 or -1(`Winevt::EventLog::Subscribe::RATE_INFINITE`).
161
+
156
162
  #### parameters
157
163
 
158
164
  |name | description |
@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
4
 
5
5
  Gem::Specification.new do |spec|
6
6
  spec.name = "fluent-plugin-windows-eventlog"
7
- spec.version = "0.3.0"
7
+ spec.version = "0.4.0"
8
8
  spec.authors = ["okahashi117", "Hiroshi Hatake", "Masahiro Nakagawa"]
9
9
  spec.email = ["naruki_okahashi@jbat.co.jp", "cosmo0920.oucc@gmail.com", "repeatedly@gmail.com"]
10
10
  spec.summary = %q{Fluentd Input plugin to read windows event log.}
@@ -22,6 +22,7 @@ Gem::Specification.new do |spec|
22
22
  spec.add_development_dependency "test-unit", "~> 3.2.0"
23
23
  spec.add_runtime_dependency "fluentd", [">= 0.14.12", "< 2"]
24
24
  spec.add_runtime_dependency "win32-eventlog"
25
- spec.add_runtime_dependency "winevt_c"
25
+ spec.add_runtime_dependency "winevt_c", ">= 0.6.0"
26
26
  spec.add_runtime_dependency "nokogiri", "~> 1.10"
27
+ spec.add_runtime_dependency "fluent-plugin-parser-winevt_xml", ">= 0.1.2"
27
28
  end
@@ -35,6 +35,8 @@ module Fluent::Plugin
35
35
  config_param :keys, :array, default: []
36
36
  config_param :read_from_head, :bool, default: false
37
37
  config_param :parse_description, :bool, default: false
38
+ config_param :render_as_xml, :bool, default: true
39
+ config_param :rate_limit, :integer, default: Winevt::EventLog::Subscribe::RATE_INFINITE
38
40
 
39
41
  config_section :storage do
40
42
  config_set_default :usage, "bookmarks"
@@ -60,12 +62,24 @@ module Fluent::Plugin
60
62
  if @keynames.empty?
61
63
  @keynames = KEY_MAP.keys
62
64
  end
65
+ @keynames.delete('Qualifiers') unless @render_as_xml
63
66
  @keynames.delete('EventData') if @parse_description
64
67
 
65
68
  @tag = tag
66
69
  @tailing = @read_from_head ? false : true
67
70
  @bookmarks_storage = storage_create(usage: "bookmarks")
68
- @parser = parser_create
71
+ @winevt_xml = false
72
+ if @render_as_xml
73
+ @parser = parser_create
74
+ @winevt_xml = @parser.respond_to?(:winevt_xml?) && @parser.winevt_xml?
75
+ class << self
76
+ alias_method :on_notify, :on_notify_xml
77
+ end
78
+ else
79
+ class << self
80
+ alias_method :on_notify, :on_notify_hash
81
+ end
82
+ end
69
83
  end
70
84
 
71
85
  def start
@@ -77,6 +91,8 @@ module Fluent::Plugin
77
91
  bookmark = Winevt::EventLog::Bookmark.new(bookmarkXml)
78
92
  subscribe.tail = @tailing
79
93
  subscribe.subscribe(ch, "*", bookmark)
94
+ subscribe.render_as_xml = @render_as_xml
95
+ subscribe.rate_limit = @rate_limit
80
96
  timer_execute("in_windows_eventlog_#{escape_channel(ch)}".to_sym, @read_interval) do
81
97
  on_notify(ch, subscribe)
82
98
  end
@@ -88,11 +104,15 @@ module Fluent::Plugin
88
104
  end
89
105
 
90
106
  def on_notify(ch, subscribe)
107
+ # for safety.
108
+ end
109
+
110
+ def on_notify_xml(ch, subscribe)
91
111
  es = Fluent::MultiEventStream.new
92
112
  subscribe.each do |xml, message, string_inserts|
93
113
  @parser.parse(xml) do |time, record|
94
114
  # record.has_key?("EventData") for none parser checking.
95
- if record.has_key?("EventData")
115
+ if @winevt_xml
96
116
  record["Description"] = message
97
117
  record["EventData"] = string_inserts
98
118
 
@@ -112,6 +132,8 @@ module Fluent::Plugin
112
132
  parse_desc(h) if @parse_description
113
133
  es.add(Fluent::Engine.now, h)
114
134
  else
135
+ record["Description"] = message
136
+ record["EventData"] = string_inserts
115
137
  # for none parser
116
138
  es.add(Fluent::Engine.now, record)
117
139
  end
@@ -121,6 +143,31 @@ module Fluent::Plugin
121
143
  @bookmarks_storage.put(ch, subscribe.bookmark)
122
144
  end
123
145
 
146
+ def on_notify_hash(ch, subscribe)
147
+ es = Fluent::MultiEventStream.new
148
+ subscribe.each do |record, message, string_inserts|
149
+ record["Description"] = message
150
+ record["EventData"] = string_inserts
151
+ h = {}
152
+ @keynames.each do |k|
153
+ type = KEY_MAP[k][1]
154
+ value = record[KEY_MAP[k][0]]
155
+ h[k]=case type
156
+ when :string
157
+ value.to_s
158
+ when :array
159
+ value.map {|v| v.to_s}
160
+ else
161
+ raise "Unknown value type: #{type}"
162
+ end
163
+ end
164
+ parse_desc(h) if @parse_description
165
+ es.add(Fluent::Engine.now, h)
166
+ end
167
+ router.emit_stream(@tag, es)
168
+ @bookmarks_storage.put(ch, subscribe.bookmark)
169
+ end
170
+
124
171
  #### These lines copied from in_windows_eventlog plugin:
125
172
  #### https://github.com/fluent/fluent-plugin-windows-eventlog/blob/528290d896a885c7721f850943daa3a43a015f3d/lib/fluent/plugin/in_windows_eventlog.rb#L192-L232
126
173
  GROUP_DELIMITER = "\r\n\r\n".freeze
@@ -23,10 +23,8 @@ unless ENV.has_key?('VERBOSE')
23
23
  end
24
24
 
25
25
  require 'fluent/test/driver/input'
26
- require 'fluent/test/driver/parser'
27
26
  require 'fluent/plugin/in_windows_eventlog'
28
27
  require 'fluent/plugin/in_windows_eventlog2'
29
- require 'fluent/plugin/parser_winevt_xml'
30
28
 
31
29
  class Test::Unit::TestCase
32
30
  end
@@ -25,6 +25,7 @@ class WindowsEventLog2InputTest < Test::Unit::TestCase
25
25
  assert_equal 2, d.instance.read_interval
26
26
  assert_equal ['application'], d.instance.channels
27
27
  assert_false d.instance.read_from_head
28
+ assert_true d.instance.render_as_xml
28
29
  end
29
30
 
30
31
  def test_parse_desc
@@ -96,6 +97,34 @@ DESC
96
97
  assert_equal(expected, record)
97
98
  end
98
99
 
100
+ class HashRendered < self
101
+ def test_write
102
+ d = create_driver(config_element("ROOT", "", {"tag" => "fluent.eventlog",
103
+ "render_as_xml" => false}, [
104
+ config_element("storage", "", {
105
+ '@type' => 'local',
106
+ 'persistent' => false
107
+ })
108
+ ]))
109
+
110
+ service = Fluent::Plugin::EventService.new
111
+
112
+ d.run(expect_emits: 1) do
113
+ service.run
114
+ end
115
+
116
+ assert(d.events.length >= 1)
117
+ event = d.events.last
118
+ record = event.last
119
+
120
+ assert_false(d.instance.render_as_xml)
121
+ assert_equal("Application", record["Channel"])
122
+ assert_equal("65500", record["EventID"])
123
+ assert_equal("4", record["Level"])
124
+ assert_equal("fluent-plugins", record["ProviderName"])
125
+ end
126
+ end
127
+
99
128
  class PersistBookMark < self
100
129
  TEST_PLUGIN_STORAGE_PATH = File.join( File.dirname(File.dirname(__FILE__)), 'tmp', 'in_windows_eventlog2', 'store' )
101
130
  CONFIG2 = config_element("ROOT", "", {"tag" => "fluent.eventlog"}, [
@@ -178,5 +207,8 @@ DESC
178
207
  # record should be {message: <RAW XML EventLog>}.
179
208
  record["message"]
180
209
  end
210
+
211
+ assert_true(record.has_key?("Description"))
212
+ assert_true(record.has_key?("EventData"))
181
213
  end
182
214
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-windows-eventlog
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.3.0
4
+ version: 0.4.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - okahashi117
@@ -10,7 +10,7 @@ authors:
10
10
  autorequire:
11
11
  bindir: bin
12
12
  cert_chain: []
13
- date: 2019-07-09 00:00:00.000000000 Z
13
+ date: 2019-10-10 00:00:00.000000000 Z
14
14
  dependencies:
15
15
  - !ruby/object:Gem::Dependency
16
16
  name: bundler
@@ -94,14 +94,14 @@ dependencies:
94
94
  requirements:
95
95
  - - ">="
96
96
  - !ruby/object:Gem::Version
97
- version: '0'
97
+ version: 0.6.0
98
98
  type: :runtime
99
99
  prerelease: false
100
100
  version_requirements: !ruby/object:Gem::Requirement
101
101
  requirements:
102
102
  - - ">="
103
103
  - !ruby/object:Gem::Version
104
- version: '0'
104
+ version: 0.6.0
105
105
  - !ruby/object:Gem::Dependency
106
106
  name: nokogiri
107
107
  requirement: !ruby/object:Gem::Requirement
@@ -116,6 +116,20 @@ dependencies:
116
116
  - - "~>"
117
117
  - !ruby/object:Gem::Version
118
118
  version: '1.10'
119
+ - !ruby/object:Gem::Dependency
120
+ name: fluent-plugin-parser-winevt_xml
121
+ requirement: !ruby/object:Gem::Requirement
122
+ requirements:
123
+ - - ">="
124
+ - !ruby/object:Gem::Version
125
+ version: 0.1.2
126
+ type: :runtime
127
+ prerelease: false
128
+ version_requirements: !ruby/object:Gem::Requirement
129
+ requirements:
130
+ - - ">="
131
+ - !ruby/object:Gem::Version
132
+ version: 0.1.2
119
133
  description: Fluentd Input plugin to read windows event log.
120
134
  email:
121
135
  - naruki_okahashi@jbat.co.jp
@@ -135,13 +149,10 @@ files:
135
149
  - fluent-plugin-winevtlog.gemspec
136
150
  - lib/fluent/plugin/in_windows_eventlog.rb
137
151
  - lib/fluent/plugin/in_windows_eventlog2.rb
138
- - lib/fluent/plugin/parser_winevt_xml.rb
139
- - test/data/eventlog.xml
140
152
  - test/generate-windows-event.rb
141
153
  - test/helper.rb
142
154
  - test/plugin/test_in_windows_eventlog2.rb
143
155
  - test/plugin/test_in_winevtlog.rb
144
- - test/plugin/test_parser_winevt_xml.rb
145
156
  homepage: https://github.com/fluent/fluent-plugin-windows-eventlog
146
157
  licenses:
147
158
  - Apache-2.0
@@ -167,9 +178,7 @@ signing_key:
167
178
  specification_version: 4
168
179
  summary: Fluentd Input plugin to read windows event log.
169
180
  test_files:
170
- - test/data/eventlog.xml
171
181
  - test/generate-windows-event.rb
172
182
  - test/helper.rb
173
183
  - test/plugin/test_in_windows_eventlog2.rb
174
184
  - test/plugin/test_in_winevtlog.rb
175
- - test/plugin/test_parser_winevt_xml.rb
@@ -1,34 +0,0 @@
1
- require 'fluent/plugin/parser'
2
- require 'nokogiri'
3
-
4
- module Fluent::Plugin
5
- class WinevtXMLparser < Parser
6
- Fluent::Plugin.register_parser('winevt_xml', self)
7
-
8
- def parse(text)
9
- record = {}
10
- doc = Nokogiri::XML(text)
11
- system_elem = doc/'Event'/'System'
12
- record["ProviderName"] = (system_elem/"Provider").attribute("Name").text rescue nil
13
- record["ProviderGUID"] = (system_elem/"Provider").attribute("Guid").text rescue nil
14
- record["EventID"] = (system_elem/'EventID').text rescue nil
15
- record["Qualifiers"] = (system_elem/'EventID').attribute("Qualifiers").text rescue nil
16
- record["Level"] = (system_elem/'Level').text rescue nil
17
- record["Task"] = (system_elem/'Task').text rescue nil
18
- record["Opcode"] = (system_elem/'Opcode').text rescue nil
19
- record["Keywords"] = (system_elem/'Keywords').text rescue nil
20
- record["TimeCreated"] = (system_elem/'TimeCreated').attribute("SystemTime").text rescue nil
21
- record["EventRecordID"] = (system_elem/'EventRecordID').text rescue nil
22
- record["ActivityID"] = (system_elem/'ActivityID').text rescue nil
23
- record["RelatedActivityID"] = (system_elem/'Correlation').attribute("ActivityID").text rescue nil
24
- record["ThreadID"] = (system_elem/'Execution').attribute("ThreadID").text rescue nil
25
- record["Channel"] = (system_elem/'Channel').text rescue nil
26
- record["Computer"] = (system_elem/"Computer").text rescue nil
27
- record["UserID"] = (system_elem/'Security').attribute("UserID").text rescue nil
28
- record["Version"] = (system_elem/'Version').text rescue nil
29
- record["EventData"] = [] # These parameters are processed in winevt_c.
30
- time = @estimate_current_event ? Fluent::EventTime.now : nil
31
- yield time, record
32
- end
33
- end
34
- end
@@ -1 +0,0 @@
1
- <Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Security-Auditing' Guid='{54849625-5478-4994-A5BA-3E3B0328C30D}'/><EventID>4624</EventID><Version>2</Version><Level>0</Level><Task>12544</Task><Opcode>0</Opcode><Keywords>0x8020000000000000</Keywords><TimeCreated SystemTime='2019-06-13T09:21:23.345889600Z'/><EventRecordID>80688</EventRecordID><Correlation ActivityID='{587F0743-1F71-0006-5007-7F58711FD501}'/><Execution ProcessID='912' ThreadID='24708'/><Channel>Security</Channel><Computer>Fluentd-Developing-Windows</Computer><Security/></System><EventData><Data Name='SubjectUserSid'>S-1-5-18</Data><Data Name='SubjectUserName'>Fluentd-Developing-Windows$</Data><Data Name='SubjectDomainName'>WORKGROUP</Data><Data Name='SubjectLogonId'>0x3e7</Data><Data Name='TargetUserSid'>S-1-5-18</Data><Data Name='TargetUserName'>SYSTEM</Data><Data Name='TargetDomainName'>NT AUTHORITY</Data><Data Name='TargetLogonId'>0x3e7</Data><Data Name='LogonType'>5</Data><Data Name='LogonProcessName'>Advapi </Data><Data Name='AuthenticationPackageName'>Negotiate</Data><Data Name='WorkstationName'>-</Data><Data Name='LogonGuid'>{00000000-0000-0000-0000-000000000000}</Data><Data Name='TransmittedServices'>-</Data><Data Name='LmPackageName'>-</Data><Data Name='KeyLength'>0</Data><Data Name='ProcessId'>0x344</Data><Data Name='ProcessName'>C:\Windows\System32\services.exe</Data><Data Name='IpAddress'>-</Data><Data Name='IpPort'>-</Data><Data Name='ImpersonationLevel'>%%1833</Data><Data Name='RestrictedAdminMode'>-</Data><Data Name='TargetOutboundUserName'>-</Data><Data Name='TargetOutboundDomainName'>-</Data><Data Name='VirtualAccount'>%%1843</Data><Data Name='TargetLinkedLogonId'>0x0</Data><Data Name='ElevatedToken'>%%1842</Data></EventData></Event>
@@ -1,42 +0,0 @@
1
- require 'helper'
2
- require 'generate-windows-event'
3
-
4
- class WinevtXMLparserTest < Test::Unit::TestCase
5
-
6
- def setup
7
- Fluent::Test.setup
8
- end
9
-
10
- CONFIG = %[]
11
- XMLLOG = File.open(File.join(__dir__, "..", "data", "eventlog.xml") )
12
-
13
- def create_driver(conf = CONFIG)
14
- Fluent::Test::Driver::Parser.new(Fluent::Plugin::WinevtXMLparser).configure(conf)
15
- end
16
-
17
- def test_parse
18
- d = create_driver
19
- xml = XMLLOG
20
- expected = {"ProviderName" => "Microsoft-Windows-Security-Auditing",
21
- "ProviderGUID" => "{54849625-5478-4994-A5BA-3E3B0328C30D}",
22
- "EventID" => "4624",
23
- "Qualifiers" => nil,
24
- "Level" => "0",
25
- "Task" => "12544",
26
- "Opcode" => "0",
27
- "Keywords" => "0x8020000000000000",
28
- "TimeCreated" => "2019-06-13T09:21:23.345889600Z",
29
- "EventRecordID" => "80688",
30
- "ActivityID" => "",
31
- "RelatedActivityID" => "{587F0743-1F71-0006-5007-7F58711FD501}",
32
- "ThreadID" => "24708",
33
- "Channel" => "Security",
34
- "Computer" => "Fluentd-Developing-Windows",
35
- "UserID" => nil,
36
- "Version" => "2",
37
- "EventData" => []}
38
- d.instance.parse(xml) do |time, record|
39
- assert_equal(expected, record)
40
- end
41
- end
42
- end