fluent-plugin-windows-eventlog 0.2.2 → 0.4.3

Sign up to get free protection for your applications and to get access to all the features.
Files changed (16) hide show
  1. checksums.yaml +5 -5
  2. data/.gitignore +14 -14
  3. data/CHANGELOG.md +32 -0
  4. data/Gemfile +4 -4
  5. data/LICENSE.txt +203 -203
  6. data/README.md +285 -132
  7. data/Rakefile +10 -10
  8. data/appveyor.yml +24 -34
  9. data/fluent-plugin-winevtlog.gemspec +28 -25
  10. data/lib/fluent/plugin/in_windows_eventlog.rb +234 -234
  11. data/lib/fluent/plugin/in_windows_eventlog2.rb +242 -0
  12. data/test/generate-windows-event.rb +47 -47
  13. data/test/helper.rb +33 -32
  14. data/test/plugin/test_in_windows_eventlog2.rb +261 -0
  15. data/test/plugin/test_in_winevtlog.rb +48 -48
  16. metadata +50 -4
data/README.md CHANGED
@@ -1,132 +1,285 @@
1
- # fluent-plugin-windows-eventlog
2
-
3
- ## Component
4
-
5
- ### fluentd Input plugin for the Windows Event Log
6
-
7
- [Fluentd](http://fluentd.org) plugin to read the Windows Event Log.
8
-
9
- ## Installation
10
- gem install fluent-plugin-windows-eventlog
11
-
12
- ## Configuration
13
-
14
- ### fluentd Input plugin for the Windows Event Log
15
-
16
- <source>
17
- @type windows_eventlog
18
- @id windows_eventlog
19
- channels application,system
20
- read_interval 2
21
- tag winevt.raw
22
- <storage>
23
- @type local # @type local is the default.
24
- persistent true # default is true. Set to false to use in-memory storage.
25
- path ./tmp/storage.json # This is required when persistent is true.
26
- # Or, please consider using <system> section's `root_dir` parameter.
27
- </storage>
28
- </source>
29
-
30
- ### parameters
31
-
32
- |name | description |
33
- |:----- |:----- |
34
- |`channels` | (option) 'application' as default. One or more of {'application', 'system', 'setup', 'security'}. If you want to read 'setup' or 'security' logs, you must launch fluentd with administrator privileges.|
35
- |`keys` | (option) A subset of [keys](#read-keys) to read. Defaults to all keys.|
36
- |`read_interval` | (option) Read interval in seconds. 2 seconds as default.|
37
- |`from_encoding` | (option) Input character encoding. `nil` as default.|
38
- |`encoding` | (option) Output character encoding. `nil` as default.|
39
- |`read_from_head` | (option) Start to read the entries from the oldest, not from when fluentd is started. Defaults to `false`.|
40
- |`<storage>` | Setting for `storage` plugin for recording read position like `in_tail`'s `pos_file`.|
41
- |`parse_description`| (option) parse `description` field and set parsed result into the record. `parse` and `string_inserts` fields are removed|
42
-
43
- #### Available keys
44
-
45
- This plugin reads the following fields from Windows Event Log entries. Use the `keys` configuration option to select a subset. No other customization is allowed for now.
46
-
47
- |key|
48
- |:----- |
49
- |`record_number` |
50
- |`time_generated`|
51
- |`time_written` |
52
- |`event_id` |
53
- |`event_type` |
54
- |`event_category`|
55
- |`source_name` |
56
- |`computer_name` |
57
- |`user` |
58
- |`description` |
59
- |`string_inserts`|
60
-
61
- #### `parse_description` details
62
-
63
- Here is an example with `parse_description true`.
64
-
65
- ```
66
- {
67
- "channel": "security",
68
- "record_number": "91698",
69
- "time_generated": "2017-08-29 20:12:29 +0000",
70
- "time_written": "2017-08-29 20:12:29 +0000",
71
- "event_id": "4798",
72
- "event_type": "audit_success",
73
- "event_category": "13824",
74
- "source_name": "Microsoft-Windows-Security-Auditing",
75
- "computer_name": "TEST",
76
- "user": "",
77
- "description": "A user's local group membership was enumerated.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-XXX\r\n\tAccount Name:\t\tTEST$\r\n\tAccount Domain:\t\tWORKGROUP\r\n\tLogon ID:\t\t0x3e7\r\n\r\nUser:\r\n\tSecurity ID:\t\tS-XXX-YYY-ZZZ\r\n\tAccount Name:\t\tAdministrator\r\n\tAccount Domain:\t\tTEST\r\n\r\nProcess Information:\r\n\tProcess ID:\t\t0x7dc\r\n\tProcess Name:\t\tC:\\Windows\\System32\\LogonUI.exe\r\n",
78
- "string_inserts": [
79
- "Administrator",
80
- "TEST",
81
- "S-XXX-YYY-ZZZ",
82
- "S-XXX",
83
- "TEST$",
84
- "WORKGROUP",
85
- "0x3e7",
86
- "0x7dc",
87
- "C:\\Windows\\System32\\LogonUI.exe"
88
- ]
89
- }
90
- ```
91
-
92
- This record is transformed to
93
-
94
- ```
95
- {
96
- "channel": "security",
97
- "record_number": "91698",
98
- "time_generated": "2017-08-29 20:12:29 +0000",
99
- "time_written": "2017-08-29 20:12:29 +0000",
100
- "event_id": "4798",
101
- "event_type": "audit_success",
102
- "event_category": "13824",
103
- "source_name": "Microsoft-Windows-Security-Auditing",
104
- "computer_name": "TEST",
105
- "user": "",
106
- "description_title": "A user's local group membership was enumerated.",
107
- "subject.security_id": "S-XXX",
108
- "subject.account_name": "TEST$",
109
- "subject.account_domain": "WORKGROUP",
110
- "subject.logon_id": "0x3e7",
111
- "user.security_id": "S-XXX-YYY-ZZZ",
112
- "user.account_name": "Administrator",
113
- "user.account_domain": "TEST",
114
- "process_information.process_id": "0x7dc",
115
- "process_information.process_name": "C:\\Windows\\System32\\LogonUI.exe\r\n"
116
- }
117
- ```
118
-
119
- NOTE: This feature assumes `description` field has following formats:
120
-
121
- - group delimiter: `\r\n\r\n`
122
- - record delimiter: `\r\n\t`
123
- - field delimiter: `\t\t`
124
-
125
- If your `description` doesn't follow this format, the parsed result is only `description_title` field with same `description` content.
126
-
127
- ## Copyright
128
- ### Copyright
129
- Copyright(C) 2014- @okahashi117
130
- ### License
131
- Apache License, Version 2.0
132
-
1
+ # fluent-plugin-windows-eventlog
2
+
3
+ ## Component
4
+
5
+ ### fluentd Input plugin for the Windows Event Log
6
+
7
+ [Fluentd](https://www.fluentd.org/) plugin to read the Windows Event Log.
8
+
9
+ ## Installation
10
+ ridk exec gem install fluent-plugin-windows-eventlog
11
+
12
+ ## Configuration
13
+
14
+ ### in_windows_eventlog
15
+
16
+ Check [in_windows_eventlog2](https://github.com/fluent/fluent-plugin-windows-eventlog#in_windows_eventlog2) first. `in_windows_eventlog` will be replaced with `in_windows_eventlog2`.
17
+
18
+ fluentd Input plugin for the Windows Event Log using old Windows Event Logging API
19
+
20
+ <source>
21
+ @type windows_eventlog
22
+ @id windows_eventlog
23
+ channels application,system
24
+ read_interval 2
25
+ tag winevt.raw
26
+ <storage>
27
+ @type local # @type local is the default.
28
+ persistent true # default is true. Set to false to use in-memory storage.
29
+ path ./tmp/storage.json # This is required when persistent is true.
30
+ # Or, please consider using <system> section's `root_dir` parameter.
31
+ </storage>
32
+ </source>
33
+
34
+ #### parameters
35
+
36
+ |name | description |
37
+ |:----- |:----- |
38
+ |`channels` | (option) 'application' as default. One or more of {'application', 'system', 'setup', 'security'}. If you want to read 'setup' or 'security' logs, you must launch fluentd with administrator privileges.|
39
+ |`keys` | (option) A subset of [keys](#read-keys) to read. Defaults to all keys.|
40
+ |`read_interval` | (option) Read interval in seconds. 2 seconds as default.|
41
+ |`from_encoding` | (option) Input character encoding. `nil` as default.|
42
+ |`encoding` | (option) Output character encoding. `nil` as default.|
43
+ |`read_from_head` | (option) Start to read the entries from the oldest, not from when fluentd is started. Defaults to `false`.|
44
+ |`<storage>` | Setting for `storage` plugin for recording read position like `in_tail`'s `pos_file`.|
45
+ |`parse_description`| (option) parse `description` field and set parsed result into the record. `parse` and `string_inserts` fields are removed|
46
+
47
+ ##### Available keys
48
+
49
+ This plugin reads the following fields from Windows Event Log entries. Use the `keys` configuration option to select a subset. No other customization is allowed for now.
50
+
51
+ |key|
52
+ |:----- |
53
+ |`record_number` |
54
+ |`time_generated`|
55
+ |`time_written` |
56
+ |`event_id` |
57
+ |`event_type` |
58
+ |`event_category`|
59
+ |`source_name` |
60
+ |`computer_name` |
61
+ |`user` |
62
+ |`description` |
63
+ |`string_inserts`|
64
+
65
+ ##### `parse_description` details
66
+
67
+ Here is an example with `parse_description true`.
68
+
69
+ ```
70
+ {
71
+ "channel": "security",
72
+ "record_number": "91698",
73
+ "time_generated": "2017-08-29 20:12:29 +0000",
74
+ "time_written": "2017-08-29 20:12:29 +0000",
75
+ "event_id": "4798",
76
+ "event_type": "audit_success",
77
+ "event_category": "13824",
78
+ "source_name": "Microsoft-Windows-Security-Auditing",
79
+ "computer_name": "TEST",
80
+ "user": "",
81
+ "description": "A user's local group membership was enumerated.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-XXX\r\n\tAccount Name:\t\tTEST$\r\n\tAccount Domain:\t\tWORKGROUP\r\n\tLogon ID:\t\t0x3e7\r\n\r\nUser:\r\n\tSecurity ID:\t\tS-XXX-YYY-ZZZ\r\n\tAccount Name:\t\tAdministrator\r\n\tAccount Domain:\t\tTEST\r\n\r\nProcess Information:\r\n\tProcess ID:\t\t0x7dc\r\n\tProcess Name:\t\tC:\\Windows\\System32\\LogonUI.exe\r\n",
82
+ "string_inserts": [
83
+ "Administrator",
84
+ "TEST",
85
+ "S-XXX-YYY-ZZZ",
86
+ "S-XXX",
87
+ "TEST$",
88
+ "WORKGROUP",
89
+ "0x3e7",
90
+ "0x7dc",
91
+ "C:\\Windows\\System32\\LogonUI.exe"
92
+ ]
93
+ }
94
+ ```
95
+
96
+ This record is transformed to
97
+
98
+ ```
99
+ {
100
+ "channel": "security",
101
+ "record_number": "91698",
102
+ "time_generated": "2017-08-29 20:12:29 +0000",
103
+ "time_written": "2017-08-29 20:12:29 +0000",
104
+ "event_id": "4798",
105
+ "event_type": "audit_success",
106
+ "event_category": "13824",
107
+ "source_name": "Microsoft-Windows-Security-Auditing",
108
+ "computer_name": "TEST",
109
+ "user": "",
110
+ "description_title": "A user's local group membership was enumerated.",
111
+ "subject.security_id": "S-XXX",
112
+ "subject.account_name": "TEST$",
113
+ "subject.account_domain": "WORKGROUP",
114
+ "subject.logon_id": "0x3e7",
115
+ "user.security_id": "S-XXX-YYY-ZZZ",
116
+ "user.account_name": "Administrator",
117
+ "user.account_domain": "TEST",
118
+ "process_information.process_id": "0x7dc",
119
+ "process_information.process_name": "C:\\Windows\\System32\\LogonUI.exe\r\n"
120
+ }
121
+ ```
122
+
123
+ NOTE: This feature assumes `description` field has following formats:
124
+
125
+ - group delimiter: `\r\n\r\n`
126
+ - record delimiter: `\r\n\t`
127
+ - field delimiter: `\t\t`
128
+
129
+ If your `description` doesn't follow this format, the parsed result is only `description_title` field with same `description` content.
130
+
131
+ ### in_windows_eventlog2
132
+
133
+ fluentd Input plugin for the Windows Event Log using newer Windows Event Logging API. This is successor to `in_windows_eventlog`. See also [this slide](https://www.slideshare.net/cosmo0920/fluentd-meetup-2019) for the details of `in_windows_eventlog2` plugin.
134
+
135
+ <source>
136
+ @type windows_eventlog2
137
+ @id windows_eventlog2
138
+ channels application,system
139
+ read_interval 2
140
+ tag winevt.raw
141
+ render_as_xml false # default is true.
142
+ rate_limit 200 # default is -1(Winevt::EventLog::Subscribe::RATE_INFINITE).
143
+ <storage>
144
+ @type local # @type local is the default.
145
+ persistent true # default is true. Set to false to use in-memory storage.
146
+ path ./tmp/storage.json # This is required when persistent is true.
147
+ # Or, please consider using <system> section's `root_dir` parameter.
148
+ </storage>
149
+ <parse>
150
+ @type winevt_xml # @type winevt_xml is the default. winevt_xml and none parsers are supported for now.
151
+ </parse>
152
+ </source>
153
+
154
+ **NOTE:** in_windows_eventlog2 always handles EventLog records as UTF-8 characters. Users don't have to specify encoding related parameters and they are not provided.
155
+
156
+ **NOTE:** When `Description` contains error message such as `The message resource is present but the message was not found in the message table.`, eventlog's resource file (.mui) related to error generating event is something wrong. This issue is also occurred in built-in Windows Event Viewer which is the part of Windows management tool.
157
+
158
+ **NOTE:** When `render_as_xml` as `false`, the dependent winevt_c gem renders Windows EventLog as Ruby Hash object directly. This reduces bottleneck to consume EventLog. Specifying `render_as_xml` as `false` should be faster consuming than `render_as_xml` as `true` case.
159
+
160
+ **NOTE:** If you encountered CPU spike due to massively huge EventLog channel, `rate_limit` parameter may help you. Currently, this paramter can handle the multiples of 10 or -1(`Winevt::EventLog::Subscribe::RATE_INFINITE`).
161
+
162
+ #### parameters
163
+
164
+ |name | description |
165
+ |:----- |:----- |
166
+ |`channels` | (option) 'application' as default. One or more of {'application', 'system', 'setup', 'security'}. If you want to read 'setup' or 'security' logs, you must launch fluentd with administrator privileges.|
167
+ |`keys` | (option) A subset of [keys](#read-keys) to read. Defaults to all keys.|
168
+ |`read_interval` | (option) Read interval in seconds. 2 seconds as default.|
169
+ |`from_encoding` | (option) Input character encoding. `nil` as default.|
170
+ |`<storage>` | Setting for `storage` plugin for recording read position like `in_tail`'s `pos_file`.|
171
+ |`<parse>` | Setting for `parser` plugin for parsing raw XML EventLog records. |
172
+ |`parse_description`| (option) parse `description` field and set parsed result into the record. `Description` and `EventData` fields are removed|
173
+
174
+ ##### Available keys
175
+
176
+ This plugin reads the following fields from Windows Event Log entries. Use the `keys` configuration option to select a subset. No other customization is allowed for now.
177
+
178
+ |key|
179
+ |:----- |
180
+ |`ProviderName` |
181
+ |`ProviderGuid` |
182
+ |`EventID` |
183
+ |`Qualifiers` |
184
+ |`Level` |
185
+ |`Task` |
186
+ |`Opcode` |
187
+ |`Keywords` |
188
+ |`TimeCreated` |
189
+ |`EventRecordId` |
190
+ |`ActivityID` |
191
+ |`RelatedActivityID`|
192
+ |`ProcessID` |
193
+ |`ThreadID` |
194
+ |`Channel` |
195
+ |`Computer` |
196
+ |`UserID` |
197
+ |`Version` |
198
+ |`Description` |
199
+ |`EventData` |
200
+
201
+ ##### `parse_description` details
202
+
203
+ Here is an example with `parse_description true`.
204
+
205
+ ```
206
+ {
207
+ "ProviderName": "Microsoft-Windows-Security-Auditing",
208
+ "ProviderGUID": "{D441060A-9695-472B-90BC-24DCA9D503A4}",
209
+ "EventID": "4798",
210
+ "Qualifiers": "",
211
+ "Level": "0",
212
+ "Task": "13824",
213
+ "Opcode": "0",
214
+ "Keywords": "0x8020000000000000",
215
+ "TimeCreated": "2019-06-19T03:10:01.982940200Z",
216
+ "EventRecordID": "87028",
217
+ "ActivityID": "",
218
+ "RelatedActivityID": "{2599DE71-2F70-44AD-9DC8-C5FF2AE8D1EF}",
219
+ "ThreadID": "16888",
220
+ "Channel": "Security",
221
+ "Computer": "DESKTOP-TEST",
222
+ "UserID": "",
223
+ "Version": "0",
224
+ "Description": "A user's local group membership was enumerated.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-X-Y-Z\r\n\tAccount Name:\t\tDESKTOP-TEST$\r\n\tAccount Domain:\t\tWORKGROUP\r\n\tLogon ID:\t\t0x3e7\r\n\r\nUser:\r\n\tSecurity ID:\t\tS-XXX-YYY-ZZZ0\r\n\tAccount Name:\t\tAdministrator\r\n\tAccount Domain:\t\tDESKTOP-TEST\r\n\r\nProcess Information:\r\n\tProcess ID:\t\t0xbac\r\n\tProcess Name:\t\tC:\\Windows\\System32\\svchost.exe\r\n",
225
+ "EventData": [
226
+ "Administrator",
227
+ "DESKTOP-TEST",
228
+ "S-XXX-YYY-ZZZ",
229
+ "S-X-Y-Z",
230
+ "DESKTOP-TEST$",
231
+ "WORKGROUP",
232
+ "0x3e7",
233
+ "0xbac",
234
+ "C:\\Windows\\System32\\svchost.exe"
235
+ ]
236
+ }
237
+ ```
238
+
239
+ This record is transformed to
240
+
241
+ ```
242
+ {
243
+ "ProviderName": "Microsoft-Windows-Security-Auditing",
244
+ "ProviderGUID": "{D441060A-9695-472B-90BC-24DCA9D503A4}",
245
+ "EventID": "4798",
246
+ "Qualifiers": "",
247
+ "Level": "0",
248
+ "Task": "13824",
249
+ "Opcode": "0",
250
+ "Keywords": "0x8020000000000000",
251
+ "TimeCreated": "2019-06-19T03:10:01.982940200Z",
252
+ "EventRecordID": "87028",
253
+ "ActivityID": "",
254
+ "RelatedActivityID": "{2599DE71-2F70-44AD-9DC8-C5FF2AE8D1EF}",
255
+ "ThreadID": "16888",
256
+ "Channel": "Security",
257
+ "Computer": "DESKTOP-TEST",
258
+ "UserID": "",
259
+ "Version": "0",
260
+ "DescriptionTitle": "A user's local group membership was enumerated.",
261
+ "subject.security_id": "S-X-Y-Z",
262
+ "subject.account_name": "DESKTOP-TEST$",
263
+ "subject.account_domain": "WORKGROUP",
264
+ "subject.logon_id": "0x3e7",
265
+ "user.security_id": "S-XXX-YYY-ZZZ",
266
+ "user.account_name": "Administrator",
267
+ "user.account_domain": "DESKTOP-TEST",
268
+ "process_information.process_id": "0xbac",
269
+ "process_information.process_name": "C:\\Windows\\System32\\svchost.exe"
270
+ }
271
+ ```
272
+
273
+ NOTE: This feature assumes `description` field has following formats:
274
+
275
+ - group delimiter: `\r\n\r\n`
276
+ - record delimiter: `\r\n\t`
277
+ - field delimiter: `\t\t`
278
+
279
+ If your `description` doesn't follow this format, the parsed result is only `description_title` field with same `description` content.
280
+
281
+ ## Copyright
282
+ ### Copyright
283
+ Copyright(C) 2014- @okahashi117
284
+ ### License
285
+ Apache License, Version 2.0
data/Rakefile CHANGED
@@ -1,10 +1,10 @@
1
- require "bundler/gem_tasks"
2
- require "rake/testtask"
3
-
4
- Rake::TestTask.new(:test) do |test|
5
- test.libs << 'lib' << 'test'
6
- test.pattern = 'test/**/test_*.rb'
7
- test.verbose = true
8
- end
9
-
10
- task default: :test
1
+ require "bundler/gem_tasks"
2
+ require "rake/testtask"
3
+
4
+ Rake::TestTask.new(:test) do |test|
5
+ test.libs << 'lib' << 'test'
6
+ test.pattern = 'test/**/test_*.rb'
7
+ test.verbose = true
8
+ end
9
+
10
+ task default: :test
data/appveyor.yml CHANGED
@@ -1,34 +1,24 @@
1
- version: '{build}'
2
-
3
- # init:
4
- # - ps: iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/enable-rdp.ps1'))
5
-
6
- install:
7
- - SET PATH=C:\Ruby%ruby_version%\bin;%PATH%
8
- - IF %ridk%==0 "%devkit%\\devkitvars.bat"
9
- - ruby --version
10
- - gem --version
11
- - IF %ridk%==1 ridk.cmd exec bundle install
12
- - IF %ridk%==0 bundle install
13
- build: off
14
- test_script:
15
- - bundle exec rake test
16
- # - bundle exec rake test TESTOPTS=-v
17
-
18
- branches:
19
- only:
20
- - master
21
-
22
- # https://www.appveyor.com/docs/installed-software/#ruby
23
- environment:
24
- matrix:
25
- - ruby_version: "23-x64"
26
- devkit: C:\Ruby23-x64\DevKit
27
- ridk: 0
28
- - ruby_version: "23"
29
- devkit: C:\Ruby23\DevKit
30
- ridk: 0
31
- - ruby_version: "24-x64"
32
- ridk: 1
33
- - ruby_version: "24"
34
- ridk: 1
1
+ version: '{build}'
2
+
3
+ # init:
4
+ # - ps: iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/enable-rdp.ps1'))
5
+
6
+ install:
7
+ - SET PATH=C:\Ruby%ruby_version%\bin;%PATH%
8
+ - ruby --version
9
+ - gem --version
10
+ - ridk.cmd exec bundle install
11
+ build: off
12
+ test_script:
13
+ - bundle exec rake test
14
+ # - bundle exec rake test TESTOPTS=-v
15
+
16
+ branches:
17
+ only:
18
+ - master
19
+
20
+ # https://www.appveyor.com/docs/installed-software/#ruby
21
+ environment:
22
+ matrix:
23
+ - ruby_version: "24-x64"
24
+ - ruby_version: "24"