logstash-codec-cef 5.0.6-java → 6.1.2-java
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +19 -0
- data/LICENSE +199 -10
- data/README.md +1 -1
- data/docs/index.asciidoc +7 -21
- data/lib/logstash/codecs/cef.rb +65 -57
- data/logstash-codec-cef.gemspec +2 -1
- data/spec/codecs/cef_spec.rb +30 -5
- metadata +16 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: b911dab6fca1f37b8edd22e8ea97759674df3301bb0afafacf14343c84237088
|
|
4
|
+
data.tar.gz: ebad9548b63d7a5f6f3bc92e74166700599b75e7425df1fd471cfa111399b098
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: b2cea62a7689f4d15791338167a5f105a5770121a0069f52607e28bb7fb1e7081eb039d4916d78dbe18f288bdc4c49a11cb69945e1721709ae5b0905c1497c0c
|
|
7
|
+
data.tar.gz: 648f0c5fd204ddc940f8d7d53cfd3bbd4d7493b607dc78058b5a11bfc5cdbeda898cd6860743be84af7c1ea5260327983300b1bbaecd314901a923c4c489acfe
|
data/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,22 @@
|
|
|
1
|
+
## 6.1.2
|
|
2
|
+
- Added error log with full payload when something bad happens in decoding a message[#84](https://github.com/logstash-plugins/logstash-codec-cef/pull/84)
|
|
3
|
+
|
|
4
|
+
## 6.1.1
|
|
5
|
+
- Improved encoding performance, especially when encoding many extension fields [#81](https://github.com/logstash-plugins/logstash-codec-cef/pull/81)
|
|
6
|
+
|
|
7
|
+
## 6.1.0
|
|
8
|
+
- Fixed CEF short to long name translation for ahost/agentHostName field, according to documentation [#75](https://github.com/logstash-plugins/logstash-codec-cef/pull/75)
|
|
9
|
+
|
|
10
|
+
## 6.0.1
|
|
11
|
+
- Fixed support for deep dot notation [#73](https://github.com/logstash-plugins/logstash-codec-cef/pull/73)
|
|
12
|
+
|
|
13
|
+
## 6.0.0
|
|
14
|
+
- Removed obsolete `sev` and `deprecated_v1_fields` fields
|
|
15
|
+
|
|
16
|
+
## 5.0.7
|
|
17
|
+
- Fixed minor doc inconsistencies (added reverse_mapping to options table, moved it to alpha order in option descriptions, fixed typo)
|
|
18
|
+
[#60](https://github.com/logstash-plugins/logstash-codec-cef/pull/60)
|
|
19
|
+
|
|
1
20
|
## 5.0.6
|
|
2
21
|
- Added reverse_mapping option, which can be used to make encoder compliant to spec [#51](https://github.com/logstash-plugins/logstash-codec-cef/pull/51)
|
|
3
22
|
|
data/LICENSE
CHANGED
|
@@ -1,13 +1,202 @@
|
|
|
1
|
-
Copyright (c) 2012-2018 Elasticsearch <http://www.elastic.co>
|
|
2
1
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
2
|
+
Apache License
|
|
3
|
+
Version 2.0, January 2004
|
|
4
|
+
http://www.apache.org/licenses/
|
|
6
5
|
|
|
7
|
-
|
|
6
|
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
8
7
|
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
8
|
+
1. Definitions.
|
|
9
|
+
|
|
10
|
+
"License" shall mean the terms and conditions for use, reproduction,
|
|
11
|
+
and distribution as defined by Sections 1 through 9 of this document.
|
|
12
|
+
|
|
13
|
+
"Licensor" shall mean the copyright owner or entity authorized by
|
|
14
|
+
the copyright owner that is granting the License.
|
|
15
|
+
|
|
16
|
+
"Legal Entity" shall mean the union of the acting entity and all
|
|
17
|
+
other entities that control, are controlled by, or are under common
|
|
18
|
+
control with that entity. For the purposes of this definition,
|
|
19
|
+
"control" means (i) the power, direct or indirect, to cause the
|
|
20
|
+
direction or management of such entity, whether by contract or
|
|
21
|
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
22
|
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
23
|
+
|
|
24
|
+
"You" (or "Your") shall mean an individual or Legal Entity
|
|
25
|
+
exercising permissions granted by this License.
|
|
26
|
+
|
|
27
|
+
"Source" form shall mean the preferred form for making modifications,
|
|
28
|
+
including but not limited to software source code, documentation
|
|
29
|
+
source, and configuration files.
|
|
30
|
+
|
|
31
|
+
"Object" form shall mean any form resulting from mechanical
|
|
32
|
+
transformation or translation of a Source form, including but
|
|
33
|
+
not limited to compiled object code, generated documentation,
|
|
34
|
+
and conversions to other media types.
|
|
35
|
+
|
|
36
|
+
"Work" shall mean the work of authorship, whether in Source or
|
|
37
|
+
Object form, made available under the License, as indicated by a
|
|
38
|
+
copyright notice that is included in or attached to the work
|
|
39
|
+
(an example is provided in the Appendix below).
|
|
40
|
+
|
|
41
|
+
"Derivative Works" shall mean any work, whether in Source or Object
|
|
42
|
+
form, that is based on (or derived from) the Work and for which the
|
|
43
|
+
editorial revisions, annotations, elaborations, or other modifications
|
|
44
|
+
represent, as a whole, an original work of authorship. For the purposes
|
|
45
|
+
of this License, Derivative Works shall not include works that remain
|
|
46
|
+
separable from, or merely link (or bind by name) to the interfaces of,
|
|
47
|
+
the Work and Derivative Works thereof.
|
|
48
|
+
|
|
49
|
+
"Contribution" shall mean any work of authorship, including
|
|
50
|
+
the original version of the Work and any modifications or additions
|
|
51
|
+
to that Work or Derivative Works thereof, that is intentionally
|
|
52
|
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
53
|
+
or by an individual or Legal Entity authorized to submit on behalf of
|
|
54
|
+
the copyright owner. For the purposes of this definition, "submitted"
|
|
55
|
+
means any form of electronic, verbal, or written communication sent
|
|
56
|
+
to the Licensor or its representatives, including but not limited to
|
|
57
|
+
communication on electronic mailing lists, source code control systems,
|
|
58
|
+
and issue tracking systems that are managed by, or on behalf of, the
|
|
59
|
+
Licensor for the purpose of discussing and improving the Work, but
|
|
60
|
+
excluding communication that is conspicuously marked or otherwise
|
|
61
|
+
designated in writing by the copyright owner as "Not a Contribution."
|
|
62
|
+
|
|
63
|
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
64
|
+
on behalf of whom a Contribution has been received by Licensor and
|
|
65
|
+
subsequently incorporated within the Work.
|
|
66
|
+
|
|
67
|
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
68
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
69
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
70
|
+
copyright license to reproduce, prepare Derivative Works of,
|
|
71
|
+
publicly display, publicly perform, sublicense, and distribute the
|
|
72
|
+
Work and such Derivative Works in Source or Object form.
|
|
73
|
+
|
|
74
|
+
3. Grant of Patent License. Subject to the terms and conditions of
|
|
75
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
76
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
77
|
+
(except as stated in this section) patent license to make, have made,
|
|
78
|
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
79
|
+
where such license applies only to those patent claims licensable
|
|
80
|
+
by such Contributor that are necessarily infringed by their
|
|
81
|
+
Contribution(s) alone or by combination of their Contribution(s)
|
|
82
|
+
with the Work to which such Contribution(s) was submitted. If You
|
|
83
|
+
institute patent litigation against any entity (including a
|
|
84
|
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
85
|
+
or a Contribution incorporated within the Work constitutes direct
|
|
86
|
+
or contributory patent infringement, then any patent licenses
|
|
87
|
+
granted to You under this License for that Work shall terminate
|
|
88
|
+
as of the date such litigation is filed.
|
|
89
|
+
|
|
90
|
+
4. Redistribution. You may reproduce and distribute copies of the
|
|
91
|
+
Work or Derivative Works thereof in any medium, with or without
|
|
92
|
+
modifications, and in Source or Object form, provided that You
|
|
93
|
+
meet the following conditions:
|
|
94
|
+
|
|
95
|
+
(a) You must give any other recipients of the Work or
|
|
96
|
+
Derivative Works a copy of this License; and
|
|
97
|
+
|
|
98
|
+
(b) You must cause any modified files to carry prominent notices
|
|
99
|
+
stating that You changed the files; and
|
|
100
|
+
|
|
101
|
+
(c) You must retain, in the Source form of any Derivative Works
|
|
102
|
+
that You distribute, all copyright, patent, trademark, and
|
|
103
|
+
attribution notices from the Source form of the Work,
|
|
104
|
+
excluding those notices that do not pertain to any part of
|
|
105
|
+
the Derivative Works; and
|
|
106
|
+
|
|
107
|
+
(d) If the Work includes a "NOTICE" text file as part of its
|
|
108
|
+
distribution, then any Derivative Works that You distribute must
|
|
109
|
+
include a readable copy of the attribution notices contained
|
|
110
|
+
within such NOTICE file, excluding those notices that do not
|
|
111
|
+
pertain to any part of the Derivative Works, in at least one
|
|
112
|
+
of the following places: within a NOTICE text file distributed
|
|
113
|
+
as part of the Derivative Works; within the Source form or
|
|
114
|
+
documentation, if provided along with the Derivative Works; or,
|
|
115
|
+
within a display generated by the Derivative Works, if and
|
|
116
|
+
wherever such third-party notices normally appear. The contents
|
|
117
|
+
of the NOTICE file are for informational purposes only and
|
|
118
|
+
do not modify the License. You may add Your own attribution
|
|
119
|
+
notices within Derivative Works that You distribute, alongside
|
|
120
|
+
or as an addendum to the NOTICE text from the Work, provided
|
|
121
|
+
that such additional attribution notices cannot be construed
|
|
122
|
+
as modifying the License.
|
|
123
|
+
|
|
124
|
+
You may add Your own copyright statement to Your modifications and
|
|
125
|
+
may provide additional or different license terms and conditions
|
|
126
|
+
for use, reproduction, or distribution of Your modifications, or
|
|
127
|
+
for any such Derivative Works as a whole, provided Your use,
|
|
128
|
+
reproduction, and distribution of the Work otherwise complies with
|
|
129
|
+
the conditions stated in this License.
|
|
130
|
+
|
|
131
|
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
132
|
+
any Contribution intentionally submitted for inclusion in the Work
|
|
133
|
+
by You to the Licensor shall be under the terms and conditions of
|
|
134
|
+
this License, without any additional terms or conditions.
|
|
135
|
+
Notwithstanding the above, nothing herein shall supersede or modify
|
|
136
|
+
the terms of any separate license agreement you may have executed
|
|
137
|
+
with Licensor regarding such Contributions.
|
|
138
|
+
|
|
139
|
+
6. Trademarks. This License does not grant permission to use the trade
|
|
140
|
+
names, trademarks, service marks, or product names of the Licensor,
|
|
141
|
+
except as required for reasonable and customary use in describing the
|
|
142
|
+
origin of the Work and reproducing the content of the NOTICE file.
|
|
143
|
+
|
|
144
|
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
145
|
+
agreed to in writing, Licensor provides the Work (and each
|
|
146
|
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
147
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
148
|
+
implied, including, without limitation, any warranties or conditions
|
|
149
|
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
150
|
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
151
|
+
appropriateness of using or redistributing the Work and assume any
|
|
152
|
+
risks associated with Your exercise of permissions under this License.
|
|
153
|
+
|
|
154
|
+
8. Limitation of Liability. In no event and under no legal theory,
|
|
155
|
+
whether in tort (including negligence), contract, or otherwise,
|
|
156
|
+
unless required by applicable law (such as deliberate and grossly
|
|
157
|
+
negligent acts) or agreed to in writing, shall any Contributor be
|
|
158
|
+
liable to You for damages, including any direct, indirect, special,
|
|
159
|
+
incidental, or consequential damages of any character arising as a
|
|
160
|
+
result of this License or out of the use or inability to use the
|
|
161
|
+
Work (including but not limited to damages for loss of goodwill,
|
|
162
|
+
work stoppage, computer failure or malfunction, or any and all
|
|
163
|
+
other commercial damages or losses), even if such Contributor
|
|
164
|
+
has been advised of the possibility of such damages.
|
|
165
|
+
|
|
166
|
+
9. Accepting Warranty or Additional Liability. While redistributing
|
|
167
|
+
the Work or Derivative Works thereof, You may choose to offer,
|
|
168
|
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
169
|
+
or other liability obligations and/or rights consistent with this
|
|
170
|
+
License. However, in accepting such obligations, You may act only
|
|
171
|
+
on Your own behalf and on Your sole responsibility, not on behalf
|
|
172
|
+
of any other Contributor, and only if You agree to indemnify,
|
|
173
|
+
defend, and hold each Contributor harmless for any liability
|
|
174
|
+
incurred by, or claims asserted against, such Contributor by reason
|
|
175
|
+
of your accepting any such warranty or additional liability.
|
|
176
|
+
|
|
177
|
+
END OF TERMS AND CONDITIONS
|
|
178
|
+
|
|
179
|
+
APPENDIX: How to apply the Apache License to your work.
|
|
180
|
+
|
|
181
|
+
To apply the Apache License to your work, attach the following
|
|
182
|
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
183
|
+
replaced with your own identifying information. (Don't include
|
|
184
|
+
the brackets!) The text should be enclosed in the appropriate
|
|
185
|
+
comment syntax for the file format. We also recommend that a
|
|
186
|
+
file or class name and description of purpose be included on the
|
|
187
|
+
same "printed page" as the copyright notice for easier
|
|
188
|
+
identification within third-party archives.
|
|
189
|
+
|
|
190
|
+
Copyright 2020 Elastic and contributors
|
|
191
|
+
|
|
192
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
|
193
|
+
you may not use this file except in compliance with the License.
|
|
194
|
+
You may obtain a copy of the License at
|
|
195
|
+
|
|
196
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
|
197
|
+
|
|
198
|
+
Unless required by applicable law or agreed to in writing, software
|
|
199
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
|
200
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
201
|
+
See the License for the specific language governing permissions and
|
|
202
|
+
limitations under the License.
|
data/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Logstash Plugin
|
|
2
2
|
|
|
3
|
-
[](https://travis-ci.com/logstash-plugins/logstash-codec-cef)
|
|
4
4
|
|
|
5
5
|
This is a plugin for [Logstash](https://github.com/elastic/logstash).
|
|
6
6
|
|
data/docs/index.asciidoc
CHANGED
|
@@ -37,6 +37,7 @@ produce an event with the payload as the 'message' field and a '_cefparsefailure
|
|
|
37
37
|
| <<plugins-{type}s-{plugin}-fields>> |<<array,array>>|No
|
|
38
38
|
| <<plugins-{type}s-{plugin}-name>> |<<string,string>>|No
|
|
39
39
|
| <<plugins-{type}s-{plugin}-product>> |<<string,string>>|No
|
|
40
|
+
| <<plugins-{type}s-{plugin}-reverse_mapping>> |<<boolean,boolean>>|No
|
|
40
41
|
| <<plugins-{type}s-{plugin}-severity>> |<<string,string>>|No
|
|
41
42
|
| <<plugins-{type}s-{plugin}-signature>> |<<string,string>>|No
|
|
42
43
|
| <<plugins-{type}s-{plugin}-vendor>> |<<string,string>>|No
|
|
@@ -68,21 +69,6 @@ This setting allows the following character sequences to have special meaning:
|
|
|
68
69
|
* `\\r` (backslash "r") - means carriage return (ASCII 0x0D)
|
|
69
70
|
* `\\n` (backslash "n") - means newline (ASCII 0x0A)
|
|
70
71
|
|
|
71
|
-
[id="plugins-{type}s-{plugin}-deprecated_v1_fields"]
|
|
72
|
-
===== `deprecated_v1_fields` (OBSOLETE)
|
|
73
|
-
|
|
74
|
-
* OBSOLETE WARNING: This configuration item is obsolete and will prevent the pipeline from starting if used
|
|
75
|
-
* Value type is <<boolean,boolean>>
|
|
76
|
-
* There is no default value for this setting.
|
|
77
|
-
|
|
78
|
-
[id="plugins-{type}s-{plugin}-reverse_mapping"]
|
|
79
|
-
===== `reverse_mapping`
|
|
80
|
-
|
|
81
|
-
* Value type is <<<boolean,boolean>>
|
|
82
|
-
* Default value is `false`
|
|
83
|
-
|
|
84
|
-
Set to true to adhere to the specifications and encode using the CEF key name (short name) for the CEF field names.
|
|
85
|
-
|
|
86
72
|
[id="plugins-{type}s-{plugin}-fields"]
|
|
87
73
|
===== `fields`
|
|
88
74
|
|
|
@@ -109,14 +95,14 @@ to help you build a new value from other parts of the event.
|
|
|
109
95
|
Device product field in CEF header. The new value can include `%{foo}` strings
|
|
110
96
|
to help you build a new value from other parts of the event.
|
|
111
97
|
|
|
112
|
-
[id="plugins-{type}s-{plugin}-sev"]
|
|
113
|
-
===== `sev` (OBSOLETE)
|
|
114
98
|
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
* There is no default value for this setting.
|
|
99
|
+
[id="plugins-{type}s-{plugin}-reverse_mapping"]
|
|
100
|
+
===== `reverse_mapping`
|
|
118
101
|
|
|
119
|
-
|
|
102
|
+
* Value type is <<boolean,boolean>>
|
|
103
|
+
* Default value is `false`
|
|
104
|
+
|
|
105
|
+
Set to true to adhere to the specifications and encode using the CEF key name (short name) for the CEF field names.
|
|
120
106
|
|
|
121
107
|
[id="plugins-{type}s-{plugin}-severity"]
|
|
122
108
|
===== `severity`
|
data/lib/logstash/codecs/cef.rb
CHANGED
|
@@ -33,9 +33,6 @@ class LogStash::Codecs::CEF < LogStash::Codecs::Base
|
|
|
33
33
|
# to help you build a new value from other parts of the event.
|
|
34
34
|
config :name, :validate => :string, :default => "Logstash"
|
|
35
35
|
|
|
36
|
-
# Obsolete severity field for CEF header
|
|
37
|
-
config :sev, :validate => :string, :obsolete => "This setting is obsolete, use :severity instead."
|
|
38
|
-
|
|
39
36
|
# Severity field in CEF header. The new value can include `%{foo}` strings
|
|
40
37
|
# to help you build a new value from other parts of the event.
|
|
41
38
|
#
|
|
@@ -53,8 +50,6 @@ class LogStash::Codecs::CEF < LogStash::Codecs::Base
|
|
|
53
50
|
# version of the CEF field names.
|
|
54
51
|
config :reverse_mapping, :validate => :boolean, :default => false
|
|
55
52
|
|
|
56
|
-
config :deprecated_v1_fields, :validate => :boolean, :obsolete => "This setting is obsolete"
|
|
57
|
-
|
|
58
53
|
# If your input puts a delimiter between each CEF event, you'll want to set
|
|
59
54
|
# this to be that delimiter.
|
|
60
55
|
#
|
|
@@ -153,7 +148,7 @@ class LogStash::Codecs::CEF < LogStash::Codecs::Base
|
|
|
153
148
|
"start" => "startTime",
|
|
154
149
|
"suid" => "sourceUserId",
|
|
155
150
|
"suser" => "sourceUserName",
|
|
156
|
-
"ahost" => "
|
|
151
|
+
"ahost" => "agentHostName",
|
|
157
152
|
"art" => "agentReceiptTime",
|
|
158
153
|
"at" => "agentType",
|
|
159
154
|
"aid" => "agentId",
|
|
@@ -194,13 +189,16 @@ class LogStash::Codecs::CEF < LogStash::Codecs::Base
|
|
|
194
189
|
# commas, periods, and square-bracketed index offsets.
|
|
195
190
|
#
|
|
196
191
|
# To support this, we look for a specific sequence of characters that are followed by an equals sign. This pattern
|
|
197
|
-
# will correctly identify all strictly-legal keys, and will also match those that include a dot "
|
|
192
|
+
# will correctly identify all strictly-legal keys, and will also match those that include a dot-joined "subkeys" and
|
|
193
|
+
# square-bracketed array indexing
|
|
198
194
|
#
|
|
199
195
|
# That sequence must begin with one or more `\w` (word: alphanumeric + underscore), which _optionally_ may be followed
|
|
200
|
-
# by "subkey"
|
|
201
|
-
#
|
|
202
|
-
#
|
|
203
|
-
|
|
196
|
+
# by one or more "subkey" sequences and an optional square-bracketed index.
|
|
197
|
+
#
|
|
198
|
+
# To be understood by this implementation, a "subkey" sequence must consist of a literal dot (`.`) followed by one or
|
|
199
|
+
# more characters that do not convey semantic meaning within CEF (e.g., literal-dot (`.`), literal-equals (`=`),
|
|
200
|
+
# whitespace (`\s`), literal-pipe (`|`), literal-backslash ('\'), or literal-square brackets (`[` or `]`)).
|
|
201
|
+
EXTENSION_KEY_PATTERN = /(?:\w+(?:\.[^\.=\s\|\\\[\]]+)*(?:\[[0-9]+\])?(?==))/
|
|
204
202
|
|
|
205
203
|
# Some CEF extension keys seen in the wild use an undocumented array-like syntax that may not be compatible with
|
|
206
204
|
# the Event API's strict-mode FieldReference parser (e.g., `fieldname[0]`).
|
|
@@ -217,6 +215,30 @@ class LogStash::Codecs::CEF < LogStash::Codecs::Base
|
|
|
217
215
|
# Cache of a scanner pattern that _captures_ extension field key/value pairs
|
|
218
216
|
EXTENSION_KEY_VALUE_SCANNER = /(#{EXTENSION_KEY_PATTERN})=(#{EXTENSION_VALUE_PATTERN})\s*/
|
|
219
217
|
|
|
218
|
+
##
|
|
219
|
+
# @see CEF#sanitize_header_field
|
|
220
|
+
HEADER_FIELD_SANITIZER_MAPPING = {
|
|
221
|
+
"\\" => "\\\\",
|
|
222
|
+
"|" => "\\|",
|
|
223
|
+
"\n" => " ",
|
|
224
|
+
"\r" => " ",
|
|
225
|
+
}
|
|
226
|
+
HEADER_FIELD_SANITIZER_PATTERN = Regexp.union(HEADER_FIELD_SANITIZER_MAPPING.keys)
|
|
227
|
+
private_constant :HEADER_FIELD_SANITIZER_MAPPING, :HEADER_FIELD_SANITIZER_PATTERN
|
|
228
|
+
|
|
229
|
+
##
|
|
230
|
+
# @see CEF#sanitize_extension_val
|
|
231
|
+
EXTENSION_VALUE_SANITIZER_MAPPING = {
|
|
232
|
+
"\\" => "\\\\",
|
|
233
|
+
"=" => "\\=",
|
|
234
|
+
"\n" => "\\n",
|
|
235
|
+
"\r" => "\\n",
|
|
236
|
+
}
|
|
237
|
+
EXTENSION_VALUE_SANITIZER_PATTERN = Regexp.union(EXTENSION_VALUE_SANITIZER_MAPPING.keys)
|
|
238
|
+
private_constant :EXTENSION_VALUE_SANITIZER_MAPPING, :EXTENSION_VALUE_SANITIZER_PATTERN
|
|
239
|
+
|
|
240
|
+
CEF_PREFIX = 'CEF:'.freeze
|
|
241
|
+
|
|
220
242
|
public
|
|
221
243
|
def initialize(params={})
|
|
222
244
|
super(params)
|
|
@@ -247,6 +269,7 @@ class LogStash::Codecs::CEF < LogStash::Codecs::Base
|
|
|
247
269
|
end
|
|
248
270
|
|
|
249
271
|
def handle(data, &block)
|
|
272
|
+
original_data = data.dup
|
|
250
273
|
event = LogStash::Event.new
|
|
251
274
|
event.set(raw_data_field, data) unless raw_data_field.nil?
|
|
252
275
|
|
|
@@ -281,14 +304,14 @@ class LogStash::Codecs::CEF < LogStash::Codecs::Base
|
|
|
281
304
|
message = unprocessed_data
|
|
282
305
|
|
|
283
306
|
# Try and parse out the syslog header if there is one
|
|
284
|
-
if event.get('cefVersion').include?
|
|
285
|
-
split_cef_version=
|
|
307
|
+
if (cef_version = event.get('cefVersion')).include?(' ')
|
|
308
|
+
split_cef_version = cef_version.rpartition(' ')
|
|
286
309
|
event.set('syslog', split_cef_version[0])
|
|
287
|
-
event.set('cefVersion',split_cef_version[2])
|
|
310
|
+
event.set('cefVersion', split_cef_version[2])
|
|
288
311
|
end
|
|
289
312
|
|
|
290
313
|
# Get rid of the CEF bit in the version
|
|
291
|
-
event.set('cefVersion', event.get('cefVersion')
|
|
314
|
+
event.set('cefVersion', delete_cef_prefix(event.get('cefVersion')))
|
|
292
315
|
|
|
293
316
|
# Use a scanning parser to capture the Extension Key/Value Pairs
|
|
294
317
|
if message && message.include?('=')
|
|
@@ -310,7 +333,8 @@ class LogStash::Codecs::CEF < LogStash::Codecs::Base
|
|
|
310
333
|
|
|
311
334
|
yield event
|
|
312
335
|
rescue => e
|
|
313
|
-
@logger.error("Failed to decode CEF payload. Generating failure event with payload in message field.",
|
|
336
|
+
@logger.error("Failed to decode CEF payload. Generating failure event with payload in message field.",
|
|
337
|
+
:exception => e.class, :message => e.message, :backtrace => e.backtrace, :original_data => original_data)
|
|
314
338
|
yield LogStash::Event.new("message" => data, "tags" => ["_cefparsefailure"])
|
|
315
339
|
end
|
|
316
340
|
|
|
@@ -319,25 +343,25 @@ class LogStash::Codecs::CEF < LogStash::Codecs::Base
|
|
|
319
343
|
# "CEF:0|Elasticsearch|Logstash|1.0|Signature|Name|Sev|"
|
|
320
344
|
|
|
321
345
|
vendor = sanitize_header_field(event.sprintf(@vendor))
|
|
322
|
-
vendor = self.class.get_config["vendor"][:default] if vendor
|
|
346
|
+
vendor = self.class.get_config["vendor"][:default] if vendor.empty?
|
|
323
347
|
|
|
324
348
|
product = sanitize_header_field(event.sprintf(@product))
|
|
325
|
-
product = self.class.get_config["product"][:default] if product
|
|
349
|
+
product = self.class.get_config["product"][:default] if product.empty?
|
|
326
350
|
|
|
327
351
|
version = sanitize_header_field(event.sprintf(@version))
|
|
328
|
-
version = self.class.get_config["version"][:default] if version
|
|
352
|
+
version = self.class.get_config["version"][:default] if version.empty?
|
|
329
353
|
|
|
330
354
|
signature = sanitize_header_field(event.sprintf(@signature))
|
|
331
|
-
signature = self.class.get_config["signature"][:default] if signature
|
|
355
|
+
signature = self.class.get_config["signature"][:default] if signature.empty?
|
|
332
356
|
|
|
333
357
|
name = sanitize_header_field(event.sprintf(@name))
|
|
334
|
-
name = self.class.get_config["name"][:default] if name
|
|
358
|
+
name = self.class.get_config["name"][:default] if name.empty?
|
|
335
359
|
|
|
336
360
|
severity = sanitize_severity(event, @severity)
|
|
337
361
|
|
|
338
362
|
# Should also probably set the fields sent
|
|
339
363
|
header = ["CEF:0", vendor, product, version, signature, name, severity].join("|")
|
|
340
|
-
values = @fields.map {|fieldname| get_value(fieldname, event)}.compact.join(" ")
|
|
364
|
+
values = @fields.map { |fieldname| get_value(fieldname, event) }.compact.join(" ")
|
|
341
365
|
|
|
342
366
|
@on_event.call(event, "#{header}|#{values}#{@delimiter}")
|
|
343
367
|
end
|
|
@@ -347,51 +371,25 @@ class LogStash::Codecs::CEF < LogStash::Codecs::Base
|
|
|
347
371
|
# Escape pipes and backslashes in the header. Equal signs are ok.
|
|
348
372
|
# Newlines are forbidden.
|
|
349
373
|
def sanitize_header_field(value)
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
value.each_char{|c|
|
|
355
|
-
case c
|
|
356
|
-
when "\\", "|"
|
|
357
|
-
output += "\\" + c
|
|
358
|
-
when "\n", "\r"
|
|
359
|
-
output += " "
|
|
360
|
-
else
|
|
361
|
-
output += c
|
|
362
|
-
end
|
|
363
|
-
}
|
|
364
|
-
|
|
365
|
-
return output
|
|
374
|
+
value.to_s
|
|
375
|
+
.gsub("\r\n", "\n")
|
|
376
|
+
.gsub(HEADER_FIELD_SANITIZER_PATTERN, HEADER_FIELD_SANITIZER_MAPPING)
|
|
366
377
|
end
|
|
367
378
|
|
|
368
379
|
# Keys must be made up of a single word, with no spaces
|
|
369
380
|
# must be alphanumeric
|
|
370
381
|
def sanitize_extension_key(value)
|
|
371
|
-
value
|
|
372
|
-
|
|
382
|
+
value.to_s
|
|
383
|
+
.gsub(/[^a-zA-Z0-9]/, "")
|
|
373
384
|
end
|
|
374
385
|
|
|
375
386
|
# Escape equal signs in the extensions. Canonicalize newlines.
|
|
376
387
|
# CEF spec leaves it up to us to choose \r or \n for newline.
|
|
377
388
|
# We choose \n as the default.
|
|
378
389
|
def sanitize_extension_val(value)
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
value.each_char{|c|
|
|
384
|
-
case c
|
|
385
|
-
when "\\", "="
|
|
386
|
-
output += "\\" + c
|
|
387
|
-
when "\n", "\r"
|
|
388
|
-
output += "\\n"
|
|
389
|
-
else
|
|
390
|
-
output += c
|
|
391
|
-
end
|
|
392
|
-
}
|
|
393
|
-
|
|
394
|
-
return output
|
|
390
|
+
value.to_s
|
|
391
|
+
.gsub("\r\n", "\n")
|
|
392
|
+
.gsub(EXTENSION_VALUE_SANITIZER_PATTERN, EXTENSION_VALUE_SANITIZER_MAPPING)
|
|
395
393
|
end
|
|
396
394
|
|
|
397
395
|
def get_value(fieldname, event)
|
|
@@ -418,7 +416,7 @@ class LogStash::Codecs::CEF < LogStash::Codecs::Base
|
|
|
418
416
|
def sanitize_severity(event, severity)
|
|
419
417
|
severity = sanitize_header_field(event.sprintf(severity)).strip
|
|
420
418
|
severity = self.class.get_config["severity"][:default] unless valid_severity?(severity)
|
|
421
|
-
severity
|
|
419
|
+
severity.to_i.to_s
|
|
422
420
|
end
|
|
423
421
|
|
|
424
422
|
def valid_severity?(sev)
|
|
@@ -429,4 +427,14 @@ class LogStash::Codecs::CEF < LogStash::Codecs::Base
|
|
|
429
427
|
rescue TypeError, ArgumentError
|
|
430
428
|
false
|
|
431
429
|
end
|
|
430
|
+
|
|
431
|
+
if Gem::Requirement.new(">= 2.5.0").satisfied_by? Gem::Version.new(RUBY_VERSION)
|
|
432
|
+
def delete_cef_prefix(cef_version)
|
|
433
|
+
cef_version.delete_prefix(CEF_PREFIX)
|
|
434
|
+
end
|
|
435
|
+
else
|
|
436
|
+
def delete_cef_prefix(cef_version)
|
|
437
|
+
cef_version.start_with?(CEF_PREFIX) ? cef_version[CEF_PREFIX.length..-1] : cef_version
|
|
438
|
+
end
|
|
439
|
+
end
|
|
432
440
|
end
|
data/logstash-codec-cef.gemspec
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
Gem::Specification.new do |s|
|
|
2
2
|
|
|
3
3
|
s.name = 'logstash-codec-cef'
|
|
4
|
-
s.version = '
|
|
4
|
+
s.version = '6.1.2'
|
|
5
5
|
s.platform = 'java'
|
|
6
6
|
s.licenses = ['Apache License (2.0)']
|
|
7
7
|
s.summary = "Reads the ArcSight Common Event Format (CEF)."
|
|
@@ -24,4 +24,5 @@ Gem::Specification.new do |s|
|
|
|
24
24
|
s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
|
|
25
25
|
|
|
26
26
|
s.add_development_dependency 'logstash-devutils'
|
|
27
|
+
s.add_development_dependency 'insist'
|
|
27
28
|
end
|
data/spec/codecs/cef_spec.rb
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
# encoding: utf-8
|
|
2
2
|
require "logstash/devutils/rspec/spec_helper"
|
|
3
|
+
require "insist"
|
|
3
4
|
require "logstash/codecs/cef"
|
|
4
5
|
require "logstash/event"
|
|
5
6
|
require "json"
|
|
@@ -213,10 +214,10 @@ describe LogStash::Codecs::CEF do
|
|
|
213
214
|
it "should encode the CEF field names to their long versions" do
|
|
214
215
|
# This is with the default value of "reverse_mapping" that is "false".
|
|
215
216
|
codec.on_event{|data, newdata| results << newdata}
|
|
216
|
-
codec.fields = [ "deviceAction", "applicationProtocol", "deviceCustomIPv6Address1", "deviceCustomIPv6Address1Label", "deviceCustomIPv6Address2", "deviceCustomIPv6Address2Label", "deviceCustomIPv6Address3", "deviceCustomIPv6Address3Label", "deviceCustomIPv6Address4", "deviceCustomIPv6Address4Label", "deviceEventCategory", "deviceCustomFloatingPoint1", "deviceCustomFloatingPoint1Label", "deviceCustomFloatingPoint2", "deviceCustomFloatingPoint2Label", "deviceCustomFloatingPoint3", "deviceCustomFloatingPoint3Label", "deviceCustomFloatingPoint4", "deviceCustomFloatingPoint4Label", "deviceCustomNumber1", "deviceCustomNumber1Label", "deviceCustomNumber2", "deviceCustomNumber2Label", "deviceCustomNumber3", "deviceCustomNumber3Label", "baseEventCount", "deviceCustomString1", "deviceCustomString1Label", "deviceCustomString2", "deviceCustomString2Label", "deviceCustomString3", "deviceCustomString3Label", "deviceCustomString4", "deviceCustomString4Label", "deviceCustomString5", "deviceCustomString5Label", "deviceCustomString6", "deviceCustomString6Label", "destinationHostName", "destinationMacAddress", "destinationNtDomain", "destinationProcessId", "destinationUserPrivileges", "destinationProcessName", "destinationPort", "destinationAddress", "destinationUserId", "destinationUserName", "deviceAddress", "deviceHostName", "deviceProcessId", "endTime", "fileName", "fileSize", "bytesIn", "message", "bytesOut", "eventOutcome", "transportProtocol", "requestUrl", "deviceReceiptTime", "sourceHostName", "sourceMacAddress", "sourceNtDomain", "sourceProcessId", "sourceUserPrivileges", "sourceProcessName", "sourcePort", "sourceAddress", "startTime", "sourceUserId", "sourceUserName", "
|
|
217
|
-
event = LogStash::Event.new("deviceAction" => "foobar", "applicationProtocol" => "foobar", "deviceCustomIPv6Address1" => "foobar", "deviceCustomIPv6Address1Label" => "foobar", "deviceCustomIPv6Address2" => "foobar", "deviceCustomIPv6Address2Label" => "foobar", "deviceCustomIPv6Address3" => "foobar", "deviceCustomIPv6Address3Label" => "foobar", "deviceCustomIPv6Address4" => "foobar", "deviceCustomIPv6Address4Label" => "foobar", "deviceEventCategory" => "foobar", "deviceCustomFloatingPoint1" => "foobar", "deviceCustomFloatingPoint1Label" => "foobar", "deviceCustomFloatingPoint2" => "foobar", "deviceCustomFloatingPoint2Label" => "foobar", "deviceCustomFloatingPoint3" => "foobar", "deviceCustomFloatingPoint3Label" => "foobar", "deviceCustomFloatingPoint4" => "foobar", "deviceCustomFloatingPoint4Label" => "foobar", "deviceCustomNumber1" => "foobar", "deviceCustomNumber1Label" => "foobar", "deviceCustomNumber2" => "foobar", "deviceCustomNumber2Label" => "foobar", "deviceCustomNumber3" => "foobar", "deviceCustomNumber3Label" => "foobar", "baseEventCount" => "foobar", "deviceCustomString1" => "foobar", "deviceCustomString1Label" => "foobar", "deviceCustomString2" => "foobar", "deviceCustomString2Label" => "foobar", "deviceCustomString3" => "foobar", "deviceCustomString3Label" => "foobar", "deviceCustomString4" => "foobar", "deviceCustomString4Label" => "foobar", "deviceCustomString5" => "foobar", "deviceCustomString5Label" => "foobar", "deviceCustomString6" => "foobar", "deviceCustomString6Label" => "foobar", "destinationHostName" => "foobar", "destinationMacAddress" => "foobar", "destinationNtDomain" => "foobar", "destinationProcessId" => "foobar", "destinationUserPrivileges" => "foobar", "destinationProcessName" => "foobar", "destinationPort" => "foobar", "destinationAddress" => "foobar", "destinationUserId" => "foobar", "destinationUserName" => "foobar", "deviceAddress" => "foobar", "deviceHostName" => "foobar", "deviceProcessId" => "foobar", "endTime" => "foobar", "fileName" => "foobar", "fileSize" => "foobar", "bytesIn" => "foobar", "message" => "foobar", "bytesOut" => "foobar", "eventOutcome" => "foobar", "transportProtocol" => "foobar", "requestUrl" => "foobar", "deviceReceiptTime" => "foobar", "sourceHostName" => "foobar", "sourceMacAddress" => "foobar", "sourceNtDomain" => "foobar", "sourceProcessId" => "foobar", "sourceUserPrivileges" => "foobar", "sourceProcessName"=> "foobar", "sourcePort" => "foobar", "sourceAddress" => "foobar", "startTime" => "foobar", "sourceUserId" => "foobar", "sourceUserName" => "foobar", "
|
|
217
|
+
codec.fields = [ "deviceAction", "applicationProtocol", "deviceCustomIPv6Address1", "deviceCustomIPv6Address1Label", "deviceCustomIPv6Address2", "deviceCustomIPv6Address2Label", "deviceCustomIPv6Address3", "deviceCustomIPv6Address3Label", "deviceCustomIPv6Address4", "deviceCustomIPv6Address4Label", "deviceEventCategory", "deviceCustomFloatingPoint1", "deviceCustomFloatingPoint1Label", "deviceCustomFloatingPoint2", "deviceCustomFloatingPoint2Label", "deviceCustomFloatingPoint3", "deviceCustomFloatingPoint3Label", "deviceCustomFloatingPoint4", "deviceCustomFloatingPoint4Label", "deviceCustomNumber1", "deviceCustomNumber1Label", "deviceCustomNumber2", "deviceCustomNumber2Label", "deviceCustomNumber3", "deviceCustomNumber3Label", "baseEventCount", "deviceCustomString1", "deviceCustomString1Label", "deviceCustomString2", "deviceCustomString2Label", "deviceCustomString3", "deviceCustomString3Label", "deviceCustomString4", "deviceCustomString4Label", "deviceCustomString5", "deviceCustomString5Label", "deviceCustomString6", "deviceCustomString6Label", "destinationHostName", "destinationMacAddress", "destinationNtDomain", "destinationProcessId", "destinationUserPrivileges", "destinationProcessName", "destinationPort", "destinationAddress", "destinationUserId", "destinationUserName", "deviceAddress", "deviceHostName", "deviceProcessId", "endTime", "fileName", "fileSize", "bytesIn", "message", "bytesOut", "eventOutcome", "transportProtocol", "requestUrl", "deviceReceiptTime", "sourceHostName", "sourceMacAddress", "sourceNtDomain", "sourceProcessId", "sourceUserPrivileges", "sourceProcessName", "sourcePort", "sourceAddress", "startTime", "sourceUserId", "sourceUserName", "agentHostName", "agentReceiptTime", "agentType", "agentId", "agentAddress", "agentVersion", "agentTimeZone", "destinationTimeZone", "sourceLongitude", "sourceLatitude", "destinationLongitude", "destinationLatitude", "categoryDeviceType", "managerReceiptTime", "agentMacAddress" ]
|
|
218
|
+
event = LogStash::Event.new("deviceAction" => "foobar", "applicationProtocol" => "foobar", "deviceCustomIPv6Address1" => "foobar", "deviceCustomIPv6Address1Label" => "foobar", "deviceCustomIPv6Address2" => "foobar", "deviceCustomIPv6Address2Label" => "foobar", "deviceCustomIPv6Address3" => "foobar", "deviceCustomIPv6Address3Label" => "foobar", "deviceCustomIPv6Address4" => "foobar", "deviceCustomIPv6Address4Label" => "foobar", "deviceEventCategory" => "foobar", "deviceCustomFloatingPoint1" => "foobar", "deviceCustomFloatingPoint1Label" => "foobar", "deviceCustomFloatingPoint2" => "foobar", "deviceCustomFloatingPoint2Label" => "foobar", "deviceCustomFloatingPoint3" => "foobar", "deviceCustomFloatingPoint3Label" => "foobar", "deviceCustomFloatingPoint4" => "foobar", "deviceCustomFloatingPoint4Label" => "foobar", "deviceCustomNumber1" => "foobar", "deviceCustomNumber1Label" => "foobar", "deviceCustomNumber2" => "foobar", "deviceCustomNumber2Label" => "foobar", "deviceCustomNumber3" => "foobar", "deviceCustomNumber3Label" => "foobar", "baseEventCount" => "foobar", "deviceCustomString1" => "foobar", "deviceCustomString1Label" => "foobar", "deviceCustomString2" => "foobar", "deviceCustomString2Label" => "foobar", "deviceCustomString3" => "foobar", "deviceCustomString3Label" => "foobar", "deviceCustomString4" => "foobar", "deviceCustomString4Label" => "foobar", "deviceCustomString5" => "foobar", "deviceCustomString5Label" => "foobar", "deviceCustomString6" => "foobar", "deviceCustomString6Label" => "foobar", "destinationHostName" => "foobar", "destinationMacAddress" => "foobar", "destinationNtDomain" => "foobar", "destinationProcessId" => "foobar", "destinationUserPrivileges" => "foobar", "destinationProcessName" => "foobar", "destinationPort" => "foobar", "destinationAddress" => "foobar", "destinationUserId" => "foobar", "destinationUserName" => "foobar", "deviceAddress" => "foobar", "deviceHostName" => "foobar", "deviceProcessId" => "foobar", "endTime" => "foobar", "fileName" => "foobar", "fileSize" => "foobar", "bytesIn" => "foobar", "message" => "foobar", "bytesOut" => "foobar", "eventOutcome" => "foobar", "transportProtocol" => "foobar", "requestUrl" => "foobar", "deviceReceiptTime" => "foobar", "sourceHostName" => "foobar", "sourceMacAddress" => "foobar", "sourceNtDomain" => "foobar", "sourceProcessId" => "foobar", "sourceUserPrivileges" => "foobar", "sourceProcessName"=> "foobar", "sourcePort" => "foobar", "sourceAddress" => "foobar", "startTime" => "foobar", "sourceUserId" => "foobar", "sourceUserName" => "foobar", "agentHostName" => "foobar", "agentReceiptTime" => "foobar", "agentType" => "foobar", "agentId" => "foobar", "agentAddress" => "foobar", "agentVersion" => "foobar", "agentTimeZone" => "foobar", "destinationTimeZone" => "foobar", "sourceLongitude" => "foobar", "sourceLatitude" => "foobar", "destinationLongitude" => "foobar", "destinationLatitude" => "foobar", "categoryDeviceType" => "foobar", "managerReceiptTime" => "foobar", "agentMacAddress" => "foobar")
|
|
218
219
|
codec.encode(event)
|
|
219
|
-
expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|deviceAction=foobar applicationProtocol=foobar deviceCustomIPv6Address1=foobar deviceCustomIPv6Address1Label=foobar deviceCustomIPv6Address2=foobar deviceCustomIPv6Address2Label=foobar deviceCustomIPv6Address3=foobar deviceCustomIPv6Address3Label=foobar deviceCustomIPv6Address4=foobar deviceCustomIPv6Address4Label=foobar deviceEventCategory=foobar deviceCustomFloatingPoint1=foobar deviceCustomFloatingPoint1Label=foobar deviceCustomFloatingPoint2=foobar deviceCustomFloatingPoint2Label=foobar deviceCustomFloatingPoint3=foobar deviceCustomFloatingPoint3Label=foobar deviceCustomFloatingPoint4=foobar deviceCustomFloatingPoint4Label=foobar deviceCustomNumber1=foobar deviceCustomNumber1Label=foobar deviceCustomNumber2=foobar deviceCustomNumber2Label=foobar deviceCustomNumber3=foobar deviceCustomNumber3Label=foobar baseEventCount=foobar deviceCustomString1=foobar deviceCustomString1Label=foobar deviceCustomString2=foobar deviceCustomString2Label=foobar deviceCustomString3=foobar deviceCustomString3Label=foobar deviceCustomString4=foobar deviceCustomString4Label=foobar deviceCustomString5=foobar deviceCustomString5Label=foobar deviceCustomString6=foobar deviceCustomString6Label=foobar destinationHostName=foobar destinationMacAddress=foobar destinationNtDomain=foobar destinationProcessId=foobar destinationUserPrivileges=foobar destinationProcessName=foobar destinationPort=foobar destinationAddress=foobar destinationUserId=foobar destinationUserName=foobar deviceAddress=foobar deviceHostName=foobar deviceProcessId=foobar endTime=foobar fileName=foobar fileSize=foobar bytesIn=foobar message=foobar bytesOut=foobar eventOutcome=foobar transportProtocol=foobar requestUrl=foobar deviceReceiptTime=foobar sourceHostName=foobar sourceMacAddress=foobar sourceNtDomain=foobar sourceProcessId=foobar sourceUserPrivileges=foobar sourceProcessName=foobar sourcePort=foobar sourceAddress=foobar startTime=foobar sourceUserId=foobar sourceUserName=foobar
|
|
220
|
+
expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|deviceAction=foobar applicationProtocol=foobar deviceCustomIPv6Address1=foobar deviceCustomIPv6Address1Label=foobar deviceCustomIPv6Address2=foobar deviceCustomIPv6Address2Label=foobar deviceCustomIPv6Address3=foobar deviceCustomIPv6Address3Label=foobar deviceCustomIPv6Address4=foobar deviceCustomIPv6Address4Label=foobar deviceEventCategory=foobar deviceCustomFloatingPoint1=foobar deviceCustomFloatingPoint1Label=foobar deviceCustomFloatingPoint2=foobar deviceCustomFloatingPoint2Label=foobar deviceCustomFloatingPoint3=foobar deviceCustomFloatingPoint3Label=foobar deviceCustomFloatingPoint4=foobar deviceCustomFloatingPoint4Label=foobar deviceCustomNumber1=foobar deviceCustomNumber1Label=foobar deviceCustomNumber2=foobar deviceCustomNumber2Label=foobar deviceCustomNumber3=foobar deviceCustomNumber3Label=foobar baseEventCount=foobar deviceCustomString1=foobar deviceCustomString1Label=foobar deviceCustomString2=foobar deviceCustomString2Label=foobar deviceCustomString3=foobar deviceCustomString3Label=foobar deviceCustomString4=foobar deviceCustomString4Label=foobar deviceCustomString5=foobar deviceCustomString5Label=foobar deviceCustomString6=foobar deviceCustomString6Label=foobar destinationHostName=foobar destinationMacAddress=foobar destinationNtDomain=foobar destinationProcessId=foobar destinationUserPrivileges=foobar destinationProcessName=foobar destinationPort=foobar destinationAddress=foobar destinationUserId=foobar destinationUserName=foobar deviceAddress=foobar deviceHostName=foobar deviceProcessId=foobar endTime=foobar fileName=foobar fileSize=foobar bytesIn=foobar message=foobar bytesOut=foobar eventOutcome=foobar transportProtocol=foobar requestUrl=foobar deviceReceiptTime=foobar sourceHostName=foobar sourceMacAddress=foobar sourceNtDomain=foobar sourceProcessId=foobar sourceUserPrivileges=foobar sourceProcessName=foobar sourcePort=foobar sourceAddress=foobar startTime=foobar sourceUserId=foobar sourceUserName=foobar agentHostName=foobar agentReceiptTime=foobar agentType=foobar agentId=foobar agentAddress=foobar agentVersion=foobar agentTimeZone=foobar destinationTimeZone=foobar sourceLongitude=foobar sourceLatitude=foobar destinationLongitude=foobar destinationLatitude=foobar categoryDeviceType=foobar managerReceiptTime=foobar agentMacAddress=foobar$/m)
|
|
220
221
|
end
|
|
221
222
|
|
|
222
223
|
context "with reverse_mapping set to true" do
|
|
@@ -224,8 +225,8 @@ describe LogStash::Codecs::CEF do
|
|
|
224
225
|
|
|
225
226
|
it "should encode the CEF field names to their short versions" do
|
|
226
227
|
codec.on_event{|data, newdata| results << newdata}
|
|
227
|
-
codec.fields = [ "deviceAction", "applicationProtocol", "deviceCustomIPv6Address1", "deviceCustomIPv6Address1Label", "deviceCustomIPv6Address2", "deviceCustomIPv6Address2Label", "deviceCustomIPv6Address3", "deviceCustomIPv6Address3Label", "deviceCustomIPv6Address4", "deviceCustomIPv6Address4Label", "deviceEventCategory", "deviceCustomFloatingPoint1", "deviceCustomFloatingPoint1Label", "deviceCustomFloatingPoint2", "deviceCustomFloatingPoint2Label", "deviceCustomFloatingPoint3", "deviceCustomFloatingPoint3Label", "deviceCustomFloatingPoint4", "deviceCustomFloatingPoint4Label", "deviceCustomNumber1", "deviceCustomNumber1Label", "deviceCustomNumber2", "deviceCustomNumber2Label", "deviceCustomNumber3", "deviceCustomNumber3Label", "baseEventCount", "deviceCustomString1", "deviceCustomString1Label", "deviceCustomString2", "deviceCustomString2Label", "deviceCustomString3", "deviceCustomString3Label", "deviceCustomString4", "deviceCustomString4Label", "deviceCustomString5", "deviceCustomString5Label", "deviceCustomString6", "deviceCustomString6Label", "destinationHostName", "destinationMacAddress", "destinationNtDomain", "destinationProcessId", "destinationUserPrivileges", "destinationProcessName", "destinationPort", "destinationAddress", "destinationUserId", "destinationUserName", "deviceAddress", "deviceHostName", "deviceProcessId", "endTime", "fileName", "fileSize", "bytesIn", "message", "bytesOut", "eventOutcome", "transportProtocol", "requestUrl", "deviceReceiptTime", "sourceHostName", "sourceMacAddress", "sourceNtDomain", "sourceProcessId", "sourceUserPrivileges", "sourceProcessName", "sourcePort", "sourceAddress", "startTime", "sourceUserId", "sourceUserName", "
|
|
228
|
-
event = LogStash::Event.new("deviceAction" => "foobar", "applicationProtocol" => "foobar", "deviceCustomIPv6Address1" => "foobar", "deviceCustomIPv6Address1Label" => "foobar", "deviceCustomIPv6Address2" => "foobar", "deviceCustomIPv6Address2Label" => "foobar", "deviceCustomIPv6Address3" => "foobar", "deviceCustomIPv6Address3Label" => "foobar", "deviceCustomIPv6Address4" => "foobar", "deviceCustomIPv6Address4Label" => "foobar", "deviceEventCategory" => "foobar", "deviceCustomFloatingPoint1" => "foobar", "deviceCustomFloatingPoint1Label" => "foobar", "deviceCustomFloatingPoint2" => "foobar", "deviceCustomFloatingPoint2Label" => "foobar", "deviceCustomFloatingPoint3" => "foobar", "deviceCustomFloatingPoint3Label" => "foobar", "deviceCustomFloatingPoint4" => "foobar", "deviceCustomFloatingPoint4Label" => "foobar", "deviceCustomNumber1" => "foobar", "deviceCustomNumber1Label" => "foobar", "deviceCustomNumber2" => "foobar", "deviceCustomNumber2Label" => "foobar", "deviceCustomNumber3" => "foobar", "deviceCustomNumber3Label" => "foobar", "baseEventCount" => "foobar", "deviceCustomString1" => "foobar", "deviceCustomString1Label" => "foobar", "deviceCustomString2" => "foobar", "deviceCustomString2Label" => "foobar", "deviceCustomString3" => "foobar", "deviceCustomString3Label" => "foobar", "deviceCustomString4" => "foobar", "deviceCustomString4Label" => "foobar", "deviceCustomString5" => "foobar", "deviceCustomString5Label" => "foobar", "deviceCustomString6" => "foobar", "deviceCustomString6Label" => "foobar", "destinationHostName" => "foobar", "destinationMacAddress" => "foobar", "destinationNtDomain" => "foobar", "destinationProcessId" => "foobar", "destinationUserPrivileges" => "foobar", "destinationProcessName" => "foobar", "destinationPort" => "foobar", "destinationAddress" => "foobar", "destinationUserId" => "foobar", "destinationUserName" => "foobar", "deviceAddress" => "foobar", "deviceHostName" => "foobar", "deviceProcessId" => "foobar", "endTime" => "foobar", "fileName" => "foobar", "fileSize" => "foobar", "bytesIn" => "foobar", "message" => "foobar", "bytesOut" => "foobar", "eventOutcome" => "foobar", "transportProtocol" => "foobar", "requestUrl" => "foobar", "deviceReceiptTime" => "foobar", "sourceHostName" => "foobar", "sourceMacAddress" => "foobar", "sourceNtDomain" => "foobar", "sourceProcessId" => "foobar", "sourceUserPrivileges" => "foobar", "sourceProcessName"=> "foobar", "sourcePort" => "foobar", "sourceAddress" => "foobar", "startTime" => "foobar", "sourceUserId" => "foobar", "sourceUserName" => "foobar", "
|
|
228
|
+
codec.fields = [ "deviceAction", "applicationProtocol", "deviceCustomIPv6Address1", "deviceCustomIPv6Address1Label", "deviceCustomIPv6Address2", "deviceCustomIPv6Address2Label", "deviceCustomIPv6Address3", "deviceCustomIPv6Address3Label", "deviceCustomIPv6Address4", "deviceCustomIPv6Address4Label", "deviceEventCategory", "deviceCustomFloatingPoint1", "deviceCustomFloatingPoint1Label", "deviceCustomFloatingPoint2", "deviceCustomFloatingPoint2Label", "deviceCustomFloatingPoint3", "deviceCustomFloatingPoint3Label", "deviceCustomFloatingPoint4", "deviceCustomFloatingPoint4Label", "deviceCustomNumber1", "deviceCustomNumber1Label", "deviceCustomNumber2", "deviceCustomNumber2Label", "deviceCustomNumber3", "deviceCustomNumber3Label", "baseEventCount", "deviceCustomString1", "deviceCustomString1Label", "deviceCustomString2", "deviceCustomString2Label", "deviceCustomString3", "deviceCustomString3Label", "deviceCustomString4", "deviceCustomString4Label", "deviceCustomString5", "deviceCustomString5Label", "deviceCustomString6", "deviceCustomString6Label", "destinationHostName", "destinationMacAddress", "destinationNtDomain", "destinationProcessId", "destinationUserPrivileges", "destinationProcessName", "destinationPort", "destinationAddress", "destinationUserId", "destinationUserName", "deviceAddress", "deviceHostName", "deviceProcessId", "endTime", "fileName", "fileSize", "bytesIn", "message", "bytesOut", "eventOutcome", "transportProtocol", "requestUrl", "deviceReceiptTime", "sourceHostName", "sourceMacAddress", "sourceNtDomain", "sourceProcessId", "sourceUserPrivileges", "sourceProcessName", "sourcePort", "sourceAddress", "startTime", "sourceUserId", "sourceUserName", "agentHostName", "agentReceiptTime", "agentType", "agentId", "agentAddress", "agentVersion", "agentTimeZone", "destinationTimeZone", "sourceLongitude", "sourceLatitude", "destinationLongitude", "destinationLatitude", "categoryDeviceType", "managerReceiptTime", "agentMacAddress" ]
|
|
229
|
+
event = LogStash::Event.new("deviceAction" => "foobar", "applicationProtocol" => "foobar", "deviceCustomIPv6Address1" => "foobar", "deviceCustomIPv6Address1Label" => "foobar", "deviceCustomIPv6Address2" => "foobar", "deviceCustomIPv6Address2Label" => "foobar", "deviceCustomIPv6Address3" => "foobar", "deviceCustomIPv6Address3Label" => "foobar", "deviceCustomIPv6Address4" => "foobar", "deviceCustomIPv6Address4Label" => "foobar", "deviceEventCategory" => "foobar", "deviceCustomFloatingPoint1" => "foobar", "deviceCustomFloatingPoint1Label" => "foobar", "deviceCustomFloatingPoint2" => "foobar", "deviceCustomFloatingPoint2Label" => "foobar", "deviceCustomFloatingPoint3" => "foobar", "deviceCustomFloatingPoint3Label" => "foobar", "deviceCustomFloatingPoint4" => "foobar", "deviceCustomFloatingPoint4Label" => "foobar", "deviceCustomNumber1" => "foobar", "deviceCustomNumber1Label" => "foobar", "deviceCustomNumber2" => "foobar", "deviceCustomNumber2Label" => "foobar", "deviceCustomNumber3" => "foobar", "deviceCustomNumber3Label" => "foobar", "baseEventCount" => "foobar", "deviceCustomString1" => "foobar", "deviceCustomString1Label" => "foobar", "deviceCustomString2" => "foobar", "deviceCustomString2Label" => "foobar", "deviceCustomString3" => "foobar", "deviceCustomString3Label" => "foobar", "deviceCustomString4" => "foobar", "deviceCustomString4Label" => "foobar", "deviceCustomString5" => "foobar", "deviceCustomString5Label" => "foobar", "deviceCustomString6" => "foobar", "deviceCustomString6Label" => "foobar", "destinationHostName" => "foobar", "destinationMacAddress" => "foobar", "destinationNtDomain" => "foobar", "destinationProcessId" => "foobar", "destinationUserPrivileges" => "foobar", "destinationProcessName" => "foobar", "destinationPort" => "foobar", "destinationAddress" => "foobar", "destinationUserId" => "foobar", "destinationUserName" => "foobar", "deviceAddress" => "foobar", "deviceHostName" => "foobar", "deviceProcessId" => "foobar", "endTime" => "foobar", "fileName" => "foobar", "fileSize" => "foobar", "bytesIn" => "foobar", "message" => "foobar", "bytesOut" => "foobar", "eventOutcome" => "foobar", "transportProtocol" => "foobar", "requestUrl" => "foobar", "deviceReceiptTime" => "foobar", "sourceHostName" => "foobar", "sourceMacAddress" => "foobar", "sourceNtDomain" => "foobar", "sourceProcessId" => "foobar", "sourceUserPrivileges" => "foobar", "sourceProcessName"=> "foobar", "sourcePort" => "foobar", "sourceAddress" => "foobar", "startTime" => "foobar", "sourceUserId" => "foobar", "sourceUserName" => "foobar", "agentHostName" => "foobar", "agentReceiptTime" => "foobar", "agentType" => "foobar", "agentId" => "foobar", "agentAddress" => "foobar", "agentVersion" => "foobar", "agentTimeZone" => "foobar", "destinationTimeZone" => "foobar", "sourceLongitude" => "foobar", "sourceLatitude" => "foobar", "destinationLongitude" => "foobar", "destinationLatitude" => "foobar", "categoryDeviceType" => "foobar", "managerReceiptTime" => "foobar", "agentMacAddress" => "foobar")
|
|
229
230
|
codec.encode(event)
|
|
230
231
|
expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|act=foobar app=foobar c6a1=foobar c6a1Label=foobar c6a2=foobar c6a2Label=foobar c6a3=foobar c6a3Label=foobar c6a4=foobar c6a4Label=foobar cat=foobar cfp1=foobar cfp1Label=foobar cfp2=foobar cfp2Label=foobar cfp3=foobar cfp3Label=foobar cfp4=foobar cfp4Label=foobar cn1=foobar cn1Label=foobar cn2=foobar cn2Label=foobar cn3=foobar cn3Label=foobar cnt=foobar cs1=foobar cs1Label=foobar cs2=foobar cs2Label=foobar cs3=foobar cs3Label=foobar cs4=foobar cs4Label=foobar cs5=foobar cs5Label=foobar cs6=foobar cs6Label=foobar dhost=foobar dmac=foobar dntdom=foobar dpid=foobar dpriv=foobar dproc=foobar dpt=foobar dst=foobar duid=foobar duser=foobar dvc=foobar dvchost=foobar dvcpid=foobar end=foobar fname=foobar fsize=foobar in=foobar msg=foobar out=foobar outcome=foobar proto=foobar request=foobar rt=foobar shost=foobar smac=foobar sntdom=foobar spid=foobar spriv=foobar sproc=foobar spt=foobar src=foobar start=foobar suid=foobar suser=foobar ahost=foobar art=foobar at=foobar aid=foobar agt=foobar av=foobar atz=foobar dtz=foobar slong=foobar slat=foobar dlong=foobar dlat=foobar catdt=foobar mrt=foobar amac=foobar$/m)
|
|
231
232
|
end
|
|
@@ -555,6 +556,16 @@ describe LogStash::Codecs::CEF do
|
|
|
555
556
|
end
|
|
556
557
|
end
|
|
557
558
|
|
|
559
|
+
let (:dots_in_keys) {'CEF:0|Vendor|Device|Version|13|my message|5|dvchost=loghost cat=traffic deviceSeverity=notice ad.nn=TEST src=192.168.0.1 destinationPort=53'}
|
|
560
|
+
it "should be OK with dots in keys" do
|
|
561
|
+
decode_one(subject, dots_in_keys) do |e|
|
|
562
|
+
insist { e.get("deviceHostName") } == "loghost"
|
|
563
|
+
insist { e.get("ad.nn") } == 'TEST'
|
|
564
|
+
insist { e.get("sourceAddress") } == '192.168.0.1'
|
|
565
|
+
insist { e.get("destinationPort") } == '53'
|
|
566
|
+
end
|
|
567
|
+
end
|
|
568
|
+
|
|
558
569
|
let (:allow_spaces_in_values) {'CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10|src=10.0.0.192 dst=12.121.122.82 spt=1232 dproc=InternetExplorer x.x.x.x'}
|
|
559
570
|
it "should be OK to have one or more spaces in values" do
|
|
560
571
|
decode_one(subject, allow_spaces_in_values) do |e|
|
|
@@ -580,6 +591,20 @@ describe LogStash::Codecs::CEF do
|
|
|
580
591
|
end
|
|
581
592
|
end
|
|
582
593
|
|
|
594
|
+
let(:preserve_complex_multiple_dot_notation_in_extension_fields) { 'CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10|src=10.0.0.192 additional.dotfieldName=new_value ad.Authentification=MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 ad.Error_,Code=3221225578 dst=12.121.122.82 ad.field[0]=field0 ad.foo.name[1]=new_name' }
|
|
595
|
+
it "should keep ad.fields" do
|
|
596
|
+
decode_one(subject, preserve_complex_multiple_dot_notation_in_extension_fields) do |e|
|
|
597
|
+
validate(e)
|
|
598
|
+
insist { e.get("sourceAddress") } == "10.0.0.192"
|
|
599
|
+
insist { e.get("destinationAddress") } == "12.121.122.82"
|
|
600
|
+
insist { e.get("[ad.field][0]") } == "field0"
|
|
601
|
+
insist { e.get("[ad.foo.name][1]") } == "new_name"
|
|
602
|
+
insist { e.get("ad.Authentification") } == "MICROSOFT_AUTHENTICATION_PACKAGE_V1_0"
|
|
603
|
+
insist { e.get('ad.Error_,Code') } == "3221225578"
|
|
604
|
+
insist { e.get("additional.dotfieldName") } == "new_value"
|
|
605
|
+
end
|
|
606
|
+
end
|
|
607
|
+
|
|
583
608
|
let (:preserve_random_values_key_value_pairs_alongside_with_additional_fields) {'CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10|src=10.0.0.192 cs4=401 random.user Admin 0 23041A10181C0000 23041810181C0000 /CN\=random.user/OU\=User Login End-Entity /CN\=TEST/OU\=Login CA TEST 34 additional.dotfieldName=new_value ad.Authentification=MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 ad.Error_,Code=3221225578 dst=12.121.122.82 ad.field[0]=field0 ad.name[1]=new_name'}
|
|
584
609
|
it "should correctly parse random values even with additional fields in message" do
|
|
585
610
|
decode_one(subject, preserve_random_values_key_value_pairs_alongside_with_additional_fields) do |e|
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: logstash-codec-cef
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version:
|
|
4
|
+
version: 6.1.2
|
|
5
5
|
platform: java
|
|
6
6
|
authors:
|
|
7
7
|
- Elastic
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date:
|
|
11
|
+
date: 2021-03-31 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -44,6 +44,20 @@ dependencies:
|
|
|
44
44
|
- - ">="
|
|
45
45
|
- !ruby/object:Gem::Version
|
|
46
46
|
version: '0'
|
|
47
|
+
- !ruby/object:Gem::Dependency
|
|
48
|
+
requirement: !ruby/object:Gem::Requirement
|
|
49
|
+
requirements:
|
|
50
|
+
- - ">="
|
|
51
|
+
- !ruby/object:Gem::Version
|
|
52
|
+
version: '0'
|
|
53
|
+
name: insist
|
|
54
|
+
prerelease: false
|
|
55
|
+
type: :development
|
|
56
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
57
|
+
requirements:
|
|
58
|
+
- - ">="
|
|
59
|
+
- !ruby/object:Gem::Version
|
|
60
|
+
version: '0'
|
|
47
61
|
description: This gem is a Logstash plugin required to be installed on top of the
|
|
48
62
|
Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This
|
|
49
63
|
gem is not a stand-alone program
|