logstash-codec-netflow 3.5.1 → 3.5.2

Sign up to get free protection for your applications and to get access to all the features.
Files changed (13) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +3 -0
  3. data/CONTRIBUTORS +1 -0
  4. data/RFC_COMPLIANCE_IPFIX.md +230 -0
  5. data/RFC_COMPLIANCE_NETFLOW_v9.md +407 -0
  6. data/docs/index.asciidoc +23 -4
  7. data/lib/logstash/codecs/netflow/util.rb +1 -1
  8. data/logstash-codec-netflow.gemspec +1 -1
  9. data/spec/codecs/ipfix_test_mikrotik_data258.dat +0 -0
  10. data/spec/codecs/ipfix_test_mikrotik_data259.dat +0 -0
  11. data/spec/codecs/ipfix_test_mikrotik_tpl.dat +0 -0
  12. data/spec/codecs/netflow_spec.rb +74 -0
  13. metadata +92 -86
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 936d05f955f0c16ab55b3a6998302ca2f283c2d2
4
- data.tar.gz: f6f16b2055779fe96113d8f61d281fb27e4673cf
3
+ metadata.gz: dea55de2fab511c14f07d52974addff5c6d9418c
4
+ data.tar.gz: 4c6a9df871fc761a276a736049e21713d6bb93b9
5
5
  SHA512:
6
- metadata.gz: e1e1bcb5abd65d89dc491122bfb99bda3b4fd20a9e17cc58753779e14b4b255c43c9d3a50ede39a5e6ed51a41999131bc1c844b2ee277863318a8724d9e995be
7
- data.tar.gz: 1358d6678af2221b0087953f0e327de3f7916dbbf2513872000a2ef261701695c69a55a17f505ffbeb74a5c4e30419cb6cfe2b539a8b1dc09adf567643877f97
6
+ metadata.gz: 680bfc1c2de24e3f2e567b103f7c3457dc1cf415dd870760e8c729727d1911735e880a97fdd077c33f813b74591f3d09e1bf9b20b05e9f64ef4f1b0a1fa51859
7
+ data.tar.gz: 9164953d8fa32cd72ddfd669f372acb0348c2574810481b7fee1076f51069aecf0489f19d1ea2a7c833a9f279dc00776bb9b9c0bb48ab9c21a4bea566ae4757f
data/CHANGELOG.md CHANGED
@@ -1,3 +1,6 @@
1
+ ## 3.5.2
2
+ - Fix some documentation issues
3
+
1
4
  ## 3.5.1
2
5
 
3
6
  - Added test for Fortigate FortiOS 5.2 (Netflow v9)
data/CONTRIBUTORS CHANGED
@@ -4,6 +4,7 @@ reports, or in general have helped logstash along its way.
4
4
  Contributors:
5
5
  * Aaron Mildenstein (untergeek)
6
6
  * Adam Kaminski (thimslugga)
7
+ * Bjørn Ruberg (bruberg)
7
8
  * Colin Surprenant (colinsurprenant)
8
9
  * Daniel Nägele (analogbyte)
9
10
  * Diyaldine Maoulida
data/RFC_COMPLIANCE_IPFIX.md ADDED
@@ -0,0 +1,230 @@
1
+ # IPFIX RFC compliance
2
+
3
+ The level of RFC compliance reached for collector-relevant requirements:
4
+
5
+ | RFC | Level |
6
+ |-----------|----------------------------------------------|
7
+ | RFC 7011 | 47% of RFC "MUST" requirements implemented |
8
+ | RFC 7011 | 19% of RFC "SHOULD" requirements implemented |
9
+ | RFC 7012 | 83% of IE data types supported
10
+ | RFC 7012 | 90% of IEs supported
11
+
12
+ ## RFC 7011 collector compliance summary
13
+
14
+ Summary of collector-relevant requirements implemented versus the total collector-relevant requirements:
15
+
16
+ | Chapter |MUST |SHOULD| MAY|
17
+ |---------------------------------------|-----|-----|-----|
18
+ | 1. Introduction | | | |
19
+ | 2. Terminology | | | |
20
+ | 3. IPFIX message format | 2/2 | 0/2 | |
21
+ | 4. Specific reporting requirements | 0/1 | | |
22
+ | 5. Timing considerations | | 0/2 | |
23
+ | 6. Linkage with the Information Model | | 0/1 | |
24
+ | 7. Variable Length IE | | | |
25
+ | 8. Template management | 4/8 | 1/5 | 1/2 |
26
+ | 9. The collecting process's side | 4/5 | 1/3 | 0/4 |
27
+ | 10. Transport protocol | 5/8 | 1/3 | 3/3 |
28
+ | 11. Security considerations | 0/8 | 1/5 | 2/3 |
29
+ | 12. Management considerations | | | |
30
+ | 13. IANA considerations | | | |
31
+
32
+ ## RFC 7012 collector compliance summary
33
+
34
+ | Chapter | MUST |SHOULD| MAY |
35
+ |-----------------------------------|------|------|-----|
36
+ | 1. Introduction | | | |
37
+ | 2. Properties of IPFIX IE | | | |
38
+ | 3. Type Space | | | |
39
+ | 4. IE identitfiers | | | |
40
+ | 5. IE | | | |
41
+ | 6. Exteding the information model | | | |
42
+ | 7. IANA considerations | | | 0/1 |
43
+ | 8. Security considerations | | | |
44
+
45
+
46
+ ## RFC7012 Information Elements data type support details
47
+
48
+ | IE data type | Support | Variable Length support |
49
+ |-----------------------|---------|-------------------------|
50
+ | octetArray | Yes | Yes |
51
+ | unsigned8 | Yes | |
52
+ | unsigned16 | Yes | |
53
+ | unsigned32 | Yes | |
54
+ | unsigned64 | Yes | |
55
+ | signed8 | Yes | |
56
+ | signed16 | Yes | |
57
+ | signed32 | Yes | |
58
+ | signed64 | Yes | |
59
+ | float32 | Yes | |
60
+ | float64 | Yes | |
61
+ | boolean | No | |
62
+ | macAddress | Yes | |
63
+ | string | Yes | Yes |
64
+ | dateTimeSeconds | Yes | |
65
+ | dateTimeMilliseconds | Yes | |
66
+ | dateTimeMicroseconds | Yes | |
67
+ | dateTimeNanoseconds | Yes | |
68
+ | ipv4Address | Yes | |
69
+ | ipv6Address | Yes | |
70
+ | basicList | No | |
71
+ | subTemplateList | No | |
72
+ | subTemplateMultiList | No | |
73
+
74
+ ## RFC7012 Information Elements support details
75
+
76
+ IE 1-433 are supported
77
+
78
+ These are not yet supported:
79
+
80
+ |id | name | data type
81
+ |---|---------------------|-------------------------
82
+ |434|mibObjectValueInteger|signed32
83
+ |435|mibObjectValueOctetString|octetArray
84
+ |436|mibObjectValueOID|octetArray
85
+ |437|mibObjectValueBits|octetArray
86
+ |438|mibObjectValueIPAddress|ipv4Address
87
+ |439|mibObjectValueCounter|unsigned64
88
+ |440|mibObjectValueGauge|unsigned32
89
+ |441|mibObjectValueTimeTicks|unsigned32
90
+ |442|mibObjectValueUnsigned|unsigned32
91
+ |443|mibObjectValueTable|subTemplateList
92
+ |444|mibObjectValueRow|subTemplateList
93
+ |445|mibObjectIdentifier|octetArray
94
+ |446|mibSubIdentifier|unsigned32
95
+ |447|mibIndexIndicator|unsigned64
96
+ |448|mibCaptureTimeSemantics|unsigned8
97
+ |449|mibContextEngineID|octetArray
98
+ |450|mibContextName|string
99
+ |451|mibObjectName|string
100
+ |452|mibObjectDescription|string
101
+ |453|mibObjectSyntax|string
102
+ |454|mibModuleName|string
103
+ |455|mobileIMSI|string
104
+ |456|mobileMSISDN|string
105
+ |457|httpStatusCode|unsigned16
106
+ |458|sourceTransportPortsLimit|unsigned16
107
+ |459|httpRequestMethod|string
108
+ |460|httpRequestHost|string
109
+ |461|httpRequestTarget|string
110
+ |462|httpMessageVersion|string
111
+ |463|natInstanceID|unsigned32
112
+ |464|internalAddressRealm|octetArray
113
+ |465|externalAddressRealm|octetArray
114
+ |466|natQuotaExceededEvent|unsigned32
115
+ |467|natThresholdEvent|unsigned32
116
+ |468|httpUserAgent|string
117
+ |469|httpContentType|string
118
+ |470|httpReasonPhrase|string
119
+
120
+
121
+ ## RFC 7011 collector compliance details
122
+
123
+ The tables below detail the collector-relevant requirements, and whether or not they are implemented:
124
+
125
+ ### 3. IPFIX Message Format
126
+
127
+ | Requirement |MUST |SHOULD| MAY|
128
+ |---------------------------------------|-----|-----|-----|
129
+ |3.1 Collecting Processes SHOULD use the Transport Session and the Observation Domain ID field to separate different export streams originating from the same Exporter.| | NO | |
130
+ |3.4.1 Collecting Processes MUST NOT assume incremental Template IDs | YES | | |
131
+ |3.4.2.1 At a minimum, Collecting Processes SHOULD support as scope the observationDomainId, exportingProcessId, meteringProcessId, templateId, lineCardId, exporterIPv4Address, exporterIPv6Address, and ingressInterface Information Elements. | | ? | |
132
+ | 3.4.2.2 As Exporting Processes are free to allocate Template IDs as they see fit, Collecting Processes MUST NOT assume incremental Template IDs, or anything about the contents of an Options Template based on its Template ID alone | YES | | |
133
+
134
+ ### 4. Specific Reporting Requirements
135
+
136
+ | Requirement |MUST |SHOULD| MAY|
137
+ |---------------------------------------|-----|-----|-----|
138
+ | The Collecting Process MUST check the possible combinations of Information Elements within the Options Template Records to correctly interpret the following Options Templates. | NO | | |
139
+
140
+ ### 5. Timing considerations
141
+
142
+ | Requirement |MUST |SHOULD| MAY|
143
+ |---------------------------------------|-----|-----|-----|
144
+ | 5.2 Collecting Processes SHOULD use the current date, or other contextual information, to properly interpret dateTimeSeconds values and the Export Time Message Header field. | | NO | |
145
+ | 5.2 Collecting Processes SHOULD use the current date, or other contextual information, to determine the NTP era in order to properly interpret dateTimeMicroseconds and dateTimeNanoseconds values in received Data Records | | NO | |
146
+
147
+ ### 6. Linkage with the Information Model
148
+
149
+ | Requirement |MUST |SHOULD| MAY|
150
+ |---------------------------------------|-----|-----|-----|
151
+ | 6.1.6 Collecting Processes SHOULD detect and ignore IPFIX Messages containing ill-formed UTF-8 string values for Information Elements | | NO | |
152
+
153
+ ### 8. Template Management
154
+
155
+ | Requirement |MUST |SHOULD| MAY|
156
+ |---------------------------------------|-----|-----|-----|
157
+ |8. The Collecting Process MUST store all received Template Record information for the duration of each Transport Session until reuse or withdrawal as described in Section 8.1, or expiry over UDP as described in Section 8.4, so that it can interpret the corresponding Data Records.| YES | | |
158
+ |8. The Collecting Process MUST NOT assume that the Template IDs from a given Exporting Process refer to the same Templates as they did in previous Transport Sessions from the same Exporting Process| NO | | |
159
+ |8. Collecting Process MUST NOT use Templates from one Transport Session to decode Data Sets in a subsequent Transport Session.| NO | | |
160
+ |8. Collecting Processes MUST properly handle Templates with multiple identical Information Elements.| ? | | |
161
+ |8. a Collecting Process MUST NOT assume that the Data Set and the associated Template Set (or Options Template Set) are exported in the same IPFIX Message| YES | | |
162
+ |8. Though a Collecting Process normally receives Template Records from the Exporting Process before receiving Data Records, this is not always the case, e.g., in the case of reordering or Collecting Process restart over UDP. In these cases, the Collecting Process MAY buffer Data Records for which it has no Templates, to wait for Template Records describing them; however, note that in the presence of Template withdrawal and redefinition (Section 8.1) this may lead to incorrect interpretation of Data Records.| | | NO |
163
+ | 8.Different Observation Domains within a Transport Session MAY use the same Template ID value to refer to different Templates; Collecting Processes MUST properly handle this case.| NO | | |
164
+ | 8.1 After receiving a Template Withdrawal, a Collecting Process MUST stop using the Template to interpret subsequently exported Data Sets. Note that this mechanism does not apply when UDP is used to transport IPFIX Messages; for that case, see Section 8.4.| NO | | |
165
+ |8.1 If a Collecting Process receives a Template Withdrawal for a Template or Options Template it does not presently have stored, this indicates a malfunctioning or improperly implemented Exporting Process. The continued receipt and interpretation of Data Records are still possible, but the Collecting Process MUST ignore the Template Withdrawal and SHOULD log the error.| | NO | |
166
+ | 8.1 If a Collecting Process receives a new Template Record or Options Template Record for an already-allocated Template ID, and that Template or Options Template is identical to the already-received Template or Options Template, it SHOULD log the retransmission | | NO | |
167
+ |8.1 If a Collecting Process receives a new Template Record or Options Template Record for an already-allocated Template ID, and that Template or Options Template is different from the already-received Template or Options Template, this indicates a malfunctioning or improperly implemented Exporting Process. The continued receipt and unambiguous interpretation of Data Records for this Template ID are no longer possible, and the Collecting Process SHOULD log the error. | | NO | |
168
+ |8.4 The Collecting Process MAY associate a lifetime with each Template received in a Transport Session. Templates not refreshed by the Exporting Process within the lifetime can then be discarded by the Collecting Process. The Template lifetime at the Collecting Process MAY be exposed by a configuration parameter or MAY be derived from observation of the interval of periodic Template retransmissions from the Exporting Process. In this latter case, the Template lifetime SHOULD default to at least 3 times the observed retransmission rate. | | | PARTIAL|
169
+ |8.4 Template Withdrawals (Section 8.1) MUST NOT be sent by Exporting Processes exporting via UDP and MUST be ignored by Collecting Processes collecting via UDP | NO | | |
170
+ |8.4 When a Collecting Process receives a new Template Record or Options Template Record via UDP for an already-allocated Template ID, and that Template or Options Template is identical to the already received Template or Options Template, it SHOULD NOT log the retransmission, as this is the normal operation of Template refresh over UDP.| | YES| |
171
+ |8.4 The Collecting Process MUST replace the Template or Options Template for that Template ID with the newly received Template or Options Template. This is the normal operation of Template ID reuse over UDP. | YES | | |
172
+ |8.4 The Collecting Process SHOULD maintain the following for all the current Template Records and Options Template Records: <IPFIX Device, Exporter source UDP port, Collector IP address, Collector destination UDP port, Observation Domain ID, Template ID, Template Definition, Last Received>. | | NO| |
173
+
174
+ ### 9. The collecting process's side
175
+
176
+ | Requirement |MUST |SHOULD| MAY|
177
+ |---------------------------------------|-----|-----|-----|
178
+ |9. The Collecting Process MUST listen for association requests / connections to start new Transport Sessions from the Exporting Process. | YES | | |
179
+ |9. The Collecting Process MUST note the Information Element identifier of any Information Element that it does not understand and MAY discard that Information Element from received Data Records.| YES | | |
180
+ |9. The Collecting Process MUST accept padding in Data Records and Template Records. | YES | | |
181
+ | 9. A Collector can detect out-of-sequence, dropped, or duplicate IPFIX Messages by tracking the Sequence Number. A Collector SHOULD provide a logging mechanism for tracking out-of- sequence IPFIX Messages. | | NO | |
182
+ | 9.1 If the Collecting Process receives a malformed IPFIX Message, it MUST discard the IPFIX Message and SHOULD log the error. | YES | YES | |
183
+ | 9.1 The Collecting Process MAY attempt to rectify the situation any way it sees fit, including: | | | NO |
184
+ | 9.1 On the other hand, the Collecting Process SHOULD stop processing IPFIX Messages from clearly malfunctioning Exporting Processes (e.g., those from which the last few IPFIX Messages have been malformed). | | NO | |
185
+ | 9.2 The Collecting Process MUST support the opening of multiple SCTP Streams | NO | | |
186
+ | 9.3 The Collecting Process MAY discard all Transport Session state after no IPFIX Messages are received from a given Exporting Process within a given Transport Session during a configurable idle timeout. | | | NO |
187
+ | 9.3 The Collecting Process SHOULD accept Data Records without the associated Template Record (or other definitions such as Common Properties) required to decode the Data Record. | | NO | |
188
+ | 9.3 If the Template Records or other definitions have not been received at the time Data Records are received, the Collecting Process MAY store the Data Records for a short period of time and decode them after the Template Records or other definitions are received | | | NO |
189
+
190
+ ### 10. Transport protocol
191
+
192
+ | Requirement |MUST |SHOULD| MAY|
193
+ |---------------------------------------|-----|-----|-----|
194
+ | 10. A Collecting Process MUST be able to handle IPFIX Message lengths of up to 65535 octets. | YES (LS>v5.1)| | |
195
+ |10. Transport Session state MUST NOT be migrated by an Exporting Process or Collecting Process among Transport Sessions using different transport protocols between the same Exporting Process and Collecting Process pair | NO | | |
196
+ |10.1 SCTP [RFC4960] using the Partially Reliable SCTP (PR-SCTP) extension as specified in [RFC3758] MUST be implemented by all compliant implementations. | NO | | |
197
+ |10.1 UDP [UDP] MAY also be implemented by compliant implementations | | | YES |
198
+ |10.1 TCP [TCP] MAY also be implemented by compliant implementations. | | | YES |
199
+ |10.1 It MUST be possible to configure both the Exporting and Collecting Processes to use different ports than the default. | YES | | |
200
+ | 10.1 By default, the Collecting Process listens for secure connections on SCTP, TCP, and/or UDP port 4740 | | | NO |
201
+ | 10.2.4 When a Collecting Process no longer wants to receive IPFIX Messages, it SHOULD shut down its end of the association. The Collecting Process SHOULD continue to receive and process IPFIX Messages until the Exporting Process has closed its end of the association. | | NO | |
202
+ |10.2.4 When a Collecting Process detects that the SCTP association has been abnormally terminated, it MUST continue to listen for a new association establishment. | NO | | |
203
+ | 10.2.4 When an Exporting Process detects that the SCTP association to the Collecting Process is abnormally terminated, it SHOULD try to re-establish the association. | | NO | |
204
+ | 10.3 UDP MAY be used in deployments where Exporters and Collectors always communicate over dedicated links that are not susceptible to congestion | | | YES |
205
+ | 10.3.2 UDP MUST NOT be used unless the application can tolerate some loss of IPFIX Messages. | | | |
206
+ | 10.4 When a Collecting Process detects that the TCP connection to the Exporting Process has terminated abnormally, it MUST continue to listen for a new connection. | YES | | |
207
+ |10.4 When a Collecting Process no longer wants to receive IPFIX Messages, it SHOULD close its end of the connection. The Collecting Process SHOULD continue to read IPFIX Messages until the Exporting Process has closed its end. | | YES | |
208
+
209
+ ### 11. Security Considerations
210
+
211
+ | Requirement |MUST |SHOULD| MAY|
212
+ |---------------------------------------|-----|-----|-----|
213
+ | 11. IPFIX Exporting Processes and Collecting Processes using UDP or SCTP MUST support DTLS version 1.0 and SHOULD support DTLS version 1.2 [RFC6347], including the mandatory ciphersuite(s) specified in each version. | NO | | |
214
+ | 11. Exporting and Collecting Processes MUST NOT request, offer, or use any version of the Secure Socket Layer (SSL), or any version of TLS prior to 1.1, due to known security vulnerabilities in prior versions of TLS| NO | | | 11.3 When using TLS or DTLS, IPFIX Exporting Processes and IPFIX Collecting Processes SHOULD be identified by a certificate containing the DNS-ID | | NO | |
215
+ | 11.3 The inclusion of Common Names (CN-IDs) in certificates identifying IPFIX Exporting Processes or Collecting Processes is NOT RECOMMENDED. | | NO | |
216
+ |11.3 To prevent man-in-the-middle attacks from impostor Exporting or Collecting Processes, the acceptance of data from an unauthorized Exporting Process, or the export of data to an unauthorized Collecting Process, mutual authentication MUST be used for both TLS and DTLS. | NO | | |
217
+ | 11.3 Collecting Processes MUST verify the reference identifiers of the Exporting Processes from which they are receiving IPFIX Messages against those stored in the certificates | NO | | |
218
+ | 11.3 Collecting Processes MUST NOT accept IPFIX Messages from non-verified Exporting Processes. | NO | | |
219
+ | 11.3 Exporting Processes and Collecting Processes MUST support the verification of certificates against an explicitly authorized list of peer certificates identified by Common Name and SHOULD support the verification of reference identifiers by matching the DNS-ID or CN-ID with a DNS lookup of the peer. | NO | | |
220
+ | 11.3 IPFIX Exporting Processes and Collecting Processes MUST use non-NULL ciphersuites for authentication, integrity, and confidentiality. | NO | | |
221
+ | 11.4 Collector rate limiting SHOULD be used to protect TLS and DTLS| |NO | |
222
+ | 11.4 SYN cookies SHOULD be used by any Collecting Process accepting TCP connections. | | YES | |
223
+ | 11.4 These rate and state limits MAY be provided by a Collecting Process, and if provided, the limits SHOULD be user configurable. | | | NO |
224
+ | 11.5 IPFIX Message traffic transported via UDP and not secured via DTLS SHOULD be protected via segregation to a dedicated network. | | | |
225
+ | 11.6 IPFIX Collecting Processes MUST detect potential IPFIX Message insertion or loss conditions by tracking the IPFIX Sequence Number and SHOULD provide a logging mechanism for reporting out-of-sequence messages. | NO | | |
226
+ | 11.6 IPFIX Exporting and Collecting Processes SHOULD log any connection attempt that fails due to authentication failure | | NO | |
227
+ | 11.6 IPFIX Exporting and Collecting Processes SHOULD detect and log any SCTP association reset or TCP connection reset. | | NO | |
228
+ | 11.7 As IPFIX uses length-prefix encodings, Collector implementors should take care to ensure the detection of inconsistent values that could impact IPFIX Message decoding, and proper operation in the presence of such inconsistent values. | | | YES |
229
+ | 11.7 Specifically, IPFIX Message, Set, and variable-length Information Element lengths must be checked for consistency to avoid buffer-sizing vulnerabilities. | | | YES |
230
+
data/RFC_COMPLIANCE_NETFLOW_v9.md ADDED
@@ -0,0 +1,407 @@
1
+ # Netflow v9 compliance
2
+
3
+ The level of RFC compliance reached for collector-relevant requirements:
4
+
5
+ | RFC | Level |
6
+ |-----------|----------------------------------------------|
7
+ | RFC 3954 | 100% of RFC "MUST" requirements implemented |
8
+ | RFC 3954 | 0% of RFC "SHOULD" requirements implemented |
9
+ | RFC 3954 | 83% of IEs 1-127 supported |
10
+ | RFC 3954 | 10% of IEs 127-32768 supported |
11
+
12
+ ## RFC 3954 collector compliance summary
13
+
14
+ Summary of collector-relevant requirements implemented versus the total collector-relevant requirements:
15
+
16
+ | Chapter |MUST |SHOULD| MAY|
17
+ |----------------------------------------------|-----|-----|-----|
18
+ | 1. Introduction | | | |
19
+ | 2. Terminology | | | |
20
+ | 3. NetFlow High-Level Picture on the Exporter| | | |
21
+ | 4. Packet layout | | | |
22
+ | 5. Export packet format | 1/1 | 0/2 | |
23
+ | 6. Options | 1/1 | | |
24
+ | 7. Template management | 3/3 | | |
25
+ | 8. Field type definitions | | | |
26
+ | 9. The collector side | 5/5 | 0/3 | |
27
+ | 10. Security considerations | | | |
28
+
29
+ ## RFC 3954 Information Elements support details
30
+
31
+ From the IEs 1-127, these are not yet supported:
32
+
33
+ |id | name
34
+ |---|--------------
35
+ |70 |MPLS_LABEL_1
36
+ |71 |MPLS_LABEL_2
37
+ |72 |MPLS_LABEL_3
38
+ |73 |MPLS_LABEL_4
39
+ |74 |MPLS_LABEL_5
40
+ |75 |MPLS_LABEL_6
41
+ |76 |MPLS_LABEL_7
42
+ |77 |MPLS_LABEL_8
43
+ |78 |MPLS_LABEL_9
44
+ |79 |MPLS_LABEL_10
45
+ |90 | MPLS PAL RD
46
+ |91 | MPLS PREFIX LEN
47
+ |92 | SRC TRAFFIC INDEX
48
+ |93 | DST TRAFFIC INDEX
49
+ |95 | APPLICATION TAG
50
+ |99 | replication factor
51
+ |102| layer2packetSectionOffset
52
+ |103| layer2packetSectionSize
53
+ |104| layer2packetSectionData
54
+
55
+ From the IEs 128-, these are not yet supported:
56
+
57
+ |id | name |data type
58
+ |---|--------------|-----
59
+ |128|bgpNextAdjacentAsNumber|unsigned32
60
+ |129|bgpPrevAdjacentAsNumber|unsigned32
61
+ |130|exporterIPv4Address|ipv4Address
62
+ |131|exporterIPv6Address|ipv6Address
63
+ |132|droppedOctetDeltaCount|unsigned64
64
+ |133|droppedPacketDeltaCount|unsigned64
65
+ |134|droppedOctetTotalCount|unsigned64
66
+ |135|droppedPacketTotalCount|unsigned64
67
+ |137|commonPropertiesId|unsigned64
68
+ |138|observationPointId|unsigned64
69
+ |139|icmpTypeCodeIPv6|unsigned16
70
+ |140|mplsTopLabelIPv6Address|ipv6Address
71
+ |141|lineCardId|unsigned32
72
+ |142|portId|unsigned32
73
+ |143|meteringProcessId|unsigned32
74
+ |144|exportingProcessId|unsigned32
75
+ |145|templateId|unsigned16
76
+ |146|wlanChannelId|unsigned8
77
+ |149|observationDomainId|unsigned32
78
+ |150|flowStartSeconds|dateTimeSeconds
79
+ |151|flowEndSeconds|dateTimeSeconds
80
+ |153|flowEndMilliseconds|dateTimeMilliseconds
81
+ |154|flowStartMicroseconds|dateTimeMicroseconds
82
+ |155|flowEndMicroseconds|dateTimeMicroseconds
83
+ |156|flowStartNanoseconds|dateTimeNanoseconds
84
+ |157|flowEndNanoseconds|dateTimeNanoseconds
85
+ |158|flowStartDeltaMicroseconds|unsigned32
86
+ |159|flowEndDeltaMicroseconds|unsigned32
87
+ |160|systemInitTimeMilliseconds|dateTimeMilliseconds
88
+ |161|flowDurationMilliseconds|unsigned32
89
+ |162|flowDurationMicroseconds|unsigned32
90
+ |163|observedFlowTotalCount|unsigned64
91
+ |164|ignoredPacketTotalCount|unsigned64
92
+ |165|ignoredOctetTotalCount|unsigned64
93
+ |166|notSentFlowTotalCount|unsigned64
94
+ |167|notSentPacketTotalCount|unsigned64
95
+ |168|notSentOctetTotalCount|unsigned64
96
+ |169|destinationIPv6Prefix|ipv6Address
97
+ |170|sourceIPv6Prefix|ipv6Address
98
+ |171|postOctetTotalCount|unsigned64
99
+ |172|postPacketTotalCount|unsigned64
100
+ |173|flowKeyIndicator|unsigned64
101
+ |174|postMCastPacketTotalCount|unsigned64
102
+ |175|postMCastOctetTotalCount|unsigned64
103
+ |184|tcpSequenceNumber|unsigned32
104
+ |185|tcpAcknowledgementNumber|unsigned32
105
+ |186|tcpWindowSize|unsigned16
106
+ |187|tcpUrgentPointer|unsigned16
107
+ |188|tcpHeaderLength|unsigned8
108
+ |189|ipHeaderLength|unsigned8
109
+ |190|totalLengthIPv4|unsigned16
110
+ |191|payloadLengthIPv6|unsigned16
111
+ |192|ipTTL|unsigned8
112
+ |193|nextHeaderIPv6|unsigned8
113
+ |196|ipPrecedence|unsigned8
114
+ |197|fragmentFlags|unsigned8
115
+ |198|octetDeltaSumOfSquares|unsigned64
116
+ |199|octetTotalSumOfSquares|unsigned64
117
+ |200|mplsTopLabelTTL|unsigned8
118
+ |202|mplsLabelStackDepth|unsigned32
119
+ |203|mplsTopLabelExp|unsigned8
120
+ |204|ipPayloadLength|unsigned32
121
+ |205|udpMessageLength|unsigned16
122
+ |206|isMulticast|unsigned8
123
+ |207|ipv4IHL|unsigned8
124
+ |208|ipv4Options|unsigned32
125
+ |209|tcpOptions|unsigned64
126
+ |210|paddingOctets|octetArray
127
+ |211|collectorIPv4Address|ipv4Address
128
+ |212|collectorIPv6Address|ipv6Address
129
+ |213|exportInterface|unsigned32
130
+ |214|exportProtocolVersion|unsigned8
131
+ |215|exportTransportProtocol|unsigned8
132
+ |216|collectorTransportPort|unsigned16
133
+ |217|exporterTransportPort|unsigned16
134
+ |218|tcpSynTotalCount|unsigned64
135
+ |219|tcpFinTotalCount|unsigned64
136
+ |220|tcpRstTotalCount|unsigned64
137
+ |221|tcpPshTotalCount|unsigned64
138
+ |222|tcpAckTotalCount|unsigned64
139
+ |223|tcpUrgTotalCount|unsigned64
140
+ |224|ipTotalLength|unsigned64
141
+ |229|natOriginatingAddressRealm|unsigned8
142
+ |230|natEvent|unsigned8
143
+ |237|postMplsTopLabelExp|unsigned8
144
+ |238|tcpWindowScale|unsigned16
145
+ |239|biflowDirection|unsigned8
146
+ |240|ethernetHeaderLength|unsigned8
147
+ |241|ethernetPayloadLength|unsigned16
148
+ |242|ethernetTotalLength|unsigned16
149
+ |243|dot1qVlanId|unsigned16
150
+ |244|dot1qPriority|unsigned8
151
+ |245|dot1qCustomerVlanId|unsigned16
152
+ |246|dot1qCustomerPriority|unsigned8
153
+ |247|metroEvcId|string
154
+ |248|metroEvcType|unsigned8
155
+ |249|pseudoWireId|unsigned32
156
+ |250|pseudoWireType|unsigned16
157
+ |251|pseudoWireControlWord|unsigned32
158
+ |252|ingressPhysicalInterface|unsigned32
159
+ |253|egressPhysicalInterface|unsigned32
160
+ |254|postDot1qVlanId|unsigned16
161
+ |255|postDot1qCustomerVlanId|unsigned16
162
+ |256|ethernetType|unsigned16
163
+ |257|postIpPrecedence|unsigned8
164
+ |258|collectionTimeMilliseconds|dateTimeMilliseconds
165
+ |259|exportSctpStreamId|unsigned16
166
+ |260|maxExportSeconds|dateTimeSeconds
167
+ |261|maxFlowEndSeconds|dateTimeSeconds
168
+ |262|messageMD5Checksum|octetArray
169
+ |263|messageScope|unsigned8
170
+ |264|minExportSeconds|dateTimeSeconds
171
+ |265|minFlowStartSeconds|dateTimeSeconds
172
+ |266|opaqueOctets|octetArray
173
+ |267|sessionScope|unsigned8
174
+ |268|maxFlowEndMicroseconds|dateTimeMicroseconds
175
+ |269|maxFlowEndMilliseconds|dateTimeMilliseconds
176
+ |270|maxFlowEndNanoseconds|dateTimeNanoseconds
177
+ |271|minFlowStartMicroseconds|dateTimeMicroseconds
178
+ |272|minFlowStartMilliseconds|dateTimeMilliseconds
179
+ |273|minFlowStartNanoseconds|dateTimeNanoseconds
180
+ |274|collectorCertificate|octetArray
181
+ |275|exporterCertificate|octetArray
182
+ |276|dataRecordsReliability|boolean
183
+ |277|observationPointType|unsigned8
184
+ |278|newConnectionDeltaCount|unsigned32
185
+ |279|connectionSumDurationSeconds|unsigned64
186
+ |280|connectionTransactionId|unsigned64
187
+ |283|natPoolId|unsigned32
188
+ |284|natPoolName|string
189
+ |285|anonymizationFlags|unsigned16
190
+ |286|anonymizationTechnique|unsigned16
191
+ |287|informationElementIndex|unsigned16
192
+ |288|p2pTechnology|string
193
+ |289|tunnelTechnology|string
194
+ |290|encryptedTechnology|string
195
+ |291|basicList|basicList
196
+ |292|subTemplateList|subTemplateList
197
+ |293|subTemplateMultiList|subTemplateMultiList
198
+ |294|bgpValidityState|unsigned8
199
+ |295|IPSecSPI|unsigned32
200
+ |296|greKey|unsigned32
201
+ |297|natType|unsigned8
202
+ |300|observationDomainName|string
203
+ |301|selectionSequenceId|unsigned64
204
+ |302|selectorId|unsigned64
205
+ |303|informationElementId|unsigned16
206
+ |304|selectorAlgorithm|unsigned16
207
+ |305|samplingPacketInterval|unsigned32
208
+ |306|samplingPacketSpace|unsigned32
209
+ |307|samplingTimeInterval|unsigned32
210
+ |308|samplingTimeSpace|unsigned32
211
+ |309|samplingSize|unsigned32
212
+ |310|samplingPopulation|unsigned32
213
+ |311|samplingProbability|float64
214
+ |312|dataLinkFrameSize|unsigned16
215
+ |313|ipHeaderPacketSection|octetArray
216
+ |314|ipPayloadPacketSection|octetArray
217
+ |315|dataLinkFrameSection|octetArray
218
+ |316|mplsLabelStackSection|octetArray
219
+ |317|mplsPayloadPacketSection|octetArray
220
+ |318|selectorIdTotalPktsObserved|unsigned64
221
+ |319|selectorIdTotalPktsSelected|unsigned64
222
+ |320|absoluteError|float64
223
+ |321|relativeError|float64
224
+ |322|observationTimeSeconds|dateTimeSeconds
225
+ |324|observationTimeMicroseconds|dateTimeMicroseconds
226
+ |325|observationTimeNanoseconds|dateTimeNanoseconds
227
+ |326|digestHashValue|unsigned64
228
+ |327|hashIPPayloadOffset|unsigned64
229
+ |328|hashIPPayloadSize|unsigned64
230
+ |329|hashOutputRangeMin|unsigned64
231
+ |330|hashOutputRangeMax|unsigned64
232
+ |331|hashSelectedRangeMin|unsigned64
233
+ |332|hashSelectedRangeMax|unsigned64
234
+ |333|hashDigestOutput|boolean
235
+ |334|hashInitialiserValue|unsigned64
236
+ |335|selectorName|string
237
+ |336|upperCILimit|float64
238
+ |337|lowerCILimit|float64
239
+ |338|confidenceLevel|float64
240
+ |339|informationElementDataType|unsigned8
241
+ |340|informationElementDescription|string
242
+ |341|informationElementName|string
243
+ |342|informationElementRangeBegin|unsigned64
244
+ |343|informationElementRangeEnd|unsigned64
245
+ |344|informationElementSemantics|unsigned8
246
+ |345|informationElementUnits|unsigned16
247
+ |346|privateEnterpriseNumber|unsigned32
248
+ |347|virtualStationInterfaceId|octetArray
249
+ |348|virtualStationInterfaceName|string
250
+ |349|virtualStationUUID|octetArray
251
+ |350|virtualStationName|string
252
+ |351|layer2SegmentId|unsigned64
253
+ |352|layer2OctetDeltaCount|unsigned64
254
+ |353|layer2OctetTotalCount|unsigned64
255
+ |354|ingressUnicastPacketTotalCount|unsigned64
256
+ |355|ingressMulticastPacketTotalCount|unsigned64
257
+ |356|ingressBroadcastPacketTotalCount|unsigned64
258
+ |357|egressUnicastPacketTotalCount|unsigned64
259
+ |358|egressBroadcastPacketTotalCount|unsigned64
260
+ |359|monitoringIntervalStartMilliSeconds|dateTimeMilliseconds
261
+ |360|monitoringIntervalEndMilliSeconds|dateTimeMilliseconds
262
+ |363|portRangeStepSize|unsigned16
263
+ |364|portRangeNumPorts|unsigned16
264
+ |368|ingressInterfaceType|unsigned32
265
+ |369|egressInterfaceType|unsigned32
266
+ |370|rtpSequenceNumber|unsigned16
267
+ |371|userName|string
268
+ |372|applicationCategoryName|string
269
+ |373|applicationSubCategoryName|string
270
+ |374|applicationGroupName|string
271
+ |375|originalFlowsPresent|unsigned64
272
+ |376|originalFlowsInitiated|unsigned64
273
+ |377|originalFlowsCompleted|unsigned64
274
+ |378|distinctCountOfSourceIPAddress|unsigned64
275
+ |379|distinctCountOfDestinationIPAddress|unsigned64
276
+ |380|distinctCountOfSourceIPv4Address|unsigned32
277
+ |381|distinctCountOfDestinationIPv4Address|unsigned32
278
+ |382|distinctCountOfSourceIPv6Address|unsigned64
279
+ |383|distinctCountOfDestinationIPv6Address|unsigned64
280
+ |384|valueDistributionMethod|unsigned8
281
+ |385|rfc3550JitterMilliseconds|unsigned32
282
+ |386|rfc3550JitterMicroseconds|unsigned32
283
+ |387|rfc3550JitterNanoseconds|unsigned32
284
+ |388|dot1qDEI|boolean
285
+ |389|dot1qCustomerDEI|boolean
286
+ |390|flowSelectorAlgorithm|unsigned16
287
+ |391|flowSelectedOctetDeltaCount|unsigned64
288
+ |392|flowSelectedPacketDeltaCount|unsigned64
289
+ |393|flowSelectedFlowDeltaCount|unsigned64
290
+ |394|selectorIDTotalFlowsObserved|unsigned64
291
+ |395|selectorIDTotalFlowsSelected|unsigned64
292
+ |396|samplingFlowInterval|unsigned64
293
+ |397|samplingFlowSpacing|unsigned64
294
+ |398|flowSamplingTimeInterval|unsigned64
295
+ |399|flowSamplingTimeSpacing|unsigned64
296
+ |400|hashFlowDomain|unsigned16
297
+ |401|transportOctetDeltaCount|unsigned64
298
+ |402|transportPacketDeltaCount|unsigned64
299
+ |403|originalExporterIPv4Address|ipv4Address
300
+ |404|originalExporterIPv6Address|ipv6Address
301
+ |405|originalObservationDomainId|unsigned32
302
+ |406|intermediateProcessId|unsigned32
303
+ |407|ignoredDataRecordTotalCount|unsigned64
304
+ |408|dataLinkFrameType|unsigned16
305
+ |409|sectionOffset|unsigned16
306
+ |410|sectionExportedOctets|unsigned16
307
+ |411|dot1qServiceInstanceTag|octetArray
308
+ |412|dot1qServiceInstanceId|unsigned32
309
+ |413|dot1qServiceInstancePriority|unsigned8
310
+ |414|dot1qCustomerSourceMacAddress|macAddress
311
+ |415|dot1qCustomerDestinationMacAddress|macAddress
312
+ |416||
313
+ |417|postLayer2OctetDeltaCount|unsigned64
314
+ |418|postMCastLayer2OctetDeltaCount|unsigned64
315
+ |419||
316
+ |420|postLayer2OctetTotalCount|unsigned64
317
+ |421|postMCastLayer2OctetTotalCount|unsigned64
318
+ |422|minimumLayer2TotalLength|unsigned64
319
+ |423|maximumLayer2TotalLength|unsigned64
320
+ |424|droppedLayer2OctetDeltaCount|unsigned64
321
+ |425|droppedLayer2OctetTotalCount|unsigned64
322
+ |426|ignoredLayer2OctetTotalCount|unsigned64
323
+ |427|notSentLayer2OctetTotalCount|unsigned64
324
+ |428|layer2OctetDeltaSumOfSquares|unsigned64
325
+ |429|layer2OctetTotalSumOfSquares|unsigned64
326
+ |430|layer2FrameDeltaCount|unsigned64
327
+ |431|layer2FrameTotalCount|unsigned64
328
+ |432|pseudoWireDestinationIPv4Address|ipv4Address
329
+ |433|ignoredLayer2FrameTotalCount|unsigned64
330
+ |434|mibObjectValueInteger|signed32
331
+ |435|mibObjectValueOctetString|octetArray
332
+ |436|mibObjectValueOID|octetArray
333
+ |437|mibObjectValueBits|octetArray
334
+ |438|mibObjectValueIPAddress|ipv4Address
335
+ |439|mibObjectValueCounter|unsigned64
336
+ |440|mibObjectValueGauge|unsigned32
337
+ |441|mibObjectValueTimeTicks|unsigned32
338
+ |442|mibObjectValueUnsigned|unsigned32
339
+ |443|mibObjectValueTable|subTemplateList
340
+ |444|mibObjectValueRow|subTemplateList
341
+ |445|mibObjectIdentifier|octetArray
342
+ |446|mibSubIdentifier|unsigned32
343
+ |447|mibIndexIndicator|unsigned64
344
+ |448|mibCaptureTimeSemantics|unsigned8
345
+ |449|mibContextEngineID|octetArray
346
+ |450|mibContextName|string
347
+ |451|mibObjectName|string
348
+ |452|mibObjectDescription|string
349
+ |453|mibObjectSyntax|string
350
+ |454|mibModuleName|string
351
+ |455|mobileIMSI|string
352
+ |456|mobileMSISDN|string
353
+ |457|httpStatusCode|unsigned16
354
+ |458|sourceTransportPortsLimit|unsigned16
355
+ |459|httpRequestMethod|string
356
+ |460|httpRequestHost|string
357
+ |461|httpRequestTarget|string
358
+ |462|httpMessageVersion|string
359
+ |463|natInstanceID|unsigned32
360
+ |464|internalAddressRealm|octetArray
361
+ |465|externalAddressRealm|octetArray
362
+ |466|natQuotaExceededEvent|unsigned32
363
+ |467|natThresholdEvent|unsigned32
364
+ |468|httpUserAgent|string
365
+ |469|httpContentType|string
366
+ |470|httpReasonPhrase|string
367
+
368
+ ## RFC 3954 collector compliance details
369
+
370
+ The tables below detail the collector-relevant requirements, and whether or not they are implemented:
371
+
372
+ ### 5. Export packet format
373
+
374
+ | Requirement |MUST |SHOULD| MAY|
375
+ |---------------------------------------|-----|-----|-----|
376
+ | 5.1 Incremental sequence counter of all Export Packets sent from the current Observation Domain by the Exporter. This value MUST be cumulative, and SHOULD be used by the Collector to identify whether any Export Packets have been missed. | | NO | |
377
+ | 5.1 NetFlow Collectors SHOULD use the combination of the source IP address and the Source ID field to separate different export streams originating from the same Exporter. | | NO | |
378
+ | 5.3 The Collector MUST use the FlowSet ID to find the corresponding Template Record and decode the Flow Records from the FlowSet. | YES | | |
379
+
380
+ ### 6. Options
381
+
382
+ | Requirement |MUST |SHOULD| MAY|
383
+ |---------------------------------------|-----|-----|-----|
384
+ | 6.2 The Collector MUST use the FlowSet ID to map the appropriate type and length to any field values that follow. | YES | | |
385
+
386
+ ### 7. Template management
387
+
388
+ | Requirement |MUST |SHOULD| MAY|
389
+ |---------------------------------------|-----|-----|-----|
390
+ | 7. the NetFlow Collector MUST store the Template Record to interpret the corresponding Flow Data Records that are received in subsequent data packets. | YES | | |
391
+ | 7. A NetFlow Collector that receives Export Packets from several Observation Domains from the same Exporter MUST be aware that the uniqueness of the Template ID is not guaranteed across Observation Domains. | YES | | |
392
+ | 7. If a Collector should receive a new definition for an already existing Template ID, it MUST discard the previous template definition and use the new one. | YES | | |
393
+
394
+ ### 9. The collector side
395
+
396
+ | Requirement |MUST |SHOULD| MAY|
397
+ |---------------------------------------|-----|-----|-----|
398
+ | 9. If the Template Records have not been received at the time Flow Data Records (or Options Data Records) are received, the Collector SHOULD store the Flow Data Records (or Options Data Records) and decode them after the Template Records are received. | | NO | |
399
+ | 9. A Collector device MUST NOT assume that the Data FlowSet and the associated Template FlowSet (or Options Template FlowSet) are exported in the same Export Packet. | YES | | |
400
+ | 9. The Collector MUST NOT assume that one and only one Template FlowSet is present in an Export Packet. | YES | | |
401
+ | 9. The Collector MUST NOT attempt to decode the Flow or Options Data Records with an expired Template. | YES | | |
402
+ | 9. At any given time the Collector SHOULD maintain the following for all the current Template Records and Options Template Records: Exporter, Observation Domain, Template ID, Template Definition, Last Received. | | NO | |
403
+ | 9. In the event of a clock configuration change on the Exporter, the Collector SHOULD discard all Template Records and Options Template Records associated with that Exporter, in order for Collector to learn the new set of fields: Exporter, Observation Domain, Template ID, Template Definition, Last Received. | | NO | |
404
+ | 9. If the Collector receives a new Template Record (for example, in the case of an Exporter restart) it MUST immediately override the existing Template Record. | YES | | |
405
+ | 9. Finally, note that the Collector MUST accept padding in the Data FlowSet and Options Template FlowSet, which means for the Flow Data Records, the Options Data Records and the Template Records. | YES | | |
406
+
407
+
data/docs/index.asciidoc CHANGED
@@ -42,10 +42,10 @@ The following Netflow/IPFIX exporters are known to work with the most recent ver
42
42
  |Cisco WLC | | y | |
43
43
  |Citrix Netscaler | | | y | Still some unknown fields, labeled netscalerUnknown<id>
44
44
  |fprobe | y | | |
45
- |Fortigate FortiOS 5.2 | | y | |
45
+ |Fortigate FortiOS | | y | |
46
46
  |ipt_NETFLOW | y | y | y |
47
47
  |Juniper MX80 | y | | | SW > 12.3R8
48
- |Mikrotik 6.35.4 | y | | n | http://wiki.mikrotik.com/wiki/Manual:IP/Traffic_Flow
48
+ |Mikrotik | y | | y | http://wiki.mikrotik.com/wiki/Manual:IP/Traffic_Flow
49
49
  |nProbe | y | y | y |
50
50
  |OpenBSD pflow | y | n | y | http://man.openbsd.org/OpenBSD-current/man4/pflow.4
51
51
  |Softflowd | y | y | y | IPFIX supported in https://github.com/djmdjm/softflowd
@@ -62,12 +62,31 @@ Example Logstash configuration that will listen on 2055/udp for Netflow v5,v9 an
62
62
  --------------------------
63
63
  input {
64
64
  udp {
65
- port => 2055
65
+ port => 2055
66
66
  codec => netflow
67
67
  }
68
68
  }
69
69
  --------------------------
70
70
 
71
+ For high-performance production environments the configuration below will decode up to 6000 flows/sec on an 8 CPU instance. If your total flowrate exceeds 6000 flows/sec, you should use multiple Logstash instances.
72
+
73
+
74
+ [source, ruby]
75
+ --------------------------
76
+ input {
77
+ udp {
78
+ port => 2055
79
+ receive_buffer_bytes => 16777216
80
+ codec => netflow
81
+ workers => 6
82
+ }
83
+ --------------------------
84
+
85
+ Make sure to increase the Linux kernel receive buffer limit:
86
+
87
+ # sysctl -w net.core.rmem_max=$((1024*1024*16))
88
+
89
+
71
90
  [id="plugins-{type}s-{plugin}-options"]
72
91
  ==== Netflow Codec Configuration Options
73
92
 
@@ -107,7 +126,7 @@ Template caches are saved as:
107
126
  * Value type is <<number,number>>
108
127
  * Default value is `4000`
109
128
 
110
- Netflow v9/v10 template cache TTL (minutes)
129
+ Netflow v9/v10 template cache TTL (seconds)
111
130
 
112
131
  [id="plugins-{type}s-{plugin}-include_flowset_id"]
113
132
  ===== `include_flowset_id`
data/lib/logstash/codecs/netflow/util.rb CHANGED
@@ -205,7 +205,7 @@ class NetflowOptionFlowset < BinData::Record
205
205
  uint16 :field_type
206
206
  uint16 :field_length, :assert => lambda { field_length > 0 }
207
207
  end
208
- string :padding, :read_length => lambda { flowset_length - 4 - scope_length - option_length - 2 - 2 -2}
208
+ string :padding, :read_length => lambda { flowset_length - 4 - scope_length - option_length - 2 - 2 - 2 }
209
209
  end
210
210
  end
211
211
 
data/logstash-codec-netflow.gemspec CHANGED
@@ -1,7 +1,7 @@
1
1
  Gem::Specification.new do |s|
2
2
 
3
3
  s.name = 'logstash-codec-netflow'
4
- s.version = '3.5.1'
4
+ s.version = '3.5.2'
5
5
  s.licenses = ['Apache License (2.0)']
6
6
  s.summary = "The netflow codec is for decoding Netflow v5/v9/v10 (IPFIX) flows."
7
7
  s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
data/spec/codecs/netflow_spec.rb CHANGED
@@ -1215,6 +1215,80 @@ describe LogStash::Codecs::Netflow do
1215
1215
 
1216
1216
  end
1217
1217
 
1218
+ context "IPFIX Mikrotik RouterOS 6.39.2" do
1219
+ let(:data) do
1220
+ packets = []
1221
+ packets << IO.read(File.join(File.dirname(__FILE__), "ipfix_test_mikrotik_tpl.dat"), :mode => "rb")
1222
+ packets << IO.read(File.join(File.dirname(__FILE__), "ipfix_test_mikrotik_data258.dat"), :mode => "rb")
1223
+ packets << IO.read(File.join(File.dirname(__FILE__), "ipfix_test_mikrotik_data259.dat"), :mode => "rb")
1224
+ end
1225
+
1226
+ let(:json_events) do
1227
+ events = []
1228
+ events << <<-END
1229
+ {
1230
+ "netflow": {
1231
+ "destinationIPv4Address": "192.168.128.17",
1232
+ "destinationTransportPort": 123,
1233
+ "flowStartSysUpTime": 2666794170,
1234
+ "tcpControlBits": 0,
1235
+ "postNATDestinationIPv4Address": "192.168.128.17",
1236
+ "flowEndSysUpTime": 2666794170,
1237
+ "sourceIPv4Address": "10.10.8.197",
1238
+ "ingressInterface": 13,
1239
+ "version": 10,
1240
+ "packetDeltaCount": 2,
1241
+ "ipVersion": 4,
1242
+ "protocolIdentifier": 17,
1243
+ "postNATSourceIPv4Address": "192.168.230.216",
1244
+ "egressInterface": 7,
1245
+ "octetDeltaCount": 152,
1246
+ "ipNextHopIPv4Address": "192.168.224.1",
1247
+ "sourceTransportPort": 123
1248
+ },
1249
+ "@timestamp": "2017-07-19T16:18:08.000Z",
1250
+ "@version": "1"
1251
+ }
1252
+ END
1253
+
1254
+ events << <<-END
1255
+ {
1256
+ "netflow": {
1257
+ "destinationTransportPort": 5678,
1258
+ "ipNextHopIPv6Address": "ff02::1",
1259
+ "flowStartSysUpTime": 2666795750,
1260
+ "tcpControlBits": 0,
1261
+ "flowEndSysUpTime": 2666795750,
1262
+ "ingressInterface": 17,
1263
+ "version": 10,
1264
+ "packetDeltaCount": 2,
1265
+ "sourceIPv6Address": "fe80::ff:fe00:1201",
1266
+ "ipVersion": 6,
1267
+ "protocolIdentifier": 17,
1268
+ "egressInterface": 0,
1269
+ "octetDeltaCount": 370,
1270
+ "sourceTransportPort": 5678,
1271
+ "destinationIPv6Address": "fe80::ff:fe00:1201"
1272
+ },
1273
+ "@timestamp": "2017-07-19T16:18:08.000Z",
1274
+ "@version": "1"
1275
+ }
1276
+ END
1277
+ events.map{|event| event.gsub(/\s+/, "")}
1278
+ end
1279
+
1280
+ it "should decode raw data" do
1281
+ expect(decode.size).to eq(46)
1282
+ expect(decode[0].get("[netflow][postNATDestinationIPv4Address]")).to eq("192.168.128.17")
1283
+ expect(decode[45].get("[netflow][ipNextHopIPv6Address]")).to eq("ff02::1")
1284
+ end
1285
+
1286
+ it "should serialize to json" do
1287
+ expect(JSON.parse(decode[0].to_json)).to eq(JSON.parse(json_events[0]))
1288
+ expect(JSON.parse(decode[45].to_json)).to eq(JSON.parse(json_events[1]))
1289
+ end
1290
+
1291
+ end
1218
1292
 
1219
1293
  context "IPFIX Netscaler with variable length fields" do
1220
1294
  let(:data) do
metadata CHANGED
@@ -1,109 +1,109 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-codec-netflow
3
3
  version: !ruby/object:Gem::Version
4
- version: 3.5.1
4
+ version: 3.5.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - Elastic
8
- autorequire:
8
+ autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2017-07-18 00:00:00.000000000 Z
11
+ date: 2017-08-15 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
- name: logstash-core-plugin-api
15
14
  requirement: !ruby/object:Gem::Requirement
16
15
  requirements:
17
- - - '>='
16
+ - - ">="
18
17
  - !ruby/object:Gem::Version
19
18
  version: '1.60'
20
- - - <=
19
+ - - "<="
21
20
  - !ruby/object:Gem::Version
22
21
  version: '2.99'
23
- type: :runtime
22
+ name: logstash-core-plugin-api
24
23
  prerelease: false
24
+ type: :runtime
25
25
  version_requirements: !ruby/object:Gem::Requirement
26
26
  requirements:
27
- - - '>='
27
+ - - ">="
28
28
  - !ruby/object:Gem::Version
29
29
  version: '1.60'
30
- - - <=
30
+ - - "<="
31
31
  - !ruby/object:Gem::Version
32
32
  version: '2.99'
33
33
  - !ruby/object:Gem::Dependency
34
- name: bindata
35
34
  requirement: !ruby/object:Gem::Requirement
36
35
  requirements:
37
- - - '>='
36
+ - - ">="
38
37
  - !ruby/object:Gem::Version
39
38
  version: 1.5.0
40
- type: :runtime
39
+ name: bindata
41
40
  prerelease: false
41
+ type: :runtime
42
42
  version_requirements: !ruby/object:Gem::Requirement
43
43
  requirements:
44
- - - '>='
44
+ - - ">="
45
45
  - !ruby/object:Gem::Version
46
46
  version: 1.5.0
47
47
  - !ruby/object:Gem::Dependency
48
- name: logstash-devutils
49
48
  requirement: !ruby/object:Gem::Requirement
50
49
  requirements:
51
- - - '>='
50
+ - - ">="
52
51
  - !ruby/object:Gem::Version
53
52
  version: 1.0.0
54
- type: :development
53
+ name: logstash-devutils
55
54
  prerelease: false
55
+ type: :development
56
56
  version_requirements: !ruby/object:Gem::Requirement
57
57
  requirements:
58
- - - '>='
58
+ - - ">="
59
59
  - !ruby/object:Gem::Version
60
60
  version: 1.0.0
61
- description: This gem is a Logstash plugin required to be installed on top of the
62
- Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This
63
- gem is not a stand-alone program
61
+ description: This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program
64
62
  email: info@elastic.co
65
63
  executables: []
66
64
  extensions: []
67
65
  extra_rdoc_files: []
68
66
  files:
67
+ - CHANGELOG.md
68
+ - CONTRIBUTORS
69
+ - Gemfile
70
+ - LICENSE
71
+ - NOTICE.TXT
72
+ - README.md
73
+ - RFC_COMPLIANCE_IPFIX.md
74
+ - RFC_COMPLIANCE_NETFLOW_v9.md
75
+ - docs/index.asciidoc
76
+ - lib/logstash/codecs/netflow.rb
69
77
  - lib/logstash/codecs/netflow/iana2yaml.rb
70
78
  - lib/logstash/codecs/netflow/ipfix.yaml
71
- - lib/logstash/codecs/netflow/util.rb
72
79
  - lib/logstash/codecs/netflow/netflow.yaml
73
- - lib/logstash/codecs/netflow.rb
80
+ - lib/logstash/codecs/netflow/util.rb
81
+ - logstash-codec-netflow.gemspec
74
82
  - spec/codecs/ipfix.dat
83
+ - spec/codecs/ipfix_test_barracuda_data256.dat
84
+ - spec/codecs/ipfix_test_barracuda_tpl.dat
85
+ - spec/codecs/ipfix_test_mikrotik_data258.dat
86
+ - spec/codecs/ipfix_test_mikrotik_data259.dat
87
+ - spec/codecs/ipfix_test_mikrotik_tpl.dat
88
+ - spec/codecs/ipfix_test_netscaler_data.dat
89
+ - spec/codecs/ipfix_test_netscaler_tpl.dat
75
90
  - spec/codecs/ipfix_test_openbsd_pflow_data.dat
76
91
  - spec/codecs/ipfix_test_openbsd_pflow_tpl.dat
92
+ - spec/codecs/ipfix_test_vmware_vds_data264.dat
93
+ - spec/codecs/ipfix_test_vmware_vds_data266.dat
94
+ - spec/codecs/ipfix_test_vmware_vds_data266_267.dat
95
+ - spec/codecs/ipfix_test_vmware_vds_tpl.dat
77
96
  - spec/codecs/netflow5.dat
78
97
  - spec/codecs/netflow5_test_invalid01.dat
79
98
  - spec/codecs/netflow5_test_invalid02.dat
80
99
  - spec/codecs/netflow5_test_juniper_mx80.dat
81
100
  - spec/codecs/netflow5_test_microtik.dat
101
+ - spec/codecs/netflow9_test_0length_fields_tpl_data.dat
82
102
  - spec/codecs/netflow9_test_cisco_asa_1_data.dat
83
103
  - spec/codecs/netflow9_test_cisco_asa_1_tpl.dat
84
104
  - spec/codecs/netflow9_test_cisco_asa_2_data.dat
85
105
  - spec/codecs/netflow9_test_cisco_asa_2_tpl_26x.dat
86
106
  - spec/codecs/netflow9_test_cisco_asa_2_tpl_27x.dat
87
- - spec/codecs/netflow9_test_invalid01.dat
88
- - spec/codecs/netflow9_test_macaddr_data.dat
89
- - spec/codecs/netflow9_test_macaddr_tpl.dat
90
- - spec/codecs/netflow9_test_nprobe_data.dat
91
- - spec/codecs/netflow9_test_nprobe_tpl.dat
92
- - spec/codecs/netflow9_test_softflowd_tpl_data.dat
93
- - spec/codecs/netflow9_test_valid01.dat
94
- - spec/codecs/netflow9_test_ubnt_edgerouter_tpl.dat
95
- - spec/codecs/netflow9_test_ubnt_edgerouter_data1024.dat
96
- - spec/codecs/netflow9_test_ubnt_edgerouter_data1025.dat
97
- - spec/codecs/ipfix_test_netscaler_data.dat
98
- - spec/codecs/ipfix_test_netscaler_tpl.dat
99
- - spec/codecs/ipfix_test_vmware_vds_data264.dat
100
- - spec/codecs/ipfix_test_vmware_vds_data266.dat
101
- - spec/codecs/ipfix_test_vmware_vds_data266_267.dat
102
- - spec/codecs/ipfix_test_vmware_vds_tpl.dat
103
- - spec/codecs/netflow9_test_0length_fields_tpl_data.dat
104
- - spec/codecs/netflow9_test_juniper_srx_tplopt.dat
105
- - spec/codecs/netflow9_test_streamcore_tpl_data256.dat
106
- - spec/codecs/netflow9_test_streamcore_tpl_data260.dat
107
107
  - spec/codecs/netflow9_test_cisco_asr9k_data256.dat
108
108
  - spec/codecs/netflow9_test_cisco_asr9k_data260.dat
109
109
  - spec/codecs/netflow9_test_cisco_asr9k_opttpl256.dat
@@ -114,82 +114,77 @@ files:
114
114
  - spec/codecs/netflow9_test_cisco_nbar_data262.dat
115
115
  - spec/codecs/netflow9_test_cisco_nbar_opttpl260.dat
116
116
  - spec/codecs/netflow9_test_cisco_nbar_tpl262.dat
117
- - spec/codecs/netflow9_test_cisco_wlc_tpl.dat
118
117
  - spec/codecs/netflow9_test_cisco_wlc_data261.dat
119
- - spec/codecs/ipfix_test_barracuda_tpl.dat
120
- - spec/codecs/ipfix_test_barracuda_data256.dat
118
+ - spec/codecs/netflow9_test_cisco_wlc_tpl.dat
119
+ - spec/codecs/netflow9_test_fortigate_fortios_521_data256.dat
121
120
  - spec/codecs/netflow9_test_fortigate_fortios_521_data257.dat
122
- - spec/codecs/netflow_spec.rb
123
121
  - spec/codecs/netflow9_test_fortigate_fortios_521_tpl.dat
124
- - spec/codecs/netflow9_test_fortigate_fortios_521_data256.dat
125
- - logstash-codec-netflow.gemspec
126
- - README.md
127
- - CHANGELOG.md
128
- - CONTRIBUTORS
129
- - Gemfile
130
- - LICENSE
131
- - NOTICE.TXT
132
- - docs/index.asciidoc
122
+ - spec/codecs/netflow9_test_invalid01.dat
123
+ - spec/codecs/netflow9_test_juniper_srx_tplopt.dat
124
+ - spec/codecs/netflow9_test_macaddr_data.dat
125
+ - spec/codecs/netflow9_test_macaddr_tpl.dat
126
+ - spec/codecs/netflow9_test_nprobe_data.dat
127
+ - spec/codecs/netflow9_test_nprobe_tpl.dat
128
+ - spec/codecs/netflow9_test_softflowd_tpl_data.dat
129
+ - spec/codecs/netflow9_test_streamcore_tpl_data256.dat
130
+ - spec/codecs/netflow9_test_streamcore_tpl_data260.dat
131
+ - spec/codecs/netflow9_test_ubnt_edgerouter_data1024.dat
132
+ - spec/codecs/netflow9_test_ubnt_edgerouter_data1025.dat
133
+ - spec/codecs/netflow9_test_ubnt_edgerouter_tpl.dat
134
+ - spec/codecs/netflow9_test_valid01.dat
135
+ - spec/codecs/netflow_spec.rb
133
136
  homepage: http://www.elastic.co/guide/en/logstash/current/index.html
134
137
  licenses:
135
138
  - Apache License (2.0)
136
139
  metadata:
137
140
  logstash_plugin: 'true'
138
141
  logstash_group: codec
139
- post_install_message:
142
+ post_install_message:
140
143
  rdoc_options: []
141
144
  require_paths:
142
145
  - lib
143
146
  required_ruby_version: !ruby/object:Gem::Requirement
144
147
  requirements:
145
- - - '>='
148
+ - - ">="
146
149
  - !ruby/object:Gem::Version
147
150
  version: '0'
148
151
  required_rubygems_version: !ruby/object:Gem::Requirement
149
152
  requirements:
150
- - - '>='
153
+ - - ">="
151
154
  - !ruby/object:Gem::Version
152
155
  version: '0'
153
156
  requirements: []
154
- rubyforge_project:
155
- rubygems_version: 2.0.14
156
- signing_key:
157
+ rubyforge_project:
158
+ rubygems_version: 2.4.8
159
+ signing_key:
157
160
  specification_version: 4
158
161
  summary: The netflow codec is for decoding Netflow v5/v9/v10 (IPFIX) flows.
159
162
  test_files:
160
163
  - spec/codecs/ipfix.dat
164
+ - spec/codecs/ipfix_test_barracuda_data256.dat
165
+ - spec/codecs/ipfix_test_barracuda_tpl.dat
166
+ - spec/codecs/ipfix_test_mikrotik_data258.dat
167
+ - spec/codecs/ipfix_test_mikrotik_data259.dat
168
+ - spec/codecs/ipfix_test_mikrotik_tpl.dat
169
+ - spec/codecs/ipfix_test_netscaler_data.dat
170
+ - spec/codecs/ipfix_test_netscaler_tpl.dat
161
171
  - spec/codecs/ipfix_test_openbsd_pflow_data.dat
162
172
  - spec/codecs/ipfix_test_openbsd_pflow_tpl.dat
173
+ - spec/codecs/ipfix_test_vmware_vds_data264.dat
174
+ - spec/codecs/ipfix_test_vmware_vds_data266.dat
175
+ - spec/codecs/ipfix_test_vmware_vds_data266_267.dat
176
+ - spec/codecs/ipfix_test_vmware_vds_tpl.dat
163
177
  - spec/codecs/netflow5.dat
164
178
  - spec/codecs/netflow5_test_invalid01.dat
165
179
  - spec/codecs/netflow5_test_invalid02.dat
166
180
  - spec/codecs/netflow5_test_juniper_mx80.dat
167
181
  - spec/codecs/netflow5_test_microtik.dat
182
+ - spec/codecs/netflow9_test_0length_fields_tpl_data.dat
168
183
  - spec/codecs/netflow9_test_cisco_asa_1_data.dat
169
184
  - spec/codecs/netflow9_test_cisco_asa_1_tpl.dat
170
185
  - spec/codecs/netflow9_test_cisco_asa_2_data.dat
171
186
  - spec/codecs/netflow9_test_cisco_asa_2_tpl_26x.dat
172
187
  - spec/codecs/netflow9_test_cisco_asa_2_tpl_27x.dat
173
- - spec/codecs/netflow9_test_invalid01.dat
174
- - spec/codecs/netflow9_test_macaddr_data.dat
175
- - spec/codecs/netflow9_test_macaddr_tpl.dat
176
- - spec/codecs/netflow9_test_nprobe_data.dat
177
- - spec/codecs/netflow9_test_nprobe_tpl.dat
178
- - spec/codecs/netflow9_test_softflowd_tpl_data.dat
179
- - spec/codecs/netflow9_test_valid01.dat
180
- - spec/codecs/netflow9_test_ubnt_edgerouter_tpl.dat
181
- - spec/codecs/netflow9_test_ubnt_edgerouter_data1024.dat
182
- - spec/codecs/netflow9_test_ubnt_edgerouter_data1025.dat
183
- - spec/codecs/ipfix_test_netscaler_data.dat
184
- - spec/codecs/ipfix_test_netscaler_tpl.dat
185
- - spec/codecs/ipfix_test_vmware_vds_data264.dat
186
- - spec/codecs/ipfix_test_vmware_vds_data266.dat
187
- - spec/codecs/ipfix_test_vmware_vds_data266_267.dat
188
- - spec/codecs/ipfix_test_vmware_vds_tpl.dat
189
- - spec/codecs/netflow9_test_0length_fields_tpl_data.dat
190
- - spec/codecs/netflow9_test_juniper_srx_tplopt.dat
191
- - spec/codecs/netflow9_test_streamcore_tpl_data256.dat
192
- - spec/codecs/netflow9_test_streamcore_tpl_data260.dat
193
188
  - spec/codecs/netflow9_test_cisco_asr9k_data256.dat
194
189
  - spec/codecs/netflow9_test_cisco_asr9k_data260.dat
195
190
  - spec/codecs/netflow9_test_cisco_asr9k_opttpl256.dat
@@ -200,11 +195,22 @@ test_files:
200
195
  - spec/codecs/netflow9_test_cisco_nbar_data262.dat
201
196
  - spec/codecs/netflow9_test_cisco_nbar_opttpl260.dat
202
197
  - spec/codecs/netflow9_test_cisco_nbar_tpl262.dat
203
- - spec/codecs/netflow9_test_cisco_wlc_tpl.dat
204
198
  - spec/codecs/netflow9_test_cisco_wlc_data261.dat
205
- - spec/codecs/ipfix_test_barracuda_tpl.dat
206
- - spec/codecs/ipfix_test_barracuda_data256.dat
199
+ - spec/codecs/netflow9_test_cisco_wlc_tpl.dat
200
+ - spec/codecs/netflow9_test_fortigate_fortios_521_data256.dat
207
201
  - spec/codecs/netflow9_test_fortigate_fortios_521_data257.dat
208
- - spec/codecs/netflow_spec.rb
209
202
  - spec/codecs/netflow9_test_fortigate_fortios_521_tpl.dat
210
- - spec/codecs/netflow9_test_fortigate_fortios_521_data256.dat
203
+ - spec/codecs/netflow9_test_invalid01.dat
204
+ - spec/codecs/netflow9_test_juniper_srx_tplopt.dat
205
+ - spec/codecs/netflow9_test_macaddr_data.dat
206
+ - spec/codecs/netflow9_test_macaddr_tpl.dat
207
+ - spec/codecs/netflow9_test_nprobe_data.dat
208
+ - spec/codecs/netflow9_test_nprobe_tpl.dat
209
+ - spec/codecs/netflow9_test_softflowd_tpl_data.dat
210
+ - spec/codecs/netflow9_test_streamcore_tpl_data256.dat
211
+ - spec/codecs/netflow9_test_streamcore_tpl_data260.dat
212
+ - spec/codecs/netflow9_test_ubnt_edgerouter_data1024.dat
213
+ - spec/codecs/netflow9_test_ubnt_edgerouter_data1025.dat
214
+ - spec/codecs/netflow9_test_ubnt_edgerouter_tpl.dat
215
+ - spec/codecs/netflow9_test_valid01.dat
216
+ - spec/codecs/netflow_spec.rb