fluent-plugin-netflow-enchanced 1.0.0.rc1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 4c76eebf8b18910e2ac39ab145f110dd01237496
4
+ data.tar.gz: e4b6d605e9fe93e9d26024e3bca00abd940e5341
5
+ SHA512:
6
+ metadata.gz: e3160293b97bece1d8dba37b608e044c90782907b17ee2c055ac4309ee72b44e2d98fb10f9e23f0b2318cf850bd6718b8ad6f62f1a2f1fd7063d9c38d6f26c5b
7
+ data.tar.gz: 5b1dabb47633997098275b699ae9c642a8956fbb0869688844f0db8790f5007a8ca625045c19f0cc65c262c2c2b6c562a95ce086744b06ae2dfbdd8c98b45556
data/.gitignore ADDED
@@ -0,0 +1,24 @@
1
+ *.gem
2
+ *.rbc
3
+ .bundle
4
+ .config
5
+ .yardoc
6
+ Gemfile.lock
7
+ InstalledFiles
8
+ _yardoc
9
+ coverage
10
+ doc/
11
+ lib/bundler/man
12
+ pkg
13
+ rdoc
14
+ spec/reports
15
+ test/tmp
16
+ test/version_tmp
17
+ tmp
18
+ # For TextMate, emacs, vim
19
+ *.tmproj
20
+ tmtags
21
+ *~
22
+ \#*
23
+ .\#*
24
+ *.swp
data/.travis.yml ADDED
@@ -0,0 +1,18 @@
1
+ language: ruby
2
+
3
+ rvm:
4
+ - 2.1
5
+ - 2.2
6
+ - 2.3.1
7
+ - 2.4.0
8
+ - ruby-head
9
+ - rbx
10
+
11
+ matrix:
12
+ allow_failures:
13
+ - rvm: ruby-head
14
+ - rvm: rbx
15
+
16
+ before_install: gem update bundler
17
+
18
+ script: bundle exec rake test
data/Gemfile ADDED
@@ -0,0 +1,3 @@
1
+ source "https://rubygems.org"
2
+
3
+ gemspec
data/README.md ADDED
@@ -0,0 +1,180 @@
1
+ # Netflow plugin for Fluentd
2
+
3
+ [![Build Status](https://travis-ci.org/repeatedly/fluent-plugin-netflow.svg)](https://travis-ci.org/repeatedly/fluent-plugin-netflow)
4
+
5
+
6
+ ## Overview
7
+
8
+ [Fluentd](http://fluentd.org/) input plugin that acts as Netflow v5/v9 collector.
9
+
10
+
11
+ ## Installation
12
+
13
+ Use RubyGems:
14
+
15
+ fluent-gem install fluent-plugin-netflow
16
+
17
+
18
+ ## Configuration
19
+
20
+ <source>
21
+ type netflow
22
+ tag netflow.event
23
+
24
+ # optional parameters
25
+ bind 192.168.0.1
26
+ port 2055
27
+ cache_ttl 6000
28
+ versions [5, 9]
29
+ definitions /path/to/custom_fields.yaml
30
+ </source>
31
+
32
+ **bind**
33
+
34
+ IP address on which the plugin will accept Netflow.
35
+ (Default: '0.0.0.0')
36
+
37
+ **port**
38
+
39
+ UDP port number on which tpe plugin will accept Netflow.
40
+ (Default: 5140)
41
+
42
+ **cache_ttl**
43
+
44
+ Template cache TTL for Netflow v9 in seconds. Templates not refreshed from the Netflow v9 exporter within the TTL are expired at the plugin.
45
+ (Default: 4000)
46
+
47
+ **versions**
48
+
49
+ Netflow versions which are acceptable.
50
+ (Default:[5, 9])
51
+
52
+ **switched_times_from_uptime**
53
+
54
+ When set to true, the plugin stores system uptime for ```first_switched``` and ```last_switched``` instead of ISO8601-formatted absolute time.
55
+ (Defaults: false)
56
+
57
+ **definitions**
58
+
59
+ YAML file containing Netflow field definitions to overfide pre-defined templates. Example is like below
60
+
61
+ ---
62
+ 4: # field value
63
+ - :uint8 # field length
64
+ - :protocol # field type
65
+
66
+
67
+ ## Performance Evaluation
68
+
69
+ Benchmark for v5 protocol on Macbook Air (Early 2014, 1.7 GHz Intel Core i7):
70
+ * 0 packets dropped in 32,000 records/second (for 3,000,000 packets)
71
+ * 45,000 records/second in maximum (for flooding netflow packets)
72
+
73
+ Tested with the packet generator below:
74
+
75
+ * https://github.com/mshindo/NetFlow-Generator
76
+ * `./flowgen -n3000000 -i50 -w1 -p5140 localhost`
77
+
78
+ And configuration:
79
+
80
+ <source>
81
+ @type netflow
82
+ tag netflow.event
83
+ bind 0.0.0.0
84
+ port 5140
85
+ switched_times_from_uptime yes
86
+ </source>
87
+ <match netflow.event>
88
+ @type flowcounter
89
+ unit minute
90
+ count_keys count # missing column for counting events only
91
+ tag flowcount
92
+ </match>
93
+ <match flowcount>
94
+ @type stdout
95
+ </match>
96
+
97
+
98
+ ## Tips
99
+
100
+ ### Use netflow parser in other plugins
101
+
102
+ ```ruby
103
+ require 'fluent/plugin/parser_netflow'
104
+
105
+ parser = Fluent::Plugin::NetflowParser.new
106
+ parser.configure(conf)
107
+
108
+ # Netflow v5
109
+ parser.call(payload) do |time, record|
110
+ # do something
111
+ end
112
+
113
+ # Netflow v9
114
+ parser.call(payload, source_ip_address) do |time, record|
115
+ # do something
116
+ end
117
+ ```
118
+
119
+ **NOTE:**
120
+ If the plugin receives Netflow v9 from multiple sources, provide ```source_ip_address``` argument to parse correctly.
121
+
122
+ ### Field definition for Netflow v9
123
+
124
+ Both option and scope fields for Netflow v9 are defined in [YAML](https://www.ietf.org/rfc/rfc3954.txt) where two parameters are described for each field value like:
125
+
126
+ ```yaml
127
+ option:
128
+ ...
129
+ 4: # field value
130
+ - :uint8 # field length
131
+ - :protocol # field type
132
+ ```
133
+
134
+ See [RFC3954 document](https://www.ietf.org/rfc/rfc3954.txt) for more details.
135
+
136
+ When int value specified for field length, the template parser in this plugin will prefer a field length in received template flowset over YAML. The int value in YAML will be used as a default value only when the length in received flowset is invalid.
137
+
138
+ ```yaml
139
+ option:
140
+ 1:
141
+ - 4 # means :unit32, which is just a default
142
+ - :in_bytes
143
+ ```
144
+
145
+ When ```:skip``` is described for a field, the template parser will learn the length from received template flowset and skip the field when data flowsets are processed.
146
+
147
+ ```yaml
148
+ option:
149
+ ...
150
+ 43:
151
+ - :skip
152
+ ```
153
+
154
+ **NOTE:**
155
+ The definitions don't exactly reflect RFC3954 in order to cover some illegal implementations which export Netflow v9 in bad field length.
156
+
157
+ ```yaml
158
+ 31:
159
+ - 3 # Some system exports in 4 bytes despite of RFC
160
+ - :ipv6_flow_label
161
+ ...
162
+ 48:
163
+ - 1 # Some system exports in 2 bytes despite of RFC
164
+ - :flow_sampler_id
165
+ ```
166
+
167
+ ### PaloAlto Netflow
168
+
169
+ PaloAlto Netflow has different field definitionas:
170
+ See this definitions for PaloAlto Netflow: https://github.com/repeatedly/fluent-plugin-netflow/issues/27#issuecomment-269197495
171
+
172
+ ### More speed ?
173
+
174
+ :bullettrain_side: Try ```switched_times_from_uptime true``` option !
175
+
176
+
177
+ ## TODO
178
+
179
+ * Netflow v9 protocol parser optimization
180
+ * Use Fluentd feature instead of own handlers
data/Rakefile ADDED
@@ -0,0 +1,14 @@
1
+
2
+ require 'bundler'
3
+ Bundler::GemHelper.install_tasks
4
+
5
+ require 'rake/testtask'
6
+
7
+ Rake::TestTask.new(:test) do |test|
8
+ test.libs << 'lib' << 'test'
9
+ test.test_files = FileList['test/**/test_*.rb']
10
+ test.verbose = true
11
+ end
12
+
13
+ task :default => [:build]
14
+
data/VERSION ADDED
@@ -0,0 +1 @@
1
+ 1.0.0.rc1
@@ -0,0 +1,9 @@
1
+ <source>
2
+ @type netflow
3
+ bind 127.0.0.1
4
+ tag example.netflow
5
+ </source>
6
+
7
+ <match example.netflow>
8
+ @type stdout
9
+ </match>
@@ -0,0 +1,24 @@
1
+ # encoding: utf-8
2
+ $:.push File.expand_path('../lib', __FILE__)
3
+
4
+ Gem::Specification.new do |gem|
5
+ gem.name = "fluent-plugin-netflow-enchanced"
6
+ gem.description = "Netflow plugin for Fluentd"
7
+ gem.homepage = "https://github.com/repeatedly/fluent-plugin-netflow"
8
+ gem.summary = gem.description
9
+ gem.version = File.read("VERSION").strip
10
+ gem.authors = ["Masahiro Nakagawa"]
11
+ gem.email = "repeatedly@gmail.com"
12
+ gem.has_rdoc = false
13
+ #gem.platform = Gem::Platform::RUBY
14
+ gem.license = 'Apache License (2.0)'
15
+ gem.files = `git ls-files`.split("\n")
16
+ gem.test_files = `git ls-files -- {test,spec,features}/*`.split("\n")
17
+ gem.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
18
+ gem.require_paths = ['lib']
19
+
20
+ gem.add_dependency "fluentd", [">= 0.14.10", "< 2"]
21
+ gem.add_dependency "bindata", "~> 2.1"
22
+ gem.add_development_dependency "rake", ">= 0.9.2"
23
+ gem.add_development_dependency "test-unit", "~> 3.0"
24
+ end
@@ -0,0 +1,80 @@
1
+ #
2
+ # Fluent
3
+ #
4
+ # Copyright (C) 2014 Masahiro Nakagawa
5
+ #
6
+ # Licensed under the Apache License, Version 2.0 (the "License");
7
+ # you may not use this file except in compliance with the License.
8
+ # You may obtain a copy of the License at
9
+ #
10
+ # http://www.apache.org/licenses/LICENSE-2.0
11
+ #
12
+ # Unless required by applicable law or agreed to in writing, software
13
+ # distributed under the License is distributed on an "AS IS" BASIS,
14
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15
+ # See the License for the specific language governing permissions and
16
+ # limitations under the License.
17
+ #
18
+
19
+ require 'fluent/plugin/input'
20
+ require 'fluent/plugin/parser_netflow'
21
+
22
+ module Fluent::Plugin
23
+ class NetflowInput < Input
24
+ Fluent::Plugin.register_input('netflow', self)
25
+
26
+ helpers :server
27
+
28
+ config_param :port, :integer, default: 5140
29
+ config_param :bind, :string, default: '0.0.0.0'
30
+ config_param :tag, :string
31
+ config_param :protocol_type, default: :udp do |val|
32
+ case val.downcase
33
+ when 'udp'
34
+ :udp
35
+ else
36
+ raise Fluent::ConfigError, "netflow input protocol type should be 'udp'"
37
+ end
38
+ end
39
+ config_param :max_bytes, :integer, default: 2048
40
+
41
+ def configure(conf)
42
+ super
43
+
44
+ @parser = Fluent::Plugin::NetflowParser.new
45
+ @parser.configure(conf)
46
+ end
47
+
48
+ def start
49
+ super
50
+ server_create(:in_netflow_server, @port, bind: @bind, proto: @protocol_type, max_bytes: @max_bytes) do |data, sock|
51
+ receive_data(sock.remote_host, data)
52
+ end
53
+ end
54
+
55
+ def shutdown
56
+ super
57
+ end
58
+
59
+ protected
60
+
61
+ def receive_data(host, data)
62
+ log.on_debug { log.debug "received logs", :host => host, :data => data }
63
+
64
+ @parser.call(data, host) { |time, record|
65
+ unless time && record
66
+ log.warn "pattern not match: #{data.inspect}"
67
+ return
68
+ end
69
+
70
+ record['host'] = host
71
+ record['process_time'] = Time.new.strftime("%Y-%m-%dT%H:%M:%S.%LZ")
72
+
73
+ router.emit(@tag, time, record)
74
+ }
75
+ rescue => e
76
+ log.warn "unexpected error on parsing", data: data.dump, error_class: e.class, error: e.message
77
+ log.warn_backtrace
78
+ end
79
+ end
80
+ end
@@ -0,0 +1,302 @@
1
+ ---
2
+ option:
3
+ 1:
4
+ - 4
5
+ - :in_bytes
6
+ 2:
7
+ - 4
8
+ - :in_pkts
9
+ 3:
10
+ - 4
11
+ - :flows
12
+ 4:
13
+ - :uint8
14
+ - :protocol
15
+ 5:
16
+ - :uint8
17
+ - :src_tos
18
+ 6:
19
+ - :uint8
20
+ - :tcp_flags
21
+ 7:
22
+ - :uint16
23
+ - :l4_src_port
24
+ 8:
25
+ - :ip4_addr
26
+ - :ipv4_src_addr
27
+ 9:
28
+ - :uint8
29
+ - :src_mask
30
+ 10:
31
+ - 2
32
+ - :input_snmp
33
+ 11:
34
+ - :uint16
35
+ - :l4_dst_port
36
+ 12:
37
+ - :ip4_addr
38
+ - :ipv4_dst_addr
39
+ 13:
40
+ - :uint8
41
+ - :dst_mask
42
+ 14:
43
+ - 2
44
+ - :output_snmp
45
+ 15:
46
+ - :ip4_addr
47
+ - :ipv4_next_hop
48
+ 16:
49
+ - 2
50
+ - :src_as
51
+ 17:
52
+ - 2
53
+ - :dst_as
54
+ 18:
55
+ - :ip4_addr
56
+ - :bgp_ipv4_next_hop
57
+ 19:
58
+ - 4
59
+ - :mul_dst_pkts
60
+ 20:
61
+ - 4
62
+ - :mul_dst_bytes
63
+ 21:
64
+ - :uint32
65
+ - :last_switched
66
+ 22:
67
+ - :uint32
68
+ - :first_switched
69
+ 23:
70
+ - 4
71
+ - :out_bytes
72
+ 24:
73
+ - 4
74
+ - :out_pkts
75
+ 25:
76
+ - :uint16
77
+ - :min_pkt_length
78
+ 26:
79
+ - :uint16
80
+ - :max_pkt_length
81
+ 27:
82
+ - :ip6_addr
83
+ - :ipv6_src_addr
84
+ 28:
85
+ - :ip6_addr
86
+ - :ipv6_dst_addr
87
+ 29:
88
+ - :uint8
89
+ - :ipv6_src_mask
90
+ 30:
91
+ - :uint8
92
+ - :ipv6_dst_mask
93
+ 31:
94
+ - 3
95
+ - :ipv6_flow_label
96
+ 32:
97
+ - :uint16
98
+ - :icmp_type
99
+ 33:
100
+ - :uint8
101
+ - :mul_igmp_type
102
+ 34:
103
+ - :uint32
104
+ - :sampling_interval
105
+ 35:
106
+ - :uint8
107
+ - :sampling_algorithm
108
+ 36:
109
+ - :uint16
110
+ - :flow_active_timeout
111
+ 37:
112
+ - :uint16
113
+ - :flow_inactive_timeout
114
+ 38:
115
+ - :uint8
116
+ - :engine_type
117
+ 39:
118
+ - :uint8
119
+ - :engine_id
120
+ 40:
121
+ - 4
122
+ - :total_bytes_exp
123
+ 41:
124
+ - 4
125
+ - :total_pkts_exp
126
+ 42:
127
+ - 4
128
+ - :total_flows_exp
129
+ 43:
130
+ - :skip
131
+ 44:
132
+ - :ip4_addr
133
+ - :ipv4_src_prefix
134
+ 45:
135
+ - :ip4_addr
136
+ - :ipv4_dst_prefix
137
+ 46:
138
+ - :uint8
139
+ - :mpls_top_label_type
140
+ 47:
141
+ - :uint32
142
+ - :mpls_top_label_ip_addr
143
+ 48:
144
+ - 1
145
+ - :flow_sampler_id
146
+ 49:
147
+ - :uint8
148
+ - :flow_sampler_mode
149
+ 50:
150
+ - :uint32
151
+ - :flow_sampler_random_interval
152
+ 51:
153
+ - :skip
154
+ 52:
155
+ - :uint8
156
+ - :min_ttl
157
+ 53:
158
+ - :uint8
159
+ - :max_ttl
160
+ 54:
161
+ - :uint16
162
+ - :ipv4_ident
163
+ 55:
164
+ - :uint8
165
+ - :dst_tos
166
+ 56:
167
+ - :mac_addr
168
+ - :in_src_mac
169
+ 57:
170
+ - :mac_addr
171
+ - :out_dst_mac
172
+ 58:
173
+ - :uint16
174
+ - :src_vlan
175
+ 59:
176
+ - :uint16
177
+ - :dst_vlan
178
+ 60:
179
+ - :uint8
180
+ - :ip_protocol_version
181
+ 61:
182
+ - :uint8
183
+ - :direction
184
+ 62:
185
+ - :ip6_addr
186
+ - :ipv6_next_hop
187
+ 63:
188
+ - :ip6_addr
189
+ - :bgp_ipv6_next_hop
190
+ 64:
191
+ - :uint32
192
+ - :ipv6_option_headers
193
+ 65:
194
+ - :skip
195
+ 66:
196
+ - :skip
197
+ 67:
198
+ - :skip
199
+ 68:
200
+ - :skip
201
+ 69:
202
+ - :skip
203
+ 70:
204
+ - :mpls_label
205
+ - :mpls_label_1
206
+ 71:
207
+ - :mpls_label
208
+ - :mpls_label_2
209
+ 72:
210
+ - :mpls_label
211
+ - :mpls_label_3
212
+ 73:
213
+ - :mpls_label
214
+ - :mpls_label_4
215
+ 74:
216
+ - :mpls_label
217
+ - :mpls_label_5
218
+ 75:
219
+ - :mpls_label
220
+ - :mpls_label_6
221
+ 76:
222
+ - :mpls_label
223
+ - :mpls_label_7
224
+ 77:
225
+ - :mpls_label
226
+ - :mpls_label_8
227
+ 78:
228
+ - :mpls_label
229
+ - :mpls_label_9
230
+ 79:
231
+ - :mpls_label
232
+ - :mpls_label_10
233
+ 80:
234
+ - :mac_addr
235
+ - :in_dst_mac
236
+ 81:
237
+ - :mac_addr
238
+ - :out_src_mac
239
+ 82:
240
+ - :string
241
+ - :if_name
242
+ 83:
243
+ - :string
244
+ - :if_desc
245
+ 84:
246
+ - :string
247
+ - :sampler_name
248
+ 89:
249
+ - :uint8
250
+ - :forwarding_status
251
+ 91:
252
+ - :uint8
253
+ - :mpls_prefix_len
254
+ 95:
255
+ - 4
256
+ - :app_id
257
+ 150:
258
+ - :uint32
259
+ - :flowStartSeconds
260
+ 151:
261
+ - :uint32
262
+ - :flowEndSeconds
263
+ 152:
264
+ - :uint64
265
+ - :flowStartMilliseconds
266
+ 153:
267
+ - :uint64
268
+ - :flowEndMilliseconds
269
+ 154:
270
+ - :uint64
271
+ - :flowStartMicroseconds
272
+ 155:
273
+ - :uint64
274
+ - :flowEndMicroseconds
275
+ 156:
276
+ - :uint64
277
+ - :flowStartNanoseconds
278
+ 157:
279
+ - :uint64
280
+ - :flowEndNanoseconds
281
+ 234:
282
+ - :uint32
283
+ - :ingress_vrf_id
284
+ 235:
285
+ - :uint32
286
+ - :egress_vrf_id
287
+ 236:
288
+ - :string
289
+ - :vrf_name
290
+
291
+ scope:
292
+ 1:
293
+ - :ip4_addr
294
+ - :system
295
+ 2:
296
+ - :skip
297
+ 3:
298
+ - :skip
299
+ 4:
300
+ - :skip
301
+ 5:
302
+ - :skip