logstash-filter-cidrtagmap 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: ed725267129ccb6206153486ba1f8d6b7d56a793
4
+ data.tar.gz: de9e163ce694db757be6ca59b24bf3e87c10a39c
5
+ SHA512:
6
+ metadata.gz: 975da1ec0f5561934688e60442986832824f973b9ce2c50efe78872aad7ee610acae6ec58a4205978091d693292e2a28d4f74f8b8a4dac534158088ceab97692
7
+ data.tar.gz: 88a06b2464fa75fd5850fe824d377fb0fbd1f13d26b2bc58434c026b1cb8c9879bc56559f4759aae9ed9ef219961aabb68471d3fc3cf65a99336423c3e559625
data/CHANGELOG.md ADDED
@@ -0,0 +1,5 @@
1
+ ## 2.0.0
2
+ - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully,
3
+ instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
4
+ - Dependency on logstash-core update to 2.0
5
+
data/CONTRIBUTORS ADDED
@@ -0,0 +1,11 @@
1
+ The following is a list of people who have contributed ideas, code, bug
2
+ reports, or in general have helped logstash along its way.
3
+
4
+ Contributors:
5
+ * Aaron Mildenstein (untergeek)
6
+ * Pier-Hugues Pellerin (ph)
7
+
8
+ Note: If you've sent us patches, bug reports, or otherwise contributed to
9
+ Logstash, and you aren't on the list above and want to be, please let us know
10
+ and we'll make sure you're here. Contributions from folks like you are what make
11
+ open source awesome.
data/DEVELOPER.md ADDED
@@ -0,0 +1,2 @@
1
+ # logstash-filter-example
2
+ Example filter plugin. This should help bootstrap your effort to write your own filter plugin!
data/Gemfile ADDED
@@ -0,0 +1,2 @@
1
+ source 'https://rubygems.org'
2
+ gemspec
data/LICENSE ADDED
@@ -0,0 +1,13 @@
1
+ Copyright (c) 2012–2016 Elasticsearch <http://www.elastic.co>
2
+
3
+ Licensed under the Apache License, Version 2.0 (the "License");
4
+ you may not use this file except in compliance with the License.
5
+ You may obtain a copy of the License at
6
+
7
+ http://www.apache.org/licenses/LICENSE-2.0
8
+
9
+ Unless required by applicable law or agreed to in writing, software
10
+ distributed under the License is distributed on an "AS IS" BASIS,
11
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ See the License for the specific language governing permissions and
13
+ limitations under the License.
data/NOTICE.TXT ADDED
@@ -0,0 +1,5 @@
1
+ Elasticsearch
2
+ Copyright 2012-2015 Elasticsearch
3
+
4
+ This product includes software developed by The Apache Software
5
+ Foundation (http://www.apache.org/).
data/README.md ADDED
@@ -0,0 +1,37 @@
1
+ This logstash filter tags netflow records according to a list of CIDR to tag mappings.
2
+
3
+ The list is an external / stand alone text file consisting of lines of the form:
4
+
5
+ ```
6
+ <network>/<mask>,<tag>
7
+ ```
8
+
9
+ The filter can be made to re-load its in-memory representation of the contents of the
10
+ map file without interrupting/restarting the logstash instance by touching a flag file.
11
+
12
+ When a netflow event matches the CIDR spec, two tags are set:
13
+
14
+ src_tag = the tag associated with the spec that matched
15
+
16
+ src_tagMatch = the CIDR spec that matched (as rendered by IPAddr.to_s)
17
+
18
+
19
+ Configuration:
20
+
21
+ ```
22
+ filter{
23
+ cidrtagmap {
24
+ mapfilepath => "cidrmap.txt"
25
+ }
26
+ }
27
+ ```
28
+
29
+ Tell the filter to reload its map
30
+
31
+ ```
32
+ touch <mapfilepath>.RELOAD
33
+ ```
34
+
35
+ Reloading is thread safe.
36
+
37
+ Put cidrtagmap.rb in $LOGSTASH/lib/logstash/filters/
@@ -0,0 +1,167 @@
1
+ require "logstash/filters/base"
2
+ require "logstash/namespace"
3
+ require 'ipaddr'
4
+
5
+ class MapEntry
6
+ attr_reader :range,:tag
7
+ def initialize(spec = "")
8
+ begin
9
+ parts = spec.split(',')
10
+ @range = IPAddr.new(parts[0])
11
+ @tag = parts[1]
12
+ return self
13
+ rescue
14
+ @logger.warn("cidrtagmap: error parsing map entry #{spec}")
15
+ return nil
16
+ end
17
+ end
18
+
19
+ def includesAddress?(ipaddr)
20
+ return @range.include?(ipaddr)
21
+ end
22
+ end
23
+
24
+ class LogStash::Filters::CIDRTagMap < LogStash::Filters::Base
25
+
26
+ config_name "cidrtagmap"
27
+
28
+ milestone 1
29
+
30
+ config :mapfilepath, :validate => :string, :default => 'cidrmap.txt'
31
+ config :asnmapfilepath, :validate => :string, :default => 'asn.txt'
32
+
33
+
34
+ private
35
+
36
+ def loadLatestMap
37
+ if File.exist?(@reloadFlagPath) or @cidrMap.nil?
38
+ @logger.debug("cidrtagmap: need to load, getting mutex")
39
+ @mutex.synchronize {
40
+ # Test again now that we have the floor just in case someone else did it already
41
+ # This is because we might have blocked on the mutex when we first encountered the condition
42
+ # We don't wrap the initial test because that'd have the effect of serializing every single
43
+ # netflow event through the test, which defeats the purpose of multiple threads.
44
+ # But we acknowledge that there's room for a race here so test again to be sure.
45
+ if File.exist?(@reloadFlagPath) or @cidrMap.nil?
46
+ if File.exist?(@reloadFlagPath)
47
+ begin
48
+ # This thread wins, clear the flag file. If someone else is waiting
49
+ # on the mutex they'll see that it's already done when they get in here.
50
+ # We want to do this right away because the longer we wait the more likely
51
+ # it is that other threads are piling up behind the mutex.
52
+ File.delete(@reloadFlagPath)
53
+ @logger.info("cidrtagmap: cleared reload flag")
54
+ rescue
55
+ @logger.warn("cidrtagmap: unable to remove #{@reloadFlagPath} - I'm probably gonna loop in a bad way")
56
+ end
57
+ end
58
+ @logger.info("cidrtagmap: loading map into memory")
59
+ begin
60
+ @mapFile = File.new(@mapfilepath,'r')
61
+ @cidrMap = @mapFile.readlines.map {
62
+ |line|
63
+ MapEntry.new(line.chomp)
64
+ }
65
+ @mapFile.close
66
+ @cidrMap = @cidrMap.reject { |item| item.nil? }
67
+ @logger.info("cidrtagmap: loaded #{@cidrMap.inspect}")
68
+ rescue
69
+ @logger.warn("cidrtagmap: error opening map file #{@mapfilepath}\n")
70
+ @mapFile = nil
71
+ end
72
+ begin
73
+ asntable = File.readlines(@asnmapfilepath)
74
+ regex = /^ (\d+?)\s+(.+?)\s+/
75
+ @asnmap = Hash[asntable.collect { |line| line.match(regex)}.select {|each| not each.nil?}.collect{|each| [each[1],each[2]] }]
76
+ rescue Exception => e
77
+ @logger.warn("cidrtagmap: error loading asn map file #{@asnmapfilepath}\n")
78
+ @logger.warn("cidrtagmap: #{e.inspect}")
79
+ end
80
+ else
81
+ @logger.debug("cidrtagmap: someone already loaded the map - I'm outta here")
82
+ end
83
+ }
84
+ end
85
+ end
86
+
87
+ def mapForIp(addrString = "")
88
+ begin
89
+ address = IPAddr.new(addrString.to_s)
90
+ matchIndex = @cidrMap.index{
91
+ |range|
92
+ range.includesAddress?(address)
93
+ }
94
+ if matchIndex
95
+ @logger.debug("cidrtagmap: match for #{address} at #{matchIndex}")
96
+ return @cidrMap[matchIndex]
97
+ else
98
+ return nil
99
+ end
100
+ rescue
101
+ @logger.warn("cidrtagmap: error attempting to map #{addrString}\n")
102
+ end
103
+ end
104
+
105
+ def asNameForNumber(as = 0)
106
+ begin
107
+ return @asnmap[as.to_s] || "UNKNOWN"
108
+ rescue
109
+ return "MAPERROR"
110
+ end
111
+ end
112
+
113
+ public
114
+ def register
115
+ @mutex = Mutex.new
116
+ @reloadFlagPath = "#{@mapfilepath}.RELOAD"
117
+ @logger.info("cidrtagmap: NOTE: touch #{@reloadFlagPath} to force map reload")
118
+ loadLatestMap
119
+ end
120
+
121
+ public
122
+ def filter(event)
123
+ return unless filter?(event)
124
+ if event['netflow']
125
+ loadLatestMap
126
+ netflow = event['netflow']
127
+ if netflow["ipv4_src_addr"]
128
+ @logger.debug("cidrtagmap: checking for src #{netflow['ipv4_src_addr']}")
129
+ src_map = mapForIp(netflow["ipv4_src_addr"])
130
+ if src_map
131
+ @logger.debug("cidrtagmap: tagging src #{netflow['ipv4_src_addr']} with #{src_map.tag}")
132
+ netflow["src_tag"] = src_map.tag
133
+ netflow['src_tagMatch'] = src_map.range.to_s
134
+ filter_matched(event)
135
+ end
136
+ end
137
+ if netflow["ipv4_dst_addr"]
138
+ @logger.debug("cidrtagmap: checking for dst #{netflow['ipv4_dst_addr']}")
139
+ dst_map = mapForIp(netflow["ipv4_dst_addr"])
140
+ if dst_map
141
+ @logger.debug("cidrtagmap: tagging dst #{netflow['ipv4_dst_addr']} with #{dst_map.tag}")
142
+ netflow["dst_tag"] = dst_map.tag
143
+ netflow["dst_tagMatch"] = dst_map.range.to_s
144
+ filter_matched(event)
145
+ end
146
+ end
147
+ if netflow["dst_as"]
148
+ @logger.debug("cidrtagmap: checking for dst_as #{netflow['dst_as']}")
149
+ dst_asname = asNameForNumber(netflow["dst_as"])
150
+ if dst_asname
151
+ @logger.debug("cidrtagmap: tagging dst_as #{netflow['dst_as']} with #{dst_asname}")
152
+ netflow["dst_as_name"] = dst_asname
153
+ filter_matched(event)
154
+ end
155
+ end
156
+ if netflow["src_as"]
157
+ @logger.debug("cidrtagmap: checking for src_as #{netflow['src_as']}")
158
+ src_asname = asNameForNumber(netflow["src_as"])
159
+ if src_asname
160
+ @logger.debug("cidrtagmap: tagging src_as #{netflow['src_as']} with #{src_asname}")
161
+ netflow["src_as_name"] = src_asname
162
+ filter_matched(event)
163
+ end
164
+ end
165
+ end
166
+ end
167
+ end
@@ -0,0 +1,23 @@
1
+ Gem::Specification.new do |s|
2
+ s.name = 'logstash-filter-cidrtagmap'
3
+ s.version = '1.1.0'
4
+ s.licenses = ['Apache License (2.0)']
5
+ s.summary = "This cidrtagmap filter replaces the contents of the message field with the specified value."
6
+ s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
7
+ s.authors = ["svdasein"]
8
+ s.email = 'daveparker01@gmail.com'
9
+ s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html"
10
+ s.require_paths = ["lib"]
11
+
12
+ # Files
13
+ s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
14
+ # Tests
15
+ s.test_files = s.files.grep(%r{^(test|spec|features)/})
16
+
17
+ # Special flag to let us know this is actually a logstash plugin
18
+ s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" }
19
+
20
+ # Gem dependencies
21
+ s.add_runtime_dependency "logstash-core-plugin-api", "~> 1.8"
22
+ s.add_development_dependency 'logstash-devutils'
23
+ end
@@ -0,0 +1,20 @@
1
+ # encoding: utf-8
2
+ require 'spec_helper'
3
+ require "logstash/filters/example"
4
+
5
+ describe LogStash::Filters::Example do
6
+ describe "Set to Hello World" do
7
+ let(:config) do <<-CONFIG
8
+ filter {
9
+ example {
10
+ message => "Hello World"
11
+ }
12
+ }
13
+ CONFIG
14
+ end
15
+
16
+ sample("message" => "some text") do
17
+ expect(subject.get("message")).to eq('Hello World')
18
+ end
19
+ end
20
+ end
@@ -0,0 +1,2 @@
1
+ # encoding: utf-8
2
+ require "logstash/devutils/rspec/spec_helper"
metadata ADDED
@@ -0,0 +1,89 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: logstash-filter-cidrtagmap
3
+ version: !ruby/object:Gem::Version
4
+ version: 1.1.0
5
+ platform: ruby
6
+ authors:
7
+ - svdasein
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2017-06-30 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: logstash-core-plugin-api
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - "~>"
18
+ - !ruby/object:Gem::Version
19
+ version: '1.8'
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - "~>"
25
+ - !ruby/object:Gem::Version
26
+ version: '1.8'
27
+ - !ruby/object:Gem::Dependency
28
+ name: logstash-devutils
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - ">="
32
+ - !ruby/object:Gem::Version
33
+ version: '0'
34
+ type: :development
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - ">="
39
+ - !ruby/object:Gem::Version
40
+ version: '0'
41
+ description: This gem is a Logstash plugin required to be installed on top of the
42
+ Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This
43
+ gem is not a stand-alone program
44
+ email: daveparker01@gmail.com
45
+ executables: []
46
+ extensions: []
47
+ extra_rdoc_files: []
48
+ files:
49
+ - CHANGELOG.md
50
+ - CONTRIBUTORS
51
+ - DEVELOPER.md
52
+ - Gemfile
53
+ - LICENSE
54
+ - NOTICE.TXT
55
+ - README.md
56
+ - lib/logstash/filters/cidrtagmap.rb
57
+ - logstash-filter-cidrtagmap.gemspec
58
+ - spec/filters/example_spec.rb
59
+ - spec/spec_helper.rb
60
+ homepage: http://www.elastic.co/guide/en/logstash/current/index.html
61
+ licenses:
62
+ - Apache License (2.0)
63
+ metadata:
64
+ logstash_plugin: 'true'
65
+ logstash_group: filter
66
+ post_install_message:
67
+ rdoc_options: []
68
+ require_paths:
69
+ - lib
70
+ required_ruby_version: !ruby/object:Gem::Requirement
71
+ requirements:
72
+ - - ">="
73
+ - !ruby/object:Gem::Version
74
+ version: '0'
75
+ required_rubygems_version: !ruby/object:Gem::Requirement
76
+ requirements:
77
+ - - ">="
78
+ - !ruby/object:Gem::Version
79
+ version: '0'
80
+ requirements: []
81
+ rubyforge_project:
82
+ rubygems_version: 2.4.8
83
+ signing_key:
84
+ specification_version: 4
85
+ summary: This cidrtagmap filter replaces the contents of the message field with the
86
+ specified value.
87
+ test_files:
88
+ - spec/filters/example_spec.rb
89
+ - spec/spec_helper.rb