logstash-filter-ip2proxy 2.3.2 → 2.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 6f13634facd68b8cccdbd6cc6117ae1ca19203010c7a26209c43d6c78ef2427b
4
- data.tar.gz: f02c8d0f4626fc8cd549393ddfc7e416f60dd144a5bad19a0b2bc8507604dbab
3
+ metadata.gz: bfc6055c80454901e9029421d264d5f55121839710680c66825abb54a4313cfa
4
+ data.tar.gz: d52dc6d54dbb10403bd7b939eb5e27cc9e457a1a254efac24bd605da95375d7b
5
5
  SHA512:
6
- metadata.gz: 8781be17fa99fdcf69042727ab4d518eac2e193e47bad8e5de72de41247959fa73e25f48ab22cff8cea43e27f3aec260b8aaf929e115c720a39fb00a940f4d0f
7
- data.tar.gz: 15b88d65bc35baedbe899129d0247896a7244aac49fc5e3229b0cc713296c8f2ff2e6fcdd138bba2fc407e652896b54e096b2c34a9d8b7a5805f693dd3c36066
6
+ metadata.gz: eb5b6b3fa711ee6dfd3eb8cd0215467a6d8b599d7f5643ece09734f4ef7be69df5604a131ecd189ed3491a90dc8e6cc0d1214dc92d8f4bfa7fdaa5bac34ec767
7
+ data.tar.gz: 85b7dc3363c7c12b547db340dc5850fe423c03f00f224291df89e378edae868907771da44c37282ad84243f877da91d065c476b5b1374828787e9643bccefa1e
data/Gemfile CHANGED
@@ -1,11 +1,11 @@
1
- source 'https://rubygems.org'
2
-
3
- gemspec
4
-
5
- logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash"
6
- use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "1"
7
-
8
- if Dir.exist?(logstash_path) && use_logstash_source
9
- gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
10
- gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
11
- end
1
+ source 'https://rubygems.org'
2
+
3
+ gemspec
4
+
5
+ logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash"
6
+ use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "1"
7
+
8
+ if Dir.exist?(logstash_path) && use_logstash_source
9
+ gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
10
+ gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
11
+ end
data/LICENSE CHANGED
@@ -1,13 +1,13 @@
1
- Copyright (c) 2022 IP2Location.com
2
-
3
- Licensed under the Apache License, Version 2.0 (the "License");
4
- you may not use this file except in compliance with the License.
5
- You may obtain a copy of the License at
6
-
7
- http://www.apache.org/licenses/LICENSE-2.0
8
-
9
- Unless required by applicable law or agreed to in writing, software
10
- distributed under the License is distributed on an "AS IS" BASIS,
11
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
- See the License for the specific language governing permissions and
1
+ Copyright (c) 2025 IP2Location.com
2
+
3
+ Licensed under the Apache License, Version 2.0 (the "License");
4
+ you may not use this file except in compliance with the License.
5
+ You may obtain a copy of the License at
6
+
7
+ http://www.apache.org/licenses/LICENSE-2.0
8
+
9
+ Unless required by applicable law or agreed to in writing, software
10
+ distributed under the License is distributed on an "AS IS" BASIS,
11
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ See the License for the specific language governing permissions and
13
13
  limitations under the License.
data/README.md CHANGED
@@ -1,85 +1,118 @@
1
- # IP2Proxy Filter Plugin
2
- This is IP2Proxy filter plugin for Logstash that enables Logstash's users to reverse search of IP address to detect VPN servers, open proxies, web proxies, Tor exit nodes, search engine robots, data center ranges and residential proxies using IP2Proxy BIN database. Other information available includes proxy type, country, state, city, ISP, domain name, usage type, AS number, AS name, threats, last seen date and provider names. The library took the proxy IP address from **IP2Proxy BIN Data** file.
3
-
4
- For the methods to use IP2Proxy filter plugin with Elastic Stack (Elasticsearch, Filebeat, Logstash, and Kibana), please take a look on this [tutorial](https://blog.ip2location.com/knowledge-base/how-to-use-ip2proxy-filter-plugin-with-elastic-stack).
5
-
6
- *Note: This plugin works in Logstash 7 and Logstash 8.*
7
-
8
-
9
- ## Dependencies (IP2PROXY BIN DATA FILE)
10
- This plugin requires IP2Proxy BIN data file to function. You may download the BIN data file at
11
- * IP2Proxy LITE BIN Data (Free): https://lite.ip2location.com
12
- * IP2Proxy Commercial BIN Data (Commercial): https://www.ip2location.com
13
-
14
-
15
- ## Installation
16
- Install this plugin by the following code:
17
- ```
18
- bin/logstash-plugin install logstash-filter-ip2proxy
19
- ```
20
-
21
-
22
- ## Config File Example
23
- ```
24
- input {
25
- beats {
26
- port => "5043"
27
- }
28
- }
29
-
30
- filter {
31
- grok {
32
- match => { "message" => "%{COMBINEDAPACHELOG}"}
33
- }
34
- ip2proxy {
35
- source => "clientip"
36
- }
37
- }
38
-
39
- output {
40
- elasticsearch {
41
- hosts => [ "localhost:9200" ]
42
- }
43
- }
44
- ```
45
-
46
-
47
- ## IP2Proxy Filter Configuration
48
- |Setting|Input type|Required|
49
- |---|---|---|
50
- |source|string|Yes|
51
- |database|a valid filesystem path|No|
52
- |use_memory_mapped|boolean|No|
53
- |use_cache|boolean|No|
54
- |hide_unsupported_fields|boolean|No|
55
-
56
- * **source** field is a required setting that containing the IP address or hostname to get the ip information.
57
- * **database** field is an optional setting that containing the path to the IP2Proxy BIN database file.
58
- * **use_memory_mapped** field is an optional setting that used to allow user to enable the use of memory mapped file. Default value is false.
59
- * **use_cache** field is an optional setting that used to allow user to enable the use of cache. Default value is true.
60
- * **hide_unsupported_fields** field is an optional setting that used to allow user to hide unsupported fields. Default value is false.
61
-
62
-
63
- ## Sample Output
64
- |Field|Description|
65
- |---|---|
66
- |ip2proxy.as|the autonomous system (AS) name of proxy's IP address or domain name|
67
- |ip2proxy.asn|the autonomous system number (ASN) of proxy's IP address or domain name|
68
- |ip2proxy.city|the city name of the proxy|
69
- |ip2proxy.country_long|the ISO3166-1 country name of the proxy|
70
- |ip2proxy.country_short|the ISO3166-1 country code (two-characters) of the proxy|
71
- |ip2proxy.domain|the domain name of proxy's IP address or domain name|
72
- |ip2proxy.is_proxy|Check whether if an IP address was a proxy. Returned value:<ul><li>-1 : errors</li><li>0 : not a proxy</li><li>1 : a proxy</li><li>2 : a data center IP address</li></ul>|
73
- |ip2proxy.isp|the ISP name of the proxy|
74
- |ip2proxy.last_seen|the last seen days ago value of proxy's IP address or domain name|
75
- |ip2proxy.provider|the VPN service provider name if available|
76
- |ip2proxy.proxy_type|the proxy type. Please visit <a href="https://www.ip2location.com/database/px11-ip-proxytype-country-region-city-isp-domain-usagetype-asn-lastseen-threat-residential-provider" target="_blank">IP2Location</a> for the list of proxy types supported|
77
- |ip2proxy.region|the ISO3166-2 region name of the proxy. Please visit <a href="https://www.ip2location.com/free/iso3166-2" target="_blank">ISO3166-2 Subdivision Code</a> for the information of ISO3166-2 supported|
78
- |ip2proxy.thread|the threat type of the proxy|
79
- |ip2proxy.usage_type|the usage type classification of the proxy. Please visit <a href="https://www.ip2location.com/database/px11-ip-proxytype-country-region-city-isp-domain-usagetype-asn-lastseen-threat-residential-provider" target="_blank">IP2Location</a> for the list of usage types supported|
80
-
81
-
82
- ## Support
83
- Email: support@ip2location.com
84
-
85
- URL: [https://www.ip2location.com](https://www.ip2location.com)
1
+ # IP2Proxy Filter Plugin
2
+ This is IP2Proxy filter plugin for Logstash that enables Logstash's users to reverse search of IP address to detect VPN servers, open proxies, web proxies, Tor exit nodes, search engine robots, data center ranges, residential proxies, consumer privacy networks, and enterprise private networks using IP2Proxy BIN database. Other information available includes proxy type, country, state, city, ISP, domain name, usage type, AS number, AS name, threats, last seen date and provider names. The library took the proxy IP address from **IP2Proxy BIN Data** file and **IP2Location.io** data.
3
+
4
+ For the methods to use IP2Proxy filter plugin with Elastic Stack (Elasticsearch, Filebeat, Logstash, and Kibana), please take a look on this [tutorial](https://blog.ip2location.com/knowledge-base/how-to-use-ip2proxy-filter-plugin-with-elastic-stack).
5
+
6
+ *Note: This plugin works in Logstash 7, 8 and 9*
7
+
8
+
9
+ ## Dependencies (IP2PROXY BIN DATA FILE)
10
+ This plugin requires IP2Proxy BIN data file to function. You may download the BIN data file at
11
+ * IP2Proxy LITE BIN Data (Free): https://lite.ip2location.com
12
+ * IP2Proxy Commercial BIN Data (Commercial): https://www.ip2location.com
13
+
14
+ ## Dependencies (IP2LOCATION.IO DATA)
15
+ This plugin requires API key to function. You may sign up for a free API key at https://www.ip2location.io/pricing.
16
+
17
+
18
+ ## Installation
19
+ Install this plugin by the following code:
20
+ ```
21
+ bin/logstash-plugin install logstash-filter-ip2proxy
22
+ ```
23
+
24
+
25
+ ## Config File Example
26
+ ```
27
+ input {
28
+ beats {
29
+ port => "5043"
30
+ }
31
+ }
32
+
33
+ filter {
34
+ grok {
35
+ match => { "message" => "%{COMBINEDAPACHELOG}"}
36
+ }
37
+ ip2proxy {
38
+ source => "[source][address]"
39
+ }
40
+ }
41
+
42
+ output {
43
+ elasticsearch {
44
+ hosts => [ "localhost:9200" ]
45
+ }
46
+ }
47
+ ```
48
+
49
+ ## Config File Example using IP2Location.io
50
+ ```
51
+ input {
52
+ beats {
53
+ port => "5043"
54
+ }
55
+ }
56
+
57
+ filter {
58
+ grok {
59
+ match => { "message" => "%{COMBINEDAPACHELOG}"}
60
+ }
61
+ ip2proxy {
62
+ source => "[source][address]"
63
+ lookup_type => "ws"
64
+ api_key => "YOUR_API_KEY"
65
+ }
66
+ }
67
+
68
+ output {
69
+ elasticsearch {
70
+ hosts => [ "localhost:9200" ]
71
+ }
72
+ }
73
+ ```
74
+
75
+
76
+ ## IP2Proxy Filter Configuration
77
+ |Setting|Input type|Required|
78
+ |---|---|---|
79
+ |source|string|Yes|
80
+ |database|a valid filesystem path|No|
81
+ |use_memory_mapped|boolean|No|
82
+ |use_cache|boolean|No|
83
+ |lookup_type|string|No|
84
+ |api_key|string|No|
85
+ |hide_unsupported_fields|boolean|No|
86
+
87
+ * **source** field is a required setting that containing the IP address or hostname to get the ip information.
88
+ * **database** field is an optional setting that containing the path to the IP2Proxy BIN database file.
89
+ * **use_memory_mapped** field is an optional setting that used to allow user to enable the use of memory mapped file. Default value is false.
90
+ * **use_cache** field is an optional setting that used to allow user to enable the use of cache. Default value is true.
91
+ * **lookup_type** field is an optional setting that used to allow user to decide the lookup method either using IP2Proxy BIN database file(db) or IP2Location.io data(ws). Default value is db.
92
+ * **api_key** field is an optional setting that used to allow user to set the API Key of the IP2Location.io lookup.
93
+ * **hide_unsupported_fields** field is an optional setting that used to allow user to hide unsupported fields. Default value is false.
94
+
95
+
96
+ ## Sample Output
97
+ |Field|Description|
98
+ |---|---|
99
+ |ip2proxy.as|the autonomous system (AS) name of proxy's IP address or domain name|
100
+ |ip2proxy.asn|the autonomous system number (ASN) of proxy's IP address or domain name|
101
+ |ip2proxy.city|the city name of the proxy|
102
+ |ip2proxy.country_long|the ISO3166-1 country name of the proxy|
103
+ |ip2proxy.country_short|the ISO3166-1 country code (two-characters) of the proxy|
104
+ |ip2proxy.domain|the domain name of proxy's IP address or domain name|
105
+ |ip2proxy.is_proxy|Check whether if an IP address was a proxy. Returned value:<ul><li>-1 : errors</li><li>0 : not a proxy</li><li>1 : a proxy</li><li>2 : a data center IP address</li></ul>|
106
+ |ip2proxy.isp|the ISP name of the proxy|
107
+ |ip2proxy.last_seen|the last seen days ago value of proxy's IP address or domain name|
108
+ |ip2proxy.provider|the VPN service provider name if available|
109
+ |ip2proxy.proxy_type|the proxy type. Please visit <a href="https://www.ip2location.com/database/px11-ip-proxytype-country-region-city-isp-domain-usagetype-asn-lastseen-threat-residential-provider" target="_blank">IP2Location</a> for the list of proxy types supported|
110
+ |ip2proxy.region|the ISO3166-2 region name of the proxy. Please visit <a href="https://www.ip2location.com/free/iso3166-2" target="_blank">ISO3166-2 Subdivision Code</a> for the information of ISO3166-2 supported|
111
+ |ip2proxy.thread|the threat type of the proxy|
112
+ |ip2proxy.usage_type|the usage type classification of the proxy. Please visit <a href="https://www.ip2location.com/database/px11-ip-proxytype-country-region-city-isp-domain-usagetype-asn-lastseen-threat-residential-provider" target="_blank">IP2Location</a> for the list of usage types supported|
113
+
114
+
115
+ ## Support
116
+ Email: support@ip2location.com
117
+
118
+ URL: [https://www.ip2location.com](https://www.ip2location.com)
@@ -1,171 +1,191 @@
1
- # encoding: utf-8
2
- require "logstash/filters/base"
3
- require "logstash/namespace"
4
-
5
- require "logstash-filter-ip2proxy_jars"
6
-
7
- class LogStash::Filters::IP2Proxy < LogStash::Filters::Base
8
- config_name "ip2proxy"
9
-
10
- # The path to the IP2Proxy.BIN database file which Logstash should use.
11
- # If not specified, this will default to the IP2PROXY-LITE-PX1.BIN database that embedded in the plugin.
12
- config :database, :validate => :path
13
-
14
- # The field containing the IP address.
15
- # If this field is an array, only the first value will be used.
16
- config :source, :validate => :string, :required => true
17
-
18
- # The field used to define iplocation as target.
19
- config :target, :validate => :string, :default => 'ip2proxy'
20
-
21
- # The field used to allow user to enable the use of cache.
22
- config :use_cache, :validate => :boolean, :default => true
23
-
24
- # The field used to allow user to enable the use of memory mapped file.
25
- config :use_memory_mapped, :validate => :boolean, :default => false
26
-
27
- # The field used to allow user to hide unsupported fields.
28
- config :hide_unsupported_fields, :validate => :boolean, :default => false
29
-
30
- # The field used to define the size of the cache. It is not required and the default value is 10 000
31
- config :cache_size, :validate => :number, :required => false, :default => 10_000
32
-
33
- public
34
- def register
35
- if @database.nil?
36
- @database = ::Dir.glob(::File.join(::File.expand_path("../../../vendor/", ::File.dirname(__FILE__)),"IP2PROXY-LITE-PX1.BIN")).first
37
-
38
- if @database.nil? || !File.exists?(@database)
39
- raise "You must specify 'database => ...' in your ip2proxy filter (I looked for '#{@database}')"
40
- end
41
- end
42
-
43
- @logger.info("Using ip2proxy database", :path => @database)
44
-
45
- @ip2proxyfilter = org.logstash.filters.IP2ProxyFilter.new(@source, @target, @database, @use_memory_mapped, @hide_unsupported_fields)
46
- end
47
-
48
- public
49
- def filter(event)
50
- ip = event.get(@source)
51
-
52
- return unless filter?(event)
53
- if @use_cache
54
- if value = IP2ProxyCache.find(event, ip, @ip2proxyfilter, @cache_size).get('ip2proxy')
55
- event.set('ip2proxy', value)
56
- filter_matched(event)
57
- else
58
- tag_iplookup_unsuccessful(event)
59
- end
60
- else
61
- if @ip2proxyfilter.handleEvent(event)
62
- filter_matched(event)
63
- else
64
- tag_iplookup_unsuccessful(event)
65
- end
66
- end
67
- end
68
-
69
- def tag_iplookup_unsuccessful(event)
70
- @logger.debug? && @logger.debug("IP #{event.get(@source)} was not found in the database", :event => event)
71
- end
72
-
73
- end # class LogStash::Filters::IP2Proxy
74
-
75
- class IP2ProxyOrderedHash
76
- ONE = 1
77
-
78
- attr_reader :times_queried # ip -> times queried
79
- attr_reader :hash
80
-
81
- def initialize
82
- @times_queried = Hash.new(0) # ip -> times queried
83
- @hash = {} # number of hits -> array of ips
84
- end
85
-
86
- def add(key)
87
- hash[ONE] ||= []
88
- hash[ONE] << key
89
- times_queried[key] = ONE
90
- end
91
-
92
- def reorder(key)
93
- number_of_queries = times_queried[key]
94
-
95
- hash[number_of_queries].delete(key)
96
- hash.delete(number_of_queries) if hash[number_of_queries].empty?
97
-
98
- hash[number_of_queries + 1] ||= []
99
- hash[number_of_queries + 1] << key
100
- end
101
-
102
- def increment(key)
103
- add(key) unless times_queried.has_key?(key)
104
- reorder(key)
105
- times_queried[key] += 1
106
- end
107
-
108
- def delete_least_used
109
- first_pile_with_something.shift.tap { |key| times_queried.delete(key) }
110
- end
111
-
112
- def first_pile_with_something
113
- hash[hash.keys.min]
114
- end
115
- end
116
-
117
- class IP2ProxyCache
118
- ONE_DAY_IN_SECONDS = 86_400
119
-
120
- @cache = {} # ip -> event
121
- @timestamps = {} # ip -> time of caching
122
- @times_queried = IP2ProxyOrderedHash.new # ip -> times queried
123
- @mutex = Mutex.new
124
-
125
- class << self
126
- attr_reader :cache
127
- attr_reader :timestamps
128
- attr_reader :times_queried
129
-
130
- def find(event, ip, filter, cache_size)
131
- synchronize do
132
- if cache.has_key?(ip)
133
- refresh_event(event, ip, filter) if too_old?(ip)
134
- else
135
- if cache_full?(cache_size)
136
- make_room
137
- end
138
- cache_event(event, ip, filter)
139
- end
140
- times_queried.increment(ip)
141
- cache[ip]
142
- end
143
- end
144
-
145
- def too_old?(ip)
146
- timestamps[ip] < Time.now - ONE_DAY_IN_SECONDS
147
- end
148
-
149
- def make_room
150
- key = times_queried.delete_least_used
151
- cache.delete(key)
152
- timestamps.delete(key)
153
- end
154
-
155
- def cache_full?(cache_size)
156
- cache.size >= cache_size
157
- end
158
-
159
- def cache_event(event, ip, filter)
160
- filter.handleEvent(event)
161
- cache[ip] = event
162
- timestamps[ip] = Time.now
163
- end
164
-
165
- def synchronize(&block)
166
- @mutex.synchronize(&block)
167
- end
168
-
169
- alias_method :refresh_event, :cache_event
170
- end
171
- end
1
+ # encoding: utf-8
2
+ require "logstash/filters/base"
3
+ require "logstash/namespace"
4
+
5
+ require "logstash-filter-ip2proxy_jars"
6
+
7
+ class LogStash::Filters::IP2Proxy < LogStash::Filters::Base
8
+ config_name "ip2proxy"
9
+
10
+ # The path to the IP2Proxy.BIN database file which Logstash should use.
11
+ # If not specified, this will default to the IP2PROXY-LITE-PX1.BIN database that embedded in the plugin.
12
+ config :database, :validate => :path
13
+
14
+ # The field containing the IP address.
15
+ # If this field is an array, only the first value will be used.
16
+ config :source, :validate => :string, :required => true
17
+
18
+ # The field used to define iplocation as target.
19
+ config :target, :validate => :string, :default => 'ip2proxy'
20
+
21
+ # The field used to allow user to enable the use of cache.
22
+ config :use_cache, :validate => :boolean, :default => true
23
+
24
+ # The field used to allow user to enable the use of memory mapped file.
25
+ config :use_memory_mapped, :validate => :boolean, :default => false
26
+
27
+ # The field used to allow user to hide unsupported fields.
28
+ config :hide_unsupported_fields, :validate => :boolean, :default => false
29
+
30
+ # The field used to define lookup type.
31
+ config :lookup_type, :validate => :string, :default => 'db'
32
+
33
+ # The field used to define the apikey of IP2location.io.
34
+ config :api_key, :validate => :string, :default => ''
35
+
36
+ # The field used to define the size of the cache. It is not required and the default value is 10 000
37
+ config :cache_size, :validate => :number, :required => false, :default => 10_000
38
+
39
+ public
40
+ def register
41
+ if @lookup_type == "ws"
42
+ @logger.info("Using IP2Location.io API")
43
+ if @api_key == ""
44
+ raise "An IP2Location.io API key is required. You may sign up for a free API key at https://www.ip2location.io/pricing."
45
+ end
46
+ else
47
+ if @database.nil?
48
+ @database = ::Dir.glob(::File.join(::File.expand_path("../../../vendor/", ::File.dirname(__FILE__)),"IP2PROXY-LITE-PX1.BIN")).first
49
+
50
+ if @database.nil? || !File.exists?(@database)
51
+ raise "You must specify 'database => ...' in your ip2proxy filter (I looked for '#{@database}')"
52
+ end
53
+ end
54
+ @logger.info("Using ip2proxy database", :path => @database)
55
+ end
56
+
57
+ @ip2proxyfilter = org.logstash.filters.IP2ProxyFilter.new(@source, @target, @database, @use_memory_mapped, @hide_unsupported_fields, @lookup_type, @api_key)
58
+ end
59
+
60
+ public
61
+ def filter(event)
62
+ ip = event.get(@source)
63
+
64
+ return unless filter?(event)
65
+ if @lookup_type == "ws"
66
+ if @ip2proxyfilter.handleEvent(event)
67
+ filter_matched(event)
68
+ else
69
+ tag_iplookup_unsuccessful(event)
70
+ end
71
+ else
72
+ if @use_cache
73
+ if value = IP2ProxyCache.find(event, ip, @ip2proxyfilter, @cache_size).get('ip2proxy')
74
+ event.set('ip2proxy', value)
75
+ filter_matched(event)
76
+ else
77
+ tag_iplookup_unsuccessful(event)
78
+ end
79
+ else
80
+ if @ip2proxyfilter.handleEvent(event)
81
+ filter_matched(event)
82
+ else
83
+ tag_iplookup_unsuccessful(event)
84
+ end
85
+ end
86
+ end
87
+ end
88
+
89
+ def tag_iplookup_unsuccessful(event)
90
+ @logger.debug? && @logger.debug("IP #{event.get(@source)} was not found in the database", :event => event)
91
+ end
92
+
93
+ end # class LogStash::Filters::IP2Proxy
94
+
95
+ class IP2ProxyOrderedHash
96
+ ONE = 1
97
+
98
+ attr_reader :times_queried # ip -> times queried
99
+ attr_reader :hash
100
+
101
+ def initialize
102
+ @times_queried = Hash.new(0) # ip -> times queried
103
+ @hash = {} # number of hits -> array of ips
104
+ end
105
+
106
+ def add(key)
107
+ hash[ONE] ||= []
108
+ hash[ONE] << key
109
+ times_queried[key] = ONE
110
+ end
111
+
112
+ def reorder(key)
113
+ number_of_queries = times_queried[key]
114
+
115
+ hash[number_of_queries].delete(key)
116
+ hash.delete(number_of_queries) if hash[number_of_queries].empty?
117
+
118
+ hash[number_of_queries + 1] ||= []
119
+ hash[number_of_queries + 1] << key
120
+ end
121
+
122
+ def increment(key)
123
+ add(key) unless times_queried.has_key?(key)
124
+ reorder(key)
125
+ times_queried[key] += 1
126
+ end
127
+
128
+ def delete_least_used
129
+ first_pile_with_something.shift.tap { |key| times_queried.delete(key) }
130
+ end
131
+
132
+ def first_pile_with_something
133
+ hash[hash.keys.min]
134
+ end
135
+ end
136
+
137
+ class IP2ProxyCache
138
+ ONE_DAY_IN_SECONDS = 86_400
139
+
140
+ @cache = {} # ip -> event
141
+ @timestamps = {} # ip -> time of caching
142
+ @times_queried = IP2ProxyOrderedHash.new # ip -> times queried
143
+ @mutex = Mutex.new
144
+
145
+ class << self
146
+ attr_reader :cache
147
+ attr_reader :timestamps
148
+ attr_reader :times_queried
149
+
150
+ def find(event, ip, filter, cache_size)
151
+ synchronize do
152
+ if cache.has_key?(ip)
153
+ refresh_event(event, ip, filter) if too_old?(ip)
154
+ else
155
+ if cache_full?(cache_size)
156
+ make_room
157
+ end
158
+ cache_event(event, ip, filter)
159
+ end
160
+ times_queried.increment(ip)
161
+ cache[ip]
162
+ end
163
+ end
164
+
165
+ def too_old?(ip)
166
+ timestamps[ip] < Time.now - ONE_DAY_IN_SECONDS
167
+ end
168
+
169
+ def make_room
170
+ key = times_queried.delete_least_used
171
+ cache.delete(key)
172
+ timestamps.delete(key)
173
+ end
174
+
175
+ def cache_full?(cache_size)
176
+ cache.size >= cache_size
177
+ end
178
+
179
+ def cache_event(event, ip, filter)
180
+ filter.handleEvent(event)
181
+ cache[ip] = event
182
+ timestamps[ip] = Time.now
183
+ end
184
+
185
+ def synchronize(&block)
186
+ @mutex.synchronize(&block)
187
+ end
188
+
189
+ alias_method :refresh_event, :cache_event
190
+ end
191
+ end
@@ -1,3 +1,4 @@
1
- require 'jar_dependencies'
2
- require_jar('com.ip2proxy.ip2proxy', 'ip2proxy', '3.4.0')
3
- require_jar('org.logstash.filters', 'logstash-filter-ip2proxy', '2.3.2')
1
+ require 'jar_dependencies'
2
+ require_jar('com.ip2proxy.ip2proxy', 'ip2proxy', '3.5.0')
3
+ require_jar('com.google.gson', 'gson', '2.11.0')
4
+ require_jar('org.logstash.filters', 'logstash-filter-ip2proxy', '2.5.0')
@@ -1,25 +1,25 @@
1
- Gem::Specification.new do |s|
2
-
3
- s.name = 'logstash-filter-ip2proxy'
4
- s.version = '2.3.2'
5
- s.licenses = ['Apache-2.0']
6
- s.summary = "Logstash filter IP2Proxy"
7
- s.description = "IP2Proxy filter plugin for Logstash enables Logstash's users to reverse search of IP address to detect VPN servers, open proxies, web proxies, Tor exit nodes, search engine robots, data center ranges and residential proxies using IP2Proxy BIN database."
8
- s.authors = ["IP2Location"]
9
- s.email = 'support@ip2location.com'
10
- s.homepage = "https://www.ip2location.com"
11
- s.require_paths = ["lib", "vendor/jar-dependencies"]
12
-
13
- # Files
14
- s.files = Dir["lib/**/*",'spec/**/*',"vendor/**/*","vendor/jar-dependencies/**/*.jar","*.gemspec","*.md","Gemfile","LICENSE"]
15
-
16
- # Tests
17
- s.test_files = s.files.grep(%r{^(test|spec|features)/})
18
-
19
- # Special flag to let us know this is actually a logstash plugin
20
- s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" }
21
-
22
- # Gem dependencies
23
- s.add_runtime_dependency "logstash-core-plugin-api", "~> 2.0"
24
- s.add_development_dependency "logstash-devutils"
25
- end
1
+ Gem::Specification.new do |s|
2
+
3
+ s.name = 'logstash-filter-ip2proxy'
4
+ s.version = '2.5.0'
5
+ s.licenses = ['Apache-2.0']
6
+ s.summary = "Logstash filter IP2Proxy"
7
+ s.description = "IP2Proxy filter plugin for Logstash enables Logstash's users to reverse search of IP address to detect VPN servers, open proxies, web proxies, Tor exit nodes, search engine robots, data center ranges and residential proxies using IP2Proxy BIN database."
8
+ s.authors = ["IP2Location"]
9
+ s.email = 'support@ip2location.com'
10
+ s.homepage = "https://www.ip2location.com"
11
+ s.require_paths = ["lib", "vendor/jar-dependencies"]
12
+
13
+ # Files
14
+ s.files = Dir["lib/**/*",'spec/**/*',"vendor/**/*","vendor/jar-dependencies/**/*.jar","*.gemspec","*.md","Gemfile","LICENSE"]
15
+
16
+ # Tests
17
+ s.test_files = s.files.grep(%r{^(test|spec|features)/})
18
+
19
+ # Special flag to let us know this is actually a logstash plugin
20
+ s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" }
21
+
22
+ # Gem dependencies
23
+ s.add_runtime_dependency "logstash-core-plugin-api", "~> 2.0"
24
+ s.add_development_dependency "logstash-devutils"
25
+ end
@@ -1,27 +1,27 @@
1
- # encoding: utf-8
2
- require_relative '../spec_helper'
3
- require "logstash/filters/ip2proxy"
4
-
5
- IP2PROXYDB = ::Dir.glob(::File.expand_path("../../vendor/", ::File.dirname(__FILE__))+"/IP2PROXY-LITE-PX1.BIN").first
6
-
7
- describe LogStash::Filters::IP2Proxy do
8
-
9
- describe "normal test" do
10
- config <<-CONFIG
11
- filter {
12
- ip2proxy {
13
- source => "ip"
14
- #database => "#{IP2PROXYDB}"
15
- }
16
- }
17
- CONFIG
18
-
19
- sample("ip" => "8.8.8.8") do
20
- expect(subject.get("ip2proxy")).not_to be_empty
21
- expect(subject.get("ip2proxy")["country_short"]).to eq("US")
22
- end
23
- end
24
-
25
- end
26
-
1
+ # encoding: utf-8
2
+ require_relative '../spec_helper'
3
+ require "logstash/filters/ip2proxy"
4
+
5
+ IP2PROXYDB = ::Dir.glob(::File.expand_path("../../vendor/", ::File.dirname(__FILE__))+"/IP2PROXY-LITE-PX1.BIN").first
6
+
7
+ describe LogStash::Filters::IP2Proxy do
8
+
9
+ describe "normal test" do
10
+ config <<-CONFIG
11
+ filter {
12
+ ip2proxy {
13
+ source => "ip"
14
+ #database => "#{IP2PROXYDB}"
15
+ }
16
+ }
17
+ CONFIG
18
+
19
+ sample("ip" => "8.8.8.8") do
20
+ expect(subject.get("ip2proxy")).not_to be_empty
21
+ expect(subject.get("ip2proxy")["country_short"]).to eq("US")
22
+ end
23
+ end
24
+
25
+ end
26
+
27
27
  end
data/spec/spec_helper.rb CHANGED
@@ -1,2 +1,2 @@
1
- # encoding: utf-8
2
- require "logstash/devutils/rspec/spec_helper"
1
+ # encoding: utf-8
2
+ require "logstash/devutils/rspec/spec_helper"
metadata CHANGED
@@ -1,14 +1,13 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-filter-ip2proxy
3
3
  version: !ruby/object:Gem::Version
4
- version: 2.3.2
4
+ version: 2.5.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - IP2Location
8
- autorequire:
9
8
  bindir: bin
10
9
  cert_chain: []
11
- date: 2022-10-14 00:00:00.000000000 Z
10
+ date: 1980-01-02 00:00:00.000000000 Z
12
11
  dependencies:
13
12
  - !ruby/object:Gem::Dependency
14
13
  name: logstash-core-plugin-api
@@ -55,16 +54,15 @@ files:
55
54
  - logstash-filter-ip2proxy.gemspec
56
55
  - spec/filters/ip2proxy_spec.rb
57
56
  - spec/spec_helper.rb
58
- - vendor/IP2PROXY-LITE-PX1.BIN
59
- - vendor/jar-dependencies/com/ip2proxy/ip2proxy/ip2proxy/3.4.0/ip2proxy-3.4.0.jar
60
- - vendor/jar-dependencies/org/logstash/filters/logstash-filter-ip2proxy/2.3.2/logstash-filter-ip2proxy-2.3.2.jar
57
+ - vendor/jar-dependencies/com/google/gson/gson/2.11.0/gson-2.11.0.jar
58
+ - vendor/jar-dependencies/com/ip2proxy/ip2proxy/ip2proxy/3.5.0/ip2proxy-3.5.0.jar
59
+ - vendor/jar-dependencies/org/logstash/filters/logstash-filter-ip2proxy/2.5.0/logstash-filter-ip2proxy-2.5.0.jar
61
60
  homepage: https://www.ip2location.com
62
61
  licenses:
63
62
  - Apache-2.0
64
63
  metadata:
65
64
  logstash_plugin: 'true'
66
65
  logstash_group: filter
67
- post_install_message:
68
66
  rdoc_options: []
69
67
  require_paths:
70
68
  - lib
@@ -80,8 +78,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
80
78
  - !ruby/object:Gem::Version
81
79
  version: '0'
82
80
  requirements: []
83
- rubygems_version: 3.3.7
84
- signing_key:
81
+ rubygems_version: 3.7.1
85
82
  specification_version: 4
86
83
  summary: Logstash filter IP2Proxy
87
84
  test_files:
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:852f55814fb14e8de5b47d416b95f1ce58757256484fda7d2d4462900db4229a
3
- size 229021771