logstash-filter-geoip 7.2.11-java → 7.3.4-java
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +26 -3
- data/VERSION +1 -0
- data/docs/index.asciidoc +58 -40
- data/lib/logstash/filters/geoip.rb +11 -1
- data/lib/logstash-filter-geoip_jars.rb +4 -3
- data/logstash-filter-geoip.gemspec +5 -3
- data/spec/filters/geoip_ecs_spec.rb +3 -3
- data/spec/filters/geoip_offline_spec.rb +1 -1
- data/spec/filters/geoip_online_spec.rb +77 -45
- data/spec/filters/geoip_spec.rb +22 -4
- data/spec/filters/test_helper.rb +35 -17
- data/vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-core/2.16.0/jackson-core-2.16.0.jar +0 -0
- data/vendor/jar-dependencies/com/maxmind/db/maxmind-db/2.1.0/maxmind-db-2.1.0.jar +0 -0
- data/vendor/jar-dependencies/com/maxmind/geoip2/geoip2/2.17.0/geoip2-2.17.0.jar +0 -0
- data/vendor/jar-dependencies/org/logstash/filters/logstash-filter-geoip/7.3.4/logstash-filter-geoip-7.3.4.jar +0 -0
- metadata +21 -22
- data/vendor/jar-dependencies/com/maxmind/db/maxmind-db/1.2.2/maxmind-db-1.2.2.jar +0 -0
- data/vendor/jar-dependencies/com/maxmind/geoip2/geoip2/2.9.0/geoip2-2.9.0.jar +0 -0
- data/vendor/jar-dependencies/org/logstash/filters/logstash-filter-geoip/6.0.0/logstash-filter-geoip-6.0.0.jar +0 -0
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 2e710c4161058e523c550ff4323492d735b2c68f4904b57a74ee519c60934e7b
|
|
4
|
+
data.tar.gz: 92f3f4d3dcff159b288e837071f46263ec3b875b67d494e67da33d043cbdc5b2
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 4098cab433c99bc666a793c6f020e8382887deb52d86bd75b3005a0b3fca04826d50d527eba58bd9440b8f284e1a01a2e087816fd8ac18c139286cb6e85bc36d
|
|
7
|
+
data.tar.gz: 2fa0fd69f9df7a3532eb411ce8a8be0cdf1864f4caa7c6c13843699b2e656965d9806e5afe319754043226f1b7ec0b373b17d9349c8ec3b3291816762178ed92
|
data/CHANGELOG.md
CHANGED
|
@@ -1,7 +1,30 @@
|
|
|
1
|
+
## 7.3.4
|
|
2
|
+
- Fix: replace deprecated `File.exists?` with `File.exist?` for Ruby 3.4 (JRuby 10) compatibility [#239](https://github.com/logstash-plugins/logstash-filter-geoip/pull/239)
|
|
3
|
+
|
|
4
|
+
## 7.3.3
|
|
5
|
+
- Re-packaging the plugin [#236](https://github.com/logstash-plugins/logstash-filter-geoip/pull/236)
|
|
6
|
+
|
|
7
|
+
## 7.3.2
|
|
8
|
+
- Add logs when MaxMind databases fail to deserialize custom field [#235](https://github.com/logstash-plugins/logstash-filter-geoip/pull/235)
|
|
9
|
+
|
|
10
|
+
## 7.3.1
|
|
11
|
+
- Avoid to crash pipelines when lookup a database with customised fields. [#225](https://github.com/logstash-plugins/logstash-filter-geoip/pull/225)
|
|
12
|
+
|
|
13
|
+
## 7.3.0
|
|
14
|
+
- Added support for MaxMind GeoIP2 Enterprise and Anonymous-IP databases ([#223](https://github.com/logstash-plugins/logstash-filter-geoip/pull/223))
|
|
15
|
+
- Updated MaxMind dependencies.
|
|
16
|
+
- Added tests for the Java classes.
|
|
17
|
+
|
|
18
|
+
## 7.2.13
|
|
19
|
+
- [DOC] Add documentation for database auto-update configuration [#210](https://github.com/logstash-plugins/logstash-filter-geoip/pull/210)
|
|
20
|
+
|
|
21
|
+
## 7.2.12
|
|
22
|
+
- [DOC] Add `http_proxy` environment variable for GeoIP service endpoint. The feature is included in 8.1.0, and was back-ported to 7.17.2 [#207](https://github.com/logstash-plugins/logstash-filter-geoip/pull/207)
|
|
23
|
+
|
|
1
24
|
## 7.2.11
|
|
2
|
-
|
|
3
|
-
|
|
4
|
-
|
|
25
|
+
- Improved compatibility with the Elastic Common Schema [#206](https://github.com/logstash-plugins/logstash-filter-geoip/pull/206)
|
|
26
|
+
- Added support for ECS's composite `region_iso_code` (`US-WA`), which _replaces_ the non-ECS `region_code` (`WA`) as a default field with City databases. To get the stand-alone `region_code` in ECS mode, you must include it in the `fields` directive.
|
|
27
|
+
- [DOC] Improve ECS-related documentation
|
|
5
28
|
|
|
6
29
|
## 7.2.10
|
|
7
30
|
- [DOC] Air-gapped environment requires both ASN and City databases [#204](https://github.com/logstash-plugins/logstash-filter-geoip/pull/204)
|
data/VERSION
ADDED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
7.3.4
|
data/docs/index.asciidoc
CHANGED
|
@@ -47,8 +47,8 @@ within 30 days of a database update.
|
|
|
47
47
|
|
|
48
48
|
The GeoIP filter plugin can manage the database for users running the Logstash default
|
|
49
49
|
distribution, or you can manage
|
|
50
|
-
database updates on your own. The behavior is controlled by the `database` setting.
|
|
51
|
-
When you use the default `database` setting
|
|
50
|
+
database updates on your own. The behavior is controlled by the `database` setting and by the auto-update feature.
|
|
51
|
+
When you use the default `database` setting and the auto-update feature is enabled, Logstash ensures that the plugin is
|
|
52
52
|
using the latest version of the database.
|
|
53
53
|
Otherwise, you are responsible for maintaining compliance.
|
|
54
54
|
|
|
@@ -59,27 +59,38 @@ database by default.
|
|
|
59
59
|
==== Database Auto-update
|
|
60
60
|
|
|
61
61
|
This plugin bundles Creative Commons (CC) license databases.
|
|
62
|
-
Logstash checks for database updates every day. It downloads the latest and can replace the old database
|
|
62
|
+
If the auto-update feature is enabled in `logstash.yml`(as it is by default), Logstash checks for database updates every day. It downloads the latest and can replace the old database
|
|
63
63
|
while the plugin is running.
|
|
64
|
-
After Logstash downloads EULA license databases, it will not fallback to CC license databases.
|
|
65
64
|
|
|
66
|
-
NOTE: If the database has never been updated successfully, as in air-gapped environments, Logstash can use CC license databases indefinitely.
|
|
65
|
+
NOTE: If the auto-update feature is disabled or the database has never been updated successfully, as in air-gapped environments, Logstash can use CC license databases indefinitely.
|
|
67
66
|
|
|
68
67
|
After Logstash has switched to a EULA licensed database, the geoip filter will
|
|
69
68
|
stop enriching events in order to maintain compliance if Logstash fails to
|
|
70
69
|
check for database updates for 30 days.
|
|
71
70
|
Events will be tagged with `_geoip_expired_database` tag to facilitate the handling of this situation.
|
|
72
71
|
|
|
72
|
+
NOTE: If the auto-update feature is enabled, Logstash upgrades from the CC database license to the EULA version on the first download.
|
|
73
|
+
|
|
73
74
|
TIP: When possible, allow Logstash to access the internet to download databases so that they are always up-to-date.
|
|
74
75
|
|
|
76
|
+
**Disable the auto-update feature**
|
|
77
|
+
|
|
78
|
+
If you work in air-gapped environment and want to disable the database auto-update feature, set the `xpack.geoip.downloader.enabled` value to `false` in `logstash.yml`.
|
|
79
|
+
|
|
80
|
+
When the auto-update feature is disabled, Logstash uses the Creative Commons (CC) license databases indefinitely, and any previously downloaded version of the EULA databases will be deleted.
|
|
81
|
+
|
|
75
82
|
[id="plugins-{type}s-{plugin}-manage_update"]
|
|
76
83
|
==== Manage your own database updates
|
|
77
84
|
|
|
78
|
-
**Use
|
|
85
|
+
**Use an HTTP proxy**
|
|
79
86
|
|
|
80
87
|
If you can't connect directly to the Elastic GeoIP endpoint, consider setting up
|
|
81
|
-
|
|
82
|
-
|
|
88
|
+
an HTTP proxy server. You can then specify the proxy with `http_proxy` environment variable.
|
|
89
|
+
|
|
90
|
+
[source,sh]
|
|
91
|
+
----
|
|
92
|
+
export http_proxy="http://PROXY_IP:PROXY_PORT"
|
|
93
|
+
----
|
|
83
94
|
|
|
84
95
|
**Use a custom endpoint (air-gapped environments)**
|
|
85
96
|
|
|
@@ -179,37 +190,44 @@ When ECS compatibility is enabled, the fields are structured to fit into an ECS
|
|
|
179
190
|
|===========================
|
|
180
191
|
| Database Field Name | ECS Field | Example
|
|
181
192
|
|
|
182
|
-
| `ip`
|
|
183
|
-
|
|
184
|
-
| `
|
|
185
|
-
| `
|
|
186
|
-
| `
|
|
187
|
-
| `
|
|
188
|
-
| `
|
|
189
|
-
| `
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
| `
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
| `
|
|
204
|
-
|
|
205
|
-
| `
|
|
206
|
-
|
|
207
|
-
| `
|
|
208
|
-
| `
|
|
209
|
-
|
|
210
|
-
| `
|
|
211
|
-
|
|
212
|
-
| `
|
|
193
|
+
| `ip` | `[ip]` | `12.34.56.78`
|
|
194
|
+
| `anonymous` | `[ip_traits][anonymous]` | `false`
|
|
195
|
+
| `anonymous_vpn` | `[ip_traits][anonymous_vpn]` | `false`
|
|
196
|
+
| `hosting_provider` | `[ip_traits][hosting_provider]` | `true`
|
|
197
|
+
| `network` | `[ip_traits][network]` | `12.34.56.78/20`
|
|
198
|
+
| `public_proxy` | `[ip_traits][public_proxy]` | `true`
|
|
199
|
+
| `residential_proxy` | `[ip_traits][residential_proxy]` | `false`
|
|
200
|
+
| `tor_exit_node` | `[ip_traits][tor_exit_node]` | `true`
|
|
201
|
+
|
|
202
|
+
| `city_name` | `[geo][city_name]` | `Seattle`
|
|
203
|
+
| `country_name` | `[geo][country_name]` | `United States`
|
|
204
|
+
| `continent_code` | `[geo][continent_code]` | `NA`
|
|
205
|
+
| `continent_name` | `[geo][continent_name]` | `North America`
|
|
206
|
+
| `country_code2` | `[geo][country_iso_code]` | `US`
|
|
207
|
+
| `country_code3` | _N/A_ | `US`
|
|
208
|
+
|
|
209
|
+
_maintained for legacy
|
|
210
|
+
support, but populated
|
|
211
|
+
with 2-character country
|
|
212
|
+
code_
|
|
213
|
+
|
|
214
|
+
| `postal_code` | `[geo][postal_code]` | `98106`
|
|
215
|
+
| `region_name` | `[geo][region_name]` | `Washington`
|
|
216
|
+
| `region_code` | `[geo][region_code]` | `WA`
|
|
217
|
+
| `region_iso_code`* | `[geo][region_iso_code]` | `US-WA`
|
|
218
|
+
| `timezone` | `[geo][timezone]` | `America/Los_Angeles`
|
|
219
|
+
| `location`* | `[geo][location]` | `{"lat": 47.6062, "lon": -122.3321}"`
|
|
220
|
+
| `latitude` | `[geo][location][lat]` | `47.6062`
|
|
221
|
+
| `longitude` | `[geo][location][lon]` | `-122.3321`
|
|
222
|
+
|
|
223
|
+
| `domain` | `[domain]` | `example.com`
|
|
224
|
+
|
|
225
|
+
| `asn` | `[as][number]` | `98765`
|
|
226
|
+
| `as_org` | `[as][organization][name]` | `Elastic, NV`
|
|
227
|
+
|
|
228
|
+
| `isp` | `[mmdb][isp]` | `InterLink Supra LLC`
|
|
229
|
+
| `dma_code` | `[mmdb][dma_code]` | `819`
|
|
230
|
+
| `organization` | `[mmdb][organization]` | `Elastic, NV`
|
|
213
231
|
|===========================
|
|
214
232
|
|
|
215
233
|
NOTE: `*` indicates a composite field, which is only populated if GeoIP lookup result contains all components.
|
|
@@ -290,7 +308,7 @@ number of cache misses and waste memory.
|
|
|
290
308
|
The path to MaxMind's database file that Logstash should use.
|
|
291
309
|
The default database is `GeoLite2-City`.
|
|
292
310
|
This plugin supports several free databases (`GeoLite2-City`, `GeoLite2-Country`, `GeoLite2-ASN`)
|
|
293
|
-
and a selection of commercially-licensed databases (`GeoIP2-City`, `GeoIP2-ISP`, `GeoIP2-Country`).
|
|
311
|
+
and a selection of commercially-licensed databases (`GeoIP2-City`, `GeoIP2-ISP`, `GeoIP2-Country`, `GeoIP2-Domain`, `GeoIP2-Enterprise`, `GeoIP2-Anonymous-IP`).
|
|
294
312
|
|
|
295
313
|
Database auto-update applies to the default distribution.
|
|
296
314
|
When `database` points to user's database path, auto-update is disabled.
|
|
@@ -171,7 +171,17 @@ class LogStash::Filters::GeoIP < LogStash::Filters::Base
|
|
|
171
171
|
end
|
|
172
172
|
|
|
173
173
|
def close
|
|
174
|
-
|
|
174
|
+
begin
|
|
175
|
+
@database_manager.unsubscribe_database_path(@default_database_type, self) if @database_manager
|
|
176
|
+
rescue => e
|
|
177
|
+
@logger.error("Error unsubscribing geoip database path", :path => @database, :exception => e)
|
|
178
|
+
end
|
|
179
|
+
|
|
180
|
+
begin
|
|
181
|
+
@geoipfilter.close if @geoipfilter
|
|
182
|
+
rescue => e
|
|
183
|
+
@logger.error("Error closing GeoIPFilter", :exception => e)
|
|
184
|
+
end
|
|
175
185
|
end
|
|
176
186
|
|
|
177
187
|
def select_database_path
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
# AUTOGENERATED BY THE GRADLE SCRIPT. DO NOT EDIT.
|
|
2
2
|
|
|
3
3
|
require 'jar_dependencies'
|
|
4
|
-
require_jar('com.maxmind.geoip2', 'geoip2', '2.
|
|
5
|
-
require_jar('com.maxmind.db', 'maxmind-db', '1.
|
|
6
|
-
require_jar('
|
|
4
|
+
require_jar('com.maxmind.geoip2', 'geoip2', '2.17.0')
|
|
5
|
+
require_jar('com.maxmind.db', 'maxmind-db', '2.1.0')
|
|
6
|
+
require_jar('com.fasterxml.jackson.core', 'jackson-core', '2.16.0')
|
|
7
|
+
require_jar('org.logstash.filters', 'logstash-filter-geoip', '7.3.4')
|
|
@@ -1,18 +1,20 @@
|
|
|
1
|
+
VERSION = File.read(File.expand_path(File.join(File.dirname(__FILE__), "VERSION"))).strip unless defined?(VERSION)
|
|
2
|
+
|
|
1
3
|
Gem::Specification.new do |s|
|
|
2
4
|
|
|
3
5
|
s.name = 'logstash-filter-geoip'
|
|
4
|
-
s.version =
|
|
6
|
+
s.version = VERSION
|
|
5
7
|
s.licenses = ['Apache License (2.0)']
|
|
6
8
|
s.summary = "Adds geographical information about an IP address"
|
|
7
9
|
s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
|
|
8
10
|
s.authors = ["Elastic"]
|
|
9
11
|
s.email = 'info@elastic.co'
|
|
10
|
-
s.homepage = "
|
|
12
|
+
s.homepage = "https://elastic.co/logstash"
|
|
11
13
|
s.platform = "java"
|
|
12
14
|
s.require_paths = ["lib", "vendor/jar-dependencies"]
|
|
13
15
|
|
|
14
16
|
# Files
|
|
15
|
-
s.files = Dir['lib/**/*','spec/**/*','vendor/**/*', 'vendor/jar-dependencies/**/*.jar', '*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT', 'maxmind-db-NOTICE.txt', 'docs/**/*']
|
|
17
|
+
s.files = Dir['lib/**/*','spec/**/*','vendor/**/*', 'vendor/jar-dependencies/**/*.jar', '*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE', 'VERSION', 'NOTICE.TXT', 'maxmind-db-NOTICE.txt', 'docs/**/*']
|
|
16
18
|
|
|
17
19
|
# Tests
|
|
18
20
|
s.test_files = s.files.grep(%r{^(test|spec|features)/})
|
|
@@ -22,7 +22,7 @@ describe LogStash::Filters::GeoIP do
|
|
|
22
22
|
let(:common_options) { {"source" => "message", "database" => CITYDB, "target" => target} }
|
|
23
23
|
|
|
24
24
|
before(:each) do
|
|
25
|
-
|
|
25
|
+
allow(plugin).to receive(:ecs_compatibility).and_return(ecs_compatibility)
|
|
26
26
|
plugin.register
|
|
27
27
|
end
|
|
28
28
|
|
|
@@ -170,7 +170,7 @@ describe LogStash::Filters::GeoIP do
|
|
|
170
170
|
|
|
171
171
|
context "ECS disabled" do
|
|
172
172
|
before do
|
|
173
|
-
|
|
173
|
+
allow(plugin).to receive(:ecs_compatibility).and_return(:disabled)
|
|
174
174
|
plugin.register
|
|
175
175
|
plugin.filter(event)
|
|
176
176
|
end
|
|
@@ -193,7 +193,7 @@ describe LogStash::Filters::GeoIP do
|
|
|
193
193
|
|
|
194
194
|
context "ECS mode" do
|
|
195
195
|
before do
|
|
196
|
-
|
|
196
|
+
allow(plugin).to receive(:ecs_compatibility).and_return(:v1)
|
|
197
197
|
end
|
|
198
198
|
|
|
199
199
|
context "`target` is unset" do
|
|
@@ -112,7 +112,7 @@ describe LogStash::Filters::GeoIP do
|
|
|
112
112
|
CONFIG
|
|
113
113
|
|
|
114
114
|
context "should return the correct sourcefield in the logging message" do
|
|
115
|
-
sample("ip" => "8.8.8.8") do
|
|
115
|
+
sample({"ip" => "8.8.8.8"}) do
|
|
116
116
|
expect { subject }.to raise_error(java.lang.IllegalArgumentException, "The database provided is invalid or corrupted.")
|
|
117
117
|
end
|
|
118
118
|
end
|
|
@@ -1,65 +1,97 @@
|
|
|
1
1
|
# encoding: utf-8
|
|
2
|
+
require 'pathname'
|
|
2
3
|
require "logstash/devutils/rspec/spec_helper"
|
|
3
4
|
require "insist"
|
|
4
5
|
require "logstash/filters/geoip"
|
|
5
6
|
require_relative 'test_helper'
|
|
6
7
|
|
|
7
8
|
describe LogStash::Filters::GeoIP do
|
|
9
|
+
context "when no database_path is given" do
|
|
8
10
|
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
11
|
+
let(:last_db_path_recorder) do
|
|
12
|
+
Module.new do
|
|
13
|
+
attr_reader :last_db_path
|
|
14
|
+
def setup_filter(db_path)
|
|
15
|
+
@last_db_path = db_path
|
|
16
|
+
super
|
|
17
|
+
end
|
|
18
|
+
end
|
|
19
|
+
end
|
|
12
20
|
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
21
|
+
let(:plugin_config) { Hash["source" => "[source][ip]", "target" => "[target]"] }
|
|
22
|
+
let(:plugin) { described_class.new(plugin_config).extend(last_db_path_recorder) }
|
|
23
|
+
let(:event) { LogStash::Event.new("source" => { "ip" => "173.9.34.107" }) }
|
|
24
|
+
|
|
25
|
+
shared_examples "event enrichment" do
|
|
26
|
+
it 'enriches events' do
|
|
27
|
+
plugin.register
|
|
28
|
+
plugin.filter(event)
|
|
29
|
+
|
|
30
|
+
expect(event.get("target")).to include('ip')
|
|
31
|
+
end
|
|
19
32
|
end
|
|
20
33
|
|
|
21
|
-
|
|
34
|
+
database_management_available = (MAJOR >= 8 || (MAJOR == 7 && MINOR >= 14)) && !LogStash::OSS
|
|
35
|
+
if database_management_available
|
|
36
|
+
context "when geoip database management is available" do
|
|
37
|
+
|
|
38
|
+
let(:mock_manager) do
|
|
39
|
+
double('LogStash::Filters::Geoip::DatabaseManager').tap do |m|
|
|
40
|
+
allow(m).to receive(:subscribe_database_path) do |db_type, explicit_path, plugin_instance|
|
|
41
|
+
explicit_path || mock_managed[db_type]
|
|
42
|
+
end
|
|
43
|
+
allow(m).to receive(:unsubscribe_database_path).with(any_args)
|
|
44
|
+
end
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
# The extension to this plugin that lives in Logstash core will _always_ provide a valid
|
|
48
|
+
# database path, and how it does so is not the concern of this plugin. We emulate this
|
|
49
|
+
# behaviour here by copying the vendored CC-licensed db's into a temporary path
|
|
50
|
+
let(:mock_managed) do
|
|
51
|
+
managed_path = Pathname.new(temp_data_path).join("managed", Time.now.to_i.to_s).tap(&:mkpath)
|
|
52
|
+
|
|
53
|
+
managed_city_db_path = Pathname.new(DEFAULT_CITY_DB_PATH).basename.expand_path(managed_path).to_path
|
|
54
|
+
FileUtils.cp(DEFAULT_CITY_DB_PATH, managed_city_db_path)
|
|
22
55
|
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
let(:event2) { LogStash::Event.new("target" => { "ip" => "55.159.212.43" }) }
|
|
56
|
+
managed_asn_db_path = Pathname.new(DEFAULT_ASN_DB_PATH).basename.expand_path(managed_path).to_path
|
|
57
|
+
FileUtils.cp(DEFAULT_ASN_DB_PATH, managed_asn_db_path)
|
|
26
58
|
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
59
|
+
{
|
|
60
|
+
'City' => managed_city_db_path,
|
|
61
|
+
'ASN' => managed_asn_db_path,
|
|
62
|
+
}
|
|
31
63
|
end
|
|
32
64
|
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
plugin.register
|
|
38
|
-
plugin.filter(event2)
|
|
39
|
-
plugin.close
|
|
40
|
-
second_dirname = get_metadata_city_database_name
|
|
65
|
+
before(:each) do
|
|
66
|
+
allow(plugin).to receive(:load_database_manager?).and_return(true)
|
|
67
|
+
stub_const("LogStash::Filters::Geoip::DatabaseManager", double("DatabaseManager.Class", :instance => mock_manager))
|
|
68
|
+
end
|
|
41
69
|
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
70
|
+
let(:temp_data_path) { Stud::Temporary.directory }
|
|
71
|
+
after(:each) do
|
|
72
|
+
FileUtils.rm_rf(temp_data_path) if File.exist?(temp_data_path)
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
it "uses a managed database" do
|
|
76
|
+
plugin.register
|
|
77
|
+
plugin.filter(event)
|
|
78
|
+
expect(plugin.last_db_path).to_not be_nil
|
|
79
|
+
expect(plugin.last_db_path).to start_with(temp_data_path)
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
include_examples "event enrichment"
|
|
45
83
|
end
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
}
|
|
57
|
-
CONFIG
|
|
58
|
-
|
|
59
|
-
sample("ip" => "173.9.34.107") do
|
|
60
|
-
insist { subject.get("geoip") }.include?("ip")
|
|
61
|
-
expect(::File.exist?(METADATA_PATH)).to be_falsey
|
|
84
|
+
else
|
|
85
|
+
context "when geoip database management is not available" do
|
|
86
|
+
|
|
87
|
+
include_examples "event enrichment"
|
|
88
|
+
|
|
89
|
+
it "uses a plugin-vendored database" do
|
|
90
|
+
plugin.register
|
|
91
|
+
expect(plugin.last_db_path).to_not be_nil
|
|
92
|
+
expect(plugin.last_db_path).to include("/vendor/")
|
|
93
|
+
end
|
|
62
94
|
end
|
|
63
95
|
end
|
|
64
|
-
end
|
|
96
|
+
end
|
|
65
97
|
end
|
data/spec/filters/geoip_spec.rb
CHANGED
|
@@ -19,13 +19,13 @@ describe LogStash::Filters::GeoIP do
|
|
|
19
19
|
end
|
|
20
20
|
end
|
|
21
21
|
|
|
22
|
-
|
|
22
|
+
shared_examples "with database manager" do
|
|
23
23
|
it "load_database_manager? should be true" do
|
|
24
24
|
expect(plugin.load_database_manager?).to be_truthy
|
|
25
25
|
end
|
|
26
|
-
end
|
|
26
|
+
end
|
|
27
27
|
|
|
28
|
-
|
|
28
|
+
shared_examples "without database manager" do
|
|
29
29
|
it "load_database_manager? should be false" do
|
|
30
30
|
expect(plugin.load_database_manager?).to be_falsey
|
|
31
31
|
end
|
|
@@ -37,6 +37,24 @@ describe LogStash::Filters::GeoIP do
|
|
|
37
37
|
expect(plugin.select_database_path).to eql(DEFAULT_CITY_DB_PATH)
|
|
38
38
|
end
|
|
39
39
|
end
|
|
40
|
-
end
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
if MAJOR >= 8 || (MAJOR == 7 && MINOR >= 14)
|
|
43
|
+
context "Logstash >= 7.14" do
|
|
44
|
+
if LogStash::OSS
|
|
45
|
+
context "OSS-only" do
|
|
46
|
+
include_examples "without database manager"
|
|
47
|
+
end
|
|
48
|
+
else
|
|
49
|
+
context "default distro" do
|
|
50
|
+
include_examples "with database manager"
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
end
|
|
54
|
+
else
|
|
55
|
+
describe "Logstash < 7.14" do
|
|
56
|
+
include_examples "without database manager"
|
|
57
|
+
end
|
|
58
|
+
end
|
|
41
59
|
end
|
|
42
60
|
end
|
data/spec/filters/test_helper.rb
CHANGED
|
@@ -2,28 +2,46 @@ require "logstash-core/logstash-core"
|
|
|
2
2
|
require "digest"
|
|
3
3
|
require "csv"
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
5
|
+
# Since we use Logstash's x-pack WITHOUT the LogStash::Runner,
|
|
6
|
+
# we must find it relative to logstash-core and add it to the load path.
|
|
7
|
+
require 'pathname'
|
|
8
|
+
logstash_core_path = Gem.loaded_specs['logstash-core']&.full_gem_path or fail("logstash-core lib not found")
|
|
9
|
+
logstash_xpack_load_path = Pathname.new(logstash_core_path).join("../x-pack/lib").cleanpath.to_s
|
|
10
|
+
if ENV['OSS'] == "true" || !File.exist?(logstash_xpack_load_path)
|
|
11
|
+
$stderr.puts("X-PACK is not available")
|
|
12
|
+
LogStash::OSS = true
|
|
13
|
+
else
|
|
14
|
+
if !$LOAD_PATH.include?(logstash_xpack_load_path)
|
|
15
|
+
$stderr.puts("ADDING LOGSTASH X-PACK to load path: #{logstash_xpack_load_path}")
|
|
16
|
+
$LOAD_PATH.unshift(logstash_xpack_load_path)
|
|
17
|
+
end
|
|
18
|
+
LogStash::OSS = false
|
|
16
19
|
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
20
|
+
# when running in a Logstash process that has a geoip extension available, it will
|
|
21
|
+
# be loaded before this plugin is instantiated. In tests, we need to find and load the
|
|
22
|
+
# appropriate extension ourselves.
|
|
23
|
+
extension = nil
|
|
24
|
+
extension ||= begin; require 'geoip_database_management/extension'; LogStash::const_get("GeoipDatabaseManagement::Extension"); rescue Exception; nil; end
|
|
25
|
+
extension ||= begin; require 'filters/geoip/extension'; LogStash::const_get("Filters::Geoip::Extension"); rescue Exception; nil; end
|
|
26
|
+
if extension
|
|
27
|
+
$stderr.puts("loading logstash extension for geoip: #{extension}")
|
|
28
|
+
extension.new.tap do |instance|
|
|
29
|
+
# the extensions require logstash/runner even though they don't need to,
|
|
30
|
+
# resulting in _all_ extensions being loaded into the registry, including
|
|
31
|
+
# those whose dependencies are not met by this plugin's dependency graph.
|
|
32
|
+
def instance.require(path)
|
|
33
|
+
super unless path == "logstash/runner"
|
|
34
|
+
end
|
|
35
|
+
end.additionals_settings(LogStash::SETTINGS)
|
|
21
36
|
else
|
|
22
|
-
|
|
37
|
+
$stderr.puts("no logstash extension for geoip is available")
|
|
23
38
|
end
|
|
24
39
|
end
|
|
25
40
|
|
|
26
|
-
|
|
41
|
+
def get_vendor_path(filename)
|
|
42
|
+
::File.join(::File.expand_path("../../vendor/", ::File.dirname(__FILE__)), filename)
|
|
43
|
+
end
|
|
44
|
+
|
|
27
45
|
DEFAULT_CITY_DB_PATH = get_vendor_path("GeoLite2-City.mmdb")
|
|
28
46
|
DEFAULT_ASN_DB_PATH = get_vendor_path("GeoLite2-ASN.mmdb")
|
|
29
47
|
|
data/vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-core/2.16.0/jackson-core-2.16.0.jar
ADDED
|
Binary file
|
|
Binary file
|
metadata
CHANGED
|
@@ -1,16 +1,16 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: logstash-filter-geoip
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 7.
|
|
4
|
+
version: 7.3.4
|
|
5
5
|
platform: java
|
|
6
6
|
authors:
|
|
7
7
|
- Elastic
|
|
8
|
-
autorequire:
|
|
9
8
|
bindir: bin
|
|
10
9
|
cert_chain: []
|
|
11
|
-
date:
|
|
10
|
+
date: 2026-02-11 00:00:00.000000000 Z
|
|
12
11
|
dependencies:
|
|
13
12
|
- !ruby/object:Gem::Dependency
|
|
13
|
+
name: logstash-core-plugin-api
|
|
14
14
|
requirement: !ruby/object:Gem::Requirement
|
|
15
15
|
requirements:
|
|
16
16
|
- - ">="
|
|
@@ -19,9 +19,8 @@ dependencies:
|
|
|
19
19
|
- - "<="
|
|
20
20
|
- !ruby/object:Gem::Version
|
|
21
21
|
version: '2.99'
|
|
22
|
-
name: logstash-core-plugin-api
|
|
23
|
-
prerelease: false
|
|
24
22
|
type: :runtime
|
|
23
|
+
prerelease: false
|
|
25
24
|
version_requirements: !ruby/object:Gem::Requirement
|
|
26
25
|
requirements:
|
|
27
26
|
- - ">="
|
|
@@ -31,70 +30,70 @@ dependencies:
|
|
|
31
30
|
- !ruby/object:Gem::Version
|
|
32
31
|
version: '2.99'
|
|
33
32
|
- !ruby/object:Gem::Dependency
|
|
33
|
+
name: logstash-mixin-ecs_compatibility_support
|
|
34
34
|
requirement: !ruby/object:Gem::Requirement
|
|
35
35
|
requirements:
|
|
36
36
|
- - "~>"
|
|
37
37
|
- !ruby/object:Gem::Version
|
|
38
38
|
version: '1.2'
|
|
39
|
-
name: logstash-mixin-ecs_compatibility_support
|
|
40
|
-
prerelease: false
|
|
41
39
|
type: :runtime
|
|
40
|
+
prerelease: false
|
|
42
41
|
version_requirements: !ruby/object:Gem::Requirement
|
|
43
42
|
requirements:
|
|
44
43
|
- - "~>"
|
|
45
44
|
- !ruby/object:Gem::Version
|
|
46
45
|
version: '1.2'
|
|
47
46
|
- !ruby/object:Gem::Dependency
|
|
47
|
+
name: logstash-devutils
|
|
48
48
|
requirement: !ruby/object:Gem::Requirement
|
|
49
49
|
requirements:
|
|
50
50
|
- - ">="
|
|
51
51
|
- !ruby/object:Gem::Version
|
|
52
52
|
version: '0'
|
|
53
|
-
name: logstash-devutils
|
|
54
|
-
prerelease: false
|
|
55
53
|
type: :development
|
|
54
|
+
prerelease: false
|
|
56
55
|
version_requirements: !ruby/object:Gem::Requirement
|
|
57
56
|
requirements:
|
|
58
57
|
- - ">="
|
|
59
58
|
- !ruby/object:Gem::Version
|
|
60
59
|
version: '0'
|
|
61
60
|
- !ruby/object:Gem::Dependency
|
|
61
|
+
name: insist
|
|
62
62
|
requirement: !ruby/object:Gem::Requirement
|
|
63
63
|
requirements:
|
|
64
64
|
- - ">="
|
|
65
65
|
- !ruby/object:Gem::Version
|
|
66
66
|
version: '0'
|
|
67
|
-
name: insist
|
|
68
|
-
prerelease: false
|
|
69
67
|
type: :development
|
|
68
|
+
prerelease: false
|
|
70
69
|
version_requirements: !ruby/object:Gem::Requirement
|
|
71
70
|
requirements:
|
|
72
71
|
- - ">="
|
|
73
72
|
- !ruby/object:Gem::Version
|
|
74
73
|
version: '0'
|
|
75
74
|
- !ruby/object:Gem::Dependency
|
|
75
|
+
name: benchmark-ips
|
|
76
76
|
requirement: !ruby/object:Gem::Requirement
|
|
77
77
|
requirements:
|
|
78
78
|
- - ">="
|
|
79
79
|
- !ruby/object:Gem::Version
|
|
80
80
|
version: '0'
|
|
81
|
-
name: benchmark-ips
|
|
82
|
-
prerelease: false
|
|
83
81
|
type: :development
|
|
82
|
+
prerelease: false
|
|
84
83
|
version_requirements: !ruby/object:Gem::Requirement
|
|
85
84
|
requirements:
|
|
86
85
|
- - ">="
|
|
87
86
|
- !ruby/object:Gem::Version
|
|
88
87
|
version: '0'
|
|
89
88
|
- !ruby/object:Gem::Dependency
|
|
89
|
+
name: logstash-core
|
|
90
90
|
requirement: !ruby/object:Gem::Requirement
|
|
91
91
|
requirements:
|
|
92
92
|
- - ">="
|
|
93
93
|
- !ruby/object:Gem::Version
|
|
94
94
|
version: 7.14.0
|
|
95
|
-
name: logstash-core
|
|
96
|
-
prerelease: false
|
|
97
95
|
type: :runtime
|
|
96
|
+
prerelease: false
|
|
98
97
|
version_requirements: !ruby/object:Gem::Requirement
|
|
99
98
|
requirements:
|
|
100
99
|
- - ">="
|
|
@@ -114,6 +113,7 @@ files:
|
|
|
114
113
|
- LICENSE
|
|
115
114
|
- NOTICE.TXT
|
|
116
115
|
- README.md
|
|
116
|
+
- VERSION
|
|
117
117
|
- docs/index.asciidoc
|
|
118
118
|
- lib/logstash-filter-geoip_jars.rb
|
|
119
119
|
- lib/logstash/filters/geoip.rb
|
|
@@ -127,16 +127,16 @@ files:
|
|
|
127
127
|
- spec/filters/test_helper.rb
|
|
128
128
|
- vendor/GeoLite2-ASN.mmdb
|
|
129
129
|
- vendor/GeoLite2-City.mmdb
|
|
130
|
-
- vendor/jar-dependencies/com/
|
|
131
|
-
- vendor/jar-dependencies/com/maxmind/
|
|
132
|
-
- vendor/jar-dependencies/
|
|
133
|
-
|
|
130
|
+
- vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-core/2.16.0/jackson-core-2.16.0.jar
|
|
131
|
+
- vendor/jar-dependencies/com/maxmind/db/maxmind-db/2.1.0/maxmind-db-2.1.0.jar
|
|
132
|
+
- vendor/jar-dependencies/com/maxmind/geoip2/geoip2/2.17.0/geoip2-2.17.0.jar
|
|
133
|
+
- vendor/jar-dependencies/org/logstash/filters/logstash-filter-geoip/7.3.4/logstash-filter-geoip-7.3.4.jar
|
|
134
|
+
homepage: https://elastic.co/logstash
|
|
134
135
|
licenses:
|
|
135
136
|
- Apache License (2.0)
|
|
136
137
|
metadata:
|
|
137
138
|
logstash_plugin: 'true'
|
|
138
139
|
logstash_group: filter
|
|
139
|
-
post_install_message:
|
|
140
140
|
rdoc_options: []
|
|
141
141
|
require_paths:
|
|
142
142
|
- lib
|
|
@@ -152,8 +152,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
152
152
|
- !ruby/object:Gem::Version
|
|
153
153
|
version: '0'
|
|
154
154
|
requirements: []
|
|
155
|
-
rubygems_version: 3.
|
|
156
|
-
signing_key:
|
|
155
|
+
rubygems_version: 3.6.3
|
|
157
156
|
specification_version: 4
|
|
158
157
|
summary: Adds geographical information about an IP address
|
|
159
158
|
test_files:
|
|
Binary file
|
|
Binary file
|