fluent-plugin-geoip 1.2.0 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +90 -211
- data/fluent-plugin-geoip.gemspec +2 -1
- data/lib/fluent/plugin/filter_geoip.rb +180 -5
- data/test/helper.rb +1 -0
- data/test/plugin/test_filter_geoip.rb +137 -69
- metadata +17 -7
- data/lib/fluent/plugin/geoip.rb +0 -171
- data/lib/fluent/plugin/out_geoip.rb +0 -56
- data/test/plugin/test_out_geoip.rb +0 -1048
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: eab0d5c4f07207bfeba15ba4c11ba6b9d4acc8538235e6b2820e7207817803cb
|
|
4
|
+
data.tar.gz: 549a11f6f954945e37301b0cc010d8272193768a81e198f474552327e0d304ec
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 7380799ed242d60ee4281870b8c2b23261428f02b5d4aa1b2908039219659fa8730b596b4713f09da6ea3c63d96e7795f7420ce0c62142f29e4542c13eb32706
|
|
7
|
+
data.tar.gz: 737f87472b5d5e145294067be686b51b984433d9d360265fef59da0afa93a21d77a12db4612af0b4d3183c4439b33eebcf47e1ab5fd98a23f8e7701152adee30
|
data/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# fluent-plugin-geoip [](https://travis-ci.org/y-ken/fluent-plugin-geoip)
|
|
2
2
|
|
|
3
|
-
Fluentd Filter
|
|
3
|
+
Fluentd Filter plugin to add information about geographical location of IP addresses with Maxmind GeoIP databases.
|
|
4
4
|
|
|
5
5
|
fluent-plugin-geoip has bundled cost-free [GeoLite2 Free Downloadable Databases](https://dev.maxmind.com/geoip/geoip2/geolite2/) and [GeoLite City database](http://dev.maxmind.com/geoip/legacy/geolite/) by default.<br />
|
|
6
6
|
Also you can use purchased [GeoIP City database](http://www.maxmind.com/en/city) ([lang:ja](http://www.maxmind.com/ja/city)) which costs starting from $50.
|
|
@@ -21,56 +21,7 @@ If you want to use this plugin with Fluentd v0.12.x or earlier use 0.8.x.
|
|
|
21
21
|
|
|
22
22
|
### Compatibility notice
|
|
23
23
|
|
|
24
|
-
We've
|
|
25
|
-
So we have dropped some features from GeoipOutput.
|
|
26
|
-
|
|
27
|
-
See also [official document](http://docs.fluentd.org/v1.0/articles/plugin-update-from-v12)
|
|
28
|
-
|
|
29
|
-
#### Fluent::Mixin::RewriteTagName
|
|
30
|
-
|
|
31
|
-
* `${tag}`, `__TAG__`
|
|
32
|
-
|
|
33
|
-
Alternative: Use `${tag}` placeholder
|
|
34
|
-
|
|
35
|
-
* `${tag_parts[n]}`, `__TAG_PARTS[n]__`
|
|
36
|
-
|
|
37
|
-
Alternative: Use `${tag[n]}` placeholder
|
|
38
|
-
|
|
39
|
-
* `${hostname}`, `__HOSTNAME__`
|
|
40
|
-
|
|
41
|
-
Alternative1: Use filter before this plugin and chunk keys:
|
|
42
|
-
```
|
|
43
|
-
<filter>
|
|
44
|
-
@type record_transformer
|
|
45
|
-
<record>
|
|
46
|
-
hostname ${hostname}
|
|
47
|
-
</record>
|
|
48
|
-
</filter>
|
|
49
|
-
<match **>
|
|
50
|
-
@type geoip
|
|
51
|
-
tag geoip.${tag[1]}.${hostname}
|
|
52
|
-
<record>
|
|
53
|
-
city ${city["host"]}
|
|
54
|
-
</record>
|
|
55
|
-
<buffer tag, hostname>
|
|
56
|
-
flush_interval 1s
|
|
57
|
-
</buffer>
|
|
58
|
-
</match>
|
|
59
|
-
```
|
|
60
|
-
|
|
61
|
-
Alternative2: Just inject hostname into record you can use `<inject>` section instead:
|
|
62
|
-
```
|
|
63
|
-
<match **>
|
|
64
|
-
@type geoip
|
|
65
|
-
tag geoip.${tag[1]}.${hostname}
|
|
66
|
-
<record>
|
|
67
|
-
city ${city["host"]}
|
|
68
|
-
</record>
|
|
69
|
-
<inject>
|
|
70
|
-
hostname_key hostname
|
|
71
|
-
</inject>
|
|
72
|
-
</match>
|
|
73
|
-
```
|
|
24
|
+
We've removed GeoipOutput since 1.3.0, because GeoipFilter is enough to add information about geographical location of IP addresse.
|
|
74
25
|
|
|
75
26
|
## Dependency
|
|
76
27
|
|
|
@@ -90,6 +41,8 @@ $ brew install geoip
|
|
|
90
41
|
$ bundle config build.geoip-c --with-geoip-dir=/usr/local/include/
|
|
91
42
|
```
|
|
92
43
|
|
|
44
|
+
See [geoip2_c](https://github.com/okkez/geoip2_c#build-requirements), if you failed to install geoip2_c.
|
|
45
|
+
|
|
93
46
|
## Installation
|
|
94
47
|
|
|
95
48
|
install with `gem` or td-agent provided command as:
|
|
@@ -116,7 +69,6 @@ Note that filter version of geoip plugin does not have handling tag feature.
|
|
|
116
69
|
@type geoip
|
|
117
70
|
|
|
118
71
|
# Specify one or more geoip lookup field which has ip address (default: host)
|
|
119
|
-
# in the case of accessing nested value, delimit keys by dot like 'host.ip'.
|
|
120
72
|
geoip_lookup_keys host
|
|
121
73
|
|
|
122
74
|
# Specify optional geoip database (using bundled GeoLiteCity databse by default)
|
|
@@ -157,6 +109,92 @@ Note that filter version of geoip plugin does not have handling tag feature.
|
|
|
157
109
|
</filter>
|
|
158
110
|
```
|
|
159
111
|
|
|
112
|
+
#### Tips: Modify records without city information
|
|
113
|
+
|
|
114
|
+
```
|
|
115
|
+
<filter access.apache>
|
|
116
|
+
@type geoip
|
|
117
|
+
geoip_lookup_keys remote_addr
|
|
118
|
+
<record>
|
|
119
|
+
city ${city.names.en["remote_addr"]} # skip adding fields if this field is null
|
|
120
|
+
latitude ${location.latitude["remote_addr"]}
|
|
121
|
+
longitude ${location.longitude["remote_addr"]}
|
|
122
|
+
country ${country.iso_code["remote_addr"]}
|
|
123
|
+
country_name ${country.names.en["remote_addr"]}
|
|
124
|
+
postal_code ${postal.code["remote_addr"]}
|
|
125
|
+
</record>
|
|
126
|
+
skip_adding_null_record true
|
|
127
|
+
</filter>
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
Skip adding fields if incoming `remote_addr`'s GeoIP data is like following:
|
|
131
|
+
|
|
132
|
+
```ruby
|
|
133
|
+
# the record does not have "city" field
|
|
134
|
+
{"continent"=>
|
|
135
|
+
{"code"=>"NA",
|
|
136
|
+
"geoname_id"=>6255149,
|
|
137
|
+
"names"=>
|
|
138
|
+
{"de"=>"Nordamerika",
|
|
139
|
+
"en"=>"North America",
|
|
140
|
+
"es"=>"Norteamérica",
|
|
141
|
+
"fr"=>"Amérique du Nord",
|
|
142
|
+
"ja"=>"北アメリカ",
|
|
143
|
+
"pt-BR"=>"América do Norte",
|
|
144
|
+
"ru"=>"Северная Америка",
|
|
145
|
+
"zh-CN"=>"北美洲"}},
|
|
146
|
+
"country"=>
|
|
147
|
+
{"geoname_id"=>6252001,
|
|
148
|
+
"iso_code"=>"US",
|
|
149
|
+
"names"=>
|
|
150
|
+
{"de"=>"USA",
|
|
151
|
+
"en"=>"United States",
|
|
152
|
+
"es"=>"Estados Unidos",
|
|
153
|
+
"fr"=>"États-Unis",
|
|
154
|
+
"ja"=>"アメリカ合衆国",
|
|
155
|
+
"pt-BR"=>"Estados Unidos",
|
|
156
|
+
"ru"=>"США",
|
|
157
|
+
"zh-CN"=>"美国"}},
|
|
158
|
+
"location"=>
|
|
159
|
+
{"accuracy_radius"=>1000, "latitude"=>37.751, "longitude"=>-97.822},
|
|
160
|
+
"registered_country"=>
|
|
161
|
+
{"geoname_id"=>6252001,
|
|
162
|
+
"iso_code"=>"US",
|
|
163
|
+
"names"=>
|
|
164
|
+
{"de"=>"USA",
|
|
165
|
+
"en"=>"United States",
|
|
166
|
+
"es"=>"Estados Unidos",
|
|
167
|
+
"fr"=>"États-Unis",
|
|
168
|
+
"ja"=>"アメリカ合衆国",
|
|
169
|
+
"pt-BR"=>"Estados Unidos",
|
|
170
|
+
"ru"=>"США",
|
|
171
|
+
"zh-CN"=>"美国"}}}
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
We can avoid this behavior changing field order in `<record>` like following:
|
|
175
|
+
|
|
176
|
+
```
|
|
177
|
+
<filter access.apache>
|
|
178
|
+
@type geoip
|
|
179
|
+
geoip_lookup_keys remote_addr
|
|
180
|
+
<record>
|
|
181
|
+
latitude ${location.latitude["remote_addr"]} # this field must not be null
|
|
182
|
+
longitude ${location.longitude["remote_addr"]}
|
|
183
|
+
country ${country.iso_code["remote_addr"]}
|
|
184
|
+
country_name ${country.names.en["remote_addr"]}
|
|
185
|
+
postal_code ${postal.code["remote_addr"]}
|
|
186
|
+
city ${city.names.en["remote_addr"]} # adding fields even if this field is null
|
|
187
|
+
</record>
|
|
188
|
+
skip_adding_null_record true
|
|
189
|
+
</filter>
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
#### Tips: nested attributes for geoip_lookup_keys
|
|
193
|
+
|
|
194
|
+
See [Record Accessor Plugin](https://docs.fluentd.org/v1.0/articles/api-plugin-helper-record_accessor#syntax)
|
|
195
|
+
|
|
196
|
+
**NOTE** Since v1.3.0 does not interpret `host.ip` as nested attribute.
|
|
197
|
+
|
|
160
198
|
#### Advanced config samples
|
|
161
199
|
|
|
162
200
|
It is a sample to get friendly geo point recdords for elasticsearch with Yajl (JSON) parser.<br />
|
|
@@ -199,51 +237,6 @@ On the case of using td-agent3 (v1-config), it have to quote `{ ... }` or `[ ...
|
|
|
199
237
|
</filter>
|
|
200
238
|
```
|
|
201
239
|
|
|
202
|
-
### For GeoipOutput
|
|
203
|
-
|
|
204
|
-
```xml
|
|
205
|
-
<match access.apache>
|
|
206
|
-
@type geoip
|
|
207
|
-
|
|
208
|
-
# Specify one or more geoip lookup field which has ip address (default: host)
|
|
209
|
-
# in the case of accessing nested value, delimit keys by dot like 'host.ip'.
|
|
210
|
-
geoip_lookup_keys host
|
|
211
|
-
|
|
212
|
-
# Specify optional geoip database (using bundled GeoLiteCity databse by default)
|
|
213
|
-
geoip_database "/path/to/your/GeoIPCity.dat"
|
|
214
|
-
# Specify optional geoip2 database
|
|
215
|
-
# geoip2_database "/path/to/your/GeoLite2-City.mmdb"
|
|
216
|
-
# Specify backend library (geoip, geoip2_compat, geoip2_c)
|
|
217
|
-
backend_library geoip
|
|
218
|
-
|
|
219
|
-
# Set adding field with placeholder (more than one settings are required.)
|
|
220
|
-
<record>
|
|
221
|
-
latitude ${location.latitude["host"]}
|
|
222
|
-
longitude ${location.longitude["host"]}
|
|
223
|
-
country ${country.iso_code["host"]}
|
|
224
|
-
country_name ${country.names.en["host"]}
|
|
225
|
-
postal_code ${postal.code["host"]}
|
|
226
|
-
region ${subdivisions.0.iso_code["host"]}
|
|
227
|
-
region_name ${subdivisions.0.names.en["host"]}
|
|
228
|
-
city ${city.names.en["host"]}
|
|
229
|
-
</record>
|
|
230
|
-
|
|
231
|
-
# Settings for tag
|
|
232
|
-
tag geoip.${tag[1]}
|
|
233
|
-
|
|
234
|
-
# To avoid get stacktrace error with `[null, null]` array for elasticsearch.
|
|
235
|
-
skip_adding_null_record true
|
|
236
|
-
|
|
237
|
-
# Set @log_level (default: warn)
|
|
238
|
-
@log_level info
|
|
239
|
-
|
|
240
|
-
<buffer tag>
|
|
241
|
-
# Set buffering time (default: 0s)
|
|
242
|
-
flush_interval 1s
|
|
243
|
-
</buffer>
|
|
244
|
-
</match>
|
|
245
|
-
```
|
|
246
|
-
|
|
247
240
|
## Tutorial
|
|
248
241
|
|
|
249
242
|
### For GeoipFilter
|
|
@@ -255,7 +248,6 @@ On the case of using td-agent3 (v1-config), it have to quote `{ ... }` or `[ ...
|
|
|
255
248
|
@type forward
|
|
256
249
|
</source>
|
|
257
250
|
|
|
258
|
-
|
|
259
251
|
<filter test.geoip>
|
|
260
252
|
@type geoip
|
|
261
253
|
geoip_lookup_keys host
|
|
@@ -290,57 +282,6 @@ $ bundle exec ruby urils/dump.rb geoip2_compat 66.102.3.80
|
|
|
290
282
|
$ bundle exec ruby urils/dump.rb geoip 66.102.3.80
|
|
291
283
|
```
|
|
292
284
|
|
|
293
|
-
### For GeoipOutput
|
|
294
|
-
|
|
295
|
-
#### configuration
|
|
296
|
-
|
|
297
|
-
```xml
|
|
298
|
-
<source>
|
|
299
|
-
@type forward
|
|
300
|
-
</source>
|
|
301
|
-
|
|
302
|
-
<match test.geoip>
|
|
303
|
-
@type copy
|
|
304
|
-
<store>
|
|
305
|
-
@type stdout
|
|
306
|
-
</store>
|
|
307
|
-
<store>
|
|
308
|
-
@type geoip
|
|
309
|
-
geoip_lookup_keys host
|
|
310
|
-
<record>
|
|
311
|
-
lat ${location.latitude["host"]}
|
|
312
|
-
lon ${location.longitude["host"]}
|
|
313
|
-
country ${country.iso_code["host"]}
|
|
314
|
-
</record>
|
|
315
|
-
tag debug.${tag[1]}
|
|
316
|
-
</store>
|
|
317
|
-
</match>
|
|
318
|
-
|
|
319
|
-
<match debug.**>
|
|
320
|
-
@type stdout
|
|
321
|
-
</match>
|
|
322
|
-
```
|
|
323
|
-
|
|
324
|
-
#### result
|
|
325
|
-
|
|
326
|
-
```bash
|
|
327
|
-
# forward record with Google's ip address.
|
|
328
|
-
$ echo '{"host":"66.102.9.80","message":"test"}' | fluent-cat test.geoip
|
|
329
|
-
|
|
330
|
-
# check the result at stdout
|
|
331
|
-
$ tail /var/log/td-agent/td-agent.log
|
|
332
|
-
2013-08-04 16:21:32 +0900 test.geoip: {"host":"66.102.9.80","message":"test"}
|
|
333
|
-
2013-08-04 16:21:32 +0900 debug.geoip: {"host":"66.102.9.80","message":"test","lat":37.4192008972168,"lon":-122.05740356445312,"country":"US"}
|
|
334
|
-
```
|
|
335
|
-
|
|
336
|
-
You can check geoip data format using [utils/dump.rb](https://github.com/okkez/fluent-plugin-geoip/utils/dump.rb).
|
|
337
|
-
|
|
338
|
-
```
|
|
339
|
-
$ bundle exec ruby urils/dump.rb geoip2 66.102.3.80
|
|
340
|
-
$ bundle exec ruby urils/dump.rb geoip2_compat 66.102.3.80
|
|
341
|
-
$ bundle exec ruby urils/dump.rb geoip 66.102.3.80
|
|
342
|
-
```
|
|
343
|
-
|
|
344
285
|
## Placeholders
|
|
345
286
|
|
|
346
287
|
### GeoIP2
|
|
@@ -465,68 +406,6 @@ On the case of getting nothing of GeoIP info (such as local IP), it will output
|
|
|
465
406
|
|
|
466
407
|
Set backend library.
|
|
467
408
|
|
|
468
|
-
### GeoipOutput
|
|
469
|
-
|
|
470
|
-
#### Plugin helpers
|
|
471
|
-
|
|
472
|
-
* [event_emitter](https://docs.fluentd.org/v1.0/articles/api-plugin-helper-event_emitter)
|
|
473
|
-
* [compat_parameters](https://docs.fluentd.org/v1.0/articles/api-plugin-helper-compat_parameters)
|
|
474
|
-
* [inject](https://docs.fluentd.org/v1.0/articles/api-plugin-helper-inject)
|
|
475
|
-
|
|
476
|
-
See also [Output Plugin Overview](https://docs.fluentd.org/v1.0/articles/output-plugin-overview)
|
|
477
|
-
|
|
478
|
-
#### Sections
|
|
479
|
-
|
|
480
|
-
* [Inject section configurations](https://docs.fluentd.org/v1.0/articles/inject-section)
|
|
481
|
-
* [Buffer section configurations](https://docs.fluentd.org/v1.0/articles/buffer-section)
|
|
482
|
-
|
|
483
|
-
#### Parameters
|
|
484
|
-
|
|
485
|
-
[Plugin Common Paramteters](https://docs.fluentd.org/v1.0/articles/plugin-common-parameters)
|
|
486
|
-
|
|
487
|
-
**geoip_database** (string) (optional)
|
|
488
|
-
|
|
489
|
-
* Default value: bundled database `GeoLiteCity.dat`
|
|
490
|
-
|
|
491
|
-
Path to GeoIP database file.
|
|
492
|
-
|
|
493
|
-
**geoip2_database** (string) (optional)
|
|
494
|
-
|
|
495
|
-
* Default value: bundled database `GeoLite2-City.mmdb`.
|
|
496
|
-
|
|
497
|
-
Path to GeoIP2 database file.
|
|
498
|
-
|
|
499
|
-
**geoip_lookup_keys** (array) (optional)
|
|
500
|
-
|
|
501
|
-
* Default value: `["host"]`
|
|
502
|
-
|
|
503
|
-
Specify one or more geoip lookup field which has IP address.
|
|
504
|
-
|
|
505
|
-
**geoip_lookup_key** (string) (optional) (deprecated)
|
|
506
|
-
|
|
507
|
-
* Default value: `nil`.
|
|
508
|
-
|
|
509
|
-
Use geoip_lookup_keys instead.
|
|
510
|
-
|
|
511
|
-
**skip_adding_null_record** (bool) (optional)
|
|
512
|
-
|
|
513
|
-
* Default value: `nil`
|
|
514
|
-
|
|
515
|
-
Skip adding geoip fields when this valaues to `true`.
|
|
516
|
-
On the case of getting nothing of GeoIP info (such as local IP), it will output the original record without changing anything.
|
|
517
|
-
|
|
518
|
-
**backend_library** (enum) (optional)
|
|
519
|
-
|
|
520
|
-
* Available values: `geoip`, `geoip2_compat`, `geoip2_c`
|
|
521
|
-
* Default value: `geoip2_c`.
|
|
522
|
-
|
|
523
|
-
Set backend library.
|
|
524
|
-
|
|
525
|
-
**tag** (string) (optional)
|
|
526
|
-
|
|
527
|
-
On using this option with tag placeholder like `tag geoip.${tag}` (test code is available at [test_out_geoip.rb](https://github.com/y-ken/fluent-plugin-geoip/blob/master/test/plugin/test_out_geoip.rb)).
|
|
528
|
-
|
|
529
|
-
|
|
530
409
|
## Articles
|
|
531
410
|
|
|
532
411
|
* [IPアドレスを元に位置情報をリアルタイムに付与する fluent-plugin-geoip v0.0.1をリリースしました #fluentd - Y-Ken Studio](http://y-ken.hatenablog.com/entry/fluent-plugin-geoip-has-released)<br />
|
data/fluent-plugin-geoip.gemspec
CHANGED
|
@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
|
|
4
4
|
|
|
5
5
|
Gem::Specification.new do |spec|
|
|
6
6
|
spec.name = "fluent-plugin-geoip"
|
|
7
|
-
spec.version = "1.
|
|
7
|
+
spec.version = "1.3.0"
|
|
8
8
|
spec.authors = ["Kentaro Yoshida"]
|
|
9
9
|
spec.email = ["y.ken.studio@gmail.com"]
|
|
10
10
|
spec.summary = %q{Fluentd Filter plugin to add information about geographical location of IP addresses with Maxmind GeoIP databases.}
|
|
@@ -20,6 +20,7 @@ Gem::Specification.new do |spec|
|
|
|
20
20
|
spec.add_development_dependency "rake"
|
|
21
21
|
spec.add_development_dependency "appraisal"
|
|
22
22
|
spec.add_development_dependency "test-unit", ">= 3.1.0"
|
|
23
|
+
spec.add_development_dependency "test-unit-rr"
|
|
23
24
|
spec.add_development_dependency "geoip2_compat"
|
|
24
25
|
|
|
25
26
|
spec.add_runtime_dependency "fluentd", [">= 0.14.8", "< 2"]
|
|
@@ -1,11 +1,37 @@
|
|
|
1
1
|
require 'fluent/plugin/filter'
|
|
2
|
-
|
|
2
|
+
|
|
3
|
+
require 'geoip'
|
|
4
|
+
require 'yajl'
|
|
5
|
+
unless {}.respond_to?(:dig)
|
|
6
|
+
begin
|
|
7
|
+
# backport_dig is faster than dig_rb so prefer backport_dig.
|
|
8
|
+
# And Fluentd v1.0.1 uses backport_dig
|
|
9
|
+
require 'backport_dig'
|
|
10
|
+
rescue LoadError
|
|
11
|
+
require 'dig_rb'
|
|
12
|
+
end
|
|
13
|
+
end
|
|
3
14
|
|
|
4
15
|
module Fluent::Plugin
|
|
5
16
|
class GeoipFilter < Fluent::Plugin::Filter
|
|
6
17
|
Fluent::Plugin.register_filter('geoip', self)
|
|
7
18
|
|
|
8
|
-
|
|
19
|
+
BACKEND_LIBRARIES = [:geoip, :geoip2_compat, :geoip2_c]
|
|
20
|
+
|
|
21
|
+
REGEXP_PLACEHOLDER_SINGLE = /^\$\{
|
|
22
|
+
(?<geoip_key>-?[^\[\]]+)
|
|
23
|
+
\[
|
|
24
|
+
(?:(?<dq>")|(?<sq>'))
|
|
25
|
+
(?<record_key>-?(?(<dq>)[^"{}]+|[^'{}]+))
|
|
26
|
+
(?(<dq>)"|')
|
|
27
|
+
\]
|
|
28
|
+
\}$/x
|
|
29
|
+
REGEXP_PLACEHOLDER_SCAN = /['"]?(\$\{[^\}]+?\})['"]?/
|
|
30
|
+
|
|
31
|
+
GEOIP_KEYS = %w(city latitude longitude country_code3 country_code country_name dma_code area_code region)
|
|
32
|
+
GEOIP2_COMPAT_KEYS = %w(city country_code country_name latitude longitude postal_code region region_name)
|
|
33
|
+
|
|
34
|
+
helpers :compat_parameters, :inject, :record_accessor
|
|
9
35
|
|
|
10
36
|
config_param :geoip_database, :string, default: File.expand_path('../../../data/GeoLiteCity.dat', __dir__)
|
|
11
37
|
config_param :geoip2_database, :string, default: File.expand_path('../../../data/GeoLite2-City.mmdb', __dir__)
|
|
@@ -15,16 +41,69 @@ module Fluent::Plugin
|
|
|
15
41
|
|
|
16
42
|
config_set_default :@log_level, "warn"
|
|
17
43
|
|
|
18
|
-
config_param :backend_library, :enum, list:
|
|
44
|
+
config_param :backend_library, :enum, list: BACKEND_LIBRARIES, default: :geoip2_c
|
|
19
45
|
|
|
20
46
|
def configure(conf)
|
|
21
47
|
compat_parameters_convert(conf, :inject)
|
|
22
48
|
super
|
|
23
|
-
|
|
49
|
+
|
|
50
|
+
@map = {}
|
|
51
|
+
if @geoip_lookup_key
|
|
52
|
+
@geoip_lookup_keys = @geoip_lookup_key.split(/\s*,\s*/)
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
@geoip_lookup_keys.each do |key|
|
|
56
|
+
if key.include?(".") && !key.start_with?("$")
|
|
57
|
+
$log.warn("#{key} is not treated as nested attributes")
|
|
58
|
+
end
|
|
59
|
+
end
|
|
60
|
+
@geoip_lookup_accessors = @geoip_lookup_keys.map {|key| [key, record_accessor_create(key)] }.to_h
|
|
61
|
+
|
|
62
|
+
if conf.keys.any? {|k| k =~ /^enable_key_/ }
|
|
63
|
+
raise Fluent::ConfigError, "geoip: 'enable_key_*' config format is obsoleted. use <record></record> directive instead."
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
# <record></record> directive
|
|
67
|
+
conf.elements.select { |element| element.name == 'record' }.each { |element|
|
|
68
|
+
element.each_pair { |k, v|
|
|
69
|
+
element.has_key?(k) # to suppress unread configuration warning
|
|
70
|
+
v = v[1..v.size-2] if quoted_value?(v)
|
|
71
|
+
@map[k] = v
|
|
72
|
+
validate_json = Proc.new {
|
|
73
|
+
begin
|
|
74
|
+
dummy_text = Yajl::Encoder.encode('dummy_text')
|
|
75
|
+
Yajl::Parser.parse(v.gsub(REGEXP_PLACEHOLDER_SCAN, dummy_text))
|
|
76
|
+
rescue Yajl::ParseError => e
|
|
77
|
+
message = "geoip: failed to parse '#{v}' as json."
|
|
78
|
+
log.error message, error: e
|
|
79
|
+
raise Fluent::ConfigError, message
|
|
80
|
+
end
|
|
81
|
+
}
|
|
82
|
+
validate_json.call if json?(v.tr('\'"\\', ''))
|
|
83
|
+
}
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
@placeholder_keys = @map.values.join.scan(REGEXP_PLACEHOLDER_SCAN).map{|placeholder| placeholder[0] }.uniq
|
|
87
|
+
@placeholder_keys.each do |key|
|
|
88
|
+
m = key.match(REGEXP_PLACEHOLDER_SINGLE)
|
|
89
|
+
geoip_key = m[:geoip_key]
|
|
90
|
+
case @backend_library
|
|
91
|
+
when :geoip
|
|
92
|
+
raise Fluent::ConfigError, "#{@backend_library}: unsupported key #{geoip_key}" unless GEOIP_KEYS.include?(geoip_key)
|
|
93
|
+
when :geoip2_compat
|
|
94
|
+
raise Fluent::ConfigError, "#{@backend_library}: unsupported key #{geoip_key}" unless GEOIP2_COMPAT_KEYS.include?(geoip_key)
|
|
95
|
+
when :geoip2_c
|
|
96
|
+
# Nothing to do.
|
|
97
|
+
# We cannot define supported key(s) before we fetch values from GeoIP2 database
|
|
98
|
+
# because geoip2_c can fetch any fields in GeoIP2 database.
|
|
99
|
+
end
|
|
100
|
+
end
|
|
101
|
+
|
|
102
|
+
@geoip = load_database
|
|
24
103
|
end
|
|
25
104
|
|
|
26
105
|
def filter(tag, time, record)
|
|
27
|
-
filtered_record =
|
|
106
|
+
filtered_record = add_geoip_field(record)
|
|
28
107
|
if filtered_record
|
|
29
108
|
record = filtered_record
|
|
30
109
|
end
|
|
@@ -35,5 +114,101 @@ module Fluent::Plugin
|
|
|
35
114
|
def multi_workers_ready?
|
|
36
115
|
true
|
|
37
116
|
end
|
|
117
|
+
|
|
118
|
+
private
|
|
119
|
+
|
|
120
|
+
def add_geoip_field(record)
|
|
121
|
+
placeholder = create_placeholder(geolocate(get_address(record)))
|
|
122
|
+
return record if @skip_adding_null_record && placeholder.values.first.nil?
|
|
123
|
+
@map.each do |record_key, value|
|
|
124
|
+
if value.match(REGEXP_PLACEHOLDER_SINGLE) #|| value.match(REGEXP_PLACEHOLDER_BRACKET_SINGLE)
|
|
125
|
+
rewrited = placeholder[value]
|
|
126
|
+
elsif json?(value)
|
|
127
|
+
rewrited = value.gsub(REGEXP_PLACEHOLDER_SCAN) {|match|
|
|
128
|
+
match = match[1..match.size-2] if quoted_value?(match)
|
|
129
|
+
Yajl::Encoder.encode(placeholder[match])
|
|
130
|
+
}
|
|
131
|
+
rewrited = parse_json(rewrited)
|
|
132
|
+
else
|
|
133
|
+
rewrited = value.gsub(REGEXP_PLACEHOLDER_SCAN, placeholder)
|
|
134
|
+
end
|
|
135
|
+
record[record_key] = rewrited
|
|
136
|
+
end
|
|
137
|
+
record
|
|
138
|
+
end
|
|
139
|
+
|
|
140
|
+
def json?(text)
|
|
141
|
+
text.match(/^\[.+\]$/) || text.match(/^\{.+\}$/)
|
|
142
|
+
end
|
|
143
|
+
|
|
144
|
+
def quoted_value?(text)
|
|
145
|
+
# to improbe compatibility with fluentd v1-config
|
|
146
|
+
text.match(/(^'.+'$|^".+"$)/)
|
|
147
|
+
end
|
|
148
|
+
|
|
149
|
+
def parse_json(message)
|
|
150
|
+
begin
|
|
151
|
+
return Yajl::Parser.parse(message)
|
|
152
|
+
rescue Yajl::ParseError => e
|
|
153
|
+
log.info "geoip: failed to parse '#{message}' as json.", error_class: e.class, error: e.message
|
|
154
|
+
return nil
|
|
155
|
+
end
|
|
156
|
+
end
|
|
157
|
+
|
|
158
|
+
def get_address(record)
|
|
159
|
+
address = {}
|
|
160
|
+
@geoip_lookup_accessors.each do |field, accessor|
|
|
161
|
+
address[field] = accessor.call(record)
|
|
162
|
+
end
|
|
163
|
+
address
|
|
164
|
+
end
|
|
165
|
+
|
|
166
|
+
def geolocate(addresses)
|
|
167
|
+
geodata = {}
|
|
168
|
+
addresses.each do |field, ip|
|
|
169
|
+
geo = nil
|
|
170
|
+
if ip
|
|
171
|
+
geo = if @geoip.respond_to?(:look_up)
|
|
172
|
+
@geoip.look_up(ip)
|
|
173
|
+
else
|
|
174
|
+
@geoip.lookup(ip)
|
|
175
|
+
end
|
|
176
|
+
end
|
|
177
|
+
geodata[field] = geo
|
|
178
|
+
end
|
|
179
|
+
geodata
|
|
180
|
+
end
|
|
181
|
+
|
|
182
|
+
def create_placeholder(geodata)
|
|
183
|
+
placeholder = {}
|
|
184
|
+
@placeholder_keys.each do |placeholder_key|
|
|
185
|
+
position = placeholder_key.match(REGEXP_PLACEHOLDER_SINGLE)
|
|
186
|
+
next if position.nil? or geodata[position[:record_key]].nil?
|
|
187
|
+
keys = [position[:record_key]] + position[:geoip_key].split('.').map(&:to_sym)
|
|
188
|
+
value = geodata.dig(*keys)
|
|
189
|
+
value = if [:latitude, :longitude].include?(keys.last)
|
|
190
|
+
value || 0.0
|
|
191
|
+
else
|
|
192
|
+
value
|
|
193
|
+
end
|
|
194
|
+
placeholder[placeholder_key] = value
|
|
195
|
+
end
|
|
196
|
+
placeholder
|
|
197
|
+
end
|
|
198
|
+
|
|
199
|
+
def load_database
|
|
200
|
+
case @backend_library
|
|
201
|
+
when :geoip
|
|
202
|
+
::GeoIP::City.new(@geoip_database, :memory, false)
|
|
203
|
+
when :geoip2_compat
|
|
204
|
+
require 'geoip2_compat'
|
|
205
|
+
GeoIP2Compat.new(@geoip2_database)
|
|
206
|
+
when :geoip2_c
|
|
207
|
+
require 'geoip2'
|
|
208
|
+
GeoIP2::Database.new(@geoip2_database)
|
|
209
|
+
end
|
|
210
|
+
rescue LoadError
|
|
211
|
+
raise Fluent::ConfigError, "You must install #{@backend_library} gem."
|
|
212
|
+
end
|
|
38
213
|
end
|
|
39
214
|
end
|