fluent-plugin-geoip 1.2.0 → 1.3.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +90 -211
- data/fluent-plugin-geoip.gemspec +2 -1
- data/lib/fluent/plugin/filter_geoip.rb +180 -5
- data/test/helper.rb +1 -0
- data/test/plugin/test_filter_geoip.rb +137 -69
- metadata +17 -7
- data/lib/fluent/plugin/geoip.rb +0 -171
- data/lib/fluent/plugin/out_geoip.rb +0 -56
- data/test/plugin/test_out_geoip.rb +0 -1048
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: eab0d5c4f07207bfeba15ba4c11ba6b9d4acc8538235e6b2820e7207817803cb
|
|
4
|
+
data.tar.gz: 549a11f6f954945e37301b0cc010d8272193768a81e198f474552327e0d304ec
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 7380799ed242d60ee4281870b8c2b23261428f02b5d4aa1b2908039219659fa8730b596b4713f09da6ea3c63d96e7795f7420ce0c62142f29e4542c13eb32706
|
|
7
|
+
data.tar.gz: 737f87472b5d5e145294067be686b51b984433d9d360265fef59da0afa93a21d77a12db4612af0b4d3183c4439b33eebcf47e1ab5fd98a23f8e7701152adee30
|
data/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# fluent-plugin-geoip [](https://travis-ci.org/y-ken/fluent-plugin-geoip)
|
|
2
2
|
|
|
3
|
-
Fluentd Filter
|
|
3
|
+
Fluentd Filter plugin to add information about geographical location of IP addresses with Maxmind GeoIP databases.
|
|
4
4
|
|
|
5
5
|
fluent-plugin-geoip has bundled cost-free [GeoLite2 Free Downloadable Databases](https://dev.maxmind.com/geoip/geoip2/geolite2/) and [GeoLite City database](http://dev.maxmind.com/geoip/legacy/geolite/) by default.<br />
|
|
6
6
|
Also you can use purchased [GeoIP City database](http://www.maxmind.com/en/city) ([lang:ja](http://www.maxmind.com/ja/city)) which costs starting from $50.
|
|
@@ -21,56 +21,7 @@ If you want to use this plugin with Fluentd v0.12.x or earlier use 0.8.x.
|
|
|
21
21
|
|
|
22
22
|
### Compatibility notice
|
|
23
23
|
|
|
24
|
-
We've
|
|
25
|
-
So we have dropped some features from GeoipOutput.
|
|
26
|
-
|
|
27
|
-
See also [official document](http://docs.fluentd.org/v1.0/articles/plugin-update-from-v12)
|
|
28
|
-
|
|
29
|
-
#### Fluent::Mixin::RewriteTagName
|
|
30
|
-
|
|
31
|
-
* `${tag}`, `__TAG__`
|
|
32
|
-
|
|
33
|
-
Alternative: Use `${tag}` placeholder
|
|
34
|
-
|
|
35
|
-
* `${tag_parts[n]}`, `__TAG_PARTS[n]__`
|
|
36
|
-
|
|
37
|
-
Alternative: Use `${tag[n]}` placeholder
|
|
38
|
-
|
|
39
|
-
* `${hostname}`, `__HOSTNAME__`
|
|
40
|
-
|
|
41
|
-
Alternative1: Use filter before this plugin and chunk keys:
|
|
42
|
-
```
|
|
43
|
-
<filter>
|
|
44
|
-
@type record_transformer
|
|
45
|
-
<record>
|
|
46
|
-
hostname ${hostname}
|
|
47
|
-
</record>
|
|
48
|
-
</filter>
|
|
49
|
-
<match **>
|
|
50
|
-
@type geoip
|
|
51
|
-
tag geoip.${tag[1]}.${hostname}
|
|
52
|
-
<record>
|
|
53
|
-
city ${city["host"]}
|
|
54
|
-
</record>
|
|
55
|
-
<buffer tag, hostname>
|
|
56
|
-
flush_interval 1s
|
|
57
|
-
</buffer>
|
|
58
|
-
</match>
|
|
59
|
-
```
|
|
60
|
-
|
|
61
|
-
Alternative2: Just inject hostname into record you can use `<inject>` section instead:
|
|
62
|
-
```
|
|
63
|
-
<match **>
|
|
64
|
-
@type geoip
|
|
65
|
-
tag geoip.${tag[1]}.${hostname}
|
|
66
|
-
<record>
|
|
67
|
-
city ${city["host"]}
|
|
68
|
-
</record>
|
|
69
|
-
<inject>
|
|
70
|
-
hostname_key hostname
|
|
71
|
-
</inject>
|
|
72
|
-
</match>
|
|
73
|
-
```
|
|
24
|
+
We've removed GeoipOutput since 1.3.0, because GeoipFilter is enough to add information about geographical location of IP addresse.
|
|
74
25
|
|
|
75
26
|
## Dependency
|
|
76
27
|
|
|
@@ -90,6 +41,8 @@ $ brew install geoip
|
|
|
90
41
|
$ bundle config build.geoip-c --with-geoip-dir=/usr/local/include/
|
|
91
42
|
```
|
|
92
43
|
|
|
44
|
+
See [geoip2_c](https://github.com/okkez/geoip2_c#build-requirements), if you failed to install geoip2_c.
|
|
45
|
+
|
|
93
46
|
## Installation
|
|
94
47
|
|
|
95
48
|
install with `gem` or td-agent provided command as:
|
|
@@ -116,7 +69,6 @@ Note that filter version of geoip plugin does not have handling tag feature.
|
|
|
116
69
|
@type geoip
|
|
117
70
|
|
|
118
71
|
# Specify one or more geoip lookup field which has ip address (default: host)
|
|
119
|
-
# in the case of accessing nested value, delimit keys by dot like 'host.ip'.
|
|
120
72
|
geoip_lookup_keys host
|
|
121
73
|
|
|
122
74
|
# Specify optional geoip database (using bundled GeoLiteCity databse by default)
|
|
@@ -157,6 +109,92 @@ Note that filter version of geoip plugin does not have handling tag feature.
|
|
|
157
109
|
</filter>
|
|
158
110
|
```
|
|
159
111
|
|
|
112
|
+
#### Tips: Modify records without city information
|
|
113
|
+
|
|
114
|
+
```
|
|
115
|
+
<filter access.apache>
|
|
116
|
+
@type geoip
|
|
117
|
+
geoip_lookup_keys remote_addr
|
|
118
|
+
<record>
|
|
119
|
+
city ${city.names.en["remote_addr"]} # skip adding fields if this field is null
|
|
120
|
+
latitude ${location.latitude["remote_addr"]}
|
|
121
|
+
longitude ${location.longitude["remote_addr"]}
|
|
122
|
+
country ${country.iso_code["remote_addr"]}
|
|
123
|
+
country_name ${country.names.en["remote_addr"]}
|
|
124
|
+
postal_code ${postal.code["remote_addr"]}
|
|
125
|
+
</record>
|
|
126
|
+
skip_adding_null_record true
|
|
127
|
+
</filter>
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
Skip adding fields if incoming `remote_addr`'s GeoIP data is like following:
|
|
131
|
+
|
|
132
|
+
```ruby
|
|
133
|
+
# the record does not have "city" field
|
|
134
|
+
{"continent"=>
|
|
135
|
+
{"code"=>"NA",
|
|
136
|
+
"geoname_id"=>6255149,
|
|
137
|
+
"names"=>
|
|
138
|
+
{"de"=>"Nordamerika",
|
|
139
|
+
"en"=>"North America",
|
|
140
|
+
"es"=>"Norteamérica",
|
|
141
|
+
"fr"=>"Amérique du Nord",
|
|
142
|
+
"ja"=>"北アメリカ",
|
|
143
|
+
"pt-BR"=>"América do Norte",
|
|
144
|
+
"ru"=>"Северная Америка",
|
|
145
|
+
"zh-CN"=>"北美洲"}},
|
|
146
|
+
"country"=>
|
|
147
|
+
{"geoname_id"=>6252001,
|
|
148
|
+
"iso_code"=>"US",
|
|
149
|
+
"names"=>
|
|
150
|
+
{"de"=>"USA",
|
|
151
|
+
"en"=>"United States",
|
|
152
|
+
"es"=>"Estados Unidos",
|
|
153
|
+
"fr"=>"États-Unis",
|
|
154
|
+
"ja"=>"アメリカ合衆国",
|
|
155
|
+
"pt-BR"=>"Estados Unidos",
|
|
156
|
+
"ru"=>"США",
|
|
157
|
+
"zh-CN"=>"美国"}},
|
|
158
|
+
"location"=>
|
|
159
|
+
{"accuracy_radius"=>1000, "latitude"=>37.751, "longitude"=>-97.822},
|
|
160
|
+
"registered_country"=>
|
|
161
|
+
{"geoname_id"=>6252001,
|
|
162
|
+
"iso_code"=>"US",
|
|
163
|
+
"names"=>
|
|
164
|
+
{"de"=>"USA",
|
|
165
|
+
"en"=>"United States",
|
|
166
|
+
"es"=>"Estados Unidos",
|
|
167
|
+
"fr"=>"États-Unis",
|
|
168
|
+
"ja"=>"アメリカ合衆国",
|
|
169
|
+
"pt-BR"=>"Estados Unidos",
|
|
170
|
+
"ru"=>"США",
|
|
171
|
+
"zh-CN"=>"美国"}}}
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
We can avoid this behavior changing field order in `<record>` like following:
|
|
175
|
+
|
|
176
|
+
```
|
|
177
|
+
<filter access.apache>
|
|
178
|
+
@type geoip
|
|
179
|
+
geoip_lookup_keys remote_addr
|
|
180
|
+
<record>
|
|
181
|
+
latitude ${location.latitude["remote_addr"]} # this field must not be null
|
|
182
|
+
longitude ${location.longitude["remote_addr"]}
|
|
183
|
+
country ${country.iso_code["remote_addr"]}
|
|
184
|
+
country_name ${country.names.en["remote_addr"]}
|
|
185
|
+
postal_code ${postal.code["remote_addr"]}
|
|
186
|
+
city ${city.names.en["remote_addr"]} # adding fields even if this field is null
|
|
187
|
+
</record>
|
|
188
|
+
skip_adding_null_record true
|
|
189
|
+
</filter>
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
#### Tips: nested attributes for geoip_lookup_keys
|
|
193
|
+
|
|
194
|
+
See [Record Accessor Plugin](https://docs.fluentd.org/v1.0/articles/api-plugin-helper-record_accessor#syntax)
|
|
195
|
+
|
|
196
|
+
**NOTE** Since v1.3.0 does not interpret `host.ip` as nested attribute.
|
|
197
|
+
|
|
160
198
|
#### Advanced config samples
|
|
161
199
|
|
|
162
200
|
It is a sample to get friendly geo point recdords for elasticsearch with Yajl (JSON) parser.<br />
|
|
@@ -199,51 +237,6 @@ On the case of using td-agent3 (v1-config), it have to quote `{ ... }` or `[ ...
|
|
|
199
237
|
</filter>
|
|
200
238
|
```
|
|
201
239
|
|
|
202
|
-
### For GeoipOutput
|
|
203
|
-
|
|
204
|
-
```xml
|
|
205
|
-
<match access.apache>
|
|
206
|
-
@type geoip
|
|
207
|
-
|
|
208
|
-
# Specify one or more geoip lookup field which has ip address (default: host)
|
|
209
|
-
# in the case of accessing nested value, delimit keys by dot like 'host.ip'.
|
|
210
|
-
geoip_lookup_keys host
|
|
211
|
-
|
|
212
|
-
# Specify optional geoip database (using bundled GeoLiteCity databse by default)
|
|
213
|
-
geoip_database "/path/to/your/GeoIPCity.dat"
|
|
214
|
-
# Specify optional geoip2 database
|
|
215
|
-
# geoip2_database "/path/to/your/GeoLite2-City.mmdb"
|
|
216
|
-
# Specify backend library (geoip, geoip2_compat, geoip2_c)
|
|
217
|
-
backend_library geoip
|
|
218
|
-
|
|
219
|
-
# Set adding field with placeholder (more than one settings are required.)
|
|
220
|
-
<record>
|
|
221
|
-
latitude ${location.latitude["host"]}
|
|
222
|
-
longitude ${location.longitude["host"]}
|
|
223
|
-
country ${country.iso_code["host"]}
|
|
224
|
-
country_name ${country.names.en["host"]}
|
|
225
|
-
postal_code ${postal.code["host"]}
|
|
226
|
-
region ${subdivisions.0.iso_code["host"]}
|
|
227
|
-
region_name ${subdivisions.0.names.en["host"]}
|
|
228
|
-
city ${city.names.en["host"]}
|
|
229
|
-
</record>
|
|
230
|
-
|
|
231
|
-
# Settings for tag
|
|
232
|
-
tag geoip.${tag[1]}
|
|
233
|
-
|
|
234
|
-
# To avoid get stacktrace error with `[null, null]` array for elasticsearch.
|
|
235
|
-
skip_adding_null_record true
|
|
236
|
-
|
|
237
|
-
# Set @log_level (default: warn)
|
|
238
|
-
@log_level info
|
|
239
|
-
|
|
240
|
-
<buffer tag>
|
|
241
|
-
# Set buffering time (default: 0s)
|
|
242
|
-
flush_interval 1s
|
|
243
|
-
</buffer>
|
|
244
|
-
</match>
|
|
245
|
-
```
|
|
246
|
-
|
|
247
240
|
## Tutorial
|
|
248
241
|
|
|
249
242
|
### For GeoipFilter
|
|
@@ -255,7 +248,6 @@ On the case of using td-agent3 (v1-config), it have to quote `{ ... }` or `[ ...
|
|
|
255
248
|
@type forward
|
|
256
249
|
</source>
|
|
257
250
|
|
|
258
|
-
|
|
259
251
|
<filter test.geoip>
|
|
260
252
|
@type geoip
|
|
261
253
|
geoip_lookup_keys host
|
|
@@ -290,57 +282,6 @@ $ bundle exec ruby urils/dump.rb geoip2_compat 66.102.3.80
|
|
|
290
282
|
$ bundle exec ruby urils/dump.rb geoip 66.102.3.80
|
|
291
283
|
```
|
|
292
284
|
|
|
293
|
-
### For GeoipOutput
|
|
294
|
-
|
|
295
|
-
#### configuration
|
|
296
|
-
|
|
297
|
-
```xml
|
|
298
|
-
<source>
|
|
299
|
-
@type forward
|
|
300
|
-
</source>
|
|
301
|
-
|
|
302
|
-
<match test.geoip>
|
|
303
|
-
@type copy
|
|
304
|
-
<store>
|
|
305
|
-
@type stdout
|
|
306
|
-
</store>
|
|
307
|
-
<store>
|
|
308
|
-
@type geoip
|
|
309
|
-
geoip_lookup_keys host
|
|
310
|
-
<record>
|
|
311
|
-
lat ${location.latitude["host"]}
|
|
312
|
-
lon ${location.longitude["host"]}
|
|
313
|
-
country ${country.iso_code["host"]}
|
|
314
|
-
</record>
|
|
315
|
-
tag debug.${tag[1]}
|
|
316
|
-
</store>
|
|
317
|
-
</match>
|
|
318
|
-
|
|
319
|
-
<match debug.**>
|
|
320
|
-
@type stdout
|
|
321
|
-
</match>
|
|
322
|
-
```
|
|
323
|
-
|
|
324
|
-
#### result
|
|
325
|
-
|
|
326
|
-
```bash
|
|
327
|
-
# forward record with Google's ip address.
|
|
328
|
-
$ echo '{"host":"66.102.9.80","message":"test"}' | fluent-cat test.geoip
|
|
329
|
-
|
|
330
|
-
# check the result at stdout
|
|
331
|
-
$ tail /var/log/td-agent/td-agent.log
|
|
332
|
-
2013-08-04 16:21:32 +0900 test.geoip: {"host":"66.102.9.80","message":"test"}
|
|
333
|
-
2013-08-04 16:21:32 +0900 debug.geoip: {"host":"66.102.9.80","message":"test","lat":37.4192008972168,"lon":-122.05740356445312,"country":"US"}
|
|
334
|
-
```
|
|
335
|
-
|
|
336
|
-
You can check geoip data format using [utils/dump.rb](https://github.com/okkez/fluent-plugin-geoip/utils/dump.rb).
|
|
337
|
-
|
|
338
|
-
```
|
|
339
|
-
$ bundle exec ruby urils/dump.rb geoip2 66.102.3.80
|
|
340
|
-
$ bundle exec ruby urils/dump.rb geoip2_compat 66.102.3.80
|
|
341
|
-
$ bundle exec ruby urils/dump.rb geoip 66.102.3.80
|
|
342
|
-
```
|
|
343
|
-
|
|
344
285
|
## Placeholders
|
|
345
286
|
|
|
346
287
|
### GeoIP2
|
|
@@ -465,68 +406,6 @@ On the case of getting nothing of GeoIP info (such as local IP), it will output
|
|
|
465
406
|
|
|
466
407
|
Set backend library.
|
|
467
408
|
|
|
468
|
-
### GeoipOutput
|
|
469
|
-
|
|
470
|
-
#### Plugin helpers
|
|
471
|
-
|
|
472
|
-
* [event_emitter](https://docs.fluentd.org/v1.0/articles/api-plugin-helper-event_emitter)
|
|
473
|
-
* [compat_parameters](https://docs.fluentd.org/v1.0/articles/api-plugin-helper-compat_parameters)
|
|
474
|
-
* [inject](https://docs.fluentd.org/v1.0/articles/api-plugin-helper-inject)
|
|
475
|
-
|
|
476
|
-
See also [Output Plugin Overview](https://docs.fluentd.org/v1.0/articles/output-plugin-overview)
|
|
477
|
-
|
|
478
|
-
#### Sections
|
|
479
|
-
|
|
480
|
-
* [Inject section configurations](https://docs.fluentd.org/v1.0/articles/inject-section)
|
|
481
|
-
* [Buffer section configurations](https://docs.fluentd.org/v1.0/articles/buffer-section)
|
|
482
|
-
|
|
483
|
-
#### Parameters
|
|
484
|
-
|
|
485
|
-
[Plugin Common Paramteters](https://docs.fluentd.org/v1.0/articles/plugin-common-parameters)
|
|
486
|
-
|
|
487
|
-
**geoip_database** (string) (optional)
|
|
488
|
-
|
|
489
|
-
* Default value: bundled database `GeoLiteCity.dat`
|
|
490
|
-
|
|
491
|
-
Path to GeoIP database file.
|
|
492
|
-
|
|
493
|
-
**geoip2_database** (string) (optional)
|
|
494
|
-
|
|
495
|
-
* Default value: bundled database `GeoLite2-City.mmdb`.
|
|
496
|
-
|
|
497
|
-
Path to GeoIP2 database file.
|
|
498
|
-
|
|
499
|
-
**geoip_lookup_keys** (array) (optional)
|
|
500
|
-
|
|
501
|
-
* Default value: `["host"]`
|
|
502
|
-
|
|
503
|
-
Specify one or more geoip lookup field which has IP address.
|
|
504
|
-
|
|
505
|
-
**geoip_lookup_key** (string) (optional) (deprecated)
|
|
506
|
-
|
|
507
|
-
* Default value: `nil`.
|
|
508
|
-
|
|
509
|
-
Use geoip_lookup_keys instead.
|
|
510
|
-
|
|
511
|
-
**skip_adding_null_record** (bool) (optional)
|
|
512
|
-
|
|
513
|
-
* Default value: `nil`
|
|
514
|
-
|
|
515
|
-
Skip adding geoip fields when this valaues to `true`.
|
|
516
|
-
On the case of getting nothing of GeoIP info (such as local IP), it will output the original record without changing anything.
|
|
517
|
-
|
|
518
|
-
**backend_library** (enum) (optional)
|
|
519
|
-
|
|
520
|
-
* Available values: `geoip`, `geoip2_compat`, `geoip2_c`
|
|
521
|
-
* Default value: `geoip2_c`.
|
|
522
|
-
|
|
523
|
-
Set backend library.
|
|
524
|
-
|
|
525
|
-
**tag** (string) (optional)
|
|
526
|
-
|
|
527
|
-
On using this option with tag placeholder like `tag geoip.${tag}` (test code is available at [test_out_geoip.rb](https://github.com/y-ken/fluent-plugin-geoip/blob/master/test/plugin/test_out_geoip.rb)).
|
|
528
|
-
|
|
529
|
-
|
|
530
409
|
## Articles
|
|
531
410
|
|
|
532
411
|
* [IPアドレスを元に位置情報をリアルタイムに付与する fluent-plugin-geoip v0.0.1をリリースしました #fluentd - Y-Ken Studio](http://y-ken.hatenablog.com/entry/fluent-plugin-geoip-has-released)<br />
|
data/fluent-plugin-geoip.gemspec
CHANGED
|
@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
|
|
4
4
|
|
|
5
5
|
Gem::Specification.new do |spec|
|
|
6
6
|
spec.name = "fluent-plugin-geoip"
|
|
7
|
-
spec.version = "1.
|
|
7
|
+
spec.version = "1.3.0"
|
|
8
8
|
spec.authors = ["Kentaro Yoshida"]
|
|
9
9
|
spec.email = ["y.ken.studio@gmail.com"]
|
|
10
10
|
spec.summary = %q{Fluentd Filter plugin to add information about geographical location of IP addresses with Maxmind GeoIP databases.}
|
|
@@ -20,6 +20,7 @@ Gem::Specification.new do |spec|
|
|
|
20
20
|
spec.add_development_dependency "rake"
|
|
21
21
|
spec.add_development_dependency "appraisal"
|
|
22
22
|
spec.add_development_dependency "test-unit", ">= 3.1.0"
|
|
23
|
+
spec.add_development_dependency "test-unit-rr"
|
|
23
24
|
spec.add_development_dependency "geoip2_compat"
|
|
24
25
|
|
|
25
26
|
spec.add_runtime_dependency "fluentd", [">= 0.14.8", "< 2"]
|
|
@@ -1,11 +1,37 @@
|
|
|
1
1
|
require 'fluent/plugin/filter'
|
|
2
|
-
|
|
2
|
+
|
|
3
|
+
require 'geoip'
|
|
4
|
+
require 'yajl'
|
|
5
|
+
unless {}.respond_to?(:dig)
|
|
6
|
+
begin
|
|
7
|
+
# backport_dig is faster than dig_rb so prefer backport_dig.
|
|
8
|
+
# And Fluentd v1.0.1 uses backport_dig
|
|
9
|
+
require 'backport_dig'
|
|
10
|
+
rescue LoadError
|
|
11
|
+
require 'dig_rb'
|
|
12
|
+
end
|
|
13
|
+
end
|
|
3
14
|
|
|
4
15
|
module Fluent::Plugin
|
|
5
16
|
class GeoipFilter < Fluent::Plugin::Filter
|
|
6
17
|
Fluent::Plugin.register_filter('geoip', self)
|
|
7
18
|
|
|
8
|
-
|
|
19
|
+
BACKEND_LIBRARIES = [:geoip, :geoip2_compat, :geoip2_c]
|
|
20
|
+
|
|
21
|
+
REGEXP_PLACEHOLDER_SINGLE = /^\$\{
|
|
22
|
+
(?<geoip_key>-?[^\[\]]+)
|
|
23
|
+
\[
|
|
24
|
+
(?:(?<dq>")|(?<sq>'))
|
|
25
|
+
(?<record_key>-?(?(<dq>)[^"{}]+|[^'{}]+))
|
|
26
|
+
(?(<dq>)"|')
|
|
27
|
+
\]
|
|
28
|
+
\}$/x
|
|
29
|
+
REGEXP_PLACEHOLDER_SCAN = /['"]?(\$\{[^\}]+?\})['"]?/
|
|
30
|
+
|
|
31
|
+
GEOIP_KEYS = %w(city latitude longitude country_code3 country_code country_name dma_code area_code region)
|
|
32
|
+
GEOIP2_COMPAT_KEYS = %w(city country_code country_name latitude longitude postal_code region region_name)
|
|
33
|
+
|
|
34
|
+
helpers :compat_parameters, :inject, :record_accessor
|
|
9
35
|
|
|
10
36
|
config_param :geoip_database, :string, default: File.expand_path('../../../data/GeoLiteCity.dat', __dir__)
|
|
11
37
|
config_param :geoip2_database, :string, default: File.expand_path('../../../data/GeoLite2-City.mmdb', __dir__)
|
|
@@ -15,16 +41,69 @@ module Fluent::Plugin
|
|
|
15
41
|
|
|
16
42
|
config_set_default :@log_level, "warn"
|
|
17
43
|
|
|
18
|
-
config_param :backend_library, :enum, list:
|
|
44
|
+
config_param :backend_library, :enum, list: BACKEND_LIBRARIES, default: :geoip2_c
|
|
19
45
|
|
|
20
46
|
def configure(conf)
|
|
21
47
|
compat_parameters_convert(conf, :inject)
|
|
22
48
|
super
|
|
23
|
-
|
|
49
|
+
|
|
50
|
+
@map = {}
|
|
51
|
+
if @geoip_lookup_key
|
|
52
|
+
@geoip_lookup_keys = @geoip_lookup_key.split(/\s*,\s*/)
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
@geoip_lookup_keys.each do |key|
|
|
56
|
+
if key.include?(".") && !key.start_with?("$")
|
|
57
|
+
$log.warn("#{key} is not treated as nested attributes")
|
|
58
|
+
end
|
|
59
|
+
end
|
|
60
|
+
@geoip_lookup_accessors = @geoip_lookup_keys.map {|key| [key, record_accessor_create(key)] }.to_h
|
|
61
|
+
|
|
62
|
+
if conf.keys.any? {|k| k =~ /^enable_key_/ }
|
|
63
|
+
raise Fluent::ConfigError, "geoip: 'enable_key_*' config format is obsoleted. use <record></record> directive instead."
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
# <record></record> directive
|
|
67
|
+
conf.elements.select { |element| element.name == 'record' }.each { |element|
|
|
68
|
+
element.each_pair { |k, v|
|
|
69
|
+
element.has_key?(k) # to suppress unread configuration warning
|
|
70
|
+
v = v[1..v.size-2] if quoted_value?(v)
|
|
71
|
+
@map[k] = v
|
|
72
|
+
validate_json = Proc.new {
|
|
73
|
+
begin
|
|
74
|
+
dummy_text = Yajl::Encoder.encode('dummy_text')
|
|
75
|
+
Yajl::Parser.parse(v.gsub(REGEXP_PLACEHOLDER_SCAN, dummy_text))
|
|
76
|
+
rescue Yajl::ParseError => e
|
|
77
|
+
message = "geoip: failed to parse '#{v}' as json."
|
|
78
|
+
log.error message, error: e
|
|
79
|
+
raise Fluent::ConfigError, message
|
|
80
|
+
end
|
|
81
|
+
}
|
|
82
|
+
validate_json.call if json?(v.tr('\'"\\', ''))
|
|
83
|
+
}
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
@placeholder_keys = @map.values.join.scan(REGEXP_PLACEHOLDER_SCAN).map{|placeholder| placeholder[0] }.uniq
|
|
87
|
+
@placeholder_keys.each do |key|
|
|
88
|
+
m = key.match(REGEXP_PLACEHOLDER_SINGLE)
|
|
89
|
+
geoip_key = m[:geoip_key]
|
|
90
|
+
case @backend_library
|
|
91
|
+
when :geoip
|
|
92
|
+
raise Fluent::ConfigError, "#{@backend_library}: unsupported key #{geoip_key}" unless GEOIP_KEYS.include?(geoip_key)
|
|
93
|
+
when :geoip2_compat
|
|
94
|
+
raise Fluent::ConfigError, "#{@backend_library}: unsupported key #{geoip_key}" unless GEOIP2_COMPAT_KEYS.include?(geoip_key)
|
|
95
|
+
when :geoip2_c
|
|
96
|
+
# Nothing to do.
|
|
97
|
+
# We cannot define supported key(s) before we fetch values from GeoIP2 database
|
|
98
|
+
# because geoip2_c can fetch any fields in GeoIP2 database.
|
|
99
|
+
end
|
|
100
|
+
end
|
|
101
|
+
|
|
102
|
+
@geoip = load_database
|
|
24
103
|
end
|
|
25
104
|
|
|
26
105
|
def filter(tag, time, record)
|
|
27
|
-
filtered_record =
|
|
106
|
+
filtered_record = add_geoip_field(record)
|
|
28
107
|
if filtered_record
|
|
29
108
|
record = filtered_record
|
|
30
109
|
end
|
|
@@ -35,5 +114,101 @@ module Fluent::Plugin
|
|
|
35
114
|
def multi_workers_ready?
|
|
36
115
|
true
|
|
37
116
|
end
|
|
117
|
+
|
|
118
|
+
private
|
|
119
|
+
|
|
120
|
+
def add_geoip_field(record)
|
|
121
|
+
placeholder = create_placeholder(geolocate(get_address(record)))
|
|
122
|
+
return record if @skip_adding_null_record && placeholder.values.first.nil?
|
|
123
|
+
@map.each do |record_key, value|
|
|
124
|
+
if value.match(REGEXP_PLACEHOLDER_SINGLE) #|| value.match(REGEXP_PLACEHOLDER_BRACKET_SINGLE)
|
|
125
|
+
rewrited = placeholder[value]
|
|
126
|
+
elsif json?(value)
|
|
127
|
+
rewrited = value.gsub(REGEXP_PLACEHOLDER_SCAN) {|match|
|
|
128
|
+
match = match[1..match.size-2] if quoted_value?(match)
|
|
129
|
+
Yajl::Encoder.encode(placeholder[match])
|
|
130
|
+
}
|
|
131
|
+
rewrited = parse_json(rewrited)
|
|
132
|
+
else
|
|
133
|
+
rewrited = value.gsub(REGEXP_PLACEHOLDER_SCAN, placeholder)
|
|
134
|
+
end
|
|
135
|
+
record[record_key] = rewrited
|
|
136
|
+
end
|
|
137
|
+
record
|
|
138
|
+
end
|
|
139
|
+
|
|
140
|
+
def json?(text)
|
|
141
|
+
text.match(/^\[.+\]$/) || text.match(/^\{.+\}$/)
|
|
142
|
+
end
|
|
143
|
+
|
|
144
|
+
def quoted_value?(text)
|
|
145
|
+
# to improbe compatibility with fluentd v1-config
|
|
146
|
+
text.match(/(^'.+'$|^".+"$)/)
|
|
147
|
+
end
|
|
148
|
+
|
|
149
|
+
def parse_json(message)
|
|
150
|
+
begin
|
|
151
|
+
return Yajl::Parser.parse(message)
|
|
152
|
+
rescue Yajl::ParseError => e
|
|
153
|
+
log.info "geoip: failed to parse '#{message}' as json.", error_class: e.class, error: e.message
|
|
154
|
+
return nil
|
|
155
|
+
end
|
|
156
|
+
end
|
|
157
|
+
|
|
158
|
+
def get_address(record)
|
|
159
|
+
address = {}
|
|
160
|
+
@geoip_lookup_accessors.each do |field, accessor|
|
|
161
|
+
address[field] = accessor.call(record)
|
|
162
|
+
end
|
|
163
|
+
address
|
|
164
|
+
end
|
|
165
|
+
|
|
166
|
+
def geolocate(addresses)
|
|
167
|
+
geodata = {}
|
|
168
|
+
addresses.each do |field, ip|
|
|
169
|
+
geo = nil
|
|
170
|
+
if ip
|
|
171
|
+
geo = if @geoip.respond_to?(:look_up)
|
|
172
|
+
@geoip.look_up(ip)
|
|
173
|
+
else
|
|
174
|
+
@geoip.lookup(ip)
|
|
175
|
+
end
|
|
176
|
+
end
|
|
177
|
+
geodata[field] = geo
|
|
178
|
+
end
|
|
179
|
+
geodata
|
|
180
|
+
end
|
|
181
|
+
|
|
182
|
+
def create_placeholder(geodata)
|
|
183
|
+
placeholder = {}
|
|
184
|
+
@placeholder_keys.each do |placeholder_key|
|
|
185
|
+
position = placeholder_key.match(REGEXP_PLACEHOLDER_SINGLE)
|
|
186
|
+
next if position.nil? or geodata[position[:record_key]].nil?
|
|
187
|
+
keys = [position[:record_key]] + position[:geoip_key].split('.').map(&:to_sym)
|
|
188
|
+
value = geodata.dig(*keys)
|
|
189
|
+
value = if [:latitude, :longitude].include?(keys.last)
|
|
190
|
+
value || 0.0
|
|
191
|
+
else
|
|
192
|
+
value
|
|
193
|
+
end
|
|
194
|
+
placeholder[placeholder_key] = value
|
|
195
|
+
end
|
|
196
|
+
placeholder
|
|
197
|
+
end
|
|
198
|
+
|
|
199
|
+
def load_database
|
|
200
|
+
case @backend_library
|
|
201
|
+
when :geoip
|
|
202
|
+
::GeoIP::City.new(@geoip_database, :memory, false)
|
|
203
|
+
when :geoip2_compat
|
|
204
|
+
require 'geoip2_compat'
|
|
205
|
+
GeoIP2Compat.new(@geoip2_database)
|
|
206
|
+
when :geoip2_c
|
|
207
|
+
require 'geoip2'
|
|
208
|
+
GeoIP2::Database.new(@geoip2_database)
|
|
209
|
+
end
|
|
210
|
+
rescue LoadError
|
|
211
|
+
raise Fluent::ConfigError, "You must install #{@backend_library} gem."
|
|
212
|
+
end
|
|
38
213
|
end
|
|
39
214
|
end
|