logstash-filter-useragent 3.1.0-java
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/CHANGELOG.md +36 -0
- data/CONTRIBUTORS +18 -0
- data/Gemfile +4 -0
- data/LICENSE +13 -0
- data/NOTICE.TXT +5 -0
- data/README.md +99 -0
- data/docs/index.asciidoc +117 -0
- data/lib/logstash-filter-useragent_jars.rb +4 -0
- data/lib/logstash/filters/useragent.rb +149 -0
- data/logstash-filter-useragent.gemspec +27 -0
- data/spec/filters/useragent_spec.rb +129 -0
- data/vendor/jar-dependencies/org/logstash/filters/logstash-filter-useragent/3.1.0/logstash-filter-useragent-3.1.0.jar +0 -0
- metadata +93 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 2882cc7695727f80d8010a52e600c24e88c44ad4
|
4
|
+
data.tar.gz: 8a174676a2edf46ce48adfc9540c28403aa2c918
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 0d2bdcc43b9fb5433a6b41c94544c3f4fe8979e79772e2973c5225d7fcf1edda14692d3cec151ea7db4647ec6c59c226f4b0c09a3ec7dde9af3a3dd7d2dfa1e1
|
7
|
+
data.tar.gz: 340fb536b59287c08e69365a4c759d76666c2c254d837df867305861d31f8c6626b7858252de844bbd236a4cb5389aa859b01f0c660fa587bd3d347c654ef7d2
|
data/CHANGELOG.md
ADDED
@@ -0,0 +1,36 @@
|
|
1
|
+
## 3.1.0
|
2
|
+
- Parser performance increase of a factor of about 2.5 by moving core parser logic from Ruby to Java
|
3
|
+
- Lower memory footprint of Java implementation allowed for increasing the default parser cache size from 1k to 100k
|
4
|
+
without an increase in memory consumption
|
5
|
+
|
6
|
+
## 3.0.3
|
7
|
+
- Move one log message from info to debug to avoid noise
|
8
|
+
|
9
|
+
## 3.0.2
|
10
|
+
- Relax constraint on logstash-core-plugin-api to >= 1.60 <= 2.99
|
11
|
+
|
12
|
+
## 3.0.1
|
13
|
+
- Republish all the gems under jruby.
|
14
|
+
## 3.0.0
|
15
|
+
- Update the plugin to the version 2.0 of the plugin api, this change is required for Logstash 5.0 compatibility. See https://github.com/elastic/logstash/issues/5141
|
16
|
+
# 2.0.8
|
17
|
+
- Revert addition of Mutex. This plugin now depends on jruby having threadsafe regexps
|
18
|
+
# 2.0.7
|
19
|
+
- Add Mutex to help on non-threadsafe JRuby versions
|
20
|
+
# 2.0.6
|
21
|
+
- Depend on logstash-core-plugin-api instead of logstash-core, removing the need to mass update plugins on major releases of logstash
|
22
|
+
# 2.0.5
|
23
|
+
- New dependency requirements for logstash-core for the 5.0 release
|
24
|
+
## 2.0.4
|
25
|
+
- Refactored field references, fixed specs and some cleanups
|
26
|
+
|
27
|
+
## 2.0.0
|
28
|
+
- Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully,
|
29
|
+
instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
|
30
|
+
- Dependency on logstash-core update to 2.0
|
31
|
+
|
32
|
+
## 2.0.1
|
33
|
+
- Add ability to replace source with target
|
34
|
+
|
35
|
+
## 1.1.0
|
36
|
+
- Add LRU cache
|
data/CONTRIBUTORS
ADDED
@@ -0,0 +1,18 @@
|
|
1
|
+
The following is a list of people who have contributed ideas, code, bug
|
2
|
+
reports, or in general have helped logstash along its way.
|
3
|
+
|
4
|
+
Contributors:
|
5
|
+
* Alex Tambellini (atambo)
|
6
|
+
* Colin Surprenant (colinsurprenant)
|
7
|
+
* Dan Everton (deverton)
|
8
|
+
* Jay Luker (lbjay)
|
9
|
+
* Jordan Sissel (jordansissel)
|
10
|
+
* Kubes (pkubat)
|
11
|
+
* Pier-Hugues Pellerin (ph)
|
12
|
+
* Richard Pijnenburg (electrical)
|
13
|
+
* Suyog Rao (suyograo)
|
14
|
+
|
15
|
+
Note: If you've sent us patches, bug reports, or otherwise contributed to
|
16
|
+
Logstash, and you aren't on the list above and want to be, please let us know
|
17
|
+
and we'll make sure you're here. Contributions from folks like you are what make
|
18
|
+
open source awesome.
|
data/Gemfile
ADDED
data/LICENSE
ADDED
@@ -0,0 +1,13 @@
|
|
1
|
+
Copyright (c) 2012–2016 Elasticsearch <http://www.elastic.co>
|
2
|
+
|
3
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
4
|
+
you may not use this file except in compliance with the License.
|
5
|
+
You may obtain a copy of the License at
|
6
|
+
|
7
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
8
|
+
|
9
|
+
Unless required by applicable law or agreed to in writing, software
|
10
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
11
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
12
|
+
See the License for the specific language governing permissions and
|
13
|
+
limitations under the License.
|
data/NOTICE.TXT
ADDED
data/README.md
ADDED
@@ -0,0 +1,99 @@
|
|
1
|
+
# Logstash Plugin
|
2
|
+
|
3
|
+
[![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-filter-useragent.svg)](https://travis-ci.org/logstash-plugins/logstash-filter-useragent)
|
4
|
+
|
5
|
+
This is a plugin for [Logstash](https://github.com/elastic/logstash).
|
6
|
+
|
7
|
+
It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
|
8
|
+
|
9
|
+
## Documentation
|
10
|
+
|
11
|
+
Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
|
12
|
+
|
13
|
+
- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
|
14
|
+
- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
|
15
|
+
|
16
|
+
## Need Help?
|
17
|
+
|
18
|
+
Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
|
19
|
+
|
20
|
+
## Developing
|
21
|
+
|
22
|
+
### 1. Plugin Developement and Testing
|
23
|
+
|
24
|
+
#### Code
|
25
|
+
- To get started, you'll need JRuby with the Bundler gem installed.
|
26
|
+
|
27
|
+
- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
|
28
|
+
|
29
|
+
- Install dependencies
|
30
|
+
```sh
|
31
|
+
bundle install
|
32
|
+
rake install_jars
|
33
|
+
```
|
34
|
+
|
35
|
+
#### Test
|
36
|
+
|
37
|
+
- Update your dependencies
|
38
|
+
|
39
|
+
```sh
|
40
|
+
bundle install
|
41
|
+
```
|
42
|
+
|
43
|
+
- Run tests
|
44
|
+
|
45
|
+
```sh
|
46
|
+
bundle exec rake test
|
47
|
+
```
|
48
|
+
|
49
|
+
### 2. Running your unpublished Plugin in Logstash
|
50
|
+
|
51
|
+
#### 2.1 Run in a local Logstash clone
|
52
|
+
|
53
|
+
- Edit Logstash `Gemfile` and add the local plugin path, for example:
|
54
|
+
```ruby
|
55
|
+
gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
|
56
|
+
```
|
57
|
+
- Install plugin
|
58
|
+
```sh
|
59
|
+
# Logstash 2.3 and higher
|
60
|
+
bin/logstash-plugin install --no-verify
|
61
|
+
|
62
|
+
# Prior to Logstash 2.3
|
63
|
+
bin/plugin install --no-verify
|
64
|
+
|
65
|
+
```
|
66
|
+
- Run Logstash with your plugin
|
67
|
+
```sh
|
68
|
+
bin/logstash -e 'filter {awesome {}}'
|
69
|
+
```
|
70
|
+
At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
|
71
|
+
|
72
|
+
#### 2.2 Run in an installed Logstash
|
73
|
+
|
74
|
+
You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
|
75
|
+
|
76
|
+
- Build your plugin gem
|
77
|
+
```sh
|
78
|
+
gem build logstash-filter-awesome.gemspec
|
79
|
+
```
|
80
|
+
- Install the plugin from the Logstash home
|
81
|
+
```sh
|
82
|
+
# Logstash 2.3 and higher
|
83
|
+
bin/logstash-plugin install --no-verify
|
84
|
+
|
85
|
+
# Prior to Logstash 2.3
|
86
|
+
bin/plugin install --no-verify
|
87
|
+
|
88
|
+
```
|
89
|
+
- Start Logstash and proceed to test the plugin
|
90
|
+
|
91
|
+
## Contributing
|
92
|
+
|
93
|
+
All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
|
94
|
+
|
95
|
+
Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.
|
96
|
+
|
97
|
+
It is more important to the community that you are able to contribute.
|
98
|
+
|
99
|
+
For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.
|
data/docs/index.asciidoc
ADDED
@@ -0,0 +1,117 @@
|
|
1
|
+
:plugin: useragent
|
2
|
+
:type: filter
|
3
|
+
|
4
|
+
///////////////////////////////////////////
|
5
|
+
START - GENERATED VARIABLES, DO NOT EDIT!
|
6
|
+
///////////////////////////////////////////
|
7
|
+
:version: %VERSION%
|
8
|
+
:release_date: %RELEASE_DATE%
|
9
|
+
:changelog_url: %CHANGELOG_URL%
|
10
|
+
:include_path: ../../../logstash/docs/include
|
11
|
+
///////////////////////////////////////////
|
12
|
+
END - GENERATED VARIABLES, DO NOT EDIT!
|
13
|
+
///////////////////////////////////////////
|
14
|
+
|
15
|
+
[id="plugins-{type}-{plugin}"]
|
16
|
+
|
17
|
+
=== Useragent
|
18
|
+
|
19
|
+
include::{include_path}/plugin_header.asciidoc[]
|
20
|
+
|
21
|
+
==== Description
|
22
|
+
|
23
|
+
Parse user agent strings into structured data based on BrowserScope data
|
24
|
+
|
25
|
+
UserAgent filter, adds information about user agent like family, operating
|
26
|
+
system, version, and device
|
27
|
+
|
28
|
+
Logstash releases ship with the regexes.yaml database made available from
|
29
|
+
ua-parser with an Apache 2.0 license. For more details on ua-parser, see
|
30
|
+
<https://github.com/tobie/ua-parser/>.
|
31
|
+
|
32
|
+
[id="plugins-{type}s-{plugin}-options"]
|
33
|
+
==== Useragent Filter Configuration Options
|
34
|
+
|
35
|
+
This plugin supports the following configuration options plus the <<plugins-{type}s-common-options>> described later.
|
36
|
+
|
37
|
+
[cols="<,<,<",options="header",]
|
38
|
+
|=======================================================================
|
39
|
+
|Setting |Input type|Required
|
40
|
+
| <<plugins-{type}s-{plugin}-lru_cache_size>> |<<number,number>>|No
|
41
|
+
| <<plugins-{type}s-{plugin}-prefix>> |<<string,string>>|No
|
42
|
+
| <<plugins-{type}s-{plugin}-regexes>> |<<string,string>>|No
|
43
|
+
| <<plugins-{type}s-{plugin}-source>> |<<string,string>>|Yes
|
44
|
+
| <<plugins-{type}s-{plugin}-target>> |<<string,string>>|No
|
45
|
+
|=======================================================================
|
46
|
+
|
47
|
+
Also see <<plugins-{type}s-common-options>> for a list of options supported by all
|
48
|
+
filter plugins.
|
49
|
+
|
50
|
+
|
51
|
+
|
52
|
+
[id="plugins-{type}s-{plugin}-lru_cache_size"]
|
53
|
+
===== `lru_cache_size`
|
54
|
+
|
55
|
+
* Value type is <<number,number>>
|
56
|
+
* Default value is `1000`
|
57
|
+
|
58
|
+
UA parsing is surprisingly expensive. This filter uses an LRU cache to take advantage of the fact that
|
59
|
+
user agents are often found adjacent to one another in log files and rarely have a random distribution.
|
60
|
+
The higher you set this the more likely an item is to be in the cache and the faster this filter will run.
|
61
|
+
However, if you set this too high you can use more memory than desired.
|
62
|
+
|
63
|
+
Experiment with different values for this option to find the best performance for your dataset.
|
64
|
+
|
65
|
+
This MUST be set to a value > 0. There is really no reason to not want this behavior, the overhead is minimal
|
66
|
+
and the speed gains are large.
|
67
|
+
|
68
|
+
It is important to note that this config value is global. That is to say all instances of the user agent filter
|
69
|
+
share the same cache. The last declared cache size will 'win'. The reason for this is that there would be no benefit
|
70
|
+
to having multiple caches for different instances at different points in the pipeline, that would just increase the
|
71
|
+
number of cache misses and waste memory.
|
72
|
+
|
73
|
+
[id="plugins-{type}s-{plugin}-prefix"]
|
74
|
+
===== `prefix`
|
75
|
+
|
76
|
+
* Value type is <<string,string>>
|
77
|
+
* Default value is `""`
|
78
|
+
|
79
|
+
A string to prepend to all of the extracted keys
|
80
|
+
|
81
|
+
[id="plugins-{type}s-{plugin}-regexes"]
|
82
|
+
===== `regexes`
|
83
|
+
|
84
|
+
* Value type is <<string,string>>
|
85
|
+
* There is no default value for this setting.
|
86
|
+
|
87
|
+
`regexes.yaml` file to use
|
88
|
+
|
89
|
+
If not specified, this will default to the `regexes.yaml` that ships
|
90
|
+
with logstash.
|
91
|
+
|
92
|
+
You can find the latest version of this here:
|
93
|
+
<https://github.com/ua-parser/uap-core/blob/master/regexes.yaml>
|
94
|
+
|
95
|
+
[id="plugins-{type}s-{plugin}-source"]
|
96
|
+
===== `source`
|
97
|
+
|
98
|
+
* This is a required setting.
|
99
|
+
* Value type is <<string,string>>
|
100
|
+
* There is no default value for this setting.
|
101
|
+
|
102
|
+
The field containing the user agent string. If this field is an
|
103
|
+
array, only the first value will be used.
|
104
|
+
|
105
|
+
[id="plugins-{type}s-{plugin}-target"]
|
106
|
+
===== `target`
|
107
|
+
|
108
|
+
* Value type is <<string,string>>
|
109
|
+
* There is no default value for this setting.
|
110
|
+
|
111
|
+
The name of the field to assign user agent data into.
|
112
|
+
|
113
|
+
If not specified user agent data will be stored in the root of the event.
|
114
|
+
|
115
|
+
|
116
|
+
|
117
|
+
include::{include_path}/{type}.asciidoc[]
|
@@ -0,0 +1,149 @@
|
|
1
|
+
# encoding: utf-8
|
2
|
+
require "java"
|
3
|
+
require "logstash-filter-useragent_jars"
|
4
|
+
require "logstash/filters/base"
|
5
|
+
require "logstash/namespace"
|
6
|
+
require "tempfile"
|
7
|
+
require "thread"
|
8
|
+
|
9
|
+
# Parse user agent strings into structured data based on BrowserScope data
|
10
|
+
#
|
11
|
+
# UserAgent filter, adds information about user agent like family, operating
|
12
|
+
# system, version, and device
|
13
|
+
#
|
14
|
+
# Logstash releases ship with the regexes.yaml database made available from
|
15
|
+
# ua-parser with an Apache 2.0 license. For more details on ua-parser, see
|
16
|
+
# <https://github.com/tobie/ua-parser/>.
|
17
|
+
class LogStash::Filters::UserAgent < LogStash::Filters::Base
|
18
|
+
|
19
|
+
config_name "useragent"
|
20
|
+
|
21
|
+
# The field containing the user agent string. If this field is an
|
22
|
+
# array, only the first value will be used.
|
23
|
+
config :source, :validate => :string, :required => true
|
24
|
+
|
25
|
+
# The name of the field to assign user agent data into.
|
26
|
+
#
|
27
|
+
# If not specified user agent data will be stored in the root of the event.
|
28
|
+
config :target, :validate => :string
|
29
|
+
|
30
|
+
# `regexes.yaml` file to use
|
31
|
+
#
|
32
|
+
# If not specified, this will default to the `regexes.yaml` that ships
|
33
|
+
# with logstash.
|
34
|
+
#
|
35
|
+
# You can find the latest version of this here:
|
36
|
+
# <https://github.com/ua-parser/uap-core/blob/master/regexes.yaml>
|
37
|
+
config :regexes, :validate => :string
|
38
|
+
|
39
|
+
# A string to prepend to all of the extracted keys
|
40
|
+
config :prefix, :validate => :string, :default => ''
|
41
|
+
|
42
|
+
# UA parsing is surprisingly expensive. This filter uses an LRU cache to take advantage of the fact that
|
43
|
+
# user agents are often found adjacent to one another in log files and rarely have a random distribution.
|
44
|
+
# The higher you set this the more likely an item is to be in the cache and the faster this filter will run.
|
45
|
+
# However, if you set this too high you can use more memory than desired.
|
46
|
+
#
|
47
|
+
# Experiment with different values for this option to find the best performance for your dataset.
|
48
|
+
#
|
49
|
+
# This MUST be set to a value > 0. There is really no reason to not want this behavior, the overhead is minimal
|
50
|
+
# and the speed gains are large.
|
51
|
+
#
|
52
|
+
# It is important to note that this config value is global. That is to say all instances of the user agent filter
|
53
|
+
# share the same cache. The last declared cache size will 'win'. The reason for this is that there would be no benefit
|
54
|
+
# to having multiple caches for different instances at different points in the pipeline, that would just increase the
|
55
|
+
# number of cache misses and waste memory.
|
56
|
+
config :lru_cache_size, :validate => :number, :default => 100_000
|
57
|
+
|
58
|
+
def register
|
59
|
+
|
60
|
+
if @regexes.nil?
|
61
|
+
@parser = org.logstash.uaparser.CachingParser.new(lru_cache_size)
|
62
|
+
else
|
63
|
+
@logger.debug("Using user agent regexes", :regexes => @regexes)
|
64
|
+
yaml = ::File.open(regexes, "rb")
|
65
|
+
@parser = org.logstash.uaparser.CachingParser.new(yaml.read, lru_cache_size)
|
66
|
+
yaml.close
|
67
|
+
end
|
68
|
+
|
69
|
+
# make @target in the format [field name] if defined, i.e. surrounded by brakets
|
70
|
+
normalized_target = (@target && @target !~ /^\[[^\[\]]+\]$/) ? "[#{@target}]" : ""
|
71
|
+
|
72
|
+
# predefine prefixed field names
|
73
|
+
@prefixed_name = "#{normalized_target}[#{@prefix}name]"
|
74
|
+
@prefixed_os = "#{normalized_target}[#{@prefix}os]"
|
75
|
+
@prefixed_os_name = "#{normalized_target}[#{@prefix}os_name]"
|
76
|
+
@prefixed_os_major = "#{normalized_target}[#{@prefix}os_major]"
|
77
|
+
@prefixed_os_minor = "#{normalized_target}[#{@prefix}os_minor]"
|
78
|
+
@prefixed_device = "#{normalized_target}[#{@prefix}device]"
|
79
|
+
@prefixed_major = "#{normalized_target}[#{@prefix}major]"
|
80
|
+
@prefixed_minor = "#{normalized_target}[#{@prefix}minor]"
|
81
|
+
@prefixed_patch = "#{normalized_target}[#{@prefix}patch]"
|
82
|
+
@prefixed_build = "#{normalized_target}[#{@prefix}build]"
|
83
|
+
end
|
84
|
+
|
85
|
+
def filter(event)
|
86
|
+
useragent = event.get(@source)
|
87
|
+
useragent = useragent.first if useragent.is_a?(Array)
|
88
|
+
|
89
|
+
return if useragent.nil? || useragent.empty?
|
90
|
+
|
91
|
+
begin
|
92
|
+
ua_data = lookup_useragent(useragent)
|
93
|
+
rescue StandardError => e
|
94
|
+
@logger.error("Uknown error while parsing user agent data", :exception => e, :field => @source, :event => event)
|
95
|
+
return
|
96
|
+
end
|
97
|
+
|
98
|
+
return unless ua_data
|
99
|
+
|
100
|
+
event.remove(@source) if @target == @source
|
101
|
+
set_fields(event, ua_data)
|
102
|
+
|
103
|
+
filter_matched(event)
|
104
|
+
end
|
105
|
+
|
106
|
+
# should be private but need to stay public for specs
|
107
|
+
# TODO: (colin) the related specs should be refactored to not rely on private methods.
|
108
|
+
def lookup_useragent(useragent)
|
109
|
+
return unless useragent
|
110
|
+
|
111
|
+
# the UserAgentParser::Parser class is not thread safe, indications are that it is probably
|
112
|
+
# caused by the underlying JRuby regex code that is not thread safe.
|
113
|
+
# see https://github.com/logstash-plugins/logstash-filter-useragent/issues/25
|
114
|
+
@parser.parse(useragent)
|
115
|
+
end
|
116
|
+
|
117
|
+
private
|
118
|
+
|
119
|
+
def set_fields(event, ua_data)
|
120
|
+
# UserAgentParser outputs as US-ASCII.
|
121
|
+
|
122
|
+
event.set(@prefixed_name, ua_data.userAgent.family.dup.force_encoding(Encoding::UTF_8))
|
123
|
+
|
124
|
+
#OSX, Android and maybe iOS parse correctly, ua-agent parsing for Windows does not provide this level of detail
|
125
|
+
|
126
|
+
# Calls in here use #dup because there's potential for later filters to modify these values
|
127
|
+
# and corrupt the cache. See uap source here for details https://github.com/ua-parser/uap-ruby/tree/master/lib/user_agent_parser
|
128
|
+
if (os = ua_data.os)
|
129
|
+
# The OS is a rich object
|
130
|
+
event.set(@prefixed_os, ua_data.os.family.dup.force_encoding(Encoding::UTF_8))
|
131
|
+
event.set(@prefixed_os_name, os.family.dup.force_encoding(Encoding::UTF_8)) if os.family
|
132
|
+
|
133
|
+
# These are all strings
|
134
|
+
if os.minor && os.major
|
135
|
+
event.set(@prefixed_os_major, os.major.dup.force_encoding(Encoding::UTF_8)) if os.major
|
136
|
+
event.set(@prefixed_os_minor, os.minor.dup.force_encoding(Encoding::UTF_8)) if os.minor
|
137
|
+
end
|
138
|
+
end
|
139
|
+
|
140
|
+
event.set(@prefixed_device, ua_data.device.to_s.dup.force_encoding(Encoding::UTF_8)) if ua_data.device
|
141
|
+
|
142
|
+
if (ua_version = ua_data.userAgent)
|
143
|
+
event.set(@prefixed_major, ua_version.major.dup.force_encoding(Encoding::UTF_8)) if ua_version.major
|
144
|
+
event.set(@prefixed_minor, ua_version.minor.dup.force_encoding(Encoding::UTF_8)) if ua_version.minor
|
145
|
+
event.set(@prefixed_patch, ua_version.patch.dup.force_encoding(Encoding::UTF_8)) if ua_version.patch
|
146
|
+
event.set(@prefixed_build, ua_version.patchMinor.dup.force_encoding(Encoding::UTF_8)) if ua_version.patchMinor
|
147
|
+
end
|
148
|
+
end
|
149
|
+
end
|
@@ -0,0 +1,27 @@
|
|
1
|
+
Gem::Specification.new do |s|
|
2
|
+
|
3
|
+
s.name = 'logstash-filter-useragent'
|
4
|
+
s.version = '3.1.0'
|
5
|
+
s.licenses = ['Apache License (2.0)']
|
6
|
+
s.summary = "Parse user agent strings into structured data based on BrowserScope data"
|
7
|
+
s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
|
8
|
+
s.authors = ["Elastic"]
|
9
|
+
s.email = 'info@elastic.co'
|
10
|
+
s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html"
|
11
|
+
s.platform = "java"
|
12
|
+
s.require_paths = ["lib", "vendor/jar-dependencies"]
|
13
|
+
|
14
|
+
# Files
|
15
|
+
s.files = Dir["lib/**/*","spec/**/*","*.gemspec","*.md","CONTRIBUTORS","Gemfile","LICENSE","NOTICE.TXT","vendor/jar-dependencies/**/*.jar","vendor/jar-dependencies/**/*.rb", "VERSION", "docs/**/*"]
|
16
|
+
|
17
|
+
# Tests
|
18
|
+
s.test_files = s.files.grep(%r{^(test|spec|features)/})
|
19
|
+
|
20
|
+
# Special flag to let us know this is actually a logstash plugin
|
21
|
+
s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" }
|
22
|
+
|
23
|
+
# Gem dependencies
|
24
|
+
s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
|
25
|
+
s.add_development_dependency 'logstash-devutils'
|
26
|
+
end
|
27
|
+
|
@@ -0,0 +1,129 @@
|
|
1
|
+
# encoding: utf-8
|
2
|
+
|
3
|
+
require "logstash/devutils/rspec/spec_helper"
|
4
|
+
require "logstash/filters/useragent"
|
5
|
+
|
6
|
+
describe LogStash::Filters::UserAgent do
|
7
|
+
|
8
|
+
describe "defaults" do
|
9
|
+
config <<-CONFIG
|
10
|
+
filter {
|
11
|
+
useragent {
|
12
|
+
source => "message"
|
13
|
+
target => "ua"
|
14
|
+
}
|
15
|
+
}
|
16
|
+
CONFIG
|
17
|
+
|
18
|
+
sample "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.63 Safari/537.31" do
|
19
|
+
insist { subject }.include?("ua")
|
20
|
+
insist { subject.get("[ua][name]") } == "Chrome"
|
21
|
+
insist { subject.get("[ua][os]") } == "Linux"
|
22
|
+
insist { subject.get("[ua][major]") } == "26"
|
23
|
+
insist { subject.get("[ua][minor]") } == "0"
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
describe "Without target field" do
|
28
|
+
config <<-CONFIG
|
29
|
+
filter {
|
30
|
+
useragent {
|
31
|
+
source => "message"
|
32
|
+
}
|
33
|
+
}
|
34
|
+
CONFIG
|
35
|
+
|
36
|
+
sample "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.63 Safari/537.31" do
|
37
|
+
insist { subject.get("name") } == "Chrome"
|
38
|
+
insist { subject.get("os") } == "Linux"
|
39
|
+
insist { subject.get("major") } == "26"
|
40
|
+
insist { subject.get("minor") } == "0"
|
41
|
+
end
|
42
|
+
end
|
43
|
+
|
44
|
+
describe "Without user agent" do
|
45
|
+
config <<-CONFIG
|
46
|
+
filter {
|
47
|
+
useragent {
|
48
|
+
source => "message"
|
49
|
+
target => "ua"
|
50
|
+
}
|
51
|
+
}
|
52
|
+
CONFIG
|
53
|
+
|
54
|
+
sample "foo" => "bar" do
|
55
|
+
reject { subject }.include?("ua")
|
56
|
+
end
|
57
|
+
|
58
|
+
sample "" do
|
59
|
+
reject { subject }.include?("ua")
|
60
|
+
end
|
61
|
+
end
|
62
|
+
|
63
|
+
describe "LRU object identity" do
|
64
|
+
let(:ua_string) { "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36" }
|
65
|
+
let(:uafilter) { LogStash::Filters::UserAgent.new("source" => "foo") }
|
66
|
+
let(:ua_data) { uafilter.lookup_useragent(ua_string) }
|
67
|
+
|
68
|
+
subject(:target) { LogStash::Event.new("foo" => ua_string) }
|
69
|
+
|
70
|
+
before do
|
71
|
+
uafilter.register
|
72
|
+
|
73
|
+
# Stub this out because this UA doesn't have this field
|
74
|
+
allow(ua_data.userAgent).to receive(:patchMinor).and_return("foo")
|
75
|
+
|
76
|
+
# expect(event).receive(:lookup_useragent)
|
77
|
+
uafilter.filter(target)
|
78
|
+
end
|
79
|
+
|
80
|
+
{
|
81
|
+
"name" => lambda {|uad| uad.userAgent.family},
|
82
|
+
"os" => lambda {|uad| uad.os.family},
|
83
|
+
"os_name" => lambda {|uad| uad.os.family},
|
84
|
+
"os_major" => lambda {|uad| uad.os.major},
|
85
|
+
"os_minor" => lambda {|uad| uad.os.minor},
|
86
|
+
"device" => lambda {|uad| uad.device.to_s},
|
87
|
+
"major" => lambda {|uad| uad.userAgent.major},
|
88
|
+
"minor" => lambda {|uad| uad.userAgent.minor},
|
89
|
+
"patch" => lambda {|uad| uad.userAgent.patch},
|
90
|
+
"build" => lambda {|uad| uad.userAgent.patchMinor}
|
91
|
+
}.each do |field, uad_getter|
|
92
|
+
context "for the #{field} field" do
|
93
|
+
let(:value) {uad_getter.call(ua_data)}
|
94
|
+
let(:target_field) { target.get(field)}
|
95
|
+
|
96
|
+
it "should not have a nil value" do
|
97
|
+
expect(target_field).to be_truthy
|
98
|
+
end
|
99
|
+
|
100
|
+
it "should have equivalent values" do
|
101
|
+
expect(target_field).to eql(value)
|
102
|
+
end
|
103
|
+
|
104
|
+
it "should dup/clone the field to prevent cache corruption" do
|
105
|
+
expect(target_field.object_id).not_to eql(value.object_id)
|
106
|
+
end
|
107
|
+
end
|
108
|
+
end
|
109
|
+
end
|
110
|
+
|
111
|
+
describe "Replace source with target" do
|
112
|
+
config <<-CONFIG
|
113
|
+
filter {
|
114
|
+
useragent {
|
115
|
+
source => "message"
|
116
|
+
target => "message"
|
117
|
+
}
|
118
|
+
}
|
119
|
+
CONFIG
|
120
|
+
|
121
|
+
sample "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.63 Safari/537.31" do
|
122
|
+
insist { subject.to_hash }.include?("message")
|
123
|
+
insist { subject.get("[message][name]") } == "Chrome"
|
124
|
+
insist { subject.get("[message][os]") } == "Linux"
|
125
|
+
insist { subject.get("[message][major]") } == "26"
|
126
|
+
insist { subject.get("[message][minor]") } == "0"
|
127
|
+
end
|
128
|
+
end
|
129
|
+
end
|
Binary file
|
metadata
ADDED
@@ -0,0 +1,93 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: logstash-filter-useragent
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 3.1.0
|
5
|
+
platform: java
|
6
|
+
authors:
|
7
|
+
- Elastic
|
8
|
+
autorequire:
|
9
|
+
bindir: bin
|
10
|
+
cert_chain: []
|
11
|
+
date: 2017-05-10 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
requirement: !ruby/object:Gem::Requirement
|
15
|
+
requirements:
|
16
|
+
- - ">="
|
17
|
+
- !ruby/object:Gem::Version
|
18
|
+
version: '1.60'
|
19
|
+
- - "<="
|
20
|
+
- !ruby/object:Gem::Version
|
21
|
+
version: '2.99'
|
22
|
+
name: logstash-core-plugin-api
|
23
|
+
prerelease: false
|
24
|
+
type: :runtime
|
25
|
+
version_requirements: !ruby/object:Gem::Requirement
|
26
|
+
requirements:
|
27
|
+
- - ">="
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: '1.60'
|
30
|
+
- - "<="
|
31
|
+
- !ruby/object:Gem::Version
|
32
|
+
version: '2.99'
|
33
|
+
- !ruby/object:Gem::Dependency
|
34
|
+
requirement: !ruby/object:Gem::Requirement
|
35
|
+
requirements:
|
36
|
+
- - ">="
|
37
|
+
- !ruby/object:Gem::Version
|
38
|
+
version: '0'
|
39
|
+
name: logstash-devutils
|
40
|
+
prerelease: false
|
41
|
+
type: :development
|
42
|
+
version_requirements: !ruby/object:Gem::Requirement
|
43
|
+
requirements:
|
44
|
+
- - ">="
|
45
|
+
- !ruby/object:Gem::Version
|
46
|
+
version: '0'
|
47
|
+
description: This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program
|
48
|
+
email: info@elastic.co
|
49
|
+
executables: []
|
50
|
+
extensions: []
|
51
|
+
extra_rdoc_files: []
|
52
|
+
files:
|
53
|
+
- CHANGELOG.md
|
54
|
+
- CONTRIBUTORS
|
55
|
+
- Gemfile
|
56
|
+
- LICENSE
|
57
|
+
- NOTICE.TXT
|
58
|
+
- README.md
|
59
|
+
- docs/index.asciidoc
|
60
|
+
- lib/logstash-filter-useragent_jars.rb
|
61
|
+
- lib/logstash/filters/useragent.rb
|
62
|
+
- logstash-filter-useragent.gemspec
|
63
|
+
- spec/filters/useragent_spec.rb
|
64
|
+
- vendor/jar-dependencies/org/logstash/filters/logstash-filter-useragent/3.1.0/logstash-filter-useragent-3.1.0.jar
|
65
|
+
homepage: http://www.elastic.co/guide/en/logstash/current/index.html
|
66
|
+
licenses:
|
67
|
+
- Apache License (2.0)
|
68
|
+
metadata:
|
69
|
+
logstash_plugin: 'true'
|
70
|
+
logstash_group: filter
|
71
|
+
post_install_message:
|
72
|
+
rdoc_options: []
|
73
|
+
require_paths:
|
74
|
+
- lib
|
75
|
+
- vendor/jar-dependencies
|
76
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
77
|
+
requirements:
|
78
|
+
- - ">="
|
79
|
+
- !ruby/object:Gem::Version
|
80
|
+
version: '0'
|
81
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
82
|
+
requirements:
|
83
|
+
- - ">="
|
84
|
+
- !ruby/object:Gem::Version
|
85
|
+
version: '0'
|
86
|
+
requirements: []
|
87
|
+
rubyforge_project:
|
88
|
+
rubygems_version: 2.4.8
|
89
|
+
signing_key:
|
90
|
+
specification_version: 4
|
91
|
+
summary: Parse user agent strings into structured data based on BrowserScope data
|
92
|
+
test_files:
|
93
|
+
- spec/filters/useragent_spec.rb
|