logstash-filter-useragent2 3.0.0-java

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 15bcd828b94689a402662e2dea628cb8cc95c94f
+  data.tar.gz: f9940c5b8ec2b47a44cbbfcee5d555823fd5609d
+SHA512:
+  metadata.gz: 38cf7c7a81e129908467570c4239c6ee2c9703224ce2c602f8f91469dda02a81271612270e080f628064bf3936dc9fdd912e29286a8e7b59cb717cf23cdbb6e5
+  data.tar.gz: 877b57f4ff746cd07261b612f074e736103c288d9487d2b00b036713f82bf2759f28f6ee73b682a1cfd182e74c8601b7955ae9247e099c90bac3d2dbc481c46b

data/CHANGELOG.md

@@ -0,0 +1,17 @@
+# 2.0.6
+  - Depend on logstash-core-plugin-api instead of logstash-core, removing the need to mass update plugins on major releases of logstash
+# 2.0.5
+  - New dependency requirements for logstash-core for the 5.0 release
+## 2.0.4
+ - Fefactored field references, fixed specs and some cleanups
+
+## 2.0.0
+ - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully,
+   instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
+ - Dependency on logstash-core update to 2.0
+
+## 2.0.1
+  - Add ability to replace source with target
+
+## 1.1.0
+  - Add LRU cache

data/CONTRIBUTORS

@@ -0,0 +1,18 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Alex Tambellini (atambo)
+* Colin Surprenant (colinsurprenant)
+* Dan Everton (deverton)
+* Jay Luker (lbjay)
+* Jordan Sissel (jordansissel)
+* Kubes (pkubat)
+* Pier-Hugues Pellerin (ph)
+* Richard Pijnenburg (electrical)
+* Suyog Rao (suyograo)
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+Elasticsearch
+Copyright 2012-2015 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,88 @@
+# Logstash Plugin
+
+[![Build Status](https://travis-ci.org/logstash-plugins/logstash-filter-useragent.svg?branch=master)](https://travis-ci.org/logstash-plugins/logstash-filter-useragent)
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+bin/plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash-filter-useragent_jars.rb

@@ -0,0 +1,4 @@
+# this is a generated file, to avoid over-writing it just delete this comment
+require 'jar_dependencies'
+
+require_jar( 'eu.bitwalker', 'UserAgentUtils', '1.19' )

data/lib/logstash/filters/useragent.rb

@@ -0,0 +1,132 @@
+# encoding: utf-8
+require "logstash/filters/base"
+require "logstash/namespace"
+require "lru_redux"
+require "tempfile"
+require "logstash-filter-useragent_jars"
+
+java_import "eu.bitwalker.useragentutils.UserAgent"
+
+# Parse user agent strings into structured data based on the Java library:
+# <https://github.com/HaraldWalker/user-agent-utils>
+#
+# UserAgent filter, adds information about user agent like family, operating
+# system, version, and device
+#
+class LogStash::Filters::UserAgent < LogStash::Filters::Base
+  LOOKUP_CACHE = LruRedux::ThreadSafeCache.new(1000)
+
+  config_name "useragent2"
+
+  # The field containing the user agent string. If this field is an
+  # array, only the first value will be used.
+  config :source, :validate => :string, :required => true
+
+  # The name of the field to assign user agent data into.
+  #
+  # If not specified user agent data will be stored in the root of the event.
+  config :target, :validate => :string
+
+  # A string to prepend to all of the extracted keys
+  config :prefix, :validate => :string, :default => ''
+
+  # UA parsing is surprisingly expensive. This filter uses an LRU cache to take advantage of the fact that
+  # user agents are often found adjacent to one another in log files and rarely have a random distribution.
+  # The higher you set this the more likely an item is to be in the cache and the faster this filter will run.
+  # However, if you set this too high you can use more memory than desired.
+  #
+  # Experiment with different values for this option to find the best performance for your dataset.
+  #
+  # This MUST be set to a value > 0. There is really no reason to not want this behavior, the overhead is minimal
+  # and the speed gains are large.
+  #
+  # It is important to note that this config value is global. That is to say all instances of the user agent filter
+  # share the same cache. The last declared cache size will 'win'. The reason for this is that there would be no benefit
+  # to having multiple caches for different instances at different points in the pipeline, that would just increase the
+  # number of cache misses and waste memory.
+  config :lru_cache_size, :validate => :number, :default => 1000
+
+  def register
+
+    LOOKUP_CACHE.max_size = @lru_cache_size
+
+    # make @target in the format [field name] if defined, i.e. surrounded by brakets
+    @normalized_target = (@target && @target !~ /^\[[^\[\]]+\]$/) ? "[#{@target}]" : ""
+
+  end
+
+  def filter(event)
+    useragent = event[@source]
+    useragent = useragent.first if useragent.is_a?(Array)
+
+    return if useragent.nil? || useragent.empty?
+
+    begin
+      ua_data = lookup_useragent(useragent)
+    rescue StandardError => e
+      @logger.error("Uknown error while parsing user agent data", :exception => e, :field => @source, :event => event)
+      return
+    end
+
+    return unless ua_data
+
+    event.remove(@source) if @target == @source
+    set_fields(event, ua_data)
+
+    filter_matched(event)
+  end
+
+  # should be private but need to stay public for specs
+  # TODO: (colin) the related specs should be refactored to not rely on private methods.
+  def lookup_useragent(useragent)
+    return unless useragent
+
+    cached = LOOKUP_CACHE[useragent]
+    return cached if cached
+
+    ua_data = parse_useragent(useragent)
+
+    LOOKUP_CACHE[useragent] = ua_data
+    ua_data
+  end
+
+  private
+
+  # Transform UserAgent object to plain object
+  def parse_useragent(useragent)
+    ua_raw_data = UserAgent.parseUserAgentString(useragent)
+    ua_data = Hash.new
+
+    os = ua_raw_data.getOperatingSystem()
+    browser = ua_raw_data.getBrowser()
+    version = ua_raw_data.getBrowserVersion()
+
+    if browser
+      ua_data["name"] = browser.getGroup().getName()
+      ua_data["fullname"] = browser.getName()
+      ua_data["vendor"] = browser.getManufacturer().getName()
+      ua_data["type"] = browser.getBrowserType().getName()
+    end
+
+    if version
+      ua_data["major"] = version.getMajorVersion()
+      ua_data["minor"] = version.getMinorVersion()
+      ua_data["version"] = version.getVersion()
+    end
+
+    if os
+      ua_data["os_fullname"] = os.getName()
+      ua_data["os_vendor"] = os.getManufacturer().getName()
+      ua_data["os"] = os.getGroup().getName()
+      ua_data["device"] = os.getDeviceType().getName()
+    end
+
+    ua_data
+  end
+
+  def set_fields(event, ua_data)
+    ua_data.each do |field, value|
+      event["#{@normalized_target}[#{@prefix}#{field}]"] = value.dup if value
+    end
+  end
+end

data/logstash-filter-useragent.gemspec

@@ -0,0 +1,31 @@
+Gem::Specification.new do |s|
+
+  s.name            = 'logstash-filter-useragent2'
+  s.version         = '3.0.0'
+  s.licenses        = ['Apache-2.0']
+  s.summary         = "Parse user agent strings into structured data based on BrowserScope data"
+  s.description     = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
+  s.authors         = ["Elastic"]
+  s.email           = 'info@elastic.co'
+  s.homepage        = "https://github.com/ebuildy/logstash-filter-useragent"
+  s.require_paths   = ["lib"]
+  s.platform        = 'java'
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+
+  # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core-plugin-api", "~> 1.0"
+  s.add_runtime_dependency 'lru_redux', '~> 1.1', '>= 1.1.0'
+
+  s.add_development_dependency 'jar-dependencies'
+  s.add_development_dependency 'logstash-devutils'
+
+  s.requirements << "jar eu.bitwalker:UserAgentUtils, 1.19"
+end

data/spec/filters/useragent_spec.rb

@@ -0,0 +1,129 @@
+# encoding: utf-8
+
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/filters/useragent"
+
+describe LogStash::Filters::UserAgent do
+
+  describe "defaults" do
+    config <<-CONFIG
+      filter {
+        useragent {
+          source => "message"
+          target => "ua"
+        }
+      }
+    CONFIG
+
+    sample "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.63 Safari/537.31" do
+      insist { subject }.include?("ua")
+      insist { subject["ua"]["name"] } == "Chrome"
+      insist { subject["ua"]["os"] } == "Linux"
+      insist { subject["ua"]["major"] } == "26"
+      insist { subject["ua"]["minor"] } == "0"
+    end
+  end
+
+  describe "Without target field" do
+    config <<-CONFIG
+      filter {
+        useragent {
+          source => "message"
+        }
+      }
+    CONFIG
+
+    sample "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.63 Safari/537.31" do
+      insist { subject["name"] } == "Chrome"
+      insist { subject["os"] } == "Linux"
+      insist { subject["major"] } == "26"
+      insist { subject["minor"] } == "0"
+    end
+  end
+
+  describe "Without user agent" do
+    config <<-CONFIG
+      filter {
+        useragent {
+          source => "message"
+          target => "ua"
+        }
+      }
+    CONFIG
+
+    sample "foo" => "bar" do
+      reject { subject }.include?("ua")
+    end
+
+    sample "" do
+      reject { subject }.include?("ua")
+    end
+  end
+
+  describe "LRU object identity" do
+    let(:ua_string) { "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36" }
+    let(:uafilter) { LogStash::Filters::UserAgent.new("source" => "foo") }
+    let(:ua_data) { uafilter.lookup_useragent(ua_string) }
+
+    subject(:target) { LogStash::Event.new("foo" => ua_string) }
+
+    before do
+      uafilter.register
+
+      # Stub this out because this UA doesn't have this field
+      allow(ua_data.version).to receive(:patch_minor).and_return("foo")
+
+      # expect(event).receive(:lookup_useragent)
+      uafilter.filter(target)
+    end
+
+    {
+      "name" => lambda {|uad| uad.name},
+      "os" => lambda {|uad| uad.os.to_s},
+      "os_name" => lambda {|uad| uad.os.name},
+      "os_major" => lambda {|uad| uad.os.version.major},
+      "os_minor" => lambda {|uad| uad.os.version.minor},
+      "device" => lambda {|uad| uad.device.to_s},
+      "major" => lambda {|uad| uad.version.major},
+      "minor" => lambda {|uad| uad.version.minor},
+      "patch" => lambda {|uad| uad.version.patch},
+      "build" => lambda {|uad| uad.version.patch_minor}
+    }.each do |field, uad_getter|
+      context "for the #{field} field" do
+        let(:value) {uad_getter.call(ua_data)}
+        let(:target_field) { target[field]}
+
+        it "should not have a nil value" do
+          expect(target_field).to be_truthy
+        end
+
+        it "should have equivalent values" do
+          expect(target_field).to eql(value)
+        end
+
+        it "should dup/clone the field to prevent cache corruption" do
+          expect(target_field.object_id).not_to eql(value.object_id)
+        end
+      end
+    end
+  end
+
+  describe "Replace source with target" do
+    config <<-CONFIG
+      filter {
+        useragent {
+          source => "message"
+          target => "message"
+        }
+      }
+    CONFIG
+
+    sample "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.63 Safari/537.31" do
+      insist { subject.to_hash }.include?("message")
+      insist { subject["message"]["name"] } == "Chrome"
+      insist { subject["message"]["os"] } == "Linux"
+      insist { subject["message"]["major"] } == "26"
+      insist { subject["message"]["minor"] } == "0"
+    end
+  end
+end

metadata

@@ -0,0 +1,122 @@
+--- !ruby/object:Gem::Specification
+name: logstash-filter-useragent2
+version: !ruby/object:Gem::Version
+  version: 3.0.0
+platform: java
+authors:
+- Elastic
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2016-04-04 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: logstash-core-plugin-api
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+- !ruby/object:Gem::Dependency
+  name: lru_redux
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.1'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.1.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.1'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.1.0
+- !ruby/object:Gem::Dependency
+  name: jar-dependencies
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: logstash-devutils
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: This gem is a logstash plugin required to be installed on top of the
+  Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not
+  a stand-alone program
+email: info@elastic.co
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CONTRIBUTORS
+- Gemfile
+- LICENSE
+- NOTICE.TXT
+- README.md
+- lib/eu/bitwalker/UserAgentUtils/1.19/UserAgentUtils-1.19.jar
+- lib/logstash-filter-useragent_jars.rb
+- lib/logstash/filters/useragent.rb
+- logstash-filter-useragent.gemspec
+- spec/filters/useragent_spec.rb
+homepage: https://github.com/ebuildy/logstash-filter-useragent
+licenses:
+- Apache-2.0
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: filter
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements:
+- jar eu.bitwalker:UserAgentUtils, 1.19
+rubyforge_project: 
+rubygems_version: 2.6.2
+signing_key: 
+specification_version: 4
+summary: Parse user agent strings into structured data based on BrowserScope data
+test_files:
+- spec/filters/useragent_spec.rb