logstash-filter-greynoise 0.1.3 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d47cd7afd1f7d7a6173276637585c6ef77f34c017a6c4d3217dfb60bb348b913
-  data.tar.gz: a18bdcb399a28ffa3849bb0d4c1937c042b39a22b7f482b0f0e997d381d70618
+  metadata.gz: 9a8f0682f3292b7e21b6aeb2200b45ad16d1e71b6f60f6c74db54e6a1a7d229d
+  data.tar.gz: 248adab329c505788c4a394a839fcaf0a63ca72b76c20a8b4407839206acf191
 SHA512:
-  metadata.gz: d8a6d98c4e3f987c99d821e310f720e3b2fde984ddd2d7087f212b9dd26351456afe17cdce768b55f9b79b04aa2e2c93a52770a316e7790146078304a2f78e37
-  data.tar.gz: 3a72cc450f862dcb0632b05458d670bf46e1f9332ad88ad0449bd2e31a215343d8683fe963b3e18e9e0739f1282a9fd710ea6b00a2c466e4f4ea13028d764846
+  metadata.gz: f10678522e0686ffd212fa53275da708daa1900bbb4e13e982e91e491f08c2815d39f3d78160a1d9150960108d4377bf9aa4e59463fd7afe4614ac0adfb6056b
+  data.tar.gz: 8d07a3c74c739141c08ce930eed271b3c0ea70aa52d4023afbcf0ee9c1970f1df2485a5a365a94958836db8aabfac63b329fbb1334122505ff668656602c7caf

data/CHANGELOG.md

@@ -6,3 +6,10 @@
   - Added user-agent  
 ## 0.1.3 
   - Removed unused dependencies  
+## 0.1.4
+  - Updated key check for len
+## 0.1.5
+  - Fixed nil key error
+## 0.1.6
+  - Added LRU cache options and filter for invalid ips
+ 

data/CONTRIBUTORS

@@ -2,7 +2,7 @@ The following is a list of people who have contributed ideas, code, bug
 reports, or in general have helped logstash along its way.
 
 Contributors:
-* nsherron90 - nsherron90@gmail.com
+* nicksherron - nsherron90@gmail.com
 
 Note: If you've sent us patches, bug reports, or otherwise contributed to
 Logstash, and you aren't on the list above and want to be, please let us know

data/DEVELOPER.md

@@ -1,2 +1,19 @@
 # logstash-filter-greynoise
-Example filter plugin. This should help bootstrap your effort to write your own filter plugin!
+
+You can build this project on your machine locally or alternative use the docker file to build and run an environment in Docker.
+
+To use Docker, run the following:
+
+```
+# build the docker container for local dev
+make build-docker
+
+# shell into docker environment
+make shell-docker
+
+# set GN key for tests
+export GN_API_KEY=<GN key>
+
+# run tests
+bundle exec rspec
+``` 

data/README.md

@@ -1,11 +1,11 @@
-# Logstash REST Filter 
+# Logstash Greynoise Filter 
 This is a filter plugin for [Logstash](https://github.com/elastic/logstash).
 
 It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
 
 ## Documentation
 
-The Greynoise filter adds information about IP addresses from logstash events via the Greynoise API.
+The GreyNoise filter adds information about IP addresses from logstash events via the GreyNoise API.
 
 GreyNoise is a system that collects and analyzes data on Internet-wide scanners.
 GreyNoise collects data on benign scanners such as Shodan.io, as well as malicious actors like SSH and telnet worms.
@@ -19,24 +19,41 @@ $LS_HOME/bin/logstash-plugin install logstash-filter-greynoise
 
 Or you can build it yourself:
 ```
-git clone https://github.com/nsherron90/logstash-filter-greynoise.git
+git clone https://github.com/nicksherron/logstash-filter-greynoise.git
 bundle install
 gem build logstash-filter-greynoise.gemspec
-$LS_HOME/bin/logstash-plugin install logstash-filter-greynoise-0.1.3.gem
+$LS_HOME/bin/logstash-plugin install logstash-filter-greynoise-0.1.7.gem
 ```
 
 ### 2. Filter Configuration
-Add the following inside the filter section of your logstash configuration:
 
 ```sh
 filter {
   greynoise {
-    ip => "ip_value"                 # string (required, reference to ip address field)
-    key => "your_greynoise_key"      # string (optional, no default)
-    target => "greynoise"            # string (optional, default = greynoise)
+    ip             => "ip_value"              # string (required, reference to ip address field)
+    full_context   => true                    # bool (optional, whether to use context lookup, default false)
+    key            => "your_greynoise_key"    # string (required)
+    target         => "greynoise"             # string (optional, default = greynoise)
+    hit_cache_size => 100                     # number (optional, default = 0)
+    hit_cache_ttl  => 6                       # number (optional, default = 60) 
   }
 }
 ```
+The GreyNoise Logstash filter plugin can be used in two different modes:
+
+##### Quick IP Enrichment
+In this mode documents (default) are enriched with a bool field `seen` which is true if GreyNoise has seen this IP or false otherwise. 
+
+This mode is faster than doing full enrichment and should be used for better performance on high-volume/-throughput event streams.
+If you have a data pipeline with multiple enrichment points, you can use the boolean field to later enrich the document with IP information from GreyNoise's context endpoint.
+
+##### Full IP Enrichment
+
+In this mode (`full_context => true`), documents are enriched with full context from GreyNoise API if the IP has been observed by GreyNoise. 
+
+This mode is slower than doing quick IP enrichment and might be reasonable for low-volume/-throughput event streams. 
+
+**NOTE**: Beware that the full context document from GreyNoise API can be large and as such the cache can grow quickly in size. Set the cache size accordingly to prevent high memory consumption by the cache.
 
 Print plugin version:
 
@@ -79,3 +96,5 @@ Programming is not a required skill. Whatever you've seen about open source and
 It is more important to the community that you are able to contribute.
 
 For more information about contributing, see the [CONTRIBUTING](https://github.com/elasticsearch/logstash/blob/master/CONTRIBUTING.md) file.
+
+See also Logstash [Elastic's filter plugin development guide](https://www.elastic.co/guide/en/logstash/current/filter-new-plugin.html).

data/lib/logstash/filters/greynoise.rb

@@ -1,59 +1,142 @@
 # encoding: utf-8
-require "logstash/filters/base"
+require 'logstash/filters/base'
 require "json"
 require "logstash/namespace"
-require 'faraday'
+require "ipaddr"
+require "lru_redux"
+require 'net/http'
+require 'uri'
 
+VERSION = "1.0.0"
+
+class InvalidAPIKey < StandardError
+end
 
 # This  filter will replace the contents of the default
 # message field with whatever you specify in the configuration.
 #
-# It is only intended to be used as an .
 class LogStash::Filters::Greynoise < LogStash::Filters::Base
 
   # Setting the config_name here is required. This is how you
   # configure this filter from your Logstash config.
   #
-   #  filter {
-   #   greynoise {
-   #     ip => "ip"
-   #   }
-   #  }
+  #  filter {
+  #   greynoise {
+  #     ip => "ip"
+  #   }
+  #  }
 
   config_name "greynoise"
 
-  # Replace the message with this value.
-
+  # ip address to use for greynoise query
   config :ip, :validate => :string, :required => true
-  config :key, :validate => :string, :required => false
+
+  # whether or not to use full context endpoint
+  config :full_context, :validate => :boolean, :default => false
+
+  # greynoise enterprise api key
+  config :key, :validate => :string, :required => true
+
+  # target top level key of hash response
   config :target, :validate => :string, :default => "greynoise"
 
+  # tag if ip address supplied is invalid
+  config :tag_on_failure, :validate => :string, :default => '_greynoise_filter_invalid_ip'
+
+  # tag if API key not valid or missing
+  config :tag_on_auth_failure, :validate => :string, :default => '_greynoise_filter_invalid_api_key'
 
+  # set the size of cache for successful requests
+  config :hit_cache_size, :validate => :number, :default => 0
+
+  # how long to cache successful requests (in seconds)
+  config :hit_cache_ttl, :validate => :number, :default => 60
 
   public
+
   def register
-  end # def register
+    if @hit_cache_size > 0
+      @hit_cache = LruRedux::TTL::ThreadSafeCache.new(@hit_cache_size, @hit_cache_ttl)
+    end
+  end
 
-  public
-  def filter(event)
 
-    if @key
-      url = "https://enterprise.api.greynoise.io/v2/noise/context/" + event.sprintf(ip)
-      uri = URI.parse(URI.encode(url.strip))
+  private
 
-      response = Faraday.get(uri, nil, 'User-Agent' => 'logstash-filter-greynoise', Key: event.sprintf(key))
+  def lookup_ip(target_ip, api_key, context = false)
+    endpoint = "quick/"
+    if context
+      endpoint = "context/"
+    end
+
+    uri = URI.parse("https://api.greynoise.io/v2/noise/" + endpoint + target_ip)
+    request = Net::HTTP::Get.new(uri)
+    request["Key"] = api_key
+    request["User-Agent"] = "logstash-filter-greynoise " + VERSION
+    req_options = {
+        use_ssl: uri.scheme == "https",
+    }
+    response = Net::HTTP.start(uri.hostname, uri.port, req_options) { |http|
+      http.request(request)
+    }
+
+    if response.is_a?(Net::HTTPSuccess)
+      result = JSON.parse(response.body)
+      unless context
+        result["seen"] = result.delete("noise")
+      end
+      result
+    elsif response.is_a?(Net::HTTPUnauthorized)
+      raise InvalidAPIKey.new
     else
-      url = "https://api.greynoise.io/v1/query/ip"
-      response = Faraday.post url, { :ip => event.sprintf(ip) }, 'User-Agent' => 'logstash-filter-greynoise'
+      nil
+    end
+  end
+
+  public
 
+  def filter(event)
+    valid = nil
+    begin
+      IPAddr.new(event.sprintf(ip))
+    rescue ArgumentError => e
+      valid = e
     end
 
-    result = JSON.parse(response.body)
+    if valid
+      @logger.error("Invalid IP address, skipping", :ip => event.sprintf(ip), :event => event.to_hash)
+      event.tag(@tag_on_failure)
+      return
+    end
 
-    event.set(@target, result)
-    # filter_matched should go in the last line of our successful code
-    filter_matched(event)
+    if @hit_cache
+      gn_result = @hit_cache[event.sprintf(ip)]
 
-  end # def filter
-end # class LogStash::Filters::Greynoise
+      # use cached data
+      if gn_result
+        event.set(@target, gn_result)
+        filter_matched(event)
+        return
+      end
+    end
+
+    # use GN API, since not found in cache
+    begin
+      gn_result = lookup_ip(event.sprintf(ip), event.sprintf(key), @full_context)
+      unless gn_result.nil?
+        if @hit_cache
+          # store in cache
+          @hit_cache[event.sprintf(ip)] = gn_result
+        end
+
+        event.set(@target, gn_result)
+        # filter_matched should go in the last line of our successful code
+        filter_matched(event)
+      end
+    rescue InvalidAPIKey => _
+      @logger.error("unauthorized - check API key")
+      event.tag(@tag_on_auth_failure)
+    end
+  end
 
+end

data/logstash-filter-greynoise.gemspec

@@ -1,10 +1,10 @@
 Gem::Specification.new do |s|
   s.name          = 'logstash-filter-greynoise'
-  s.version       = '0.1.3'
+  s.version       = '1.0.0'
   s.licenses      = ['Apache-2.0']
-  s.summary = 'This greynoise filter takes contents in the ip field and returns greynoise api data (see https://greynoise.io/ for more info).'
+  s.summary = 'This greynoise filter takes contents in the IP field and returns GreyNoise API data (see https://developer.greynoise.io/ for more info).'
   s.description     = 'This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install logstash-filter-greynoise. This gem is not a stand-alone program'
-  s.homepage = 'https://github.com/nsherron90/logstash-filter-greynoise'
+  s.homepage = 'https://github.com/nicksherron/logstash-filter-greynoise'
   s.authors       = ['nsherron90']
   s.email         = 'nsherron90@gmail.com'
   s.require_paths = ['lib']
@@ -18,8 +18,7 @@ Gem::Specification.new do |s|
   s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" }
 
   # Gem dependencies
-  s.add_runtime_dependency 'logstash-core-plugin-api', '~> 2.0'
   s.add_development_dependency 'logstash-devutils'
-  s.add_runtime_dependency  'faraday', '~> 0.9.2'
-
+  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
+  s.add_runtime_dependency 'lru_redux', "~> 1.1.0"
 end

data/spec/filters/greynoise_spec.rb

@@ -3,33 +3,47 @@ require_relative '../spec_helper'
 require "logstash/filters/greynoise"
 
 describe LogStash::Filters::Greynoise do
-
   describe "defaults" do
-    let(:config) do <<-CONFIG
+    let(:config) do
+      api_key = ENV["GN_API_KEY"]
+      <<-CONFIG
       filter {
         greynoise {
-          ip => "ip"
+          ip => "%{ip}"          
+          key => "#{api_key}"          
         }
       }
-    CONFIG
-    # end
+      CONFIG
+    end
 
     sample("ip" => "8.8.8.8") do
       insist { subject }.include?("greynoise")
-
-      expected_fields = %w(greynoise.ip greynoise.seen)
+      expected_fields = %w(ip seen code)
       expected_fields.each do |f|
         insist { subject.get("greynoise") }.include?(f)
       end
+      insist { subject.get("greynoise").get("code").equal?("0x05") }
+    end
+
+    sample("ip" => "4.2.1.A") do
+      insist { subject.get("tags") }.include?("_greynoise_filter_invalid_ip")
+    end
+  end
+
+  describe "invalid_key" do
+    let(:config) do
+      <<-CONFIG
+      filter {
+        greynoise {
+          ip => "%{ip}"          
+          key => "BAD_KEY"          
+        }
+      }
+      CONFIG
     end
+
+    sample("ip" => "8.8.8.8") do
+      insist { subject.get("tags") }.include?("_greynoise_filter_invalid_api_key")
     end
   end
 end
-#
-#
-#     sample("message" => "some text") do
-#       expect(subject).to include("message")
-#       expect(subject.get('message')).to eq('Hello World')
-#     end
-#   end
-# end

metadata

@@ -1,57 +1,63 @@
 --- !ruby/object:Gem::Specification
 name: logstash-filter-greynoise
 version: !ruby/object:Gem::Version
-  version: 0.1.3
+  version: 1.0.0
 platform: ruby
 authors:
 - nsherron90
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2019-04-29 00:00:00.000000000 Z
+date: 2020-12-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: logstash-core-plugin-api
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '2.0'
-  type: :runtime
+        version: '0'
+  name: logstash-devutils
+  type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '2.0'
+        version: '0'
 - !ruby/object:Gem::Dependency
-  name: logstash-devutils
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+  name: logstash-core-plugin-api
+  type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
 - !ruby/object:Gem::Dependency
-  name: faraday
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.9.2
+        version: 1.1.0
+  name: lru_redux
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.9.2
+        version: 1.1.0
 description: This gem is a Logstash plugin required to be installed on top of the
   Logstash core pipeline using $LS_HOME/bin/logstash-plugin install logstash-filter-greynoise.
   This gem is not a stand-alone program
@@ -70,13 +76,13 @@ files:
 - logstash-filter-greynoise.gemspec
 - spec/filters/greynoise_spec.rb
 - spec/spec_helper.rb
-homepage: https://github.com/nsherron90/logstash-filter-greynoise
+homepage: https://github.com/nicksherron/logstash-filter-greynoise
 licenses:
 - Apache-2.0
 metadata:
   logstash_plugin: 'true'
   logstash_group: filter
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -91,11 +97,11 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.3
-signing_key: 
+rubygems_version: 3.0.6
+signing_key:
 specification_version: 4
-summary: This greynoise filter takes contents in the ip field and returns greynoise
-  api data (see https://greynoise.io/ for more info).
+summary: This greynoise filter takes contents in the IP field and returns GreyNoise
+  API data (see https://developer.greynoise.io/ for more info).
 test_files:
 - spec/filters/greynoise_spec.rb
 - spec/spec_helper.rb