logstash-filter-weblookup 0.1.1 → 0.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2c96b1c02dc9936abcbed2834158d6df2d37b65423b92e0ddc0fb676b77c4b00
-  data.tar.gz: 49283b96b0ab8c0f00638c965897e01c0a6f29d8a413b7275bbe650c804849fb
+  metadata.gz: c5839cc6259028cb20d7eb1395f0b37f739c12508489bf34c241506bb9d5ac33
+  data.tar.gz: 9b65c6f9f316daf2974f5e982d79a87a2108c465502c4f3649db7f2ce1f11994
 SHA512:
-  metadata.gz: fba0a7b28e64f24b4cc9733a161a7a368579edc0f0de2317992ee530811c196d11bef2f8fc2a328745f7c9b7cb531748d27f4632486171659f3fecfb0c0f5f26
-  data.tar.gz: 50517ea043a5b254db9e74ffed413d2cdaf0f6b4c3bb3ee1ce3890288b69be7fc1bc9591b50fec9649f49b18d96b99fe6d3df73673194060bcba44f4ce51179f
+  metadata.gz: 96f53b391486a826b7a9b1c22c99c97a1f473b1a1284dcd7fa0300f75737b02279a5da8e08948a3c80aaf7343218987ebe19f6b8223972cce5074399c4fac2fc
+  data.tar.gz: 38387275e213910b857db6364deaffdf85f8fb2cfb2fc8315d6b918e3d905992da7171a6452cca6204c0023c27f3883c6864869c14dad916ae23f5971d233349

data/README.md

@@ -1,86 +1,25 @@
 # Logstash Plugin
 
-This is a plugin for [Logstash](https://github.com/elastic/logstash).
-
-It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+This gem is a plugin for [Logstash](https://github.com/elastic/logstash). During filter it takes one or more fields and uses that as input to query additional information. The original purpose is to enrich IP addresses with matching subnet, netname and hostname, but it is generic so that any field can be looked up. The function is similar to the translate filter's dictionary lookup, which supports files and regex. The jdbc_streaming filter plugin is also very useful if the data resides in a database. This plugins features are web based lookups and redis caching, for fast lookups.
 
 ## Documentation
 
-Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
-
-- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
-- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
-
-## Need Help?
-
-Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
-
-## Developing
-
-### 1. Plugin Developement and Testing
-
-#### Code
-- To get started, you'll need JRuby with the Bundler gem installed.
-
-- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
-
-- Install dependencies
-```sh
-bundle install
-```
-
-#### Test
+weblookup {
+    fields => ['[client][ip]']
+    destinations => ['net']
+    url => "http://localhost/ripe.php?ip=<item>"
+    use_redis => true
+    redis_path => "/var/run/redis/redis-server.sock"
+    normalize => true
+    newroot => "[records][properties]"
+    roottostrip => "[records]"
+}
 
-- Update your dependencies
+Where <item> will be replaced by the value of client.ip
 
-```sh
-bundle install
-```
+The first three components are needed for the plugin, the others are optional. use_redis and redis_path are for caching the response, this speedsup the requists. It's also possible to hardcode values here, but I'm not using it myself yet. normalize, newroot and roottostrip probably would be better in a separte plugin, but for now weblookup can move the json objects inside the roottostrip into it's own root, by default elasticsearch uses _source as invisible root.
 
-- Run tests
-
-```sh
-bundle exec rspec
-```
-
-### 2. Running your unpublished Plugin in Logstash
-
-#### 2.1 Run in a local Logstash clone
-
-- Edit Logstash `Gemfile` and add the local plugin path, for example:
-```ruby
-gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
-```
-- Install plugin
-```sh
-bin/logstash-plugin install --no-verify
-```
-- Run Logstash with your plugin
-```sh
-bin/logstash -e 'filter {awesome {}}'
-```
-At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
-
-#### 2.2 Run in an installed Logstash
-
-You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
-
-- Build your plugin gem
-```sh
-gem build logstash-filter-awesome.gemspec
-```
-- Install the plugin from the Logstash home
-```sh
-bin/logstash-plugin install /your/local/plugin/logstash-filter-awesome.gem
-```
-- Start Logstash and proceed to test the plugin
-
-## Contributing
-
-All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
-
-Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+## Need Help?
 
-It is more important to the community that you are able to contribute.
+Need help? Raise an issue on https://github.com/janmg/logstash-filter-weblookup 
 
-For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/filters/weblookup.rb

@@ -108,8 +108,12 @@ def filter(event)
     else
         fields.each_with_index do |field, index|
             # @logger.info(event.get("["+field+"]"))
-            json = parse(event.get(field).to_s)
-            event.set("["+destinations[index]+"]", json)
+            begin
+             json = parse(event.get(field).to_s)
+             event.set("["+destinations[index]+"]", json)
+            rescue Exception => e
+             @logger.error(" caught: #{e.message}")
+            end 
         end
     end
     if @normalize
@@ -127,16 +131,18 @@ def parse(field)
     begin
         json = JSON.parse(x)
     rescue JSON::ParserError
-        json = x
-    end			
+        json = JSON.parse("{\"ip\": \""+field+"\"}")
+    end
+    # @logger.info("json parse option for field #{field} / #{json}")
 end
 
 def find(item)
-    res = nil
-    # Is item in list? (list is an optional hash)
-    unless list.nil? 
-        return list[item]
-    end
+    res = "{}"
+    # Is item in list? (list is an optional array)
+    #unless list.nil? 
+        # What if the list exists, but item is not on the list?
+    #    return list[item]
+    #end
     # Is item in redis?
     unless @red.nil?
         res = @red.get(item)
@@ -150,7 +156,11 @@ def find(item)
     current_uri.query_values = @params.merge({@ip => item})
     #logger.info(@uri.to_s)
     @connpool.with do |conn|
-        res = conn.request_get(current_uri).read_body
+        http_response = conn.request_get(current_uri)
+	res = http_response.read_body if http_response.is_a?(Net::HTTPSuccess)
+	if res.eql? "null"
+            res = "{}"
+        end
 	#logger.info(res.to_s)
         unless @red.nil?
             @red.set(item, res)
@@ -193,4 +203,4 @@ def yml_loader(data)
     get_map.merge!(YAML.load_file(data))
 end
 
-end # class LogStash::Filters::Lookup
+end

data/logstash-filter-weblookup.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name          = 'logstash-filter-weblookup'
-  s.version       = '0.1.1'
+  s.version       = '0.1.2'
   s.licenses      = ['Apache-2.0']
   s.summary       = 'This logstash filter plugin takes one or more fields and enriches with a lookup value from a list, redis cache or webservice'
   s.description   = <<-EOF
@@ -20,7 +20,7 @@ EOF
   s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" }
 
   # Gem dependencies
-  s.add_runtime_dependency 'logstash-core-plugin-api', '~> 2.0'
+  s.add_runtime_dependency 'logstash-core-plugin-api', '~> 2.1'
   s.add_runtime_dependency 'connection_pool', '~> 2.2'
   #s.add_runtime_dependency 'addressable', '~> 2.3.8'
   s.add_development_dependency 'logstash-devutils', '~> 0'

metadata

@@ -1,21 +1,21 @@
 --- !ruby/object:Gem::Specification
 name: logstash-filter-weblookup
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.2
 platform: ruby
 authors:
 - Jan Geertsma
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2019-10-31 00:00:00.000000000 Z
+date: 2019-11-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '2.0'
+        version: '2.1'
   name: logstash-core-plugin-api
   prerelease: false
   type: :runtime
@@ -23,7 +23,7 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '2.0'
+        version: '2.1'
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
     requirements:
@@ -96,8 +96,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project:
-rubygems_version: 2.7.9
+rubygems_version: 3.0.6
 signing_key:
 specification_version: 4
 summary: This logstash filter plugin takes one or more fields and enriches with a