logstash-filter-cloudfoundry 0.3.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: bbd641a5d1f0798692f1b8906ed06f63afc2c633
-  data.tar.gz: ea84e455e4d94075cff2437494e375f865336249
+  metadata.gz: b4c0d8f5f199a6b09e45a1f58658c40ee57fd4db
+  data.tar.gz: 8ad64182692ce3bc36b4b4859aa83d5d46cb20e2
 SHA512:
-  metadata.gz: 92fe5abf5c2038c2b6104aa0a56d61438631d5a09781ae8643855aebb0794d7b098bfec730c08fa7cf928d802e770995b7632e7e271152999113fe6047d29fc3
-  data.tar.gz: 9bdbb110d18ee58e2783e8227dfc34d32a83972ce4e9a1d985e2a796cbc4f8cfe959af5206c7152c6d43943f37ff119b5bdf00ce587418d3bf691702f7c6c004
+  metadata.gz: 29b465b97d6d1bc2f0e47f7572ec4b951f76b18c794d6fa0b0d2b4b4564b140553076673ac33f85a4cf3f01e3a905cf26b4863bea55398d26ea434fae1206ff2
+  data.tar.gz: 591f8897ff8490ec8a52072bdf8e4d5df853dae96f15fb1e948bc447064726de5a0d04cc60c101183734ef132ed5851c767f374e45adde564ee13c6a9335c721

data/README.md

@@ -6,15 +6,50 @@ It is fully free and fully open source. The license is Apache 2.0, meaning you a
 
 ## Documentation
 
-Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elasticsearch.org/guide/en/logstash/current/).
-
-- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
-- For more asciidoc formatting tips, see the excellent reference here https://github.com/elasticsearch/docs#asciidoc-guide
-
-## Need Help?
-
-Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
-
+The Cloud Foundry filter will add the following meta-data to an application log
+- Org name
+- Space name
+- Application name
+
+Cloud Foundry only provides an applications GUID when shipping logs directly from the loggregator when using a syslog drain. This filter will use that GUID and the Cloud Foundry CLI to look up information about your application. 
+
+That being said, for this filter to work you will need the CF CLI installed on your system. (https://github.com/cloudfoundry/cli).
+
+This filter only processes 1 event at a time so the use of this filter can significantly slow down your pipeline's throughput if you have a high latency network. When the filter is initialized a cache will be created that will containt an applications GUID and relevant data. This is put in place to minimize the number of connections the filter will need to make. Instead of preforming a look up on every single Cloud Foundry log, the filter will look up the log the first time and refer to the cache for subsiquent calls (until the item is removed from the cache). The cache parameters should be configured according to your network and pipelines preformance. 
+
+In the event of a network or Cloud Foundry outage, the filter has some safeguards to protect your pipelines throughput. If the Cloud Foundry endpoint becomes unreachable you can set a timeout period before the CF CLI tries to communicate with the Cloud Foundry endpoint again. 
+
+This filter can be used by any user in the Cloud Foundry environemnt that has the "space developer" role for the applications you want to collect data from. Additionall, this filter supports paralle CF CLI logins. Meaning, if you have multiple Cloud Foundry endpoints, this filter can gracefully handle collecting data from both of them at the time. 
+
+Below is a list of the available config fields
+- cf_api : The Cloud Foundry API endpoint
+- cf_user: A valid Cloud Foundry user that has premission to the applications you want data for
+- cf_password: The users password
+- cf_org: A valid Cloud Foundry org that a user has premission to (required for a successful login)
+- cf_space: A valid Cloud Foundry space in the selected org that a user has premission to (required for a successful login)
+- skip_ssl_validation: A boolean flag to skip ssl validation on login
+- cache_flush_time: How often you want the job to clean out the cache to run
+- cace_age_time: A cache items time to live
+- cf_retry_cli_timeout: After a failed attempt to reach the Cloud Foundry endpoint, how long should the filter wait before using the cf   CLI again
+- cache_invalid_guids: If the Cloud Foundry API receives an invalid guid, cache it so the plugin won't waste resources continuously      trying to look it up
+ 
+Here are some example configurations:
+```sh
+filter{
+  cloudfoundry{
+    cf_api      => "https://api.cf-domain.com"
+    cf_user     => "username"
+    cf_password => "password"
+    cf_org      => "system"
+    cf_space    => "apps_manager"
+    skip_ssl_validation  => true
+    cache_flush_time     => "10m"
+    cache_age_time       => 600
+    cf_retry_cli_timeout => 120
+    cache_invalid_guids  => false
+  }
+}
+```
 ## Developing
 
 ### 1. Plugin Developement and Testing
@@ -83,4 +118,4 @@ Programming is not a required skill. Whatever you've seen about open source and
 
 It is more important to the community that you are able to contribute.
 
-For more information about contributing, see the [CONTRIBUTING](https://github.com/elasticsearch/logstash/blob/master/CONTRIBUTING.md) file.
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elasticsearch/logstash/blob/master/CONTRIBUTING.md) file.

data/logstash-filter-cloudfoundry.gemspec

@@ -1,12 +1,12 @@
 Gem::Specification.new do |s|
   s.name          = 'logstash-filter-cloudfoundry'
-  s.version       = '0.3.0'
+  s.version       = '0.4.0'
   s.licenses      = ['Apache License (2.0)']
   s.summary       = "Plugin used to assign meta-data to cloud foundry logs"
-  s.description   = "This filter will add App, Space, and Org metadata to CloudFoundry logs. This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
+  s.description   = "This filter will add App, Space, and Org metadata to CloudFoundry logs. See github README for more information. This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
   s.authors       = ["Tyler Stigliano"]
   s.email         = 'info@elastic.co'
-  s.homepage      = "https://github.com/logstash-plugins/logstash-filter-example/"
+  s.homepage      = "https://github.com/ahahtyler/logstash-filter-cloudfoundry"
   s.require_paths = ["lib"]
 
   # Files

data/spec/filters/cloudfoundry_spec.rb

@@ -1,20 +1,207 @@
-require 'spec_helper'
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"
 require "logstash/filters/cloudfoundry"
 
 describe LogStash::Filters::CloudFoundry do
 
-  describe "Set to Hello World" do
-    let(:config) do <<-CONFIG
-      filter {
-        cloudfoundry {
-        }
-      }
-    CONFIG
+  subject(:plugin) { LogStash::Filters::CloudFoundry.new(config) }
+  let(:config) { Hash.new }
+
+  let(:doc)   { "" }
+  let(:event) { LogStash::Event.new("message" => doc) }
+
+  #Received invalid config inputs
+  describe "registration" do
+
+    before(:each) do
+      plugin.register
+    end
+
+    describe "got blank required data" do
+      let(:config) do {
+          "cf_api"      => "",
+          "cf_user"     => "",
+          "cf_password" => "",
+          "cf_org"      => "",
+          "cf_space"    => ""}
+      end
+
+      it "failed variables are set" do
+        expect(plugin.instance_variable_get(:@cf_logged_in)).to eq(false)
+        expect(plugin.instance_variable_get(:@cf_retry_cli)).to eq(false)
+      end
+    end
+
+    describe "got bad data" do
+      let(:config) do {
+          "cf_api"      => "bad data",
+          "cf_user"     => "bad data",
+          "cf_password" => "bad data",
+          "cf_org"      => "bad data",
+          "cf_space"    => "bad data"}
+      end
+
+      it "failed variables are set" do
+        expect(plugin.instance_variable_get(:@cf_logged_in)).to eq(false)
+        expect(plugin.instance_variable_get(:@cf_retry_cli)).to eq(false)
+      end
+
+      it "CF HOME variable was set" do
+        expect(File.directory?(plugin.instance_variable_get(:@cf_path))).to be(true)
+      end
+    end
+
+    describe "got valid data without retry" do
+      let(:config) do {
+          "cf_api"      => "https://api.run.pivotal.io",
+          "cf_user"     => "username",
+          "cf_password" => "password",
+          "cf_org"      => "ITCS",
+          "cf_space"    => "development"}
+      end
+
+      it "successful variables are set" do
+        expect(plugin.instance_variable_get(:@cf_logged_in)).to eq(true)
+        expect(plugin.instance_variable_get(:@cf_retry_cli)).to eq(false)
+      end
+
+      it "CF HOME variable was set" do
+        expect(File.directory?(plugin.instance_variable_get(:@cf_path))).to be(true)
+      end
+    end
+
+    describe "got valid data with retry" do
+      let(:config) do {
+          "cf_api"      => "https://api.run.pivotal.io",
+          "cf_user"     => "username",
+          "cf_password" => "password",
+          "cf_org"      => "ITCS",
+          "cf_space"    => "development",
+          "cf_retry_cli_timeout" => 10}
+      end
+
+      it "successful variables are set" do
+        expect(plugin.instance_variable_get(:@cf_logged_in)).to eq(true)
+        expect(plugin.instance_variable_get(:@cf_retry_cli)).to eq(true)
+      end
+
+      it "CF HOME variable was set" do
+        expect(File.directory?(plugin.instance_variable_get(:@cf_path))).to be(true)
+      end
+    end
+
+  end
+
+  describe "filtering" do
+
+    before(:each) do
+      plugin.register
     end
 
-    sample("message" => "some text") do
-      expect(subject).to include("message")
-      expect(subject['message']).to eq('Hello World')
+    describe "got valid data" do
+      let(:config) do {
+          "cf_api"      => "https://api.run.pivotal.io",
+          "cf_user"     => "username",
+          "cf_password" => "password",
+          "cf_org"      => "ITCS",
+          "cf_space"    => "development"}
+      end
+
+      let(:doc) { "abc loggregator 964db4e9-4f9c-42a2-9296-f8381ce459eb sample log" }
+
+      it "extract all the values" do
+        plugin.filter(event)
+        expect(event.get("orgname")).to eq("ITCS")
+        expect(event.get("spacename")).to eq("development")
+        expect(event.get("appname")).to eq("spring-music")
+        expect(event.get("appguid")).to eq("964db4e9-4f9c-42a2-9296-f8381ce459eb")
+      end
+
+      it "extract all values from cache" do
+        plugin.filter(event)
+        expect(plugin.instance_variable_get(:@app_cache)[event.get("appguid")].nil?).to be(false)
+      end
+    end
+
+    describe "got valid data from different space" do
+      let(:config) do {
+          "cf_api"      => "https://api.run.pivotal.io",
+          "cf_user"     => "username",
+          "cf_password" => "password",
+          "cf_org"      => "ITCS",
+          "cf_space"    => "test"}
+      end
+
+      let(:doc) { "abc loggregator 964db4e9-4f9c-42a2-9296-f8381ce459eb sample log" }
+
+      it "extract all the values" do
+        plugin.filter(event)
+        expect(event.get("orgname")).to eq("ITCS")
+        expect(event.get("spacename")).to eq("development")
+        expect(event.get("appname")).to eq("spring-music")
+        expect(event.get("appguid")).to eq("964db4e9-4f9c-42a2-9296-f8381ce459eb")
+      end
+
+      it "extract all values from cache" do
+        plugin.filter(event)
+        expect(plugin.instance_variable_get(:@app_cache)[event.get("appguid")].nil?).to be(false)
+      end
     end
+
+    describe "got valid data with invalid guid" do
+      let(:config) do {
+          "cf_api"      => "https://api.run.pivotal.io",
+          "cf_user"     => "username",
+          "cf_password" => "password",
+          "cf_org"      => "ITCS",
+          "cf_space"    => "test",
+          "cache_invalid_guids" => false}
+      end
+
+      let(:doc) { "abc loggregator aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee sample log" }
+
+      it "make sure cache is empty" do
+        plugin.filter(event)
+        expect(plugin.instance_variable_get(:@app_cache)[event.get("appguid")].nil?).to be(true)
+      end
+
+    end
+
+    describe "got valid data with invalid guid" do
+      let(:config) do {
+          "cf_api"      => "https://api.run.pivotal.io",
+          "cf_user"     => "username",
+          "cf_password" => "password",
+          "cf_org"      => "ITCS",
+          "cf_space"    => "test",
+          "cache_invalid_guids" => true}
+      end
+
+      let(:doc) { "abc loggregator aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee sample log" }
+
+      it "make sure cache has value" do
+        plugin.filter(event)
+        expect(plugin.instance_variable_get(:@app_cache)[event.get("appguid")].nil?).to be(false)
+      end
+    end
+
+    describe "got valid data with no guid" do
+      let(:config) do {
+          "cf_api"      => "https://api.run.pivotal.io",
+          "cf_user"     => "username",
+          "cf_password" => "password",
+          "cf_org"      => "ITCS",
+          "cf_space"    => "test"}
+      end
+
+      let(:doc) { "sample log" }
+
+      it "make sure flags are set" do
+        plugin.filter(event)
+        expect(event.get("appguid").nil?).to eq(true)
+      end
+    end
+
   end
+
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-filter-cloudfoundry
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.4.0
 platform: ruby
 authors:
 - Tyler Stigliano
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-09-24 00:00:00.000000000 Z
+date: 2016-09-28 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: logstash-core
@@ -45,9 +45,9 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 description: This filter will add App, Space, and Org metadata to CloudFoundry logs.
-  This gem is a logstash plugin required to be installed on top of the Logstash core
-  pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone
-  program
+  See github README for more information. This gem is a logstash plugin required to
+  be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install
+  gemname. This gem is not a stand-alone program
 email: info@elastic.co
 executables: []
 extensions: []
@@ -67,7 +67,7 @@ files:
 - lib/logstash/filters/cloudfoundry.rb
 - logstash-filter-cloudfoundry.gemspec
 - spec/filters/cloudfoundry_spec.rb
-homepage: https://github.com/logstash-plugins/logstash-filter-example/
+homepage: https://github.com/ahahtyler/logstash-filter-cloudfoundry
 licenses:
 - Apache License (2.0)
 metadata: