logstash-filter-cloudfoundry 0.3.0 → 0.4.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +45 -10
- data/logstash-filter-cloudfoundry.gemspec +3 -3
- data/spec/filters/cloudfoundry_spec.rb +198 -11
- metadata +6 -6
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: b4c0d8f5f199a6b09e45a1f58658c40ee57fd4db
|
4
|
+
data.tar.gz: 8ad64182692ce3bc36b4b4859aa83d5d46cb20e2
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 29b465b97d6d1bc2f0e47f7572ec4b951f76b18c794d6fa0b0d2b4b4564b140553076673ac33f85a4cf3f01e3a905cf26b4863bea55398d26ea434fae1206ff2
|
7
|
+
data.tar.gz: 591f8897ff8490ec8a52072bdf8e4d5df853dae96f15fb1e948bc447064726de5a0d04cc60c101183734ef132ed5851c767f374e45adde564ee13c6a9335c721
|
data/README.md
CHANGED
@@ -6,15 +6,50 @@ It is fully free and fully open source. The license is Apache 2.0, meaning you a
|
|
6
6
|
|
7
7
|
## Documentation
|
8
8
|
|
9
|
-
|
10
|
-
|
11
|
-
-
|
12
|
-
-
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
9
|
+
The Cloud Foundry filter will add the following meta-data to an application log
|
10
|
+
- Org name
|
11
|
+
- Space name
|
12
|
+
- Application name
|
13
|
+
|
14
|
+
Cloud Foundry only provides an applications GUID when shipping logs directly from the loggregator when using a syslog drain. This filter will use that GUID and the Cloud Foundry CLI to look up information about your application.
|
15
|
+
|
16
|
+
That being said, for this filter to work you will need the CF CLI installed on your system. (https://github.com/cloudfoundry/cli).
|
17
|
+
|
18
|
+
This filter only processes 1 event at a time so the use of this filter can significantly slow down your pipeline's throughput if you have a high latency network. When the filter is initialized a cache will be created that will containt an applications GUID and relevant data. This is put in place to minimize the number of connections the filter will need to make. Instead of preforming a look up on every single Cloud Foundry log, the filter will look up the log the first time and refer to the cache for subsiquent calls (until the item is removed from the cache). The cache parameters should be configured according to your network and pipelines preformance.
|
19
|
+
|
20
|
+
In the event of a network or Cloud Foundry outage, the filter has some safeguards to protect your pipelines throughput. If the Cloud Foundry endpoint becomes unreachable you can set a timeout period before the CF CLI tries to communicate with the Cloud Foundry endpoint again.
|
21
|
+
|
22
|
+
This filter can be used by any user in the Cloud Foundry environemnt that has the "space developer" role for the applications you want to collect data from. Additionall, this filter supports paralle CF CLI logins. Meaning, if you have multiple Cloud Foundry endpoints, this filter can gracefully handle collecting data from both of them at the time.
|
23
|
+
|
24
|
+
Below is a list of the available config fields
|
25
|
+
- cf_api : The Cloud Foundry API endpoint
|
26
|
+
- cf_user: A valid Cloud Foundry user that has premission to the applications you want data for
|
27
|
+
- cf_password: The users password
|
28
|
+
- cf_org: A valid Cloud Foundry org that a user has premission to (required for a successful login)
|
29
|
+
- cf_space: A valid Cloud Foundry space in the selected org that a user has premission to (required for a successful login)
|
30
|
+
- skip_ssl_validation: A boolean flag to skip ssl validation on login
|
31
|
+
- cache_flush_time: How often you want the job to clean out the cache to run
|
32
|
+
- cace_age_time: A cache items time to live
|
33
|
+
- cf_retry_cli_timeout: After a failed attempt to reach the Cloud Foundry endpoint, how long should the filter wait before using the cf CLI again
|
34
|
+
- cache_invalid_guids: If the Cloud Foundry API receives an invalid guid, cache it so the plugin won't waste resources continuously trying to look it up
|
35
|
+
|
36
|
+
Here are some example configurations:
|
37
|
+
```sh
|
38
|
+
filter{
|
39
|
+
cloudfoundry{
|
40
|
+
cf_api => "https://api.cf-domain.com"
|
41
|
+
cf_user => "username"
|
42
|
+
cf_password => "password"
|
43
|
+
cf_org => "system"
|
44
|
+
cf_space => "apps_manager"
|
45
|
+
skip_ssl_validation => true
|
46
|
+
cache_flush_time => "10m"
|
47
|
+
cache_age_time => 600
|
48
|
+
cf_retry_cli_timeout => 120
|
49
|
+
cache_invalid_guids => false
|
50
|
+
}
|
51
|
+
}
|
52
|
+
```
|
18
53
|
## Developing
|
19
54
|
|
20
55
|
### 1. Plugin Developement and Testing
|
@@ -83,4 +118,4 @@ Programming is not a required skill. Whatever you've seen about open source and
|
|
83
118
|
|
84
119
|
It is more important to the community that you are able to contribute.
|
85
120
|
|
86
|
-
For more information about contributing, see the [CONTRIBUTING](https://github.com/elasticsearch/logstash/blob/master/CONTRIBUTING.md) file.
|
121
|
+
For more information about contributing, see the [CONTRIBUTING](https://github.com/elasticsearch/logstash/blob/master/CONTRIBUTING.md) file.
|
@@ -1,12 +1,12 @@
|
|
1
1
|
Gem::Specification.new do |s|
|
2
2
|
s.name = 'logstash-filter-cloudfoundry'
|
3
|
-
s.version = '0.
|
3
|
+
s.version = '0.4.0'
|
4
4
|
s.licenses = ['Apache License (2.0)']
|
5
5
|
s.summary = "Plugin used to assign meta-data to cloud foundry logs"
|
6
|
-
s.description = "This filter will add App, Space, and Org metadata to CloudFoundry logs. This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
|
6
|
+
s.description = "This filter will add App, Space, and Org metadata to CloudFoundry logs. See github README for more information. This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
|
7
7
|
s.authors = ["Tyler Stigliano"]
|
8
8
|
s.email = 'info@elastic.co'
|
9
|
-
s.homepage = "https://github.com/
|
9
|
+
s.homepage = "https://github.com/ahahtyler/logstash-filter-cloudfoundry"
|
10
10
|
s.require_paths = ["lib"]
|
11
11
|
|
12
12
|
# Files
|
@@ -1,20 +1,207 @@
|
|
1
|
-
|
1
|
+
# encoding: utf-8
|
2
|
+
require "logstash/devutils/rspec/spec_helper"
|
2
3
|
require "logstash/filters/cloudfoundry"
|
3
4
|
|
4
5
|
describe LogStash::Filters::CloudFoundry do
|
5
6
|
|
6
|
-
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
7
|
+
subject(:plugin) { LogStash::Filters::CloudFoundry.new(config) }
|
8
|
+
let(:config) { Hash.new }
|
9
|
+
|
10
|
+
let(:doc) { "" }
|
11
|
+
let(:event) { LogStash::Event.new("message" => doc) }
|
12
|
+
|
13
|
+
#Received invalid config inputs
|
14
|
+
describe "registration" do
|
15
|
+
|
16
|
+
before(:each) do
|
17
|
+
plugin.register
|
18
|
+
end
|
19
|
+
|
20
|
+
describe "got blank required data" do
|
21
|
+
let(:config) do {
|
22
|
+
"cf_api" => "",
|
23
|
+
"cf_user" => "",
|
24
|
+
"cf_password" => "",
|
25
|
+
"cf_org" => "",
|
26
|
+
"cf_space" => ""}
|
27
|
+
end
|
28
|
+
|
29
|
+
it "failed variables are set" do
|
30
|
+
expect(plugin.instance_variable_get(:@cf_logged_in)).to eq(false)
|
31
|
+
expect(plugin.instance_variable_get(:@cf_retry_cli)).to eq(false)
|
32
|
+
end
|
33
|
+
end
|
34
|
+
|
35
|
+
describe "got bad data" do
|
36
|
+
let(:config) do {
|
37
|
+
"cf_api" => "bad data",
|
38
|
+
"cf_user" => "bad data",
|
39
|
+
"cf_password" => "bad data",
|
40
|
+
"cf_org" => "bad data",
|
41
|
+
"cf_space" => "bad data"}
|
42
|
+
end
|
43
|
+
|
44
|
+
it "failed variables are set" do
|
45
|
+
expect(plugin.instance_variable_get(:@cf_logged_in)).to eq(false)
|
46
|
+
expect(plugin.instance_variable_get(:@cf_retry_cli)).to eq(false)
|
47
|
+
end
|
48
|
+
|
49
|
+
it "CF HOME variable was set" do
|
50
|
+
expect(File.directory?(plugin.instance_variable_get(:@cf_path))).to be(true)
|
51
|
+
end
|
52
|
+
end
|
53
|
+
|
54
|
+
describe "got valid data without retry" do
|
55
|
+
let(:config) do {
|
56
|
+
"cf_api" => "https://api.run.pivotal.io",
|
57
|
+
"cf_user" => "username",
|
58
|
+
"cf_password" => "password",
|
59
|
+
"cf_org" => "ITCS",
|
60
|
+
"cf_space" => "development"}
|
61
|
+
end
|
62
|
+
|
63
|
+
it "successful variables are set" do
|
64
|
+
expect(plugin.instance_variable_get(:@cf_logged_in)).to eq(true)
|
65
|
+
expect(plugin.instance_variable_get(:@cf_retry_cli)).to eq(false)
|
66
|
+
end
|
67
|
+
|
68
|
+
it "CF HOME variable was set" do
|
69
|
+
expect(File.directory?(plugin.instance_variable_get(:@cf_path))).to be(true)
|
70
|
+
end
|
71
|
+
end
|
72
|
+
|
73
|
+
describe "got valid data with retry" do
|
74
|
+
let(:config) do {
|
75
|
+
"cf_api" => "https://api.run.pivotal.io",
|
76
|
+
"cf_user" => "username",
|
77
|
+
"cf_password" => "password",
|
78
|
+
"cf_org" => "ITCS",
|
79
|
+
"cf_space" => "development",
|
80
|
+
"cf_retry_cli_timeout" => 10}
|
81
|
+
end
|
82
|
+
|
83
|
+
it "successful variables are set" do
|
84
|
+
expect(plugin.instance_variable_get(:@cf_logged_in)).to eq(true)
|
85
|
+
expect(plugin.instance_variable_get(:@cf_retry_cli)).to eq(true)
|
86
|
+
end
|
87
|
+
|
88
|
+
it "CF HOME variable was set" do
|
89
|
+
expect(File.directory?(plugin.instance_variable_get(:@cf_path))).to be(true)
|
90
|
+
end
|
91
|
+
end
|
92
|
+
|
93
|
+
end
|
94
|
+
|
95
|
+
describe "filtering" do
|
96
|
+
|
97
|
+
before(:each) do
|
98
|
+
plugin.register
|
13
99
|
end
|
14
100
|
|
15
|
-
|
16
|
-
|
17
|
-
|
101
|
+
describe "got valid data" do
|
102
|
+
let(:config) do {
|
103
|
+
"cf_api" => "https://api.run.pivotal.io",
|
104
|
+
"cf_user" => "username",
|
105
|
+
"cf_password" => "password",
|
106
|
+
"cf_org" => "ITCS",
|
107
|
+
"cf_space" => "development"}
|
108
|
+
end
|
109
|
+
|
110
|
+
let(:doc) { "abc loggregator 964db4e9-4f9c-42a2-9296-f8381ce459eb sample log" }
|
111
|
+
|
112
|
+
it "extract all the values" do
|
113
|
+
plugin.filter(event)
|
114
|
+
expect(event.get("orgname")).to eq("ITCS")
|
115
|
+
expect(event.get("spacename")).to eq("development")
|
116
|
+
expect(event.get("appname")).to eq("spring-music")
|
117
|
+
expect(event.get("appguid")).to eq("964db4e9-4f9c-42a2-9296-f8381ce459eb")
|
118
|
+
end
|
119
|
+
|
120
|
+
it "extract all values from cache" do
|
121
|
+
plugin.filter(event)
|
122
|
+
expect(plugin.instance_variable_get(:@app_cache)[event.get("appguid")].nil?).to be(false)
|
123
|
+
end
|
124
|
+
end
|
125
|
+
|
126
|
+
describe "got valid data from different space" do
|
127
|
+
let(:config) do {
|
128
|
+
"cf_api" => "https://api.run.pivotal.io",
|
129
|
+
"cf_user" => "username",
|
130
|
+
"cf_password" => "password",
|
131
|
+
"cf_org" => "ITCS",
|
132
|
+
"cf_space" => "test"}
|
133
|
+
end
|
134
|
+
|
135
|
+
let(:doc) { "abc loggregator 964db4e9-4f9c-42a2-9296-f8381ce459eb sample log" }
|
136
|
+
|
137
|
+
it "extract all the values" do
|
138
|
+
plugin.filter(event)
|
139
|
+
expect(event.get("orgname")).to eq("ITCS")
|
140
|
+
expect(event.get("spacename")).to eq("development")
|
141
|
+
expect(event.get("appname")).to eq("spring-music")
|
142
|
+
expect(event.get("appguid")).to eq("964db4e9-4f9c-42a2-9296-f8381ce459eb")
|
143
|
+
end
|
144
|
+
|
145
|
+
it "extract all values from cache" do
|
146
|
+
plugin.filter(event)
|
147
|
+
expect(plugin.instance_variable_get(:@app_cache)[event.get("appguid")].nil?).to be(false)
|
148
|
+
end
|
18
149
|
end
|
150
|
+
|
151
|
+
describe "got valid data with invalid guid" do
|
152
|
+
let(:config) do {
|
153
|
+
"cf_api" => "https://api.run.pivotal.io",
|
154
|
+
"cf_user" => "username",
|
155
|
+
"cf_password" => "password",
|
156
|
+
"cf_org" => "ITCS",
|
157
|
+
"cf_space" => "test",
|
158
|
+
"cache_invalid_guids" => false}
|
159
|
+
end
|
160
|
+
|
161
|
+
let(:doc) { "abc loggregator aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee sample log" }
|
162
|
+
|
163
|
+
it "make sure cache is empty" do
|
164
|
+
plugin.filter(event)
|
165
|
+
expect(plugin.instance_variable_get(:@app_cache)[event.get("appguid")].nil?).to be(true)
|
166
|
+
end
|
167
|
+
|
168
|
+
end
|
169
|
+
|
170
|
+
describe "got valid data with invalid guid" do
|
171
|
+
let(:config) do {
|
172
|
+
"cf_api" => "https://api.run.pivotal.io",
|
173
|
+
"cf_user" => "username",
|
174
|
+
"cf_password" => "password",
|
175
|
+
"cf_org" => "ITCS",
|
176
|
+
"cf_space" => "test",
|
177
|
+
"cache_invalid_guids" => true}
|
178
|
+
end
|
179
|
+
|
180
|
+
let(:doc) { "abc loggregator aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee sample log" }
|
181
|
+
|
182
|
+
it "make sure cache has value" do
|
183
|
+
plugin.filter(event)
|
184
|
+
expect(plugin.instance_variable_get(:@app_cache)[event.get("appguid")].nil?).to be(false)
|
185
|
+
end
|
186
|
+
end
|
187
|
+
|
188
|
+
describe "got valid data with no guid" do
|
189
|
+
let(:config) do {
|
190
|
+
"cf_api" => "https://api.run.pivotal.io",
|
191
|
+
"cf_user" => "username",
|
192
|
+
"cf_password" => "password",
|
193
|
+
"cf_org" => "ITCS",
|
194
|
+
"cf_space" => "test"}
|
195
|
+
end
|
196
|
+
|
197
|
+
let(:doc) { "sample log" }
|
198
|
+
|
199
|
+
it "make sure flags are set" do
|
200
|
+
plugin.filter(event)
|
201
|
+
expect(event.get("appguid").nil?).to eq(true)
|
202
|
+
end
|
203
|
+
end
|
204
|
+
|
19
205
|
end
|
206
|
+
|
20
207
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: logstash-filter-cloudfoundry
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.4.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Tyler Stigliano
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2016-09-
|
11
|
+
date: 2016-09-28 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: logstash-core
|
@@ -45,9 +45,9 @@ dependencies:
|
|
45
45
|
- !ruby/object:Gem::Version
|
46
46
|
version: '0'
|
47
47
|
description: This filter will add App, Space, and Org metadata to CloudFoundry logs.
|
48
|
-
This gem is a logstash plugin required to
|
49
|
-
pipeline using $LS_HOME/bin/plugin install
|
50
|
-
program
|
48
|
+
See github README for more information. This gem is a logstash plugin required to
|
49
|
+
be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install
|
50
|
+
gemname. This gem is not a stand-alone program
|
51
51
|
email: info@elastic.co
|
52
52
|
executables: []
|
53
53
|
extensions: []
|
@@ -67,7 +67,7 @@ files:
|
|
67
67
|
- lib/logstash/filters/cloudfoundry.rb
|
68
68
|
- logstash-filter-cloudfoundry.gemspec
|
69
69
|
- spec/filters/cloudfoundry_spec.rb
|
70
|
-
homepage: https://github.com/
|
70
|
+
homepage: https://github.com/ahahtyler/logstash-filter-cloudfoundry
|
71
71
|
licenses:
|
72
72
|
- Apache License (2.0)
|
73
73
|
metadata:
|