fluent-plugin-kubernetes_tagged_remote_syslog 0.3.5 → 0.3.6
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.travis.yml +0 -2
- data/CHANGELOG.md +8 -0
- data/LICENSE.txt +1 -1
- data/README.md +66 -2
- data/VERSION +1 -1
- data/fluent-plugin-kubernetes_tagged_remote_syslog.gemspec +3 -1
- data/lib/fluent/plugin/out_kubernetes_tagged_remote_syslog.rb +6 -7
- metadata +5 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 601c5afbbc0db25d4e4f755e3271f474e15705e6
|
4
|
+
data.tar.gz: 02bf9e9589975c2755d5d2fe6aa74c2cc7af2b6d
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 0f4d666b250c8286321f4685d18176f2ad959adebad8bd371940f2279086a7031142c69c10262b66185860093fec711d60cd1591324d2d8ce5f84c4f870abca1
|
7
|
+
data.tar.gz: cda027c1edfbcdd0bfeb5740cd79c24f5bfe691f97f42dc30c305ca9fa4284f13afa1954f3f756869f46d6a5f4737f23c317e22d8f1ff3909734f53b727dd489
|
data/.travis.yml
CHANGED
data/CHANGELOG.md
CHANGED
data/LICENSE.txt
CHANGED
data/README.md
CHANGED
@@ -1,17 +1,50 @@
|
|
1
1
|
# fluent-plugin-kubernetes_tagged_remote_syslog
|
2
2
|
|
3
|
-
<!-- [![Build Status](https://travis-ci.org/
|
3
|
+
<!-- [![Build Status](https://travis-ci.org/kvitajakub/fluent-plugin-kubernetes_tagged_remote_syslog.svg?branch=master)](https://travis-ci.org/kvitajakub/fluent-plugin-kubernetes_tagged_remote_syslog) -->
|
4
4
|
|
5
5
|
[Fluentd](http://fluentd.org) plugin for output to remote syslog service (e.g. [Papertrail](http://papertrailapp.com/)). It is meant to use in Kubernetes environment with [kubernetes_metadata_filter](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter) plugin annotating the messages.
|
6
6
|
|
7
|
+
This plugin was created because regular remote syslog Fluentd plugins doesn't work well with kubernetes annotated messages. `hostname`/`program` fields have not helpful values `localhost`/`fluentd` and actual message is unreadable compact JSON.
|
8
|
+
|
7
9
|
## Installation
|
8
10
|
|
9
11
|
```bash
|
10
12
|
fluent-gem install fluent-plugin-kubernetes_tagged_remote_syslog
|
11
13
|
```
|
12
14
|
|
13
|
-
##
|
15
|
+
## Example input/output
|
16
|
+
Default input should be JSON message annotated with kubernetes info:
|
17
|
+
```
|
18
|
+
{
|
19
|
+
"log": "2015/05/05 19:54:41 log me\n",
|
20
|
+
"stream": "stderr",
|
21
|
+
"docker": {
|
22
|
+
"id": "df14e0d5ae4c07284fa636d739c8fc2e6b52bc344658de7d3f08c36a2e804115",
|
23
|
+
}
|
24
|
+
"kubernetes": {
|
25
|
+
"host": "jimmi-redhat.localnet",
|
26
|
+
"pod_name":"fabric8-console-controller-98rqc",
|
27
|
+
"pod_id": "c76927af-f563-11e4-b32d-54ee7527188d",
|
28
|
+
"container_name": "fabric8-console-container",
|
29
|
+
"namespace_name": "default",
|
30
|
+
"namespace_id": "23437884-8e08-4d95-850b-e94378c9b2fd",
|
31
|
+
"labels": {
|
32
|
+
"component": "fabric8Console"
|
33
|
+
}
|
34
|
+
}
|
35
|
+
}
|
36
|
+
```
|
37
|
+
This will become:
|
38
|
+
```
|
39
|
+
default fabric8-console-controller-98rqc: 2015/05/05 19:54:41 \n
|
40
|
+
```
|
41
|
+
|
42
|
+
And at the message receive Papertrail will add a timestamp in front of the message:
|
43
|
+
```
|
44
|
+
May 05 19:54:45 default fabric8-console-controller-98rqc: 2015/05/05 19:54:41 log me\n
|
45
|
+
```
|
14
46
|
|
47
|
+
## Usage
|
15
48
|
```
|
16
49
|
<match foo>
|
17
50
|
type kubernetes_tagged_remote_syslog
|
@@ -21,6 +54,37 @@
|
|
21
54
|
hostname 'namespace_name'
|
22
55
|
</match>
|
23
56
|
```
|
57
|
+
`hostname` value will look for the key with that name in message and adds it as hostname. If no key is found it will use the value of the key itself.
|
58
|
+
`program` value works in the same way as `hostname`.
|
59
|
+
|
60
|
+
## Dropping messages from namespaces you don't need
|
61
|
+
Parameter `skip_namespaces` will drop messages from selected namespaces. This should be technically in its own filter plugin, but that would require installing and configuring two pluggins so I just dropped it here.
|
62
|
+
|
63
|
+
Default configuration will drop messages from `kube-system` and `deis` namespaces.
|
64
|
+
|
65
|
+
## Special Use Case - Deis Workflow
|
66
|
+
|
67
|
+
I am running this plugin myself as part of [Deis Workflow Fluentd](https://github.com/deis/fluentd) DaemonSet. This requires editing `values.yaml`:
|
68
|
+
```
|
69
|
+
...
|
70
|
+
fluentd:
|
71
|
+
syslog:
|
72
|
+
# Configure the following ONLY if using Fluentd to send log messages to both
|
73
|
+
# the Logger component and external syslog endpoint
|
74
|
+
# external syslog endpoint url
|
75
|
+
host: ""
|
76
|
+
# external syslog endpoint port
|
77
|
+
port: ""
|
78
|
+
daemon_environment:
|
79
|
+
FLUENTD_PLUGIN_1: "fluent-plugin-kubernetes_tagged_remote_syslog"
|
80
|
+
CUSTOM_STORE_1: "<store> \n
|
81
|
+
@type kubernetes_tagged_remote_syslog \n
|
82
|
+
host logs5.papertrailapp.com \n
|
83
|
+
port REDACTED \n
|
84
|
+
</store>"
|
85
|
+
...
|
86
|
+
```
|
87
|
+
Plugin will be installed with `fluent-gem` from [rubygems.com](https://rubygems.org/).
|
24
88
|
|
25
89
|
## License
|
26
90
|
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.3.
|
1
|
+
0.3.6
|
@@ -7,11 +7,13 @@ Gem::Specification.new do |spec|
|
|
7
7
|
spec.version = File.read("VERSION").strip
|
8
8
|
spec.authors = ["Richard Lee, Jakub Kvita"]
|
9
9
|
spec.email = ["kvitajakub@gmail.com"]
|
10
|
-
spec.summary = %q{Fluentd output plugin for remote syslog
|
10
|
+
spec.summary = %q{Fluentd output plugin for remote syslog. This is meant for processing kubernetes annotated messages.}
|
11
11
|
spec.description = spec.description
|
12
12
|
spec.homepage = "https://github.com/kvitajakub/fluent-plugin-kubernetes_tagged_remote_syslog"
|
13
13
|
spec.license = "MIT"
|
14
14
|
|
15
|
+
spec.required_ruby_version = '>= 2.3.0'
|
16
|
+
|
15
17
|
spec.files = `git ls-files -z`.split("\x0")
|
16
18
|
spec.test_files = spec.files.grep(%r{^(test|spec|features)/})
|
17
19
|
spec.require_paths = ["lib"]
|
@@ -23,25 +23,24 @@ module Fluent
|
|
23
23
|
end
|
24
24
|
|
25
25
|
def emit(tag, es, chain)
|
26
|
-
es.each do |
|
27
|
-
record.each_pair do |
|
26
|
+
es.each do |_time, record|
|
27
|
+
record.each_pair do |_k, v|
|
28
28
|
if v.is_a?(String)
|
29
29
|
v.force_encoding('utf-8')
|
30
30
|
end
|
31
31
|
end
|
32
32
|
|
33
|
-
next if
|
34
|
-
skip_namespaces.include?(record['kubernetes']['namespace_name'])
|
33
|
+
next if skip_namespaces.include?(record.dig('kubernetes', 'namespace_name'))
|
35
34
|
|
36
35
|
@loggers[tag] ||= RemoteSyslogLogger::UdpSender.new(
|
37
36
|
@host,
|
38
37
|
@port,
|
39
38
|
facility: record['facility'] || @facility,
|
40
39
|
severity: record['severity'] || @severity,
|
41
|
-
program: (record
|
42
|
-
local_hostname: (record
|
40
|
+
program: (record.dig('kubernetes', @program) || @program)[0...31],
|
41
|
+
local_hostname: (record.dig('kubernetes', @hostname) || @hostname)[0...31])
|
43
42
|
|
44
|
-
@loggers[tag].transmit record['log'].to_s
|
43
|
+
@loggers[tag].transmit((if record.key?('log') then record['log'] else record end).to_s)
|
45
44
|
end
|
46
45
|
chain.next
|
47
46
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-kubernetes_tagged_remote_syslog
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.3.
|
4
|
+
version: 0.3.6
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Richard Lee, Jakub Kvita
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2017-07-
|
11
|
+
date: 2017-07-12 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: rake
|
@@ -111,7 +111,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
111
111
|
requirements:
|
112
112
|
- - ">="
|
113
113
|
- !ruby/object:Gem::Version
|
114
|
-
version:
|
114
|
+
version: 2.3.0
|
115
115
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
116
116
|
requirements:
|
117
117
|
- - ">="
|
@@ -122,7 +122,8 @@ rubyforge_project:
|
|
122
122
|
rubygems_version: 2.5.1
|
123
123
|
signing_key:
|
124
124
|
specification_version: 4
|
125
|
-
summary: Fluentd output plugin for remote syslog
|
125
|
+
summary: Fluentd output plugin for remote syslog. This is meant for processing kubernetes
|
126
|
+
annotated messages.
|
126
127
|
test_files:
|
127
128
|
- test/plugin/out_kubernetes_tagged_remote_syslog.rb
|
128
129
|
- test/test_helper.rb
|