fluent-plugin-papertrail 0.2.4 → 0.2.5.pre.dev
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/Makefile +0 -4
- data/README.md +5 -36
- data/fluent-plugin-papertrail.gemspec +1 -1
- data/lib/fluent/plugin/out_papertrail.rb +12 -4
- metadata +4 -10
- data/docker/Dockerfile +0 -45
- data/docker/conf/fluent.conf +0 -18
- data/docker/conf/kubernetes.conf +0 -171
- data/docker/conf/systemd.conf +0 -61
- data/docker/plugins/parser_kubernetes.rb +0 -65
- data/kubernetes/fluentd-daemonset-papertrail.yaml +0 -85
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 297b4941b70100a4be1be6fb5e35f14263553202
|
4
|
+
data.tar.gz: 80fccb4690f183f69aabc9da7a793fbbe357e19d
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 4ed8cae9ae77236c3c64d3f4537e2a5274a67050bcf5478a5d60a90a0ac9cc795cb54c79936eb5ec0a6459c1825a8ff5fe2981e1ac9fbd1f482b1f7f9545ccea
|
7
|
+
data.tar.gz: 9a575ad74e1759185008d11196d712f010f954ba9467ea2829668c461b712790fc99cc4c7f0d439eb23a5f18f2cfff96e7bb3ea24c5db8444bfd12160e223a4f
|
data/Makefile
CHANGED
@@ -10,7 +10,3 @@ release: bundle
|
|
10
10
|
rm -rf ${REPO_NAME}-*.gem
|
11
11
|
bundle exec gem build ${REPO_NAME}.gemspec
|
12
12
|
bundle exec gem push ${REPO_NAME}-*.gem
|
13
|
-
|
14
|
-
release-docker:
|
15
|
-
cd docker; docker build -t quay.io/solarwinds/fluentd-kubernetes:$(TAG) .
|
16
|
-
docker push quay.io/solarwinds/fluentd-kubernetes:$(TAG)
|
data/README.md
CHANGED
@@ -1,10 +1,10 @@
|
|
1
1
|
# Fluent::Plugin::Papertrail
|
2
2
|
|
3
|
-
[![Gem Version](https://badge.fury.io/rb/fluent-plugin-papertrail.svg)](https://badge.fury.io/rb/fluent-plugin-papertrail) [![
|
3
|
+
[![Gem Version](https://badge.fury.io/rb/fluent-plugin-papertrail.svg)](https://badge.fury.io/rb/fluent-plugin-papertrail) [![CircleCI](https://circleci.com/gh/solarwinds/fluent-plugin-papertrail/tree/master.svg?style=shield)](https://circleci.com/gh/solarwinds/fluent-plugin-papertrail/tree/master)
|
4
4
|
|
5
5
|
## Description
|
6
6
|
|
7
|
-
This repository contains the Fluentd Papertrail Output Plugin
|
7
|
+
This repository contains the Fluentd Papertrail Output Plugin.
|
8
8
|
|
9
9
|
## Installation
|
10
10
|
|
@@ -42,7 +42,7 @@ This plugin expects the following fields to be set for each Fluent record:
|
|
42
42
|
hostname The source hostname for papertrail logging
|
43
43
|
```
|
44
44
|
|
45
|
-
The following example is a `record_transformer` filter, from the
|
45
|
+
The following example is a `record_transformer` filter, from the Kubernetes assets [in the Solarwinds fluentd-deployment repo](https://github.com/solarwinds/fluentd-deployment/blob/master/docker/conf/kubernetes.conf), that is used along with the [fluent-plugin-kubernetes_metadata_filter](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter) to populate the required fields for our plugin:
|
46
46
|
```yaml
|
47
47
|
<filter kubernetes.**>
|
48
48
|
type kubernetes_metadata
|
@@ -81,21 +81,9 @@ If you want to change any of these parameters simply add them to a match stanza.
|
|
81
81
|
</match>
|
82
82
|
```
|
83
83
|
|
84
|
-
## Kubernetes
|
84
|
+
## Kubernetes Annotations
|
85
85
|
|
86
|
-
|
87
|
-
|
88
|
-
To deploy this plugin as a DaemonSet to your Kubernetes cluster, just adjust the `FLUENT_*` environment variables in `kubernetes/fluentd-daemonset-papertrail.yaml` and push it to your cluster with:
|
89
|
-
|
90
|
-
```
|
91
|
-
kubectl apply -f kubernetes/fluentd-daemonset-papertrail.yaml
|
92
|
-
```
|
93
|
-
|
94
|
-
The Dockerfile that generates [the image used in this DaemonSet](https://quay.io/repository/solarwinds/fluentd-kubernetes), can be found at `docker/Dockerfile`.
|
95
|
-
|
96
|
-
### Annotations
|
97
|
-
|
98
|
-
You can redirect logs to alternate Papertrail destinations by adding annotations to your Pods or Namespaces:
|
86
|
+
If you're running this plugin in Kubernetes with the kubernetes_metadata_filter plugin enabled you can redirect logs to alternate Papertrail destinations by adding annotations to your Pods or Namespaces:
|
99
87
|
|
100
88
|
```
|
101
89
|
solarwinds.io/papertrail_host: 'logs0.papertrailapp.com'
|
@@ -104,21 +92,6 @@ solarwinds.io/papertrail_port: '12345'
|
|
104
92
|
|
105
93
|
If both the Pod and Namespace have annotations for any running Pod, the Pod's annotation is used.
|
106
94
|
|
107
|
-
### Audit Logs
|
108
|
-
|
109
|
-
If you'd like to redirect Kubernetes API Server Audit logs to a seperate Papertrail destination, add the following to your `fluent.conf`:
|
110
|
-
```
|
111
|
-
<match kube-apiserver-audit>
|
112
|
-
type papertrail
|
113
|
-
num_threads 4
|
114
|
-
|
115
|
-
papertrail_host "#{ENV['FLUENT_PAPERTRAIL_AUDIT_HOST']}"
|
116
|
-
papertrail_port "#{ENV['FLUENT_PAPERTRAIL_AUDIT_PORT']}"
|
117
|
-
</match>
|
118
|
-
```
|
119
|
-
|
120
|
-
This requires you to configure an [audit policy file](https://kubernetes.io/docs/tasks/debug-application-cluster/audit/) on your cluster.
|
121
|
-
|
122
95
|
## Development
|
123
96
|
|
124
97
|
This plugin is targeting Ruby 2.4 and Fluentd v1.0, although it should work with older versions of both.
|
@@ -136,10 +109,6 @@ To release a new version, update the version number in the [GemSpec](fluent-plug
|
|
136
109
|
|
137
110
|
`make release`
|
138
111
|
|
139
|
-
### Release in [Quay.io](https://quay.io/repository/solarwinds/fluentd-kubernetes)
|
140
|
-
|
141
|
-
`make release-docker TAG=$(VERSION)`
|
142
|
-
|
143
112
|
## Contributing
|
144
113
|
|
145
114
|
Bug reports and pull requests are welcome on GitHub at: https://github.com/solarwinds/fluent-plugin-papertrail
|
@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
|
4
4
|
|
5
5
|
Gem::Specification.new do |spec|
|
6
6
|
spec.name = "fluent-plugin-papertrail"
|
7
|
-
spec.version = "0.2.
|
7
|
+
spec.version = "0.2.5-dev"
|
8
8
|
spec.authors = ["Jonathan Lozinski", "Alex Ouzounis", "Chris Rust"]
|
9
9
|
spec.email = ["jonathan.lozinski@solarwinds.com", "alex.ouzounis@solarwinds.com", "chris.rust@solarwinds.com"]
|
10
10
|
|
@@ -12,10 +12,14 @@ module Fluent
|
|
12
12
|
config_param :default_hostname, :string, default: 'unidentified'
|
13
13
|
# overriding default flush_interval (60 sec) from Fluent::BufferedOutput
|
14
14
|
config_param :flush_interval, :time, default: 1
|
15
|
+
config_param :discard_unannotated_pod_logs, :bool, default: false
|
15
16
|
|
16
17
|
# register as 'papertrail' fluent plugin
|
17
18
|
Fluent::Plugin.register_output('papertrail', self)
|
18
19
|
|
20
|
+
# declare const string for nullifying token if we decide to discard records
|
21
|
+
DISCARD_STRING = 'DISCARD'
|
22
|
+
|
19
23
|
def configure(conf)
|
20
24
|
super
|
21
25
|
# create initial sockets hash and socket based on config param
|
@@ -33,8 +37,10 @@ module Fluent
|
|
33
37
|
def write(chunk)
|
34
38
|
chunk.msgpack_each {|(tag, time, record)|
|
35
39
|
socket_key = pick_socket(record)
|
36
|
-
|
37
|
-
|
40
|
+
unless socket_key.eql? "#{DISCARD_STRING}:#{DISCARD_STRING}"
|
41
|
+
packet = create_packet(tag, time, record)
|
42
|
+
send_to_papertrail(packet, socket_key)
|
43
|
+
end
|
38
44
|
}
|
39
45
|
end
|
40
46
|
|
@@ -78,6 +84,10 @@ module Fluent
|
|
78
84
|
record.dig('kubernetes', 'namespace_annotations', 'solarwinds_io/papertrail_port')
|
79
85
|
host = record['kubernetes']['namespace_annotations']['solarwinds_io/papertrail_host']
|
80
86
|
port = record['kubernetes']['namespace_annotations']['solarwinds_io/papertrail_port']
|
87
|
+
# else if it is a kubernetes log and we're discarding unannotated logs
|
88
|
+
elsif record.dig('kubernetes') && @discard_unannotated_pod_logs
|
89
|
+
host = DISCARD_STRING
|
90
|
+
port = DISCARD_STRING
|
81
91
|
# else use pre-configured destination
|
82
92
|
else
|
83
93
|
host = @papertrail_host
|
@@ -92,7 +102,6 @@ module Fluent
|
|
92
102
|
def send_to_papertrail(packet, socket_key)
|
93
103
|
if @sockets[socket_key].nil?
|
94
104
|
err_msg = "Unable to create socket with #{socket_key}"
|
95
|
-
log.error err_msg
|
96
105
|
raise SocketFailureError, err_msg
|
97
106
|
else
|
98
107
|
begin
|
@@ -100,7 +109,6 @@ module Fluent
|
|
100
109
|
@sockets[socket_key].puts packet.assemble
|
101
110
|
rescue => e
|
102
111
|
err_msg = "Error writing to #{socket_key}: #{e}"
|
103
|
-
log.error err_msg
|
104
112
|
# socket failed, reset to nil to recreate for the next write
|
105
113
|
@sockets[socket_key] = nil
|
106
114
|
raise SocketFailureError, err_msg, e.backtrace
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-papertrail
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.2.
|
4
|
+
version: 0.2.5.pre.dev
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Jonathan Lozinski
|
@@ -10,7 +10,7 @@ authors:
|
|
10
10
|
autorequire:
|
11
11
|
bindir: exe
|
12
12
|
cert_chain: []
|
13
|
-
date: 2018-
|
13
|
+
date: 2018-07-13 00:00:00.000000000 Z
|
14
14
|
dependencies:
|
15
15
|
- !ruby/object:Gem::Dependency
|
16
16
|
name: fluentd
|
@@ -132,13 +132,7 @@ files:
|
|
132
132
|
- Makefile
|
133
133
|
- README.md
|
134
134
|
- Rakefile
|
135
|
-
- docker/Dockerfile
|
136
|
-
- docker/conf/fluent.conf
|
137
|
-
- docker/conf/kubernetes.conf
|
138
|
-
- docker/conf/systemd.conf
|
139
|
-
- docker/plugins/parser_kubernetes.rb
|
140
135
|
- fluent-plugin-papertrail.gemspec
|
141
|
-
- kubernetes/fluentd-daemonset-papertrail.yaml
|
142
136
|
- lib/fluent/plugin/out_papertrail.rb
|
143
137
|
homepage: https://github.com/solarwinds/fluent-plugin-papertrail
|
144
138
|
licenses:
|
@@ -155,9 +149,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
155
149
|
version: '0'
|
156
150
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
157
151
|
requirements:
|
158
|
-
- - "
|
152
|
+
- - ">"
|
159
153
|
- !ruby/object:Gem::Version
|
160
|
-
version:
|
154
|
+
version: 1.3.1
|
161
155
|
requirements: []
|
162
156
|
rubyforge_project:
|
163
157
|
rubygems_version: 2.5.2.2
|
data/docker/Dockerfile
DELETED
@@ -1,45 +0,0 @@
|
|
1
|
-
FROM fluent/fluentd:v1.1.1-debian
|
2
|
-
|
3
|
-
USER root
|
4
|
-
WORKDIR /home/fluent
|
5
|
-
ENV PATH /home/fluent/.gem/ruby/2.3.0/bin:$PATH
|
6
|
-
ENV BUILDDEPS="sudo make gcc g++ libc-dev ruby-dev libffi-dev"
|
7
|
-
|
8
|
-
RUN apt-get update \
|
9
|
-
&& apt-get upgrade -y \
|
10
|
-
&& apt-get install -y --no-install-recommends \
|
11
|
-
$BUILDDEPS
|
12
|
-
|
13
|
-
RUN echo 'gem: --no-document' >> /etc/gemrc \
|
14
|
-
&& gem install fluent-plugin-systemd -v 0.3.1 \
|
15
|
-
&& gem install fluent-plugin-rewrite-tag-filter -v 2.0.2 \
|
16
|
-
&& gem install fluent-plugin-papertrail -v 0.2.3 \
|
17
|
-
&& gem install fluent-plugin-loggly-anno -v 0.0.2 \
|
18
|
-
&& gem install fluent-plugin-kubernetes_metadata_input -v 0.21.11 \
|
19
|
-
&& gem install fluent-plugin-kubernetes_metadata_filter -v 2.0.0
|
20
|
-
|
21
|
-
RUN SUDO_FORCE_REMOVE=yes \
|
22
|
-
apt-get purge -y --auto-remove \
|
23
|
-
-o APT::AutoRemove::RecommendsImportant=false \
|
24
|
-
$BUILDDEPS \
|
25
|
-
&& rm -rf /var/lib/apt/lists/* \
|
26
|
-
&& gem sources --clear-all \
|
27
|
-
&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
|
28
|
-
|
29
|
-
# Copy configuration files
|
30
|
-
COPY ./conf/fluent.conf /fluentd/etc/
|
31
|
-
COPY ./conf/systemd.conf /fluentd/etc/
|
32
|
-
COPY ./conf/kubernetes.conf /fluentd/etc/
|
33
|
-
|
34
|
-
# Copy plugins
|
35
|
-
COPY plugins /fluentd/plugins/
|
36
|
-
|
37
|
-
# Environment variables
|
38
|
-
ENV FLUENTD_OPT=""
|
39
|
-
ENV FLUENTD_CONF="fluent.conf"
|
40
|
-
|
41
|
-
# Overriding entrypoint, otherwise parent image causes to run as user fluent
|
42
|
-
ENTRYPOINT []
|
43
|
-
|
44
|
-
# Run Fluentd
|
45
|
-
CMD exec fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT
|
data/docker/conf/fluent.conf
DELETED
@@ -1,18 +0,0 @@
|
|
1
|
-
@include systemd.conf
|
2
|
-
@include kubernetes.conf
|
3
|
-
|
4
|
-
## Capture audit logs
|
5
|
-
#<match kube-apiserver-audit>
|
6
|
-
# @type papertrail
|
7
|
-
#
|
8
|
-
# papertrail_host "#{ENV['FLUENT_PAPERTRAIL_AUDIT_HOST']}"
|
9
|
-
# papertrail_port "#{ENV['FLUENT_PAPERTRAIL_AUDIT_PORT']}"
|
10
|
-
#</match>
|
11
|
-
|
12
|
-
<match **>
|
13
|
-
@type papertrail
|
14
|
-
|
15
|
-
papertrail_host "#{ENV['FLUENT_PAPERTRAIL_HOST']}"
|
16
|
-
papertrail_port "#{ENV['FLUENT_PAPERTRAIL_PORT']}"
|
17
|
-
|
18
|
-
</match>
|
data/docker/conf/kubernetes.conf
DELETED
@@ -1,171 +0,0 @@
|
|
1
|
-
<match fluent.**>
|
2
|
-
@type null
|
3
|
-
</match>
|
4
|
-
|
5
|
-
<source>
|
6
|
-
@type tail
|
7
|
-
path /var/log/containers/*.log
|
8
|
-
pos_file /var/log/fluentd-containers.log.pos
|
9
|
-
time_format %Y-%m-%dT%H:%M:%S.%NZ
|
10
|
-
tag kubernetes.*
|
11
|
-
format json
|
12
|
-
read_from_head true
|
13
|
-
</source>
|
14
|
-
|
15
|
-
<source>
|
16
|
-
@type tail
|
17
|
-
format /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
|
18
|
-
time_format %Y-%m-%d %H:%M:%S
|
19
|
-
path /var/log/salt/minion
|
20
|
-
pos_file /var/log/fluentd-salt.pos
|
21
|
-
tag salt
|
22
|
-
</source>
|
23
|
-
|
24
|
-
<source>
|
25
|
-
@type tail
|
26
|
-
format syslog
|
27
|
-
path /var/log/startupscript.log
|
28
|
-
pos_file /var/log/fluentd-startupscript.log.pos
|
29
|
-
tag startupscript
|
30
|
-
</source>
|
31
|
-
|
32
|
-
<source>
|
33
|
-
@type tail
|
34
|
-
format /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
|
35
|
-
path /var/log/docker.log
|
36
|
-
pos_file /var/log/fluentd-docker.log.pos
|
37
|
-
tag docker
|
38
|
-
</source>
|
39
|
-
|
40
|
-
<source>
|
41
|
-
@type tail
|
42
|
-
format none
|
43
|
-
path /var/log/etcd.log
|
44
|
-
pos_file /var/log/fluentd-etcd.log.pos
|
45
|
-
tag etcd
|
46
|
-
</source>
|
47
|
-
|
48
|
-
<source>
|
49
|
-
@type tail
|
50
|
-
format kubernetes
|
51
|
-
multiline_flush_interval 5s
|
52
|
-
path /var/log/kubelet.log
|
53
|
-
pos_file /var/log/fluentd-kubelet.log.pos
|
54
|
-
tag kubelet
|
55
|
-
</source>
|
56
|
-
|
57
|
-
<source>
|
58
|
-
@type tail
|
59
|
-
format kubernetes
|
60
|
-
multiline_flush_interval 5s
|
61
|
-
path /var/log/kube-proxy.log
|
62
|
-
pos_file /var/log/fluentd-kube-proxy.log.pos
|
63
|
-
tag kube-proxy
|
64
|
-
</source>
|
65
|
-
|
66
|
-
<source>
|
67
|
-
@type tail
|
68
|
-
format kubernetes
|
69
|
-
multiline_flush_interval 5s
|
70
|
-
path /var/log/kube-apiserver.log
|
71
|
-
pos_file /var/log/fluentd-kube-apiserver.log.pos
|
72
|
-
tag kube-apiserver
|
73
|
-
</source>
|
74
|
-
|
75
|
-
<source>
|
76
|
-
@type tail
|
77
|
-
format kubernetes
|
78
|
-
multiline_flush_interval 5s
|
79
|
-
path /var/log/kube-controller-manager.log
|
80
|
-
pos_file /var/log/fluentd-kube-controller-manager.log.pos
|
81
|
-
tag kube-controller-manager
|
82
|
-
</source>
|
83
|
-
|
84
|
-
<source>
|
85
|
-
@type tail
|
86
|
-
format kubernetes
|
87
|
-
multiline_flush_interval 5s
|
88
|
-
path /var/log/kube-scheduler.log
|
89
|
-
pos_file /var/log/fluentd-kube-scheduler.log.pos
|
90
|
-
tag kube-scheduler
|
91
|
-
</source>
|
92
|
-
|
93
|
-
<source>
|
94
|
-
@type tail
|
95
|
-
format kubernetes
|
96
|
-
multiline_flush_interval 5s
|
97
|
-
path /var/log/rescheduler.log
|
98
|
-
pos_file /var/log/fluentd-rescheduler.log.pos
|
99
|
-
tag rescheduler
|
100
|
-
</source>
|
101
|
-
|
102
|
-
<source>
|
103
|
-
@type tail
|
104
|
-
format kubernetes
|
105
|
-
multiline_flush_interval 5s
|
106
|
-
path /var/log/glbc.log
|
107
|
-
pos_file /var/log/fluentd-glbc.log.pos
|
108
|
-
tag glbc
|
109
|
-
</source>
|
110
|
-
|
111
|
-
<source>
|
112
|
-
@type tail
|
113
|
-
format kubernetes
|
114
|
-
multiline_flush_interval 5s
|
115
|
-
path /var/log/cluster-autoscaler.log
|
116
|
-
pos_file /var/log/fluentd-cluster-autoscaler.log.pos
|
117
|
-
tag cluster-autoscaler
|
118
|
-
</source>
|
119
|
-
|
120
|
-
# Example:
|
121
|
-
# 2017-02-09T00:15:57.992775796Z AUDIT: id="90c73c7c-97d6-4b65-9461-f94606ff825f" ip="104.132.1.72" method="GET" user="kubecfg" as="<self>" asgroups="<lookup>" namespace="default" uri="/api/v1/namespaces/default/pods"
|
122
|
-
# 2017-02-09T00:15:57.993528822Z AUDIT: id="90c73c7c-97d6-4b65-9461-f94606ff825f" response="200"
|
123
|
-
<source>
|
124
|
-
@type tail
|
125
|
-
format multiline
|
126
|
-
multiline_flush_interval 5s
|
127
|
-
format_firstline /^\S+\s+AUDIT:/
|
128
|
-
# Fields must be explicitly captured by name to be parsed into the record.
|
129
|
-
# Fields may not always be present, and order may change, so this just looks
|
130
|
-
# for a list of key="\"quoted\" value" pairs separated by spaces.
|
131
|
-
# Unknown fields are ignored.
|
132
|
-
# Note: We can't separate query/response lines as format1/format2 because
|
133
|
-
# they don't always come one after the other for a given query.
|
134
|
-
format1 /^(?<time>\S+) AUDIT:(?: (?:id="(?<id>(?:[^"\\]|\\.)*)"|ip="(?<ip>(?:[^"\\]|\\.)*)"|method="(?<method>(?:[^"\\]|\\.)*)"|user="(?<user>(?:[^"\\]|\\.)*)"|groups="(?<groups>(?:[^"\\]|\\.)*)"|as="(?<as>(?:[^"\\]|\\.)*)"|asgroups="(?<asgroups>(?:[^"\\]|\\.)*)"|namespace="(?<namespace>(?:[^"\\]|\\.)*)"|uri="(?<uri>(?:[^"\\]|\\.)*)"|response="(?<response>(?:[^"\\]|\\.)*)"|\w+="(?:[^"\\]|\\.)*"))*/
|
135
|
-
time_format %FT%T.%L%Z
|
136
|
-
path /var/log/kubernetes/kube-apiserver-audit.log
|
137
|
-
pos_file /var/log/kube-apiserver-audit.log.pos
|
138
|
-
tag kube-apiserver-audit
|
139
|
-
</source>
|
140
|
-
|
141
|
-
<filter kubernetes.**>
|
142
|
-
@type kubernetes_metadata
|
143
|
-
annotation_match ["solarwinds.io/*"]
|
144
|
-
</filter>
|
145
|
-
|
146
|
-
<filter kube-apiserver-audit>
|
147
|
-
@type record_transformer
|
148
|
-
enable_ruby true
|
149
|
-
<record>
|
150
|
-
hostname #{ENV['FLUENT_HOSTNAME']}
|
151
|
-
program kube-apiserver-audit
|
152
|
-
severity info
|
153
|
-
facility local0
|
154
|
-
message ${record}
|
155
|
-
</record>
|
156
|
-
</filter>
|
157
|
-
|
158
|
-
# append namespace and pod name to hostname, so that logs in Papertrail are filterable by each
|
159
|
-
# use container name as program name, but trim it to 32 characters to match remote_syslog spec
|
160
|
-
<filter kubernetes.**>
|
161
|
-
@type record_transformer
|
162
|
-
enable_ruby true
|
163
|
-
<record>
|
164
|
-
hostname #{ENV['FLUENT_HOSTNAME']}-${record["kubernetes"]["namespace_name"]}-${record["kubernetes"]["pod_name"]}
|
165
|
-
program ${record["kubernetes"]["container_name"][0..31]}
|
166
|
-
severity info
|
167
|
-
facility local0
|
168
|
-
message ${record['log']}
|
169
|
-
</record>
|
170
|
-
remove_keys ["log"]
|
171
|
-
</filter>
|
data/docker/conf/systemd.conf
DELETED
@@ -1,61 +0,0 @@
|
|
1
|
-
<source>
|
2
|
-
@type systemd
|
3
|
-
tag systemd
|
4
|
-
read_from_head true
|
5
|
-
<storage>
|
6
|
-
@type local
|
7
|
-
persistent true
|
8
|
-
path /var/log/fluentd-journald-systemd.pos
|
9
|
-
</storage>
|
10
|
-
</source>
|
11
|
-
|
12
|
-
# rewrite tags as systemd.* for the specefic SYSTEMD_UNIT, then we can filter specifically on kubelet and docker below
|
13
|
-
<match systemd>
|
14
|
-
@type rewrite_tag_filter
|
15
|
-
<rule>
|
16
|
-
key SYSTEMD_UNIT
|
17
|
-
pattern /^kubelet.service$/
|
18
|
-
tag systemd.kubelet
|
19
|
-
</rule>
|
20
|
-
<rule>
|
21
|
-
key SYSTEMD_UNIT
|
22
|
-
pattern /^docker.service$/
|
23
|
-
tag systemd.docker
|
24
|
-
</rule>
|
25
|
-
</match>
|
26
|
-
|
27
|
-
# toss all other systemd logs in the bin
|
28
|
-
<match systemd>
|
29
|
-
@type null
|
30
|
-
</match>
|
31
|
-
|
32
|
-
# transform systemd logs to Papertrail format
|
33
|
-
<filter systemd.kubelet>
|
34
|
-
@type record_transformer
|
35
|
-
enable_ruby true
|
36
|
-
<record>
|
37
|
-
hostname "#{ENV['FLUENT_HOSTNAME']}-${record['HOSTNAME']}"
|
38
|
-
program kubelet
|
39
|
-
severity info
|
40
|
-
facility local0
|
41
|
-
message ${record['MESSAGE']}
|
42
|
-
</record>
|
43
|
-
</filter>
|
44
|
-
|
45
|
-
<filter systemd.docker>
|
46
|
-
@type parser
|
47
|
-
format /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
|
48
|
-
reserve_data true
|
49
|
-
key_name MESSAGE
|
50
|
-
</filter>
|
51
|
-
|
52
|
-
<filter systemd.docker>
|
53
|
-
@type record_transformer
|
54
|
-
enable_ruby true
|
55
|
-
<record>
|
56
|
-
hostname "#{ENV['FLUENT_HOSTNAME']}-${record['HOSTNAME']}"
|
57
|
-
program docker
|
58
|
-
severity info
|
59
|
-
facility local0
|
60
|
-
</record>
|
61
|
-
</filter>
|
@@ -1,65 +0,0 @@
|
|
1
|
-
#
|
2
|
-
# Fluentd
|
3
|
-
#
|
4
|
-
# Licensed under the Apache License, Version 2.0 (the "License");
|
5
|
-
# you may not use this file except in compliance with the License.
|
6
|
-
# You may obtain a copy of the License at
|
7
|
-
#
|
8
|
-
# http://www.apache.org/licenses/LICENSE-2.0
|
9
|
-
#
|
10
|
-
# Unless required by applicable law or agreed to in writing, software
|
11
|
-
# distributed under the License is distributed on an "AS IS" BASIS,
|
12
|
-
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
13
|
-
# See the License for the specific language governing permissions and
|
14
|
-
# limitations under the License.
|
15
|
-
#
|
16
|
-
|
17
|
-
# The following Fluentd parser plugin, aims to simplify the parsing of multiline
|
18
|
-
# logs found in Kubernetes nodes. Since many log files shared the same format and
|
19
|
-
# in order to simplify the configuration, this plugin provides a 'kubernetes' format
|
20
|
-
# parser (built on top of MultilineParser).
|
21
|
-
#
|
22
|
-
# When tailing files, this 'kubernetes' format should be applied to the following
|
23
|
-
# log file sources:
|
24
|
-
#
|
25
|
-
# - /var/log/kubelet.log
|
26
|
-
# - /var/log/kube-proxy.log
|
27
|
-
# - /var/log/kube-apiserver.log
|
28
|
-
# - /var/log/kube-controller-manager.log
|
29
|
-
# - /var/log/kube-scheduler.log
|
30
|
-
# - /var/log/rescheduler.log
|
31
|
-
# - /var/log/glbc.log
|
32
|
-
# - /var/log/cluster-autoscaler.log
|
33
|
-
#
|
34
|
-
# Usage:
|
35
|
-
#
|
36
|
-
# ---- fluentd.conf ----
|
37
|
-
#
|
38
|
-
# <source>
|
39
|
-
# type tail
|
40
|
-
# format kubernetes
|
41
|
-
# path ./kubelet.log
|
42
|
-
# read_from_head yes
|
43
|
-
# tag kubelet
|
44
|
-
# </source>
|
45
|
-
#
|
46
|
-
# ---- EOF ---
|
47
|
-
|
48
|
-
require 'fluent/parser'
|
49
|
-
|
50
|
-
module Fluent
|
51
|
-
class KubernetesParser < Fluent::TextParser::MultilineParser
|
52
|
-
Fluent::Plugin.register_parser("kubernetes", self)
|
53
|
-
|
54
|
-
CONF_FORMAT_FIRSTLINE = %q{/^\w\d{4}/}
|
55
|
-
CONF_FORMAT1 = %q{/^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/}
|
56
|
-
CONF_TIME_FORMAT = "%m%d %H:%M:%S.%N"
|
57
|
-
|
58
|
-
def configure(conf)
|
59
|
-
conf['format_firstline'] = CONF_FORMAT_FIRSTLINE
|
60
|
-
conf['format1'] = CONF_FORMAT1
|
61
|
-
conf['time_format'] = CONF_TIME_FORMAT
|
62
|
-
super
|
63
|
-
end
|
64
|
-
end
|
65
|
-
end
|
@@ -1,85 +0,0 @@
|
|
1
|
-
apiVersion: extensions/v1beta1
|
2
|
-
kind: DaemonSet
|
3
|
-
metadata:
|
4
|
-
name: fluentd-papertrail-global-logging
|
5
|
-
namespace: kube-system
|
6
|
-
labels:
|
7
|
-
k8s-app: fluentd-logging
|
8
|
-
version: v1
|
9
|
-
kubernetes.io/cluster-service: "true"
|
10
|
-
spec:
|
11
|
-
updateStrategy:
|
12
|
-
type: RollingUpdate
|
13
|
-
template:
|
14
|
-
metadata:
|
15
|
-
labels:
|
16
|
-
k8s-app: fluentd-logging
|
17
|
-
version: v1
|
18
|
-
kubernetes.io/cluster-service: "true"
|
19
|
-
spec:
|
20
|
-
tolerations:
|
21
|
-
- key: node-role.kubernetes.io/master
|
22
|
-
effect: NoSchedule
|
23
|
-
containers:
|
24
|
-
- name: fluentd
|
25
|
-
image: quay.io/solarwinds/fluentd-kubernetes:v1.1.1-debian-papertrail-0.2.3
|
26
|
-
imagePullPolicy: Always
|
27
|
-
env:
|
28
|
-
- name: FLUENT_PAPERTRAIL_HOST
|
29
|
-
value: 'logsN.papertrailapp.com'
|
30
|
-
- name: FLUENT_PAPERTRAIL_PORT
|
31
|
-
value: 'NNNNN'
|
32
|
-
- name: FLUENT_HOSTNAME
|
33
|
-
value: 'my-cluster-name'
|
34
|
-
- name: K8S_NODE_NAME
|
35
|
-
valueFrom:
|
36
|
-
fieldRef:
|
37
|
-
fieldPath: spec.nodeName
|
38
|
-
resources:
|
39
|
-
limits:
|
40
|
-
cpu: 200m
|
41
|
-
memory: 400Mi
|
42
|
-
requests:
|
43
|
-
cpu: 200m
|
44
|
-
memory: 400Mi
|
45
|
-
volumeMounts:
|
46
|
-
- name: varlog
|
47
|
-
mountPath: /var/log
|
48
|
-
- name: varlibdockercontainers
|
49
|
-
mountPath: /var/lib/docker/containers
|
50
|
-
readOnly: true
|
51
|
-
- name: config
|
52
|
-
mountPath: /fluentd/etc/fluent.conf
|
53
|
-
subPath: fluent.conf
|
54
|
-
terminationGracePeriodSeconds: 30
|
55
|
-
volumes:
|
56
|
-
- name: varlog
|
57
|
-
hostPath:
|
58
|
-
path: /var/log
|
59
|
-
- name: varlibdockercontainers
|
60
|
-
hostPath:
|
61
|
-
path: /var/lib/docker/containers
|
62
|
-
- name: config
|
63
|
-
configMap:
|
64
|
-
name: fluentd-papertrail-global-logging-config
|
65
|
-
items:
|
66
|
-
- key: fluent.conf
|
67
|
-
path: fluent.conf
|
68
|
-
---
|
69
|
-
apiVersion: v1
|
70
|
-
kind: ConfigMap
|
71
|
-
metadata:
|
72
|
-
name: fluentd-papertrail-global-logging-config
|
73
|
-
namespace: kube-system
|
74
|
-
data:
|
75
|
-
fluent.conf: |-
|
76
|
-
@include kubernetes.conf
|
77
|
-
@include systemd.conf
|
78
|
-
|
79
|
-
<match **>
|
80
|
-
type papertrail
|
81
|
-
num_threads 4
|
82
|
-
|
83
|
-
papertrail_host "#{ENV['FLUENT_PAPERTRAIL_HOST']}"
|
84
|
-
papertrail_port "#{ENV['FLUENT_PAPERTRAIL_PORT']}"
|
85
|
-
</match>
|