fluent-plugin-papertrail 0.1.1 → 0.1.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/.circleci/config.yml +1 -1
- data/Makefile +10 -19
- data/README.md +29 -9
- data/docker/Dockerfile +50 -0
- data/docker/conf/fluent.conf +18 -0
- data/docker/conf/kubernetes.conf +169 -0
- data/docker/conf/systemd.conf +51 -0
- data/docker/plugins/parser_kubernetes.rb +65 -0
- data/fluent-plugin-papertrail.gemspec +3 -3
- data/kubernetes/fluentd-daemonset-papertrail.yaml +96 -0
- metadata +21 -10
- data/Dockerfile.scratch +0 -21
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 13f071d87f88a75ef2453e21eafd29feebff7b91
|
4
|
+
data.tar.gz: 4abf3a957a68b902098a944577a1d2c3c5ec75ca
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 9fcdbd9fd6d970117b86d27c6455019ce459da0a5315521e37c0f0ddf21d5d182f2f5cbbd8de785cefb44ac744a70d667432cc887cf54ad9859a0c647a401378
|
7
|
+
data.tar.gz: 18dfa2b58cae05762579db20ddff31dbee6efc3b40f42797cdff7d5e9e41b42e4a98dca5a07c1b635f11b3b7d84ddbba929e10dae075c53dc16da595d7fd261f
|
data/.circleci/config.yml
CHANGED
data/Makefile
CHANGED
@@ -1,25 +1,16 @@
|
|
1
1
|
REPO_NAME=fluent-plugin-papertrail
|
2
|
-
SCRATCH_CONTAINER_DOCKERFILE=Dockerfile.scratch
|
3
|
-
SCRATCH_IMAGE_NAME=${REPO_NAME}_scratch
|
4
|
-
SCRATCH_CONTAINER_NAME=${REPO_NAME}_scratch
|
5
|
-
SCRATCH_CONTAINER_DOCKER_OPTS=--rm -v $(PWD):/home -v $(PWD)/vendor/bundle:/usr/local/bundle -w=/home
|
6
2
|
|
7
|
-
|
8
|
-
|
3
|
+
bundle:
|
4
|
+
bundle install
|
9
5
|
|
10
|
-
|
11
|
-
|
6
|
+
test: bundle
|
7
|
+
bundle exec rake test
|
12
8
|
|
13
|
-
|
14
|
-
docker run ${SCRATCH_CONTAINER_DOCKER_OPTS} --name ${SCRATCH_CONTAINER_NAME} ${SCRATCH_IMAGE_NAME} bundle exec rake test
|
15
|
-
|
16
|
-
release: install
|
9
|
+
release: bundle
|
17
10
|
rm -rf ${REPO_NAME}-*.gem
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
clean:
|
22
|
-
docker rm ${SCRATCH_CONTAINER_NAME}
|
11
|
+
bundle exec gem build ${REPO_NAME}.gemspec
|
12
|
+
bundle exec gem push ${REPO_NAME}-*.gem
|
23
13
|
|
24
|
-
|
25
|
-
docker
|
14
|
+
release-docker:
|
15
|
+
cd docker; docker build -t quay.io/solarwinds/fluentd-kubernetes:$(TAG) .
|
16
|
+
docker push quay.io/solarwinds/fluentd-kubernetes:$(TAG)
|
data/README.md
CHANGED
@@ -1,6 +1,10 @@
|
|
1
1
|
# Fluent::Plugin::Papertrail
|
2
2
|
|
3
|
-
|
3
|
+
[![Gem Version](https://badge.fury.io/rb/fluent-plugin-papertrail.svg)](https://badge.fury.io/rb/fluent-plugin-papertrail) [![Docker Repository on Quay](https://quay.io/repository/solarwinds/fluentd-kubernetes/status "Docker Repository on Quay")](https://quay.io/repository/solarwinds/fluentd-kubernetes) [![CircleCI](https://circleci.com/gh/solarwinds/fluent-plugin-papertrail/tree/master.svg?style=shield)](https://circleci.com/gh/solarwinds/fluent-plugin-papertrail/tree/master)
|
4
|
+
|
5
|
+
## Description
|
6
|
+
|
7
|
+
This repository contains the Fluentd Papertrail Output Plugin and the Docker and Kubernetes assets for deploying that combined Fluentd, Papertrail, Kubernetes log aggregation toolset to your cluster.
|
4
8
|
|
5
9
|
## Installation
|
6
10
|
|
@@ -57,7 +61,7 @@ the [fluent-plugin-kubernetes_metadata_filter](https://github.com/fabric8io/flue
|
|
57
61
|
```
|
58
62
|
|
59
63
|
### Advanced Configuration
|
60
|
-
This plugin inherits a few useful config parameters from Fluent's BufferedOutput class.
|
64
|
+
This plugin inherits a few useful config parameters from Fluent's `BufferedOutput` class.
|
61
65
|
|
62
66
|
Parameters for flushing the buffer, based on size and time, are `buffer_chunk_limit` and `flush_interval`, respectively. This plugin overrides the inherited default `flush_interval` to `1`, causing the fluent buffer to flush to Papertrail every second.
|
63
67
|
|
@@ -74,23 +78,39 @@ If you want to change any of these parameters simply add them to a match stanza.
|
|
74
78
|
</match>
|
75
79
|
```
|
76
80
|
|
81
|
+
## Kubernetes
|
82
|
+
|
83
|
+
This repo also includes Kubernetes and Docker assets which do all of the heavy lifting for you.
|
84
|
+
|
85
|
+
If you'd like to deploy this plugin as a DaemonSet to your Kubernetes cluster, just adjust the `FLUENT_*` environment variables in `kubernetes/fluentd-daemonset-papertrail.yaml` and push it to your cluster with:
|
86
|
+
|
87
|
+
```
|
88
|
+
kubectl apply -f kubernetes/fluentd-daemonset-papertrail.yaml
|
89
|
+
```
|
90
|
+
|
91
|
+
The Dockerfile that generates [the image used in this DaemonSet](https://quay.io/repository/solarwinds/fluentd-kubernetes), can be found at `docker/Dockerfile`.
|
92
|
+
|
77
93
|
## Development
|
78
94
|
|
79
|
-
|
80
|
-
We have a [Dockerfile](Dockerfile.scratch) where we build a scratch image that contains all the dependencies.
|
81
|
-
We have a [Makefile](Makefile) to wrap the common functions and make life easier.
|
95
|
+
This plugin is targeting Ruby 2.4 and Fluentd v1.0, although it should work with older versions of both.
|
82
96
|
|
83
|
-
|
84
|
-
|
97
|
+
We have a [Makefile](Makefile) to wrap common functions and make life easier.
|
98
|
+
|
99
|
+
### Install Dependencies
|
100
|
+
`make bundle`
|
85
101
|
|
86
102
|
### Test
|
87
103
|
`make test`
|
88
104
|
|
89
|
-
### Release in [RubyGems](
|
90
|
-
To release a new version, update the version number in the [
|
105
|
+
### Release in [RubyGems](https://rubygems.org/gems/fluent-plugin-papertrail)
|
106
|
+
To release a new version, update the version number in the [GemSpec](fluent-plugin-papertrail.gemspec) and then, run:
|
91
107
|
|
92
108
|
`make release`
|
93
109
|
|
110
|
+
### Release in [Quay.io](https://quay.io/repository/solarwinds/fluentd-kubernetes)
|
111
|
+
|
112
|
+
`make release-docker TAG=$(VERSION)`
|
113
|
+
|
94
114
|
## Contributing
|
95
115
|
|
96
116
|
Bug reports and pull requests are welcome on GitHub at https://github.com/solarwinds/fluent-plugin-papertrail.
|
data/docker/Dockerfile
ADDED
@@ -0,0 +1,50 @@
|
|
1
|
+
FROM fluent/fluentd:v1.1.1-debian
|
2
|
+
|
3
|
+
USER root
|
4
|
+
WORKDIR /home/fluent
|
5
|
+
ENV PATH /home/fluent/.gem/ruby/2.3.0/bin:$PATH
|
6
|
+
|
7
|
+
RUN buildDeps="sudo make gcc g++ libc-dev ruby-dev libffi-dev" \
|
8
|
+
&& apt-get update \
|
9
|
+
&& apt-get upgrade -y \
|
10
|
+
&& apt-get install \
|
11
|
+
-y --no-install-recommends \
|
12
|
+
$buildDeps \
|
13
|
+
&& echo 'gem: --no-document' >> /etc/gemrc \
|
14
|
+
&& gem install fluent-plugin-secure-forward \
|
15
|
+
&& gem install fluent-plugin-record-reformer \
|
16
|
+
&& gem install fluent-plugin-systemd -v 0.3.1 \
|
17
|
+
&& gem install fluent-plugin-rewrite-tag-filter -v 1.6.0 \
|
18
|
+
&& gem install fluent-plugin-papertrail -v 0.1.2.pre.dev \
|
19
|
+
&& gem install fluent-plugin-kubernetes_metadata_filter \
|
20
|
+
&& gem install ffi \
|
21
|
+
&& SUDO_FORCE_REMOVE=yes \
|
22
|
+
apt-get purge -y --auto-remove \
|
23
|
+
-o APT::AutoRemove::RecommendsImportant=false \
|
24
|
+
$buildDeps \
|
25
|
+
&& rm -rf /var/lib/apt/lists/* \
|
26
|
+
&& gem sources --clear-all \
|
27
|
+
&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
|
28
|
+
|
29
|
+
# Copy configuration files
|
30
|
+
COPY ./conf/fluent.conf /fluentd/etc/
|
31
|
+
COPY ./conf/systemd.conf /fluentd/etc/
|
32
|
+
COPY ./conf/kubernetes.conf /fluentd/etc/
|
33
|
+
|
34
|
+
# Copy plugins
|
35
|
+
COPY plugins /fluentd/plugins/
|
36
|
+
|
37
|
+
# Environment variables
|
38
|
+
ENV FLUENTD_OPT=""
|
39
|
+
ENV FLUENTD_CONF="fluent.conf"
|
40
|
+
|
41
|
+
# jemalloc is memory optimization only available for td-agent
|
42
|
+
# td-agent is provided and QA'ed by treasuredata as rpm/deb/.. package
|
43
|
+
# -> td-agent (stable) vs fluentd (edge)
|
44
|
+
#ENV LD_PRELOAD="/usr/lib/libjemalloc.so.2"
|
45
|
+
|
46
|
+
# Overriding entrypoint, otherwise parent image causes to run as user fluent
|
47
|
+
ENTRYPOINT []
|
48
|
+
|
49
|
+
# Run Fluentd
|
50
|
+
CMD exec fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT
|
@@ -0,0 +1,18 @@
|
|
1
|
+
@include systemd.conf
|
2
|
+
@include kubernetes.conf
|
3
|
+
|
4
|
+
## Capture audit logs
|
5
|
+
#<match kube-apiserver-audit>
|
6
|
+
# type papertrail
|
7
|
+
#
|
8
|
+
# papertrail_host "#{ENV['FLUENT_PAPERTRAIL_AUDIT_HOST']}"
|
9
|
+
# papertrail_port "#{ENV['FLUENT_PAPERTRAIL_AUDIT_PORT']}"
|
10
|
+
#</match>
|
11
|
+
|
12
|
+
<match **>
|
13
|
+
type papertrail
|
14
|
+
|
15
|
+
papertrail_host "#{ENV['FLUENT_PAPERTRAIL_HOST']}"
|
16
|
+
papertrail_port "#{ENV['FLUENT_PAPERTRAIL_PORT']}"
|
17
|
+
|
18
|
+
</match>
|
@@ -0,0 +1,169 @@
|
|
1
|
+
<match fluent.**>
|
2
|
+
type null
|
3
|
+
</match>
|
4
|
+
|
5
|
+
<source>
|
6
|
+
type tail
|
7
|
+
path /var/log/containers/*.log
|
8
|
+
pos_file /var/log/fluentd-containers.log.pos
|
9
|
+
time_format %Y-%m-%dT%H:%M:%S.%NZ
|
10
|
+
tag kubernetes.*
|
11
|
+
format json
|
12
|
+
read_from_head true
|
13
|
+
</source>
|
14
|
+
|
15
|
+
<source>
|
16
|
+
type tail
|
17
|
+
format /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
|
18
|
+
time_format %Y-%m-%d %H:%M:%S
|
19
|
+
path /var/log/salt/minion
|
20
|
+
pos_file /var/log/fluentd-salt.pos
|
21
|
+
tag salt
|
22
|
+
</source>
|
23
|
+
|
24
|
+
<source>
|
25
|
+
type tail
|
26
|
+
format syslog
|
27
|
+
path /var/log/startupscript.log
|
28
|
+
pos_file /var/log/fluentd-startupscript.log.pos
|
29
|
+
tag startupscript
|
30
|
+
</source>
|
31
|
+
|
32
|
+
<source>
|
33
|
+
type tail
|
34
|
+
format /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
|
35
|
+
path /var/log/docker.log
|
36
|
+
pos_file /var/log/fluentd-docker.log.pos
|
37
|
+
tag docker
|
38
|
+
</source>
|
39
|
+
|
40
|
+
<source>
|
41
|
+
type tail
|
42
|
+
format none
|
43
|
+
path /var/log/etcd.log
|
44
|
+
pos_file /var/log/fluentd-etcd.log.pos
|
45
|
+
tag etcd
|
46
|
+
</source>
|
47
|
+
|
48
|
+
<source>
|
49
|
+
type tail
|
50
|
+
format kubernetes
|
51
|
+
multiline_flush_interval 5s
|
52
|
+
path /var/log/kubelet.log
|
53
|
+
pos_file /var/log/fluentd-kubelet.log.pos
|
54
|
+
tag kubelet
|
55
|
+
</source>
|
56
|
+
|
57
|
+
<source>
|
58
|
+
type tail
|
59
|
+
format kubernetes
|
60
|
+
multiline_flush_interval 5s
|
61
|
+
path /var/log/kube-proxy.log
|
62
|
+
pos_file /var/log/fluentd-kube-proxy.log.pos
|
63
|
+
tag kube-proxy
|
64
|
+
</source>
|
65
|
+
|
66
|
+
<source>
|
67
|
+
type tail
|
68
|
+
format kubernetes
|
69
|
+
multiline_flush_interval 5s
|
70
|
+
path /var/log/kube-apiserver.log
|
71
|
+
pos_file /var/log/fluentd-kube-apiserver.log.pos
|
72
|
+
tag kube-apiserver
|
73
|
+
</source>
|
74
|
+
|
75
|
+
<source>
|
76
|
+
type tail
|
77
|
+
format kubernetes
|
78
|
+
multiline_flush_interval 5s
|
79
|
+
path /var/log/kube-controller-manager.log
|
80
|
+
pos_file /var/log/fluentd-kube-controller-manager.log.pos
|
81
|
+
tag kube-controller-manager
|
82
|
+
</source>
|
83
|
+
|
84
|
+
<source>
|
85
|
+
type tail
|
86
|
+
format kubernetes
|
87
|
+
multiline_flush_interval 5s
|
88
|
+
path /var/log/kube-scheduler.log
|
89
|
+
pos_file /var/log/fluentd-kube-scheduler.log.pos
|
90
|
+
tag kube-scheduler
|
91
|
+
</source>
|
92
|
+
|
93
|
+
<source>
|
94
|
+
type tail
|
95
|
+
format kubernetes
|
96
|
+
multiline_flush_interval 5s
|
97
|
+
path /var/log/rescheduler.log
|
98
|
+
pos_file /var/log/fluentd-rescheduler.log.pos
|
99
|
+
tag rescheduler
|
100
|
+
</source>
|
101
|
+
|
102
|
+
<source>
|
103
|
+
type tail
|
104
|
+
format kubernetes
|
105
|
+
multiline_flush_interval 5s
|
106
|
+
path /var/log/glbc.log
|
107
|
+
pos_file /var/log/fluentd-glbc.log.pos
|
108
|
+
tag glbc
|
109
|
+
</source>
|
110
|
+
|
111
|
+
<source>
|
112
|
+
type tail
|
113
|
+
format kubernetes
|
114
|
+
multiline_flush_interval 5s
|
115
|
+
path /var/log/cluster-autoscaler.log
|
116
|
+
pos_file /var/log/fluentd-cluster-autoscaler.log.pos
|
117
|
+
tag cluster-autoscaler
|
118
|
+
</source>
|
119
|
+
|
120
|
+
# Example:
|
121
|
+
# 2017-02-09T00:15:57.992775796Z AUDIT: id="90c73c7c-97d6-4b65-9461-f94606ff825f" ip="104.132.1.72" method="GET" user="kubecfg" as="<self>" asgroups="<lookup>" namespace="default" uri="/api/v1/namespaces/default/pods"
|
122
|
+
# 2017-02-09T00:15:57.993528822Z AUDIT: id="90c73c7c-97d6-4b65-9461-f94606ff825f" response="200"
|
123
|
+
<source>
|
124
|
+
type tail
|
125
|
+
format multiline
|
126
|
+
multiline_flush_interval 5s
|
127
|
+
format_firstline /^\S+\s+AUDIT:/
|
128
|
+
# Fields must be explicitly captured by name to be parsed into the record.
|
129
|
+
# Fields may not always be present, and order may change, so this just looks
|
130
|
+
# for a list of key="\"quoted\" value" pairs separated by spaces.
|
131
|
+
# Unknown fields are ignored.
|
132
|
+
# Note: We can't separate query/response lines as format1/format2 because
|
133
|
+
# they don't always come one after the other for a given query.
|
134
|
+
format1 /^(?<time>\S+) AUDIT:(?: (?:id="(?<id>(?:[^"\\]|\\.)*)"|ip="(?<ip>(?:[^"\\]|\\.)*)"|method="(?<method>(?:[^"\\]|\\.)*)"|user="(?<user>(?:[^"\\]|\\.)*)"|groups="(?<groups>(?:[^"\\]|\\.)*)"|as="(?<as>(?:[^"\\]|\\.)*)"|asgroups="(?<asgroups>(?:[^"\\]|\\.)*)"|namespace="(?<namespace>(?:[^"\\]|\\.)*)"|uri="(?<uri>(?:[^"\\]|\\.)*)"|response="(?<response>(?:[^"\\]|\\.)*)"|\w+="(?:[^"\\]|\\.)*"))*/
|
135
|
+
time_format %FT%T.%L%Z
|
136
|
+
path /var/log/kubernetes/kube-apiserver-audit.log
|
137
|
+
pos_file /var/log/kube-apiserver-audit.log.pos
|
138
|
+
tag kube-apiserver-audit
|
139
|
+
</source>
|
140
|
+
|
141
|
+
<filter kubernetes.**>
|
142
|
+
type kubernetes_metadata
|
143
|
+
</filter>
|
144
|
+
|
145
|
+
<filter kube-apiserver-audit>
|
146
|
+
type record_transformer
|
147
|
+
enable_ruby true
|
148
|
+
<record>
|
149
|
+
hostname #{ENV['FLUENT_HOSTNAME']}
|
150
|
+
program kube-apiserver-audit
|
151
|
+
severity info
|
152
|
+
facility local0
|
153
|
+
message ${record}
|
154
|
+
</record>
|
155
|
+
</filter>
|
156
|
+
|
157
|
+
# append namespace and pod name to hostname, so that logs in Papertrail are filterable by each
|
158
|
+
# use container name as program name, but trim it to 32 characters to match remote_syslog spec
|
159
|
+
<filter kubernetes.**>
|
160
|
+
type record_transformer
|
161
|
+
enable_ruby true
|
162
|
+
<record>
|
163
|
+
hostname #{ENV['FLUENT_HOSTNAME']}-${record["kubernetes"]["namespace_name"]}-${record["kubernetes"]["pod_name"]}
|
164
|
+
program ${record["kubernetes"]["container_name"][0..31]}
|
165
|
+
severity info
|
166
|
+
facility local0
|
167
|
+
message ${record['log']}
|
168
|
+
</record>
|
169
|
+
</filter>
|
@@ -0,0 +1,51 @@
|
|
1
|
+
<source>
|
2
|
+
@type systemd
|
3
|
+
pos_file /var/log/fluentd-journald-systemd.pos
|
4
|
+
read_from_head true
|
5
|
+
strip_underscores true
|
6
|
+
tag systemd
|
7
|
+
</source>
|
8
|
+
|
9
|
+
# rewrite tags as systemd.* for the specefic SYSTEMD_UNIT, then we can filter specifically on kubelet and docker below
|
10
|
+
<match systemd>
|
11
|
+
@type rewrite_tag_filter
|
12
|
+
rewriterule1 SYSTEMD_UNIT ^kubelet.service$ systemd.kubelet
|
13
|
+
rewriterule2 SYSTEMD_UNIT ^docker.service$ systemd.docker
|
14
|
+
</match>
|
15
|
+
|
16
|
+
# toss all other systemd logs in the bin
|
17
|
+
<match systemd>
|
18
|
+
@type null
|
19
|
+
</match>
|
20
|
+
|
21
|
+
# transform systemd logs to Papertrail format
|
22
|
+
<filter systemd.kubelet>
|
23
|
+
@type record_transformer
|
24
|
+
enable_ruby true
|
25
|
+
<record>
|
26
|
+
hostname "#{ENV['FLUENT_HOSTNAME']}-${record['HOSTNAME']}"
|
27
|
+
program kubelet
|
28
|
+
severity info
|
29
|
+
facility local0
|
30
|
+
message ${record['MESSAGE']}
|
31
|
+
</record>
|
32
|
+
</filter>
|
33
|
+
|
34
|
+
<filter systemd.docker>
|
35
|
+
type parser
|
36
|
+
format /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
|
37
|
+
reserve_data true
|
38
|
+
key_name MESSAGE
|
39
|
+
suppress_parse_error_log true
|
40
|
+
</filter>
|
41
|
+
|
42
|
+
<filter systemd.docker>
|
43
|
+
@type record_transformer
|
44
|
+
enable_ruby true
|
45
|
+
<record>
|
46
|
+
hostname "#{ENV['FLUENT_HOSTNAME']}-${record['HOSTNAME']}"
|
47
|
+
program docker
|
48
|
+
severity info
|
49
|
+
facility local0
|
50
|
+
</record>
|
51
|
+
</filter>
|
@@ -0,0 +1,65 @@
|
|
1
|
+
#
|
2
|
+
# Fluentd
|
3
|
+
#
|
4
|
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
5
|
+
# you may not use this file except in compliance with the License.
|
6
|
+
# You may obtain a copy of the License at
|
7
|
+
#
|
8
|
+
# http://www.apache.org/licenses/LICENSE-2.0
|
9
|
+
#
|
10
|
+
# Unless required by applicable law or agreed to in writing, software
|
11
|
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
12
|
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
13
|
+
# See the License for the specific language governing permissions and
|
14
|
+
# limitations under the License.
|
15
|
+
#
|
16
|
+
|
17
|
+
# The following Fluentd parser plugin, aims to simplify the parsing of multiline
|
18
|
+
# logs found in Kubernetes nodes. Since many log files shared the same format and
|
19
|
+
# in order to simplify the configuration, this plugin provides a 'kubernetes' format
|
20
|
+
# parser (built on top of MultilineParser).
|
21
|
+
#
|
22
|
+
# When tailing files, this 'kubernetes' format should be applied to the following
|
23
|
+
# log file sources:
|
24
|
+
#
|
25
|
+
# - /var/log/kubelet.log
|
26
|
+
# - /var/log/kube-proxy.log
|
27
|
+
# - /var/log/kube-apiserver.log
|
28
|
+
# - /var/log/kube-controller-manager.log
|
29
|
+
# - /var/log/kube-scheduler.log
|
30
|
+
# - /var/log/rescheduler.log
|
31
|
+
# - /var/log/glbc.log
|
32
|
+
# - /var/log/cluster-autoscaler.log
|
33
|
+
#
|
34
|
+
# Usage:
|
35
|
+
#
|
36
|
+
# ---- fluentd.conf ----
|
37
|
+
#
|
38
|
+
# <source>
|
39
|
+
# type tail
|
40
|
+
# format kubernetes
|
41
|
+
# path ./kubelet.log
|
42
|
+
# read_from_head yes
|
43
|
+
# tag kubelet
|
44
|
+
# </source>
|
45
|
+
#
|
46
|
+
# ---- EOF ---
|
47
|
+
|
48
|
+
require 'fluent/parser'
|
49
|
+
|
50
|
+
module Fluent
|
51
|
+
class KubernetesParser < Fluent::TextParser::MultilineParser
|
52
|
+
Fluent::Plugin.register_parser("kubernetes", self)
|
53
|
+
|
54
|
+
CONF_FORMAT_FIRSTLINE = %q{/^\w\d{4}/}
|
55
|
+
CONF_FORMAT1 = %q{/^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/}
|
56
|
+
CONF_TIME_FORMAT = "%m%d %H:%M:%S.%N"
|
57
|
+
|
58
|
+
def configure(conf)
|
59
|
+
conf['format_firstline'] = CONF_FORMAT_FIRSTLINE
|
60
|
+
conf['format1'] = CONF_FORMAT1
|
61
|
+
conf['time_format'] = CONF_TIME_FORMAT
|
62
|
+
super
|
63
|
+
end
|
64
|
+
end
|
65
|
+
end
|
@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
|
4
4
|
|
5
5
|
Gem::Specification.new do |spec|
|
6
6
|
spec.name = "fluent-plugin-papertrail"
|
7
|
-
spec.version = "0.1.
|
7
|
+
spec.version = "0.1.2"
|
8
8
|
spec.authors = ["Jonathan Lozinski", "Alex Ouzounis", "Chris Rust"]
|
9
9
|
spec.email = ["jonathan.lozinski@solarwinds.com", "alex.ouzounis@solarwinds.com", "chris.rust@solarwinds.com"]
|
10
10
|
|
@@ -20,8 +20,8 @@ Gem::Specification.new do |spec|
|
|
20
20
|
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
|
21
21
|
spec.require_paths = ["lib"]
|
22
22
|
|
23
|
-
spec.add_dependency "fluentd",
|
24
|
-
spec.add_dependency "fluent-mixin-config-placeholders", "~> 0.
|
23
|
+
spec.add_dependency "fluentd", '>= 0.10', '< 2'
|
24
|
+
spec.add_dependency "fluent-mixin-config-placeholders", "~> 0.4.0"
|
25
25
|
spec.add_dependency "syslog_protocol"
|
26
26
|
|
27
27
|
spec.add_development_dependency "bundler", "~> 1.12"
|
@@ -0,0 +1,96 @@
|
|
1
|
+
# Uncomment sections below to allow audit logs to go to a separate host
|
2
|
+
apiVersion: extensions/v1beta1
|
3
|
+
kind: DaemonSet
|
4
|
+
metadata:
|
5
|
+
name: fluentd-papertrail-global-logging
|
6
|
+
namespace: kube-system
|
7
|
+
labels:
|
8
|
+
k8s-app: fluentd-logging
|
9
|
+
version: v1
|
10
|
+
kubernetes.io/cluster-service: "true"
|
11
|
+
spec:
|
12
|
+
updateStrategy:
|
13
|
+
type: RollingUpdate
|
14
|
+
template:
|
15
|
+
metadata:
|
16
|
+
labels:
|
17
|
+
k8s-app: fluentd-logging
|
18
|
+
version: v1
|
19
|
+
kubernetes.io/cluster-service: "true"
|
20
|
+
spec:
|
21
|
+
tolerations:
|
22
|
+
- key: node-role.kubernetes.io/master
|
23
|
+
effect: NoSchedule
|
24
|
+
containers:
|
25
|
+
- name: fluentd
|
26
|
+
image: quay.io/solarwinds/fluentd-kubernetes:v0.12.33-debian-papertrail
|
27
|
+
imagePullPolicy: Always
|
28
|
+
env:
|
29
|
+
- name: FLUENT_PAPERTRAIL_HOST
|
30
|
+
value: 'logsN.papertrailapp.com'
|
31
|
+
- name: FLUENT_PAPERTRAIL_PORT
|
32
|
+
value: 'NNNNN'
|
33
|
+
# - name: FLUENT_PAPERTRAIL_AUDIT_HOST
|
34
|
+
# value: 'logsN.papertrailapp.com'
|
35
|
+
# - name: FLUENT_PAPERTRAIL_AUDIT_PORT
|
36
|
+
# value: 'NNNNN'
|
37
|
+
- name: FLUENT_HOSTNAME
|
38
|
+
value: 'my-cluster-name'
|
39
|
+
resources:
|
40
|
+
limits:
|
41
|
+
cpu: 200m
|
42
|
+
memory: 400Mi
|
43
|
+
requests:
|
44
|
+
cpu: 200m
|
45
|
+
memory: 400Mi
|
46
|
+
volumeMounts:
|
47
|
+
- name: varlog
|
48
|
+
mountPath: /var/log
|
49
|
+
- name: varlibdockercontainers
|
50
|
+
mountPath: /var/lib/docker/containers
|
51
|
+
readOnly: true
|
52
|
+
- name: config
|
53
|
+
mountPath: /fluentd/etc/fluent.conf
|
54
|
+
subPath: fluent.conf
|
55
|
+
terminationGracePeriodSeconds: 30
|
56
|
+
volumes:
|
57
|
+
- name: varlog
|
58
|
+
hostPath:
|
59
|
+
path: /var/log
|
60
|
+
- name: varlibdockercontainers
|
61
|
+
hostPath:
|
62
|
+
path: /var/lib/docker/containers
|
63
|
+
- name: config
|
64
|
+
configMap:
|
65
|
+
name: fluentd-papertrail-global-logging-config
|
66
|
+
items:
|
67
|
+
- key: fluent.conf
|
68
|
+
path: fluent.conf
|
69
|
+
---
|
70
|
+
apiVersion: v1
|
71
|
+
kind: ConfigMap
|
72
|
+
metadata:
|
73
|
+
name: fluentd-papertrail-global-logging-config
|
74
|
+
namespace: kube-system
|
75
|
+
data:
|
76
|
+
fluent.conf: |-
|
77
|
+
@include kubernetes.conf
|
78
|
+
@include systemd.conf
|
79
|
+
|
80
|
+
# # Capture audit logs
|
81
|
+
# <match kube-apiserver-audit>
|
82
|
+
# type papertrail
|
83
|
+
# num_threads 4
|
84
|
+
#
|
85
|
+
# papertrail_host "#{ENV['FLUENT_PAPERTRAIL_AUDIT_HOST']}"
|
86
|
+
# papertrail_port "#{ENV['FLUENT_PAPERTRAIL_AUDIT_PORT']}"
|
87
|
+
# </match>
|
88
|
+
#
|
89
|
+
# Capture all unmatched tags
|
90
|
+
<match **>
|
91
|
+
type papertrail
|
92
|
+
num_threads 4
|
93
|
+
|
94
|
+
papertrail_host "#{ENV['FLUENT_PAPERTRAIL_HOST']}"
|
95
|
+
papertrail_port "#{ENV['FLUENT_PAPERTRAIL_PORT']}"
|
96
|
+
</match>
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-papertrail
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.
|
4
|
+
version: 0.1.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Jonathan Lozinski
|
@@ -10,36 +10,42 @@ authors:
|
|
10
10
|
autorequire:
|
11
11
|
bindir: exe
|
12
12
|
cert_chain: []
|
13
|
-
date: 2018-
|
13
|
+
date: 2018-03-10 00:00:00.000000000 Z
|
14
14
|
dependencies:
|
15
15
|
- !ruby/object:Gem::Dependency
|
16
16
|
name: fluentd
|
17
17
|
requirement: !ruby/object:Gem::Requirement
|
18
18
|
requirements:
|
19
|
-
- - "
|
19
|
+
- - ">="
|
20
|
+
- !ruby/object:Gem::Version
|
21
|
+
version: '0.10'
|
22
|
+
- - "<"
|
20
23
|
- !ruby/object:Gem::Version
|
21
|
-
version:
|
24
|
+
version: '2'
|
22
25
|
type: :runtime
|
23
26
|
prerelease: false
|
24
27
|
version_requirements: !ruby/object:Gem::Requirement
|
25
28
|
requirements:
|
26
|
-
- - "
|
29
|
+
- - ">="
|
30
|
+
- !ruby/object:Gem::Version
|
31
|
+
version: '0.10'
|
32
|
+
- - "<"
|
27
33
|
- !ruby/object:Gem::Version
|
28
|
-
version:
|
34
|
+
version: '2'
|
29
35
|
- !ruby/object:Gem::Dependency
|
30
36
|
name: fluent-mixin-config-placeholders
|
31
37
|
requirement: !ruby/object:Gem::Requirement
|
32
38
|
requirements:
|
33
39
|
- - "~>"
|
34
40
|
- !ruby/object:Gem::Version
|
35
|
-
version: 0.
|
41
|
+
version: 0.4.0
|
36
42
|
type: :runtime
|
37
43
|
prerelease: false
|
38
44
|
version_requirements: !ruby/object:Gem::Requirement
|
39
45
|
requirements:
|
40
46
|
- - "~>"
|
41
47
|
- !ruby/object:Gem::Version
|
42
|
-
version: 0.
|
48
|
+
version: 0.4.0
|
43
49
|
- !ruby/object:Gem::Dependency
|
44
50
|
name: syslog_protocol
|
45
51
|
requirement: !ruby/object:Gem::Requirement
|
@@ -121,7 +127,6 @@ extra_rdoc_files: []
|
|
121
127
|
files:
|
122
128
|
- ".circleci/config.yml"
|
123
129
|
- ".gitignore"
|
124
|
-
- Dockerfile.scratch
|
125
130
|
- Gemfile
|
126
131
|
- LICENSE
|
127
132
|
- Makefile
|
@@ -129,7 +134,13 @@ files:
|
|
129
134
|
- Rakefile
|
130
135
|
- bin/console
|
131
136
|
- bin/setup
|
137
|
+
- docker/Dockerfile
|
138
|
+
- docker/conf/fluent.conf
|
139
|
+
- docker/conf/kubernetes.conf
|
140
|
+
- docker/conf/systemd.conf
|
141
|
+
- docker/plugins/parser_kubernetes.rb
|
132
142
|
- fluent-plugin-papertrail.gemspec
|
143
|
+
- kubernetes/fluentd-daemonset-papertrail.yaml
|
133
144
|
- lib/fluent/plugin/out_papertrail.rb
|
134
145
|
homepage: https://github.com/solarwinds/fluent-plugin-papertrail
|
135
146
|
licenses:
|
@@ -151,7 +162,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
151
162
|
version: '0'
|
152
163
|
requirements: []
|
153
164
|
rubyforge_project:
|
154
|
-
rubygems_version: 2.
|
165
|
+
rubygems_version: 2.6.14
|
155
166
|
signing_key:
|
156
167
|
specification_version: 4
|
157
168
|
summary: Remote Syslog Output Fluentd plugin for papertrail
|
data/Dockerfile.scratch
DELETED
@@ -1,21 +0,0 @@
|
|
1
|
-
# vim:set ft=dockerfile:
|
2
|
-
FROM ruby:2.3.6-alpine
|
3
|
-
|
4
|
-
MAINTAINER Alex Ouzounis <alex.ouzounis@solarwinds.com>
|
5
|
-
|
6
|
-
ENV NOKOGIRI_USE_SYSTEM_LIBRARIES=1
|
7
|
-
|
8
|
-
RUN \
|
9
|
-
apk add --update \
|
10
|
-
ruby-bigdecimal \
|
11
|
-
ruby-bundler \
|
12
|
-
ruby-rdoc \
|
13
|
-
build-base \
|
14
|
-
ruby-dev \
|
15
|
-
git \
|
16
|
-
openssh \
|
17
|
-
ca-certificates \
|
18
|
-
\
|
19
|
-
&& bundle config build.nokogiri --use-system-libraries \
|
20
|
-
\
|
21
|
-
&& rm -rf /var/cache/apk/* /tmp
|