fluent-plugin-papertrail 0.2.4 → 0.2.5.pre.dev

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 3413dd7623c0aab24484393559a57199331c9830
4
- data.tar.gz: 0f21d2987c62bc31dd38c35bcec83dcce3bc6821
3
+ metadata.gz: 297b4941b70100a4be1be6fb5e35f14263553202
4
+ data.tar.gz: 80fccb4690f183f69aabc9da7a793fbbe357e19d
5
5
  SHA512:
6
- metadata.gz: f31eab178bfbed278862bbc68e18f2493e1fc9ab60b84a8f2dbb67ed4a3f8a0b74cb1587c5111cae65ebc14ad9d351731a37eb86178ab06ba366402d41ef7156
7
- data.tar.gz: 33eec86f6cbee0ff4f0bdc91c41028f3d3371aece5b71c6b82d8dcfc8db9c61e8248d0556f692e347c05055e4ee77976bcea9631bc91e987e5a68e24d647b519
6
+ metadata.gz: 4ed8cae9ae77236c3c64d3f4537e2a5274a67050bcf5478a5d60a90a0ac9cc795cb54c79936eb5ec0a6459c1825a8ff5fe2981e1ac9fbd1f482b1f7f9545ccea
7
+ data.tar.gz: 9a575ad74e1759185008d11196d712f010f954ba9467ea2829668c461b712790fc99cc4c7f0d439eb23a5f18f2cfff96e7bb3ea24c5db8444bfd12160e223a4f
data/Makefile CHANGED
@@ -10,7 +10,3 @@ release: bundle
10
10
  rm -rf ${REPO_NAME}-*.gem
11
11
  bundle exec gem build ${REPO_NAME}.gemspec
12
12
  bundle exec gem push ${REPO_NAME}-*.gem
13
-
14
- release-docker:
15
- cd docker; docker build -t quay.io/solarwinds/fluentd-kubernetes:$(TAG) .
16
- docker push quay.io/solarwinds/fluentd-kubernetes:$(TAG)
data/README.md CHANGED
@@ -1,10 +1,10 @@
1
1
  # Fluent::Plugin::Papertrail
2
2
 
3
- [![Gem Version](https://badge.fury.io/rb/fluent-plugin-papertrail.svg)](https://badge.fury.io/rb/fluent-plugin-papertrail) [![Docker Repository on Quay](https://quay.io/repository/solarwinds/fluentd-kubernetes/status "Docker Repository on Quay")](https://quay.io/repository/solarwinds/fluentd-kubernetes) [![CircleCI](https://circleci.com/gh/solarwinds/fluent-plugin-papertrail/tree/master.svg?style=shield)](https://circleci.com/gh/solarwinds/fluent-plugin-papertrail/tree/master)
3
+ [![Gem Version](https://badge.fury.io/rb/fluent-plugin-papertrail.svg)](https://badge.fury.io/rb/fluent-plugin-papertrail) [![CircleCI](https://circleci.com/gh/solarwinds/fluent-plugin-papertrail/tree/master.svg?style=shield)](https://circleci.com/gh/solarwinds/fluent-plugin-papertrail/tree/master)
4
4
 
5
5
  ## Description
6
6
 
7
- This repository contains the Fluentd Papertrail Output Plugin and the Docker and Kubernetes assets for deploying that combined Fluentd, Papertrail, Kubernetes log aggregation toolset to your cluster.
7
+ This repository contains the Fluentd Papertrail Output Plugin.
8
8
 
9
9
  ## Installation
10
10
 
@@ -42,7 +42,7 @@ This plugin expects the following fields to be set for each Fluent record:
42
42
  hostname The source hostname for papertrail logging
43
43
  ```
44
44
 
45
- The following example is a `record_transformer` filter, from the [Kubernetes assets](docker/conf/kubernetes.conf) in this repo, that is used along with the [fluent-plugin-kubernetes_metadata_filter](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter) to populate the required fields for our plugin:
45
+ The following example is a `record_transformer` filter, from the Kubernetes assets [in the Solarwinds fluentd-deployment repo](https://github.com/solarwinds/fluentd-deployment/blob/master/docker/conf/kubernetes.conf), that is used along with the [fluent-plugin-kubernetes_metadata_filter](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter) to populate the required fields for our plugin:
46
46
  ```yaml
47
47
  <filter kubernetes.**>
48
48
  type kubernetes_metadata
@@ -81,21 +81,9 @@ If you want to change any of these parameters simply add them to a match stanza.
81
81
  </match>
82
82
  ```
83
83
 
84
- ## Kubernetes
84
+ ## Kubernetes Annotations
85
85
 
86
- This repo includes a Kubernetes DaemonSet and accompanying Docker container which will stream all of your Kubernetes logs, containers and services, to Papertrail.
87
-
88
- To deploy this plugin as a DaemonSet to your Kubernetes cluster, just adjust the `FLUENT_*` environment variables in `kubernetes/fluentd-daemonset-papertrail.yaml` and push it to your cluster with:
89
-
90
- ```
91
- kubectl apply -f kubernetes/fluentd-daemonset-papertrail.yaml
92
- ```
93
-
94
- The Dockerfile that generates [the image used in this DaemonSet](https://quay.io/repository/solarwinds/fluentd-kubernetes), can be found at `docker/Dockerfile`.
95
-
96
- ### Annotations
97
-
98
- You can redirect logs to alternate Papertrail destinations by adding annotations to your Pods or Namespaces:
86
+ If you're running this plugin in Kubernetes with the kubernetes_metadata_filter plugin enabled you can redirect logs to alternate Papertrail destinations by adding annotations to your Pods or Namespaces:
99
87
 
100
88
  ```
101
89
  solarwinds.io/papertrail_host: 'logs0.papertrailapp.com'
@@ -104,21 +92,6 @@ solarwinds.io/papertrail_port: '12345'
104
92
 
105
93
  If both the Pod and Namespace have annotations for any running Pod, the Pod's annotation is used.
106
94
 
107
- ### Audit Logs
108
-
109
- If you'd like to redirect Kubernetes API Server Audit logs to a seperate Papertrail destination, add the following to your `fluent.conf`:
110
- ```
111
- <match kube-apiserver-audit>
112
- type papertrail
113
- num_threads 4
114
-
115
- papertrail_host "#{ENV['FLUENT_PAPERTRAIL_AUDIT_HOST']}"
116
- papertrail_port "#{ENV['FLUENT_PAPERTRAIL_AUDIT_PORT']}"
117
- </match>
118
- ```
119
-
120
- This requires you to configure an [audit policy file](https://kubernetes.io/docs/tasks/debug-application-cluster/audit/) on your cluster.
121
-
122
95
  ## Development
123
96
 
124
97
  This plugin is targeting Ruby 2.4 and Fluentd v1.0, although it should work with older versions of both.
@@ -136,10 +109,6 @@ To release a new version, update the version number in the [GemSpec](fluent-plug
136
109
 
137
110
  `make release`
138
111
 
139
- ### Release in [Quay.io](https://quay.io/repository/solarwinds/fluentd-kubernetes)
140
-
141
- `make release-docker TAG=$(VERSION)`
142
-
143
112
  ## Contributing
144
113
 
145
114
  Bug reports and pull requests are welcome on GitHub at: https://github.com/solarwinds/fluent-plugin-papertrail
@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
4
 
5
5
  Gem::Specification.new do |spec|
6
6
  spec.name = "fluent-plugin-papertrail"
7
- spec.version = "0.2.4"
7
+ spec.version = "0.2.5-dev"
8
8
  spec.authors = ["Jonathan Lozinski", "Alex Ouzounis", "Chris Rust"]
9
9
  spec.email = ["jonathan.lozinski@solarwinds.com", "alex.ouzounis@solarwinds.com", "chris.rust@solarwinds.com"]
10
10
 
@@ -12,10 +12,14 @@ module Fluent
12
12
  config_param :default_hostname, :string, default: 'unidentified'
13
13
  # overriding default flush_interval (60 sec) from Fluent::BufferedOutput
14
14
  config_param :flush_interval, :time, default: 1
15
+ config_param :discard_unannotated_pod_logs, :bool, default: false
15
16
 
16
17
  # register as 'papertrail' fluent plugin
17
18
  Fluent::Plugin.register_output('papertrail', self)
18
19
 
20
+ # declare const string for nullifying token if we decide to discard records
21
+ DISCARD_STRING = 'DISCARD'
22
+
19
23
  def configure(conf)
20
24
  super
21
25
  # create initial sockets hash and socket based on config param
@@ -33,8 +37,10 @@ module Fluent
33
37
  def write(chunk)
34
38
  chunk.msgpack_each {|(tag, time, record)|
35
39
  socket_key = pick_socket(record)
36
- packet = create_packet(tag, time, record)
37
- send_to_papertrail(packet, socket_key)
40
+ unless socket_key.eql? "#{DISCARD_STRING}:#{DISCARD_STRING}"
41
+ packet = create_packet(tag, time, record)
42
+ send_to_papertrail(packet, socket_key)
43
+ end
38
44
  }
39
45
  end
40
46
 
@@ -78,6 +84,10 @@ module Fluent
78
84
  record.dig('kubernetes', 'namespace_annotations', 'solarwinds_io/papertrail_port')
79
85
  host = record['kubernetes']['namespace_annotations']['solarwinds_io/papertrail_host']
80
86
  port = record['kubernetes']['namespace_annotations']['solarwinds_io/papertrail_port']
87
+ # else if it is a kubernetes log and we're discarding unannotated logs
88
+ elsif record.dig('kubernetes') && @discard_unannotated_pod_logs
89
+ host = DISCARD_STRING
90
+ port = DISCARD_STRING
81
91
  # else use pre-configured destination
82
92
  else
83
93
  host = @papertrail_host
@@ -92,7 +102,6 @@ module Fluent
92
102
  def send_to_papertrail(packet, socket_key)
93
103
  if @sockets[socket_key].nil?
94
104
  err_msg = "Unable to create socket with #{socket_key}"
95
- log.error err_msg
96
105
  raise SocketFailureError, err_msg
97
106
  else
98
107
  begin
@@ -100,7 +109,6 @@ module Fluent
100
109
  @sockets[socket_key].puts packet.assemble
101
110
  rescue => e
102
111
  err_msg = "Error writing to #{socket_key}: #{e}"
103
- log.error err_msg
104
112
  # socket failed, reset to nil to recreate for the next write
105
113
  @sockets[socket_key] = nil
106
114
  raise SocketFailureError, err_msg, e.backtrace
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-papertrail
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.4
4
+ version: 0.2.5.pre.dev
5
5
  platform: ruby
6
6
  authors:
7
7
  - Jonathan Lozinski
@@ -10,7 +10,7 @@ authors:
10
10
  autorequire:
11
11
  bindir: exe
12
12
  cert_chain: []
13
- date: 2018-05-09 00:00:00.000000000 Z
13
+ date: 2018-07-13 00:00:00.000000000 Z
14
14
  dependencies:
15
15
  - !ruby/object:Gem::Dependency
16
16
  name: fluentd
@@ -132,13 +132,7 @@ files:
132
132
  - Makefile
133
133
  - README.md
134
134
  - Rakefile
135
- - docker/Dockerfile
136
- - docker/conf/fluent.conf
137
- - docker/conf/kubernetes.conf
138
- - docker/conf/systemd.conf
139
- - docker/plugins/parser_kubernetes.rb
140
135
  - fluent-plugin-papertrail.gemspec
141
- - kubernetes/fluentd-daemonset-papertrail.yaml
142
136
  - lib/fluent/plugin/out_papertrail.rb
143
137
  homepage: https://github.com/solarwinds/fluent-plugin-papertrail
144
138
  licenses:
@@ -155,9 +149,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
155
149
  version: '0'
156
150
  required_rubygems_version: !ruby/object:Gem::Requirement
157
151
  requirements:
158
- - - ">="
152
+ - - ">"
159
153
  - !ruby/object:Gem::Version
160
- version: '0'
154
+ version: 1.3.1
161
155
  requirements: []
162
156
  rubyforge_project:
163
157
  rubygems_version: 2.5.2.2
@@ -1,45 +0,0 @@
1
- FROM fluent/fluentd:v1.1.1-debian
2
-
3
- USER root
4
- WORKDIR /home/fluent
5
- ENV PATH /home/fluent/.gem/ruby/2.3.0/bin:$PATH
6
- ENV BUILDDEPS="sudo make gcc g++ libc-dev ruby-dev libffi-dev"
7
-
8
- RUN apt-get update \
9
- && apt-get upgrade -y \
10
- && apt-get install -y --no-install-recommends \
11
- $BUILDDEPS
12
-
13
- RUN echo 'gem: --no-document' >> /etc/gemrc \
14
- && gem install fluent-plugin-systemd -v 0.3.1 \
15
- && gem install fluent-plugin-rewrite-tag-filter -v 2.0.2 \
16
- && gem install fluent-plugin-papertrail -v 0.2.3 \
17
- && gem install fluent-plugin-loggly-anno -v 0.0.2 \
18
- && gem install fluent-plugin-kubernetes_metadata_input -v 0.21.11 \
19
- && gem install fluent-plugin-kubernetes_metadata_filter -v 2.0.0
20
-
21
- RUN SUDO_FORCE_REMOVE=yes \
22
- apt-get purge -y --auto-remove \
23
- -o APT::AutoRemove::RecommendsImportant=false \
24
- $BUILDDEPS \
25
- && rm -rf /var/lib/apt/lists/* \
26
- && gem sources --clear-all \
27
- && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
28
-
29
- # Copy configuration files
30
- COPY ./conf/fluent.conf /fluentd/etc/
31
- COPY ./conf/systemd.conf /fluentd/etc/
32
- COPY ./conf/kubernetes.conf /fluentd/etc/
33
-
34
- # Copy plugins
35
- COPY plugins /fluentd/plugins/
36
-
37
- # Environment variables
38
- ENV FLUENTD_OPT=""
39
- ENV FLUENTD_CONF="fluent.conf"
40
-
41
- # Overriding entrypoint, otherwise parent image causes to run as user fluent
42
- ENTRYPOINT []
43
-
44
- # Run Fluentd
45
- CMD exec fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT
@@ -1,18 +0,0 @@
1
- @include systemd.conf
2
- @include kubernetes.conf
3
-
4
- ## Capture audit logs
5
- #<match kube-apiserver-audit>
6
- # @type papertrail
7
- #
8
- # papertrail_host "#{ENV['FLUENT_PAPERTRAIL_AUDIT_HOST']}"
9
- # papertrail_port "#{ENV['FLUENT_PAPERTRAIL_AUDIT_PORT']}"
10
- #</match>
11
-
12
- <match **>
13
- @type papertrail
14
-
15
- papertrail_host "#{ENV['FLUENT_PAPERTRAIL_HOST']}"
16
- papertrail_port "#{ENV['FLUENT_PAPERTRAIL_PORT']}"
17
-
18
- </match>
@@ -1,171 +0,0 @@
1
- <match fluent.**>
2
- @type null
3
- </match>
4
-
5
- <source>
6
- @type tail
7
- path /var/log/containers/*.log
8
- pos_file /var/log/fluentd-containers.log.pos
9
- time_format %Y-%m-%dT%H:%M:%S.%NZ
10
- tag kubernetes.*
11
- format json
12
- read_from_head true
13
- </source>
14
-
15
- <source>
16
- @type tail
17
- format /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
18
- time_format %Y-%m-%d %H:%M:%S
19
- path /var/log/salt/minion
20
- pos_file /var/log/fluentd-salt.pos
21
- tag salt
22
- </source>
23
-
24
- <source>
25
- @type tail
26
- format syslog
27
- path /var/log/startupscript.log
28
- pos_file /var/log/fluentd-startupscript.log.pos
29
- tag startupscript
30
- </source>
31
-
32
- <source>
33
- @type tail
34
- format /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
35
- path /var/log/docker.log
36
- pos_file /var/log/fluentd-docker.log.pos
37
- tag docker
38
- </source>
39
-
40
- <source>
41
- @type tail
42
- format none
43
- path /var/log/etcd.log
44
- pos_file /var/log/fluentd-etcd.log.pos
45
- tag etcd
46
- </source>
47
-
48
- <source>
49
- @type tail
50
- format kubernetes
51
- multiline_flush_interval 5s
52
- path /var/log/kubelet.log
53
- pos_file /var/log/fluentd-kubelet.log.pos
54
- tag kubelet
55
- </source>
56
-
57
- <source>
58
- @type tail
59
- format kubernetes
60
- multiline_flush_interval 5s
61
- path /var/log/kube-proxy.log
62
- pos_file /var/log/fluentd-kube-proxy.log.pos
63
- tag kube-proxy
64
- </source>
65
-
66
- <source>
67
- @type tail
68
- format kubernetes
69
- multiline_flush_interval 5s
70
- path /var/log/kube-apiserver.log
71
- pos_file /var/log/fluentd-kube-apiserver.log.pos
72
- tag kube-apiserver
73
- </source>
74
-
75
- <source>
76
- @type tail
77
- format kubernetes
78
- multiline_flush_interval 5s
79
- path /var/log/kube-controller-manager.log
80
- pos_file /var/log/fluentd-kube-controller-manager.log.pos
81
- tag kube-controller-manager
82
- </source>
83
-
84
- <source>
85
- @type tail
86
- format kubernetes
87
- multiline_flush_interval 5s
88
- path /var/log/kube-scheduler.log
89
- pos_file /var/log/fluentd-kube-scheduler.log.pos
90
- tag kube-scheduler
91
- </source>
92
-
93
- <source>
94
- @type tail
95
- format kubernetes
96
- multiline_flush_interval 5s
97
- path /var/log/rescheduler.log
98
- pos_file /var/log/fluentd-rescheduler.log.pos
99
- tag rescheduler
100
- </source>
101
-
102
- <source>
103
- @type tail
104
- format kubernetes
105
- multiline_flush_interval 5s
106
- path /var/log/glbc.log
107
- pos_file /var/log/fluentd-glbc.log.pos
108
- tag glbc
109
- </source>
110
-
111
- <source>
112
- @type tail
113
- format kubernetes
114
- multiline_flush_interval 5s
115
- path /var/log/cluster-autoscaler.log
116
- pos_file /var/log/fluentd-cluster-autoscaler.log.pos
117
- tag cluster-autoscaler
118
- </source>
119
-
120
- # Example:
121
- # 2017-02-09T00:15:57.992775796Z AUDIT: id="90c73c7c-97d6-4b65-9461-f94606ff825f" ip="104.132.1.72" method="GET" user="kubecfg" as="<self>" asgroups="<lookup>" namespace="default" uri="/api/v1/namespaces/default/pods"
122
- # 2017-02-09T00:15:57.993528822Z AUDIT: id="90c73c7c-97d6-4b65-9461-f94606ff825f" response="200"
123
- <source>
124
- @type tail
125
- format multiline
126
- multiline_flush_interval 5s
127
- format_firstline /^\S+\s+AUDIT:/
128
- # Fields must be explicitly captured by name to be parsed into the record.
129
- # Fields may not always be present, and order may change, so this just looks
130
- # for a list of key="\"quoted\" value" pairs separated by spaces.
131
- # Unknown fields are ignored.
132
- # Note: We can't separate query/response lines as format1/format2 because
133
- # they don't always come one after the other for a given query.
134
- format1 /^(?<time>\S+) AUDIT:(?: (?:id="(?<id>(?:[^"\\]|\\.)*)"|ip="(?<ip>(?:[^"\\]|\\.)*)"|method="(?<method>(?:[^"\\]|\\.)*)"|user="(?<user>(?:[^"\\]|\\.)*)"|groups="(?<groups>(?:[^"\\]|\\.)*)"|as="(?<as>(?:[^"\\]|\\.)*)"|asgroups="(?<asgroups>(?:[^"\\]|\\.)*)"|namespace="(?<namespace>(?:[^"\\]|\\.)*)"|uri="(?<uri>(?:[^"\\]|\\.)*)"|response="(?<response>(?:[^"\\]|\\.)*)"|\w+="(?:[^"\\]|\\.)*"))*/
135
- time_format %FT%T.%L%Z
136
- path /var/log/kubernetes/kube-apiserver-audit.log
137
- pos_file /var/log/kube-apiserver-audit.log.pos
138
- tag kube-apiserver-audit
139
- </source>
140
-
141
- <filter kubernetes.**>
142
- @type kubernetes_metadata
143
- annotation_match ["solarwinds.io/*"]
144
- </filter>
145
-
146
- <filter kube-apiserver-audit>
147
- @type record_transformer
148
- enable_ruby true
149
- <record>
150
- hostname #{ENV['FLUENT_HOSTNAME']}
151
- program kube-apiserver-audit
152
- severity info
153
- facility local0
154
- message ${record}
155
- </record>
156
- </filter>
157
-
158
- # append namespace and pod name to hostname, so that logs in Papertrail are filterable by each
159
- # use container name as program name, but trim it to 32 characters to match remote_syslog spec
160
- <filter kubernetes.**>
161
- @type record_transformer
162
- enable_ruby true
163
- <record>
164
- hostname #{ENV['FLUENT_HOSTNAME']}-${record["kubernetes"]["namespace_name"]}-${record["kubernetes"]["pod_name"]}
165
- program ${record["kubernetes"]["container_name"][0..31]}
166
- severity info
167
- facility local0
168
- message ${record['log']}
169
- </record>
170
- remove_keys ["log"]
171
- </filter>
@@ -1,61 +0,0 @@
1
- <source>
2
- @type systemd
3
- tag systemd
4
- read_from_head true
5
- <storage>
6
- @type local
7
- persistent true
8
- path /var/log/fluentd-journald-systemd.pos
9
- </storage>
10
- </source>
11
-
12
- # rewrite tags as systemd.* for the specefic SYSTEMD_UNIT, then we can filter specifically on kubelet and docker below
13
- <match systemd>
14
- @type rewrite_tag_filter
15
- <rule>
16
- key SYSTEMD_UNIT
17
- pattern /^kubelet.service$/
18
- tag systemd.kubelet
19
- </rule>
20
- <rule>
21
- key SYSTEMD_UNIT
22
- pattern /^docker.service$/
23
- tag systemd.docker
24
- </rule>
25
- </match>
26
-
27
- # toss all other systemd logs in the bin
28
- <match systemd>
29
- @type null
30
- </match>
31
-
32
- # transform systemd logs to Papertrail format
33
- <filter systemd.kubelet>
34
- @type record_transformer
35
- enable_ruby true
36
- <record>
37
- hostname "#{ENV['FLUENT_HOSTNAME']}-${record['HOSTNAME']}"
38
- program kubelet
39
- severity info
40
- facility local0
41
- message ${record['MESSAGE']}
42
- </record>
43
- </filter>
44
-
45
- <filter systemd.docker>
46
- @type parser
47
- format /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
48
- reserve_data true
49
- key_name MESSAGE
50
- </filter>
51
-
52
- <filter systemd.docker>
53
- @type record_transformer
54
- enable_ruby true
55
- <record>
56
- hostname "#{ENV['FLUENT_HOSTNAME']}-${record['HOSTNAME']}"
57
- program docker
58
- severity info
59
- facility local0
60
- </record>
61
- </filter>
@@ -1,65 +0,0 @@
1
- #
2
- # Fluentd
3
- #
4
- # Licensed under the Apache License, Version 2.0 (the "License");
5
- # you may not use this file except in compliance with the License.
6
- # You may obtain a copy of the License at
7
- #
8
- # http://www.apache.org/licenses/LICENSE-2.0
9
- #
10
- # Unless required by applicable law or agreed to in writing, software
11
- # distributed under the License is distributed on an "AS IS" BASIS,
12
- # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
- # See the License for the specific language governing permissions and
14
- # limitations under the License.
15
- #
16
-
17
- # The following Fluentd parser plugin, aims to simplify the parsing of multiline
18
- # logs found in Kubernetes nodes. Since many log files shared the same format and
19
- # in order to simplify the configuration, this plugin provides a 'kubernetes' format
20
- # parser (built on top of MultilineParser).
21
- #
22
- # When tailing files, this 'kubernetes' format should be applied to the following
23
- # log file sources:
24
- #
25
- # - /var/log/kubelet.log
26
- # - /var/log/kube-proxy.log
27
- # - /var/log/kube-apiserver.log
28
- # - /var/log/kube-controller-manager.log
29
- # - /var/log/kube-scheduler.log
30
- # - /var/log/rescheduler.log
31
- # - /var/log/glbc.log
32
- # - /var/log/cluster-autoscaler.log
33
- #
34
- # Usage:
35
- #
36
- # ---- fluentd.conf ----
37
- #
38
- # <source>
39
- # type tail
40
- # format kubernetes
41
- # path ./kubelet.log
42
- # read_from_head yes
43
- # tag kubelet
44
- # </source>
45
- #
46
- # ---- EOF ---
47
-
48
- require 'fluent/parser'
49
-
50
- module Fluent
51
- class KubernetesParser < Fluent::TextParser::MultilineParser
52
- Fluent::Plugin.register_parser("kubernetes", self)
53
-
54
- CONF_FORMAT_FIRSTLINE = %q{/^\w\d{4}/}
55
- CONF_FORMAT1 = %q{/^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/}
56
- CONF_TIME_FORMAT = "%m%d %H:%M:%S.%N"
57
-
58
- def configure(conf)
59
- conf['format_firstline'] = CONF_FORMAT_FIRSTLINE
60
- conf['format1'] = CONF_FORMAT1
61
- conf['time_format'] = CONF_TIME_FORMAT
62
- super
63
- end
64
- end
65
- end
@@ -1,85 +0,0 @@
1
- apiVersion: extensions/v1beta1
2
- kind: DaemonSet
3
- metadata:
4
- name: fluentd-papertrail-global-logging
5
- namespace: kube-system
6
- labels:
7
- k8s-app: fluentd-logging
8
- version: v1
9
- kubernetes.io/cluster-service: "true"
10
- spec:
11
- updateStrategy:
12
- type: RollingUpdate
13
- template:
14
- metadata:
15
- labels:
16
- k8s-app: fluentd-logging
17
- version: v1
18
- kubernetes.io/cluster-service: "true"
19
- spec:
20
- tolerations:
21
- - key: node-role.kubernetes.io/master
22
- effect: NoSchedule
23
- containers:
24
- - name: fluentd
25
- image: quay.io/solarwinds/fluentd-kubernetes:v1.1.1-debian-papertrail-0.2.3
26
- imagePullPolicy: Always
27
- env:
28
- - name: FLUENT_PAPERTRAIL_HOST
29
- value: 'logsN.papertrailapp.com'
30
- - name: FLUENT_PAPERTRAIL_PORT
31
- value: 'NNNNN'
32
- - name: FLUENT_HOSTNAME
33
- value: 'my-cluster-name'
34
- - name: K8S_NODE_NAME
35
- valueFrom:
36
- fieldRef:
37
- fieldPath: spec.nodeName
38
- resources:
39
- limits:
40
- cpu: 200m
41
- memory: 400Mi
42
- requests:
43
- cpu: 200m
44
- memory: 400Mi
45
- volumeMounts:
46
- - name: varlog
47
- mountPath: /var/log
48
- - name: varlibdockercontainers
49
- mountPath: /var/lib/docker/containers
50
- readOnly: true
51
- - name: config
52
- mountPath: /fluentd/etc/fluent.conf
53
- subPath: fluent.conf
54
- terminationGracePeriodSeconds: 30
55
- volumes:
56
- - name: varlog
57
- hostPath:
58
- path: /var/log
59
- - name: varlibdockercontainers
60
- hostPath:
61
- path: /var/lib/docker/containers
62
- - name: config
63
- configMap:
64
- name: fluentd-papertrail-global-logging-config
65
- items:
66
- - key: fluent.conf
67
- path: fluent.conf
68
- ---
69
- apiVersion: v1
70
- kind: ConfigMap
71
- metadata:
72
- name: fluentd-papertrail-global-logging-config
73
- namespace: kube-system
74
- data:
75
- fluent.conf: |-
76
- @include kubernetes.conf
77
- @include systemd.conf
78
-
79
- <match **>
80
- type papertrail
81
- num_threads 4
82
-
83
- papertrail_host "#{ENV['FLUENT_PAPERTRAIL_HOST']}"
84
- papertrail_port "#{ENV['FLUENT_PAPERTRAIL_PORT']}"
85
- </match>