fluent-plugin-papertrail 0.2.4 → 0.2.5.pre.dev

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 3413dd7623c0aab24484393559a57199331c9830
4
- data.tar.gz: 0f21d2987c62bc31dd38c35bcec83dcce3bc6821
3
+ metadata.gz: 297b4941b70100a4be1be6fb5e35f14263553202
4
+ data.tar.gz: 80fccb4690f183f69aabc9da7a793fbbe357e19d
5
5
  SHA512:
6
- metadata.gz: f31eab178bfbed278862bbc68e18f2493e1fc9ab60b84a8f2dbb67ed4a3f8a0b74cb1587c5111cae65ebc14ad9d351731a37eb86178ab06ba366402d41ef7156
7
- data.tar.gz: 33eec86f6cbee0ff4f0bdc91c41028f3d3371aece5b71c6b82d8dcfc8db9c61e8248d0556f692e347c05055e4ee77976bcea9631bc91e987e5a68e24d647b519
6
+ metadata.gz: 4ed8cae9ae77236c3c64d3f4537e2a5274a67050bcf5478a5d60a90a0ac9cc795cb54c79936eb5ec0a6459c1825a8ff5fe2981e1ac9fbd1f482b1f7f9545ccea
7
+ data.tar.gz: 9a575ad74e1759185008d11196d712f010f954ba9467ea2829668c461b712790fc99cc4c7f0d439eb23a5f18f2cfff96e7bb3ea24c5db8444bfd12160e223a4f
data/Makefile CHANGED
@@ -10,7 +10,3 @@ release: bundle
10
10
  rm -rf ${REPO_NAME}-*.gem
11
11
  bundle exec gem build ${REPO_NAME}.gemspec
12
12
  bundle exec gem push ${REPO_NAME}-*.gem
13
-
14
- release-docker:
15
- cd docker; docker build -t quay.io/solarwinds/fluentd-kubernetes:$(TAG) .
16
- docker push quay.io/solarwinds/fluentd-kubernetes:$(TAG)
data/README.md CHANGED
@@ -1,10 +1,10 @@
1
1
  # Fluent::Plugin::Papertrail
2
2
 
3
- [![Gem Version](https://badge.fury.io/rb/fluent-plugin-papertrail.svg)](https://badge.fury.io/rb/fluent-plugin-papertrail) [![Docker Repository on Quay](https://quay.io/repository/solarwinds/fluentd-kubernetes/status "Docker Repository on Quay")](https://quay.io/repository/solarwinds/fluentd-kubernetes) [![CircleCI](https://circleci.com/gh/solarwinds/fluent-plugin-papertrail/tree/master.svg?style=shield)](https://circleci.com/gh/solarwinds/fluent-plugin-papertrail/tree/master)
3
+ [![Gem Version](https://badge.fury.io/rb/fluent-plugin-papertrail.svg)](https://badge.fury.io/rb/fluent-plugin-papertrail) [![CircleCI](https://circleci.com/gh/solarwinds/fluent-plugin-papertrail/tree/master.svg?style=shield)](https://circleci.com/gh/solarwinds/fluent-plugin-papertrail/tree/master)
4
4
 
5
5
  ## Description
6
6
 
7
- This repository contains the Fluentd Papertrail Output Plugin and the Docker and Kubernetes assets for deploying that combined Fluentd, Papertrail, Kubernetes log aggregation toolset to your cluster.
7
+ This repository contains the Fluentd Papertrail Output Plugin.
8
8
 
9
9
  ## Installation
10
10
 
@@ -42,7 +42,7 @@ This plugin expects the following fields to be set for each Fluent record:
42
42
  hostname The source hostname for papertrail logging
43
43
  ```
44
44
 
45
- The following example is a `record_transformer` filter, from the [Kubernetes assets](docker/conf/kubernetes.conf) in this repo, that is used along with the [fluent-plugin-kubernetes_metadata_filter](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter) to populate the required fields for our plugin:
45
+ The following example is a `record_transformer` filter, from the Kubernetes assets [in the Solarwinds fluentd-deployment repo](https://github.com/solarwinds/fluentd-deployment/blob/master/docker/conf/kubernetes.conf), that is used along with the [fluent-plugin-kubernetes_metadata_filter](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter) to populate the required fields for our plugin:
46
46
  ```yaml
47
47
  <filter kubernetes.**>
48
48
  type kubernetes_metadata
@@ -81,21 +81,9 @@ If you want to change any of these parameters simply add them to a match stanza.
81
81
  </match>
82
82
  ```
83
83
 
84
- ## Kubernetes
84
+ ## Kubernetes Annotations
85
85
 
86
- This repo includes a Kubernetes DaemonSet and accompanying Docker container which will stream all of your Kubernetes logs, containers and services, to Papertrail.
87
-
88
- To deploy this plugin as a DaemonSet to your Kubernetes cluster, just adjust the `FLUENT_*` environment variables in `kubernetes/fluentd-daemonset-papertrail.yaml` and push it to your cluster with:
89
-
90
- ```
91
- kubectl apply -f kubernetes/fluentd-daemonset-papertrail.yaml
92
- ```
93
-
94
- The Dockerfile that generates [the image used in this DaemonSet](https://quay.io/repository/solarwinds/fluentd-kubernetes), can be found at `docker/Dockerfile`.
95
-
96
- ### Annotations
97
-
98
- You can redirect logs to alternate Papertrail destinations by adding annotations to your Pods or Namespaces:
86
+ If you're running this plugin in Kubernetes with the kubernetes_metadata_filter plugin enabled you can redirect logs to alternate Papertrail destinations by adding annotations to your Pods or Namespaces:
99
87
 
100
88
  ```
101
89
  solarwinds.io/papertrail_host: 'logs0.papertrailapp.com'
@@ -104,21 +92,6 @@ solarwinds.io/papertrail_port: '12345'
104
92
 
105
93
  If both the Pod and Namespace have annotations for any running Pod, the Pod's annotation is used.
106
94
 
107
- ### Audit Logs
108
-
109
- If you'd like to redirect Kubernetes API Server Audit logs to a seperate Papertrail destination, add the following to your `fluent.conf`:
110
- ```
111
- <match kube-apiserver-audit>
112
- type papertrail
113
- num_threads 4
114
-
115
- papertrail_host "#{ENV['FLUENT_PAPERTRAIL_AUDIT_HOST']}"
116
- papertrail_port "#{ENV['FLUENT_PAPERTRAIL_AUDIT_PORT']}"
117
- </match>
118
- ```
119
-
120
- This requires you to configure an [audit policy file](https://kubernetes.io/docs/tasks/debug-application-cluster/audit/) on your cluster.
121
-
122
95
  ## Development
123
96
 
124
97
  This plugin is targeting Ruby 2.4 and Fluentd v1.0, although it should work with older versions of both.
@@ -136,10 +109,6 @@ To release a new version, update the version number in the [GemSpec](fluent-plug
136
109
 
137
110
  `make release`
138
111
 
139
- ### Release in [Quay.io](https://quay.io/repository/solarwinds/fluentd-kubernetes)
140
-
141
- `make release-docker TAG=$(VERSION)`
142
-
143
112
  ## Contributing
144
113
 
145
114
  Bug reports and pull requests are welcome on GitHub at: https://github.com/solarwinds/fluent-plugin-papertrail
@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
4
 
5
5
  Gem::Specification.new do |spec|
6
6
  spec.name = "fluent-plugin-papertrail"
7
- spec.version = "0.2.4"
7
+ spec.version = "0.2.5-dev"
8
8
  spec.authors = ["Jonathan Lozinski", "Alex Ouzounis", "Chris Rust"]
9
9
  spec.email = ["jonathan.lozinski@solarwinds.com", "alex.ouzounis@solarwinds.com", "chris.rust@solarwinds.com"]
10
10
 
@@ -12,10 +12,14 @@ module Fluent
12
12
  config_param :default_hostname, :string, default: 'unidentified'
13
13
  # overriding default flush_interval (60 sec) from Fluent::BufferedOutput
14
14
  config_param :flush_interval, :time, default: 1
15
+ config_param :discard_unannotated_pod_logs, :bool, default: false
15
16
 
16
17
  # register as 'papertrail' fluent plugin
17
18
  Fluent::Plugin.register_output('papertrail', self)
18
19
 
20
+ # declare const string for nullifying token if we decide to discard records
21
+ DISCARD_STRING = 'DISCARD'
22
+
19
23
  def configure(conf)
20
24
  super
21
25
  # create initial sockets hash and socket based on config param
@@ -33,8 +37,10 @@ module Fluent
33
37
  def write(chunk)
34
38
  chunk.msgpack_each {|(tag, time, record)|
35
39
  socket_key = pick_socket(record)
36
- packet = create_packet(tag, time, record)
37
- send_to_papertrail(packet, socket_key)
40
+ unless socket_key.eql? "#{DISCARD_STRING}:#{DISCARD_STRING}"
41
+ packet = create_packet(tag, time, record)
42
+ send_to_papertrail(packet, socket_key)
43
+ end
38
44
  }
39
45
  end
40
46
 
@@ -78,6 +84,10 @@ module Fluent
78
84
  record.dig('kubernetes', 'namespace_annotations', 'solarwinds_io/papertrail_port')
79
85
  host = record['kubernetes']['namespace_annotations']['solarwinds_io/papertrail_host']
80
86
  port = record['kubernetes']['namespace_annotations']['solarwinds_io/papertrail_port']
87
+ # else if it is a kubernetes log and we're discarding unannotated logs
88
+ elsif record.dig('kubernetes') && @discard_unannotated_pod_logs
89
+ host = DISCARD_STRING
90
+ port = DISCARD_STRING
81
91
  # else use pre-configured destination
82
92
  else
83
93
  host = @papertrail_host
@@ -92,7 +102,6 @@ module Fluent
92
102
  def send_to_papertrail(packet, socket_key)
93
103
  if @sockets[socket_key].nil?
94
104
  err_msg = "Unable to create socket with #{socket_key}"
95
- log.error err_msg
96
105
  raise SocketFailureError, err_msg
97
106
  else
98
107
  begin
@@ -100,7 +109,6 @@ module Fluent
100
109
  @sockets[socket_key].puts packet.assemble
101
110
  rescue => e
102
111
  err_msg = "Error writing to #{socket_key}: #{e}"
103
- log.error err_msg
104
112
  # socket failed, reset to nil to recreate for the next write
105
113
  @sockets[socket_key] = nil
106
114
  raise SocketFailureError, err_msg, e.backtrace
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-papertrail
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.4
4
+ version: 0.2.5.pre.dev
5
5
  platform: ruby
6
6
  authors:
7
7
  - Jonathan Lozinski
@@ -10,7 +10,7 @@ authors:
10
10
  autorequire:
11
11
  bindir: exe
12
12
  cert_chain: []
13
- date: 2018-05-09 00:00:00.000000000 Z
13
+ date: 2018-07-13 00:00:00.000000000 Z
14
14
  dependencies:
15
15
  - !ruby/object:Gem::Dependency
16
16
  name: fluentd
@@ -132,13 +132,7 @@ files:
132
132
  - Makefile
133
133
  - README.md
134
134
  - Rakefile
135
- - docker/Dockerfile
136
- - docker/conf/fluent.conf
137
- - docker/conf/kubernetes.conf
138
- - docker/conf/systemd.conf
139
- - docker/plugins/parser_kubernetes.rb
140
135
  - fluent-plugin-papertrail.gemspec
141
- - kubernetes/fluentd-daemonset-papertrail.yaml
142
136
  - lib/fluent/plugin/out_papertrail.rb
143
137
  homepage: https://github.com/solarwinds/fluent-plugin-papertrail
144
138
  licenses:
@@ -155,9 +149,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
155
149
  version: '0'
156
150
  required_rubygems_version: !ruby/object:Gem::Requirement
157
151
  requirements:
158
- - - ">="
152
+ - - ">"
159
153
  - !ruby/object:Gem::Version
160
- version: '0'
154
+ version: 1.3.1
161
155
  requirements: []
162
156
  rubyforge_project:
163
157
  rubygems_version: 2.5.2.2
@@ -1,45 +0,0 @@
1
- FROM fluent/fluentd:v1.1.1-debian
2
-
3
- USER root
4
- WORKDIR /home/fluent
5
- ENV PATH /home/fluent/.gem/ruby/2.3.0/bin:$PATH
6
- ENV BUILDDEPS="sudo make gcc g++ libc-dev ruby-dev libffi-dev"
7
-
8
- RUN apt-get update \
9
- && apt-get upgrade -y \
10
- && apt-get install -y --no-install-recommends \
11
- $BUILDDEPS
12
-
13
- RUN echo 'gem: --no-document' >> /etc/gemrc \
14
- && gem install fluent-plugin-systemd -v 0.3.1 \
15
- && gem install fluent-plugin-rewrite-tag-filter -v 2.0.2 \
16
- && gem install fluent-plugin-papertrail -v 0.2.3 \
17
- && gem install fluent-plugin-loggly-anno -v 0.0.2 \
18
- && gem install fluent-plugin-kubernetes_metadata_input -v 0.21.11 \
19
- && gem install fluent-plugin-kubernetes_metadata_filter -v 2.0.0
20
-
21
- RUN SUDO_FORCE_REMOVE=yes \
22
- apt-get purge -y --auto-remove \
23
- -o APT::AutoRemove::RecommendsImportant=false \
24
- $BUILDDEPS \
25
- && rm -rf /var/lib/apt/lists/* \
26
- && gem sources --clear-all \
27
- && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
28
-
29
- # Copy configuration files
30
- COPY ./conf/fluent.conf /fluentd/etc/
31
- COPY ./conf/systemd.conf /fluentd/etc/
32
- COPY ./conf/kubernetes.conf /fluentd/etc/
33
-
34
- # Copy plugins
35
- COPY plugins /fluentd/plugins/
36
-
37
- # Environment variables
38
- ENV FLUENTD_OPT=""
39
- ENV FLUENTD_CONF="fluent.conf"
40
-
41
- # Overriding entrypoint, otherwise parent image causes to run as user fluent
42
- ENTRYPOINT []
43
-
44
- # Run Fluentd
45
- CMD exec fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT
@@ -1,18 +0,0 @@
1
- @include systemd.conf
2
- @include kubernetes.conf
3
-
4
- ## Capture audit logs
5
- #<match kube-apiserver-audit>
6
- # @type papertrail
7
- #
8
- # papertrail_host "#{ENV['FLUENT_PAPERTRAIL_AUDIT_HOST']}"
9
- # papertrail_port "#{ENV['FLUENT_PAPERTRAIL_AUDIT_PORT']}"
10
- #</match>
11
-
12
- <match **>
13
- @type papertrail
14
-
15
- papertrail_host "#{ENV['FLUENT_PAPERTRAIL_HOST']}"
16
- papertrail_port "#{ENV['FLUENT_PAPERTRAIL_PORT']}"
17
-
18
- </match>
@@ -1,171 +0,0 @@
1
- <match fluent.**>
2
- @type null
3
- </match>
4
-
5
- <source>
6
- @type tail
7
- path /var/log/containers/*.log
8
- pos_file /var/log/fluentd-containers.log.pos
9
- time_format %Y-%m-%dT%H:%M:%S.%NZ
10
- tag kubernetes.*
11
- format json
12
- read_from_head true
13
- </source>
14
-
15
- <source>
16
- @type tail
17
- format /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
18
- time_format %Y-%m-%d %H:%M:%S
19
- path /var/log/salt/minion
20
- pos_file /var/log/fluentd-salt.pos
21
- tag salt
22
- </source>
23
-
24
- <source>
25
- @type tail
26
- format syslog
27
- path /var/log/startupscript.log
28
- pos_file /var/log/fluentd-startupscript.log.pos
29
- tag startupscript
30
- </source>
31
-
32
- <source>
33
- @type tail
34
- format /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
35
- path /var/log/docker.log
36
- pos_file /var/log/fluentd-docker.log.pos
37
- tag docker
38
- </source>
39
-
40
- <source>
41
- @type tail
42
- format none
43
- path /var/log/etcd.log
44
- pos_file /var/log/fluentd-etcd.log.pos
45
- tag etcd
46
- </source>
47
-
48
- <source>
49
- @type tail
50
- format kubernetes
51
- multiline_flush_interval 5s
52
- path /var/log/kubelet.log
53
- pos_file /var/log/fluentd-kubelet.log.pos
54
- tag kubelet
55
- </source>
56
-
57
- <source>
58
- @type tail
59
- format kubernetes
60
- multiline_flush_interval 5s
61
- path /var/log/kube-proxy.log
62
- pos_file /var/log/fluentd-kube-proxy.log.pos
63
- tag kube-proxy
64
- </source>
65
-
66
- <source>
67
- @type tail
68
- format kubernetes
69
- multiline_flush_interval 5s
70
- path /var/log/kube-apiserver.log
71
- pos_file /var/log/fluentd-kube-apiserver.log.pos
72
- tag kube-apiserver
73
- </source>
74
-
75
- <source>
76
- @type tail
77
- format kubernetes
78
- multiline_flush_interval 5s
79
- path /var/log/kube-controller-manager.log
80
- pos_file /var/log/fluentd-kube-controller-manager.log.pos
81
- tag kube-controller-manager
82
- </source>
83
-
84
- <source>
85
- @type tail
86
- format kubernetes
87
- multiline_flush_interval 5s
88
- path /var/log/kube-scheduler.log
89
- pos_file /var/log/fluentd-kube-scheduler.log.pos
90
- tag kube-scheduler
91
- </source>
92
-
93
- <source>
94
- @type tail
95
- format kubernetes
96
- multiline_flush_interval 5s
97
- path /var/log/rescheduler.log
98
- pos_file /var/log/fluentd-rescheduler.log.pos
99
- tag rescheduler
100
- </source>
101
-
102
- <source>
103
- @type tail
104
- format kubernetes
105
- multiline_flush_interval 5s
106
- path /var/log/glbc.log
107
- pos_file /var/log/fluentd-glbc.log.pos
108
- tag glbc
109
- </source>
110
-
111
- <source>
112
- @type tail
113
- format kubernetes
114
- multiline_flush_interval 5s
115
- path /var/log/cluster-autoscaler.log
116
- pos_file /var/log/fluentd-cluster-autoscaler.log.pos
117
- tag cluster-autoscaler
118
- </source>
119
-
120
- # Example:
121
- # 2017-02-09T00:15:57.992775796Z AUDIT: id="90c73c7c-97d6-4b65-9461-f94606ff825f" ip="104.132.1.72" method="GET" user="kubecfg" as="<self>" asgroups="<lookup>" namespace="default" uri="/api/v1/namespaces/default/pods"
122
- # 2017-02-09T00:15:57.993528822Z AUDIT: id="90c73c7c-97d6-4b65-9461-f94606ff825f" response="200"
123
- <source>
124
- @type tail
125
- format multiline
126
- multiline_flush_interval 5s
127
- format_firstline /^\S+\s+AUDIT:/
128
- # Fields must be explicitly captured by name to be parsed into the record.
129
- # Fields may not always be present, and order may change, so this just looks
130
- # for a list of key="\"quoted\" value" pairs separated by spaces.
131
- # Unknown fields are ignored.
132
- # Note: We can't separate query/response lines as format1/format2 because
133
- # they don't always come one after the other for a given query.
134
- format1 /^(?<time>\S+) AUDIT:(?: (?:id="(?<id>(?:[^"\\]|\\.)*)"|ip="(?<ip>(?:[^"\\]|\\.)*)"|method="(?<method>(?:[^"\\]|\\.)*)"|user="(?<user>(?:[^"\\]|\\.)*)"|groups="(?<groups>(?:[^"\\]|\\.)*)"|as="(?<as>(?:[^"\\]|\\.)*)"|asgroups="(?<asgroups>(?:[^"\\]|\\.)*)"|namespace="(?<namespace>(?:[^"\\]|\\.)*)"|uri="(?<uri>(?:[^"\\]|\\.)*)"|response="(?<response>(?:[^"\\]|\\.)*)"|\w+="(?:[^"\\]|\\.)*"))*/
135
- time_format %FT%T.%L%Z
136
- path /var/log/kubernetes/kube-apiserver-audit.log
137
- pos_file /var/log/kube-apiserver-audit.log.pos
138
- tag kube-apiserver-audit
139
- </source>
140
-
141
- <filter kubernetes.**>
142
- @type kubernetes_metadata
143
- annotation_match ["solarwinds.io/*"]
144
- </filter>
145
-
146
- <filter kube-apiserver-audit>
147
- @type record_transformer
148
- enable_ruby true
149
- <record>
150
- hostname #{ENV['FLUENT_HOSTNAME']}
151
- program kube-apiserver-audit
152
- severity info
153
- facility local0
154
- message ${record}
155
- </record>
156
- </filter>
157
-
158
- # append namespace and pod name to hostname, so that logs in Papertrail are filterable by each
159
- # use container name as program name, but trim it to 32 characters to match remote_syslog spec
160
- <filter kubernetes.**>
161
- @type record_transformer
162
- enable_ruby true
163
- <record>
164
- hostname #{ENV['FLUENT_HOSTNAME']}-${record["kubernetes"]["namespace_name"]}-${record["kubernetes"]["pod_name"]}
165
- program ${record["kubernetes"]["container_name"][0..31]}
166
- severity info
167
- facility local0
168
- message ${record['log']}
169
- </record>
170
- remove_keys ["log"]
171
- </filter>
@@ -1,61 +0,0 @@
1
- <source>
2
- @type systemd
3
- tag systemd
4
- read_from_head true
5
- <storage>
6
- @type local
7
- persistent true
8
- path /var/log/fluentd-journald-systemd.pos
9
- </storage>
10
- </source>
11
-
12
- # rewrite tags as systemd.* for the specefic SYSTEMD_UNIT, then we can filter specifically on kubelet and docker below
13
- <match systemd>
14
- @type rewrite_tag_filter
15
- <rule>
16
- key SYSTEMD_UNIT
17
- pattern /^kubelet.service$/
18
- tag systemd.kubelet
19
- </rule>
20
- <rule>
21
- key SYSTEMD_UNIT
22
- pattern /^docker.service$/
23
- tag systemd.docker
24
- </rule>
25
- </match>
26
-
27
- # toss all other systemd logs in the bin
28
- <match systemd>
29
- @type null
30
- </match>
31
-
32
- # transform systemd logs to Papertrail format
33
- <filter systemd.kubelet>
34
- @type record_transformer
35
- enable_ruby true
36
- <record>
37
- hostname "#{ENV['FLUENT_HOSTNAME']}-${record['HOSTNAME']}"
38
- program kubelet
39
- severity info
40
- facility local0
41
- message ${record['MESSAGE']}
42
- </record>
43
- </filter>
44
-
45
- <filter systemd.docker>
46
- @type parser
47
- format /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
48
- reserve_data true
49
- key_name MESSAGE
50
- </filter>
51
-
52
- <filter systemd.docker>
53
- @type record_transformer
54
- enable_ruby true
55
- <record>
56
- hostname "#{ENV['FLUENT_HOSTNAME']}-${record['HOSTNAME']}"
57
- program docker
58
- severity info
59
- facility local0
60
- </record>
61
- </filter>
@@ -1,65 +0,0 @@
1
- #
2
- # Fluentd
3
- #
4
- # Licensed under the Apache License, Version 2.0 (the "License");
5
- # you may not use this file except in compliance with the License.
6
- # You may obtain a copy of the License at
7
- #
8
- # http://www.apache.org/licenses/LICENSE-2.0
9
- #
10
- # Unless required by applicable law or agreed to in writing, software
11
- # distributed under the License is distributed on an "AS IS" BASIS,
12
- # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
- # See the License for the specific language governing permissions and
14
- # limitations under the License.
15
- #
16
-
17
- # The following Fluentd parser plugin, aims to simplify the parsing of multiline
18
- # logs found in Kubernetes nodes. Since many log files shared the same format and
19
- # in order to simplify the configuration, this plugin provides a 'kubernetes' format
20
- # parser (built on top of MultilineParser).
21
- #
22
- # When tailing files, this 'kubernetes' format should be applied to the following
23
- # log file sources:
24
- #
25
- # - /var/log/kubelet.log
26
- # - /var/log/kube-proxy.log
27
- # - /var/log/kube-apiserver.log
28
- # - /var/log/kube-controller-manager.log
29
- # - /var/log/kube-scheduler.log
30
- # - /var/log/rescheduler.log
31
- # - /var/log/glbc.log
32
- # - /var/log/cluster-autoscaler.log
33
- #
34
- # Usage:
35
- #
36
- # ---- fluentd.conf ----
37
- #
38
- # <source>
39
- # type tail
40
- # format kubernetes
41
- # path ./kubelet.log
42
- # read_from_head yes
43
- # tag kubelet
44
- # </source>
45
- #
46
- # ---- EOF ---
47
-
48
- require 'fluent/parser'
49
-
50
- module Fluent
51
- class KubernetesParser < Fluent::TextParser::MultilineParser
52
- Fluent::Plugin.register_parser("kubernetes", self)
53
-
54
- CONF_FORMAT_FIRSTLINE = %q{/^\w\d{4}/}
55
- CONF_FORMAT1 = %q{/^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/}
56
- CONF_TIME_FORMAT = "%m%d %H:%M:%S.%N"
57
-
58
- def configure(conf)
59
- conf['format_firstline'] = CONF_FORMAT_FIRSTLINE
60
- conf['format1'] = CONF_FORMAT1
61
- conf['time_format'] = CONF_TIME_FORMAT
62
- super
63
- end
64
- end
65
- end
@@ -1,85 +0,0 @@
1
- apiVersion: extensions/v1beta1
2
- kind: DaemonSet
3
- metadata:
4
- name: fluentd-papertrail-global-logging
5
- namespace: kube-system
6
- labels:
7
- k8s-app: fluentd-logging
8
- version: v1
9
- kubernetes.io/cluster-service: "true"
10
- spec:
11
- updateStrategy:
12
- type: RollingUpdate
13
- template:
14
- metadata:
15
- labels:
16
- k8s-app: fluentd-logging
17
- version: v1
18
- kubernetes.io/cluster-service: "true"
19
- spec:
20
- tolerations:
21
- - key: node-role.kubernetes.io/master
22
- effect: NoSchedule
23
- containers:
24
- - name: fluentd
25
- image: quay.io/solarwinds/fluentd-kubernetes:v1.1.1-debian-papertrail-0.2.3
26
- imagePullPolicy: Always
27
- env:
28
- - name: FLUENT_PAPERTRAIL_HOST
29
- value: 'logsN.papertrailapp.com'
30
- - name: FLUENT_PAPERTRAIL_PORT
31
- value: 'NNNNN'
32
- - name: FLUENT_HOSTNAME
33
- value: 'my-cluster-name'
34
- - name: K8S_NODE_NAME
35
- valueFrom:
36
- fieldRef:
37
- fieldPath: spec.nodeName
38
- resources:
39
- limits:
40
- cpu: 200m
41
- memory: 400Mi
42
- requests:
43
- cpu: 200m
44
- memory: 400Mi
45
- volumeMounts:
46
- - name: varlog
47
- mountPath: /var/log
48
- - name: varlibdockercontainers
49
- mountPath: /var/lib/docker/containers
50
- readOnly: true
51
- - name: config
52
- mountPath: /fluentd/etc/fluent.conf
53
- subPath: fluent.conf
54
- terminationGracePeriodSeconds: 30
55
- volumes:
56
- - name: varlog
57
- hostPath:
58
- path: /var/log
59
- - name: varlibdockercontainers
60
- hostPath:
61
- path: /var/lib/docker/containers
62
- - name: config
63
- configMap:
64
- name: fluentd-papertrail-global-logging-config
65
- items:
66
- - key: fluent.conf
67
- path: fluent.conf
68
- ---
69
- apiVersion: v1
70
- kind: ConfigMap
71
- metadata:
72
- name: fluentd-papertrail-global-logging-config
73
- namespace: kube-system
74
- data:
75
- fluent.conf: |-
76
- @include kubernetes.conf
77
- @include systemd.conf
78
-
79
- <match **>
80
- type papertrail
81
- num_threads 4
82
-
83
- papertrail_host "#{ENV['FLUENT_PAPERTRAIL_HOST']}"
84
- papertrail_port "#{ENV['FLUENT_PAPERTRAIL_PORT']}"
85
- </match>