fluent-plugin-kubernetes_sumologic 0.0.1

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: ad8e3aa9817fe1b08fe011c5ce3965a0d8111ff9c54cbd0d06adcc6c179ef22c
4
+ data.tar.gz: 803b1a5027bd5f9863be35eb07c0c67e17c745bf73d6e97e548d49037e290c69
5
+ SHA512:
6
+ metadata.gz: 3614d3f18a3cee94a4c9caddacba78037deee9217fe022fe621ab061c3e43cdc7438b96621cb02697bc28dc893b22024c93096aed80b450ce520bce2e2999e4c
7
+ data.tar.gz: adb72a5a1d8db4d0e1b10e1ea8e91275b21081c6fc7c57e5967086768fb64e397b0378cef20b1042b073299f4fb278e560f23fbf11844fd194b9567b2233ed0c
data/.gitignore ADDED
@@ -0,0 +1,22 @@
1
+ # Ruby ignores
2
+ *.bridgesupport
3
+ *.gem
4
+ *.rbc
5
+ .dat*
6
+ .bundle/*
7
+ .ruby-gemset
8
+ .ruby-version
9
+ .rvmrc
10
+ bin/*
11
+ coverage
12
+ Gemfile.lock
13
+ TAGS
14
+
15
+ # Mac OSX ignores
16
+ .DS_Store
17
+
18
+ # VS Code ignores
19
+ .vscode/*
20
+
21
+ # idea ignores
22
+ .idea
data/.travis.yml ADDED
@@ -0,0 +1,13 @@
1
+ lang: ruby
2
+ services: docker
3
+ before_install: gem install bundler
4
+ script: ci/build.sh
5
+ deploy:
6
+ provider: rubygems
7
+ skip_cleanup: true
8
+ api_key:
9
+ secure: utz+1f2fW1CTu3rTcvpqz6kcQQak4+1Fb4MvlCHKE4nqjC2ujhxjagzXydFwBVuyrw+/VvFfNshZuO5Drz3qpvcLAqPd7ptru3bnYDiwMTFBiVg3SSVtTFphlNKuwNwM4alT/7Jhi3eW0LVXYAXPA3nhh7fOkTbJb9J2FcKD1/ty1ybgFVeUJlaJ8pcqA0Q4PD/dW/6CMkzt4x2CDgsE+MsufFg2SqGelxty1vWIk4oBa6PsDgAoauX8d9KUsa5MRFi7OWTrZviDKwfHRrTb99jW/k+4iQX0+gwIWDOQwTrajQreNIShvwciiA71MdQ60uFNPQYLretrt2C1cIyeB6Vyt/ozGREo6JfLpCNiwwS2PmWknDsa4nQFytrE6wH9qRQz89Q8vkr4dMlneVIgUDkQBVe7nVuatKOBMwN+ZxHLoDdVcGzLz2wWIzc7FcIiUWFphOiWmZB8G4+St+zOyVCTM25qHby9xs0DDveXyCLXCL2CCip7lG7ZnBaWrmW43YvLKKip2NGmlmi6oecsvT8IbCMxJoj5hGgJXbzYC8dAJ0ZATuVbQuzn94hhPwWWOLkcJ3xqg8jzEwoiMhYvzjvywTlKlc+lgglXIvQVfA4HXmttYX1WkbXuQD4CtdUgdB5JznsaVQo3n9UwD1iiAk/kuwWyKOOC+AmSX2SKlb0=
10
+ gem: fluent-plugin-kubernetes_sumologic
11
+ on:
12
+ tags: true
13
+ repo: SumoLogic/fluentd-kubernetes-sumologic
data/CHANGELOG.md ADDED
@@ -0,0 +1,101 @@
1
+ # Change Log
2
+
3
+ ## 2.3.0
4
+ - Polish "reduce metadata" feature with keeping `namespace_name` [PR #108](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/108)
5
+ - Bug fix - missing params in `containers.**` [PR #109](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/109)
6
+
7
+ ## 2.2.0
8
+ - bump base image to 1.3.2
9
+ - upgrade Sumo Logic FluentD Output plugin to 1.4.0, expose timestamp_key configuration
10
+
11
+ ## 2.1.0
12
+ - [Correctly remove the dynamic replicaset from the pod_name](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/100)
13
+ - [Control adding time and stream](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/96)
14
+ - [Reduce k8s metadata logs](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/92)
15
+
16
+ ## 2.0.0
17
+ - remove duplicate control plane logs to resolve #79. Considered a breaking change as this requires the Kubernetes App in Sumo Logic to be updated as it previously used these logs. Please [see here](README.md#upgrading-to-v200) for more information.
18
+ - set kubernetes metadata filter plugin logging to warn to remove noisy logs
19
+ - upgrade docker image to fluentd 1.2.6
20
+ - add rewrite-tag-filter and prometheus plugins
21
+ - test for existence of labels key in kubernetes metadata
22
+
23
+ ## 1.2.0
24
+ - change version scheme
25
+ - [Add labels to k8s_metadata](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/83)
26
+ - [Add source_host to conf.d/systemd/source.containers.conf](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/84)
27
+ - minor refactoring and testing
28
+ - upgrade fluent-plugin-sumologic_output to latest (1.3.1)
29
+ - expose additional configuration options to provide more control over fluent-plugin-kubernetes_metadata_filter using default values (fixes #80 and #35)
30
+
31
+ ## 1.18
32
+ - add FluentD S3 Output plugin to give more output options.
33
+
34
+ ## 1.17
35
+ - [fix #69 by upgrading sumologic fluentd output to 1.1.1 which contains fix to handle json parsing correctly](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/commit/498bd09fc2f353f0986ef3a9c583e4fb8ce7b401)
36
+
37
+ ## 1.16
38
+ -[Enable ability to configure enable_stat_watcher on in_tail plugins to workaround issue where inotify may hang.](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/commit/b963866aea0587d079913f87ca1a60b4d8afb982)
39
+
40
+ ## 1.15
41
+ -[Expose environment variable to configure proxy_uri in sumo out configuration](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/issues/65)
42
+ -[Allow container path for in_tail plugin to be configured via environment variable. Default to current value for backwards compatibility.](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/issues/64)
43
+
44
+ ## 1.14
45
+ -[kubernetes_sumologic plugin always adds a timestamp field to json logs](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/issues/61)
46
+
47
+ ## 1.13
48
+ -[Add ntp and timesyncd to systemd services](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/59#pullrequestreview-115374648)
49
+ -[Updates to new Audit Logging format](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/commit/90cc454927055cd337a91942b285e6b57264e8c5)
50
+
51
+ ## 1.12
52
+ -[Allow hand picked data to be sent to sumo when certain exclusions apply](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/58)
53
+
54
+ ## 1.11
55
+ -[upgrade metadata filter plugin to address #54](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/commit/eb7a69ce5b36a9d8150bd0a0b62f98fb18d35367)
56
+
57
+ ## 1.10
58
+ - [add missing exclude options to kubelet logs](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/commit/ced3304f1e64173554a8dee6367c785945f3ce99)
59
+ - [Use multi-stage build to eliminate build tools from final image](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/57)
60
+
61
+ ## 1.9
62
+ - base docker image was quite stale, this release bumps to 1.1.3 and anchors the plugins to specific versions. Same versions from 1.8 used, this just enforces consistency.
63
+ - ensure read_from_head in systemd respects setting via environment variable
64
+
65
+ ## 1.8
66
+ - [Change default FLUSH_INTERVAL to 5s](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/commit/7b100306d6c84335ee0d4ec6724a3218e8028893)
67
+ - [Handle possible timeout from Concat Plugin and ensure all logs resume the flow thru rest of pipeline when this occurs](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/commit/7b100306d6c84335ee0d4ec6724a3218e8028893)
68
+ - [Allow the time_key field to be defined via environment variables](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/53)
69
+
70
+ ## 1.7
71
+ - [Fix typo in sumologic kubernetes filter](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/51)
72
+
73
+ ## v1.6
74
+ - upgrade fluentd-sumo_output plugin to latest 1.0, adds support for millisecond precision.
75
+ - add support for kubernetes audit log
76
+
77
+ ## v1.5
78
+
79
+ - [Setting up tolerations](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/43)
80
+ - [add some etcdadm services](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/41)
81
+ - [Add RBAC permissions ](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/40)
82
+
83
+ ## v1.4
84
+
85
+ - [add support for multi-line log messages ](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/33)
86
+
87
+ ## v1.3
88
+
89
+ - [fix empty? checks on nil values by switching defaults to empty strings](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/32)
90
+
91
+ ## v1.2
92
+
93
+ - [Enable monitoring endpoints](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/28)
94
+
95
+ ## v1.1
96
+
97
+ - [Add support for systemd](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/pull/21).
98
+
99
+ ## v1.0
100
+
101
+ - Initial tag
data/Dockerfile ADDED
@@ -0,0 +1,70 @@
1
+ FROM fluent/fluentd:v1.3.2-debian AS builder
2
+
3
+ ENV PATH /home/fluent/.gem/ruby/2.3.0/bin:$PATH
4
+
5
+ # New fluent image dynamically creates user in entrypoint
6
+ RUN [ -f /bin/entrypoint.sh ] && /bin/entrypoint.sh echo || : && \
7
+ apt-get update && \
8
+ apt-get install -y build-essential ruby-dev libffi-dev libsystemd-dev && \
9
+ gem install fluent-plugin-s3 -v 1.1.4 && \
10
+ gem install fluent-plugin-systemd -v 0.3.1 && \
11
+ gem install fluent-plugin-record-reformer -v 0.9.1 && \
12
+ gem install fluent-plugin-kubernetes_metadata_filter -v 1.0.2 && \
13
+ gem install fluent-plugin-sumologic_output -v 1.4.0 && \
14
+ gem install fluent-plugin-concat -v 2.3.0 && \
15
+ gem install fluent-plugin-rewrite-tag-filter -v 2.1.0 && \
16
+ gem install fluent-plugin-prometheus -v 1.1.0 && \
17
+ rm -rf /home/fluent/.gem/ruby/2.3.0/cache/*.gem && \
18
+ gem sources -c && \
19
+ apt-get remove --purge -y build-essential ruby-dev libffi-dev libsystemd-dev && \
20
+ rm -rf /var/lib/apt/lists/*
21
+
22
+ FROM fluent/fluentd:v1.3.2-debian
23
+
24
+ WORKDIR /home/fluent
25
+ ENV PATH /home/fluent/.gem/ruby/2.3.0/bin:$PATH
26
+
27
+ RUN mkdir -p /mnt/pos
28
+ EXPOSE 24284
29
+
30
+ RUN mkdir -p /fluentd/conf.d && \
31
+ mkdir -p /fluentd/etc && \
32
+ mkdir -p /fluentd/plugins
33
+
34
+ # Default settings
35
+ ENV LOG_FORMAT "json"
36
+ ENV FLUSH_INTERVAL "5s"
37
+ ENV NUM_THREADS "1"
38
+ ENV SOURCE_CATEGORY "%{namespace}/%{pod_name}"
39
+ ENV SOURCE_CATEGORY_PREFIX "kubernetes/"
40
+ ENV SOURCE_CATEGORY_REPLACE_DASH "/"
41
+ ENV SOURCE_NAME "%{namespace}.%{pod}.%{container}"
42
+ ENV KUBERNETES_META "true"
43
+ ENV KUBERNETES_META_REDUCE "false"
44
+ ENV READ_FROM_HEAD "true"
45
+ ENV FLUENTD_SOURCE "file"
46
+ ENV FLUENTD_USER_CONFIG_DIR "/fluentd/conf.d/user"
47
+ ENV MULTILINE_START_REGEXP "/^\w{3} \d{1,2}, \d{4}/"
48
+ ENV CONCAT_SEPARATOR ""
49
+ ENV AUDIT_LOG_PATH "/mnt/log/kube-apiserver-audit.log"
50
+ ENV TIME_KEY "time"
51
+ ENV ADD_TIMESTAMP "true"
52
+ ENV TIMESTAMP_KEY "timestamp"
53
+ ENV ADD_STREAM "true"
54
+ ENV ADD_TIME "true"
55
+ ENV CONTAINER_LOGS_PATH "/mnt/log/containers/*.log"
56
+ ENV ENABLE_STAT_WATCHER "true"
57
+ ENV K8S_METADATA_FILTER_WATCH "true"
58
+ ENV K8S_METADATA_FILTER_VERIFY_SSL "true"
59
+ ENV K8S_METADATA_FILTER_BEARER_CACHE_SIZE "1000"
60
+ ENV K8S_METADATA_FILTER_BEARER_CACHE_TTL "3600"
61
+ ENV VERIFY_SSL "true"
62
+
63
+ COPY --from=builder /var/lib/gems /var/lib/gems
64
+ COPY ./conf.d/ /fluentd/conf.d/
65
+ COPY ./etc/* /fluentd/etc/
66
+ COPY ./entrypoint.sh /fluentd/
67
+ COPY ./fluent-plugin-kubernetes_sumologic*.gem ./
68
+ RUN gem install fluent-plugin-kubernetes_sumologic
69
+
70
+ ENTRYPOINT ["/fluentd/entrypoint.sh"]
data/Gemfile ADDED
@@ -0,0 +1,9 @@
1
+ source 'https://rubygems.org'
2
+
3
+ group :test do
4
+ gem 'codecov'
5
+ gem 'simplecov'
6
+ gem 'webmock'
7
+ end
8
+
9
+ gemspec
data/LICENSE ADDED
@@ -0,0 +1,201 @@
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License" shall mean the terms and conditions for use, reproduction,
10
+ and distribution as defined by Sections 1 through 9 of this document.
11
+
12
+ "Licensor" shall mean the copyright owner or entity authorized by
13
+ the copyright owner that is granting the License.
14
+
15
+ "Legal Entity" shall mean the union of the acting entity and all
16
+ other entities that control, are controlled by, or are under common
17
+ control with that entity. For the purposes of this definition,
18
+ "control" means (i) the power, direct or indirect, to cause the
19
+ direction or management of such entity, whether by contract or
20
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
21
+ outstanding shares, or (iii) beneficial ownership of such entity.
22
+
23
+ "You" (or "Your") shall mean an individual or Legal Entity
24
+ exercising permissions granted by this License.
25
+
26
+ "Source" form shall mean the preferred form for making modifications,
27
+ including but not limited to software source code, documentation
28
+ source, and configuration files.
29
+
30
+ "Object" form shall mean any form resulting from mechanical
31
+ transformation or translation of a Source form, including but
32
+ not limited to compiled object code, generated documentation,
33
+ and conversions to other media types.
34
+
35
+ "Work" shall mean the work of authorship, whether in Source or
36
+ Object form, made available under the License, as indicated by a
37
+ copyright notice that is included in or attached to the work
38
+ (an example is provided in the Appendix below).
39
+
40
+ "Derivative Works" shall mean any work, whether in Source or Object
41
+ form, that is based on (or derived from) the Work and for which the
42
+ editorial revisions, annotations, elaborations, or other modifications
43
+ represent, as a whole, an original work of authorship. For the purposes
44
+ of this License, Derivative Works shall not include works that remain
45
+ separable from, or merely link (or bind by name) to the interfaces of,
46
+ the Work and Derivative Works thereof.
47
+
48
+ "Contribution" shall mean any work of authorship, including
49
+ the original version of the Work and any modifications or additions
50
+ to that Work or Derivative Works thereof, that is intentionally
51
+ submitted to Licensor for inclusion in the Work by the copyright owner
52
+ or by an individual or Legal Entity authorized to submit on behalf of
53
+ the copyright owner. For the purposes of this definition, "submitted"
54
+ means any form of electronic, verbal, or written communication sent
55
+ to the Licensor or its representatives, including but not limited to
56
+ communication on electronic mailing lists, source code control systems,
57
+ and issue tracking systems that are managed by, or on behalf of, the
58
+ Licensor for the purpose of discussing and improving the Work, but
59
+ excluding communication that is conspicuously marked or otherwise
60
+ designated in writing by the copyright owner as "Not a Contribution."
61
+
62
+ "Contributor" shall mean Licensor and any individual or Legal Entity
63
+ on behalf of whom a Contribution has been received by Licensor and
64
+ subsequently incorporated within the Work.
65
+
66
+ 2. Grant of Copyright License. Subject to the terms and conditions of
67
+ this License, each Contributor hereby grants to You a perpetual,
68
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69
+ copyright license to reproduce, prepare Derivative Works of,
70
+ publicly display, publicly perform, sublicense, and distribute the
71
+ Work and such Derivative Works in Source or Object form.
72
+
73
+ 3. Grant of Patent License. Subject to the terms and conditions of
74
+ this License, each Contributor hereby grants to You a perpetual,
75
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76
+ (except as stated in this section) patent license to make, have made,
77
+ use, offer to sell, sell, import, and otherwise transfer the Work,
78
+ where such license applies only to those patent claims licensable
79
+ by such Contributor that are necessarily infringed by their
80
+ Contribution(s) alone or by combination of their Contribution(s)
81
+ with the Work to which such Contribution(s) was submitted. If You
82
+ institute patent litigation against any entity (including a
83
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
84
+ or a Contribution incorporated within the Work constitutes direct
85
+ or contributory patent infringement, then any patent licenses
86
+ granted to You under this License for that Work shall terminate
87
+ as of the date such litigation is filed.
88
+
89
+ 4. Redistribution. You may reproduce and distribute copies of the
90
+ Work or Derivative Works thereof in any medium, with or without
91
+ modifications, and in Source or Object form, provided that You
92
+ meet the following conditions:
93
+
94
+ (a) You must give any other recipients of the Work or
95
+ Derivative Works a copy of this License; and
96
+
97
+ (b) You must cause any modified files to carry prominent notices
98
+ stating that You changed the files; and
99
+
100
+ (c) You must retain, in the Source form of any Derivative Works
101
+ that You distribute, all copyright, patent, trademark, and
102
+ attribution notices from the Source form of the Work,
103
+ excluding those notices that do not pertain to any part of
104
+ the Derivative Works; and
105
+
106
+ (d) If the Work includes a "NOTICE" text file as part of its
107
+ distribution, then any Derivative Works that You distribute must
108
+ include a readable copy of the attribution notices contained
109
+ within such NOTICE file, excluding those notices that do not
110
+ pertain to any part of the Derivative Works, in at least one
111
+ of the following places: within a NOTICE text file distributed
112
+ as part of the Derivative Works; within the Source form or
113
+ documentation, if provided along with the Derivative Works; or,
114
+ within a display generated by the Derivative Works, if and
115
+ wherever such third-party notices normally appear. The contents
116
+ of the NOTICE file are for informational purposes only and
117
+ do not modify the License. You may add Your own attribution
118
+ notices within Derivative Works that You distribute, alongside
119
+ or as an addendum to the NOTICE text from the Work, provided
120
+ that such additional attribution notices cannot be construed
121
+ as modifying the License.
122
+
123
+ You may add Your own copyright statement to Your modifications and
124
+ may provide additional or different license terms and conditions
125
+ for use, reproduction, or distribution of Your modifications, or
126
+ for any such Derivative Works as a whole, provided Your use,
127
+ reproduction, and distribution of the Work otherwise complies with
128
+ the conditions stated in this License.
129
+
130
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
131
+ any Contribution intentionally submitted for inclusion in the Work
132
+ by You to the Licensor shall be under the terms and conditions of
133
+ this License, without any additional terms or conditions.
134
+ Notwithstanding the above, nothing herein shall supersede or modify
135
+ the terms of any separate license agreement you may have executed
136
+ with Licensor regarding such Contributions.
137
+
138
+ 6. Trademarks. This License does not grant permission to use the trade
139
+ names, trademarks, service marks, or product names of the Licensor,
140
+ except as required for reasonable and customary use in describing the
141
+ origin of the Work and reproducing the content of the NOTICE file.
142
+
143
+ 7. Disclaimer of Warranty. Unless required by applicable law or
144
+ agreed to in writing, Licensor provides the Work (and each
145
+ Contributor provides its Contributions) on an "AS IS" BASIS,
146
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147
+ implied, including, without limitation, any warranties or conditions
148
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149
+ PARTICULAR PURPOSE. You are solely responsible for determining the
150
+ appropriateness of using or redistributing the Work and assume any
151
+ risks associated with Your exercise of permissions under this License.
152
+
153
+ 8. Limitation of Liability. In no event and under no legal theory,
154
+ whether in tort (including negligence), contract, or otherwise,
155
+ unless required by applicable law (such as deliberate and grossly
156
+ negligent acts) or agreed to in writing, shall any Contributor be
157
+ liable to You for damages, including any direct, indirect, special,
158
+ incidental, or consequential damages of any character arising as a
159
+ result of this License or out of the use or inability to use the
160
+ Work (including but not limited to damages for loss of goodwill,
161
+ work stoppage, computer failure or malfunction, or any and all
162
+ other commercial damages or losses), even if such Contributor
163
+ has been advised of the possibility of such damages.
164
+
165
+ 9. Accepting Warranty or Additional Liability. While redistributing
166
+ the Work or Derivative Works thereof, You may choose to offer,
167
+ and charge a fee for, acceptance of support, warranty, indemnity,
168
+ or other liability obligations and/or rights consistent with this
169
+ License. However, in accepting such obligations, You may act only
170
+ on Your own behalf and on Your sole responsibility, not on behalf
171
+ of any other Contributor, and only if You agree to indemnify,
172
+ defend, and hold each Contributor harmless for any liability
173
+ incurred by, or claims asserted against, such Contributor by reason
174
+ of your accepting any such warranty or additional liability.
175
+
176
+ END OF TERMS AND CONDITIONS
177
+
178
+ APPENDIX: How to apply the Apache License to your work.
179
+
180
+ To apply the Apache License to your work, attach the following
181
+ boilerplate notice, with the fields enclosed by brackets "{}"
182
+ replaced with your own identifying information. (Don't include
183
+ the brackets!) The text should be enclosed in the appropriate
184
+ comment syntax for the file format. We also recommend that a
185
+ file or class name and description of purpose be included on the
186
+ same "printed page" as the copyright notice for easier
187
+ identification within third-party archives.
188
+
189
+ Copyright {yyyy} {name of copyright owner}
190
+
191
+ Licensed under the Apache License, Version 2.0 (the "License");
192
+ you may not use this file except in compliance with the License.
193
+ You may obtain a copy of the License at
194
+
195
+ http://www.apache.org/licenses/LICENSE-2.0
196
+
197
+ Unless required by applicable law or agreed to in writing, software
198
+ distributed under the License is distributed on an "AS IS" BASIS,
199
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200
+ See the License for the specific language governing permissions and
201
+ limitations under the License.
data/README.md ADDED
@@ -0,0 +1,545 @@
1
+ [![Build Status](https://travis-ci.org/SumoLogic/fluentd-kubernetes-sumologic.svg?branch=master)](https://travis-ci.org/SumoLogic/fluentd-kubernetes-sumologic) [![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/SumoLogic/fluentd-output-sumologic/issues)
2
+
3
+ This page describes the Sumo Kubernetes [Fluentd](http://www.fluentd.org/) plugin.
4
+
5
+ ## Support
6
+ The code in this repository has been developed in collaboration with the Sumo Logic community and is not supported via standard Sumo Logic Support channels. For any issues or questions please submit an issue within the GitHub repository. The maintainers of this project will work directly with the community to answer any questions, address bugs, or review any requests for new features.
7
+
8
+ ## Installation
9
+
10
+ The plugin runs as a Kubernetes [DaemonSet](http://kubernetes.io/docs/admin/daemons/); it runs an instance of the plugin on each host in a cluster. Each plugin instance pulls system, kubelet, docker daemon, and container logs from the host and sends them, in JSON or text format, to an HTTP endpoint on a hosted collector in the [Sumo](http://www.sumologic.com) service. Note the plugin with default configuration requires Kubernetes >=1.8. See [the section below on running this on Kubernetes <1.8](#running-on-kubernetes-versions-<1.8)
11
+
12
+ - [Step 1 Create hosted collector and HTTP source in Sumo](#step-1--create-hosted-collector-and-http-source-in-sumo)
13
+ - [Step 2 Create a Kubernetes secret](#step-2--create-a-kubernetes-secret)
14
+ - [Step 3 Install the Sumo Kubernetes FluentD plugin](#step-3--install-the-sumo-kubernetes-fluentd-plugin)
15
+ * [Option A Install plugin using kubectl](#option-a--install-plugin-using-kubectl)
16
+ * [Option B Helm chart](#option-b--helm-chart)
17
+ - [Environment variables](#environment-variables)
18
+ + [Override environment variables using annotations](#override-environment-variables-using-annotations)
19
+ + [Exclude data using annotations](#exclude-data-using-annotations)
20
+ + [Include excluded using annotations](#include-excluded-using-annotations)
21
+ - [Step 4 Set up Heapster for metric collection](#step-4-set-up-heapster-for-metric-collection)
22
+ * [Kubernetes ConfigMap](#kubernetes-configmap)
23
+ * [Kubernetes Service](#kubernetes-service)
24
+ * [Kubernetes Deployment](#kubernetes-deployment)
25
+ - [Log data](#log-data)
26
+ * [Docker](#docker)
27
+ * [Kubelet](#kubelet)
28
+ * [Containers](#containers)
29
+ - [Taints and Tolerations](#taints-and-tolerations)
30
+ - [Running On OpenShift](#running-on-openshift)
31
+
32
+
33
+
34
+ ![deployment](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/blob/master/screenshots/kubernetes.png)
35
+
36
+ # Step 1 Create hosted collector and HTTP source in Sumo
37
+
38
+ In this step you create, on the Sumo service, an HTTP endpoint to receive your logs. This process involves creating an HTTP source on a hosted collector in Sumo. In Sumo, collectors use sources to receive data.
39
+
40
+ 1. If you don’t already have a Sumo account, you can create one by clicking the **Free Trial** button on https://www.sumologic.com/.
41
+ 2. Create a hosted collector, following the instructions on [Configure a Hosted Collector](https://help.sumologic.com/Send-Data/Hosted-Collectors/Configure-a-Hosted-Collector) in Sumo help. (If you already have a Sumo hosted collector that you want to use, skip this step.)
42
+ 3. Create an HTTP source on the collector you created in the previous step. For instructions, see [HTTP Logs and Metrics Source](https://help.sumologic.com/Send-Data/Sources/02Sources-for-Hosted-Collectors/HTTP-Source) in Sumo help.
43
+ 4. When you have configured the HTTP source, Sumo will display the URL of the HTTP endpoint. Make a note of the URL. You will use it when you configure the Kubernetes service to send data to Sumo.
44
+
45
+ # Step 2 Create a Kubernetes secret
46
+
47
+ Create a secret in Kubernetes with the HTTP source URL. If you want to change the secret name, you must modify the Kubernetes manifest accordingly.
48
+
49
+ `kubectl create secret generic sumologic --from-literal=collector-url=INSERT_HTTP_URL`
50
+
51
+ You should see the confirmation message
52
+
53
+ `secret "sumologic" created.`
54
+
55
+ # Step 3 Install the Sumo Kubernetes FluentD plugin
56
+
57
+ Follow the instructions in Option A below to install the plugin using `kubectl`. If you prefer to use a Helm chart, see Option B.
58
+
59
+ Before you start, see [Environment variables](#environment-variables) for information about settings you can customize, and how to use annotations to override selected environment variables and exclude data from being sent to Sumo.
60
+
61
+ ## Option A Install plugin using kubectl
62
+
63
+ See the sample Kubernetes DaemonSet and Role in [fluentd.yaml](/daemonset/rbac/fluentd.yaml).
64
+
65
+ 1. Clone the [GitHub repo](https://github.com/SumoLogic/fluentd-kubernetes-sumologic).
66
+
67
+ 2. In `fluentd-kubernetes-sumologic`, install the chart using `kubectl`.
68
+
69
+ Which `.yaml` file you should use depends on whether or not you are running RBAC for authorization. RBAC is enabled by default as of Kubernetes 1.6. Note the plugin with default configuration requires Kubernetes >=1.8. See the section below on [running this on Kubernetes <1.8](#running-on-kubernetes-versions-<1.8)
70
+
71
+ **Non-RBAC (Kubernetes 1.5 and below)**
72
+
73
+ `kubectl create -f /daemonset/nonrbac/fluentd.yaml`
74
+
75
+ **RBAC (Kubernetes 1.6 and above)** <br/><br/>`kubectl create -f /daemonset/rbac/fluentd.yaml`
76
+
77
+
78
+ **Note** if you modified the command in Step 2 to use a different name, update the `.yaml` file to use the correct secret.
79
+
80
+ Logs should begin flowing into Sumo within a few minutes of plugin installation.
81
+
82
+ ## Option B Helm chart
83
+ If you use Helm to manage your Kubernetes resources, there is a Helm chart for the plugin at https://github.com/kubernetes/charts/tree/master/stable/sumologic-fluentd.
84
+
85
+ # Environment variables
86
+
87
+ Environment | Variable Description
88
+ ----------- | --------------------
89
+ `AUDIT_LOG_PATH`|Define the path to the [Kubernetes Audit Log](https://kubernetes.io/docs/tasks/debug-application-cluster/audit/) <br/><br/> Default: `/mnt/log/kube-apiserver-audit.log`
90
+ `CONCAT_SEPARATOR` |The character to use to delimit lines within the final concatenated message. Most multi-line messages contain a newline at the end of each line. <br/><br/> Default: ""
91
+ `EXCLUDE_CONTAINER_REGEX` |A regular expression for containers. Matching containers will be excluded from Sumo. The logs will still be sent to FluentD.
92
+ `EXCLUDE_FACILITY_REGEX`|A regular expression for syslog [facilities](https://en.wikipedia.org/wiki/Syslog#Facility). Matching facilities will be excluded from Sumo. The logs will still be sent to FluentD.
93
+ `EXCLUDE_HOST_REGEX`|A regular expression for hosts. Matching hosts will be excluded from Sumo. The logs will still be sent to FluentD.
94
+ `EXCLUDE_NAMESPACE_REGEX`|A regular expression for `namespaces`. Matching `namespaces` will be excluded from Sumo. The logs will still be sent to FluentD.
95
+ `EXCLUDE_PATH`|Files matching this pattern will be ignored by the `in_tail` plugin, and will not be sent to Kubernetes or Sumo. This can be a comma-separated list as well. See [in_tail](http://docs.fluentd.org/v0.12/articles/in_tail#excludepath) documentation for more information. <br/><br/> For example, defining `EXCLUDE_PATH` as shown below excludes all files matching `/var/log/containers/*.log`, <br/><br/>`...`<br/><br/>`env:`<br/>   - `name: EXCLUDE_PATH`<br/>   `value: "[\"/var/log/containers/*.log\"]"`
96
+ `EXCLUDE_POD_REGEX`|A regular expression for pods. Matching pods will be excluded from Sumo. The logs will still be sent to FluentD.
97
+ `EXCLUDE_PRIORITY_REGEX`|A regular expression for syslog [priorities](https://en.wikipedia.org/wiki/Syslog#Severity_level). Matching priorities will be excluded from Sumo. The logs will still be sent to FluentD.
98
+ `EXCLUDE_UNIT_REGEX` |A regular expression for `systemd` units. Matching units will be excluded from Sumo. The logs will still be sent to FluentD.
99
+ `FLUENTD_SOURCE`|Fluentd can tail files or query `systemd`. Allowable values: `file`, `Systemd`. <br/><br/>Default: `file`
100
+ `FLUENTD_USER_CONFIG_DIR`|A directory of user-defined fluentd configuration files, which must be in the `*.conf` directory in the container.
101
+ `FLUSH_INTERVAL` |How frequently to push logs to Sumo.<br/><br/>Default: `5s`
102
+ `KUBERNETES_META`|Include or exclude Kubernetes metadata such as `namespace` and `pod_name` if using JSON log format. <br/><br/>Default: `true`
103
+ `KUBERNETES_META_REDUCE`| Reduces redundant Kubernetes metadata, see [_Reducing Kubernetes Metadata_](#reducing-kubernetes-metadata). <br></br>Default: `false`
104
+ `LOG_FORMAT`|Format in which to post logs to Sumo. Allowable values:<br/><br/>`text`—Logs will appear in SumoLogic in text format.<br/>`json`—Logs will appear in SumoLogic in json format.<br/>`json_merge`—Same as json but if the container logs in json format to stdout it will merge in the container json log at the root level and remove the log field.<br/><br/>Default: `json`
105
+ `MULTILINE_START_REGEXP`|The regular expression for the `concat` plugin to use when merging multi-line messages. Defaults to Julian dates, for example, Jul 29, 2017.
106
+ `NUM_THREADS`|Set the number of HTTP threads to Sumo. It might be necessary to do so in heavy-logging clusters. <br/><br/>Default: `1`
107
+ `READ_FROM_HEAD`|Start to read the logs from the head of file, not bottom. Only applies to containers log files. See in_tail doc for more information.<br/><br/>Default: `true`
108
+ `SOURCE_CATEGORY` |Set the `_sourceCategory` metadata field in Sumo. <br/><br/>Default: `"%{namespace}/%{pod_name}"`
109
+ `SOURCE_CATEGORY_PREFIX`|Prepends a string that identifies the cluster to the `_sourceCategory` metadata field in Sumo.<br/><br/>Default: `kubernetes/`
110
+ `SOURCE_CATEGORY_REPLACE_DASH` |Used to replace a dash (-) character with another character. <br/><br/>Default: `/`<br/><br/>For example, a Pod called `travel-nginx-3629474229-dirmo` within namespace `app` will appear in Sumo with `_sourceCategory=app/travel/nginx`.
111
+ `SOURCE_HOST`|Set the `_sourceHost` metadata field in Sumo.<br/><br/>Default: `""`
112
+ `SOURCE_NAME`|Set the `_sourceName` metadata field in Sumo. <br/><br/> Default: `"%{namespace}.%{pod}.%{container}"`
113
+ `TIME_KEY`|The field name for json formatted sources that should be used as the time. See [time_key](https://docs.fluentd.org/v0.12/articles/formatter_json#time_key-(string,-optional,-defaults-to-%E2%80%9Ctime%E2%80%9D)). Default: `time`
114
+ `ADD_TIMESTAMP`|Option to control adding timestamp to logs. Default: `true`
115
+ `TIMESTAMP_KEY`|Field name when add_timestamp is on. Default: `timestamp`
116
+ `ADD_STREAM`|Option to control adding stream to logs. Default: `true`
117
+ `ADD_TIME`|Option to control adding time to logs. Default: `true`
118
+ `CONTAINER_LOGS_PATH`|Specify the path in_tail should watch for container logs. Default: `/mnt/log/containers/*.log`
119
+ `PROXY_URI`|Add the uri of the proxy environment if present.
120
+ `ENABLE_STAT_WATCHER`|Option to control the enabling of [stat_watcher](https://docs.fluentd.org/v1.0/articles/in_tail#enable_stat_watcher). Default: `true`
121
+ `K8S_METADATA_FILTER_WATCH`|Option to control the enabling of [metadata filter plugin watch](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter#configuration). Default: `true`
122
+ `K8S_METADATA_FILTER_CA_FILE`|Option to control the enabling of [metadata filter plugin ca_file](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter#configuration).
123
+ `K8S_METADATA_FILTER_VERIFY_SSL`|Option to control the enabling of [metadata filter plugin verify_ssl](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter#configuration). Default: `true`
124
+ `K8S_METADATA_FILTER_CLIENT_CERT`|Option to control the enabling of [metadata filter plugin client_cert](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter#configuration).
125
+ `K8S_METADATA_FILTER_CLIENT_KEY`|Option to control the enabling of [metadata filter plugin client_key](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter#configuration).
126
+ `K8S_METADATA_FILTER_BEARER_TOKEN_FILE`|Option to control the enabling of [metadata filter plugin bearer_token_file](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter#configuration).
127
+ `K8S_METADATA_FILTER_BEARER_CACHE_SIZE`|Option to control the enabling of [metadata filter plugin cache_size](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter#configuration). Default: `1000`
128
+ `K8S_METADATA_FILTER_BEARER_CACHE_TTL`|Option to control the enabling of [metadata filter plugin cache_ttl](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter#configuration). Default: `3600`
129
+ `K8S_NODE_NAME`|If set, improves [caching of pod metadata](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter#environment-variables-for-kubernetes) and reduces API calls.
130
+ `VERIFY_SSL`|Verify ssl certificate of sumologic endpoint. Default: `true`
131
+
132
+
133
+ The following table show which environment variables affect which Fluentd sources.
134
+
135
+ | Environment Variable | Containers | Docker | Kubernetes | Systemd |
136
+ |----------------------|------------|--------|------------|---------|
137
+ | `EXCLUDE_CONTAINER_REGEX` | ✔ | ✘ | ✘ | ✘ |
138
+ | `EXCLUDE_FACILITY_REGEX` | ✘ | ✘ | ✘ | ✔ |
139
+ | `EXCLUDE_HOST_REGEX `| ✔ | ✘ | ✘ | ✔ |
140
+ | `EXCLUDE_NAMESPACE_REGEX` | ✔ | ✘ | ✔ | ✘ |
141
+ | `EXCLUDE_PATH` | ✔ | ✔ | ✔ | ✘ |
142
+ | `EXCLUDE_PRIORITY_REGEX` | ✘ | ✘ | ✘ | ✔ |
143
+ | `EXCLUDE_POD_REGEX` | ✔ | ✘ | ✘ | ✘ |
144
+ | `EXCLUDE_UNIT_REGEX` | ✘ | ✘ | ✘ | ✔ |
145
+ | `TIME_KEY` | ✔ | ✘ | ✘ | ✘ |
146
+
147
+ ### FluentD stops processing logs
148
+ When dealing with large volumes of data (TB's from what we have seen), FluentD may stop processing logs, but continue to run. This issue seems to be caused by the [scalability of the inotify process](https://github.com/fluent/fluentd/issues/1630) that is packaged with the FluentD in_tail plugin. If you encounter this situation, setting the `ENABLE_STAT_WATCHER` to `false` should resolve this issue.
149
+
150
+ ### Reducing Kubernetes metadata
151
+
152
+ You can use the `KUBERNETES_META_REDUCE` environment variable (global) or the `sumologic.com/kubernetes_meta_reduce` annotation (per pod) to reduce the amount of Kubernetes metadata included with each log line under the `kubernetes` field.
153
+
154
+ When set, FluentD will remove the following properties:
155
+
156
+ * `pod_id`
157
+ * `container_id`
158
+ * `namespace_id`
159
+ * `master_url`
160
+ * `labels`
161
+ * `annotations`
162
+
163
+ Logs will still include:
164
+
165
+ * `pod_name`
166
+ * `container_name`
167
+ * `namespace_name`
168
+ * `host`
169
+
170
+ These fields still allow you to uniquely identify a pod and look up additional details with the Kubernetes API.
171
+
172
+ ```yaml
173
+ apiVersion: v1
174
+ kind: ReplicationController
175
+ metadata:
176
+ name: nginx
177
+ spec:
178
+ replicas: 1
179
+ selector:
180
+ app: mywebsite
181
+ template:
182
+ metadata:
183
+ name: nginx
184
+ labels:
185
+ app: mywebsite
186
+ annotations:
187
+ sumologic.com/kubernetes_meta_reduce: "true"
188
+ spec:
189
+ containers:
190
+ - name: nginx
191
+ image: nginx
192
+ ports:
193
+ - containerPort: 80
194
+ ```
195
+
196
+
197
+ ### Override environment variables using annotations
198
+ You can override the `LOG_FORMAT`, `KUBERNETES_META_REDUCE`, `SOURCE_CATEGORY` and `SOURCE_NAME` environment variables, per pod, using [Kubernetes annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/). For example:
199
+
200
+ ```
201
+ apiVersion: v1
202
+ kind: ReplicationController
203
+ metadata:
204
+ name: nginx
205
+ spec:
206
+ replicas: 1
207
+ selector:
208
+ app: mywebsite
209
+ template:
210
+ metadata:
211
+ name: nginx
212
+ labels:
213
+ app: mywebsite
214
+ annotations:
215
+ sumologic.com/format: "text"
216
+ sumologic.com/kubernetes_meta_reduce: "true"
217
+ sumologic.com/sourceCategory: "mywebsite/nginx"
218
+ sumologic.com/sourceName: "mywebsite_nginx"
219
+ spec:
220
+ containers:
221
+ - name: nginx
222
+ image: nginx
223
+ ports:
224
+ - containerPort: 80
225
+ ```
226
+
227
+ ### Exclude data using annotations
228
+
229
+ You can also use the `sumologic.com/exclude` annotation to exclude data from Sumo. This data is sent to FluentD, but not to Sumo.
230
+
231
+ ```
232
+ apiVersion: v1
233
+ kind: ReplicationController
234
+ metadata:
235
+ name: nginx
236
+ spec:
237
+ replicas: 1
238
+ selector:
239
+ app: mywebsite
240
+ template:
241
+ metadata:
242
+ name: nginx
243
+ labels:
244
+ app: mywebsite
245
+ annotations:
246
+ sumologic.com/format: "text"
247
+ sumologic.com/sourceCategory: "mywebsite/nginx"
248
+ sumologic.com/sourceName: "mywebsite_nginx"
249
+ sumologic.com/exclude: "true"
250
+ spec:
251
+ containers:
252
+ - name: nginx
253
+ image: nginx
254
+ ports:
255
+ - containerPort: 80
256
+ ```
257
+
258
+ ### Include excluded using annotations
259
+
260
+ If you excluded a whole namespace, but still need one or few pods to be still included for shipping to Sumologic, you can use the `sumologic.com/include` annotation to include data to Sumo. It takes precedence over the exclusion described above.
261
+
262
+ ```
263
+ apiVersion: v1
264
+ kind: ReplicationController
265
+ metadata:
266
+ name: nginx
267
+ spec:
268
+ replicas: 1
269
+ selector:
270
+ app: mywebsite
271
+ template:
272
+ metadata:
273
+ name: nginx
274
+ labels:
275
+ app: mywebsite
276
+ annotations:
277
+ sumologic.com/format: "text"
278
+ sumologic.com/sourceCategory: "mywebsite/nginx"
279
+ sumologic.com/sourceName: "mywebsite_nginx"
280
+ sumologic.com/include: "true"
281
+ spec:
282
+ containers:
283
+ - name: nginx
284
+ image: nginx
285
+ ports:
286
+ - containerPort: 80
287
+ ```
288
+
289
+ # Step 4 Set up Heapster for metric collection
290
+
291
+ The recommended way to collect metrics from Kubernetes clusters is to use Heapster and a Sumo collector with a Graphite source.
292
+
293
+ Heapster aggregates metrics across a Kubenetes cluster. Heapster runs as a pod in the cluster, and discovers all nodes in the cluster and queries usage information from each node's `kubelet`—the on-machine Kubernetes agent.
294
+
295
+ Heapster provides metrics at the cluster, node and pod level.
296
+
297
+ 1. Install Heapster in your Kubernetes cluster and configure a Graphite Sink to send the data in Graphite format to Sumo. For instructions, see
298
+ https://github.com/kubernetes/heapster/blob/master/docs/sink-configuration.md#graphitecarbon. Assuming you have used the below YAML files to configure your system, then the sink option in graphite would be `--sink=graphite:tcp://sumo-graphite.kube-system.svc:2003`. You may need to change this depending on the namespace you run the deployment in, the name of the service or the port number for your Graphite source.
299
+
300
+ 2. Use the Sumo Docker container. For instructions, see https://hub.docker.com/r/sumologic/collector/.
301
+
302
+ 3. The following sections contain an example configmap, which contains the `sources.json` configuration, an example service, and an example deployment. Create these manifests in Kubernetes using `kubectl`.
303
+
304
+
305
+ ## Kubernetes ConfigMap
306
+ ```
307
+ kind: ConfigMap
308
+ apiVersion: v1
309
+ metadata:
310
+ name: "sumo-sources"
311
+ data:
312
+ sources.json: |-
313
+ {
314
+ "api.version": "v1",
315
+ "sources": [
316
+ {
317
+ "name": "SOURCE_NAME",
318
+ "category": "SOURCE_CATEGORY",
319
+ "automaticDateParsing": true,
320
+ "contentType": "Graphite",
321
+ "timeZone": "UTC",
322
+ "encoding": "UTF-8",
323
+ "protocol": "TCP",
324
+ "port": 2003,
325
+ "sourceType": "Graphite"
326
+ }
327
+ ]
328
+ }
329
+
330
+ ```
331
+ ## Kubernetes Service
332
+ ```
333
+ apiVersion: v1
334
+ kind: Service
335
+ metadata:
336
+ name: sumo-graphite
337
+ spec:
338
+ ports:
339
+ - port: 2003
340
+ selector:
341
+ app: sumo-graphite
342
+ ```
343
+ ## Kubernetes Deployment
344
+ ```
345
+ apiVersion: extensions/v1beta1
346
+ kind: Deployment
347
+ metadata:
348
+ labels:
349
+ app: sumo-graphite
350
+ name: sumo-graphite
351
+ spec:
352
+ replicas: 2
353
+ template:
354
+ metadata:
355
+ labels:
356
+ app: sumo-graphite
357
+ spec:
358
+ volumes:
359
+ - name: sumo-sources
360
+ configMap:
361
+ name: sumo-sources
362
+ items:
363
+ - key: sources.json
364
+ path: sources.json
365
+ containers:
366
+ - name: sumo-graphite
367
+ image: sumologic/collector:latest
368
+ ports:
369
+ - containerPort: 2003
370
+ volumeMounts:
371
+ - mountPath: /sumo
372
+ name: sumo-sources
373
+ env:
374
+ - name: SUMO_ACCESS_ID
375
+ value: <SUMO_ACCESS_ID>
376
+ - name: SUMO_ACCESS_KEY
377
+ value: <SUMO_ACCESS_KEY>
378
+ - name: SUMO_SOURCES_JSON
379
+ value: /sumo/sources.json
380
+
381
+ ```
382
+
383
+ # Templating Kubernetes metadata
384
+ The following Kubernetes metadata is available for string templating:
385
+
386
+ | String template | Description |
387
+ | --------------- | ------------------------------------------------------ |
388
+ | `%{namespace}` | Namespace name |
389
+ | `%{pod}` | Full pod name (e.g. `travel-products-4136654265-zpovl`) |
390
+ | `%{pod_name}` | Friendly pod name (e.g. `travel-products`) |
391
+ | `%{pod_id}` | The pod's uid (a UUID) |
392
+ | `%{container}` | Container name |
393
+ | `%{source_host}` | Host |
394
+ | `%{label:foo}` | The value of label `foo` |
395
+
396
+ ## Missing labels
397
+ Unlike the other templates, labels are not guaranteed to exist, so missing labels interpolate as `"undefined"`.
398
+
399
+ For example, if you have only the label `app: travel` but you define `SOURCE_NAME="%{label:app}@%{label:version}"`, the source name will appear as `travel@undefined`.
400
+
401
+ # Log data
402
+ After performing the configuration described above, your logs should start streaming to SumoLogic in `json` or text format with the appropriate metadata. If you are using `json` format you can auto extract fields, for example `_sourceCategory=some/app | json auto`.
403
+
404
+ ## Docker
405
+ ![Docker Logs](/screenshots/docker.png)
406
+
407
+ ## Kubelet
408
+ Note that Kubelet logs are only collected if you are using systemd. Kubernetes no longer outputs the kubelet logs to a file.
409
+ ![Docker Logs](/screenshots/kubelet.png)
410
+
411
+ ## Containers
412
+ ![Docker Logs](/screenshots/container.png)
413
+
414
+ # Taints and Tolerations
415
+ By default, the fluentd pods will schedule on, and therefore collect logs from, any worker nodes that do not have a taint and any master node that does not have a taint beyond the default master taint. If you would like to schedule pods on all nodes, regardless of taints, uncomment the following line from fluentd.yaml before applying it.
416
+
417
+ ```
418
+ tolerations:
419
+ #- operator: "Exists"
420
+ ```
421
+
422
+ # Running On OpenShift
423
+
424
+ This daemonset setting mounts /var/log as service account FluentD so you need to run containers as privileged container. Here is command example:
425
+
426
+ ```
427
+ oc adm policy add-scc-to-user privileged system:serviceaccount:logging:fluentd
428
+ oc adm policy add-cluster-role-to-user cluster-reader system:serviceaccount:logging:fluentd
429
+ oc label node —all logging-sumologic-fluentd=true
430
+ oc patch ds sumologic-fluentd -p "spec:
431
+ template:
432
+ spec:
433
+ containers:
434
+ - image: sumologic/fluentd-kubernetes-sumologic:latest
435
+ name: fluentd
436
+ securityContext:
437
+ privileged: true"
438
+ oc delete pod -l name = fluentd-sumologic
439
+ ```
440
+
441
+ ## Running on Kubernetes versions <1.8
442
+
443
+ In order to run this plugin on Kubernetes <1.8 you will need to make some changes the yaml file prior to deploying it.
444
+
445
+ Replace:
446
+
447
+ ```
448
+ - name: pos-files
449
+ hostPath:
450
+ path: /var/run/fluentd-pos
451
+ type: ""
452
+ ```
453
+ With:
454
+
455
+ ```
456
+ - name: pos-files
457
+ emptyDir: {}
458
+ ```
459
+
460
+ ## Output to S3
461
+
462
+ If you need to also send data to S3 (i.e. as a secondary backup/audit trail) the image includes the `fluent-plugin-s3` plugin. In order to send the logs from FluentD to multiple outputs, you must use the `copy` plugin. This image comes with an [OOB configuration](conf.d/out.sumo.conf) to output the logs to Sumo Logic. In order to output to multiple destinations, you need to modify that existing configuration.
463
+
464
+ **Example:** Send all logs to S3 and Sumo:
465
+
466
+ ```
467
+ <match **>
468
+ @type copy
469
+ <store>
470
+ @type sumologic
471
+ log_key log
472
+ endpoint "#{ENV['COLLECTOR_URL']}"
473
+ verify_ssl "#{ENV['VERIFY_SSL']}"
474
+ log_format "#{ENV['LOG_FORMAT']}"
475
+ flush_interval "#{ENV['FLUSH_INTERVAL']}"
476
+ num_threads "#{ENV['NUM_THREADS']}"
477
+ open_timeout 60
478
+ add_timestamp "#{ENV['ADD_TIMESTAMP']}"
479
+ proxy_uri "#{ENV['PROXY_URI']}"
480
+ </store>
481
+ <store>
482
+ @type s3
483
+
484
+ aws_key_id YOUR_AWS_KEY_ID
485
+ aws_sec_key YOUR_AWS_SECRET_KEY
486
+ s3_bucket YOUR_S3_BUCKET_NAME
487
+ s3_region us-west-1
488
+ path logs/
489
+ buffer_path /var/log/fluent/s3
490
+
491
+ time_slice_format %Y%m%d%H
492
+ time_slice_wait 10m
493
+ utc
494
+
495
+ buffer_chunk_limit 256m
496
+ </store>
497
+ </match>
498
+ ```
499
+
500
+ You can replace the OOB configuration by creating a new Docker image from our image or by using a configmap to inject the new configuration to the pod.
501
+
502
+ More details about the S3 plugin can be found [in the docs](https://docs.fluentd.org/v0.12/articles/out_s3).
503
+
504
+ ## Upgrading to v2.0.0
505
+
506
+ In version 2.0.0, some legacy FluentD configuration has been removed that could lead to [duplicate logs being ingested into Sumo Logic](https://github.com/SumoLogic/fluentd-kubernetes-sumologic/issues/79). These logs were control plane components. This version was done as a major release as it breaks the current version of the [Kubernetes App](https://help.sumologic.com/Send-Data/Applications-and-Other-Data-Sources/Kubernetes/Install_the_Kubernetes_App_and_View_the_Dashboards) you may have installed in Sumo Logic.
507
+
508
+ After upgrading to this version, you will need to reinstall the [Kubernetes App](https://help.sumologic.com/Send-Data/Applications-and-Other-Data-Sources/Kubernetes/Install_the_Kubernetes_App_and_View_the_Dashboards) in Sumo Logic. If you do not some of the panels in the dashboards will not render properly.
509
+
510
+ If you have other content outside the app (Partitions, Scheduled Views, Field Extraction Rules or Scheduled Searches and Alerts), these may need to be updated after upgrading to v2.0.0. The logs, while the same content, have a different format and the same parsing logic and metadata may not apply.
511
+
512
+ The previous log format that is removed in v2.0.0:
513
+ ```json
514
+ {
515
+ "timestamp": 1538776281387,
516
+ "severity": "I",
517
+ "pid": "1",
518
+ "source": "wrap.go:42",
519
+ "message": "GET /api/v1/namespaces/kube-system/endpoints/kube-scheduler: (3.514372ms) 200 [[kube-scheduler/v1.10.5 (linux/amd64) kubernetes/32ac1c9/leader-election] 127.0.0.1:46290]"
520
+ }
521
+ ```
522
+ Is replaced by the following version. It is the same log line in a different format enriched with the same metadata the plugin applies to all pod logs.
523
+ ```json
524
+ {
525
+ "timestamp": 1538776282152,
526
+ "log": "I1005 21:51:21.387204 1 wrap.go:42] GET /api/v1/namespaces/kube-system/endpoints/kube-scheduler: (3.514372ms) 200 [[kube-scheduler/v1.10.5 (linux/amd64) kubernetes/32ac1c9/leader-election] 127.0.0.1:46290]",
527
+ "stream": "stdout",
528
+ "time": "2018-10-05T21:51:21.387477546Z",
529
+ "docker": {
530
+ "container_id": "a442fd2982dfdc09ab6235941f8d661a0a5c8df5e1d21f23ff48a9923ac14739"
531
+ },
532
+ "kubernetes": {
533
+ "container_name": "kube-apiserver",
534
+ "namespace_name": "kube-system",
535
+ "pod_name": "kube-apiserver-ip-172-20-122-71.us-west-2.compute.internal",
536
+ "pod_id": "80fa5e13-c8b9-11e8-a456-0a8c1424d0d4",
537
+ "labels": {
538
+ "k8s-app": "kube-apiserver"
539
+ },
540
+ "host": "ip-172-20-122-71.us-west-2.compute.internal",
541
+ "master_url": "https://100.64.0.1:443/api",
542
+ "namespace_id": "9b9b75b7-aa16-11e8-9d62-06df85b5d3bc"
543
+ }
544
+ }
545
+ ```