fluent-plugin-tkgi-metadata-parser 1.0.0 → 1.0.1

This diff has not been reviewed by any users.
Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 4d9138b2b8ed47185ddcfb4c75f543c68693109c7b8e401dd5ed227061171edc
4
- data.tar.gz: 59ee969a172811b858b8c1916ed83c82a01e8a5c31046c284502d1d76511e5d0
3
+ metadata.gz: 4e664538aada392e7de09f4dc2b9765eac852066b4b6b5b987f8259753592cff
4
+ data.tar.gz: 191b486c973a9bcbcf578e5f8ce9f2cdb2acdc45c0174c4cacf5d935b5a463ff
5
5
  SHA512:
6
- metadata.gz: 6b05ec91e09d2f97bfb4fcac3693b58ce58be6d3c0a5d8d83d2a4647e2a6601d06f3237ed01fa1689d7afc4242545b17ee872e78a6623c7ad2531c83ebb2982a
7
- data.tar.gz: 900cc106dcdc93aa7b6e43c8652559b6c0e2b5cabe61e509cef28dd803624921e77af760dd56fc18b12b827334b4dba773b51317f37dac875c76d781fc2541ce
6
+ metadata.gz: 06c2484b2d244a1859ea62684c8525e6925c91ad6791e4475d46b2e5710a8c59dd8d0b95c05594b299d01bbf7cbf05a0465ca3b9ee44165249c2e716da2d9a29
7
+ data.tar.gz: dc46f4049c3519acd508ad933e0e4080284bd563495c32e78ab1ac699cd4e383cbe2a6cd70e09e3bfbf62bec1e0a8feb395f09cb749e9a46c41b1f09a7a6e824
data/README.md CHANGED
@@ -1 +1,121 @@
1
- # fluent-plugin-tkgi-metadata-parser
1
+ # fluent-plugin-tkgi-metadata-parser
2
+
3
+ ## Description
4
+
5
+ This Fluentd parser plugin parses metadata received from Tanzu Kubernetes Grid Integrated Edition (TKGI) or Tanzu Kubernetes Grid (TKG).
6
+
7
+ TKG(I) uses [RFC 5424 - The Syslog Protocol](https://tools.ietf.org/html/rfc5424), this plugin parses specifically the syslog5424_sd field if you use standard parser to parse syslog5424 logs initially, but you can use any field which consists of Kubernetes metadata.
8
+
9
+ e.g
10
+ ```log
11
+ "syslog5424_sd":"[kubernetes@47450 pod-template-hash=\"6cdc894687\" app.kubernetes.io/component=\"operator\" app.kubernetes.io/instance=\"prometheus\" app.kubernetes.io/managed-by=\"Helm\" app.kubernetes.io/name=\"kube-prometheus\" helm.sh/chart=\"kube-prometheus-3.2.0\" namespace_name=\"prometheus\" object_name=\"prometheus-kube-prometheus-operator-6cdc894687-c56bf\" container_name=\"prometheus-operator\" vm_id=\"04c2872a-768a-4577-944a-87842407f582\"]"
12
+ ```
13
+
14
+ ## Installation
15
+
16
+ ```shell
17
+ $ td-agent-gem install fluent-plugin-tkgi-metadata-parser
18
+ ```
19
+
20
+
21
+ ## Usage
22
+
23
+ To parse log initially when it comes to Fluentd you need to apply syslog5424 parser, example below uses [grok parser plugin](https://github.com/fluent/fluent-plugin-grok-parser) but feel free to use any other parser of your choice.
24
+
25
+ ```conf
26
+ <source>
27
+ @type tcp
28
+ port 6514
29
+ key_name message
30
+ <parse>
31
+ @type grok
32
+ grok_failue_key _grokparsefailure
33
+ <grok>
34
+ pattern %{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:syslog5424_ts}|-) +(?:%{HOSTNAME:syslog5424_host}|-) +(?:%{NOTSPACE:syslog5424_app}|-) +(?:%{NOTSPACE:syslog5424_proc}|-) +(?:%{WORD:syslog5424_msgid}|-) +(?:%{SYSLOG5424SD:syslog5424_sd}|-|) +%{GREEDYDATA:syslog5424_msg}
35
+ </grok>
36
+ </parse>
37
+ tag tkgi
38
+ </source>
39
+ ```
40
+
41
+ ### Plugin configuration
42
+
43
+ ```conf
44
+ <filter tkgi>
45
+ @type parser
46
+ key_name syslog5424_sd
47
+ reserve_data true
48
+ reserve_time true
49
+ <parse>
50
+ @type tkgi_metadata
51
+ es_mode true
52
+ </parse>
53
+ </filter>
54
+ ```
55
+
56
+ ### Once parsed the output looks like this:
57
+
58
+ ```json
59
+ {
60
+ "_index": "k8s-2020.12.31",
61
+ "_type": "_doc",
62
+ "_id": "2zqJunYBHg9f6MMRmev2",
63
+ "_version": 1,
64
+ "_score": null,
65
+ "_source": {
66
+ "syslog5424_pri": "14",
67
+ "syslog5424_ver": "1",
68
+ "syslog5424_ts": "2020-12-31T20:41:24.224539+00:00",
69
+ "syslog5424_host": "XXXX",
70
+ "syslog5424_app": "pod.log/prometheus/prometheus-kube-prometheus-op",
71
+ "syslog5424_proc": "-",
72
+ "syslog5424_msg": "something very important",
73
+ "app_kubernetes_io/component": "operator",
74
+ "app_kubernetes_io/instance": "prometheus",
75
+ "app_kubernetes_io/managed-by": "Helm",
76
+ "app_kubernetes_io/name": "kube-prometheus",
77
+ "helm_sh/chart": "kube-prometheus-3.2.0",
78
+ "pod-template-hash": "6cdc894687",
79
+ "namespace_name": "prometheus",
80
+ "object_name": "prometheus-kube-prometheus-operator-6cdc894687-c56bf",
81
+ "container_name": "prometheus-operator",
82
+ "vm_id": "04c2872a-768a-4577-944a-87842407f582",
83
+ "source": "kubernetes",
84
+ "source_id": "47450",
85
+ "@timestamp": "2020-12-31T14:41:25.003082896-06:00",
86
+ "tag": "tkgi"
87
+ }
88
+ ```
89
+
90
+ ## Optional Parameters
91
+
92
+ - **delimiter**
93
+
94
+ delimiter which separate each key-value pairs.
95
+ whitespaces or tabs can be given in quotes: ie, "` `" or "`\t`" .
96
+ By default it is "` `".
97
+
98
+ - **es_mode**
99
+
100
+ When using Elasticsearch as a storage database, it expects mapping to be an object and not text, due to Kubernetes label/annotation naming conventions you might get the following error:
101
+
102
+ ```
103
+ <Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError: 400 - Rejected by Elasticsearch [error type]: mapper_parsing_exception [reason]: 'Could not dynamically add mapping for field [app.kubernetes.io/component]. Existing mapping for [app] must be of type object but found [text].'>
104
+ ```
105
+
106
+ This issue is caused by label/annotation dots (`.`) creating hierarchy in Elasticsearch documents. If an annotation has a structure like this: `example.annotation.data: some-data` and a different one contains `example.annotation: value` their mapping will conflict, as `example.annotation` is both an object and a keyword.
107
+
108
+ To avoid this, when set to `true` this plugin replaces (`.`) with (`_`)
109
+
110
+ By default, it is `false`
111
+
112
+ ## Contributing
113
+
114
+ 1. Fork it
115
+ 2. Create your feature branch (`git checkout -b my-new-feature`)
116
+ 3. Commit your changes (`git commit -am 'Add some feature'`)
117
+ 4. Push to the branch (`git push origin my-new-feature`)
118
+ 5. Create a new Pull Request
119
+
120
+ ## TODO:
121
+ Remove grok parser dependency to parse syslog initially.
@@ -1,10 +1,10 @@
1
1
  $:.push File.expand_path('../lib', __FILE__)
2
2
  Gem::Specification.new do |spec|
3
3
  spec.name = 'fluent-plugin-tkgi-metadata-parser'
4
- spec.version = '1.0.0'
4
+ spec.version = '1.0.1'
5
5
  spec.authors = ['Saurabh Kulkarni']
6
6
  spec.email = ['saurabh.kl@outlook.com']
7
- spec.summary = 'FluentD parser plugin to parse TKGI metadata'
7
+ spec.summary = 'Fluentd parser plugin to parse TKGI metadata'
8
8
  spec.homepage = 'https://github.com/srbhklkrn/fluent-plugin-tkgi-metadata-parser'
9
9
  spec.license = 'MIT'
10
10
  spec.platform = Gem::Platform::RUBY
@@ -1,3 +1,25 @@
1
+ # MIT License
2
+
3
+ # Copyright (c) 2020 Saurabh Kulkarni
4
+
5
+ # Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ # of this software and associated documentation files (the "Software"), to deal
7
+ # in the Software without restriction, including without limitation the rights
8
+ # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ # copies of the Software, and to permit persons to whom the Software is
10
+ # furnished to do so, subject to the following conditions:
11
+
12
+ # The above copyright notice and this permission notice shall be included in all
13
+ # copies or substantial portions of the Software.
14
+
15
+ # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ # SOFTWARE.
22
+
1
23
  require "fluent/plugin/parser"
2
24
 
3
25
  module Fluent::Plugin
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-tkgi-metadata-parser
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.0
4
+ version: 1.0.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Saurabh Kulkarni
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2020-12-31 00:00:00.000000000 Z
11
+ date: 2021-01-03 00:00:00.000000000 Z
12
12
  dependencies: []
13
13
  description:
14
14
  email:
@@ -42,8 +42,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
42
42
  - !ruby/object:Gem::Version
43
43
  version: '0'
44
44
  requirements: []
45
- rubygems_version: 3.1.2
45
+ rubygems_version: 3.0.3
46
46
  signing_key:
47
47
  specification_version: 4
48
- summary: FluentD parser plugin to parse TKGI metadata
48
+ summary: Fluentd parser plugin to parse TKGI metadata
49
49
  test_files: []