fluent-plugin-systemd 1.0.0.rc3 → 1.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +67 -46
- metadata +4 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 05b2387f068e7c0ec23f74f61bb5d1b031fbd670
|
4
|
+
data.tar.gz: 438b9f07ea3b23d6af05077ec460f51e413002e4
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 23d0e490a9befa2d01ea8fd6092dbed234ea8a31e3433cb458a584195d0ba6a08376881641df75db4fb139887e869b66c935a58ce7daf796d24bea8b533020fa
|
7
|
+
data.tar.gz: 7ac5767e188b7ef944c62415a7e13eb100effa83f05784377b7fa071b9a6cbc52eaf72f7f45a5d593d0302474e4ca858d746a616ae1ec2a8522c62f20cc09dc8
|
data/README.md
CHANGED
@@ -4,8 +4,8 @@
|
|
4
4
|
|
5
5
|
## Overview
|
6
6
|
|
7
|
-
**systemd** input plugin
|
8
|
-
**systemd** filter plugin
|
7
|
+
* **systemd** input plugin to read logs from the systemd journal
|
8
|
+
* **systemd** filter plugin for basic manipulation of systemd journal entries
|
9
9
|
|
10
10
|
## Support
|
11
11
|
|
@@ -21,19 +21,22 @@ Join the #plugin-systemd channel on the [Fluentd Slack](http://slack.fluentd.org
|
|
21
21
|
| > 0.1.0 | >= 0.14.11, < 2 | 3 | >= 2.1 |
|
22
22
|
| 0.0.x | ~> 0.12.0 | 2 | >= 1.9 |
|
23
23
|
|
24
|
-
* The
|
25
|
-
* The 0.0.x series (compatible with fluentd v0.12, and td-agent 2) is
|
26
|
-
* The next version is currently under development on the [1.0.0 branch](https://github.com/reevoo/fluent-plugin-systemd/tree/1.0.0) it's progress is tracked [here](https://github.com/reevoo/fluent-plugin-systemd/issues/53)
|
24
|
+
* The 1.x.x series is developed from this branch (master)
|
25
|
+
* The 0.0.x series (compatible with fluentd v0.12, and td-agent 2) is maintained on the [0.0.x branch](https://github.com/reevoo/fluent-plugin-systemd/tree/0.0.x)
|
27
26
|
|
28
27
|
## Installation
|
29
28
|
|
30
29
|
Simply use RubyGems:
|
31
30
|
|
32
|
-
gem install fluent-plugin-systemd -v 0.
|
31
|
+
gem install fluent-plugin-systemd -v 1.0.0
|
33
32
|
|
34
33
|
or
|
35
34
|
|
36
|
-
td-agent-gem install fluent-plugin-systemd -v 0.
|
35
|
+
td-agent-gem install fluent-plugin-systemd -v 1.0.0
|
36
|
+
|
37
|
+
## Upgrading
|
38
|
+
|
39
|
+
If you are upgrading to version 1.0 from a previous version of this plugin take a look at the [upgrade documentation](docs/upgrading.md). A number of deprecated config options were removed so you might need to update your configuration.
|
37
40
|
|
38
41
|
## Input Plugin Configuration
|
39
42
|
|
@@ -49,7 +52,6 @@ or
|
|
49
52
|
path kube-proxy.pos
|
50
53
|
</storage>
|
51
54
|
<entry>
|
52
|
-
field_map {"MESSAGE": "log", "_PID": ["process", "pid"], "_CMDLINE": "process", "_COMM": "cmd"}
|
53
55
|
fields_strip_underscores true
|
54
56
|
fields_lowercase true
|
55
57
|
</entry>
|
@@ -65,7 +67,7 @@ Path to the systemd journal, defaults to `/var/log/journal`
|
|
65
67
|
|
66
68
|
**`filters`**
|
67
69
|
|
68
|
-
_This parameter name is
|
70
|
+
_This parameter name is depreciated and should be renamed to `matches`_
|
69
71
|
|
70
72
|
**`matches`**
|
71
73
|
|
@@ -73,40 +75,21 @@ Expects an array of hashes defining desired matches to filter the log
|
|
73
75
|
messages with. When this property is not specified, this plugin will default to
|
74
76
|
reading all logs from the journal.
|
75
77
|
|
76
|
-
See [matching details](docs/
|
78
|
+
See [matching details](docs/matching.md) for a more exhaustive
|
77
79
|
description of this property and how to use it.
|
78
80
|
|
79
|
-
**`pos_file`**
|
80
|
-
|
81
|
-
_This parameter is deprecated and will be removed in favour of storage in v1.0._
|
82
|
-
|
83
|
-
|
84
|
-
Path to pos file, stores the journald cursor. File is created if does not exist.
|
85
|
-
|
86
81
|
**`storage`**
|
87
82
|
|
88
83
|
Configuration for a [storage plugin](http://docs.fluentd.org/v0.14/articles/storage-plugin-overview) used to store the journald cursor.
|
89
84
|
|
90
|
-
_Upgrading from `pos_file`_
|
91
|
-
|
92
|
-
If `pos_file` is specified in addition to a storage plugin with persistent set to true, the cursor will be
|
93
|
-
copied from the `pos_file` on startup, and the old `pos_file` removed.
|
94
|
-
|
95
85
|
**`read_from_head`**
|
96
86
|
|
97
87
|
If true reads all available journal from head, otherwise starts reading from tail,
|
98
|
-
ignored if
|
99
|
-
|
100
|
-
**`strip_underscores`**
|
101
|
-
|
102
|
-
_This parameter is deprecated and will be removed in favour of entry in v1.0._
|
103
|
-
|
104
|
-
If true strips underscores from the beginning of systemd field names.
|
105
|
-
May be useful if outputting to kibana, as underscore prefixed fields are unindexed there.
|
88
|
+
ignored if cursor exists in storage (and is valid). Defaults to false.
|
106
89
|
|
107
90
|
**`entry`**
|
108
91
|
|
109
|
-
Optional configuration for an
|
92
|
+
Optional configuration for an embedded systemd entry filter. See the [Filter Plugin Configuration](#filter-plugin-configuration) for config reference.
|
110
93
|
|
111
94
|
**`tag`**
|
112
95
|
|
@@ -114,19 +97,20 @@ _Required_
|
|
114
97
|
|
115
98
|
A tag that will be added to events generated by this input.
|
116
99
|
|
117
|
-
### Example
|
118
|
-
|
119
|
-
For an example of a full working setup including the plugin, [take a look at](https://github.com/assemblyline/fluentd)
|
120
100
|
|
121
101
|
## Filter Plugin Configuration
|
122
102
|
|
123
|
-
|
124
|
-
|
125
|
-
|
126
|
-
|
127
|
-
|
128
|
-
|
129
|
-
|
103
|
+
```
|
104
|
+
<filter kube-proxy>
|
105
|
+
@type systemd_entry
|
106
|
+
field_map {"MESSAGE": "log", "_PID": ["process", "pid"], "_CMDLINE": "process", "_COMM": "cmd"}
|
107
|
+
field_map_strict false
|
108
|
+
fields_lowercase true
|
109
|
+
fields_strip_underscores true
|
110
|
+
</filter>
|
111
|
+
```
|
112
|
+
|
113
|
+
_Note that the following configurations can be embedded in a systemd source block, within an entry block, you only need to use a filter directly for more complicated workflows._
|
130
114
|
|
131
115
|
**`field_map`**
|
132
116
|
|
@@ -151,7 +135,7 @@ If true, lowercase all non-mapped fields. Defaults to false.
|
|
151
135
|
|
152
136
|
If true, strip leading underscores from all non-mapped fields. Defaults to false.
|
153
137
|
|
154
|
-
### Example
|
138
|
+
### Filter Example
|
155
139
|
|
156
140
|
Given a systemd journal source entry:
|
157
141
|
```
|
@@ -178,26 +162,63 @@ The resulting entry using the above sample configuration:
|
|
178
162
|
|
179
163
|
## Common Issues
|
180
164
|
|
181
|
-
> ### When I look at fluentd logs, everything looks fine but no journal logs are read
|
165
|
+
> ### When I look at fluentd logs, everything looks fine but no journal logs are read ?
|
182
166
|
|
183
|
-
This is commonly caused when the user running fluentd does not have
|
167
|
+
This is commonly caused when the user running fluentd does not have the correct permissions
|
184
168
|
to read the systemd journal.
|
185
169
|
|
186
|
-
|
170
|
+
According to the [systemd documentation](https://www.freedesktop.org/software/systemd/man/systemd-journald.service.html):
|
187
171
|
> Journal files are, by default, owned and readable by the "systemd-journal" system group but are not writable. Adding a user to this group thus enables her/him to read the journal files.
|
188
172
|
|
173
|
+
> ### How can I deal with multi-line logs ?
|
174
|
+
|
175
|
+
Ideally you want to ensure that your logs are saved to the systemd journal as a single entry regardless of how many lines they span.
|
176
|
+
|
177
|
+
It is possible for applications to naively support this (but only if they have tight integration with systemd it seems) see: https://github.com/systemd/systemd/issues/5188.
|
178
|
+
|
179
|
+
Typically you would not be able to this, so another way is to configure your logger to replace newline characters with something else. See this blog post for an example configuring a Java logging library to do this https://fabianlee.org/2018/03/09/java-collapsing-multiline-stack-traces-into-a-single-log-event-using-spring-backed-by-logback-or-log4j2/
|
180
|
+
|
181
|
+
Another strategy would be to use a plugin like [fluent-plugin-concat](https://github.com/fluent-plugins-nursery/fluent-plugin-concat) to combine multi line logs into a single event, this is more tricky though because you need to be able to identify the first and last lines of a multi line message with a regex.
|
182
|
+
|
183
|
+
> ### How can I use this plugin inside of a docker container ?
|
184
|
+
|
185
|
+
* Install the [systemd dependencies](#dependencies) if required
|
186
|
+
* You can use an [offical fluentd docker](https://github.com/fluent/fluentd-docker-image) image as a base, (choose the debian based version, as alpine linux doesn't support systemd).
|
187
|
+
* Bind mount `/var/log/journal` into your container.
|
188
|
+
|
189
|
+
### Example
|
190
|
+
|
191
|
+
For an example of a full working setup including the plugin, take a look at [the fluentd kubernetes daemonset](https://github.com/fluent/fluentd-kubernetes-daemonset)
|
189
192
|
|
190
193
|
## Dependencies
|
191
194
|
|
192
195
|
This plugin depends on libsystemd
|
193
196
|
|
197
|
+
On Debian or Ubuntu you might need to install the libsystemd0 package:
|
198
|
+
|
199
|
+
```
|
200
|
+
apt-get install libsystemd0
|
201
|
+
```
|
202
|
+
|
203
|
+
On CentOS or RHEL you might need to install the systemd package:
|
204
|
+
|
205
|
+
```
|
206
|
+
yum install -y systemd
|
207
|
+
```
|
208
|
+
|
209
|
+
If you want to do this in a CentOS docker image you might first need to remove the `fakesystemd` package.
|
210
|
+
|
211
|
+
```
|
212
|
+
yum remove -y fakesystemd
|
213
|
+
```
|
214
|
+
|
194
215
|
## Running the tests
|
195
216
|
|
196
217
|
To run the tests with docker on several distros simply run `rake`
|
197
218
|
|
198
219
|
For systems with systemd installed you can run the tests against your installed libsystemd with `rake test`
|
199
220
|
|
200
|
-
##
|
221
|
+
## License
|
201
222
|
|
202
223
|
[Apache-2.0](LICENCE)
|
203
224
|
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-systemd
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.0.0
|
4
|
+
version: 1.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Ed Robinson
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2018-05-
|
11
|
+
date: 2018-05-10 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
@@ -127,9 +127,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
127
127
|
version: '0'
|
128
128
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
129
129
|
requirements:
|
130
|
-
- - "
|
130
|
+
- - ">="
|
131
131
|
- !ruby/object:Gem::Version
|
132
|
-
version:
|
132
|
+
version: '0'
|
133
133
|
requirements: []
|
134
134
|
rubyforge_project:
|
135
135
|
rubygems_version: 2.6.13
|