fluent-plugin-grafana-loki 1.2.7 → 1.2.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +1 -211
- data/bin/setup +1 -1
- data/lib/fluent/plugin/out_loki.rb +17 -8
- metadata +4 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 33ae09dd97c7f753b7bc507a701df03e4630c9561681c88fcdfb02446157b920
|
4
|
+
data.tar.gz: 1eb0e85e3136bcb251a861cca28693394c90615e403cc02306650607ec9ca98b
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 3e52e7c93fcd20ea3ce687a0c74b0273de4367ede9addd14e5c59bbba477f769db1d1a8bd8bb84dc5e464dfa095fb7e9ce3214504535c4fbd903744e1b410348
|
7
|
+
data.tar.gz: 2d7bbca865bb13f2a2b8dc3a9e62ae84a9740fd1a09413569a8b892a4453ebada71205d678e6824885145b4d2bfdd694423d0db9ec3360d32a14423fc1852716
|
data/README.md
CHANGED
@@ -1,216 +1,6 @@
|
|
1
1
|
# fluent-plugin-grafana-loki
|
2
2
|
|
3
|
-
[Fluentd](https://fluentd.org/) output plugin to ship logs to a Loki server.
|
4
|
-
|
5
|
-
This plugin offers two line formats and uses protobuf to send compressed data to Loki.
|
6
|
-
|
7
|
-
Key features:
|
8
|
-
* extra_labels - labels to be added to every line of a logfile, useful for designating environments
|
9
|
-
* label - This section allows you to specify labels from your log fields
|
10
|
-
|
11
|
-
## Installation
|
12
|
-
|
13
|
-
```
|
14
|
-
$ gem install fluent-plugin-grafana-loki
|
15
|
-
```
|
16
|
-
|
17
|
-
## Usage
|
18
|
-
In your Fluentd configuration, use `@type loki`. Additional configuration is optional, default values would look like this:
|
19
|
-
```
|
20
|
-
<match **>
|
21
|
-
@type loki
|
22
|
-
url "https://logs-us-west1.grafana.net"
|
23
|
-
username "#{ENV['LOKI_USERNAME']}"
|
24
|
-
password "#{ENV['LOKI_PASSWORD']}"
|
25
|
-
extra_labels {"env":"dev"}
|
26
|
-
flush_interval 10s
|
27
|
-
flush_at_shutdown true
|
28
|
-
buffer_chunk_limit 1m
|
29
|
-
</match>
|
30
|
-
```
|
31
|
-
|
32
|
-
### Using labels
|
33
|
-
|
34
|
-
Simple label from top level attribute
|
35
|
-
```
|
36
|
-
<match mytag>
|
37
|
-
@type loki
|
38
|
-
# ...
|
39
|
-
<label>
|
40
|
-
fluentd_worker
|
41
|
-
</label>
|
42
|
-
# ...
|
43
|
-
</match>
|
44
|
-
```
|
45
|
-
|
46
|
-
You can rewrite the label keys as well as the following
|
47
|
-
|
48
|
-
```
|
49
|
-
<match mytag>
|
50
|
-
@type loki
|
51
|
-
# ...
|
52
|
-
<label>
|
53
|
-
worker fluentd_worker
|
54
|
-
</label>
|
55
|
-
# ...
|
56
|
-
</match>
|
57
|
-
```
|
58
|
-
|
59
|
-
You can use record accessor syntax for nested field. https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-record_accessor#syntax
|
60
|
-
|
61
|
-
```
|
62
|
-
<match mytag>
|
63
|
-
@type loki
|
64
|
-
# ...
|
65
|
-
<label>
|
66
|
-
container $.kubernetes.container
|
67
|
-
</label>
|
68
|
-
# ...
|
69
|
-
</match>
|
70
|
-
```
|
71
|
-
|
72
|
-
### Extracting Kubernetes labels
|
73
|
-
|
74
|
-
As Kubernetes labels are a list of nested key-value pairs there is a separate option to extract them.
|
75
|
-
Note that special characters like "`. - /`" will be overwritten with `_`.
|
76
|
-
Use with the `remove_keys kubernetes` option to eliminate metadata from the log.
|
77
|
-
```
|
78
|
-
<match mytag>
|
79
|
-
@type loki
|
80
|
-
# ...
|
81
|
-
extract_kubernetes_labels true
|
82
|
-
remove_keys kubernetes
|
83
|
-
<label>
|
84
|
-
container $.kubernetes.container
|
85
|
-
</label>
|
86
|
-
# ...
|
87
|
-
</match>
|
88
|
-
```
|
89
|
-
|
90
|
-
### Multi-worker usage
|
91
|
-
|
92
|
-
Loki doesn't currently support out-of-order inserts - if you try to insert a log entry an earlier timestamp after a log entry with with identical labels but a later timestamp, the insert will fail with `HTTP status code: 500, message: rpc error: code = Unknown desc = Entry out of order`. Therefore, in order to use this plugin in a multi worker Fluentd setup, you'll need to include the worker ID in the labels.
|
93
|
-
|
94
|
-
For example, using [fluent-plugin-record-modifier](https://github.com/repeatedly/fluent-plugin-record-modifier):
|
95
|
-
```
|
96
|
-
<filter mytag>
|
97
|
-
@type record_modifier
|
98
|
-
<record>
|
99
|
-
fluentd_worker "#{worker_id}"
|
100
|
-
</record>
|
101
|
-
</filter>
|
102
|
-
|
103
|
-
<match mytag>
|
104
|
-
@type loki
|
105
|
-
# ...
|
106
|
-
<label>
|
107
|
-
fluentd_worker
|
108
|
-
</label>
|
109
|
-
# ...
|
110
|
-
</match>
|
111
|
-
```
|
112
|
-
|
113
|
-
### Using multiple buffer flush threads
|
114
|
-
|
115
|
-
Similarly, when using `flush_thread_count` > 1 in the [`buffer`](https://docs.fluentd.org/configuration/buffer-section#flushing-parameters)
|
116
|
-
section, a thread identifier must be added as a label to ensure that log chunks flushed in parallel to loki by fluentd always have increasing
|
117
|
-
times for their unique label sets.
|
118
|
-
|
119
|
-
This plugin automatically adds a `fluentd_thread` label with the name of the buffer flush thread when `flush_thread_count` > 1.
|
120
|
-
|
121
|
-
## Docker Image
|
122
|
-
|
123
|
-
There is a Docker image `grafana/fluent-plugin-grafana-loki:master` which contains default configuration files to git log information
|
124
|
-
a host's `/var/log` dir, and from the host's Journald. To use it, you can set the `LOKI_URL`, `LOKI_USERNAME`, and `LOKI_PASSWORD` environment variables (you can leave the USERNAME and PASSWORD blank if they're not used.)
|
125
|
-
|
126
|
-
|
127
|
-
A Docker Swarm Compose configuration that will work looks like:
|
128
|
-
|
129
|
-
```
|
130
|
-
services:
|
131
|
-
fluentd:
|
132
|
-
image: grafana/fluent-plugin-grafana-loki:master
|
133
|
-
command:
|
134
|
-
- "fluentd"
|
135
|
-
- "-v"
|
136
|
-
- "-p"
|
137
|
-
- "/fluentd/plugins"
|
138
|
-
environment:
|
139
|
-
LOKI_URL: http://loki:3100
|
140
|
-
LOKI_USERNAME:
|
141
|
-
LOKI_PASSWORD:
|
142
|
-
deploy:
|
143
|
-
mode: global
|
144
|
-
configs:
|
145
|
-
- source: loki_config
|
146
|
-
target: /fluentd/etc/loki/loki.conf
|
147
|
-
networks:
|
148
|
-
- loki
|
149
|
-
volumes:
|
150
|
-
- host_logs:/var/log
|
151
|
-
# Needed for journald log ingestion:
|
152
|
-
- /etc/machine-id:/etc/machine-id
|
153
|
-
- /dev/log:/dev/log
|
154
|
-
- /var/run/systemd/journal/:/var/run/systemd/journal/
|
155
|
-
logging:
|
156
|
-
options:
|
157
|
-
tag: infra.monitoring
|
158
|
-
```
|
159
|
-
|
160
|
-
## Configuration
|
161
|
-
|
162
|
-
### url
|
163
|
-
The url of the Loki server to send logs to. When sending data the publish path (`/api/prom/push`) will automatically be appended.
|
164
|
-
By default the url is set to `https://logs-us-west1.grafana.net`, the url of the Grafana Labs preview (hosted Loki)[https://grafana.com/loki] service.
|
165
|
-
|
166
|
-
#### Proxy Support
|
167
|
-
|
168
|
-
Starting with version 0.8.0, this gem uses excon, which supports proxy with environment variables - https://github.com/excon/excon#proxy-support
|
169
|
-
|
170
|
-
### username / password
|
171
|
-
Specify a username and password if the Loki server requires authentication.
|
172
|
-
If using the GrafanaLab's hosted Loki, the username needs to be set to your instanceId and the password should be a Grafana.com api key.
|
173
|
-
|
174
|
-
### tenant
|
175
|
-
Loki is a multi-tenant log storage platform and all requests sent must include a tenant. For some installations the tenant will be set automatically by an authenticating proxy. Otherwise you can define a tenant to be passed through. The tenant can be any string value.
|
176
|
-
|
177
|
-
### client certificate verification
|
178
|
-
Specify a pair of client certificate and private key with `cert` and `key` if a reverse proxy with client certificate verification is configured in front of Loki. `ca_cert` can also be specified if the server uses custom certificate authority.
|
179
|
-
|
180
|
-
```
|
181
|
-
<match **>
|
182
|
-
@type loki
|
183
|
-
|
184
|
-
url "https://loki"
|
185
|
-
|
186
|
-
cert /path/to/certificate.pem
|
187
|
-
key /path/to/key.key
|
188
|
-
ca_cert /path/to/ca.pem
|
189
|
-
|
190
|
-
...
|
191
|
-
</match>
|
192
|
-
```
|
193
|
-
|
194
|
-
### output format
|
195
|
-
Loki is intended to index and group log streams using only a small set of labels. It is not intended for full-text indexing. When sending logs to Loki the majority of log message will be sent as a single log "line".
|
196
|
-
|
197
|
-
There are few configurations settings to control the output format.
|
198
|
-
- extra_labels: (default: nil) set of labels to include with every Loki stream. eg `{"env":"dev", "datacenter": "dc1"}`
|
199
|
-
- remove_keys: (default: nil) comma separated list of needless record keys to remove. All other keys will be placed into the log line. You can use [record_accessor syntax](https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-record_accessor#syntax).
|
200
|
-
- line_format: format to use when flattening the record to a log line. Valid values are "json" or "key_value". If set to "json" the log line sent to Loki will be the fluentd record (excluding any keys extracted out as labels) dumped as json. If set to "key_value", the log line will be each item in the record concatenated together (separated by a single space) in the format `<key>=<value>`.
|
201
|
-
- drop_single_key: if set to true and after extracting label_keys a record only has a single key remaining, the log line sent to Loki will just be the value of the record key.
|
202
|
-
|
203
|
-
### Buffer options
|
204
|
-
|
205
|
-
`fluentd-plugin-loki` extends [Fluentd's builtin Output plugin](https://docs.fluentd.org/v1.0/articles/output-plugin-overview) and use `compat_parameters` plugin helper. It adds the following options:
|
206
|
-
|
207
|
-
```
|
208
|
-
buffer_type memory
|
209
|
-
flush_interval 10s
|
210
|
-
retry_limit 17
|
211
|
-
retry_wait 1.0
|
212
|
-
num_threads 1
|
213
|
-
```
|
3
|
+
[Fluentd](https://fluentd.org/) output plugin to ship logs to a Loki server. See [docs/client/fluentd/README.md](../../docs/clients/fluentd/README.md) for detailed information.
|
214
4
|
|
215
5
|
## Development
|
216
6
|
|
data/bin/setup
CHANGED
@@ -17,7 +17,6 @@
|
|
17
17
|
|
18
18
|
require 'fluent/plugin/output'
|
19
19
|
require 'net/http'
|
20
|
-
require 'uri'
|
21
20
|
require 'yajl'
|
22
21
|
require 'time'
|
23
22
|
|
@@ -70,9 +69,14 @@ module Fluent
|
|
70
69
|
config_set_default :chunk_keys, []
|
71
70
|
end
|
72
71
|
|
73
|
-
def configure(conf)
|
72
|
+
def configure(conf) # rubocop:disable Metrics/CyclomaticComplexity
|
74
73
|
compat_parameters_convert(conf, :buffer)
|
75
74
|
super
|
75
|
+
@uri = URI.parse(@url + '/loki/api/v1/push')
|
76
|
+
unless @uri.is_a?(URI::HTTP) || @uri.is_a?(URI::HTTPS)
|
77
|
+
raise Fluent::ConfigError, 'url parameter must be valid HTTP'
|
78
|
+
end
|
79
|
+
|
76
80
|
@record_accessors = {}
|
77
81
|
conf.elements.select { |element| element.name == 'label' }.each do |element|
|
78
82
|
element.each_pair do |k, v|
|
@@ -127,27 +131,26 @@ module Fluent
|
|
127
131
|
body = { 'streams' => payload }
|
128
132
|
|
129
133
|
# add ingest path to loki url
|
130
|
-
uri = URI.parse(url + '/loki/api/v1/push')
|
131
134
|
|
132
135
|
req = Net::HTTP::Post.new(
|
133
|
-
uri.request_uri
|
136
|
+
@uri.request_uri
|
134
137
|
)
|
135
138
|
req.add_field('Content-Type', 'application/json')
|
136
139
|
req.add_field('X-Scope-OrgID', @tenant) if @tenant
|
137
140
|
req.body = Yajl.dump(body)
|
138
141
|
req.basic_auth(@username, @password) if @username
|
139
142
|
|
140
|
-
opts = ssl_opts(uri)
|
143
|
+
opts = ssl_opts(@uri)
|
141
144
|
|
142
145
|
log.debug "sending #{req.body.length} bytes to loki"
|
143
|
-
res = Net::HTTP.start(uri.
|
146
|
+
res = Net::HTTP.start(@uri.host, @uri.port, **opts) { |http| http.request(req) }
|
144
147
|
unless res&.is_a?(Net::HTTPSuccess)
|
145
148
|
res_summary = if res
|
146
149
|
"#{res.code} #{res.message} #{res.body}"
|
147
150
|
else
|
148
151
|
'res=nil'
|
149
152
|
end
|
150
|
-
log.warn "failed to #{req.method} #{uri} (#{res_summary})"
|
153
|
+
log.warn "failed to #{req.method} #{@uri} (#{res_summary})"
|
151
154
|
log.warn Yajl.dump(body)
|
152
155
|
|
153
156
|
end
|
@@ -215,7 +218,13 @@ module Fluent
|
|
215
218
|
end
|
216
219
|
|
217
220
|
def to_nano(time)
|
218
|
-
time
|
221
|
+
# time is a Fluent::EventTime object, or an Integer which represents unix timestamp (seconds from Epoch)
|
222
|
+
# https://docs.fluentd.org/plugin-development/api-plugin-output#chunk-each-and-block
|
223
|
+
if time.is_a?(Fluent::EventTime)
|
224
|
+
time.to_i * (10**9) + time.nsec
|
225
|
+
else
|
226
|
+
time.to_i * (10**9)
|
227
|
+
end
|
219
228
|
end
|
220
229
|
|
221
230
|
def record_to_line(record)
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-grafana-loki
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.2.
|
4
|
+
version: 1.2.9
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- woodsaj
|
@@ -10,7 +10,7 @@ authors:
|
|
10
10
|
autorequire:
|
11
11
|
bindir: bin
|
12
12
|
cert_chain: []
|
13
|
-
date: 2020-
|
13
|
+
date: 2020-02-18 00:00:00.000000000 Z
|
14
14
|
dependencies:
|
15
15
|
- !ruby/object:Gem::Dependency
|
16
16
|
name: bundler
|
@@ -60,7 +60,7 @@ dependencies:
|
|
60
60
|
requirements:
|
61
61
|
- - ">="
|
62
62
|
- !ruby/object:Gem::Version
|
63
|
-
version:
|
63
|
+
version: 1.9.0
|
64
64
|
- - "<"
|
65
65
|
- !ruby/object:Gem::Version
|
66
66
|
version: '2'
|
@@ -70,7 +70,7 @@ dependencies:
|
|
70
70
|
requirements:
|
71
71
|
- - ">="
|
72
72
|
- !ruby/object:Gem::Version
|
73
|
-
version:
|
73
|
+
version: 1.9.0
|
74
74
|
- - "<"
|
75
75
|
- !ruby/object:Gem::Version
|
76
76
|
version: '2'
|