fluent-plugin-grafana-loki 1.2.7 → 1.2.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 6272020eefeef15ea9b65b49ede71e60c08c298c7c56d2ca366a95130fbf3f3a
4
- data.tar.gz: b3699f4d8dcc778bfc63d6d2a0fe456ecfd95fb55dfbf6048af0846fdd741259
3
+ metadata.gz: 33ae09dd97c7f753b7bc507a701df03e4630c9561681c88fcdfb02446157b920
4
+ data.tar.gz: 1eb0e85e3136bcb251a861cca28693394c90615e403cc02306650607ec9ca98b
5
5
  SHA512:
6
- metadata.gz: 15cd081338b9da5c2b8ea7212d4ebd0cb455782cb02722992bdd671e6715d960c6a1bfb828d14a763c60175cc82d156f912a624dde81072eaeb85c726b8a06c5
7
- data.tar.gz: dbec238d8c2e2b52c5508cc1cebe9a55677dfbb6de26de33af2b45399742fd57b8a5a727b6e763560b4f775eafa5172c4fed86e4cd9162ad06b90e2233708851
6
+ metadata.gz: 3e52e7c93fcd20ea3ce687a0c74b0273de4367ede9addd14e5c59bbba477f769db1d1a8bd8bb84dc5e464dfa095fb7e9ce3214504535c4fbd903744e1b410348
7
+ data.tar.gz: 2d7bbca865bb13f2a2b8dc3a9e62ae84a9740fd1a09413569a8b892a4453ebada71205d678e6824885145b4d2bfdd694423d0db9ec3360d32a14423fc1852716
data/README.md CHANGED
@@ -1,216 +1,6 @@
1
1
  # fluent-plugin-grafana-loki
2
2
 
3
- [Fluentd](https://fluentd.org/) output plugin to ship logs to a Loki server.
4
-
5
- This plugin offers two line formats and uses protobuf to send compressed data to Loki.
6
-
7
- Key features:
8
- * extra_labels - labels to be added to every line of a logfile, useful for designating environments
9
- * label - This section allows you to specify labels from your log fields
10
-
11
- ## Installation
12
-
13
- ```
14
- $ gem install fluent-plugin-grafana-loki
15
- ```
16
-
17
- ## Usage
18
- In your Fluentd configuration, use `@type loki`. Additional configuration is optional, default values would look like this:
19
- ```
20
- <match **>
21
- @type loki
22
- url "https://logs-us-west1.grafana.net"
23
- username "#{ENV['LOKI_USERNAME']}"
24
- password "#{ENV['LOKI_PASSWORD']}"
25
- extra_labels {"env":"dev"}
26
- flush_interval 10s
27
- flush_at_shutdown true
28
- buffer_chunk_limit 1m
29
- </match>
30
- ```
31
-
32
- ### Using labels
33
-
34
- Simple label from top level attribute
35
- ```
36
- <match mytag>
37
- @type loki
38
- # ...
39
- <label>
40
- fluentd_worker
41
- </label>
42
- # ...
43
- </match>
44
- ```
45
-
46
- You can rewrite the label keys as well as the following
47
-
48
- ```
49
- <match mytag>
50
- @type loki
51
- # ...
52
- <label>
53
- worker fluentd_worker
54
- </label>
55
- # ...
56
- </match>
57
- ```
58
-
59
- You can use record accessor syntax for nested field. https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-record_accessor#syntax
60
-
61
- ```
62
- <match mytag>
63
- @type loki
64
- # ...
65
- <label>
66
- container $.kubernetes.container
67
- </label>
68
- # ...
69
- </match>
70
- ```
71
-
72
- ### Extracting Kubernetes labels
73
-
74
- As Kubernetes labels are a list of nested key-value pairs there is a separate option to extract them.
75
- Note that special characters like "`. - /`" will be overwritten with `_`.
76
- Use with the `remove_keys kubernetes` option to eliminate metadata from the log.
77
- ```
78
- <match mytag>
79
- @type loki
80
- # ...
81
- extract_kubernetes_labels true
82
- remove_keys kubernetes
83
- <label>
84
- container $.kubernetes.container
85
- </label>
86
- # ...
87
- </match>
88
- ```
89
-
90
- ### Multi-worker usage
91
-
92
- Loki doesn't currently support out-of-order inserts - if you try to insert a log entry an earlier timestamp after a log entry with with identical labels but a later timestamp, the insert will fail with `HTTP status code: 500, message: rpc error: code = Unknown desc = Entry out of order`. Therefore, in order to use this plugin in a multi worker Fluentd setup, you'll need to include the worker ID in the labels.
93
-
94
- For example, using [fluent-plugin-record-modifier](https://github.com/repeatedly/fluent-plugin-record-modifier):
95
- ```
96
- <filter mytag>
97
- @type record_modifier
98
- <record>
99
- fluentd_worker "#{worker_id}"
100
- </record>
101
- </filter>
102
-
103
- <match mytag>
104
- @type loki
105
- # ...
106
- <label>
107
- fluentd_worker
108
- </label>
109
- # ...
110
- </match>
111
- ```
112
-
113
- ### Using multiple buffer flush threads
114
-
115
- Similarly, when using `flush_thread_count` > 1 in the [`buffer`](https://docs.fluentd.org/configuration/buffer-section#flushing-parameters)
116
- section, a thread identifier must be added as a label to ensure that log chunks flushed in parallel to loki by fluentd always have increasing
117
- times for their unique label sets.
118
-
119
- This plugin automatically adds a `fluentd_thread` label with the name of the buffer flush thread when `flush_thread_count` > 1.
120
-
121
- ## Docker Image
122
-
123
- There is a Docker image `grafana/fluent-plugin-grafana-loki:master` which contains default configuration files to git log information
124
- a host's `/var/log` dir, and from the host's Journald. To use it, you can set the `LOKI_URL`, `LOKI_USERNAME`, and `LOKI_PASSWORD` environment variables (you can leave the USERNAME and PASSWORD blank if they're not used.)
125
-
126
-
127
- A Docker Swarm Compose configuration that will work looks like:
128
-
129
- ```
130
- services:
131
- fluentd:
132
- image: grafana/fluent-plugin-grafana-loki:master
133
- command:
134
- - "fluentd"
135
- - "-v"
136
- - "-p"
137
- - "/fluentd/plugins"
138
- environment:
139
- LOKI_URL: http://loki:3100
140
- LOKI_USERNAME:
141
- LOKI_PASSWORD:
142
- deploy:
143
- mode: global
144
- configs:
145
- - source: loki_config
146
- target: /fluentd/etc/loki/loki.conf
147
- networks:
148
- - loki
149
- volumes:
150
- - host_logs:/var/log
151
- # Needed for journald log ingestion:
152
- - /etc/machine-id:/etc/machine-id
153
- - /dev/log:/dev/log
154
- - /var/run/systemd/journal/:/var/run/systemd/journal/
155
- logging:
156
- options:
157
- tag: infra.monitoring
158
- ```
159
-
160
- ## Configuration
161
-
162
- ### url
163
- The url of the Loki server to send logs to. When sending data the publish path (`/api/prom/push`) will automatically be appended.
164
- By default the url is set to `https://logs-us-west1.grafana.net`, the url of the Grafana Labs preview (hosted Loki)[https://grafana.com/loki] service.
165
-
166
- #### Proxy Support
167
-
168
- Starting with version 0.8.0, this gem uses excon, which supports proxy with environment variables - https://github.com/excon/excon#proxy-support
169
-
170
- ### username / password
171
- Specify a username and password if the Loki server requires authentication.
172
- If using the GrafanaLab's hosted Loki, the username needs to be set to your instanceId and the password should be a Grafana.com api key.
173
-
174
- ### tenant
175
- Loki is a multi-tenant log storage platform and all requests sent must include a tenant. For some installations the tenant will be set automatically by an authenticating proxy. Otherwise you can define a tenant to be passed through. The tenant can be any string value.
176
-
177
- ### client certificate verification
178
- Specify a pair of client certificate and private key with `cert` and `key` if a reverse proxy with client certificate verification is configured in front of Loki. `ca_cert` can also be specified if the server uses custom certificate authority.
179
-
180
- ```
181
- <match **>
182
- @type loki
183
-
184
- url "https://loki"
185
-
186
- cert /path/to/certificate.pem
187
- key /path/to/key.key
188
- ca_cert /path/to/ca.pem
189
-
190
- ...
191
- </match>
192
- ```
193
-
194
- ### output format
195
- Loki is intended to index and group log streams using only a small set of labels. It is not intended for full-text indexing. When sending logs to Loki the majority of log message will be sent as a single log "line".
196
-
197
- There are few configurations settings to control the output format.
198
- - extra_labels: (default: nil) set of labels to include with every Loki stream. eg `{"env":"dev", "datacenter": "dc1"}`
199
- - remove_keys: (default: nil) comma separated list of needless record keys to remove. All other keys will be placed into the log line. You can use [record_accessor syntax](https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-record_accessor#syntax).
200
- - line_format: format to use when flattening the record to a log line. Valid values are "json" or "key_value". If set to "json" the log line sent to Loki will be the fluentd record (excluding any keys extracted out as labels) dumped as json. If set to "key_value", the log line will be each item in the record concatenated together (separated by a single space) in the format `<key>=<value>`.
201
- - drop_single_key: if set to true and after extracting label_keys a record only has a single key remaining, the log line sent to Loki will just be the value of the record key.
202
-
203
- ### Buffer options
204
-
205
- `fluentd-plugin-loki` extends [Fluentd's builtin Output plugin](https://docs.fluentd.org/v1.0/articles/output-plugin-overview) and use `compat_parameters` plugin helper. It adds the following options:
206
-
207
- ```
208
- buffer_type memory
209
- flush_interval 10s
210
- retry_limit 17
211
- retry_wait 1.0
212
- num_threads 1
213
- ```
3
+ [Fluentd](https://fluentd.org/) output plugin to ship logs to a Loki server. See [docs/client/fluentd/README.md](../../docs/clients/fluentd/README.md) for detailed information.
214
4
 
215
5
  ## Development
216
6
 
data/bin/setup CHANGED
@@ -3,5 +3,5 @@ set -euo pipefail
3
3
  IFS=$'\n\t'
4
4
  set -vx
5
5
 
6
- sudo gem install bundle
6
+ gem install bundler
7
7
  bundle install
@@ -17,7 +17,6 @@
17
17
 
18
18
  require 'fluent/plugin/output'
19
19
  require 'net/http'
20
- require 'uri'
21
20
  require 'yajl'
22
21
  require 'time'
23
22
 
@@ -70,9 +69,14 @@ module Fluent
70
69
  config_set_default :chunk_keys, []
71
70
  end
72
71
 
73
- def configure(conf)
72
+ def configure(conf) # rubocop:disable Metrics/CyclomaticComplexity
74
73
  compat_parameters_convert(conf, :buffer)
75
74
  super
75
+ @uri = URI.parse(@url + '/loki/api/v1/push')
76
+ unless @uri.is_a?(URI::HTTP) || @uri.is_a?(URI::HTTPS)
77
+ raise Fluent::ConfigError, 'url parameter must be valid HTTP'
78
+ end
79
+
76
80
  @record_accessors = {}
77
81
  conf.elements.select { |element| element.name == 'label' }.each do |element|
78
82
  element.each_pair do |k, v|
@@ -127,27 +131,26 @@ module Fluent
127
131
  body = { 'streams' => payload }
128
132
 
129
133
  # add ingest path to loki url
130
- uri = URI.parse(url + '/loki/api/v1/push')
131
134
 
132
135
  req = Net::HTTP::Post.new(
133
- uri.request_uri
136
+ @uri.request_uri
134
137
  )
135
138
  req.add_field('Content-Type', 'application/json')
136
139
  req.add_field('X-Scope-OrgID', @tenant) if @tenant
137
140
  req.body = Yajl.dump(body)
138
141
  req.basic_auth(@username, @password) if @username
139
142
 
140
- opts = ssl_opts(uri)
143
+ opts = ssl_opts(@uri)
141
144
 
142
145
  log.debug "sending #{req.body.length} bytes to loki"
143
- res = Net::HTTP.start(uri.hostname, uri.port, **opts) { |http| http.request(req) }
146
+ res = Net::HTTP.start(@uri.host, @uri.port, **opts) { |http| http.request(req) }
144
147
  unless res&.is_a?(Net::HTTPSuccess)
145
148
  res_summary = if res
146
149
  "#{res.code} #{res.message} #{res.body}"
147
150
  else
148
151
  'res=nil'
149
152
  end
150
- log.warn "failed to #{req.method} #{uri} (#{res_summary})"
153
+ log.warn "failed to #{req.method} #{@uri} (#{res_summary})"
151
154
  log.warn Yajl.dump(body)
152
155
 
153
156
  end
@@ -215,7 +218,13 @@ module Fluent
215
218
  end
216
219
 
217
220
  def to_nano(time)
218
- time.to_i * (10**9) + time.nsec
221
+ # time is a Fluent::EventTime object, or an Integer which represents unix timestamp (seconds from Epoch)
222
+ # https://docs.fluentd.org/plugin-development/api-plugin-output#chunk-each-and-block
223
+ if time.is_a?(Fluent::EventTime)
224
+ time.to_i * (10**9) + time.nsec
225
+ else
226
+ time.to_i * (10**9)
227
+ end
219
228
  end
220
229
 
221
230
  def record_to_line(record)
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-grafana-loki
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.2.7
4
+ version: 1.2.9
5
5
  platform: ruby
6
6
  authors:
7
7
  - woodsaj
@@ -10,7 +10,7 @@ authors:
10
10
  autorequire:
11
11
  bindir: bin
12
12
  cert_chain: []
13
- date: 2020-01-17 00:00:00.000000000 Z
13
+ date: 2020-02-18 00:00:00.000000000 Z
14
14
  dependencies:
15
15
  - !ruby/object:Gem::Dependency
16
16
  name: bundler
@@ -60,7 +60,7 @@ dependencies:
60
60
  requirements:
61
61
  - - ">="
62
62
  - !ruby/object:Gem::Version
63
- version: 0.14.10
63
+ version: 1.9.0
64
64
  - - "<"
65
65
  - !ruby/object:Gem::Version
66
66
  version: '2'
@@ -70,7 +70,7 @@ dependencies:
70
70
  requirements:
71
71
  - - ">="
72
72
  - !ruby/object:Gem::Version
73
- version: 0.14.10
73
+ version: 1.9.0
74
74
  - - "<"
75
75
  - !ruby/object:Gem::Version
76
76
  version: '2'