fluent-plugin-grafana-loki 1.2.9 → 1.2.15

Sign up to get free protection for your applications and to get access to all the features.
Files changed (4) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +69 -3
  3. data/lib/fluent/plugin/out_loki.rb +81 -41
  4. metadata +52 -10
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 33ae09dd97c7f753b7bc507a701df03e4630c9561681c88fcdfb02446157b920
4
- data.tar.gz: 1eb0e85e3136bcb251a861cca28693394c90615e403cc02306650607ec9ca98b
3
+ metadata.gz: 9da0220850d940db558feefbf825cd3e1b5a73c73f0a2e35516c98a9bc86f4f8
4
+ data.tar.gz: 50f3c8cfda6747c98cf0335f6cf60705b828e34e44c5372ac9ff5c13f59caf96
5
5
  SHA512:
6
- metadata.gz: 3e52e7c93fcd20ea3ce687a0c74b0273de4367ede9addd14e5c59bbba477f769db1d1a8bd8bb84dc5e464dfa095fb7e9ce3214504535c4fbd903744e1b410348
7
- data.tar.gz: 2d7bbca865bb13f2a2b8dc3a9e62ae84a9740fd1a09413569a8b892a4453ebada71205d678e6824885145b4d2bfdd694423d0db9ec3360d32a14423fc1852716
6
+ metadata.gz: 64ee54f30a6cb6da3c5b2725eebe9f3a8b25e41ba51bcf8027d876d33080ffbd831371252ba05f39a3dced2fad6d94a9b35f918d4be060e25f3d690223005a06
7
+ data.tar.gz: f6c507d56f16056306e8f53688a5a509cb1425947b57dba7e143aa4b2496dcb438adc38be2a1b9bde4051ef8c522214f514149888bb93fcd38760f08db143968
data/README.md CHANGED
@@ -1,10 +1,12 @@
1
- # fluent-plugin-grafana-loki
1
+ # Fluentd output plugin
2
2
 
3
- [Fluentd](https://fluentd.org/) output plugin to ship logs to a Loki server. See [docs/client/fluentd/README.md](../../docs/clients/fluentd/README.md) for detailed information.
3
+ [Fluentd](https://fluentd.org/) is a data collector for unified logging layer, it can be configured with the Loki output plugin, provided in this folder, to ship logs to Loki.
4
+
5
+ See [docs/client/fluentd/README.md](../../docs/sources/clients/fluentd/_index.md) for detailed information.
4
6
 
5
7
  ## Development
6
8
 
7
- After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
9
+ After checking out the repo, run `bin/setup` to install dependencies. Then, run `bundle exec rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
8
10
 
9
11
  To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `fluent-plugin-grafana-loki.gemspec`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
10
12
 
@@ -13,6 +15,70 @@ To create the gem: `gem build fluent-plugin-grafana-loki.gemspec`
13
15
  Useful additions:
14
16
  `gem install rubocop`
15
17
 
18
+ ## Testing
19
+
20
+ Start Loki using:
21
+
22
+ ```bash
23
+ docker run -it -p 3100:3100 grafana/loki:latest
24
+ ```
25
+
26
+ Verify that Loki accept and stores logs:
27
+
28
+ ```bash
29
+ curl -H "Content-Type: application/json" -XPOST -s "http://localhost:3100/loki/api/v1/push" --data-raw "{\"streams\": [{\"stream\": {\"job\": \"test\"}, \"values\": [[\"$(date +%s)000000000\", \"fizzbuzz\"]]}]}"
30
+ curl "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query={job="test"}' --data-urlencode 'step=300' | jq .data.result
31
+ ```
32
+
33
+ The expected output is:
34
+
35
+ ```json
36
+ [
37
+ {
38
+ "stream": {
39
+ "job": "test"
40
+ },
41
+ "values": [
42
+ [
43
+ "1588337198000000000",
44
+ "fizzbuzz"
45
+ ]
46
+ ]
47
+ }
48
+ ]
49
+ ```
50
+
51
+ Start and send test logs with Fluentd using:
52
+
53
+ ```bash
54
+ LOKI_URL=http://{{ IP }}:3100 make fluentd-test
55
+ ```
56
+
57
+ Verify that syslogs are being feeded into Loki:
58
+
59
+ ```bash
60
+ curl "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query={job="fluentd"}' --data-urlencode 'step=300' | jq .data.result
61
+ ```
62
+
63
+ The expected output is:
64
+
65
+ ```json
66
+ [
67
+ {
68
+ "stream": {
69
+ "job": "fluentd"
70
+ },
71
+ "values": [
72
+ [
73
+ "1588336950379591919",
74
+ "log=\"May 1 14:42:30 ibuprofen avahi-daemon[859]: New relevant interface vethb503225.IPv6 for mDNS.\""
75
+ ],
76
+ ...
77
+ ]
78
+ }
79
+ ]
80
+ ```
81
+
16
82
  ## Copyright
17
83
 
18
84
  * Copyright(c) 2018- Grafana Labs
@@ -15,6 +15,7 @@
15
15
  # See the License for the specific language governing permissions and
16
16
  # limitations under the License.
17
17
 
18
+ require 'fluent/env'
18
19
  require 'fluent/plugin/output'
19
20
  require 'net/http'
20
21
  require 'yajl'
@@ -26,26 +27,34 @@ module Fluent
26
27
  class LokiOutput < Fluent::Plugin::Output # rubocop:disable Metrics/ClassLength
27
28
  Fluent::Plugin.register_output('loki', self)
28
29
 
30
+ class LogPostError < StandardError; end
31
+
29
32
  helpers :compat_parameters, :record_accessor
30
33
 
31
34
  attr_accessor :record_accessors
32
35
 
33
36
  DEFAULT_BUFFER_TYPE = 'memory'
34
37
 
35
- desc 'url of loki server'
36
- config_param :url, :string, default: 'https://logs-us-west1.grafana.net'
38
+ desc 'Loki API base URL'
39
+ config_param :url, :string, default: 'https://logs-prod-us-central1.grafana.net'
37
40
 
38
- desc 'BasicAuth credentials'
41
+ desc 'Authentication: basic auth credentials'
39
42
  config_param :username, :string, default: nil
40
43
  config_param :password, :string, default: nil, secret: true
41
44
 
42
- desc 'Client certificate'
45
+ desc 'Authentication: Authorization header with Bearer token scheme'
46
+ config_param :bearer_token_file, :string, default: nil
47
+
48
+ desc 'TLS: parameters for presenting a client certificate'
43
49
  config_param :cert, :string, default: nil
44
50
  config_param :key, :string, default: nil
45
51
 
46
- desc 'TLS'
52
+ desc 'TLS: CA certificate file for server certificate verification'
47
53
  config_param :ca_cert, :string, default: nil
48
54
 
55
+ desc 'TLS: disable server certificate verification'
56
+ config_param :insecure_tls, :bool, default: false
57
+
49
58
  desc 'Loki tenant id'
50
59
  config_param :tenant, :string, default: nil
51
60
 
@@ -74,7 +83,7 @@ module Fluent
74
83
  super
75
84
  @uri = URI.parse(@url + '/loki/api/v1/push')
76
85
  unless @uri.is_a?(URI::HTTP) || @uri.is_a?(URI::HTTPS)
77
- raise Fluent::ConfigError, 'url parameter must be valid HTTP'
86
+ raise Fluent::ConfigError, 'URL parameter must have HTTP/HTTPS scheme'
78
87
  end
79
88
 
80
89
  @record_accessors = {}
@@ -90,24 +99,42 @@ module Fluent
90
99
  @remove_keys_accessors.push(record_accessor_create(key))
91
100
  end
92
101
 
93
- if ssl_cert?
94
- load_ssl
95
- validate_ssl_key
102
+ # If configured, load and validate client certificate (and corresponding key)
103
+ if client_cert_configured?
104
+ load_client_cert
105
+ validate_client_cert_key
96
106
  end
97
107
 
108
+ raise "bearer_token_file #{@bearer_token_file} not found" if !@bearer_token_file.nil? && !File.exist?(@bearer_token_file)
109
+
110
+ @auth_token_bearer = nil
111
+ if !@bearer_token_file.nil?
112
+ if !File.exist?(@bearer_token_file)
113
+ raise "bearer_token_file #{@bearer_token_file} not found"
114
+ end
115
+
116
+ # Read the file once, assume long-lived authentication token.
117
+ @auth_token_bearer = File.read(@bearer_token_file)
118
+ if @auth_token_bearer.empty?
119
+ raise "bearer_token_file #{@bearer_token_file} is empty"
120
+ end
121
+ log.info "will use Bearer token from bearer_token_file #{@bearer_token_file} in Authorization header"
122
+ end
123
+
124
+
98
125
  raise "CA certificate file #{@ca_cert} not found" if !@ca_cert.nil? && !File.exist?(@ca_cert)
99
126
  end
100
127
 
101
- def ssl_cert?
128
+ def client_cert_configured?
102
129
  !@key.nil? && !@cert.nil?
103
130
  end
104
131
 
105
- def load_ssl
132
+ def load_client_cert
106
133
  @cert = OpenSSL::X509::Certificate.new(File.read(@cert)) if @cert
107
134
  @key = OpenSSL::PKey.read(File.read(@key)) if @key
108
135
  end
109
136
 
110
- def validate_ssl_key
137
+ def validate_client_cert_key
111
138
  if !@key.is_a?(OpenSSL::PKey::RSA) && !@key.is_a?(OpenSSL::PKey::DSA)
112
139
  raise "Unsupported private key type #{key.class}"
113
140
  end
@@ -117,58 +144,52 @@ module Fluent
117
144
  true
118
145
  end
119
146
 
120
- def http_opts(uri)
121
- opts = {
122
- use_ssl: uri.scheme == 'https'
123
- }
124
- opts
125
- end
126
-
127
147
  # flush a chunk to loki
128
148
  def write(chunk)
129
149
  # streams by label
130
150
  payload = generic_to_loki(chunk)
131
151
  body = { 'streams' => payload }
132
152
 
133
- # add ingest path to loki url
153
+ tenant = extract_placeholders(@tenant, chunk) if @tenant
134
154
 
135
- req = Net::HTTP::Post.new(
136
- @uri.request_uri
137
- )
138
- req.add_field('Content-Type', 'application/json')
139
- req.add_field('X-Scope-OrgID', @tenant) if @tenant
140
- req.body = Yajl.dump(body)
141
- req.basic_auth(@username, @password) if @username
155
+ # add ingest path to loki url
156
+ res = loki_http_request(body, tenant)
142
157
 
143
- opts = ssl_opts(@uri)
158
+ if res.is_a?(Net::HTTPSuccess)
159
+ log.debug "POST request was responded to with status code #{res.code}"
160
+ return
161
+ end
144
162
 
145
- log.debug "sending #{req.body.length} bytes to loki"
146
- res = Net::HTTP.start(@uri.host, @uri.port, **opts) { |http| http.request(req) }
147
- unless res&.is_a?(Net::HTTPSuccess)
148
- res_summary = if res
149
- "#{res.code} #{res.message} #{res.body}"
150
- else
151
- 'res=nil'
152
- end
153
- log.warn "failed to #{req.method} #{@uri} (#{res_summary})"
154
- log.warn Yajl.dump(body)
163
+ res_summary = "#{res.code} #{res.message} #{res.body}"
164
+ log.warn "failed to write post to #{@uri} (#{res_summary})"
165
+ log.debug Yajl.dump(body)
155
166
 
156
- end
167
+ # Only retry 429 and 500s
168
+ raise(LogPostError, res_summary) if res.is_a?(Net::HTTPTooManyRequests) || res.is_a?(Net::HTTPServerError)
157
169
  end
158
170
 
159
- def ssl_opts(uri)
171
+ def http_request_opts(uri)
160
172
  opts = {
161
173
  use_ssl: uri.scheme == 'https'
162
174
  }
163
175
 
176
+ # Optionally disable server server certificate verification.
177
+ if @insecure_tls
178
+ opts = opts.merge(
179
+ verify_mode: OpenSSL::SSL::VERIFY_NONE
180
+ )
181
+ end
182
+
183
+ # Optionally present client certificate
164
184
  if !@cert.nil? && !@key.nil?
165
185
  opts = opts.merge(
166
- verify_mode: OpenSSL::SSL::VERIFY_PEER,
167
186
  cert: @cert,
168
187
  key: @key
169
188
  )
170
189
  end
171
190
 
191
+ # For server certificate verification: set custom CA bundle.
192
+ # Only takes effect when `insecure_tls` is not set.
172
193
  unless @ca_cert.nil?
173
194
  opts = opts.merge(
174
195
  ca_file: @ca_cert
@@ -186,6 +207,25 @@ module Fluent
186
207
 
187
208
  private
188
209
 
210
+ def loki_http_request(body, tenant)
211
+ req = Net::HTTP::Post.new(
212
+ @uri.request_uri
213
+ )
214
+ req.add_field('Content-Type', 'application/json')
215
+ req.add_field('Authorization', "Bearer #{@auth_token_bearer}") if !@auth_token_bearer.nil?
216
+ req.add_field('X-Scope-OrgID', tenant) if tenant
217
+ req.body = Yajl.dump(body)
218
+ req.basic_auth(@username, @password) if @username
219
+
220
+ opts = http_request_opts(@uri)
221
+
222
+ msg = "sending #{req.body.length} bytes to loki"
223
+ msg += " (tenant: \"#{tenant}\")" if tenant
224
+ log.debug msg
225
+
226
+ Net::HTTP.start(@uri.host, @uri.port, **opts) { |http| http.request(req) }
227
+ end
228
+
189
229
  def numeric?(val)
190
230
  !Float(val).nil?
191
231
  rescue StandardError
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-grafana-loki
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.2.9
4
+ version: 1.2.15
5
5
  platform: ruby
6
6
  authors:
7
7
  - woodsaj
@@ -10,8 +10,28 @@ authors:
10
10
  autorequire:
11
11
  bindir: bin
12
12
  cert_chain: []
13
- date: 2020-02-18 00:00:00.000000000 Z
13
+ date: 2020-10-16 00:00:00.000000000 Z
14
14
  dependencies:
15
+ - !ruby/object:Gem::Dependency
16
+ name: fluentd
17
+ requirement: !ruby/object:Gem::Requirement
18
+ requirements:
19
+ - - ">="
20
+ - !ruby/object:Gem::Version
21
+ version: 1.9.3
22
+ - - "<"
23
+ - !ruby/object:Gem::Version
24
+ version: '2'
25
+ type: :runtime
26
+ prerelease: false
27
+ version_requirements: !ruby/object:Gem::Requirement
28
+ requirements:
29
+ - - ">="
30
+ - !ruby/object:Gem::Version
31
+ version: 1.9.3
32
+ - - "<"
33
+ - !ruby/object:Gem::Version
34
+ version: '2'
15
35
  - !ruby/object:Gem::Dependency
16
36
  name: bundler
17
37
  requirement: !ruby/object:Gem::Requirement
@@ -55,25 +75,47 @@ dependencies:
55
75
  - !ruby/object:Gem::Version
56
76
  version: '3.0'
57
77
  - !ruby/object:Gem::Dependency
58
- name: fluentd
78
+ name: rubocop-rspec
59
79
  requirement: !ruby/object:Gem::Requirement
60
80
  requirements:
61
81
  - - ">="
62
82
  - !ruby/object:Gem::Version
63
- version: 1.9.0
64
- - - "<"
83
+ version: '0'
84
+ type: :development
85
+ prerelease: false
86
+ version_requirements: !ruby/object:Gem::Requirement
87
+ requirements:
88
+ - - ">="
65
89
  - !ruby/object:Gem::Version
66
- version: '2'
67
- type: :runtime
90
+ version: '0'
91
+ - !ruby/object:Gem::Dependency
92
+ name: simplecov
93
+ requirement: !ruby/object:Gem::Requirement
94
+ requirements:
95
+ - - ">="
96
+ - !ruby/object:Gem::Version
97
+ version: '0'
98
+ type: :development
68
99
  prerelease: false
69
100
  version_requirements: !ruby/object:Gem::Requirement
70
101
  requirements:
71
102
  - - ">="
72
103
  - !ruby/object:Gem::Version
73
- version: 1.9.0
74
- - - "<"
104
+ version: '0'
105
+ - !ruby/object:Gem::Dependency
106
+ name: test-unit
107
+ requirement: !ruby/object:Gem::Requirement
108
+ requirements:
109
+ - - ">="
75
110
  - !ruby/object:Gem::Version
76
- version: '2'
111
+ version: '0'
112
+ type: :development
113
+ prerelease: false
114
+ version_requirements: !ruby/object:Gem::Requirement
115
+ requirements:
116
+ - - ">="
117
+ - !ruby/object:Gem::Version
118
+ version: '0'
77
119
  description: Output plugin to ship logs to a Grafana Loki server
78
120
  email:
79
121
  - awoods@grafana.com