phihos-fluent-plugin-prometheus 2.0.3
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/.github/workflows/linux.yml +34 -0
- data/.gitignore +16 -0
- data/.rspec +2 -0
- data/.travis.yml +14 -0
- data/ChangeLog +43 -0
- data/Gemfile +4 -0
- data/LICENSE +202 -0
- data/README.md +537 -0
- data/Rakefile +7 -0
- data/fluent-plugin-prometheus.gemspec +22 -0
- data/lib/fluent/plugin/filter_prometheus.rb +43 -0
- data/lib/fluent/plugin/in_prometheus/async_wrapper.rb +47 -0
- data/lib/fluent/plugin/in_prometheus.rb +230 -0
- data/lib/fluent/plugin/in_prometheus_monitor.rb +107 -0
- data/lib/fluent/plugin/in_prometheus_output_monitor.rb +234 -0
- data/lib/fluent/plugin/in_prometheus_tail_monitor.rb +98 -0
- data/lib/fluent/plugin/out_prometheus.rb +42 -0
- data/lib/fluent/plugin/prometheus/data_store.rb +93 -0
- data/lib/fluent/plugin/prometheus/placeholder_expander.rb +132 -0
- data/lib/fluent/plugin/prometheus.rb +418 -0
- data/lib/fluent/plugin/prometheus_metrics.rb +77 -0
- data/misc/fluentd_sample.conf +170 -0
- data/misc/nginx_proxy.conf +22 -0
- data/misc/prometheus.yaml +13 -0
- data/misc/prometheus_alerts.yaml +59 -0
- data/spec/fluent/plugin/filter_prometheus_spec.rb +118 -0
- data/spec/fluent/plugin/in_prometheus_monitor_spec.rb +42 -0
- data/spec/fluent/plugin/in_prometheus_spec.rb +225 -0
- data/spec/fluent/plugin/in_prometheus_tail_monitor_spec.rb +42 -0
- data/spec/fluent/plugin/out_prometheus_spec.rb +139 -0
- data/spec/fluent/plugin/prometheus/placeholder_expander_spec.rb +110 -0
- data/spec/fluent/plugin/prometheus_metrics_spec.rb +138 -0
- data/spec/fluent/plugin/shared.rb +248 -0
- data/spec/spec_helper.rb +10 -0
- metadata +176 -0
data/README.md
ADDED
@@ -0,0 +1,537 @@
|
|
1
|
+
# fluent-plugin-prometheus, a plugin for [Fluentd](https://www.fluentd.org)
|
2
|
+
|
3
|
+
[![Build Status](https://travis-ci.org/fluent/fluent-plugin-prometheus.svg?branch=master)](https://travis-ci.org/fluent/fluent-plugin-prometheus)
|
4
|
+
|
5
|
+
A fluent plugin that instruments metrics from records and exposes them via web interface. Intended to be used together with a [Prometheus server](https://github.com/prometheus/prometheus).
|
6
|
+
|
7
|
+
## Requirements
|
8
|
+
|
9
|
+
| fluent-plugin-prometheus | fluentd | ruby |
|
10
|
+
|--------------------------|------------|--------|
|
11
|
+
| 1.x.y | >= v1.9.1 | >= 2.4 |
|
12
|
+
| 1.[0-7].y | >= v0.14.8 | >= 2.1 |
|
13
|
+
| 0.x.y | >= v0.12.0 | >= 1.9 |
|
14
|
+
|
15
|
+
Since v1.8.0, fluent-plugin-prometheus uses [http_server helper](https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-http_server) to launch HTTP server.
|
16
|
+
If you want to handle lots of connections, install `async-http` gem.
|
17
|
+
|
18
|
+
## Installation
|
19
|
+
|
20
|
+
Add this line to your application's Gemfile:
|
21
|
+
|
22
|
+
```ruby
|
23
|
+
gem 'fluent-plugin-prometheus'
|
24
|
+
```
|
25
|
+
|
26
|
+
And then execute:
|
27
|
+
|
28
|
+
$ bundle
|
29
|
+
|
30
|
+
Or install it yourself as:
|
31
|
+
|
32
|
+
$ gem install fluent-plugin-prometheus
|
33
|
+
|
34
|
+
## Usage
|
35
|
+
|
36
|
+
fluentd-plugin-prometheus includes 6 plugins.
|
37
|
+
|
38
|
+
- `prometheus` input plugin
|
39
|
+
- `prometheus_monitor` input plugin
|
40
|
+
- `prometheus_output_monitor` input plugin
|
41
|
+
- `prometheus_tail_monitor` input plugin
|
42
|
+
- `prometheus` output plugin
|
43
|
+
- `prometheus` filter plugin
|
44
|
+
|
45
|
+
See [sample configuration](./misc/fluentd_sample.conf), or try [tutorial](#try-plugin-with-nginx).
|
46
|
+
|
47
|
+
### prometheus input plugin
|
48
|
+
|
49
|
+
You have to configure this plugin to expose metrics collected by other Prometheus plugins.
|
50
|
+
This plugin provides a metrics HTTP endpoint to be scraped by a Prometheus server on 24231/tcp(default).
|
51
|
+
|
52
|
+
With following configuration, you can access http://localhost:24231/metrics on a server where fluentd running.
|
53
|
+
|
54
|
+
```
|
55
|
+
<source>
|
56
|
+
@type prometheus
|
57
|
+
</source>
|
58
|
+
```
|
59
|
+
|
60
|
+
More configuration parameters:
|
61
|
+
|
62
|
+
- `bind`: binding interface (default: '0.0.0.0')
|
63
|
+
- `port`: listen port (default: 24231)
|
64
|
+
- `metrics_path`: metrics HTTP endpoint (default: /metrics)
|
65
|
+
- `aggregated_metrics_path`: metrics HTTP endpoint (default: /aggregated_metrics)
|
66
|
+
|
67
|
+
When using multiple workers, each worker binds to port + `fluent_worker_id`.
|
68
|
+
To scrape metrics from all workers at once, you can access http://localhost:24231/aggregated_metrics.
|
69
|
+
|
70
|
+
#### TLS setting
|
71
|
+
|
72
|
+
Use `<trasnport tls>`. See [transport config article](https://docs.fluentd.org/configuration/transport-section) for more details.
|
73
|
+
|
74
|
+
```
|
75
|
+
<source>
|
76
|
+
@type prometheus
|
77
|
+
<transport tls>
|
78
|
+
# TLS parameters...
|
79
|
+
</transport
|
80
|
+
</source>
|
81
|
+
```
|
82
|
+
|
83
|
+
### prometheus_monitor input plugin
|
84
|
+
|
85
|
+
This plugin collects internal metrics in Fluentd. The metrics are similar to/part of [monitor_agent](https://docs.fluentd.org/input/monitor_agent).
|
86
|
+
|
87
|
+
|
88
|
+
#### Exposed metrics
|
89
|
+
|
90
|
+
- `fluentd_status_buffer_queue_length`
|
91
|
+
- `fluentd_status_buffer_total_bytes`
|
92
|
+
- `fluentd_status_retry_count`
|
93
|
+
- `fluentd_status_buffer_newest_timekey` from fluentd v1.4.2
|
94
|
+
- `fluentd_status_buffer_oldest_timekey` from fluentd v1.4.2
|
95
|
+
|
96
|
+
#### Configuration
|
97
|
+
|
98
|
+
With following configuration, those metrics are collected.
|
99
|
+
|
100
|
+
```
|
101
|
+
<source>
|
102
|
+
@type prometheus_monitor
|
103
|
+
</source>
|
104
|
+
```
|
105
|
+
|
106
|
+
More configuration parameters:
|
107
|
+
|
108
|
+
- `<labels>`: additional labels for this metric (optional). See [Labels](#labels)
|
109
|
+
- `interval`: interval to update monitor_agent information in seconds (default: 5)
|
110
|
+
|
111
|
+
### prometheus_output_monitor input plugin
|
112
|
+
|
113
|
+
This plugin collects internal metrics for output plugin in Fluentd. This is similar to `prometheus_monitor` plugin, but specialized for output plugin. There are Many metrics `prometheus_monitor` does not include, such as `num_errors`, `retry_wait` and so on.
|
114
|
+
|
115
|
+
#### Exposed metrics
|
116
|
+
|
117
|
+
Metrics for output
|
118
|
+
|
119
|
+
- `fluentd_output_status_retry_count`
|
120
|
+
- `fluentd_output_status_num_errors`
|
121
|
+
- `fluentd_output_status_emit_count`
|
122
|
+
- `fluentd_output_status_retry_wait`
|
123
|
+
- current retry_wait computed from last retry time and next retry time
|
124
|
+
- `fluentd_output_status_emit_records`
|
125
|
+
- `fluentd_output_status_write_count`
|
126
|
+
- `fluentd_output_status_rollback_count`
|
127
|
+
- `fluentd_output_status_flush_time_count` in milliseconds from fluentd v1.6.0
|
128
|
+
- `fluentd_output_status_slow_flush_count` from fluentd v1.6.0
|
129
|
+
|
130
|
+
Metrics for buffer
|
131
|
+
|
132
|
+
- `fluentd_output_status_buffer_total_bytes`
|
133
|
+
- `fluentd_output_status_buffer_stage_length` from fluentd v1.6.0
|
134
|
+
- `fluentd_output_status_buffer_stage_byte_size` from fluentd v1.6.0
|
135
|
+
- `fluentd_output_status_buffer_queue_length`
|
136
|
+
- `fluentd_output_status_buffer_queue_byte_size` from fluentd v1.6.0
|
137
|
+
- `fluentd_output_status_buffer_newest_timekey` from fluentd v1.6.0
|
138
|
+
- `fluentd_output_status_buffer_oldest_timekey` from fluentd v1.6.0
|
139
|
+
- `fluentd_output_status_buffer_available_space_ratio` from fluentd v1.6.0
|
140
|
+
|
141
|
+
#### Configuration
|
142
|
+
|
143
|
+
With following configuration, those metrics are collected.
|
144
|
+
|
145
|
+
```
|
146
|
+
<source>
|
147
|
+
@type prometheus_output_monitor
|
148
|
+
</source>
|
149
|
+
```
|
150
|
+
|
151
|
+
More configuration parameters:
|
152
|
+
|
153
|
+
- `<labels>`: additional labels for this metric (optional). See [Labels](#labels)
|
154
|
+
- `interval`: interval to update monitor_agent information in seconds (default: 5)
|
155
|
+
- `gauge_all`: Specify metric type. If `true`, use `gauge` type. If `false`, use `counter` type. Since v2, this parameter will be removed and use `counter` type.
|
156
|
+
|
157
|
+
### prometheus_tail_monitor input plugin
|
158
|
+
|
159
|
+
This plugin collects internal metrics for in_tail plugin in Fluentd. in_tail plugin holds internal state for files that the plugin is watching. The state is sometimes important to monitor plugins work correctly.
|
160
|
+
|
161
|
+
This plugin uses internal class of Fluentd, so it's easy to break.
|
162
|
+
|
163
|
+
#### Exposed metrics
|
164
|
+
|
165
|
+
- `fluentd_tail_file_position`
|
166
|
+
- Current bytes which plugin reads from the file
|
167
|
+
- `fluentd_tail_file_inode`
|
168
|
+
- inode of the file
|
169
|
+
|
170
|
+
Default labels:
|
171
|
+
|
172
|
+
- `plugin_id`: a value set for a plugin in configuration.
|
173
|
+
- `type`: plugin name. `in_tail` only for now.
|
174
|
+
- `path`: file path
|
175
|
+
|
176
|
+
#### Configuration
|
177
|
+
|
178
|
+
With following configuration, those metrics are collected.
|
179
|
+
|
180
|
+
```
|
181
|
+
<source>
|
182
|
+
@type prometheus_tail_monitor
|
183
|
+
</source>
|
184
|
+
```
|
185
|
+
|
186
|
+
More configuration parameters:
|
187
|
+
|
188
|
+
- `<labels>`: additional labels for this metric (optional). See [Labels](#labels)
|
189
|
+
- `interval`: interval to update monitor_agent information in seconds (default: 5)
|
190
|
+
|
191
|
+
### prometheus output/filter plugin
|
192
|
+
|
193
|
+
Both output/filter plugins instrument metrics from records. Both plugins have no impact against values of each records, just read.
|
194
|
+
|
195
|
+
Assuming you have following configuration and receiving message,
|
196
|
+
|
197
|
+
```
|
198
|
+
<match message>
|
199
|
+
@type stdout
|
200
|
+
</match>
|
201
|
+
```
|
202
|
+
|
203
|
+
```
|
204
|
+
message {
|
205
|
+
"foo": 100,
|
206
|
+
"bar": 200,
|
207
|
+
"baz": 300
|
208
|
+
}
|
209
|
+
```
|
210
|
+
|
211
|
+
In filter plugin style,
|
212
|
+
|
213
|
+
```
|
214
|
+
<filter message>
|
215
|
+
@type prometheus
|
216
|
+
<metric>
|
217
|
+
name message_foo_counter
|
218
|
+
type counter
|
219
|
+
desc The total number of foo in message.
|
220
|
+
key foo
|
221
|
+
</metric>
|
222
|
+
</filter>
|
223
|
+
|
224
|
+
<match message>
|
225
|
+
@type stdout
|
226
|
+
</match>
|
227
|
+
```
|
228
|
+
|
229
|
+
In output plugin style:
|
230
|
+
|
231
|
+
```
|
232
|
+
<filter message>
|
233
|
+
@type prometheus
|
234
|
+
<metric>
|
235
|
+
name message_foo_counter
|
236
|
+
type counter
|
237
|
+
desc The total number of foo in message.
|
238
|
+
key foo
|
239
|
+
</metric>
|
240
|
+
</filter>
|
241
|
+
|
242
|
+
<match message>
|
243
|
+
@type copy
|
244
|
+
<store>
|
245
|
+
@type prometheus
|
246
|
+
<metric>
|
247
|
+
name message_foo_counter
|
248
|
+
type counter
|
249
|
+
desc The total number of foo in message.
|
250
|
+
key foo
|
251
|
+
</metric>
|
252
|
+
</store>
|
253
|
+
<store>
|
254
|
+
@type stdout
|
255
|
+
</store>
|
256
|
+
</match>
|
257
|
+
```
|
258
|
+
|
259
|
+
With above configuration, the plugin collects a metric named `message_foo_counter` from key `foo` of each records.
|
260
|
+
|
261
|
+
You can access nested keys in records via dot or bracket notation (https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-record_accessor#syntax), for example: `$.kubernetes.namespace`, `$['key1'][0]['key2']`. The record accessor is enable only if the value starts with `$.` or `$[`.
|
262
|
+
|
263
|
+
See Supported Metric Type and Labels for more configuration parameters.
|
264
|
+
|
265
|
+
## Supported Metric Types
|
266
|
+
|
267
|
+
For details of each metric type, see [Prometheus documentation](http://prometheus.io/docs/concepts/metric_types/). Also see [metric name guide](http://prometheus.io/docs/practices/naming/).
|
268
|
+
|
269
|
+
### counter type
|
270
|
+
|
271
|
+
```
|
272
|
+
<metric>
|
273
|
+
name message_foo_counter
|
274
|
+
type counter
|
275
|
+
desc The total number of foo in message.
|
276
|
+
key foo
|
277
|
+
<labels>
|
278
|
+
tag ${tag}
|
279
|
+
host ${hostname}
|
280
|
+
foo bar
|
281
|
+
</labels>
|
282
|
+
</metric>
|
283
|
+
```
|
284
|
+
|
285
|
+
- `name`: metric name (required)
|
286
|
+
- `type`: metric type (required)
|
287
|
+
- `desc`: description of this metric (required)
|
288
|
+
- `key`: key name of record for instrumentation (**optional**)
|
289
|
+
- `retention`: time in seconds to remove a metric after not being updated (optional). See [Retention](#retention)
|
290
|
+
- `retention_check_interval`: time in seconds to check for expired metrics (optional). Has no effect when `retention` not set. See [Retention](#retention)
|
291
|
+
- `<labels>`: additional labels for this metric (optional). See [Labels](#labels)
|
292
|
+
|
293
|
+
If key is empty, the metric values is treated as 1, so the counter increments by 1 on each record regardless of contents of the record.
|
294
|
+
|
295
|
+
### gauge type
|
296
|
+
|
297
|
+
```
|
298
|
+
<metric>
|
299
|
+
name message_foo_gauge
|
300
|
+
type gauge
|
301
|
+
desc The total number of foo in message.
|
302
|
+
key foo
|
303
|
+
<labels>
|
304
|
+
tag ${tag}
|
305
|
+
host ${hostname}
|
306
|
+
foo bar
|
307
|
+
</labels>
|
308
|
+
</metric>
|
309
|
+
```
|
310
|
+
|
311
|
+
- `name`: metric name (required)
|
312
|
+
- `type`: metric type (required)
|
313
|
+
- `desc`: description of metric (required)
|
314
|
+
- `key`: key name of record for instrumentation (required)
|
315
|
+
- `retention`: time in seconds to remove a metric after not being updated (optional). See [Retention](#retention)
|
316
|
+
- `retention_check_interval`: time in seconds to check for expired metrics (optional). Has no effect when `retention` not set. See [Retention](#retention)
|
317
|
+
- `<labels>`: additional labels for this metric (optional). See [Labels](#labels)
|
318
|
+
|
319
|
+
### summary type
|
320
|
+
|
321
|
+
```
|
322
|
+
<metric>
|
323
|
+
name message_foo
|
324
|
+
type summary
|
325
|
+
desc The summary of foo in message.
|
326
|
+
key foo
|
327
|
+
<labels>
|
328
|
+
tag ${tag}
|
329
|
+
host ${hostname}
|
330
|
+
foo bar
|
331
|
+
</labels>
|
332
|
+
</metric>
|
333
|
+
```
|
334
|
+
|
335
|
+
- `name`: metric name (required)
|
336
|
+
- `type`: metric type (required)
|
337
|
+
- `desc`: description of metric (required)
|
338
|
+
- `key`: key name of record for instrumentation (required)
|
339
|
+
- `retention`: time in seconds to remove a metric after not being updated (optional). See [Retention](#retention)
|
340
|
+
- `retention_check_interval`: time in seconds to check for expired metrics (optional). Has no effect when `retention` not set. See [Retention](#retention)
|
341
|
+
- `<labels>`: additional labels for this metric (optional). See [Labels](#labels)
|
342
|
+
|
343
|
+
### histogram type
|
344
|
+
|
345
|
+
```
|
346
|
+
<metric>
|
347
|
+
name message_foo
|
348
|
+
type histogram
|
349
|
+
desc The histogram of foo in message.
|
350
|
+
key foo
|
351
|
+
buckets 0.1, 1, 5, 10
|
352
|
+
<labels>
|
353
|
+
tag ${tag}
|
354
|
+
host ${hostname}
|
355
|
+
foo bar
|
356
|
+
</labels>
|
357
|
+
</metric>
|
358
|
+
```
|
359
|
+
|
360
|
+
- `name`: metric name (required)
|
361
|
+
- `type`: metric type (required)
|
362
|
+
- `desc`: description of metric (required)
|
363
|
+
- `key`: key name of record for instrumentation (required)
|
364
|
+
- `buckets`: buckets of record for instrumentation (optional)
|
365
|
+
- `retention`: time in seconds to remove a metric after not being updated (optional). See [Retention](#retention)
|
366
|
+
- `retention_check_interval`: time in seconds to check for expired metrics (optional). Has no effect when `retention` not set. See [Retention](#retention)
|
367
|
+
- `<labels>`: additional labels for this metric (optional). See [Labels](#labels)
|
368
|
+
|
369
|
+
## Labels
|
370
|
+
|
371
|
+
See [Prometheus Data Model](http://prometheus.io/docs/concepts/data_model/) first.
|
372
|
+
|
373
|
+
You can add labels with static value or dynamic value from records. In `prometheus_monitor` input plugin, you can't use label value from records.
|
374
|
+
|
375
|
+
### labels section
|
376
|
+
|
377
|
+
```
|
378
|
+
<labels>
|
379
|
+
key1 value1
|
380
|
+
key2 value2
|
381
|
+
</labels>
|
382
|
+
```
|
383
|
+
|
384
|
+
All labels sections has same format. Each lines have key/value for label.
|
385
|
+
|
386
|
+
You can access nested fields in records via dot or bracket notation (https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-record_accessor#syntax), for example: `$.kubernetes.namespace`, `$['key1'][0]['key2']`. The record accessor is enable only if the value starts with `$.` or `$[`. Other values are handled as raw string as is and may be expanded by placeholder described later.
|
387
|
+
|
388
|
+
You can use placeholder for label values. The placeholders will be expanded from reserved values and records.
|
389
|
+
If you specify `${hostname}`, it will be expanded by value of a hostname where fluentd runs.
|
390
|
+
The placeholder for records is deprecated. Use record accessor syntax instead.
|
391
|
+
|
392
|
+
Reserved placeholders are:
|
393
|
+
|
394
|
+
- `${hostname}`: hostname
|
395
|
+
- `${worker_id}`: fluent worker id
|
396
|
+
- `${tag}`: tag name
|
397
|
+
- only available in Prometheus output/filter plugin
|
398
|
+
- `${tag_parts[N]}` refers to the Nth part of the tag.
|
399
|
+
- only available in Prometheus output/filter plugin
|
400
|
+
- `${tag_prefix[N]}` refers to the [0..N] part of the tag.
|
401
|
+
- only available in Prometheus output/filter plugin
|
402
|
+
- `${tag_suffix[N]}` refers to the [`tagsize`-1-N..] part of the tag.
|
403
|
+
- where `tagsize` is the size of tag which is splitted with `.` (when tag is `1.2.3`, then `tagsize` is 3)
|
404
|
+
- only available in Prometheus output/filter plugin
|
405
|
+
|
406
|
+
### top-level labels and labels inside metric
|
407
|
+
|
408
|
+
Prometheus output/filter plugin can have multiple metric section. Top-level labels section specifies labels for all metrics. Labels section inside metric section specifies labels for the metric. Both are specified, labels are merged.
|
409
|
+
|
410
|
+
```
|
411
|
+
<filter message>
|
412
|
+
@type prometheus
|
413
|
+
<metric>
|
414
|
+
name message_foo_counter
|
415
|
+
type counter
|
416
|
+
desc The total number of foo in message.
|
417
|
+
key foo
|
418
|
+
<labels>
|
419
|
+
key foo
|
420
|
+
data_type ${type}
|
421
|
+
</labels>
|
422
|
+
</metric>
|
423
|
+
<metric>
|
424
|
+
name message_bar_counter
|
425
|
+
type counter
|
426
|
+
desc The total number of bar in message.
|
427
|
+
key bar
|
428
|
+
<labels>
|
429
|
+
key bar
|
430
|
+
</labels>
|
431
|
+
</metric>
|
432
|
+
<labels>
|
433
|
+
tag ${tag}
|
434
|
+
hostname ${hostname}
|
435
|
+
</labels>
|
436
|
+
</filter>
|
437
|
+
```
|
438
|
+
|
439
|
+
In this case, `message_foo_counter` has `tag`, `hostname`, `key` and `data_type` labels.
|
440
|
+
|
441
|
+
## Retention
|
442
|
+
|
443
|
+
By default metrics with all encountered label combinations are preserved until the next restart of fluentd.
|
444
|
+
Even if a label combination did not receive any update for a long time.
|
445
|
+
That behavior is not always desirable e.g. when the contents of of fields change for good and the metric becomes idle.
|
446
|
+
For these metrics you can set `retention` and `retention_check_interval` like this:
|
447
|
+
|
448
|
+
```
|
449
|
+
<metric>
|
450
|
+
name message_foo_counter
|
451
|
+
type counter
|
452
|
+
desc The total number of foo in message.
|
453
|
+
key foo
|
454
|
+
retention 3600 # 1h
|
455
|
+
retention_check_interval 1800 # 30m
|
456
|
+
<labels>
|
457
|
+
bar ${bar}
|
458
|
+
</labels>
|
459
|
+
</metric>
|
460
|
+
```
|
461
|
+
|
462
|
+
If `${bar}` was `baz` one time but after that no records with that value were processed, then after one hour the metric
|
463
|
+
`foo{bar="baz"}` might be removed.
|
464
|
+
When this actually happens depends on `retention_check_interval` (default 60).
|
465
|
+
It causes a background thread to check every 30 minutes for expired metrics.
|
466
|
+
So worst case the metrics are removed 30 minutes after expiration.
|
467
|
+
You can set this value as low as `1`, but that may put more stress on your CPU.
|
468
|
+
|
469
|
+
## Try plugin with nginx
|
470
|
+
|
471
|
+
Checkout repository and setup.
|
472
|
+
|
473
|
+
```
|
474
|
+
$ git clone git://github.com/fluent/fluent-plugin-prometheus
|
475
|
+
$ cd fluent-plugin-prometheus
|
476
|
+
$ bundle install --path vendor/bundle
|
477
|
+
```
|
478
|
+
|
479
|
+
Download pre-compiled Prometheus binary and start it. It listens on 9090.
|
480
|
+
|
481
|
+
```
|
482
|
+
$ wget https://github.com/prometheus/prometheus/releases/download/v1.5.2/prometheus-1.5.2.linux-amd64.tar.gz -O - | tar zxf -
|
483
|
+
$ ./prometheus-1.5.2.linux-amd64/prometheus -config.file=./misc/prometheus.yaml -storage.local.path=./prometheus/metrics
|
484
|
+
```
|
485
|
+
|
486
|
+
Install Nginx for sample metrics. It listens on 80 and 9999.
|
487
|
+
|
488
|
+
```
|
489
|
+
$ sudo apt-get install -y nginx
|
490
|
+
$ sudo cp misc/nginx_proxy.conf /etc/nginx/sites-enabled/proxy
|
491
|
+
$ sudo chmod 777 /var/log/nginx && sudo chmod +r /var/log/nginx/*.log
|
492
|
+
$ sudo service nginx restart
|
493
|
+
```
|
494
|
+
|
495
|
+
Start fluentd with sample configuration. It listens on 24231.
|
496
|
+
|
497
|
+
```
|
498
|
+
$ bundle exec fluentd -c misc/fluentd_sample.conf -v
|
499
|
+
```
|
500
|
+
|
501
|
+
Generate some records by accessing nginx.
|
502
|
+
|
503
|
+
```
|
504
|
+
$ curl http://localhost/
|
505
|
+
$ curl http://localhost:9999/
|
506
|
+
```
|
507
|
+
|
508
|
+
Confirm that some metrics are exported via Fluentd.
|
509
|
+
|
510
|
+
```
|
511
|
+
$ curl http://localhost:24231/metrics
|
512
|
+
```
|
513
|
+
|
514
|
+
Then, make a graph on Prometheus UI. http://localhost:9090/
|
515
|
+
|
516
|
+
## Contributing
|
517
|
+
|
518
|
+
1. Fork it ( https://github.com/fluent/fluent-plugin-prometheus/fork )
|
519
|
+
2. Create your feature branch (`git checkout -b my-new-feature`)
|
520
|
+
3. Commit your changes (`git commit -am 'Add some feature'`)
|
521
|
+
4. Push to the branch (`git push origin my-new-feature`)
|
522
|
+
5. Create a new Pull Request
|
523
|
+
|
524
|
+
|
525
|
+
## Copyright
|
526
|
+
|
527
|
+
<table>
|
528
|
+
<tr>
|
529
|
+
<td>Author</td><td>Masahiro Sano <sabottenda@gmail.com></td>
|
530
|
+
</tr>
|
531
|
+
<tr>
|
532
|
+
<td>Copyright</td><td>Copyright (c) 2015- Masahiro Sano</td>
|
533
|
+
</tr>
|
534
|
+
<tr>
|
535
|
+
<td>License</td><td>Apache License, Version 2.0</td>
|
536
|
+
</tr>
|
537
|
+
</table>
|
data/Rakefile
ADDED
@@ -0,0 +1,22 @@
|
|
1
|
+
Gem::Specification.new do |spec|
|
2
|
+
spec.name = "phihos-fluent-plugin-prometheus"
|
3
|
+
spec.version = "2.0.3"
|
4
|
+
spec.authors = ["Philipp Hossner"]
|
5
|
+
spec.email = ["philipp.hossner@posteo.de"]
|
6
|
+
spec.summary = %q{A fluent plugin that collects metrics and exposes for Prometheus.}
|
7
|
+
spec.description = %q{A fluent plugin that collects metrics and exposes for Prometheus.}
|
8
|
+
spec.homepage = "https://github.com/phihos/fluent-plugin-prometheus"
|
9
|
+
spec.license = "Apache-2.0"
|
10
|
+
|
11
|
+
spec.files = `git ls-files -z`.split("\x0")
|
12
|
+
spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
|
13
|
+
spec.test_files = spec.files.grep(%r{^(test|spec|features)/})
|
14
|
+
spec.require_paths = ["lib"]
|
15
|
+
|
16
|
+
spec.add_dependency "fluentd", ">= 1.9.1", "< 2"
|
17
|
+
spec.add_dependency "prometheus-client", ">= 2.1.0"
|
18
|
+
spec.add_development_dependency "bundler"
|
19
|
+
spec.add_development_dependency "rake"
|
20
|
+
spec.add_development_dependency "rspec"
|
21
|
+
spec.add_development_dependency "test-unit"
|
22
|
+
end
|
@@ -0,0 +1,43 @@
|
|
1
|
+
require 'fluent/plugin/prometheus'
|
2
|
+
require 'fluent/plugin/filter'
|
3
|
+
|
4
|
+
module Fluent::Plugin
|
5
|
+
class PrometheusFilter < Fluent::Plugin::Filter
|
6
|
+
Fluent::Plugin.register_filter('prometheus', self)
|
7
|
+
include Fluent::Plugin::PrometheusLabelParser
|
8
|
+
include Fluent::Plugin::Prometheus
|
9
|
+
|
10
|
+
helpers :thread
|
11
|
+
|
12
|
+
def initialize
|
13
|
+
super
|
14
|
+
@registry = ::Prometheus::Client.registry
|
15
|
+
end
|
16
|
+
|
17
|
+
def multi_workers_ready?
|
18
|
+
true
|
19
|
+
end
|
20
|
+
|
21
|
+
def configure(conf)
|
22
|
+
super
|
23
|
+
labels = parse_labels_elements(conf)
|
24
|
+
@metrics = Fluent::Plugin::Prometheus.parse_metrics_elements(conf, @registry, labels)
|
25
|
+
end
|
26
|
+
|
27
|
+
def start
|
28
|
+
super
|
29
|
+
Fluent::Plugin::Prometheus.start_retention_threads(
|
30
|
+
@metrics,
|
31
|
+
@registry,
|
32
|
+
method(:thread_create),
|
33
|
+
method(:thread_current_running?),
|
34
|
+
@log
|
35
|
+
)
|
36
|
+
end
|
37
|
+
|
38
|
+
def filter(tag, time, record)
|
39
|
+
instrument_single(tag, time, record, @metrics)
|
40
|
+
record
|
41
|
+
end
|
42
|
+
end
|
43
|
+
end
|
@@ -0,0 +1,47 @@
|
|
1
|
+
require 'async'
|
2
|
+
|
3
|
+
module Fluent::Plugin
|
4
|
+
class PrometheusInput
|
5
|
+
module AsyncWrapper
|
6
|
+
def do_request(host:, port:, secure:)
|
7
|
+
endpoint =
|
8
|
+
if secure
|
9
|
+
context = OpenSSL::SSL::SSLContext.new
|
10
|
+
context.verify_mode = OpenSSL::SSL::VERIFY_NONE
|
11
|
+
Async::HTTP::Endpoint.parse("https://#{host}:#{port}", ssl_context: context)
|
12
|
+
else
|
13
|
+
Async::HTTP::Endpoint.parse("http://#{host}:#{port}")
|
14
|
+
end
|
15
|
+
|
16
|
+
Async::HTTP::Client.open(endpoint) do |client|
|
17
|
+
yield(AsyncHttpWrapper.new(client))
|
18
|
+
end
|
19
|
+
end
|
20
|
+
|
21
|
+
Response = Struct.new(:code, :body, :headers)
|
22
|
+
|
23
|
+
class AsyncHttpWrapper
|
24
|
+
def initialize(http)
|
25
|
+
@http = http
|
26
|
+
end
|
27
|
+
|
28
|
+
def get(path)
|
29
|
+
error = nil
|
30
|
+
response = Async::Task.current.async {
|
31
|
+
begin
|
32
|
+
@http.get(path)
|
33
|
+
rescue => e # Async::Reactor rescue all error. handle it by itself
|
34
|
+
error = e
|
35
|
+
end
|
36
|
+
}.wait
|
37
|
+
|
38
|
+
if error
|
39
|
+
raise error
|
40
|
+
end
|
41
|
+
|
42
|
+
Response.new(response.status.to_s, response.read || '', response.headers)
|
43
|
+
end
|
44
|
+
end
|
45
|
+
end
|
46
|
+
end
|
47
|
+
end
|