fluent-plugin-http-pull 0.8.1 → 0.8.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/README.md +222 -75
- data/fluent-plugin-http-pull.gemspec +1 -1
- data/lib/fluent/plugin/in_http_pull.rb +1 -1
- metadata +3 -3
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
|
-
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 001a669d9fa3eccbeaa6f6db2e532e27fe85bfba518463a8dceeac3246abeca6
|
|
4
|
+
data.tar.gz: 19effaa1d73dd98100104ce3aaf3966d23bbf6a63ec76ee6338a33d789d237fb
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 9d3647df586360e9df56a3aae6379c9cdb9df729a55af9440f441c1d6c1da47527ec576de3038de635a8ca262f3fd052b88990a60971ca00aab11a61fa1d11c0
|
|
7
|
+
data.tar.gz: 6198fc5e08f7832a52524f8ef043e8cf10baba91ec2c42ec5312d2f47bbe79409e08eb296e0997912406ffaa9564d7cb766faf416f756fe4126da0342d4337f2
|
data/README.md
CHANGED
|
@@ -8,7 +8,37 @@
|
|
|
8
8
|
|
|
9
9
|
[Fluentd](http://fluentd.org/) input plugin to pull log from rest api.
|
|
10
10
|
|
|
11
|
-
Many of modern server application offer status reporting API via http (even 'fluentd' too).
|
|
11
|
+
Many of modern server application offer status reporting API via http (even 'fluentd' too).
|
|
12
|
+
This plugin will help to gathering status log from these status api.
|
|
13
|
+
|
|
14
|
+
* [Installation](#installation)
|
|
15
|
+
* [Usage](#usage)
|
|
16
|
+
* [Basic Usage](#basic-usage)
|
|
17
|
+
* [Monitor Status Code](#monitoring-http-status-code-only)
|
|
18
|
+
* [Override User Agent](#override-user-agent)
|
|
19
|
+
* [HTTP Basic Auth](#http-basic-auth)
|
|
20
|
+
* [HTTP Proxy](#http-proxy)
|
|
21
|
+
* [Logging Response Header](#logging-http-response-header)
|
|
22
|
+
* [Custom Request Header](#custom-request-header)
|
|
23
|
+
* [HTTPS Support](#https-support)
|
|
24
|
+
* [Configuration](#configuration)
|
|
25
|
+
* [tag](#tag-string-required)
|
|
26
|
+
* [url](#url-string-required)
|
|
27
|
+
* [agent](#agent-string-optional-default-fluent-plugin-http-pull)
|
|
28
|
+
* [interval](#interval-time-required)
|
|
29
|
+
* [format](#format-required)
|
|
30
|
+
* [status_only](#status_only-bool-optional-default-false)
|
|
31
|
+
* [http_method](#http_method-enum-optional-default-get)
|
|
32
|
+
* [timeout](#timeout-time-optional-default-10s)
|
|
33
|
+
* [proxy](#proxy-string-optional-default-nil)
|
|
34
|
+
* [user](#user-string-optional-default-nil)
|
|
35
|
+
* [password](#password-string-optional-default-nil)
|
|
36
|
+
* [response_header](#response_header-section-optional-default-nil)
|
|
37
|
+
* [request_header](#response_header-section-optional-default-nil)
|
|
38
|
+
* [verify_ssl](#verify_ssl-bool-optional-default-true)
|
|
39
|
+
* [ca_path](#ca_path-string-optional-defualt-nil)
|
|
40
|
+
* [ca_file](#ca_file-string-optional-defualt-nil)
|
|
41
|
+
* [Copyright](#copyright)
|
|
12
42
|
|
|
13
43
|
## Installation
|
|
14
44
|
|
|
@@ -32,87 +62,227 @@ And then execute:
|
|
|
32
62
|
$ bundle
|
|
33
63
|
```
|
|
34
64
|
|
|
35
|
-
##
|
|
65
|
+
## Usage
|
|
36
66
|
|
|
37
|
-
|
|
67
|
+
### basic usage
|
|
68
|
+
|
|
69
|
+
In your Fluentd configuration, use @type http_pull.
|
|
70
|
+
`tag`, `url`, `interval`, `format` is mandatory configuration.
|
|
71
|
+
|
|
72
|
+
```
|
|
73
|
+
<source>
|
|
74
|
+
@type http_pull
|
|
75
|
+
|
|
76
|
+
tag status
|
|
77
|
+
url http://your-infrastructure/api/status.json
|
|
78
|
+
interval 1s
|
|
79
|
+
|
|
80
|
+
format json
|
|
81
|
+
</source>
|
|
82
|
+
|
|
83
|
+
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
In this example, a response of your infrastructure parsed as json.
|
|
87
|
+
A result contains `url`, `status`, `message`. `message` is a response of your infrastructure.
|
|
88
|
+
|
|
89
|
+
```
|
|
90
|
+
{
|
|
91
|
+
"url": "http://your-infrastructure/api/status.json",
|
|
92
|
+
"status": 200,
|
|
93
|
+
"message": {
|
|
94
|
+
// response of your infra structure
|
|
95
|
+
}
|
|
96
|
+
}
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
You can found more examples in this document.
|
|
38
100
|
|
|
39
101
|
### Monitoring http status code only
|
|
102
|
+
|
|
103
|
+
If you need only http status code, not response body, You can turn off response
|
|
104
|
+
body parser to set `status_only` is true. Remember that `format` is mandatory.
|
|
105
|
+
In this case, you must set `format` is none.
|
|
106
|
+
|
|
107
|
+
|
|
40
108
|
```
|
|
41
109
|
<source>
|
|
42
|
-
|
|
110
|
+
@type http_pull
|
|
43
111
|
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
112
|
+
tag fluentd.status
|
|
113
|
+
url http://your-infrastructure/healthcheck
|
|
114
|
+
interval 1s
|
|
47
115
|
|
|
48
|
-
|
|
49
|
-
|
|
116
|
+
status_only true
|
|
117
|
+
format none
|
|
50
118
|
</source>
|
|
51
119
|
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
120
|
+
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/healthcheck","status":200}
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
### Override User Agent
|
|
124
|
+
|
|
125
|
+
You can set the User Agent by specifying the `agent` option.
|
|
55
126
|
|
|
56
|
-
# 2017-05-17 21:40:40.413219000 +0900 test: {"url":"http://www.google.com","status":200}
|
|
57
|
-
# 2017-05-17 21:40:41.298215000 +0900 test: {"url":"http://www.google.com","status":200}
|
|
58
|
-
# 2017-05-17 21:40:42.310993000 +0900 test: {"url":"http://www.google.com","status":200}
|
|
59
|
-
# 2017-05-17 21:40:43.305947000 +0900 test: {"url":"http://www.google.com","status":200}
|
|
60
127
|
```
|
|
128
|
+
<source>
|
|
129
|
+
@type http_pull
|
|
130
|
+
|
|
131
|
+
tag status
|
|
132
|
+
url http://your-infrastructure/api/status.json
|
|
133
|
+
interval 1s
|
|
134
|
+
|
|
135
|
+
format json
|
|
136
|
+
agent infrastructure_monitor
|
|
137
|
+
</source>
|
|
138
|
+
|
|
139
|
+
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
### HTTP basic auth
|
|
143
|
+
|
|
144
|
+
If your infrastructure is protected by HTTP basic auth,
|
|
145
|
+
You can use `user`, `password` options to provide authentication information.
|
|
61
146
|
|
|
62
|
-
### Monitoring fluentd itself
|
|
63
147
|
```
|
|
64
148
|
<source>
|
|
65
|
-
|
|
149
|
+
@type http_pull
|
|
150
|
+
|
|
151
|
+
tag status
|
|
152
|
+
url http://your-infrastructure/api/status.json
|
|
153
|
+
interval 1s
|
|
66
154
|
|
|
67
|
-
|
|
155
|
+
format json
|
|
156
|
+
user foo
|
|
157
|
+
passwrd bar
|
|
68
158
|
</source>
|
|
69
159
|
|
|
160
|
+
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
### HTTP proxy
|
|
164
|
+
|
|
165
|
+
You can send your requests via proxy server.
|
|
166
|
+
|
|
167
|
+
```
|
|
70
168
|
<source>
|
|
71
|
-
|
|
169
|
+
@type http_pull
|
|
72
170
|
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
171
|
+
tag status
|
|
172
|
+
url http://your-infrastructure/api/status.json
|
|
173
|
+
proxy http://your-proxy-server:3128
|
|
174
|
+
interval 1s
|
|
76
175
|
|
|
77
|
-
|
|
176
|
+
format json
|
|
78
177
|
</source>
|
|
79
178
|
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
179
|
+
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
### Logging HTTP response header
|
|
183
|
+
|
|
184
|
+
If you wish to monitoring not only response body but also response header,
|
|
185
|
+
provide name of header in `response_header` sections.
|
|
83
186
|
|
|
84
|
-
# 2017-05-17 21:41:47.872951000 +0900 fluentd.status: {"url":"http://localhost:24220/api/plugins.json","status":200,"message":{"plugins":[{"plugin_id":"object:1e7e3d...
|
|
85
|
-
# 2017-05-17 21:41:48.955316000 +0900 fluentd.status: {"url":"http://localhost:24220/api/plugins.json","status":200,"message":{"plugins":[{"plugin_id":"object:1e7e3d...
|
|
86
|
-
# 2017-05-17 21:41:50.033628000 +0900 fluentd.status: {"url":"http://localhost:24220/api/plugins.json","status":200,"message":{"plugins":[{"plugin_id":"object:1e7e3d...
|
|
87
|
-
# 2017-05-17 21:41:51.107372000 +0900 fluentd.status: {"url":"http://localhost:24220/api/plugins.json","status":200,"message":{"plugins":[{"plugin_id":"object:1e7e3d...
|
|
88
187
|
```
|
|
188
|
+
<source>
|
|
189
|
+
@type http_pull
|
|
190
|
+
|
|
191
|
+
tag status
|
|
192
|
+
url http://your-infrastructure/api/status.json
|
|
193
|
+
interval 1s
|
|
194
|
+
|
|
195
|
+
format json
|
|
89
196
|
|
|
197
|
+
<response_header>
|
|
198
|
+
header Content-Type
|
|
199
|
+
</response_header>
|
|
200
|
+
|
|
201
|
+
<response_header>
|
|
202
|
+
header Content-Length
|
|
203
|
+
</response_header>
|
|
204
|
+
</source>
|
|
205
|
+
|
|
206
|
+
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
|
|
210
|
+
### Custom request header
|
|
211
|
+
|
|
212
|
+
The custom request header also supported.
|
|
213
|
+
This will help you when your infrastructure needs authentication headers or something like that.
|
|
90
214
|
|
|
91
|
-
### Monitoring elasticsearch cluster health
|
|
92
215
|
```
|
|
93
216
|
<source>
|
|
94
|
-
|
|
217
|
+
@type http_pull
|
|
218
|
+
|
|
219
|
+
tag status
|
|
220
|
+
url http://your-infrastructure/api/status.json
|
|
221
|
+
interval 1s
|
|
95
222
|
|
|
96
|
-
|
|
97
|
-
url http://localhost:9200/_cluster/health
|
|
98
|
-
interval 1s
|
|
223
|
+
format json
|
|
99
224
|
|
|
100
|
-
|
|
225
|
+
<request_header>
|
|
226
|
+
header API_ACCESS_KEY
|
|
227
|
+
value hatsune
|
|
228
|
+
</request_header>
|
|
229
|
+
|
|
230
|
+
<response_header>
|
|
231
|
+
header API_ACCESS_KEY_SECRET
|
|
232
|
+
value miku
|
|
233
|
+
</response_header>
|
|
101
234
|
</source>
|
|
102
235
|
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
236
|
+
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
|
|
237
|
+
```
|
|
238
|
+
|
|
239
|
+
|
|
240
|
+
### HTTPS support
|
|
241
|
+
|
|
242
|
+
If your infrastructure has https endpoints, just use https url for monitoring them.
|
|
243
|
+
|
|
244
|
+
#### self-signed SSL
|
|
245
|
+
|
|
246
|
+
|
|
247
|
+
If your infrastructure has https endpoints secured by self signed certification,
|
|
248
|
+
you can provide custom certification file via `ca_path`, `ca_file` option.
|
|
106
249
|
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
250
|
+
```
|
|
251
|
+
<source>
|
|
252
|
+
@type http_pull
|
|
253
|
+
|
|
254
|
+
tag status
|
|
255
|
+
url https://your-infrastructure/api/status.json
|
|
256
|
+
interval 1s
|
|
257
|
+
ca_path /somewhere/ca/stored
|
|
258
|
+
ca_file /somewhere/ca/stored/server.crt
|
|
259
|
+
|
|
260
|
+
format json
|
|
261
|
+
</source>
|
|
262
|
+
|
|
263
|
+
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
And, disable SSL verification also supported. (not recommended)
|
|
267
|
+
|
|
268
|
+
```
|
|
269
|
+
<source>
|
|
270
|
+
@type http_pull
|
|
271
|
+
|
|
272
|
+
tag status
|
|
273
|
+
url https://your-infrastructure/api/status.json
|
|
274
|
+
interval 1s
|
|
275
|
+
verify_ssl false
|
|
276
|
+
|
|
277
|
+
format json
|
|
278
|
+
</source>
|
|
279
|
+
|
|
280
|
+
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
|
|
111
281
|
```
|
|
112
282
|
|
|
113
283
|
## Configuration
|
|
114
284
|
|
|
115
|
-
### Basic
|
|
285
|
+
### Basic Configuration
|
|
116
286
|
|
|
117
287
|
#### tag (string) (required)
|
|
118
288
|
|
|
@@ -143,7 +313,7 @@ for more detail.
|
|
|
143
313
|
|
|
144
314
|
If `status_only` is true, body is not parsed.
|
|
145
315
|
|
|
146
|
-
|
|
316
|
+
#### http_method (enum) (optional, default: :get)
|
|
147
317
|
|
|
148
318
|
The http request method for each requests. Avaliable options are listed below.
|
|
149
319
|
|
|
@@ -157,13 +327,13 @@ If `status_only` is true, `http_method` was override to `head`
|
|
|
157
327
|
|
|
158
328
|
The timeout of each request.
|
|
159
329
|
|
|
160
|
-
### Proxy
|
|
330
|
+
### Proxy Configuration
|
|
161
331
|
|
|
162
332
|
#### proxy (string) (optional, default: nil)
|
|
163
333
|
|
|
164
334
|
The HTTP proxy URL to use for each requests
|
|
165
335
|
|
|
166
|
-
### Basic
|
|
336
|
+
### Basic Auth Configuration
|
|
167
337
|
|
|
168
338
|
#### user (string) (optional, default: nil)
|
|
169
339
|
|
|
@@ -173,7 +343,7 @@ The user for basic auth
|
|
|
173
343
|
|
|
174
344
|
The password for basic auth
|
|
175
345
|
|
|
176
|
-
### Req/Resp
|
|
346
|
+
### Req/Resp Header Configuration
|
|
177
347
|
|
|
178
348
|
#### response_header (section) (optional, default: nil)
|
|
179
349
|
|
|
@@ -183,7 +353,7 @@ The name of response header for capture.
|
|
|
183
353
|
|
|
184
354
|
The name, value pair of custom reuqest header.
|
|
185
355
|
|
|
186
|
-
### SSL
|
|
356
|
+
### SSL Configuration
|
|
187
357
|
|
|
188
358
|
#### verify_ssl (bool) (optional, default: true)
|
|
189
359
|
|
|
@@ -195,33 +365,10 @@ The absolute path of directory where ca_file stored. Should be used with `ca_fil
|
|
|
195
365
|
|
|
196
366
|
#### ca_file (string) (optional, defualt: nil)
|
|
197
367
|
|
|
198
|
-
The
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
## In case of remote error
|
|
202
|
-
|
|
203
|
-
### Can receive response from remote
|
|
204
|
-
|
|
205
|
-
```
|
|
206
|
-
{
|
|
207
|
-
"url": url of remote
|
|
208
|
-
"status": status code of response
|
|
209
|
-
"error": "RestClient::NotFound: 404 Not Found" or something similar
|
|
210
|
-
}
|
|
211
|
-
```
|
|
212
|
-
|
|
213
|
-
### All the other case
|
|
214
|
-
|
|
215
|
-
```
|
|
216
|
-
{
|
|
217
|
-
"url": url of remote
|
|
218
|
-
"status": 0
|
|
219
|
-
"error": "Errno::ECONNREFUSED: Connection refused - connect(2) for "localhost" port 12345" or something similar
|
|
220
|
-
}
|
|
221
|
-
```
|
|
368
|
+
The absolute path of ca_file. Should be used with `ca_path`.
|
|
222
369
|
|
|
223
370
|
## Copyright
|
|
224
371
|
|
|
225
372
|
* Copyright(c) 2017- filepang
|
|
226
373
|
* License
|
|
227
|
-
|
|
374
|
+
* Apache License, Version 2.0
|
|
@@ -57,7 +57,7 @@ module Fluent
|
|
|
57
57
|
config_param :user, :string, default: nil
|
|
58
58
|
|
|
59
59
|
desc 'password of basic auth'
|
|
60
|
-
config_param :password, :string, default: nil
|
|
60
|
+
config_param :password, :string, default: nil, secret: true
|
|
61
61
|
|
|
62
62
|
# req/res header options
|
|
63
63
|
config_section :response_header, param_name: :response_headers, multi: true do
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: fluent-plugin-http-pull
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.8.
|
|
4
|
+
version: 0.8.2
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- filepang
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2018-
|
|
11
|
+
date: 2018-08-25 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: rake
|
|
@@ -178,7 +178,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
178
178
|
version: '0'
|
|
179
179
|
requirements: []
|
|
180
180
|
rubyforge_project:
|
|
181
|
-
rubygems_version: 2.6
|
|
181
|
+
rubygems_version: 2.7.6
|
|
182
182
|
signing_key:
|
|
183
183
|
specification_version: 4
|
|
184
184
|
summary: fluent-plugin-http-pull
|