fluent-plugin-http-pull 0.8.1 → 0.8.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +5 -5
- data/README.md +222 -75
- data/fluent-plugin-http-pull.gemspec +1 -1
- data/lib/fluent/plugin/in_http_pull.rb +1 -1
- metadata +3 -3
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
|
-
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 001a669d9fa3eccbeaa6f6db2e532e27fe85bfba518463a8dceeac3246abeca6
|
|
4
|
+
data.tar.gz: 19effaa1d73dd98100104ce3aaf3966d23bbf6a63ec76ee6338a33d789d237fb
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 9d3647df586360e9df56a3aae6379c9cdb9df729a55af9440f441c1d6c1da47527ec576de3038de635a8ca262f3fd052b88990a60971ca00aab11a61fa1d11c0
|
|
7
|
+
data.tar.gz: 6198fc5e08f7832a52524f8ef043e8cf10baba91ec2c42ec5312d2f47bbe79409e08eb296e0997912406ffaa9564d7cb766faf416f756fe4126da0342d4337f2
|
data/README.md
CHANGED
|
@@ -8,7 +8,37 @@
|
|
|
8
8
|
|
|
9
9
|
[Fluentd](http://fluentd.org/) input plugin to pull log from rest api.
|
|
10
10
|
|
|
11
|
-
Many of modern server application offer status reporting API via http (even 'fluentd' too).
|
|
11
|
+
Many of modern server application offer status reporting API via http (even 'fluentd' too).
|
|
12
|
+
This plugin will help to gathering status log from these status api.
|
|
13
|
+
|
|
14
|
+
* [Installation](#installation)
|
|
15
|
+
* [Usage](#usage)
|
|
16
|
+
* [Basic Usage](#basic-usage)
|
|
17
|
+
* [Monitor Status Code](#monitoring-http-status-code-only)
|
|
18
|
+
* [Override User Agent](#override-user-agent)
|
|
19
|
+
* [HTTP Basic Auth](#http-basic-auth)
|
|
20
|
+
* [HTTP Proxy](#http-proxy)
|
|
21
|
+
* [Logging Response Header](#logging-http-response-header)
|
|
22
|
+
* [Custom Request Header](#custom-request-header)
|
|
23
|
+
* [HTTPS Support](#https-support)
|
|
24
|
+
* [Configuration](#configuration)
|
|
25
|
+
* [tag](#tag-string-required)
|
|
26
|
+
* [url](#url-string-required)
|
|
27
|
+
* [agent](#agent-string-optional-default-fluent-plugin-http-pull)
|
|
28
|
+
* [interval](#interval-time-required)
|
|
29
|
+
* [format](#format-required)
|
|
30
|
+
* [status_only](#status_only-bool-optional-default-false)
|
|
31
|
+
* [http_method](#http_method-enum-optional-default-get)
|
|
32
|
+
* [timeout](#timeout-time-optional-default-10s)
|
|
33
|
+
* [proxy](#proxy-string-optional-default-nil)
|
|
34
|
+
* [user](#user-string-optional-default-nil)
|
|
35
|
+
* [password](#password-string-optional-default-nil)
|
|
36
|
+
* [response_header](#response_header-section-optional-default-nil)
|
|
37
|
+
* [request_header](#response_header-section-optional-default-nil)
|
|
38
|
+
* [verify_ssl](#verify_ssl-bool-optional-default-true)
|
|
39
|
+
* [ca_path](#ca_path-string-optional-defualt-nil)
|
|
40
|
+
* [ca_file](#ca_file-string-optional-defualt-nil)
|
|
41
|
+
* [Copyright](#copyright)
|
|
12
42
|
|
|
13
43
|
## Installation
|
|
14
44
|
|
|
@@ -32,87 +62,227 @@ And then execute:
|
|
|
32
62
|
$ bundle
|
|
33
63
|
```
|
|
34
64
|
|
|
35
|
-
##
|
|
65
|
+
## Usage
|
|
36
66
|
|
|
37
|
-
|
|
67
|
+
### basic usage
|
|
68
|
+
|
|
69
|
+
In your Fluentd configuration, use @type http_pull.
|
|
70
|
+
`tag`, `url`, `interval`, `format` is mandatory configuration.
|
|
71
|
+
|
|
72
|
+
```
|
|
73
|
+
<source>
|
|
74
|
+
@type http_pull
|
|
75
|
+
|
|
76
|
+
tag status
|
|
77
|
+
url http://your-infrastructure/api/status.json
|
|
78
|
+
interval 1s
|
|
79
|
+
|
|
80
|
+
format json
|
|
81
|
+
</source>
|
|
82
|
+
|
|
83
|
+
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
In this example, a response of your infrastructure parsed as json.
|
|
87
|
+
A result contains `url`, `status`, `message`. `message` is a response of your infrastructure.
|
|
88
|
+
|
|
89
|
+
```
|
|
90
|
+
{
|
|
91
|
+
"url": "http://your-infrastructure/api/status.json",
|
|
92
|
+
"status": 200,
|
|
93
|
+
"message": {
|
|
94
|
+
// response of your infra structure
|
|
95
|
+
}
|
|
96
|
+
}
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
You can found more examples in this document.
|
|
38
100
|
|
|
39
101
|
### Monitoring http status code only
|
|
102
|
+
|
|
103
|
+
If you need only http status code, not response body, You can turn off response
|
|
104
|
+
body parser to set `status_only` is true. Remember that `format` is mandatory.
|
|
105
|
+
In this case, you must set `format` is none.
|
|
106
|
+
|
|
107
|
+
|
|
40
108
|
```
|
|
41
109
|
<source>
|
|
42
|
-
|
|
110
|
+
@type http_pull
|
|
43
111
|
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
112
|
+
tag fluentd.status
|
|
113
|
+
url http://your-infrastructure/healthcheck
|
|
114
|
+
interval 1s
|
|
47
115
|
|
|
48
|
-
|
|
49
|
-
|
|
116
|
+
status_only true
|
|
117
|
+
format none
|
|
50
118
|
</source>
|
|
51
119
|
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
120
|
+
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/healthcheck","status":200}
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
### Override User Agent
|
|
124
|
+
|
|
125
|
+
You can set the User Agent by specifying the `agent` option.
|
|
55
126
|
|
|
56
|
-
# 2017-05-17 21:40:40.413219000 +0900 test: {"url":"http://www.google.com","status":200}
|
|
57
|
-
# 2017-05-17 21:40:41.298215000 +0900 test: {"url":"http://www.google.com","status":200}
|
|
58
|
-
# 2017-05-17 21:40:42.310993000 +0900 test: {"url":"http://www.google.com","status":200}
|
|
59
|
-
# 2017-05-17 21:40:43.305947000 +0900 test: {"url":"http://www.google.com","status":200}
|
|
60
127
|
```
|
|
128
|
+
<source>
|
|
129
|
+
@type http_pull
|
|
130
|
+
|
|
131
|
+
tag status
|
|
132
|
+
url http://your-infrastructure/api/status.json
|
|
133
|
+
interval 1s
|
|
134
|
+
|
|
135
|
+
format json
|
|
136
|
+
agent infrastructure_monitor
|
|
137
|
+
</source>
|
|
138
|
+
|
|
139
|
+
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
### HTTP basic auth
|
|
143
|
+
|
|
144
|
+
If your infrastructure is protected by HTTP basic auth,
|
|
145
|
+
You can use `user`, `password` options to provide authentication information.
|
|
61
146
|
|
|
62
|
-
### Monitoring fluentd itself
|
|
63
147
|
```
|
|
64
148
|
<source>
|
|
65
|
-
|
|
149
|
+
@type http_pull
|
|
150
|
+
|
|
151
|
+
tag status
|
|
152
|
+
url http://your-infrastructure/api/status.json
|
|
153
|
+
interval 1s
|
|
66
154
|
|
|
67
|
-
|
|
155
|
+
format json
|
|
156
|
+
user foo
|
|
157
|
+
passwrd bar
|
|
68
158
|
</source>
|
|
69
159
|
|
|
160
|
+
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
### HTTP proxy
|
|
164
|
+
|
|
165
|
+
You can send your requests via proxy server.
|
|
166
|
+
|
|
167
|
+
```
|
|
70
168
|
<source>
|
|
71
|
-
|
|
169
|
+
@type http_pull
|
|
72
170
|
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
171
|
+
tag status
|
|
172
|
+
url http://your-infrastructure/api/status.json
|
|
173
|
+
proxy http://your-proxy-server:3128
|
|
174
|
+
interval 1s
|
|
76
175
|
|
|
77
|
-
|
|
176
|
+
format json
|
|
78
177
|
</source>
|
|
79
178
|
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
179
|
+
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
### Logging HTTP response header
|
|
183
|
+
|
|
184
|
+
If you wish to monitoring not only response body but also response header,
|
|
185
|
+
provide name of header in `response_header` sections.
|
|
83
186
|
|
|
84
|
-
# 2017-05-17 21:41:47.872951000 +0900 fluentd.status: {"url":"http://localhost:24220/api/plugins.json","status":200,"message":{"plugins":[{"plugin_id":"object:1e7e3d...
|
|
85
|
-
# 2017-05-17 21:41:48.955316000 +0900 fluentd.status: {"url":"http://localhost:24220/api/plugins.json","status":200,"message":{"plugins":[{"plugin_id":"object:1e7e3d...
|
|
86
|
-
# 2017-05-17 21:41:50.033628000 +0900 fluentd.status: {"url":"http://localhost:24220/api/plugins.json","status":200,"message":{"plugins":[{"plugin_id":"object:1e7e3d...
|
|
87
|
-
# 2017-05-17 21:41:51.107372000 +0900 fluentd.status: {"url":"http://localhost:24220/api/plugins.json","status":200,"message":{"plugins":[{"plugin_id":"object:1e7e3d...
|
|
88
187
|
```
|
|
188
|
+
<source>
|
|
189
|
+
@type http_pull
|
|
190
|
+
|
|
191
|
+
tag status
|
|
192
|
+
url http://your-infrastructure/api/status.json
|
|
193
|
+
interval 1s
|
|
194
|
+
|
|
195
|
+
format json
|
|
89
196
|
|
|
197
|
+
<response_header>
|
|
198
|
+
header Content-Type
|
|
199
|
+
</response_header>
|
|
200
|
+
|
|
201
|
+
<response_header>
|
|
202
|
+
header Content-Length
|
|
203
|
+
</response_header>
|
|
204
|
+
</source>
|
|
205
|
+
|
|
206
|
+
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
|
|
210
|
+
### Custom request header
|
|
211
|
+
|
|
212
|
+
The custom request header also supported.
|
|
213
|
+
This will help you when your infrastructure needs authentication headers or something like that.
|
|
90
214
|
|
|
91
|
-
### Monitoring elasticsearch cluster health
|
|
92
215
|
```
|
|
93
216
|
<source>
|
|
94
|
-
|
|
217
|
+
@type http_pull
|
|
218
|
+
|
|
219
|
+
tag status
|
|
220
|
+
url http://your-infrastructure/api/status.json
|
|
221
|
+
interval 1s
|
|
95
222
|
|
|
96
|
-
|
|
97
|
-
url http://localhost:9200/_cluster/health
|
|
98
|
-
interval 1s
|
|
223
|
+
format json
|
|
99
224
|
|
|
100
|
-
|
|
225
|
+
<request_header>
|
|
226
|
+
header API_ACCESS_KEY
|
|
227
|
+
value hatsune
|
|
228
|
+
</request_header>
|
|
229
|
+
|
|
230
|
+
<response_header>
|
|
231
|
+
header API_ACCESS_KEY_SECRET
|
|
232
|
+
value miku
|
|
233
|
+
</response_header>
|
|
101
234
|
</source>
|
|
102
235
|
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
236
|
+
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
|
|
237
|
+
```
|
|
238
|
+
|
|
239
|
+
|
|
240
|
+
### HTTPS support
|
|
241
|
+
|
|
242
|
+
If your infrastructure has https endpoints, just use https url for monitoring them.
|
|
243
|
+
|
|
244
|
+
#### self-signed SSL
|
|
245
|
+
|
|
246
|
+
|
|
247
|
+
If your infrastructure has https endpoints secured by self signed certification,
|
|
248
|
+
you can provide custom certification file via `ca_path`, `ca_file` option.
|
|
106
249
|
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
250
|
+
```
|
|
251
|
+
<source>
|
|
252
|
+
@type http_pull
|
|
253
|
+
|
|
254
|
+
tag status
|
|
255
|
+
url https://your-infrastructure/api/status.json
|
|
256
|
+
interval 1s
|
|
257
|
+
ca_path /somewhere/ca/stored
|
|
258
|
+
ca_file /somewhere/ca/stored/server.crt
|
|
259
|
+
|
|
260
|
+
format json
|
|
261
|
+
</source>
|
|
262
|
+
|
|
263
|
+
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
And, disable SSL verification also supported. (not recommended)
|
|
267
|
+
|
|
268
|
+
```
|
|
269
|
+
<source>
|
|
270
|
+
@type http_pull
|
|
271
|
+
|
|
272
|
+
tag status
|
|
273
|
+
url https://your-infrastructure/api/status.json
|
|
274
|
+
interval 1s
|
|
275
|
+
verify_ssl false
|
|
276
|
+
|
|
277
|
+
format json
|
|
278
|
+
</source>
|
|
279
|
+
|
|
280
|
+
# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}
|
|
111
281
|
```
|
|
112
282
|
|
|
113
283
|
## Configuration
|
|
114
284
|
|
|
115
|
-
### Basic
|
|
285
|
+
### Basic Configuration
|
|
116
286
|
|
|
117
287
|
#### tag (string) (required)
|
|
118
288
|
|
|
@@ -143,7 +313,7 @@ for more detail.
|
|
|
143
313
|
|
|
144
314
|
If `status_only` is true, body is not parsed.
|
|
145
315
|
|
|
146
|
-
|
|
316
|
+
#### http_method (enum) (optional, default: :get)
|
|
147
317
|
|
|
148
318
|
The http request method for each requests. Avaliable options are listed below.
|
|
149
319
|
|
|
@@ -157,13 +327,13 @@ If `status_only` is true, `http_method` was override to `head`
|
|
|
157
327
|
|
|
158
328
|
The timeout of each request.
|
|
159
329
|
|
|
160
|
-
### Proxy
|
|
330
|
+
### Proxy Configuration
|
|
161
331
|
|
|
162
332
|
#### proxy (string) (optional, default: nil)
|
|
163
333
|
|
|
164
334
|
The HTTP proxy URL to use for each requests
|
|
165
335
|
|
|
166
|
-
### Basic
|
|
336
|
+
### Basic Auth Configuration
|
|
167
337
|
|
|
168
338
|
#### user (string) (optional, default: nil)
|
|
169
339
|
|
|
@@ -173,7 +343,7 @@ The user for basic auth
|
|
|
173
343
|
|
|
174
344
|
The password for basic auth
|
|
175
345
|
|
|
176
|
-
### Req/Resp
|
|
346
|
+
### Req/Resp Header Configuration
|
|
177
347
|
|
|
178
348
|
#### response_header (section) (optional, default: nil)
|
|
179
349
|
|
|
@@ -183,7 +353,7 @@ The name of response header for capture.
|
|
|
183
353
|
|
|
184
354
|
The name, value pair of custom reuqest header.
|
|
185
355
|
|
|
186
|
-
### SSL
|
|
356
|
+
### SSL Configuration
|
|
187
357
|
|
|
188
358
|
#### verify_ssl (bool) (optional, default: true)
|
|
189
359
|
|
|
@@ -195,33 +365,10 @@ The absolute path of directory where ca_file stored. Should be used with `ca_fil
|
|
|
195
365
|
|
|
196
366
|
#### ca_file (string) (optional, defualt: nil)
|
|
197
367
|
|
|
198
|
-
The
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
## In case of remote error
|
|
202
|
-
|
|
203
|
-
### Can receive response from remote
|
|
204
|
-
|
|
205
|
-
```
|
|
206
|
-
{
|
|
207
|
-
"url": url of remote
|
|
208
|
-
"status": status code of response
|
|
209
|
-
"error": "RestClient::NotFound: 404 Not Found" or something similar
|
|
210
|
-
}
|
|
211
|
-
```
|
|
212
|
-
|
|
213
|
-
### All the other case
|
|
214
|
-
|
|
215
|
-
```
|
|
216
|
-
{
|
|
217
|
-
"url": url of remote
|
|
218
|
-
"status": 0
|
|
219
|
-
"error": "Errno::ECONNREFUSED: Connection refused - connect(2) for "localhost" port 12345" or something similar
|
|
220
|
-
}
|
|
221
|
-
```
|
|
368
|
+
The absolute path of ca_file. Should be used with `ca_path`.
|
|
222
369
|
|
|
223
370
|
## Copyright
|
|
224
371
|
|
|
225
372
|
* Copyright(c) 2017- filepang
|
|
226
373
|
* License
|
|
227
|
-
|
|
374
|
+
* Apache License, Version 2.0
|
|
@@ -57,7 +57,7 @@ module Fluent
|
|
|
57
57
|
config_param :user, :string, default: nil
|
|
58
58
|
|
|
59
59
|
desc 'password of basic auth'
|
|
60
|
-
config_param :password, :string, default: nil
|
|
60
|
+
config_param :password, :string, default: nil, secret: true
|
|
61
61
|
|
|
62
62
|
# req/res header options
|
|
63
63
|
config_section :response_header, param_name: :response_headers, multi: true do
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: fluent-plugin-http-pull
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.8.
|
|
4
|
+
version: 0.8.2
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- filepang
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2018-
|
|
11
|
+
date: 2018-08-25 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: rake
|
|
@@ -178,7 +178,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
178
178
|
version: '0'
|
|
179
179
|
requirements: []
|
|
180
180
|
rubyforge_project:
|
|
181
|
-
rubygems_version: 2.6
|
|
181
|
+
rubygems_version: 2.7.6
|
|
182
182
|
signing_key:
|
|
183
183
|
specification_version: 4
|
|
184
184
|
summary: fluent-plugin-http-pull
|