logstash-input-http_poller 5.0.0 → 5.2.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: dae0f4077b0291250677f8c0dec67e0c713bba9d6994b4b2406fbffb47f92459
4
- data.tar.gz: 05a45a34f59b148c8c04c73501f06fe1d7c41e3bde6ab6f814ca0e7a07a9f1e7
3
+ metadata.gz: 512535463aaa56747ba5e9d6a264887588966a6df43518955deccb19f2b7acae
4
+ data.tar.gz: 6c9b5b63198545b35e019b8e4f70edcec9f74ea9e8b4e53d6982b3841f48598b
5
5
  SHA512:
6
- metadata.gz: 9522d889fe4f3ae718783963437ed7c360e59755979d9e9db98cb5aef4569e1c4ffb10ca9f92d2f0de0d9c072104edf78b1d1c6b55a2209fdade646c2b14bd01
7
- data.tar.gz: 7dd5226d0a736cc25a4189a3751d3bf4bfa6f11c028858f2fc35cea2286e3b35ee5237b349df143d66f5316391bf33ed07302352d3be04d36be4dca770da5105
6
+ metadata.gz: 52a07cfe047ab7774d5ed1279ac9c5434031ccd86f57a6b93996345c8f7c1e6fe4792b8525dee40ef0be14832a2bfb85be192a80021f67a59adae507a0e01b5f
7
+ data.tar.gz: e614b85fd0558f4bfda62c8cb5df8811249cd6ed2cfa9ebfeb67caab08feb133a00e29737e2a806c471b59bf4efd75ac5404f06be38488f8e70987ebe8441ef6
data/CHANGELOG.md CHANGED
@@ -1,3 +1,15 @@
1
+ ## 5.2.0
2
+ - Feat: support ssl_verification_mode option [#131](https://github.com/logstash-plugins/logstash-input-http_poller/pull/131)
3
+
4
+ ## 5.1.0
5
+ - Add ECS support [#129](https://github.com/logstash-plugins/logstash-input-http_poller/pull/129)
6
+
7
+ ## 5.0.2
8
+ - [DOC]Expanded url option to include Manticore keys [#119](https://github.com/logstash-plugins/logstash-input-http_poller/pull/119)
9
+
10
+ ## 5.0.1
11
+ - Fixed minor doc and doc formatting issues [#107](https://github.com/logstash-plugins/logstash-input-http_poller/pull/107)
12
+
1
13
  ## 5.0.0
2
14
  - Removed obsolete field `interval`
3
15
 
@@ -15,7 +27,7 @@
15
27
  - Docs: Remove row in overview table to fix build error
16
28
 
17
29
  ## 4.0.2
18
- - Don't bleed URLs credentials on startup and on exception #82
30
+ - Don't bleed URLs credentials on startup and on exception [#82](https://github.com/logstash-plugins/logstash-input-http_poller/pull/82)
19
31
 
20
32
  ## 4.0.1
21
33
  - Fix some documentation issues
data/LICENSE CHANGED
@@ -1,3 +1,4 @@
1
+
1
2
  Apache License
2
3
  Version 2.0, January 2004
3
4
  http://www.apache.org/licenses/
@@ -178,7 +179,7 @@
178
179
  APPENDIX: How to apply the Apache License to your work.
179
180
 
180
181
  To apply the Apache License to your work, attach the following
181
- boilerplate notice, with the fields enclosed by brackets "{}"
182
+ boilerplate notice, with the fields enclosed by brackets "[]"
182
183
  replaced with your own identifying information. (Don't include
183
184
  the brackets!) The text should be enclosed in the appropriate
184
185
  comment syntax for the file format. We also recommend that a
@@ -186,7 +187,7 @@
186
187
  same "printed page" as the copyright notice for easier
187
188
  identification within third-party archives.
188
189
 
189
- Copyright {yyyy} {name of copyright owner}
190
+ Copyright 2020 Elastic and contributors
190
191
 
191
192
  Licensed under the Apache License, Version 2.0 (the "License");
192
193
  you may not use this file except in compliance with the License.
@@ -199,4 +200,3 @@
199
200
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200
201
  See the License for the specific language governing permissions and
201
202
  limitations under the License.
202
-
data/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Logstash HTTP input plugin
2
2
 
3
- [![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-input-http_poller.svg)](https://travis-ci.org/logstash-plugins/logstash-input-http_poller)
3
+ [![Travis Build Status](https://travis-ci.com/logstash-plugins/logstash-input-http_poller.svg)](https://travis-ci.com/logstash-plugins/logstash-input-http_poller)
4
4
 
5
5
  This plugin is based off [logstash-input-rest](https://github.com/maximede/logstash-input-rest) by @maximede.
6
6
 
data/docs/index.asciidoc CHANGED
@@ -87,6 +87,36 @@ The above snippet will create two files `downloaded_cert.pem` and `downloaded_tr
87
87
  ----------------------------------
88
88
 
89
89
 
90
+ [id="plugins-{type}s-{plugin}-ecs_metadata"]
91
+ ==== Event Metadata and the Elastic Common Schema (ECS)
92
+
93
+ This input will add metadata about the HTTP connection itself to each event.
94
+
95
+ When ECS compatibility is disabled, metadata was added to a variety of non-standard top-level fields, which has the potential to create confusion and schema conflicts downstream.
96
+
97
+ With ECS Compatibility Mode, we can ensure a pipeline maintains access to this metadata throughout the event's lifecycle without polluting the top-level namespace.
98
+
99
+ Here’s how ECS compatibility mode affects output.
100
+ [cols="<l,<l,e,<e"]
101
+ |=======================================================================
102
+ | ECS disabled | ECS v1 | Availability | Description
103
+
104
+ | [@metadata][host] | [@metadata][input][http_poller][request][host][hostname] | Always | Hostname
105
+ | [@metadata][code] | [@metadata][input][http_poller][response][status_code] | When server responds a valid status code | HTTP response code
106
+ | [@metadata][response_headers] | [@metadata][input][http_poller][response][headers] | When server responds with headers | HTTP headers of the response
107
+ | [@metadata][response_message] | [@metadata][input][http_poller][response][status_message] | When server responds with status line | message of status line of HTTP headers
108
+ | [@metadata][runtime_seconds] | [@metadata][input][http_poller][response][elapsed_time_ns] | When server responds a valid status code | elapsed time of calling endpoint. ECS v1 shows in nanoseconds.
109
+ | [http_request_failure][runtime_seconds] | [event][duration] | When server throws exception | elapsed time of calling endpoint. ECS v1 shows in nanoseconds.
110
+ | [@metadata][times_retried] | [@metadata][input][http_poller][request][retry_count] | When the poller calls server successfully | retry count from http client library
111
+ | [@metadata][name] / [http_request_failure][name] | [@metadata][input][http_poller][request][name] | Always | The key of `urls` from poller config
112
+ | [@metadata][request] / [http_request_failure][request]| [@metadata][input][http_poller][request][original] | Always | The whole object of `urls` from poller config
113
+ | [http_request_failure][error] | [error][message] | When server throws exception | Error message
114
+ | [http_request_failure][backtrace] | [error][stack_trace] | When server throws exception | Stack trace of error
115
+ | -- | [url][full] | When server throws exception | The URL of the endpoint
116
+ | -- | [http][request][method] | When server throws exception | HTTP request method
117
+ | -- | [host][hostname] | When server throws exception | Hostname
118
+ |=======================================================================
119
+
90
120
  [id="plugins-{type}s-{plugin}-options"]
91
121
  ==== Http_poller Input Configuration Options
92
122
 
@@ -95,20 +125,20 @@ This plugin supports the following configuration options plus the <<plugins-{typ
95
125
  [cols="<,<,<",options="header",]
96
126
  |=======================================================================
97
127
  |Setting |Input type|Required
98
- | <<plugins-{type}s-{plugin}-user>> |<<string,string>>|no
99
- | <<plugins-{type}s-{plugin}-password>> |<<password,password>>|No
100
128
  | <<plugins-{type}s-{plugin}-automatic_retries>> |<<number,number>>|No
101
129
  | <<plugins-{type}s-{plugin}-cacert>> |a valid filesystem path|No
102
130
  | <<plugins-{type}s-{plugin}-client_cert>> |a valid filesystem path|No
103
131
  | <<plugins-{type}s-{plugin}-client_key>> |a valid filesystem path|No
104
132
  | <<plugins-{type}s-{plugin}-connect_timeout>> |<<number,number>>|No
105
133
  | <<plugins-{type}s-{plugin}-cookies>> |<<boolean,boolean>>|No
134
+ | <<plugins-{type}s-{plugin}-ecs_compatibility>> | <<string,string>>|No
106
135
  | <<plugins-{type}s-{plugin}-follow_redirects>> |<<boolean,boolean>>|No
107
136
  | <<plugins-{type}s-{plugin}-keepalive>> |<<boolean,boolean>>|No
108
137
  | <<plugins-{type}s-{plugin}-keystore>> |a valid filesystem path|No
109
138
  | <<plugins-{type}s-{plugin}-keystore_password>> |<<password,password>>|No
110
139
  | <<plugins-{type}s-{plugin}-keystore_type>> |<<string,string>>|No
111
140
  | <<plugins-{type}s-{plugin}-metadata_target>> |<<string,string>>|No
141
+ | <<plugins-{type}s-{plugin}-password>> |<<password,password>>|No
112
142
  | <<plugins-{type}s-{plugin}-pool_max>> |<<number,number>>|No
113
143
  | <<plugins-{type}s-{plugin}-pool_max_per_route>> |<<number,number>>|No
114
144
  | <<plugins-{type}s-{plugin}-proxy>> |<<,>>|No
@@ -116,11 +146,13 @@ This plugin supports the following configuration options plus the <<plugins-{typ
116
146
  | <<plugins-{type}s-{plugin}-retry_non_idempotent>> |<<boolean,boolean>>|No
117
147
  | <<plugins-{type}s-{plugin}-schedule>> |<<hash,hash>>|Yes
118
148
  | <<plugins-{type}s-{plugin}-socket_timeout>> |<<number,number>>|No
149
+ | <<plugins-{type}s-{plugin}-ssl_verification_mode>> |<<string,string>>|No
119
150
  | <<plugins-{type}s-{plugin}-target>> |<<string,string>>|No
120
151
  | <<plugins-{type}s-{plugin}-truststore>> |a valid filesystem path|No
121
152
  | <<plugins-{type}s-{plugin}-truststore_password>> |<<password,password>>|No
122
153
  | <<plugins-{type}s-{plugin}-truststore_type>> |<<string,string>>|No
123
154
  | <<plugins-{type}s-{plugin}-urls>> |<<hash,hash>>|Yes
155
+ | <<plugins-{type}s-{plugin}-user>> |<<string,string>>|No
124
156
  | <<plugins-{type}s-{plugin}-validate_after_inactivity>> |<<number,number>>|No
125
157
  |=======================================================================
126
158
 
@@ -129,23 +161,6 @@ input plugins.
129
161
 
130
162
  &nbsp;
131
163
 
132
- [id="plugins-{type}s-{plugin}-user"]
133
- ===== `user`
134
-
135
- * Value type is <<string,string>>
136
- * There is no default value for this setting.
137
-
138
- Username to use with HTTP authentication for ALL requests. Note that you can also set this per-URL.
139
- If you set this you must also set the `password` option.
140
-
141
- [id="plugins-{type}s-{plugin}-password"]
142
- ===== `password`
143
-
144
- * Value type is <<password,password>>
145
- * There is no default value for this setting.
146
-
147
- Password to be used in conjunction with the username for HTTP authentication.
148
-
149
164
  [id="plugins-{type}s-{plugin}-automatic_retries"]
150
165
  ===== `automatic_retries`
151
166
 
@@ -197,6 +212,81 @@ Timeout (in seconds) to wait for a connection to be established. Default is `10s
197
212
  Enable cookie support. With this enabled the client will persist cookies
198
213
  across requests as a normal web browser would. Enabled by default
199
214
 
215
+ [id="plugins-{type}s-{plugin}-ecs_compatibility"]
216
+ ===== `ecs_compatibility`
217
+
218
+ * Value type is <<string,string>>
219
+ * Supported values are:
220
+ ** `disabled`: unstructured data added at root level
221
+ ** `v1`: uses `error`, `url` and `http` fields that are compatible with Elastic Common Schema
222
+
223
+ Controls this plugin's compatibility with the
224
+ {ecs-ref}[Elastic Common Schema (ECS)].
225
+ See <<plugins-{type}s-{plugin}-ecs_metadata>> for detailed information.
226
+
227
+ Example output:
228
+
229
+ **Sample output: ECS disabled**
230
+ [source,text]
231
+ -----
232
+ {
233
+ "http_poller_data" => {
234
+ "@version" => "1",
235
+ "@timestamp" => 2021-01-01T00:43:22.388Z,
236
+ "status" => "UP"
237
+ },
238
+ "@version" => "1",
239
+ "@timestamp" => 2021-01-01T00:43:22.389Z,
240
+ }
241
+ -----
242
+
243
+ **Sample output: ECS enabled**
244
+ [source,text]
245
+ -----
246
+ {
247
+ "http_poller_data" => {
248
+ "status" => "UP",
249
+ "@version" => "1",
250
+ "event" => {
251
+ "original" => "{\"status\":\"UP\"}"
252
+ },
253
+ "@timestamp" => 2021-01-01T00:40:59.558Z
254
+ },
255
+ "@version" => "1",
256
+ "@timestamp" => 2021-01-01T00:40:59.559Z
257
+ }
258
+ -----
259
+
260
+ **Sample error output: ECS enabled**
261
+ [source,text]
262
+ ----
263
+ {
264
+ "@timestamp" => 2021-07-09T09:53:48.721Z,
265
+ "@version" => "1",
266
+ "host" => {
267
+ "hostname" => "MacBook-Pro"
268
+ },
269
+ "http" => {
270
+ "request" => {
271
+ "method" => "get"
272
+ }
273
+ },
274
+ "event" => {
275
+ "duration" => 259019
276
+ },
277
+ "error" => {
278
+ "stack_trace" => nil,
279
+ "message" => "Connection refused (Connection refused)"
280
+ },
281
+ "url" => {
282
+ "full" => "http://localhost:8080/actuator/health"
283
+ },
284
+ "tags" => [
285
+ [0] "_http_request_failure"
286
+ ]
287
+ }
288
+ ----
289
+
200
290
  [id="plugins-{type}s-{plugin}-follow_redirects"]
201
291
  ===== `follow_redirects`
202
292
 
@@ -249,6 +339,14 @@ If you'd like to work with the request/response metadata.
249
339
  Set this value to the name of the field you'd like to store a nested
250
340
  hash of metadata.
251
341
 
342
+ [id="plugins-{type}s-{plugin}-password"]
343
+ ===== `password`
344
+
345
+ * Value type is <<password,password>>
346
+ * There is no default value for this setting.
347
+
348
+ Password to be used in conjunction with <<plugins-{type}s-{plugin}-user>> for HTTP authentication.
349
+
252
350
  [id="plugins-{type}s-{plugin}-pool_max"]
253
351
  ===== `pool_max`
254
352
 
@@ -316,6 +414,22 @@ See: rufus/scheduler for details about different schedule options and value stri
316
414
 
317
415
  Timeout (in seconds) to wait for data on the socket. Default is `10s`
318
416
 
417
+ [id="plugins-{type}s-{plugin}-ssl_verification_mode"]
418
+ ===== `ssl_verification_mode`
419
+
420
+ * Value type is <<string,string>>
421
+ * Supported values are: `full`, `none`
422
+ * Default value is `full`
423
+
424
+ Controls the verification of server certificates.
425
+ The `full` option verifies that the provided certificate is signed by a trusted authority (CA)
426
+ and also that the server’s hostname (or IP address) matches the names identified within the certificate.
427
+
428
+ The `none` setting performs no verification of the server’s certificate.
429
+ This mode disables many of the security benefits of SSL/TLS and should only be used after cautious consideration.
430
+ It is primarily intended as a temporary diagnostic mechanism when attempting to resolve TLS errors.
431
+ Using `none` in production environments is strongly discouraged.
432
+
319
433
  [id="plugins-{type}s-{plugin}-target"]
320
434
  ===== `target`
321
435
 
@@ -324,6 +438,10 @@ Timeout (in seconds) to wait for data on the socket. Default is `10s`
324
438
 
325
439
  Define the target field for placing the received data. If this setting is omitted, the data will be stored at the root (top level) of the event.
326
440
 
441
+ TIP: When ECS is enabled, set `target` in the codec (if the codec has a `target` option).
442
+ Example: `codec => json { target => "TARGET_FIELD_NAME" }`
443
+
444
+
327
445
  [id="plugins-{type}s-{plugin}-truststore"]
328
446
  ===== `truststore`
329
447
 
@@ -357,7 +475,45 @@ Specify the truststore type here. One of `JKS` or `PKCS12`. Default is `JKS`
357
475
  * There is no default value for this setting.
358
476
 
359
477
  A Hash of urls in this format : `"name" => "url"`.
360
- The name and the url will be passed in the outputed event
478
+ The name and the url will be passed in the outputted event.
479
+
480
+ The values in urls can be either:
481
+
482
+ * a string url (which will be issued as an HTTP GET).
483
+ * a sub-hash containing many useful keys provided by the Manticore backend:
484
+ ** url: the String url
485
+ ** method: (optional) the HTTP method to use (defaults to GET)
486
+ ** user: (optional) the HTTP Basic Auth user. The user must be under
487
+ an auth sub-hash for Manticore, but this plugin also accepts it either way.
488
+ ** password: (optional) the HTTP Basic Auth password. The password
489
+ must be under an auth sub-hash for Manticore, but this plugin accepts it either way.
490
+ ** headers: a hash containing key-value pairs of headers.
491
+ ** body: a string (supported only on POST and PUT requests)
492
+ ** possibly other options mentioned in the
493
+ https://www.rubydoc.info/github/cheald/manticore/Manticore/Client#http-instance_method[Manticore docs].
494
+ Note that Manticore options that are not explicitly documented above are not
495
+ thoroughly tested and therefore liable to break in unexpected ways if we
496
+ replace the backend.
497
+
498
+ *Notes:*
499
+
500
+ * Passwords specified as a part of `urls` are prone to exposure in plugin log output.
501
+ The plugin does not declare them as passwords, and therefore doesn't wrap them in
502
+ leak-reducing wrappers as we do elsewhere.
503
+ * We don't guarantee that boolean-type options like Manticore's `follow_redirects` are supported
504
+ correctly. The strings `true` or `false` may get passed through, and in ruby any
505
+ string is "truthy."
506
+ * Our implementation of this plugin precludes the ability to specify auth[:eager]
507
+ as anything other than true
508
+
509
+ [id="plugins-{type}s-{plugin}-user"]
510
+ ===== `user`
511
+
512
+ * Value type is <<string,string>>
513
+ * There is no default value for this setting.
514
+
515
+ Username to use with HTTP authentication for ALL requests. Note that you can also set this per-URL.
516
+ If you set this you must also set the <<plugins-{type}s-{plugin}-password>> option.
361
517
 
362
518
  [id="plugins-{type}s-{plugin}-validate_after_inactivity"]
363
519
  ===== `validate_after_inactivity`
@@ -365,15 +521,18 @@ The name and the url will be passed in the outputed event
365
521
  * Value type is <<number,number>>
366
522
  * Default value is `200`
367
523
 
368
- How long to wait before checking if the connection is stale before executing a request on a connection using keepalive.
369
- # You may want to set this lower, possibly to 0 if you get connection errors regularly
370
- Quoting the Apache commons docs (this client is based Apache Commmons):
371
- 'Defines period of inactivity in milliseconds after which persistent connections must be re-validated prior to being leased to the consumer. Non-positive value passed to this method disables connection validation. This check helps detect connections that have become stale (half-closed) while kept inactive in the pool.'
372
- See https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/conn/PoolingHttpClientConnectionManager.html#setValidateAfterInactivity(int)[these docs for more info]
373
-
524
+ How long to wait before checking for a stale connection to determine if a keepalive request is needed.
525
+ Consider setting this value lower than the default, possibly to 0, if you get connection errors regularly.
374
526
 
527
+ This client is based on Apache Commons' HTTP implementation. Here's how the
528
+ https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/conn/PoolingHttpClientConnectionManager.html#setValidateAfterInactivity(int)[Apache
529
+ Commons documentation] describes this option: "Defines period of inactivity in
530
+ milliseconds after which persistent connections must be re-validated prior to
531
+ being leased to the consumer. Non-positive value passed to this method disables
532
+ connection validation. This check helps detect connections that have become
533
+ stale (half-closed) while kept inactive in the pool."
375
534
 
376
535
  [id="plugins-{type}s-{plugin}-common-options"]
377
536
  include::{include_path}/{type}.asciidoc[]
378
537
 
379
- :default_codec!:
538
+ :default_codec!:
@@ -5,9 +5,18 @@ require "logstash/plugin_mixins/http_client"
5
5
  require "socket" # for Socket.gethostname
6
6
  require "manticore"
7
7
  require "rufus/scheduler"
8
+ require "logstash/plugin_mixins/ecs_compatibility_support"
9
+ require 'logstash/plugin_mixins/ecs_compatibility_support/target_check'
10
+ require 'logstash/plugin_mixins/validator_support/field_reference_validation_adapter'
11
+ require 'logstash/plugin_mixins/event_support/event_factory_adapter'
8
12
 
9
13
  class LogStash::Inputs::HTTP_Poller < LogStash::Inputs::Base
10
14
  include LogStash::PluginMixins::HttpClient
15
+ include LogStash::PluginMixins::ECSCompatibilitySupport(:disabled, :v1, :v8 => :v1)
16
+ include LogStash::PluginMixins::ECSCompatibilitySupport::TargetCheck
17
+ include LogStash::PluginMixins::EventSupport::EventFactoryAdapter
18
+
19
+ extend LogStash::PluginMixins::ValidatorSupport::FieldReferenceValidationAdapter
11
20
 
12
21
  config_name "http_poller"
13
22
 
@@ -28,7 +37,7 @@ class LogStash::Inputs::HTTP_Poller < LogStash::Inputs::Base
28
37
  config :schedule, :validate => :hash, :required => true
29
38
 
30
39
  # Define the target field for placing the received data. If this setting is omitted, the data will be stored at the root (top level) of the event.
31
- config :target, :validate => :string
40
+ config :target, :validate => :field_reference
32
41
 
33
42
  # If you'd like to work with the request/response metadata.
34
43
  # Set this value to the name of the field you'd like to store a nested
@@ -42,6 +51,7 @@ class LogStash::Inputs::HTTP_Poller < LogStash::Inputs::Base
42
51
 
43
52
  @logger.info("Registering http_poller Input", :type => @type, :schedule => @schedule, :timeout => @timeout)
44
53
 
54
+ setup_ecs_field!
45
55
  setup_requests!
46
56
  end
47
57
 
@@ -55,6 +65,35 @@ class LogStash::Inputs::HTTP_Poller < LogStash::Inputs::Base
55
65
  @requests = Hash[@urls.map {|name, url| [name, normalize_request(url)] }]
56
66
  end
57
67
 
68
+ private
69
+ # In the context of ECS, there are two type of events in this plugin, valid HTTP response and failure
70
+ # For a valid HTTP response, `url`, `request_method` and `host` are metadata of request.
71
+ # The call could retrieve event which contain `[url]`, `[http][request][method]`, `[host][hostname]` data
72
+ # Therefore, metadata should not write to those fields
73
+ # For a failure, `url`, `request_method` and `host` are primary data of the event because the plugin owns this event,
74
+ # so it writes to url.*, http.*, host.*
75
+ def setup_ecs_field!
76
+ @request_host_field = ecs_select[disabled: "[#{metadata_target}][host]", v1: "[#{metadata_target}][input][http_poller][request][host][hostname]"]
77
+ @response_code_field = ecs_select[disabled: "[#{metadata_target}][code]", v1: "[#{metadata_target}][input][http_poller][response][status_code]"]
78
+ @response_headers_field = ecs_select[disabled: "[#{metadata_target}][response_headers]", v1: "[#{metadata_target}][input][http_poller][response][headers]"]
79
+ @response_message_field = ecs_select[disabled: "[#{metadata_target}][response_message]", v1: "[#{metadata_target}][input][http_poller][response][status_message]"]
80
+ @response_time_s_field = ecs_select[disabled: "[#{metadata_target}][runtime_seconds]", v1: nil]
81
+ @response_time_ns_field = ecs_select[disabled: nil, v1: "[#{metadata_target}][input][http_poller][response][elapsed_time_ns]"]
82
+ @request_retry_count_field = ecs_select[disabled: "[#{metadata_target}][times_retried]", v1: "[#{metadata_target}][input][http_poller][request][retry_count]"]
83
+ @request_name_field = ecs_select[disabled: "[#{metadata_target}][name]", v1: "[#{metadata_target}][input][http_poller][request][name]"]
84
+ @original_request_field = ecs_select[disabled: "[#{metadata_target}][request]", v1: "[#{metadata_target}][input][http_poller][request][original]"]
85
+
86
+ @error_msg_field = ecs_select[disabled: "[http_request_failure][error]", v1: "[error][message]"]
87
+ @stack_trace_field = ecs_select[disabled: "[http_request_failure][backtrace]", v1: "[error][stack_trace]"]
88
+ @fail_original_request_field = ecs_select[disabled: "[http_request_failure][request]", v1: nil]
89
+ @fail_request_name_field = ecs_select[disabled: "[http_request_failure][name]", v1: nil]
90
+ @fail_response_time_s_field = ecs_select[disabled: "[http_request_failure][runtime_seconds]", v1: nil]
91
+ @fail_response_time_ns_field = ecs_select[disabled: nil, v1: "[event][duration]"]
92
+ @fail_request_url_field = ecs_select[disabled: nil, v1: "[url][full]"]
93
+ @fail_request_method_field = ecs_select[disabled: nil, v1: "[http][request][method]"]
94
+ @fail_request_host_field = ecs_select[disabled: nil, v1: "[host][hostname]"]
95
+ end
96
+
58
97
  private
59
98
  def normalize_request(url_or_spec)
60
99
  if url_or_spec.is_a?(String)
@@ -151,10 +190,14 @@ class LogStash::Inputs::HTTP_Poller < LogStash::Inputs::Base
151
190
 
152
191
  method, *request_opts = request
153
192
  client.async.send(method, *request_opts).
154
- on_success {|response| handle_success(queue, name, request, response, Time.now - started)}.
155
- on_failure {|exception|
156
- handle_failure(queue, name, request, exception, Time.now - started)
157
- }
193
+ on_success {|response| handle_success(queue, name, request, response, Time.now - started) }.
194
+ on_failure {|exception| handle_failure(queue, name, request, exception, Time.now - started) }
195
+ end
196
+
197
+ private
198
+ # time diff in float to nanoseconds
199
+ def to_nanoseconds(time_diff)
200
+ (time_diff * 1000000).to_i
158
201
  end
159
202
 
160
203
  private
@@ -164,11 +207,11 @@ class LogStash::Inputs::HTTP_Poller < LogStash::Inputs::Base
164
207
  # responses come up as "" which will cause the codec to not yield anything
165
208
  if body && body.size > 0
166
209
  decode_and_flush(@codec, body) do |decoded|
167
- event = @target ? LogStash::Event.new(@target => decoded.to_hash) : decoded
210
+ event = @target ? targeted_event_factory.new_event(decoded.to_hash) : decoded
168
211
  handle_decoded_event(queue, name, request, response, event, execution_time)
169
212
  end
170
213
  else
171
- event = ::LogStash::Event.new
214
+ event = event_factory.new_event
172
215
  handle_decoded_event(queue, name, request, response, event, execution_time)
173
216
  end
174
217
  end
@@ -197,20 +240,10 @@ class LogStash::Inputs::HTTP_Poller < LogStash::Inputs::Base
197
240
  private
198
241
  # Beware, on old versions of manticore some uncommon failures are not handled
199
242
  def handle_failure(queue, name, request, exception, execution_time)
200
- event = LogStash::Event.new
201
- apply_metadata(event, name, request)
202
-
243
+ event = event_factory.new_event
203
244
  event.tag("_http_request_failure")
204
-
205
- # This is also in the metadata, but we send it anyone because we want this
206
- # persisted by default, whereas metadata isn't. People don't like mysterious errors
207
- event.set("http_request_failure", {
208
- "request" => structure_request(request),
209
- "name" => name,
210
- "error" => exception.to_s,
211
- "backtrace" => exception.backtrace,
212
- "runtime_seconds" => execution_time
213
- })
245
+ apply_metadata(event, name, request, nil, execution_time)
246
+ apply_failure_fields(event, name, request, exception, execution_time)
214
247
 
215
248
  queue << event
216
249
  rescue StandardError, java.lang.Exception => e
@@ -231,29 +264,39 @@ class LogStash::Inputs::HTTP_Poller < LogStash::Inputs::Base
231
264
  end
232
265
 
233
266
  private
234
- def apply_metadata(event, name, request, response=nil, execution_time=nil)
267
+ def apply_metadata(event, name, request, response, execution_time)
235
268
  return unless @metadata_target
236
- event.set(@metadata_target, event_metadata(name, request, response, execution_time))
237
- end
238
-
239
- private
240
- def event_metadata(name, request, response=nil, execution_time=nil)
241
- m = {
242
- "name" => name,
243
- "host" => @host,
244
- "request" => structure_request(request),
245
- }
246
269
 
247
- m["runtime_seconds"] = execution_time
270
+ event.set(@request_host_field, @host)
271
+ event.set(@response_time_s_field, execution_time) if @response_time_s_field
272
+ event.set(@response_time_ns_field, to_nanoseconds(execution_time)) if @response_time_ns_field
273
+ event.set(@request_name_field, name)
274
+ event.set(@original_request_field, structure_request(request))
248
275
 
249
276
  if response
250
- m["code"] = response.code
251
- m["response_headers"] = response.headers
252
- m["response_message"] = response.message
253
- m["times_retried"] = response.times_retried
277
+ event.set(@response_code_field, response.code)
278
+ event.set(@response_headers_field, response.headers)
279
+ event.set(@response_message_field, response.message)
280
+ event.set(@request_retry_count_field, response.times_retried)
254
281
  end
282
+ end
255
283
 
256
- m
284
+ private
285
+ def apply_failure_fields(event, name, request, exception, execution_time)
286
+ # This is also in the metadata, but we send it anyone because we want this
287
+ # persisted by default, whereas metadata isn't. People don't like mysterious errors
288
+ event.set(@fail_original_request_field, structure_request(request)) if @fail_original_request_field
289
+ event.set(@fail_request_name_field, name) if @fail_request_name_field
290
+
291
+ method, url, _ = request
292
+ event.set(@fail_request_url_field, url) if @fail_request_url_field
293
+ event.set(@fail_request_method_field, method.to_s) if @fail_request_method_field
294
+ event.set(@fail_request_host_field, @host) if @fail_request_host_field
295
+
296
+ event.set(@fail_response_time_s_field, execution_time) if @fail_response_time_s_field
297
+ event.set(@fail_response_time_ns_field, to_nanoseconds(execution_time)) if @fail_response_time_ns_field
298
+ event.set(@error_msg_field, exception.to_s)
299
+ event.set(@stack_trace_field, exception.backtrace)
257
300
  end
258
301
 
259
302
  private
@@ -1,6 +1,6 @@
1
1
  Gem::Specification.new do |s|
2
2
  s.name = 'logstash-input-http_poller'
3
- s.version = '5.0.0'
3
+ s.version = '5.2.0'
4
4
  s.licenses = ['Apache License (2.0)']
5
5
  s.summary = "Decodes the output of an HTTP API into events"
6
6
  s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
@@ -20,9 +20,12 @@ Gem::Specification.new do |s|
20
20
  # Gem dependencies
21
21
  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
22
22
  s.add_runtime_dependency 'logstash-codec-plain'
23
- s.add_runtime_dependency 'logstash-mixin-http_client', "~> 7"
23
+ s.add_runtime_dependency "logstash-mixin-http_client", ">= 7.1.0"
24
24
  s.add_runtime_dependency 'stud', "~> 0.0.22"
25
25
  s.add_runtime_dependency 'rufus-scheduler', "~>3.0.9"
26
+ s.add_runtime_dependency 'logstash-mixin-ecs_compatibility_support', '~>1.3'
27
+ s.add_runtime_dependency 'logstash-mixin-event_support', '~> 1.0', '>= 1.0.1'
28
+ s.add_runtime_dependency 'logstash-mixin-validator_support', '~> 1.0'
26
29
 
27
30
  s.add_development_dependency 'logstash-codec-json'
28
31
  s.add_development_dependency 'logstash-codec-line'
@@ -1,9 +1,11 @@
1
1
  require "logstash/devutils/rspec/spec_helper"
2
+ require "logstash/devutils/rspec/shared_examples"
2
3
  require 'logstash/inputs/http_poller'
3
4
  require 'flores/random'
4
5
  require "timecop"
5
6
  # Workaround for the bug reported in https://github.com/jruby/jruby/issues/4637
6
7
  require 'rspec/matchers/built_in/raise_error.rb'
8
+ require 'logstash/plugin_mixins/ecs_compatibility_support/spec_helper'
7
9
 
8
10
  describe LogStash::Inputs::HTTP_Poller do
9
11
  let(:metadata_target) { "_http_poller_metadata" }
@@ -28,6 +30,7 @@ describe LogStash::Inputs::HTTP_Poller do
28
30
  }
29
31
  let(:klass) { LogStash::Inputs::HTTP_Poller }
30
32
 
33
+
31
34
  describe "instances" do
32
35
  subject { klass.new(default_opts) }
33
36
 
@@ -132,10 +135,10 @@ describe LogStash::Inputs::HTTP_Poller do
132
135
 
133
136
  it "should properly set the auth parameter" do
134
137
  expect(normalized[2][:auth]).to eq({
135
- :user => auth["user"],
136
- :pass => auth["password"],
137
- :eager => true
138
- })
138
+ :user => auth["user"],
139
+ :pass => auth["password"],
140
+ :eager => true
141
+ })
139
142
  end
140
143
  end
141
144
  end
@@ -143,14 +146,14 @@ describe LogStash::Inputs::HTTP_Poller do
143
146
  # Legacy way of doing things, kept for backwards compat.
144
147
  describe "auth with nested auth hash" do
145
148
  let(:url) { {"url" => "http://localhost", "method" => "get", "auth" => auth} }
146
-
149
+
147
150
  include_examples("auth")
148
151
  end
149
152
 
150
153
  # The new 'right' way to do things
151
154
  describe "auth with direct auth options" do
152
155
  let(:url) { {"url" => "http://localhost", "method" => "get", "user" => auth["user"], "password" => auth["password"]} }
153
-
156
+
154
157
  include_examples("auth")
155
158
  end
156
159
  end
@@ -254,7 +257,7 @@ describe LogStash::Inputs::HTTP_Poller do
254
257
  instance.stop
255
258
  runner.kill
256
259
  runner.join
257
- expect(queue.size).to eq(3)
260
+ expect(queue.size).to be_between(2, 3)
258
261
  end
259
262
  end
260
263
 
@@ -283,216 +286,267 @@ describe LogStash::Inputs::HTTP_Poller do
283
286
  end
284
287
  end
285
288
 
286
- describe "events" do
287
- shared_examples("matching metadata") {
288
- let(:metadata) { event.get(metadata_target) }
289
-
290
- it "should have the correct name" do
291
- expect(metadata["name"]).to eql(name)
289
+ describe "events", :ecs_compatibility_support, :aggregate_failures do
290
+ ecs_compatibility_matrix(:disabled, :v1, :v8 => :v1) do |ecs_select|
291
+ before do
292
+ allow_any_instance_of(described_class).to receive(:ecs_compatibility).and_return(ecs_compatibility)
292
293
  end
293
294
 
294
- it "should have the correct request url" do
295
- if url.is_a?(Hash) # If the url was specified as a complex test the whole thing
296
- expect(metadata["request"]).to eql(url)
297
- else # Otherwise we have to make some assumptions
298
- expect(metadata["request"]["url"]).to eql(url)
299
- end
300
- end
295
+ shared_examples("matching metadata") {
296
+ let(:metadata) { event.get(metadata_target) }
301
297
 
302
- it "should have the correct code" do
303
- expect(metadata["code"]).to eql(code)
304
- end
305
- }
298
+ it "should have the correct name" do
299
+ field = ecs_select[disabled: "[#{metadata_target}][name]", v1: "[#{metadata_target}][input][http_poller][request][name]"]
300
+ expect(event.get(field)).to eql(name)
301
+ end
306
302
 
307
- shared_examples "unprocessable_requests" do
308
- let(:poller) { LogStash::Inputs::HTTP_Poller.new(settings) }
309
- subject(:event) {
310
- poller.send(:run_once, queue)
311
- queue.pop(true)
312
- }
303
+ it "should have the correct request url" do
304
+ if url.is_a?(Hash) # If the url was specified as a complex test the whole thing
305
+ http_client_field = ecs_select[disabled: "[#{metadata_target}][request]",
306
+ v1: "[#{metadata_target}][input][http_poller][request][original]"]
307
+ expect(event.get(http_client_field)).to eql(url)
308
+ else # Otherwise we have to make some assumptions
309
+ url_field = ecs_select[disabled: "[#{metadata_target}][request][url]",
310
+ v1: "[#{metadata_target}][input][http_poller][request][original][url]"]
311
+ expect(event.get(url_field)).to eql(url)
312
+ end
313
+ end
313
314
 
314
- before do
315
- poller.register
316
- allow(poller).to receive(:handle_failure).and_call_original
317
- allow(poller).to receive(:handle_success)
318
- event # materialize the subject
319
- end
315
+ it "should have the correct code" do
316
+ expect(event.get(ecs_select[disabled: "[#{metadata_target}][code]",
317
+ v1: "[#{metadata_target}][input][http_poller][response][status_code]"]))
318
+ .to eql(code)
319
+ end
320
320
 
321
- it "should enqueue a message" do
322
- expect(event).to be_a(LogStash::Event)
323
- end
321
+ it "should have the correct host" do
322
+ expect(event.get(ecs_select[disabled: "[#{metadata_target}][host]",
323
+ v1: "[#{metadata_target}][input][http_poller][request][host][hostname]"]))
324
+ .not_to be_nil
325
+ end
326
+ }
324
327
 
325
- it "should enqueue a message with 'http_request_failure' set" do
326
- expect(event.get("http_request_failure")).to be_a(Hash)
327
- end
328
+ shared_examples "unprocessable_requests" do
329
+ let(:poller) { LogStash::Inputs::HTTP_Poller.new(settings) }
330
+ subject(:event) {
331
+ poller.send(:run_once, queue)
332
+ queue.pop(true)
333
+ }
328
334
 
329
- it "should tag the event with '_http_request_failure'" do
330
- expect(event.get("tags")).to include('_http_request_failure')
331
- end
335
+ before do
336
+ poller.register
337
+ allow(poller).to receive(:handle_failure).and_call_original
338
+ allow(poller).to receive(:handle_success)
339
+ event # materialize the subject
340
+ end
332
341
 
333
- it "should invoke handle failure exactly once" do
334
- expect(poller).to have_received(:handle_failure).once
335
- end
342
+ it "should enqueue a message" do
343
+ expect(event).to be_a(LogStash::Event)
344
+ end
336
345
 
337
- it "should not invoke handle success at all" do
338
- expect(poller).not_to have_received(:handle_success)
339
- end
346
+ it "should enqueue a message with 'http_request_failure' set" do
347
+ if ecs_compatibility == :disabled
348
+ expect(event.get("http_request_failure")).to be_a(Hash)
349
+ expect(event.get("[http_request_failure][runtime_seconds]")).to be_a(Float)
350
+ else
351
+ expect(event.get("http_request_failure")).to be_nil
352
+ expect(event.get("error")).to be_a(Hash)
353
+ expect(event.get("[event][duration]")).to be_a(Integer)
354
+ expect(event.get("[url][full]")).to eq(url)
355
+ expect(event.get("[http][request][method]")).to be_a(String)
356
+ expect(event.get("[host][hostname]")).to be_a(String)
357
+ end
358
+ end
340
359
 
341
- include_examples("matching metadata")
342
- end
360
+ it "should tag the event with '_http_request_failure'" do
361
+ expect(event.get("tags")).to include('_http_request_failure')
362
+ end
343
363
 
344
- context "with a non responsive server" do
345
- context "due to a non-existant host" do # Fail with handlers
346
- let(:name) { default_name }
347
- let(:url) { "http://thouetnhoeu89ueoueohtueohtneuohn" }
348
- let(:code) { nil } # no response expected
364
+ it "should invoke handle failure exactly once" do
365
+ expect(poller).to have_received(:handle_failure).once
366
+ end
349
367
 
350
- let(:settings) { default_opts.merge("urls" => { name => url}) }
368
+ it "should not invoke handle success at all" do
369
+ expect(poller).not_to have_received(:handle_success)
370
+ end
351
371
 
352
- include_examples("unprocessable_requests")
372
+ include_examples("matching metadata")
353
373
  end
354
374
 
355
- context "due to a bogus port number" do # fail with return?
356
- let(:invalid_port) { Flores::Random.integer(65536..1000000) }
375
+ context "with a non responsive server" do
376
+ context "due to a non-existant host" do # Fail with handlers
377
+ let(:name) { default_name }
378
+ let(:url) { "http://thouetnhoeu89ueoueohtueohtneuohn" }
379
+ let(:code) { nil } # no response expected
357
380
 
358
- let(:name) { default_name }
359
- let(:url) { "http://127.0.0.1:#{invalid_port}" }
360
- let(:settings) { default_opts.merge("urls" => {name => url}) }
361
- let(:code) { nil } # No response expected
381
+ let(:settings) { default_opts.merge("urls" => { name => url}) }
362
382
 
363
- include_examples("unprocessable_requests")
364
- end
365
- end
383
+ include_examples("unprocessable_requests")
384
+ end
366
385
 
367
- describe "a valid request and decoded response" do
368
- let(:payload) { {"a" => 2, "hello" => ["a", "b", "c"]} }
369
- let(:response_body) { LogStash::Json.dump(payload) }
370
- let(:opts) { default_opts }
371
- let(:instance) {
372
- klass.new(opts)
373
- }
374
- let(:name) { default_name }
375
- let(:url) { default_url }
376
- let(:code) { 202 }
386
+ context "due to a bogus port number" do # fail with return?
387
+ let(:invalid_port) { Flores::Random.integer(65536..1000000) }
377
388
 
378
- subject(:event) {
379
- queue.pop(true)
380
- }
389
+ let(:name) { default_name }
390
+ let(:url) { "http://127.0.0.1:#{invalid_port}" }
391
+ let(:settings) { default_opts.merge("urls" => {name => url}) }
392
+ let(:code) { nil } # No response expected
381
393
 
382
- before do
383
- instance.register
384
- u = url.is_a?(Hash) ? url["url"] : url # handle both complex specs and simple string URLs
385
- instance.client.stub(u,
386
- :body => response_body,
387
- :code => code
388
- )
389
- allow(instance).to receive(:decorate)
390
- instance.send(:run_once, queue)
394
+ include_examples("unprocessable_requests")
395
+ end
391
396
  end
392
397
 
393
- it "should have a matching message" do
394
- expect(event.to_hash).to include(payload)
395
- end
398
+ describe "a valid request and decoded response" do
399
+ let(:payload) { {"a" => 2, "hello" => ["a", "b", "c"]} }
400
+ let(:response_body) { LogStash::Json.dump(payload) }
401
+ let(:opts) { default_opts }
402
+ let(:instance) {
403
+ klass.new(opts)
404
+ }
405
+ let(:name) { default_name }
406
+ let(:url) { default_url }
407
+ let(:code) { 202 }
396
408
 
397
- it "should decorate the event" do
398
- expect(instance).to have_received(:decorate).once
399
- end
409
+ subject(:event) {
410
+ queue.pop(true)
411
+ }
400
412
 
401
- include_examples("matching metadata")
402
-
403
- context "with an empty body" do
404
- let(:response_body) { "" }
405
- it "should return an empty event" do
413
+ before do
414
+ instance.register
415
+ u = url.is_a?(Hash) ? url["url"] : url # handle both complex specs and simple string URLs
416
+ instance.client.stub(u,
417
+ :body => response_body,
418
+ :code => code
419
+ )
420
+ allow(instance).to receive(:decorate)
406
421
  instance.send(:run_once, queue)
407
- expect(event.get("[_http_poller_metadata][response_headers][content-length]")).to eql("0")
408
422
  end
409
- end
410
423
 
411
- context "with metadata omitted" do
412
- let(:opts) {
413
- opts = default_opts.clone
414
- opts.delete("metadata_target")
415
- opts
416
- }
424
+ it "should have a matching message" do
425
+ expect(event.to_hash).to include(payload)
426
+ end
417
427
 
418
- it "should not have any metadata on the event" do
419
- instance.send(:run_once, queue)
420
- expect(event.get(metadata_target)).to be_nil
428
+ it "should decorate the event" do
429
+ expect(instance).to have_received(:decorate).once
421
430
  end
422
- end
423
431
 
424
- context "with a complex URL spec" do
425
- let(:url) {
426
- {
427
- "method" => "get",
428
- "url" => default_url,
429
- "headers" => {
430
- "X-Fry" => "I'm having one of those things, like a headache, with pictures..."
432
+ include_examples("matching metadata")
433
+
434
+ context "with an empty body" do
435
+ let(:response_body) { "" }
436
+ it "should return an empty event" do
437
+ instance.send(:run_once, queue)
438
+ headers_field = ecs_select[disabled: "[#{metadata_target}][response_headers]",
439
+ v1: "[#{metadata_target}][input][http_poller][response][headers]"]
440
+ expect(event.get("#{headers_field}[content-length]")).to eql("0")
441
+ end
442
+ end
443
+
444
+ context "with metadata omitted" do
445
+ let(:opts) {
446
+ opts = default_opts.clone
447
+ opts.delete("metadata_target")
448
+ opts
449
+ }
450
+
451
+ it "should not have any metadata on the event" do
452
+ instance.send(:run_once, queue)
453
+ expect(event.get(metadata_target)).to be_nil
454
+ end
455
+ end
456
+
457
+ context "with a complex URL spec" do
458
+ let(:url) {
459
+ {
460
+ "method" => "get",
461
+ "url" => default_url,
462
+ "headers" => {
463
+ "X-Fry" => "I'm having one of those things, like a headache, with pictures..."
464
+ }
431
465
  }
432
466
  }
433
- }
434
- let(:opts) {
435
- {
436
- "schedule" => {
437
- "cron" => "* * * * * UTC"
467
+ let(:opts) {
468
+ {
469
+ "schedule" => {
470
+ "cron" => "* * * * * UTC"
471
+ },
472
+ "urls" => {
473
+ default_name => url
438
474
  },
439
- "urls" => {
440
- default_name => url
441
- },
442
- "codec" => "json",
443
- "metadata_target" => metadata_target
475
+ "codec" => "json",
476
+ "metadata_target" => metadata_target
477
+ }
444
478
  }
445
- }
446
479
 
447
- include_examples("matching metadata")
480
+ include_examples("matching metadata")
448
481
 
449
- it "should have a matching message" do
450
- expect(event.to_hash).to include(payload)
482
+ it "should have a matching message" do
483
+ expect(event.to_hash).to include(payload)
484
+ end
451
485
  end
452
- end
453
486
 
454
- context "with a specified target" do
455
- let(:target) { "mytarget" }
456
- let(:opts) { default_opts.merge("target" => target) }
487
+ context "with a specified target" do
488
+ let(:target) { "mytarget" }
489
+ let(:opts) { default_opts.merge("target" => target) }
457
490
 
458
- it "should store the event info in the target" do
459
- # When events go through the pipeline they are java-ified
460
- # this normalizes the payload to java types
461
- payload_normalized = LogStash::Json.load(LogStash::Json.dump(payload))
462
- expect(event.get(target)).to include(payload_normalized)
491
+ it "should store the event info in the target" do
492
+ # When events go through the pipeline they are java-ified
493
+ # this normalizes the payload to java types
494
+ payload_normalized = LogStash::Json.load(LogStash::Json.dump(payload))
495
+ expect(event.get(target)).to include(payload_normalized)
496
+ end
463
497
  end
464
- end
465
498
 
466
- context 'using a line codec' do
467
- let(:opts) do
468
- default_opts.merge({"codec" => "line"})
469
- end
470
- subject(:events) do
471
- [].tap do |events|
472
- events << queue.pop until queue.empty?
499
+ context "with default metadata target" do
500
+ let(:metadata_target) { "@metadata" }
501
+
502
+ it "should store the metadata info in @metadata" do
503
+ if ecs_compatibility == :disabled
504
+ expect(event.get("[@metadata][response_headers]")).to be_a(Hash)
505
+ expect(event.get("[@metadata][runtime_seconds]")).to be_a(Float)
506
+ expect(event.get("[@metadata][times_retried]")).to eq(0)
507
+ expect(event.get("[@metadata][name]")).to eq(default_name)
508
+ expect(event.get("[@metadata][request]")).to be_a(Hash)
509
+ else
510
+ expect(event.get("[@metadata][input][http_poller][response][headers]")).to be_a(Hash)
511
+ expect(event.get("[@metadata][input][http_poller][response][elapsed_time_ns]")).to be_a(Integer)
512
+ expect(event.get("[@metadata][input][http_poller][request][retry_count]")).to eq(0)
513
+ expect(event.get("[@metadata][input][http_poller][request][name]")).to eq(default_name)
514
+ expect(event.get("[@metadata][input][http_poller][request][original]")).to be_a(Hash)
515
+ end
473
516
  end
474
517
  end
475
518
 
476
- context 'when response has a trailing newline' do
477
- let(:response_body) { "one\ntwo\nthree\nfour\n" }
478
- it 'emits all events' do
479
- expect(events.size).to equal(4)
480
- messages = events.map{|e| e.get('message')}
481
- expect(messages).to include('one')
482
- expect(messages).to include('two')
483
- expect(messages).to include('three')
484
- expect(messages).to include('four')
519
+ context 'using a line codec' do
520
+ let(:opts) do
521
+ default_opts.merge({"codec" => "line"})
485
522
  end
486
- end
487
- context 'when response has no trailing newline' do
488
- let(:response_body) { "one\ntwo\nthree\nfour" }
489
- it 'emits all events' do
490
- expect(events.size).to equal(4)
491
- messages = events.map{|e| e.get('message')}
492
- expect(messages).to include('one')
493
- expect(messages).to include('two')
494
- expect(messages).to include('three')
495
- expect(messages).to include('four')
523
+ subject(:events) do
524
+ [].tap do |events|
525
+ events << queue.pop until queue.empty?
526
+ end
527
+ end
528
+
529
+ context 'when response has a trailing newline' do
530
+ let(:response_body) { "one\ntwo\nthree\nfour\n" }
531
+ it 'emits all events' do
532
+ expect(events.size).to equal(4)
533
+ messages = events.map{|e| e.get('message')}
534
+ expect(messages).to include('one')
535
+ expect(messages).to include('two')
536
+ expect(messages).to include('three')
537
+ expect(messages).to include('four')
538
+ end
539
+ end
540
+ context 'when response has no trailing newline' do
541
+ let(:response_body) { "one\ntwo\nthree\nfour" }
542
+ it 'emits all events' do
543
+ expect(events.size).to equal(4)
544
+ messages = events.map{|e| e.get('message')}
545
+ expect(messages).to include('one')
546
+ expect(messages).to include('two')
547
+ expect(messages).to include('three')
548
+ expect(messages).to include('four')
549
+ end
496
550
  end
497
551
  end
498
552
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-input-http_poller
3
3
  version: !ruby/object:Gem::Version
4
- version: 5.0.0
4
+ version: 5.2.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Elastic
@@ -9,7 +9,7 @@ authors:
9
9
  autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
- date: 2019-01-30 00:00:00.000000000 Z
12
+ date: 2022-02-01 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  requirement: !ruby/object:Gem::Requirement
@@ -48,17 +48,17 @@ dependencies:
48
48
  - !ruby/object:Gem::Dependency
49
49
  requirement: !ruby/object:Gem::Requirement
50
50
  requirements:
51
- - - "~>"
51
+ - - ">="
52
52
  - !ruby/object:Gem::Version
53
- version: '7'
53
+ version: 7.1.0
54
54
  name: logstash-mixin-http_client
55
55
  prerelease: false
56
56
  type: :runtime
57
57
  version_requirements: !ruby/object:Gem::Requirement
58
58
  requirements:
59
- - - "~>"
59
+ - - ">="
60
60
  - !ruby/object:Gem::Version
61
- version: '7'
61
+ version: 7.1.0
62
62
  - !ruby/object:Gem::Dependency
63
63
  requirement: !ruby/object:Gem::Requirement
64
64
  requirements:
@@ -87,6 +87,54 @@ dependencies:
87
87
  - - "~>"
88
88
  - !ruby/object:Gem::Version
89
89
  version: 3.0.9
90
+ - !ruby/object:Gem::Dependency
91
+ requirement: !ruby/object:Gem::Requirement
92
+ requirements:
93
+ - - "~>"
94
+ - !ruby/object:Gem::Version
95
+ version: '1.3'
96
+ name: logstash-mixin-ecs_compatibility_support
97
+ prerelease: false
98
+ type: :runtime
99
+ version_requirements: !ruby/object:Gem::Requirement
100
+ requirements:
101
+ - - "~>"
102
+ - !ruby/object:Gem::Version
103
+ version: '1.3'
104
+ - !ruby/object:Gem::Dependency
105
+ requirement: !ruby/object:Gem::Requirement
106
+ requirements:
107
+ - - "~>"
108
+ - !ruby/object:Gem::Version
109
+ version: '1.0'
110
+ - - ">="
111
+ - !ruby/object:Gem::Version
112
+ version: 1.0.1
113
+ name: logstash-mixin-event_support
114
+ prerelease: false
115
+ type: :runtime
116
+ version_requirements: !ruby/object:Gem::Requirement
117
+ requirements:
118
+ - - "~>"
119
+ - !ruby/object:Gem::Version
120
+ version: '1.0'
121
+ - - ">="
122
+ - !ruby/object:Gem::Version
123
+ version: 1.0.1
124
+ - !ruby/object:Gem::Dependency
125
+ requirement: !ruby/object:Gem::Requirement
126
+ requirements:
127
+ - - "~>"
128
+ - !ruby/object:Gem::Version
129
+ version: '1.0'
130
+ name: logstash-mixin-validator_support
131
+ prerelease: false
132
+ type: :runtime
133
+ version_requirements: !ruby/object:Gem::Requirement
134
+ requirements:
135
+ - - "~>"
136
+ - !ruby/object:Gem::Version
137
+ version: '1.0'
90
138
  - !ruby/object:Gem::Dependency
91
139
  requirement: !ruby/object:Gem::Requirement
92
140
  requirements:
@@ -195,8 +243,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
195
243
  - !ruby/object:Gem::Version
196
244
  version: '0'
197
245
  requirements: []
198
- rubyforge_project:
199
- rubygems_version: 2.6.13
246
+ rubygems_version: 3.1.6
200
247
  signing_key:
201
248
  specification_version: 4
202
249
  summary: Decodes the output of an HTTP API into events