logstash-input-httpclient 0.2.0 → 0.9.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (4) hide show
  1. checksums.yaml +4 -4
  2. data/lib/logstash/inputs/httpclient.rb +160 -99
  3. data/logstash-input-httpclient.gemspec +1 -1
  4. metadata +2 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 34f259c605e9064ef201bcf35e71b9aa20b82dc8
4
- data.tar.gz: b9d6f9eab9ad654bc368dbd708f7f7ad7193cf01
3
+ metadata.gz: e5937770c62c0b1d858d952beaf403efcf716b49
4
+ data.tar.gz: d505ba3fece17a54e9746dc6550bcc9c271a2fb0
5
5
  SHA512:
6
- metadata.gz: f58d38b040dda5c5bd5fbdbeb91b8438ac2c1962c1164f49b7d18c822cff00c76bf5822250b3ab55822db38f9005aefd8ecc5e60344d06fe1b6100c78d7b7525
7
- data.tar.gz: 69193c447b6aa56d3cfc7c03a1c829c46d76f0e6197f2e1a3fce12d46e0e820694fb999619896bcbd0a96b6cbdf7e81acbaa37d8f89b0789dadadbf682ce1b97
6
+ metadata.gz: 6dbe5f66ee634e65bb45fae4705af0dc11094bafcb7ec967aad550f3395094eb77a28482656df653b2e11144da4f3696a0c16918fb001806349f8fd68b896abc
7
+ data.tar.gz: 79c3b89b1fca5643b5ccf5e5c2b83148904a3f06fb42f6fc737fa54457ce4f5d6e2d86d779192acbb8454cd5b92bcf12940c91d86f2d1bc9ce987859daa64842
data/lib/logstash/inputs/httpclient.rb CHANGED
@@ -4,116 +4,177 @@ require "logstash/namespace"
4
4
  require "stud/interval"
5
5
  require "net/http"
6
6
 
7
- # Query a http server with GET requests at a regular interval.
7
+ # Query a HTTP server with GET requests at a regular interval and process the events returned in the response body.
8
8
  #
9
9
  #[%hardbreaks]
10
10
  # HTTP GET requests are performed using the provided url (which may contain URL parameters).
11
- # The HTTP Response body will be added to "message" after being decoded by the provided codec.
11
+ # The HTTP Response body will be decoded by the provided coded and the events will be sent to logstash for filtering/output.
12
+ #
13
+ # HTTP Header Information (advanced use cases):
14
+ # There are several additional HTTP headers that can be useful if needed (otherwise they can be safely ignored):
15
+ #
16
+ # **Response Headers** (from the server)
17
+ # `X-More-Events-Available` If set to `true`, then there will be no wait time before the next HTTP GET Request is executed to get more events.
18
+ # `X-Messages-Batch-Id` Can be set to any value to identify this 'batch' of messages. The client will add a `X-Successful-Batch-Ids` header to the next request if this batch is successfully sent to logstash.
19
+ #
20
+ # **Request Headers** (these provide stats/info about the **previous** HTTP Request and the events processed from it)
21
+ # `X-Logstash-Num-Events` The number of events that were processed from the previous request
22
+ # `X-Logstash-Http-Request-Total-Secs` The time (in seconds) that it took for the HTTP Request to finish.
23
+ # `X-Logstash-Codec-Parse-Total-Secs` The time (in seconds) that it took for the HTTP Response body to be parsed and decoded.
24
+ # `X-Logstash-Queue-Total-Secs` The time (in seconds) that it took to queue all of the events (logstash uses a fixed queue length of 20, so it blocks when full).
25
+ # `X-Successful-Batch-Ids` If the server provided a `X-Messages-Batch-Id` header in the previous request, that value is set in this header to acknowledge the messages were successfully sent to logstash.
26
+
12
27
  # There is an additional header named `X-Logstash-Avg-Queue-Secs` added to each HTTP request. This is the
13
28
  # average seconds it took for the past 20 events to be accepted by Logstash's bounded input queue.
14
29
  # The HTTP server could use this as a indication of how many events the Logstash instance is able to process (throughput).
15
30
 
16
31
 
17
32
  class LogStash::Inputs::HttpClient < LogStash::Inputs::Base
18
- config_name "httpclient"
19
-
20
- # If undefined, Logstash will complain, even if codec is unused.
21
- default :codec, "plain"
22
-
23
- #[%hardbreaks]
24
- # The full url to execute GET requests on.
25
- # It may contain URL parameters (you are responsible for URL encoding them)
26
- # e.g. `https://mywebservice.int/`
27
- # e.g. `http://mywebservice.int?queue=logstash_events&numevents=10`
28
- #
29
- # IMPORTANT: make sure you end with a slash (if not using url params). `http://mywebservice.int` is not valid, it must be `http://myswebservice.int/`
30
- # todo: auto-append slash if user forgets
31
- config :url, :validate => :string, :required => true
33
+ config_name "httpclient"
32
34
 
33
- # Set how frequently we should query the url for messages
34
- #
35
- # The default, `10`, means send a message every 10 seconds.
36
- config :interval, :validate => :number, :default => 10
37
-
38
- # The name of the object that http response meta-data (headers, response code, etc) should be added to.
39
- config :response_object_name, :validate => :string, :default => "http_response"
40
-
41
- # Should the http headers be added to the `response_object_name`?
42
- #
43
- # If true, there will be a "headers" with all the response headers/values.
44
- config :include_response_headers, :validate => :boolean, :default => false
45
-
46
- # Should the http response code be added to the `response_object_name`?
47
- #
48
- # If true, there will be a "code" with all the response headers/values.
49
- config :include_response_code, :validate => :boolean, :default => true
50
-
51
- # Should the http request elapsed time be added to the `response_object_name`?
52
- #
53
- # If true, there will be a "took_secs" with the time it took for the http request to complete
54
- config :include_http_request_time, :validate => :boolean, :default => true
55
-
56
- public
57
- def register
58
- @uri = URI(@url)
59
- end # def register
35
+ # If undefined, Logstash will complain, even if codec is unused.
36
+ default :codec, "plain"
60
37
 
61
- def run(queue)
62
- queue_times = Array.new #track an average of how long it's taking to queue events into logstash
63
- #::start creates a connection to the HTTP server and keeps it alive for the duration
64
- Net::HTTP.start @uri.host, @uri.port, :use_ssl => @uri.scheme == 'https' do |http|
65
- while true
66
- begin
67
- sleepIval = @interval
68
- begin
69
- http_start = Time.now
70
- request = Net::HTTP::Get.new(@uri.request_uri)
71
- request ["X-Logstash-Avg-Queue-Secs"] = arr_avg(queue_times, 20, 3)
72
- response = http.request request # Net::HTTPResponse object
73
- http_elapsed = Time.now - http_start
74
- rescue => e
75
- @logger.warn("Http request failed, will retry", :exception => e)
76
- @logger.warn(e.backtrace)
77
- sleep(sleepIval)
78
- retry
79
- end
38
+ #[%hardbreaks]
39
+ # The full url to execute GET requests on.
40
+ # It may contain URL parameters (you are responsible for URL encoding them)
41
+ # e.g. `https://mywebservice.mycompany.int/`
42
+ # e.g. `http://mywebservice.mycompany.int?type=logstash_events&numevents=10`
43
+ #
44
+ config :url, :validate => :string, :required => true
80
45
 
81
- if response["X-More-Events-Available"] == "true"
82
- sleepIval = 0 #don't wait if the server indicated there are more events ready now
83
- end
46
+ # Set how frequently we should query the url for messages
47
+ #
48
+ # The default, `10`, means execute a HTTP GET request every 10 seconds.
49
+ # If the server adds a `X-More-Events-Available: true` header to the response,
50
+ #a new request will be executed immediately and this wait time will be ignored.
51
+ config :interval, :validate => :number, :default => 10
52
+
53
+ # If desired, meta-data about the HTTP Response can be included in the event.
54
+ # This is the name of the object that http response meta-data (headers, response code, etc) will be added to.
55
+ config :response_object_name, :validate => :string, :default => "http_response"
56
+
57
+ # Should the http response headers be added to the `response_object_name`?
58
+ #
59
+ # If true, there will be a "headers" object with all the response headers/values.
60
+ config :include_response_headers, :validate => :boolean, :default => false
61
+
62
+ # Should the http response code be added to the `response_object_name`?
63
+ #
64
+ # If true, there will be a "code" with the response code.
65
+ config :include_response_code, :validate => :boolean, :default => true
66
+
67
+ # Should the http request elapsed time be added to the `response_object_name`?
68
+ #
69
+ # If true, there will be a "took_secs" with the time it took for the http request to complete
70
+ config :include_http_request_time, :validate => :boolean, :default => true
71
+
72
+ # The server should return this HTTP response code to indicate no messages were found/retuned.
73
+ # When this status code is seen, the message body will not be processed or added to the Logstash input queue.
74
+ # Defaults to 204 (No Content)
75
+ config :no_messages_response_code, :validate => :number, :default => 204
76
+
77
+ public
78
+ def register
79
+ @uri = URI(@url)
80
+ end # def register
84
81
 
85
- @codec.decode(response.body) do |event|
86
- event[@response_object_name] = {}
87
- if @include_response_headers
88
- event[@response_object_name]["headers"] = response
89
- end
90
- if @include_response_code
91
- event[@response_object_name]["code"] = response.code
92
- end
93
- if @include_http_request_time
94
- event[@response_object_name]["took_secs"] = http_elapsed
95
- end
96
- decorate(event)
97
- queue_start = Time.now
98
- queue << event
99
- queue_elapsed = Time.now - queue_start
100
- queue_times.push queue_elapsed
101
- end
102
- ensure
103
- sleep(@interval)
104
- end
105
- end #interval loop
106
- end #HTTP keepalive
107
- end # def run
108
-
109
- def arr_avg(arr,cutoff,precision)
110
- if(arr.size == 0)
111
- return 0
112
- end
113
- if(arr.size > cutoff)
114
- arr.drop(arr.length-cutoff) #removes from beginning of array
115
- end
116
- #get the average
117
- (arr.inject{ |sum, el| sum + el }.to_f / arr.size).round(precision)
118
- end
82
+ def run(queue)
83
+ begin
84
+ num_events = 0
85
+ queue_time = nil #track an average of how long it's taking to queue events into logstash
86
+ codec_parse_time = nil
87
+ http_request_time = nil
88
+ batch_id = nil
89
+
90
+ @logger.debug("Initializing http connection")
91
+ #Net::HTTP.start creates a connection to the HTTP server and keeps it alive for the duration. Throws an EOFError when connection breaks.
92
+ Net::HTTP.start @uri.host, @uri.port, :use_ssl => @uri.scheme == 'https' do |http|
93
+ while true
94
+ http_start = Time.now
95
+ sleepIval = @interval
96
+ @logger.debug("Executing http get request: " + @uri.host + @uri.request_uri)
97
+ request = Net::HTTP::Get.new(@uri.request_uri)
98
+
99
+ #include stats from processing the previous batch as custom headers
100
+ if num_events > 0 #0 if there wasn't anything processed in the last batch
101
+ #send the times/stats (from the previous batch) as custom headers to the server
102
+ request ["X-Logstash-Num-Events"] = num_events
103
+ request ["X-Logstash-Http-Request-Total-Secs"] = http_request_time
104
+ request ["X-Logstash-Codec-Parse-Total-Secs"] = codec_parse_time
105
+ request ["X-Logstash-Queue-Total-Secs"] = queue_time
106
+ end
107
+ if !batch_id.nil?
108
+ request ["X-Successful-Batch-Ids"] = batch_id #send the previous successfully processed batch id so the server knows those messages were all processed by logstash
109
+ end
110
+ response = http.request request # execute GET request and return Net::HTTPResponse object
111
+ if response["X-More-Events-Available"] == "true"
112
+ sleepIval = 0 #don't wait if the server indicated there are more events ready now
113
+ end
114
+ batch_id = response["X-Messages-Batch-Id"] #save the batch id of the event in this response body, so it can be ack'd in the next http request
115
+ http_request_time = Time.now - http_start
116
+
117
+
118
+ #check if events were returned and process them using the defined @codec
119
+ num_events = 0
120
+ if response.code.to_i == @no_messages_response_code
121
+ #HTTP Response code indicated there are no messages. Skip processing response and sleep.
122
+ batch_id = nil
123
+ elsif response.code.to_i != 200
124
+ @logger.warn("Received non-200 response code: #{response.code}. Will not process this response.")
125
+ elsif response.body.nil?
126
+ @logger.warn("Received normal #{response.code} response code, but the response body is empty. Will not process this response.")
127
+ else #normal HTTP 200 OK response body -- process this response
128
+ codec_start = Time.now
129
+ events = Array.new #will store the decoded & decorated event(s) from the message body in this temporary in-memory array
130
+ @codec.decode(response.body) do |event|
131
+ event[@response_object_name] = {}
132
+ if @include_response_headers
133
+ event[@response_object_name]["headers"] = response
134
+ end
135
+ if @include_response_code
136
+ event[@response_object_name]["code"] = response.code.to_i
137
+ end
138
+ if @include_http_request_time
139
+ event[@response_object_name]["took_secs"] = http_request_time
140
+ end
141
+ decorate(event)
142
+ events.push(event)
143
+ end #end decoding events in response body
144
+ num_events = events.length
145
+ codec_parse_time = Time.now - codec_start
146
+
147
+ #now queue the events to Logstash's bounded input queue and time how long it takes
148
+ queue_start = Time.now
149
+ events.each{ |event|
150
+ #@logger.info("Queueing event")
151
+ queue << event #blocks if queue is full
152
+ #@logger.info("Event has been queued")
153
+ }
154
+ queue_time = Time.now - queue_start
155
+ @logger.info("HTTP GET request processed #{events.length} events. http_request_secs=#{http_request_time}, codec_parse_secs=#{codec_parse_time}, queueing_secs=#{queue_time}, queue_size=#{queue.length}/#{queue.max}, queue_num_threads_waiting=#{queue.num_waiting}. Server=" + response["Server"] + ". Request=" + @uri.host + @uri.request_uri)
156
+ end #end normal processing of HTTP response body
157
+
158
+ #TODO: should we provide an option to post back (ack) immetiately once we sucecssfully queue all the messages in the response
159
+
160
+ if(sleepIval > 0)
161
+ @logger.debug("Waiting #{sleepIval} seconds before next http request")
162
+ sleep(sleepIval)
163
+ end
164
+ end #while loop
165
+ end #HTTP keepalive
166
+ rescue => x
167
+ if x.instance_of?(EOFError)
168
+ #this seems to happen if the http keepalive times out and server disconnects
169
+ @logger.warn("HTTP Connection has closed (EOFError), will reset http client connection and try again.", :exception => x)
170
+ sleep(1)
171
+ else
172
+ @logger.warn("Error occurred, resetting http client connection and trying again in 10 seconds.", :exception => x)
173
+ @logger.warn(x.backtrace)
174
+ sleep(10) #todo: configurable sleep time for errors?
175
+ end
176
+ retry
177
+ end #end begin outside of the loop
178
+ @logger.warn("Unexpected: HTTP client has ended")
179
+ end # def run
119
180
  end # class LogStash::Inputs::Example
data/logstash-input-httpclient.gemspec CHANGED
@@ -1,6 +1,6 @@
1
1
  Gem::Specification.new do |s|
2
2
  s.name = 'logstash-input-httpclient'
3
- s.version = '0.2.0'
3
+ s.version = '0.9.0'
4
4
  s.licenses = ['Apache License (2.0)']
5
5
  s.summary = "This input queries a http url at a regular interval and sends the responses to logstash."
6
6
  s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-input-httpclient
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.0
4
+ version: 0.9.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - bradvido
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2015-06-08 00:00:00.000000000 Z
11
+ date: 2015-06-22 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: logstash-core