sensu 1.0.4 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 7340072ff38575eed6bc61c3ebdf267e6256b43e
-  data.tar.gz: 3abac29f261e891ddc35c8999dc3bd44c7bdc4dd
+  metadata.gz: ed1314dc2d888635865717c37c4e71d77da17b28
+  data.tar.gz: 5f90de786a5691adf798879beca63a4d11732f38
 SHA512:
-  metadata.gz: 98d1abcbd269002e00bcc0e78753ecdcca406771f945aa589d330fcea5acfabde81ce87eea3f359839c3b7e4a140b157c03d2b37a19716257e53f387e0d0983d
-  data.tar.gz: 3d70a5834d1e8d8fadd6893f53bebad799a54a235a24c5d77dff62899957a74334bc8d65dbecac854cd8cf8b26e195ecf470b5c2fa2dfc44bc81132ab7b4aa50
+  metadata.gz: 6ad6541d3bba0982e9b0f397349f29889fc597a294ffdaad6e7dba49aa8af8830745d30a45061adbb67385a21ec2248ca8dc32640b73700f8278671280edb021
+  data.tar.gz: 60f3e3a064bcdcc8255ba3f0a52d4798ddbbf2a67b9588048b43e1b648ed1ebfe4d0cd747947df4980ea9616003e006937b9c6d99647e27db1ae3970667722e7

data/CHANGELOG.md

@@ -1,3 +1,61 @@
+## 1.1.0 - 2017-09-27
+
+### Features
+
+Check hooks, commands run by the Sensu client in response to the result of
+the check command execution. The Sensu client will execute the appropriate
+configured hook command, depending on the check execution status (e.g. 1).
+Valid hook names include (in order of precedence): "1"-"255", "ok",
+"warning", "critical", "unknown", and "non-zero". The check hook command
+output, status, executed timestamp, and duration are captured and
+published in the check result. Check hook commands can optionally receive
+JSON serialized Sensu client and check definition data via STDIN.
+
+Check STDIN. A boolean check definition attribute, `"stdin"`, when set to
+`true` instructs the Sensu client to write JSON serialized Sensu client
+and check definition data to the check command process STDIN. This
+attribute cannot be used with existing Sensu check plugins, nor Nagios
+plugins etc, as the Sensu client will wait indefinitely for the check
+process to read and close STDIN.
+
+Splayed proxy check request publishing. Users can now splay proxy check
+requests (optional), evenly, over a window of time, determined by the
+check interval and a configurable splay coverage percentage. For
+example, if a check has an interval of 60s and a configured splay
+coverage of 90%, its proxy check requests would be splayed evenly over a
+time window of 60s * 90%, 54s, leaving 6s for the last proxy check
+execution before the the next round of proxy check requests for the same
+check. Proxy check request splayed publishing can be configured with two
+new check definition attributes, within the `proxy_requests` scope,
+`splay` (boolean) to enable it and `splay_coverage` (integer percentage,
+defaults to `90`).
+
+Configurable check output truncation (for storage in Redis). Check output
+truncation can be manually enabled/disabled with the check definition
+attribute "truncate_output", e.g.`"truncate_output": false`. The output
+truncation length can be configured with the check definition attribute
+"truncate_output_length", e.g. `"truncate_output_length": 1024`. Check
+output truncation is still enabled by default for metric checks, with
+`"type": "metric"`.
+
+Sensu client HTTP socket basic authentication can how be applied to all
+endpoints (not just `/settings`), via the client definition http_socket
+attribute "protect_all_endpoints", e.g. `"protect_all_endpoints": true`.
+
+### Other
+
+Improved check TTL monitoring performance.
+
+The Sensu extension run log event log level is now set to debug (instead
+of info) when the run output is empty and the status is 0.
+
+### Fixes
+
+Added initial timestamp to proxy client definitions. The Uchiwa and Sensu
+dashboards will no longer display "Invalid Date".
+
+Deleting check history when deleting an associated check result.
+
 ## 1.0.3 - 2017-08-25
 
 ### Fixes

data/README.md

@@ -1,8 +1,8 @@
 ![sensu](https://raw.github.com/sensu/sensu/master/sensu-logo.png)
 
-[![Build Status](https://img.shields.io/travis/sensu/sensu.png)](https://travis-ci.org/sensu/sensu)
+[![Build Status](https://img.shields.io/travis/sensu/sensu.svg)](https://travis-ci.org/sensu/sensu)
 [![Gem Version](https://img.shields.io/gem/v/sensu.svg)](https://github.com/sensu/sensu/blob/master/CHANGELOG.md)
-![MIT Licensed](https://img.shields.io/github/license/sensu/sensu.svg)
+[![MIT Licensed](https://img.shields.io/github/license/sensu/sensu.svg)](https://raw.githubusercontent.com/sensu/sensu/master/MIT-LICENSE.txt)
 
 # Sensu
 
@@ -27,7 +27,7 @@ on configuring and operating Sensu.
 ## Getting Help
 
 If you have questions not covered by the documentation, the Sensu community
-is here to help. Please check out our [chat][4] on Freenode IRC, or the
+is here to help. Please check out our [chat][4] on Slack, or the
 [sensu-users][5] discussion list.
 
 Commercial support is also available. See the [support section of our website][6] for more detail.
@@ -45,7 +45,7 @@ Sensu Core is released under the [MIT license][8].
 [1]: https://sensuapp.org/enterprise
 [2]: https://sensuapp.org/docs/latest/platforms/
 [3]: http://sensuapp.org/docs/latest/overview
-[4]: http://webchat.freenode.net/?channels=%23sensu
+[4]: https://slack.sensu.io/
 [5]: http://groups.google.com/group/sensu-users
 [6]: https://sensuapp.org/support
 [7]: https://github.com/sensu/sensu/blob/master/CONTRIBUTING.md

data/lib/sensu/api/http_handler.rb

@@ -1,4 +1,3 @@
-require "sensu/utilities"
 require "sensu/api/routes"
 require "sensu/api/utilities/filter_response_content"
 
@@ -22,11 +21,9 @@ module Sensu
       # @result [Hash]
       def request_details
         return @request_details if @request_details
-        @request_id = @http.fetch(:x_request_id, random_uuid)
         @request_start_time = Time.now.to_f
         _, remote_address = Socket.unpack_sockaddr_in(get_peername)
         @request_details = {
-          :request_id => @request_id,
           :remote_address => remote_address,
           :user_agent => @http[:user_agent],
           :method => @http_request_method,
@@ -47,9 +44,7 @@ module Sensu
       end
 
       # Log the HTTP response. This method calculates the
-      # request/response time. The debug log level is used for the
-      # response body log event, as it is generally very verbose and
-      # unnecessary in most cases.
+      # request/response time.
       def log_response
         @logger.info("api response", {
           :request => request_details,
@@ -57,12 +52,6 @@ module Sensu
           :content_length => @response.content.to_s.bytesize,
           :time => (Time.now.to_f - @request_start_time).round(3)
         })
-        @logger.debug("api response body", {
-          :request => {
-            :request_id => @request_id
-          },
-          :content => @response.content
-        })
       end
 
       # Parse the HTTP request URI using a regular expression,
@@ -159,13 +148,6 @@ module Sensu
         end
       end
 
-      # Set the HTTP response headers, including the request ID and
-      # cors headers (via `set_cores_headers()`).
-      def set_headers
-        @response.headers["X-Request-ID"] = @request_id
-        set_cors_headers
-      end
-
       # Paginate the provided items. This method uses two HTTP query
       # parameters to determine how to paginate the items, `limit` and
       # `offset`. The parameter `limit` specifies how many items are
@@ -410,7 +392,7 @@ module Sensu
         log_request
         parse_parameters
         create_response
-        set_headers
+        set_cors_headers
         if authorized?
           if connected?
             route_request

data/lib/sensu/api/routes/results.rb

@@ -105,7 +105,9 @@ module Sensu
             if result_exists
               @redis.srem("result:#{client_name}", check_name) do
                 @redis.del(result_key) do
-                  no_content!
+                  @redis.del("history:#{client_name}:#{check_name}") do
+                    no_content!
+                  end
                 end
               end
             else

data/lib/sensu/client/http_socket.rb

@@ -87,6 +87,14 @@ module Sensu
         false
       end
 
+      def unauthorized_response
+        @logger.warn("http socket refusing to serve unauthorized request")
+        @response.headers["WWW-Authenticate"] = 'Basic realm="Sensu Client Restricted Area"'
+        send_response(401, "Unauthorized", {
+          :response => "You must be authenticated using your http_options user and password settings"
+        })
+      end
+
       def send_response(status, status_string, content)
         @logger.debug("http socket sending response", {
           :status => status,
@@ -100,6 +108,9 @@ module Sensu
       end
 
       def process_request_info
+        if @settings[:client][:http_socket][:protect_all_endpoints]
+          return unauthorized_response unless authorized?
+        end
         transport_info do |info|
           send_response(200, "OK", {
             :sensu => {
@@ -111,6 +122,9 @@ module Sensu
       end
 
       def process_request_results
+        if @settings[:client][:http_socket][:protect_all_endpoints]
+          return unauthorized_response unless authorized?
+        end
         if @http[:content_type] and @http[:content_type] == "application/json" and @http_content
           begin
             check = Sensu::JSON::load(@http_content)
@@ -125,19 +139,12 @@ module Sensu
       end
 
       def process_request_settings
-        if authorized?
-          @logger.info("http socket responding to request for configuration settings")
-          if @http_query_string and @http_query_string.downcase.include?("redacted=false")
-            send_response(200, "OK", @settings.to_hash)
-          else
-            send_response(200, "OK", redact_sensitive(@settings.to_hash))
-          end
+        return unauthorized_response unless authorized?
+        @logger.info("http socket responding to request for configuration settings")
+        if @http_query_string and @http_query_string.downcase.include?("redacted=false")
+          send_response(200, "OK", @settings.to_hash)
         else
-          @logger.warn("http socket refusing to serve unauthorized settings request")
-          @response.headers["WWW-Authenticate"] = 'Basic realm="Sensu Client Restricted Area"'
-          send_response(401, "Unauthorized", {
-            :response => "You must be authenticated using your http_options user and password settings"
-          })
+          send_response(200, "OK", redact_sensitive(@settings.to_hash))
         end
       end
 
@@ -181,7 +188,7 @@ module Sensu
               :http_request_uri => @http_request_uri
             })
             send_response(405, "Method Not Allowed", {
-              :response => "Valid methods for this endpoint: #{reqdef['methods'].keys}"
+              :response => "Valid methods for this endpoint: #{endpoint['methods'].keys}"
             })
           end
         else

data/lib/sensu/client/process.rb

@@ -110,13 +110,66 @@ module Sensu
         [check[:source], check[:name]].compact.join(":")
       end
 
+      # Execute a check hook, capturing its output (STDOUT/ERR),
+      # exit status code, executed timestamp, and duration. This
+      # method determines which hook command to run by inspecting the
+      # check execution status. Check hook command tokens are
+      # substituted with the associated client attribute values, via
+      # `substitute_tokens()`. If there are unmatched check attribute
+      # value tokens, the check hook will not be executed, instead the
+      # hook command output will be set to report the unmatched
+      # tokens. Hook commands may expect/read and utilize JSON
+      # serialized Sensu client and check data via STDIN, if the hook
+      # definition includes `"stdin": true` (default is `false`). A
+      # hook may have a configured execution timeout, e.g. `"timeout":
+      # 30`, if one is not specified, the timeout defaults to 60
+      # seconds.
+      #
+      # @param check [Hash]
+      # @yield [check] callback/block called after executing the check
+      #   hook (if any).
+      def execute_check_hook(check)
+        @logger.debug("attempting to execute check hook", :check => check)
+        severity = SEVERITIES[check[:status]] || "unknown"
+        hook = check[:hooks][check[:status].to_s] || check[:hooks][severity.to_sym]
+        if hook.nil? && check[:status] != 0
+          hook = check[:hooks]["non-zero"]
+        end
+        if hook
+          command, unmatched_tokens = substitute_tokens(hook[:command].dup, @settings[:client])
+          started = Time.now.to_f
+          hook[:executed] = started.to_i
+          if unmatched_tokens.empty?
+            options = {:timeout => hook.fetch(:timeout, 60)}
+            if hook[:stdin]
+              options[:data] = Sensu::JSON.dump({
+                :client => @settings[:client],
+                :check => check
+              })
+            end
+            Spawn.process(command, options) do |output, status|
+              hook[:duration] = ("%.3f" % (Time.now.to_f - started)).to_f
+              hook[:output] = output
+              hook[:status] = status
+              yield(check)
+            end
+          else
+            hook[:output] = "Unmatched client token(s): " + unmatched_tokens.join(", ")
+            hook[:status] = 3
+            yield(check)
+          end
+        else
+          yield(check)
+        end
+      end
+
       # Execute a check command, capturing its output (STDOUT/ERR),
       # exit status code, execution duration, timestamp, and publish
       # the result. This method guards against multiple executions for
       # the same check. Check attribute value tokens are substituted
       # with the associated client attribute values, via
-      # `object_substitute_tokens()`. The original check command is
-      # always published, to guard against publishing
+      # `object_substitute_tokens()`. The original check command and
+      # hooks are always published, to guard against publishing
       # sensitive/redacted client attribute values. If there are
       # unmatched check attribute value tokens, the check will not be
       # executed, instead a check result will be published reporting
@@ -129,16 +182,31 @@ module Sensu
         unless @checks_in_progress.include?(in_progress_key)
           @checks_in_progress << in_progress_key
           substituted, unmatched_tokens = object_substitute_tokens(check.dup, @settings[:client])
-          check = substituted.merge(:command => check[:command])
+          check = substituted.merge(:command => check[:command], :hooks => check[:hooks])
+          check.delete(:hooks) if check[:hooks].nil?
           started = Time.now.to_f
           check[:executed] = started.to_i
           if unmatched_tokens.empty?
-            Spawn.process(substituted[:command], :timeout => check[:timeout]) do |output, status|
+            options = {:timeout => check[:timeout]}
+            if check[:stdin]
+              options[:data] = Sensu::JSON.dump({
+                :client => @settings[:client],
+                :check => check
+              })
+            end
+            Spawn.process(substituted[:command], options) do |output, status|
               check[:duration] = ("%.3f" % (Time.now.to_f - started)).to_f
               check[:output] = output
               check[:status] = status
-              publish_check_result(check)
-              @checks_in_progress.delete(in_progress_key)
+              if check[:hooks] && !check[:hooks].empty?
+                execute_check_hook(check) do |check|
+                  publish_check_result(check)
+                  @checks_in_progress.delete(in_progress_key)
+                end
+              else
+                publish_check_result(check)
+                @checks_in_progress.delete(in_progress_key)
+              end
             end
           else
             check[:output] = "Unmatched client token(s): " + unmatched_tokens.join(", ")
@@ -178,8 +246,15 @@ module Sensu
             check[:duration] = ("%.3f" % (Time.now.to_f - started)).to_f
             check[:output] = output
             check[:status] = status
-            publish_check_result(check)
-            @checks_in_progress.delete(in_progress_key)
+            if check[:hooks] && !check[:hooks].empty?
+              execute_check_hook(check) do |check|
+                publish_check_result(check)
+                @checks_in_progress.delete(in_progress_key)
+              end
+            else
+              publish_check_result(check)
+              @checks_in_progress.delete(in_progress_key)
+            end
           end
         else
           @logger.warn("previous check extension execution in progress", :check => check)

data/lib/sensu/constants.rb

@@ -1,7 +1,7 @@
 module Sensu
   unless defined?(Sensu::VERSION)
     # Sensu release version.
-    VERSION = "1.0.4".freeze
+    VERSION = "1.1.0".freeze
 
     # Sensu release information.
     RELEASE_INFO = {

data/lib/sensu/daemon.rb

@@ -4,7 +4,7 @@ gem "eventmachine", "1.2.5"
 
 gem "sensu-json", "2.1.0"
 gem "sensu-logger", "1.2.1"
-gem "sensu-settings", "10.3.0"
+gem "sensu-settings", "10.9.0"
 gem "sensu-extension", "1.5.1"
 gem "sensu-extensions", "1.9.0"
 gem "sensu-transport", "7.0.2"

data/lib/sensu/server/handle.rb

@@ -126,7 +126,8 @@ module Sensu
       # @param event_id [String] event UUID
       def handler_extension(handler, event_data, event_id)
         handler.safe_run(event_data) do |output, status|
-          @logger.info("handler extension output", {
+          log_level = (output.empty? && status.zero?) ? :debug : :info
+          @logger.send(log_level, "handler extension output", {
             :extension => handler.definition,
             :event => { :id => event_id },
             :output => output,

data/lib/sensu/server/process.rb

@@ -320,16 +320,17 @@ module Sensu
         end
       end
 
-      # Truncate check output. For metric checks, (`"type":
-      # "metric"`), check output is truncated to a single line and a
-      # maximum of 255 characters. Check output is currently left
-      # unmodified for standard checks.
+      # Truncate check output. Metric checks (`"type": "metric"`), or
+      # checks with `"truncate_output": true`, have their output
+      # truncated to a single line and a maximum character length of
+      # 255 by default. The maximum character length can be change by
+      # the `"truncate_output_length"` check definition attribute.
       #
       # @param check [Hash]
       # @return [Hash] check with truncated output.
       def truncate_check_output(check)
-        case check[:type]
-        when METRIC_CHECK_TYPE
+        if check[:truncate_output] ||
+           (check[:type] == METRIC_CHECK_TYPE && check[:truncate_output] != false)
           begin
             output_lines = check[:output].split("\n")
           rescue ArgumentError
@@ -341,8 +342,9 @@ module Sensu
             output_lines = utf8_output.split("\n")
           end
           output = output_lines.first || check[:output]
-          if output_lines.length > 1 || output.length > 255
-            output = output[0..255] + "\n..."
+          truncate_output_length = check.fetch(:truncate_output_length, 255)
+          if output_lines.length > 1 || output.length > truncate_output_length
+            output = output[0..truncate_output_length] + "\n..."
           end
           check.merge(:output => output)
         else
@@ -603,7 +605,8 @@ module Sensu
           :address => "unknown",
           :subscriptions => ["client:#{name}"],
           :keepalives => false,
-          :version => VERSION
+          :version => VERSION,
+          :timestamp => Time.now.to_i
         }
       end
 
@@ -792,45 +795,129 @@ module Sensu
         end
       end
 
+      # Determine and return clients from the registry that match a
+      # set of attributes.
+      #
+      # @param clients [Array] of client names.
+      # @param attributes [Hash]
+      # @yield [Array] callback/block called after determining the
+      #   matching clients, returning them as a block parameter.
+      def determine_matching_clients(clients, attributes)
+        client_keys = clients.map { |client_name| "client:#{client_name}" }
+        @redis.mget(*client_keys) do |client_json_objects|
+          matching_clients = []
+          client_json_objects.each do |client_json|
+            unless client_json.nil?
+              client = Sensu::JSON.load(client_json)
+              if attributes_match?(client, attributes)
+                matching_clients << client
+              end
+            end
+          end
+          yield(matching_clients)
+        end
+      end
+
+      # Publish a proxy check request for a client. This method
+      # substitutes client tokens in the check definition prior to
+      # publish the check request. If there are unmatched client
+      # tokens, a warning is logged, and a check request is not
+      # published.
+      #
+      # @param client [Hash] definition.
+      # @param check [Hash] definition.
+      def publish_proxy_check_request(client, check)
+        @logger.debug("creating a proxy check request", {
+          :client => client,
+          :check => check
+        })
+        proxy_check, unmatched_tokens = object_substitute_tokens(check.dup, client)
+        if unmatched_tokens.empty?
+          proxy_check[:source] ||= client[:name]
+          publish_check_request(proxy_check)
+        else
+          @logger.warn("failed to publish a proxy check request", {
+            :reason => "unmatched client tokens",
+            :unmatched_tokens => unmatched_tokens,
+            :client => client,
+            :check => check
+          })
+        end
+      end
+
+      # Publish proxy check requests for one or more clients. This
+      # method can optionally splay proxy check requests, evenly, over
+      # a period of time, determined by the check interval and a
+      # configurable splay coverage percentage. For example, splay
+      # proxy check requests over 60s * 90%, 54s, leaving 6s for the
+      # last proxy check execution before the the next round of proxy
+      # check requests for the same check. The
+      # `publish_proxy_check_request() method is used to publish the
+      # proxy check requests.
+      #
+      # @param clients [Array] of client definitions.
+      # @param check [Hash] definition.
+      def publish_proxy_check_requests(clients, check)
+        client_count = clients.length
+        splay = 0
+        if check[:proxy_requests][:splay]
+          interval = check[:interval]
+          if check[:cron]
+            interval = determine_check_cron_time(check)
+          end
+          unless interval.nil?
+            splay_coverage = check[:proxy_requests].fetch(:splay_coverage, 90)
+            splay = interval * (splay_coverage / 100.0) / client_count
+          end
+        end
+        splay_timer = 0
+        clients.each do |client|
+          unless splay == 0
+            EM::Timer.new(splay_timer) do
+              publish_proxy_check_request(client, check)
+            end
+            splay_timer += splay
+          else
+            publish_proxy_check_request(client, check)
+          end
+        end
+      end
+
       # Create and publish one or more proxy check requests. This
       # method iterates through the Sensu client registry for clients
       # that matched provided proxy request client attributes. A proxy
       # check request is created for each client in the registry that
       # matches the proxy request client attributes. Proxy check
       # requests have their client tokens subsituted by the associated
-      # client attributes values. The check requests are published to
-      # the Transport via `publish_check_request()`.
+      # client attributes values. The `determine_matching_clients()`
+      # method is used to fetch and inspect each slide of clients from
+      # the registry, returning those that match the configured proxy
+      # request client attributes. A relatively small clients slice
+      # size (20) is used to reduce the number of clients inspected
+      # within a single tick of the EM reactor. The
+      # `publish_proxy_check_requests()` method is used to iterate
+      # through the matching Sensu clients, creating their own unique
+      # proxy check request, substituting client tokens, and then
+      # publishing them to the targetted subscriptions.
       #
       # @param check [Hash] definition.
-      def publish_proxy_check_requests(check)
+      def create_proxy_check_requests(check)
         client_attributes = check[:proxy_requests][:client_attributes]
         unless client_attributes.empty?
           @redis.smembers("clients") do |clients|
-            clients.each do |client_name|
-              @redis.get("client:#{client_name}") do |client_json|
-                unless client_json.nil?
-                  client = Sensu::JSON.load(client_json)
-                  if attributes_match?(client, client_attributes)
-                    @logger.debug("creating a proxy check request", {
-                      :client => client,
-                      :check => check
-                    })
-                    proxy_check, unmatched_tokens = object_substitute_tokens(check.dup, client)
-                    if unmatched_tokens.empty?
-                      proxy_check[:source] ||= client[:name]
-                      publish_check_request(proxy_check)
-                    else
-                      @logger.warn("failed to publish a proxy check request", {
-                        :reason => "unmatched client tokens",
-                        :unmatched_tokens => unmatched_tokens,
-                        :client => client,
-                        :check => check
-                      })
-                    end
-                  end
+            client_count = clients.length
+            proxy_check_requests = Proc.new do |matching_clients, slice_start, slice_size|
+              unless slice_start > client_count - 1
+                clients_slice = clients.slice(slice_start..slice_size)
+                determine_matching_clients(clients_slice, client_attributes) do |additional_clients|
+                  matching_clients += additional_clients
+                  proxy_check_requests.call(matching_clients, slice_start + 20, slice_size + 20)
                 end
+              else
+                publish_proxy_check_requests(matching_clients, check)
               end
             end
+            proxy_check_requests.call([], 0, 19)
           end
         end
       end
@@ -846,7 +933,7 @@ module Sensu
         Proc.new do
           unless check_subdued?(check)
             if check[:proxy_requests]
-              publish_proxy_check_requests(check)
+              create_proxy_check_requests(check)
             else
               publish_check_request(check)
             end
@@ -1064,46 +1151,83 @@ module Sensu
         end
       end
 
+      # Create check TTL results. This method will retrieve check
+      # results from the registry and determine the time since their
+      # last check execution (in seconds). If the time since last
+      # execution is equal to or greater than the defined check TTL, a
+      # warning check result is published with the appropriate check
+      # output.
+      #
+      # @param ttl_keys [Array] of TTL keys.
+      # @param interval [Integer] to use for the check TTL result
+      #   interval.
+      # @yield [] callback/block called after the check TTL results
+      #   have been created.
+      def create_check_ttl_results(ttl_keys, interval=30)
+        result_keys = ttl_keys.map { |ttl_key| "result:#{ttl_key}" }
+        @redis.mget(*result_keys) do |result_json_objects|
+          result_json_objects.each_with_index do |result_json, index|
+            unless result_json.nil?
+              check = Sensu::JSON.load(result_json)
+              next unless check[:ttl] && check[:executed] && !check[:force_resolve]
+              time_since_last_execution = Time.now.to_i - check[:executed]
+              if time_since_last_execution >= check[:ttl]
+                client_name = ttl_keys[index].split(":").first
+                keepalive_event_exists?(client_name) do |event_exists|
+                  unless event_exists
+                    check[:output] = "Last check execution was "
+                    check[:output] << "#{time_since_last_execution} seconds ago"
+                    check[:status] = check[:ttl_status] || 1
+                    check[:interval] = interval
+                    publish_check_result(client_name, check)
+                  end
+                end
+              end
+            else
+              @redis.srem("ttl", result_key)
+            end
+          end
+          yield
+        end
+      end
+
       # Determine stale check results, those that have not executed in
       # a specified amount of time (check TTL). This method iterates
       # through stored check results that have a defined TTL value (in
-      # seconds). The time since last check execution (in seconds) is
-      # calculated for each check result. If the time since last
-      # execution is equal to or greater than the check TTL, a warning
-      # check result is published with the appropriate check output.
-      def determine_stale_check_results(interval = 30)
-        @logger.info("determining stale check results")
-        @redis.smembers("ttl") do |result_keys|
-          result_keys.each do |result_key|
-            @redis.get("result:#{result_key}") do |result_json|
-              unless result_json.nil?
-                check = Sensu::JSON.load(result_json)
-                next unless check[:ttl] && check[:executed] && !check[:force_resolve]
-                time_since_last_execution = Time.now.to_i - check[:executed]
-                if time_since_last_execution >= check[:ttl]
-                  client_name = result_key.split(":").first
-                  keepalive_event_exists?(client_name) do |event_exists|
-                    unless event_exists
-                      check[:output] = "Last check execution was "
-                      check[:output] << "#{time_since_last_execution} seconds ago"
-                      check[:status] = check[:ttl_status] || 1
-                      check[:interval] = interval
-                      publish_check_result(client_name, check)
-                    end
-                  end
-                end
-              else
-                @redis.srem("ttl", result_key)
+      # seconds). The `create_check_ttl_results()` method is used to
+      # inspect each check result, calculating their time since last
+      # check execution (in seconds). If the time since last execution
+      # is equal to or greater than the check TTL, a warning check
+      # result is published with the appropriate check output. A
+      # relatively small check results slice size (20) is used to
+      # reduce the number of check results inspected within a single
+      # tick of the EM reactor.
+      #
+      # @param interval [Integer] to use for the check TTL result
+      #   interval.
+      def determine_stale_check_results(interval=30)
+        @logger.info("determining stale check results (ttl)")
+        @redis.smembers("ttl") do |ttl_keys|
+          ttl_key_count = ttl_keys.length
+          ttl_check_results = Proc.new do |slice_start, slice_size|
+            unless slice_start > ttl_key_count - 1
+              ttl_keys_slice = ttl_keys.slice(slice_start..slice_size)
+              create_check_ttl_results(ttl_keys_slice, interval) do
+                ttl_check_results.call(slice_start + 20, slice_size + 20)
               end
             end
           end
+          ttl_check_results.call(0, 19)
         end
       end
 
       # Set up the check result monitor, a periodic timer to run
       # `determine_stale_check_results()` every 30 seconds. The timer
       # is stored in the timers hash under `:tasks`.
-      def setup_check_result_monitor(interval = 30)
+      #
+      # @param interval [Integer] to use for the check TTL result
+      #   interval.
+      def setup_check_result_monitor(interval=30)
         @logger.debug("monitoring check results")
         @timers[:tasks][:check_result_monitor] << EM::PeriodicTimer.new(interval) do
           determine_stale_check_results(interval)

data/sensu.gemspec

@@ -15,7 +15,7 @@ Gem::Specification.new do |s|
   s.add_dependency "eventmachine", "1.2.5"
   s.add_dependency "sensu-json", "2.1.0"
   s.add_dependency "sensu-logger", "1.2.1"
-  s.add_dependency "sensu-settings", "10.3.0"
+  s.add_dependency "sensu-settings", "10.9.0"
   s.add_dependency "sensu-extension", "1.5.1"
   s.add_dependency "sensu-extensions", "1.9.0"
   s.add_dependency "sensu-transport", "7.0.2"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: sensu
 version: !ruby/object:Gem::Version
-  version: 1.0.4
+  version: 1.1.0
 platform: ruby
 authors:
 - Sean Porter
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2017-10-18 00:00:00.000000000 Z
+date: 2017-09-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: eventmachine
@@ -59,14 +59,14 @@ dependencies:
     requirements:
     - - '='
       - !ruby/object:Gem::Version
-        version: 10.3.0
+        version: 10.9.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '='
       - !ruby/object:Gem::Version
-        version: 10.3.0
+        version: 10.9.0
 - !ruby/object:Gem::Dependency
   name: sensu-extension
   requirement: !ruby/object:Gem::Requirement