app_status 0.1.1 → 1.0.0

data/README.md

@@ -3,6 +3,39 @@
 AppStatus is a Rails engine which makes it easy to expose application status
 data in a way easily consumed by Nagios or other monitoring packages.
 
+## Why?
+
+Defining health checks outside of your application (like in Nagios)
+has a few different problems.
+
+  1. The people who maintain nagios aren't necessarily
+     the same people who maintain the application.
+  1. Keeping the 2 systems in sync can be non-trivial with a fast-changing
+     application.
+  1. Failing to monitor new features, or monitoring the wrong things, leads
+     to a false sense of security.
+
+Instead, app_status lets you define your health checks right in the application
+itself and expose the results as a JSON service which is easy for Nagios
+to consume.
+
+The benefits basically come down to 1 major thing: Nagios doesn't need to know
+anything about your application. All Nagios needs is a 'healthy/not healthy'
+status report.
+
+This is good because:
+
+  1. As your app's feature set changes, you can deploy updated health checks
+     at the same time. No need for coordinated updates between the app and
+     the monitoring system.
+  1. Credentials for external services (like databases) can stay with your
+     app. Nagios doesn't need them.
+  1. You don't need nrpe to do local process checks. Your application can do
+     them for itself.
+  1. Your health checks can be testable methods just like all your other code.
+  1. You don't need to duplicate complex queries & other business logic over
+     to Nagios.
+
 ## Installation
 
 ### `Gemfile`
@@ -13,8 +46,6 @@ gem 'app_status'
 
 ### `config/routes.rb`
 
-Wire it up.
-
 ```ruby
 mount AppStatus::Engine, at: "/status"
 ```
@@ -30,19 +61,32 @@ This exposes the following URLs
 
 This is where you set up the checks which you want to be run when
 someone hits the URL above. Set up some calls which evaluate the health
-of your application and call `add` for each one.
+of your application and call `add_check` for each one.
+
+`add_check` expects a service name, plus a block to be evaluated to determine
+the health of that service. The block should return either a status value, or
+a 2-element array with status and some details.
 
 ```ruby
 AppStatus::CheckCollection.configure do |c|
-  value = some_service_check
-  c.add(:name => 'some_service', :status => :ok, :details => value)
+
+  c.add_check('some_service') do
+    details = do_something_to_check_your_service
+    status = (details != "FAIL") ? :ok : :critical
+    [status, details]
+  end
+
+  c.add_check('failing_service') do
+    :critical # you can return just a status if desired.
+  end
 end
 ```
 
-The checks that you set up here are not run when you configure them. They're
-run whenever someone hits the check URL.
+The details string should be concise. `app_status` does its best to provide
+readable output, and Nagios does its best to make this impossible to actually
+do well.
 
-Status values (in ascending order of seriousness)
+Valid status values (in ascending order of seriousness) are:
   - :ok
   - :warning
   - :critical
@@ -50,8 +94,9 @@ Status values (in ascending order of seriousness)
 
 These are set up to be compatible with Nagios.
 
-Details doesn't have to be a string. It can be anything which is serializable
-as JSON.
+Keep in mind that anyone who hits your status URL can cause your checks to run,
+so if they expose sensitive data or are a potential DOS vector you should
+probably protect them with some kind of authentication.
 
 ## Usage
 
@@ -62,33 +107,34 @@ Output will look something like this:
 {
   "status": "critical",
   "status_code": 2,
-  "run_time_ms": 52,
+  "ms": 52,
   "finished": "2013-10-03T21:28:10Z",
   "checks": {
     "some_service": {
       "status": "ok",
       "status_code": 0,
-      "details": "Looks good!"
+      "details": "Looks good!",
+      "ms": 30
     },
     "failing_service": {
       "status": "critical",
       "status_code": 2,
-      "details": "Oh noes!"
+      "details": "",
+      "ms": 20
     }
   }
 }
 ```
 
-The overall status will be the worst status which is actually observed in your
-individual checks.
+The overall status will be the worst value observed in your individual checks.
 
 ## Nagios Integration
 
-[bin/check_app_status.rb](https://github.com/alexdean/app_status/blob/master/bin/check_app_status.rb)
+[check_app_status.rb](check_app_status.rb)
 is a Nagios check script which can be used to monitor the output from `app_status`
 
 ```
-$ bin/check_app_status.rb --help
+$ ./check_app_status.rb --help
 Nagios check script for app_status. See https://github.com/alexdean/app_status
     -v, --verbose                    Output more information
     -V, --version                    Output version information
@@ -102,14 +148,18 @@ The script's exit status is derived from the overall status returned by the
 server. Individual detail items will be grouped by status for display.
 (Unknowns are displayed together, then criticals, then warnings, then OKs.)
 
-Sample output (using verbose mode)
+Sample output
 
 ```
-$ bin/check_app_status.rb --url http://localhost:3000/status -v
-2013-10-03T20:54:16-05:00  options: {:timeout=>10, :url=>"http://localhost:3000/status"}
-2013-10-03T20:54:16-05:00  timeout: 10s
-2013-10-03T20:54:16-05:00  response body: {"status":"warning","status_code":1,"run_time_ms":0,"finished":"2013-10-04T01:54:16Z","details":{"some_service":{"status":"ok","status_code":0,"details":"Looks good!"},"failing_service":{"status":"warning","status_code":1,"details":"Oh noes!"}}}
+$ ./check_app_status.rb --url http://localhost:3000/status
+
+CRIT failed_service
+--- failed_service: shit's on fire yo, 501ms
+
+WARN problematic_service
+--- problematic_service: not looking good, 2001ms
 
-WARN: failing_service:'Oh noes!'
-OK: some_service:'Looks good!'
+OK ok_process, ok_process_2
+--- ok_process: these are some details, 0ms
+--- ok_process_2: more details on another process, 0ms
 ```

data/app/models/app_status/check_collection.rb

@@ -4,7 +4,7 @@ module AppStatus
 
   class CheckCollection
 
-    @@config_proc = nil
+    @@checks = HashWithIndifferentAccess.new
 
     # Add checks here.
     #
@@ -24,11 +24,11 @@ module AppStatus
     #
     #   end
     def self.configure(&block)
-      @@config_proc = block
+      yield self
     end
 
     def self.clear_checks!
-      @@config_proc = nil
+      @@checks = HashWithIndifferentAccess.new
     end
 
     def initialize
@@ -39,7 +39,7 @@ module AppStatus
          unknown: 3
       }.freeze
 
-      @checks = HashWithIndifferentAccess.new
+      @check_results = HashWithIndifferentAccess.new
       @eval_finished = nil
       @eval_time = 0
     end
@@ -50,24 +50,15 @@ module AppStatus
     # example:
     #   value = some_service_check
     #   c.add(:name => 'some_service', :status => :ok, :details => value)
-    def add(options={})
-      raise ArgumentError, ":name option is required." if ! options[:name]
-      raise ArgumentError, ":status option is required." if ! options[:status]
-
-      name = options[:name].to_sym
-      status = options[:status].to_sym
-      details = options[:details].to_s
-
-      # blow up if someone sends us options we don't understand.
-      other_options = options.keys - [:name, :status, :details]
-      if other_options.size > 0
-        raise ArgumentError, "Unrecognized option(s) for '#{name}' check: #{other_options.join(',')}"
-      end
+    def self.add_check(name, &block)
+      raise ArgumentError, ":name option is required." if ! name
+      # raise ArgumentError, ":status option is required." if ! options[:status]
 
-      raise ArgumentError, "'#{status}' is not a valid status for check '#{name}'." if ! valid_status?(status)
-      raise ArgumentError, "Check name '#{name}' has already been added." if @checks.keys.include?(name)
+      name = name.to_sym
+      raise ArgumentError, "Check name '#{name}' has already been added." if @@checks.keys.include?(name.to_s)
+      raise ArgumentError, "No check defined for '#{name}'." if ! block_given?
 
-      @checks[name] = {status: status, status_code: @valid_status[status], details: details}
+      @@checks[name] = block
     end
 
     def valid_status?(status)
@@ -77,28 +68,47 @@ module AppStatus
     # run the checks added via configure
     # results of the checks are available via as_json
     def evaluate!
-      start = Time.now
-      @checks = {}
-      @@config_proc.call(self) if @@config_proc
+      eval_start = Time.now
+      @check_results = {}
+      @@checks.each do |name,proc|
+        check_start = Time.now
+        status, details = proc.call
+        check_time = (Time.now - check_start) * 1000
+
+        status = status.to_sym if status
+        details = details.to_s if details
+
+        if ! valid_status?(status)
+          details = "Check returned invalid status '#{status}'. #{details}".strip
+          status = :unknown
+        end
+        @check_results[name] = {
+          status: status,
+          status_code: @valid_status[status],
+          details: details,
+          ms: check_time.to_i
+        }
+      end
+
       @eval_finished = Time.now.utc
-      @eval_time = (Time.now - start) * 1000
+      @eval_time = (Time.now - eval_start) * 1000
     end
 
     def as_json
-      if @checks.size == 0
+      if @check_results.size == 0
         max_status = :unknown
         max_int = @valid_status[max_status]
       else
-        max_int = @checks.inject([]){ |memo,val| memo << val[1][:status_code]; memo}.max
+        max_int = @check_results.inject([]){ |memo,val| memo << val[1][:status_code]; memo}.max
         max_status = @valid_status.invert[max_int]
       end
 
       HashWithIndifferentAccess.new({
         status: max_status,
         status_code: max_int,
-        run_time_ms: @eval_time.to_i,
+        ms: @eval_time.to_i,
         finished: @eval_finished.iso8601,
-        checks: @checks
+        checks: @check_results
       })
     end
   end

data/lib/app_status/version.rb

@@ -1,3 +1,3 @@
 module AppStatus
-  VERSION = "0.1.1"
+  VERSION = "1.0.0"
 end

data/spec/controllers/status_controller_spec.rb

@@ -4,9 +4,8 @@ describe AppStatus::StatusController do
 
   describe "GET index" do
     before(:each) do
-      AppStatus::CheckCollection.configure do |c|
-        c.add(name: 'some_service', status: :ok, details: 'foo')
-      end
+      AppStatus::CheckCollection.clear_checks!
+      AppStatus::CheckCollection.add_check('some_service') {[:ok, 'foo']}
     end
 
     it "should render json" do

data/spec/dummy/config/environments/development.rb

@@ -23,11 +23,11 @@ Dummy::Application.configure do
   config.action_dispatch.best_standards_support = :builtin
 
   # Raise exception on mass assignment protection for Active Record models
-  config.active_record.mass_assignment_sanitizer = :strict
+  # config.active_record.mass_assignment_sanitizer = :strict
 
   # Log the query plan for queries taking more than this (works
   # with SQLite, MySQL, and PostgreSQL)
-  config.active_record.auto_explain_threshold_in_seconds = 0.5
+  # config.active_record.auto_explain_threshold_in_seconds = 0.5
 
   # Do not compress assets
   config.assets.compress = false

data/spec/dummy/log/test.log

@@ -41,3 +41,12 @@ Processing by AppStatus::StatusController#index as JSON
 Completed 200 OK in 1ms (Views: 0.3ms)
 Processing by AppStatus::StatusController#index as HTML
 Completed 200 OK in 18ms (Views: 17.4ms)
+Processing by AppStatus::StatusController#index as HTML
+  Rendered /Users/alex/Code/app_status/app/views/app_status/status/index.html.haml within layouts/app_status/application (0.2ms)
+Completed 200 OK in 23ms (Views: 22.3ms)
+Processing by AppStatus::StatusController#index as JSON
+Completed 200 OK in 0ms (Views: 0.2ms)
+Processing by AppStatus::StatusController#index as JSON
+Completed 200 OK in 1ms (Views: 0.3ms)
+Processing by AppStatus::StatusController#index as HTML
+Completed 200 OK in 16ms (Views: 16.0ms)

data/spec/models/check_collection_spec.rb

@@ -6,81 +6,98 @@ describe AppStatus::CheckCollection do
     AppStatus::CheckCollection.clear_checks!
   end
 
-  describe "add" do
+  describe "configure" do
 
-    describe "validations" do
-      it "should raise an error if :name is not supplied" do
-        AppStatus::CheckCollection.configure {|c| c.add status: :ok }
-        c = AppStatus::CheckCollection.new
-        expect {
-          c.evaluate!
-        }.to raise_error(ArgumentError, ":name option is required.")
+    it "should yield itself" do
+      AppStatus::CheckCollection.configure do |c|
+        c.should eq AppStatus::CheckCollection
       end
+    end
 
-      it "should raise an error if :status is not supplied" do
-        AppStatus::CheckCollection.configure {|c| c.add name: 'foo' }
-        c = AppStatus::CheckCollection.new
-        expect {
-          c.evaluate!
-        }.to raise_error(ArgumentError, ":status option is required.")
-      end
+  end
+
+  describe "clear_checks!" do
+    it "should remove all checks" do
+      c = AppStatus::CheckCollection.new
+      AppStatus::CheckCollection.add_check('test') { nil }
+      c.evaluate!
+      c.as_json['checks'].size.should eq 1
 
-      it "should raise an error if an unrecognized option is supplied" do
-        AppStatus::CheckCollection.configure {|c| c.add name: 'foo', status: :ok, glork: '', ping: '' }
-        c = AppStatus::CheckCollection.new
+      AppStatus::CheckCollection.clear_checks!
+      c.evaluate!
+      c.as_json['checks'].size.should eq 0
+    end
+  end
+
+  describe "add_check" do
+
+    describe "validations" do
+      it "should raise an error if name is not supplied" do
         expect {
-          c.evaluate!
-        }.to raise_error(ArgumentError, "Unrecognized option(s) for 'foo' check: glork,ping")
+          AppStatus::CheckCollection.add_check
+        }.to raise_error(ArgumentError, "wrong number of arguments (0 for 1)")
       end
 
-      it "should raise an error if :status is unrecognized" do
-        AppStatus::CheckCollection.configure {|c| c.add name: 'foo', status: 'test' }
-        c = AppStatus::CheckCollection.new
+      it "should raise an error if block is not supplied" do
         expect {
-          c.evaluate!
-        }.to raise_error(ArgumentError, "'test' is not a valid status for check 'foo'.")
+          AppStatus::CheckCollection.add_check 'some_service'
+        }.to raise_error(ArgumentError, "No check defined for 'some_service'.")
       end
 
       it "should raise an error if a :name is used multiple times" do
-        AppStatus::CheckCollection.configure do |c|
-          c.add name: 'foo', status: :ok
-          c.add name: 'foo', status: :critical
-        end
-        c = AppStatus::CheckCollection.new
+        AppStatus::CheckCollection.add_check('foo') {[:ok, 'ok']}
         expect {
-          c.evaluate!
+          AppStatus::CheckCollection.add_check('foo') {[:ok, 'ok']}
         }.to raise_error(ArgumentError, "Check name 'foo' has already been added.")
       end
     end
 
   end
 
+
   describe "evaluate!" do
+
     it "should run configured checks each time it is called" do
       counter = 0
-      check = lambda { counter += 1; counter }
-      AppStatus::CheckCollection.configure do |c|
-        num_calls = check.call
-        c.add name: 'something', status: :ok, details: num_calls
-      end
+      AppStatus::CheckCollection.add_check('test') { counter += 1; [:ok, counter] }
 
       c = AppStatus::CheckCollection.new
 
       Timecop.freeze '2013-10-04T12:00:00Z' do
         c.evaluate!
         c.as_json[:finished].should eq '2013-10-04T12:00:00Z'
-        c.as_json[:checks][:something][:details].should eq "1"
+        c.as_json[:checks][:test][:details].should eq "1"
       end
 
       Timecop.freeze '2013-10-04T01:00:00Z' do
         c.evaluate!
         c.as_json[:finished].should eq '2013-10-04T01:00:00Z'
-        c.as_json[:checks][:something][:details].should eq "2"
+        c.as_json[:checks][:test][:details].should eq "2"
       end
     end
+
+    it "should set :unknown status for a check which does not return a status" do
+      AppStatus::CheckCollection.add_check('test') { nil }
+
+      c = AppStatus::CheckCollection.new
+      c.evaluate!
+      c.as_json[:checks][:test][:status].should eq :unknown
+      c.as_json[:checks][:test][:details].should eq "Check returned invalid status ''."
+    end
+
+    it "should set :unknown status for a check which returns an invalid status" do
+      AppStatus::CheckCollection.add_check('test') { 'huh?' }
+
+      c = AppStatus::CheckCollection.new
+      c.evaluate!
+      c.as_json[:checks][:test][:status].should eq :unknown
+      c.as_json[:checks][:test][:details].should eq "Check returned invalid status 'huh?'."
+    end
+
   end
 
   describe "as_json" do
+
     it "should use :unknown status if no checks are configured" do
       c = AppStatus::CheckCollection.new
       c.evaluate!
@@ -89,17 +106,86 @@ describe AppStatus::CheckCollection do
       data[:status].should eq :unknown
       data[:checks].should eq({})
     end
-  end
 
-  describe "configure" do
-    it "should add checks to be evaluated later" do
+    it "should set overall status to match the worst status among configured checks" do
+      c = AppStatus::CheckCollection.new
+
+      AppStatus::CheckCollection.add_check('a') { :ok }
+
+      c.evaluate!
+      c.as_json['status'].should eq :ok
+      c.as_json['checks'].size.should eq 1
+
+      AppStatus::CheckCollection.add_check('b') { :warning }
+
+      c.evaluate!
+      c.as_json['status'].should eq :warning
+      c.as_json['checks'].size.should eq 2
+
+      AppStatus::CheckCollection.add_check('c') { :critical }
+
+      c.evaluate!
+      c.as_json['status'].should eq :critical
+      c.as_json['checks'].size.should eq 3
+
+      AppStatus::CheckCollection.add_check('d') { :unknown }
+
+      c.evaluate!
+      c.as_json['status'].should eq :unknown
+      c.as_json['checks'].size.should eq 4
+    end
+
+    it "should include details on all checks" do
       AppStatus::CheckCollection.configure do |c|
-        c.add name: 'something', status: :ok
+        c.add_check('test1') { [:ok, 'looks good'] }
+        c.add_check('test2') { [:huh, 'invalid'] }
+        c.add_check('test3') { [:warning, 'not good'] }
+        c.add_check('test4') { [:critical, 'on fire'] }
+        c.add_check('test5') { [:unknown, 'no idea'] }
       end
 
       c = AppStatus::CheckCollection.new
-      c.evaluate!
-      data = c.as_json
+      Timecop.freeze('2013-10-05T12:00:00Z') { c.evaluate! }
+
+      c.as_json.should eq({
+        "status" => :unknown,
+        "status_code" => 3,
+        "ms" => instance_of(Fixnum),
+        "finished" => "2013-10-05T12:00:00Z",
+        "checks" => {
+          "test1" => {
+            "status" => :ok,
+            "status_code" => 0,
+            "details" => "looks good",
+            "ms" => instance_of(Fixnum)
+          },
+          "test2" => {
+            "status" => :unknown,
+            "status_code" => 3,
+            "details" => "Check returned invalid status 'huh'. invalid",
+            "ms" => instance_of(Fixnum)
+          },
+          "test3" => {
+            "status" => :warning,
+            "status_code" => 1,
+            "details" => "not good",
+            "ms" => instance_of(Fixnum)
+          },
+          "test4" => {
+            "status" => :critical,
+            "status_code" => 2,
+            "details" => "on fire",
+            "ms" => instance_of(Fixnum)
+          },
+          "test5" => {
+            "status" => :unknown,
+            "status_code" => 3,
+            "details" => "no idea",
+            "ms" => instance_of(Fixnum)
+          }
+        }
+      })
     end
+
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: app_status
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 1.0.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-10-04 00:00:00.000000000 Z
+date: 2013-10-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails
@@ -168,6 +168,7 @@ files:
 - spec/dummy/config/locales/en.yml
 - spec/dummy/config/routes.rb
 - spec/dummy/config.ru
+- spec/dummy/log/development.log
 - spec/dummy/log/test.log
 - spec/dummy/public/404.html
 - spec/dummy/public/422.html
@@ -192,7 +193,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 2540672613882673214
+      hash: 4001732300146768489
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -201,7 +202,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 2540672613882673214
+      hash: 4001732300146768489
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.24
@@ -231,6 +232,7 @@ test_files:
 - spec/dummy/config/locales/en.yml
 - spec/dummy/config/routes.rb
 - spec/dummy/config.ru
+- spec/dummy/log/development.log
 - spec/dummy/log/test.log
 - spec/dummy/public/404.html
 - spec/dummy/public/422.html