flapjack 0.4.10

data/LICENCE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 Lindsay Holmwood
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,205 @@
+Flapjack
+========
+
+Flapjack is highly scalable and distributed monitoring system. 
+
+It understands the Nagios plugin format, and can easily be scaled 
+from 1 server to 1000. 
+
+
+
+Setup dependencies (Ubuntu Hardy)
+---------------------------------
+
+Add the following lines to `/etc/apt/sources.list`
+
+    deb http://ppa.launchpad.net/ubuntu-ruby-backports/ubuntu hardy main
+    deb http://ppa.launchpad.net/auxesis/ppa/ubuntu hardy main
+
+Add GPG keys for the repos: 
+
+    sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 288BA53BCB7DA731
+
+Update your package list:
+
+    sudo apt-get update 
+
+Install Ruby dependencies: 
+
+    sudo apt-get install build-essential libsqlite3-dev
+
+Install rubygems + beanstalkd:
+
+    sudo apt-get install rubygems beanstalkd
+
+Set `ENABLED=1` in `/etc/default/beanstalkd`.
+
+Start beanstalkd: 
+
+    sudo /etc/init.d/beanstalkd start
+
+
+Setup dependencies (everyone else)
+----------------------------------
+
+Install the following software through your package manager or from source: 
+
+ - beanstalkd (from http://xph.us/software/beanstalkd/)
+ - libevent (from http://monkey.org/~provos/libevent/)
+
+
+Installation
+------------
+
+Add GitHub's RubyGems server to your Gem sources: 
+
+    sudo gem sources -a http://gems.github.com
+
+Install the Flapjack gem: 
+
+    sudo gem install auxesis-flapjack
+
+Then run the magic configuration script to set up init scripts: 
+
+    sudo install-flapjack-systemwide
+
+The script will prompt you if it wants to do anything destructive. 
+
+
+Running 
+-------
+
+Make sure beanstalkd is running.
+
+You'll want to set up `/etc/flapjack/recipients.yaml` so notifications can be sent via 
+`flapjack-notifier`: 
+
+    ---
+    - :name: Jane Doe
+      :email: "jane@doe.com"
+      :phone: "+61 444 222 111"
+      :pager: "61444222111"
+      :jid: "jane@doe.com"
+
+You also need to set up `/etc/flapjack/flapjack-notifier.yaml`: 
+
+    --- 
+    :notifiers: 
+      :mailer: 
+        :from_address: notifications@my-domain.com
+      :xmpp: 
+        :jid: notifications@my-domain.com
+        :password: foo
+    :database_uri: "sqlite3:///var/lib/flapjack/flapjack.db"
+
+Start up a cluster of workers: 
+
+    flapjack-worker-manager start
+
+This will spin up 5 workers in the background. You can specify how many workers 
+to start: 
+
+    flapjack-worker-manager start --workers=10
+
+Each of the `flapjack-worker`s will output to syslog (check in /var/log/messages).
+
+Start up the notifier: 
+
+    flapjack-notifier-manager start --recipients /etc/flapjack/recipients.yaml
+
+Currently there are email and XMPP notifiers. 
+
+You'll want to get a copy of (http://github.com/auxesis/flapjack-admin/)[flapjack-admin]
+to set up some checks, then run its' populator to get them into Flapjack. 
+
+What things do 
+--------------
+
+  * `flapjack-worker` => executes checks, reports results  
+  * `flapjack-worker-manager` => starts/stops a cluster of `flapjack-worker`
+  * `flapjack-notifier` => gets results, notifies people if necessary  
+  * `flapjack-stats` => gets stats from beanstalkd tubes (useful for benchmarks + performance analysis)  
+
+
+init scripts
+------------
+
+You can use the provided init scripts to start Flapjack on boot. 
+
+To start: 
+
+    /etc/init.d/flapjack-workers start
+    /etc/init.d/flapjack-notifier start
+
+To set Flapjack to start on boot (Ubuntu): 
+
+    sudo update-rc.d flapjack-workers defaults
+    sudo update-rc.d flapjack-notifier defaults
+
+Config for the init scripts can be found in `/etc/defaults`.
+
+
+
+Developing
+----------
+
+You can write your own notifiers and place them in `lib/flapjack/notifiers/`.
+
+Your notifier just needs to implement the `notify` method, and take in a hash:
+
+    class Sms
+      def initialize(opts={})
+        # you may want to set from address here
+      end
+
+      def notify(opts={})
+        who = opts[:who]
+        result = opts[:result]
+        # sms to your hearts content
+      end
+    end
+
+
+Testing
+-------
+
+Tests are in `spec/`.
+
+To run tests:
+
+    $ rake spec
+
+
+Architecture
+------------
+
+           -------------------
+           | web interface / |
+           | dsl / flat file |
+           -------------------
+          /
+          |
+          |
+    -------------                  ------------
+    | populator |---          -----| notifier |
+    -------------  |          |    ------------
+                   |          |
+                  --------------
+                  | beanstalkd |
+                  --------------
+                        |
+       -----------------------------------
+       |                |                |
+    ----------      ----------       ----------
+    | worker |      | worker |       | worker |
+    ----------      ----------       ----------
+
+
+- populator determines checks that need to occur, and puts checks onto `jobs` tube
+  - populator fetches jobs from a database 
+  - populator can be swapped out to get checks from a dsl or flat files
+- workers pop a check off the beanstalk, perform check, put result onto `results` tube, 
+  re-add check to `jobs` tube with a delay. 
+- notifier pops results off `results` tube, notifies as necessary
+
+

data/Rakefile

@@ -0,0 +1,84 @@
+#!/usr/bin/env ruby 
+
+require 'rubygems'
+require 'fileutils'
+require 'spec/rake/spectask'
+
+# integration tests for cli utils
+begin 
+  require 'cucumber/rake/task'
+
+  Cucumber::Rake::Task.new do |task|
+    task.cucumber_opts = "--require features/"
+  end
+rescue LoadError
+end
+
+Spec::Rake::SpecTask.new do |t|
+  t.spec_opts = ["--options", "spec/spec.opts"]
+end
+
+
+desc "freeze deps"
+task :deps do 
+
+  deps = {'beanstalk-client' => ">= 1.0.2",
+          'log4r' => ">= 1.0.5",
+          'xmpp4r-simple' => ">= 0.8.8",
+          'mailfactory' => ">= 1.4.0"}
+
+  puts "\ninstalling dependencies. this will take a few minutes."
+
+  deps.each_pair do |dep, version|
+    puts "\ninstalling #{dep} (#{version})"
+    system("gem install #{dep} --version '#{version}' -i gems --no-rdoc --no-ri")
+  end
+
+end
+
+desc "generate list of files for gemspec"
+task "gengemfiles" do 
+  executables = `git ls-files bin/*`.split.map {|bin| bin.gsub(/^bin\//, '')}             
+  files = `git ls-files`.split.delete_if {|file| file =~ /^(spec\/|\.gitignore)/}
+  puts
+  puts "Copy and paste into flapjack.gemspec:"
+  puts
+  puts "    s.executables = #{executables.inspect}"
+  puts "    s.files = #{files.inspect}"
+  puts
+  puts
+end
+
+desc "build gem"
+task :build do 
+  system("gem build flapjack.gemspec")
+  
+  FileUtils.mkdir_p('pkg')
+  puts
+  puts "Flapjack gems:"
+  Dir.glob("flapjack-*.gem").each do |gem|
+    dest = File.join('pkg', gem)
+    FileUtils.mv gem, dest
+    puts "  " + dest
+  end
+end
+
+
+if require 'yard'
+  
+  YARD::Rake::YardocTask.new do |t|
+    t.files   = ['lib/**/*.rb']
+    t.options = ['--output-dir=doc/', '--readme=README.md']
+  end
+  
+end
+
+desc "display FIXMEs in the codebase"
+task :fixmes do 
+  output = `grep -nR FIXME lib/* spec/* bin/`
+  output.split("\n").each do |line|
+    parts = line.split(':')
+    puts "#{parts[0].strip} +#{parts[1].strip}"
+    puts "  - #{parts[3].strip}"
+  end
+end

data/TODO.md

@@ -0,0 +1,26 @@
+ * create events for failed checks
+ * build option to specify notifier(s) directory
+ * documentation!
+   * user
+   * developer
+   * step-by-step install guide
+   * scaling guide
+   * integrating with collectd guide
+   * writing custom populators guide
+ * build benchmarks for flapjack-{worker,notifier}
+ * write puppet manifests
+ * package with pallet
+ * generate .deb/.rpms
+   * build packages for gem dependencies
+ * sandbox flapjack-worker
+   * provide config option for specifying sandbox dir
+ * provide common interface for loading checks into beanstalk (extract from populator)
+ * write check generator
+   * include a collection of common functions 
+     (logging to rrd, retreiving values, executing check)
+ * write zeroconf/avahi notifier
+ * write growl notifier
+ * write way to customise notifier messages (email body, xmpp format)
+ * write sms notifier
+ * patch beanstalk-client to recognise DRAINING status 
+ * http://www.kitchensoap.com/2009/10/05/meanwhile-more-meta-metrics/

data/bin/flapjack-notifier

@@ -0,0 +1,46 @@
+#!/usr/bin/env ruby
+
+$: << File.dirname(__FILE__) + '/../lib' unless $:.include?(File.dirname(__FILE__) + '/../lib/')
+
+require 'rubygems'
+require 'beanstalk-client'
+require 'ostruct'
+require 'optparse'
+require 'log4r'
+require 'log4r/outputter/syslogoutputter'
+require 'flapjack/result'
+require 'flapjack/notifier'
+require 'daemons'
+require 'flapjack/patches'
+
+# command line options are in here
+require 'flapjack/cli/notifier'
+
+# boot up the notifier
+@options = Flapjack::NotifierOptions.parse(ARGV)
+
+ncli = Flapjack::NotifierCLI.new
+ncli.setup_loggers
+ncli.setup_recipients(:filename => @options.recipients)
+ncli.setup_config(:filename => @options.config_filename)
+ncli.setup_notifier
+ncli.setup_database(:database_uri => @options.database_uri)
+
+begin 
+
+  ncli.results_queue = Beanstalk::Pool.new(["#{@options.host}:#{@options.port}"], 'results')
+  ncli.log.info("established connection to beanstalkd on #{@options.host}...")
+
+  # process results
+  ncli.process_loop
+
+rescue Beanstalk::NotConnected
+  ncli.log.error("Couldn't connect to the Beanstalk!")
+
+  timeout = 5 
+  ncli.log.error("Retrying in #{timeout} seconds.")
+  sleep timeout
+
+  ncli.log.error("Retrying...")
+  retry
+end

data/bin/flapjack-notifier-manager

@@ -0,0 +1,45 @@
+#!/usr/bin/env ruby
+
+$: << File.dirname(__FILE__) + '/../lib' unless $:.include?(File.dirname(__FILE__) + '/../lib/')
+
+require 'rubygems'
+require 'daemons'
+require 'fileutils'
+require 'flapjack/cli/notifier_manager'
+require 'flapjack/patches' # for Daemons
+
+# reassign ARGV so we don't mess with it directly
+args = ARGV
+args << '--help' if args.size == 0
+options = Flapjack::NotifierManagerOptions.parse(args)
+
+worker_path = File.join(File.dirname(__FILE__), 'flapjack-notifier')
+
+# set up pid dir
+pid_dir = "/var/run/flapjack/"
+unless File.exists?(pid_dir)
+  puts "#{pid_dir} doesn't exist."
+  exit 2
+end
+
+unless File.writable?(pid_dir)
+  puts "Can't write to #{pid_dir} - check permissions?"
+  exit 2
+end
+
+# construct arguments
+daemon_args = args
+if args.first != "stop"
+  # if we're not stopping the daemon, pass options to it
+  daemon_args += ['--', '--beanstalk', options.host, 
+                        '--port', options.port.to_s,
+                        '--recipients', options.recipients,
+                        '--config', File.expand_path(options.config_filename)]
+end
+
+# boot
+Daemons.run(worker_path, :ARGV => daemon_args,
+                         :multiple => false, 
+                         :dir_mode => :normal,
+                         :dir => pid_dir)
+

data/bin/flapjack-stats

@@ -0,0 +1,22 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+require 'beanstalk-client'
+
+begin 
+  beanstalk = Beanstalk::Pool.new(['localhost:11300'])
+  loop do 
+    time = Time.now
+    beanstalk.list_tubes['localhost:11300'].each do |tube_name|
+      next if tube_name == 'default'
+      stats = beanstalk.stats_tube(tube_name)
+      puts "#{time.to_i} [#{tube_name}] total: #{stats['total-jobs']}, waiting: #{stats['current-waiting']}, ready: #{stats['current-jobs-ready']}"
+    end
+    sleep 1
+  end
+rescue Beanstalk::NotConnected
+  puts "beanstalk isn't up!"
+  exit 2
+end
+  
+