flapjack 0.4.12 → 0.5.1

data/README.md

@@ -6,7 +6,11 @@ Flapjack is highly scalable and distributed monitoring system.
 It understands the Nagios plugin format, and can easily be scaled 
 from 1 server to 1000. 
 
+WARNING
+=======
 
+**These install instructions are probably wrong. Follow them best you can, and
+please report bugs as you find them!**
 
 Setup dependencies (Ubuntu Hardy)
 ---------------------------------
@@ -51,13 +55,14 @@ Install the following software through your package manager or from source:
 Installation
 ------------
 
-Add GitHub's RubyGems server to your Gem sources: 
 
-    sudo gem sources -a http://gems.github.com
+Add the Gemcutter RubyGems server to your Gem sources: 
+
+    sudo gem sources -a http://gemcutter.org
 
 Install the Flapjack gem: 
 
-    sudo gem install auxesis-flapjack
+    sudo gem install flapjack
 
 Then run the magic configuration script to set up init scripts: 
 
@@ -71,26 +76,51 @@ Running
 
 Make sure beanstalkd is running.
 
-You'll want to set up `/etc/flapjack/recipients.yaml` so notifications can be sent via 
+You'll want to set up `/etc/flapjack/recipients.conf` so notifications can be sent via 
 `flapjack-notifier`: 
 
-    ---
-    - :name: Jane Doe
-      :email: "jane@doe.com"
-      :phone: "+61 444 222 111"
-      :pager: "61444222111"
-      :jid: "jane@doe.com"
-
-You also need to set up `/etc/flapjack/flapjack-notifier.yaml`: 
+    [John Doe]
+      email = johndoe@example.org
+      jid = john@example.org
+      phone = +61 400 111 222
+      pager = 61400111222
+    
+    [Jane Doe]
+      email = janedoe@example.org
+      jid = jane@example.org
+      phone = +61 222 333 111
+      pager = 61222333111
+
+You also need to set up `/etc/flapjack/notifier.conf`: 
+
+    # top level
+    [notifier]
+      notifier-directories = /usr/lib/flapjack/notifiers/ /path/to/my/notifiers
+      notifiers = mailer, xmpp
+    
+    # persistence layers
+    [persistence]
+      backend = data_mapper
+      uri = sqlite3:///tmp/flapjack.db
+    
+    # message transports
+    [transport]
+      backend = beanstalkd
+      host = localhost
+      port = 11300
+    
+    # notifiers
+    [mailer-notifier]
+      from_address = foo@bar.com
+    
+    [xmpp-notifier]
+      jid = foo@bar.com
+      password = barfoo
+    
+    # filters
+    [filters]
+      chain = ok, downtime, any_parents_failed
 
-    --- 
-    :notifiers: 
-      :mailer: 
-        :from_address: notifications@my-domain.com
-      :xmpp: 
-        :jid: notifications@my-domain.com
-        :password: foo
-    :database_uri: "sqlite3:///var/lib/flapjack/flapjack.db"
 
 Start up a cluster of workers: 
 
@@ -105,12 +135,12 @@ Each of the `flapjack-worker`s will output to syslog (check in /var/log/messages
 
 Start up the notifier: 
 
-    flapjack-notifier-manager start --recipients /etc/flapjack/recipients.yaml
+    flapjack-notifier-manager start --recipients /etc/flapjack/recipients.conf
 
 Currently there are email and XMPP notifiers. 
 
 You'll want to get a copy of (http://github.com/auxesis/flapjack-admin/)[flapjack-admin]
-to set up some checks, then run its' populator to get them into Flapjack. 
+to set up some checks, then run its populator to get them into Flapjack. 
 
 What things do 
 --------------
@@ -119,6 +149,7 @@ What things do
   * `flapjack-worker-manager` => starts/stops a cluster of `flapjack-worker`
   * `flapjack-notifier` => gets results, notifies people if necessary  
   * `flapjack-stats` => gets stats from beanstalkd tubes (useful for benchmarks + performance analysis)  
+  * `flapjack-benchmark` => benchmarks various persistence/transport backend combinations
 
 
 init scripts
@@ -140,46 +171,35 @@ Config for the init scripts can be found in `/etc/defaults`.
 
 
 
-Developing
-----------
-
-You can write your own notifiers and place them in `lib/flapjack/notifiers/`.
-
-Your notifier just needs to implement the `notify` method, and take in a hash:
-
-    class Sms
-      def initialize(opts={})
-        # you may want to set from address here
-      end
-
-      def notify(opts={})
-        who = opts[:who]
-        result = opts[:result]
-        # sms to your hearts content
-      end
-    end
-
-
 Testing
 -------
 
-Tests are in `spec/`.
+Tests are in `spec/` and `features/`.
 
 To run tests:
 
     $ rake spec
+    $ rake cucumber
 
 
 Architecture
 ------------
 
-           -------------------
-           | web interface / |
-           | dsl / flat file |
-           -------------------
-          /
-          |
-          |
+    -------------------        -------------------
+    | web interface / |        | visualisation / |
+    | dsl / flat file |        | reporting       |
+    -------------------        -------------------
+              |                   |
+               \                 /
+                \               /
+                 ---------------
+                 | persistence |
+                 ---------------
+                        |
+          -------------------------------
+          |                             |
+          |                             |
+          |                             |
     -------------                  ------------
     | populator |---          -----| notifier |
     -------------  |          |    ------------
@@ -202,4 +222,11 @@ Architecture
   re-add check to `jobs` tube with a delay. 
 - notifier pops results off `results` tube, notifies as necessary
 
+Releasing
+---------
 
+ 1. bump version number + release date in gemspec
+ 1. create a new tag with `git tag -a -m "releasing 0.9" 0.9`
+ 1. push to github with `git push --tags`
+ 1. create a new gem with `rake build`
+ 1. push gem to Gemcutter with `rake push`

data/Rakefile

@@ -19,23 +19,6 @@ Spec::Rake::SpecTask.new do |t|
 end
 
 
-desc "freeze deps"
-task :deps do 
-
-  deps = {'beanstalk-client' => ">= 1.0.2",
-          'log4r' => ">= 1.0.5",
-          'xmpp4r-simple' => ">= 0.8.8",
-          'mailfactory' => ">= 1.4.0"}
-
-  puts "\ninstalling dependencies. this will take a few minutes."
-
-  deps.each_pair do |dep, version|
-    puts "\ninstalling #{dep} (#{version})"
-    system("gem install #{dep} --version '#{version}' -i gems --no-rdoc --no-ri")
-  end
-
-end
-
 desc "generate list of files for gemspec"
 task "gengemfiles" do 
   executables = `git ls-files bin/*`.split.map {|bin| bin.gsub(/^bin\//, '')}             
@@ -64,15 +47,6 @@ task :build do
 end
 
 
-if require 'yard'
-  
-  YARD::Rake::YardocTask.new do |t|
-    t.files   = ['lib/**/*.rb']
-    t.options = ['--output-dir=doc/', '--readme=README.md']
-  end
-  
-end
-
 desc "display FIXMEs in the codebase"
 task :fixmes do 
   output = `grep -nR FIXME lib/* spec/* bin/`
@@ -82,3 +56,81 @@ task :fixmes do
     puts "  - #{parts[3].strip}"
   end
 end
+
+desc "build a tarball suitable for building packages from"
+task :tarball do 
+  require 'open-uri'
+  require 'tmpdir'
+  tmpdir = Dir.mktmpdir
+
+  current_commit  = `git log`.first.split[1][0..5]
+  release_tarball = "/tmp/flapjack-#{current_commit}.tar.gz"
+  flapjack_path   = File.join(tmpdir, "flapjack-#{current_commit}")
+  repo_path       = File.expand_path(File.dirname(__FILE__))
+  ignores         = %w(.gitignore .git)
+
+  # create new staging directory
+  command = "git clone #{repo_path} #{flapjack_path}"
+  `#{command}`
+
+  # remove files we don't want to publish
+  ignores.each do |filename|
+    path = File.join(flapjack_path, filename)
+    FileUtils.rm_rf(path)
+  end
+
+  deps = { :daemons => {
+             :source => "http://rubyforge.org/frs/download.php/34223/daemons-1.0.10.tgz",
+             :filename => "daemons-1.0.10.tar.gz"
+           },
+           :beanstalkd_client => {
+             :source => "http://github.com/kr/beanstalk-client-ruby/tarball/v1.0.2",
+             :filename => "beanstalkd-client-1.0.2.tar.gz"
+           },
+           :yajl_ruby => {
+             :source => "http://github.com/brianmario/yajl-ruby/tarball/0.6.4",
+             :filename => "yajl-ruby-0.6.4.tar.gz"
+           },
+           :tmail => {
+             :source => "http://rubyforge.org/frs/download.php/35297/tmail-1.2.3.1.tgz",
+             :filename => "tmail-1.2.3.1.tar.gz"
+           },
+           :xmpp4r => {
+             :source => "http://download.gna.org/xmpp4r/xmpp4r-0.4.tgz",
+             :filename => "xmpp4r-0.4.tar.gz"
+           },
+           :log4r => {
+             :source => "http://qa.debian.org/watch/sf.php/log4r/log4r-1.0.5.tgz",
+             :filename => "log4r-1.0.5.tar.gz"
+           }
+  }
+  deps.each_pair do |name, details|
+    puts "Pulling in #{name} dependency..."
+    # save file
+    saved_file = "/tmp/#{details[:filename]}"
+    File.open(saved_file, 'w') do |f|
+      f << open(details[:source]).read 
+    end
+
+    `tar zxf #{saved_file} -C #{tmpdir}`
+  end
+
+  # build tarball
+  under = File.dirname(tmpdir)
+  staging = File.join("/tmp", "flapjack-#{current_commit}")
+  FileUtils.mv(tmpdir, staging)
+  actual = File.basename(staging)
+  command = "cd #{under} ; tar czf #{release_tarball} #{actual}"
+  system(command)
+
+  puts "Release tarball with deps at #{release_tarball}"
+end
+
+desc "dump out statements to create sqlite3 schema"
+task :dm_debug do 
+  require 'lib/flapjack/persistence/data_mapper'
+
+  DataMapper.logger.set_log(STDOUT, :debug)
+  DataMapper.setup(:default, "sqlite3:///tmp/sqlite3.db")
+  DataMapper.auto_migrate!
+end

data/TODO.md

@@ -1,53 +1,36 @@
- * create events for failed checks
+ * refactor couchdb backend to be less bongtastic
+ * write "action" hooks API on notifications
 
- * rethink Notifier/NotifierCLI split
-   Notifier + NotifierCLI are tightly coupled, which makes it difficult to refactor
-   follow Puppet's lead with Puppet::Application (NotifierCLI translates to Flapjack::Notifier::Application)
-   implement simple interface application interface => Flapjack::Notifier::Application.run(options)
+ * write migration tool for sqlite3 => couchdb
 
+ * write persistence backend setup tool (create couchdb/sqlite3 database)
 
- * reduce notifier dependencies (e.g. xmpp4r over xmpp4r-simple)
- * make notification/escalation logic pluggable (to reduce packaging dependencies)
-
- * release Flapjack as a distribution-consumable tarball
- * automate building of release tarball that optionally pulls in dependencies
  * add lintian-like checks for verifying packagability (see http://pkg-ruby-extras.alioth.debian.org/upstream-devs.html)
- 
- * build benchmarks for flapjack-{worker,notifier}
-
- * setup wiki.flapjack-project.com
- * documentation!
-   * user
-   * developer
-   * step-by-step install guide
-   * scaling guide
-   * integrating with collectd guide
-   * writing custom populators guide
- * write puppet manifests
- 
- * build option to specify notifier(s) directory
  * sandbox flapjack-worker
-   * provide config option for specifying sandbox dir
+ * write beanstalkd.yreserve to simplify code
+ * write beanstalkd.jput, beanstalkd.jreserve for native json api
+
+ * build config file/cli options proxy
+ * build easily runnable benchmarks for flapjack-{worker,notifier}
+
+ * update installation guide
+ * clean up persistence API to be more user friendly
 
+ * write puppet manifests
  * provide common interface for loading checks into beanstalk (extract from populator)
- * make message queueing interface more abstract (support for AMQP/RabbitMQ)
  
  * write zeroconf/avahi notifier
  * write growl notifier
  * write sms notifier
- * write way to customise notifier messages (email body, xmpp format)
+ * allow customisation of notifier messages (body, header)
 
  * http://www.kitchensoap.com/2009/10/05/meanwhile-more-meta-metrics/
 
- * write beanstalkd.yreserve to simplify code
- * write beanstalkd.jput, beanstalkd.jreserve for native json api
- 
  * add support to worker and notifier for multiple beanstalkds
+ * patch beanstalk-client to recognise DRAINING status 
 
  * write check generator
    * include a collection of common functions 
      (logging to rrd, retreiving values, executing check)
- * patch beanstalk-client to recognise DRAINING status 
-
 
 

data/bin/flapjack-benchmark

@@ -0,0 +1,50 @@
+#!/usr/bin/env ruby
+
+$: << File.dirname(__FILE__) + '/../lib' unless $:.include?(File.dirname(__FILE__) + '/../lib/')
+
+require 'flapjack/cli/notifier'
+require 'flapjack/applications/notifier'
+require 'daemons'
+require 'flapjack/patches'
+
+# setup 
+@options = Flapjack::Notifier::Options.parse(ARGV)
+
+log = Log4r::Logger.new("notifier")
+log.add(Log4r::SyslogOutputter.new("notifier"))
+
+app = Flapjack::Notifier::Application.run(:log => log, 
+                                          :notifiers => {},
+                                          :filters => [],
+                                          :queue_backend => {:type => :beanstalkd},
+                                          :persistence => {:type => :data_mapper,
+                                                           :uri => "sqlite3:///tmp/flapjack.db"})
+
+mockapp = Flapjack::Notifier::Application.run(:log => log,
+                                              :notifiers => {},
+                                              :queue_backend => {:type => :mockbackend,
+                                                                 :basedir => File.join(File.dirname(__FILE__), '..', 'spec', 'transports')},
+                                              :persistence => {:type => :mockbackend,
+                                                               :basedir => File.join(File.dirname(__FILE__), '..', 'spec', 'persistence')})
+
+# populate 
+require 'beanstalk-client'
+beanstalk = Beanstalk::Pool.new(["localhost:11300"], 'results')
+
+puts "Writing 1000 passing checks onto work queue."
+2000.times do 
+  result = {:output => "some output", :check_id => 1, :retval => 0}
+  beanstalk.yput(result)
+end
+
+# benchmark
+require 'benchmark' 
+
+Benchmark.bm(30) do |b|
+  b.report("beanstalkd + datamapper:") do
+    1000.times {app.process_result}
+  end
+  b.report("mock + mock:") do
+    1000.times {mockapp.process_result}
+  end
+end

data/bin/flapjack-notifier

@@ -2,45 +2,20 @@
 
 $: << File.dirname(__FILE__) + '/../lib' unless $:.include?(File.dirname(__FILE__) + '/../lib/')
 
-require 'rubygems'
-require 'beanstalk-client'
-require 'ostruct'
-require 'optparse'
-require 'log4r'
-require 'log4r/outputter/syslogoutputter'
-require 'flapjack/result'
-require 'flapjack/notifier'
+require 'flapjack/cli/notifier'
+require 'flapjack/applications/notifier'
 require 'daemons'
 require 'flapjack/patches'
 
-# command line options are in here
-require 'flapjack/cli/notifier'
-
-# boot up the notifier
-@options = Flapjack::NotifierOptions.parse(ARGV)
-
-ncli = Flapjack::NotifierCLI.new
-ncli.setup_loggers
-ncli.setup_recipients(:filename => @options.recipients)
-ncli.setup_config(:filename => @options.config_filename)
-ncli.setup_notifier
-ncli.setup_database(:database_uri => @options.database_uri)
-
-begin 
-
-  ncli.results_queue = Beanstalk::Pool.new(["#{@options.host}:#{@options.port}"], 'results')
-  ncli.log.info("established connection to beanstalkd on #{@options.host}...")
-
-  # process results
-  ncli.process_loop
+@options = Flapjack::Notifier::Options.parse(ARGV)
 
-rescue Beanstalk::NotConnected
-  ncli.log.error("Couldn't connect to the Beanstalk!")
+log = Log4r::Logger.new("notifier")
+log.add(Log4r::StdoutOutputter.new("notifier"))
+log.add(Log4r::SyslogOutputter.new("notifier"))
 
-  timeout = 5 
-  ncli.log.error("Retrying in #{timeout} seconds.")
-  sleep timeout
+app = Flapjack::Notifier::Application.run(:log => log, 
+                                          :notifiers => @options.notifiers,
+                                          :transport => @options.transport,
+                                          :persistence => @options.persistence)
+app.main()
 
-  ncli.log.error("Retrying...")
-  retry
-end