flapjack 0.5.5 → 0.6.23

data/.gitignore

@@ -14,3 +14,13 @@ pkg/*
 !etc/flapjack/recipients.yaml
 spec/test.db
 features/support/tmp/*
+features/support/etc/netsaint/*
+log/*
+vendor
+.bundle
+etc/flapjack_config.yaml
+coverage
+tmp/pids/*
+tmp/spec*
+flapjack-*.gem
+

data/.rbenv-version

@@ -0,0 +1 @@
+1.9.3-p125

data/.rspec

@@ -0,0 +1,10 @@
+--color
+--format Fuubar
+--format html
+--out tmp/spec.html
+--require ./spec/support/uncolored_doc_formatter.rb
+--format UncoloredDocFormatter
+--out tmp/spec_doc.txt
+--require ./spec/support/profile_all_formatter.rb
+--format ProfileAllFormatter
+--out tmp/spec_profile.txt

data/Gemfile

@@ -0,0 +1,18 @@
+source :rubygems
+
+gemspec :name => 'flapjack'
+
+group :test do
+  gem 'rspec'
+  gem 'cucumber'
+  gem 'delorean'
+  gem 'rack-test'
+  gem 'resque_spec'
+  gem 'webmock'
+  gem 'guard'
+  gem 'guard-rspec'
+  gem 'guard-cucumber'
+  gem 'fuubar'
+  gem 'fuubar-cucumber'
+  gem 'simplecov', :require => false
+end

data/Guardfile

@@ -0,0 +1,14 @@
+
+guard 'rspec', :cli => '--format Fuubar --format html --out tmp/spec.html --require ./spec/support/uncolored_doc_formatter.rb --format UncoloredDocFormatter --out tmp/spec_doc.txt --require ./spec/support/profile_all_formatter.rb --format ProfileAllFormatter --out tmp/spec_profile.txt', :version => 2 do
+  watch(%r{^spec/.+_spec\.rb$})
+  watch(%r{^lib/(.+)\.rb$})     { |m| "spec/lib/#{m[1]}_spec.rb" }
+  watch('spec/spec_helper.rb')  { "spec" }
+end
+
+# NB: seems to be buggy with the default --progress formatter, was
+# causing failures to parse the features
+guard 'cucumber', :cli => '--no-profile --color --format fuubar --strict' do
+  watch(%r{^features/.+\.feature$})
+  watch(%r{^features/support/.+$})          { 'features' }
+  watch(%r{^features/step_definitions/(.+)_steps\.rb$}) { |m| Dir[File.join("**/#{m[1]}.feature")][0] || 'features' }
+end

data/README.md

@@ -1,232 +1,211 @@
 Flapjack
 ========
 
-Flapjack is highly scalable and distributed monitoring system. 
+Flapjack is a highly scalable and distributed monitoring notification system.
 
-It understands the Nagios plugin format, and can easily be scaled 
-from 1 server to 1000. 
+Flapjack provides a scalable method for dealing with events representing changes in system state (OK -> WARNING -> CRITICAL transitions) and alerting appropriate people as necessary.
 
-WARNING
-=======
+At its core, flapjack process events received from external check execution engines, such as Nagios. Nagios provides a 'perfdata' event output channel, which writes to a named pipe. `flapjack-nagios-receiver` then reads from this named pipe, converts each line to JSON and adds them to the events queue. `executive` picks up the events and processes them - deciding when and who to notifify about problems, recoveries, acknowledgements etc. Additional check engines can be supported by adding additional receiver processes similar to the nagios receiver.
 
-**These install instructions are probably wrong. Follow them best you can, and
-please report bugs as you find them!**
 
-Setup dependencies (Ubuntu Hardy)
----------------------------------
+What things do
+--------------
 
-Add the following lines to `/etc/apt/sources.list`
+Executables:
 
-    deb http://ppa.launchpad.net/ubuntu-ruby-backports/ubuntu hardy main
-    deb http://ppa.launchpad.net/auxesis/ppa/ubuntu hardy main
+  * `flapjack` => starts multiple components ('pikelets') within the one ruby process as specified in the configuration file.
+  * `flapjack-nagios-receiver` => reads nagios check output on standard input and places them on the events queue in redis as JSON blobs. Currently unable to be run in-process with `flapjack`
 
-Add GPG keys for the repos: 
+There are more fun executables in the bin directory.
 
-    sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 288BA53BCB7DA731
+Pikelets:
 
-Update your package list:
+  * `executive` => processes events off a queue (a redis list) and decides what actions to take (alert, record state changes, etc)
+  * `email_notifier` => generates email notifications (resque, mail)
+  * `sms_notifier` => generates sms notifications (resque)
+  * `jabber_gateway` => connects to an XMPP (jabber) server, sends notifications (to rooms and individuals), handles acknowledgements from jabber users and other commands (blather)
+  * `web` => web UI (sinatra, thin)
+  * `api` => HTTP API server (sinatra, thin)
 
-    sudo apt-get update 
+Pikelets are flapjack components which can be run within the same ruby process, or as separate processes.
 
-Install Ruby dependencies: 
+The simplest configuration will have one `flapjack` process running executive, web, and some notification gateways, and one `flapjack-nagios-receiver` process receiving events from Nagios and placing them on the events queue for processing by executive.
 
-    sudo apt-get install build-essential libsqlite3-dev
 
-Install rubygems + beanstalkd:
+Developing
+----------
 
-    sudo apt-get install rubygems beanstalkd
+Clone the repository:
 
-Set `ENABLED=1` in `/etc/default/beanstalkd`.
+    git clone https://auxesis@github.com/auxesis/flapjack.git
 
-Start beanstalkd: 
+Install development dependencies:
 
-    sudo /etc/init.d/beanstalkd start
+    gem install bundler
+    bundle install
 
+Run the tests:
 
-Setup dependencies (everyone else)
-----------------------------------
+    cucumber features/
+    rspec spec
 
-Install the following software through your package manager or from source: 
+Testing
+-------
 
- - beanstalkd (from http://xph.us/software/beanstalkd/)
- - libevent (from http://monkey.org/~provos/libevent/)
+Feature tests live in `features/`.
 
+To run feature tests:
 
-Installation
-------------
+    $ cucumber features
 
+There's some rspec unit tests as well, run these like so:
 
-Add the Gemcutter RubyGems server to your Gem sources: 
+    $ rspec spec
 
-    sudo gem sources -a http://gemcutter.org
+NB, if the cucumber tests fail with a spurious lexing error on line 2 of events.feature, then try this:
 
-Install the Flapjack gem: 
+    $ cucumber -f fuubar features
 
-    sudo gem install flapjack
 
-Then run the magic configuration script to set up init scripts: 
 
-    sudo install-flapjack-systemwide
+Releasing
+---------
 
-The script will prompt you if it wants to do anything destructive. 
+Gem releases are handled with [Bundler](http://gembundler.com/rubygems.html).
 
+To build the gem, run:
 
-Running 
--------
+    gem build flapjack.gemspec
 
-Make sure beanstalkd is running.
-
-You'll want to set up `/etc/flapjack/recipients.conf` so notifications can be sent via 
-`flapjack-notifier`: 
-
-    [John Doe]
-      email = johndoe@example.org
-      jid = john@example.org
-      phone = +61 400 111 222
-      pager = 61400111222
-    
-    [Jane Doe]
-      email = janedoe@example.org
-      jid = jane@example.org
-      phone = +61 222 333 111
-      pager = 61222333111
-
-You also need to set up `/etc/flapjack/notifier.conf`: 
-
-    # top level
-    [notifier]
-      notifier-directories = /usr/lib/flapjack/notifiers/ /path/to/my/notifiers
-      notifiers = mailer, xmpp
-    
-    # persistence layers
-    [persistence]
-      backend = data_mapper
-      uri = sqlite3:///tmp/flapjack.db
-    
-    # message transports
-    [transport]
-      backend = beanstalkd
-      host = localhost
-      port = 11300
-    
-    # notifiers
-    [mailer-notifier]
-      from_address = foo@bar.com
-    
-    [xmpp-notifier]
-      jid = foo@bar.com
-      password = barfoo
-    
-    # filters
-    [filters]
-      chain = ok, downtime, any_parents_failed
-
-
-Start up a cluster of workers: 
-
-    flapjack-worker-manager start
-
-This will spin up 5 workers in the background. You can specify how many workers 
-to start: 
-
-    flapjack-worker-manager start --workers=10
-
-Each of the `flapjack-worker`s will output to syslog (check in /var/log/messages).
-
-Start up the notifier: 
-
-    flapjack-notifier-manager start --recipients /etc/flapjack/recipients.conf
-
-Currently there are email and XMPP notifiers. 
-
-You'll want to get a copy of (http://github.com/auxesis/flapjack-admin/)[flapjack-admin]
-to set up some checks, then run its populator to get them into Flapjack. 
-
-What things do 
---------------
+Configuring
+===========
 
-  * `flapjack-worker` => executes checks, reports results  
-  * `flapjack-worker-manager` => starts/stops a cluster of `flapjack-worker`
-  * `flapjack-notifier` => gets results, notifies people if necessary  
-  * `flapjack-stats` => gets stats from beanstalkd tubes (useful for benchmarks + performance analysis)  
-  * `flapjack-benchmark` => benchmarks various persistence/transport backend combinations
+```
+cp etc/flapjack-config.yaml.example etc/flapjack-config.yaml
+```
 
+Edit the configuration to suit (redis connection, smtp server, jabber server, sms gateway, etc)
 
-init scripts
-------------
+The default FLAPJACK_ENV is "development" if there is no environment variable set.
 
-You can use the provided init scripts to start Flapjack on boot. 
+Nagios config
+-------------
 
-To start: 
+You need a Nagios prior to version 3.3 as this breaks perfdata output for checks which don't generate performance data (stuff after a | in the check output). We are developing and running against Nagios version 3.2.3 with success.
 
-    /etc/init.d/flapjack-workers start
-    /etc/init.d/flapjack-notifier start
+nagios.cfg config file changes:
 
-To set Flapjack to start on boot (Ubuntu): 
+```
+# modified lines:
+enable_notifications=0
+host_perfdata_file=/var/cache/nagios3/event_stream.fifo
+service_perfdata_file=/var/cache/nagios3/event_stream.fifo
+host_perfdata_file_template=[HOSTPERFDATA]\t$TIMET$\t$HOSTNAME$\tHOST\t$HOSTSTATE$\t$HOSTEXECUTIONTIME$\t$HOSTLATENCY$\t$HOSTOUTPUT$\t$HOSTPERFDATA$
+service_perfdata_file_template=[SERVICEPERFDATA]\t$TIMET$\t$HOSTNAME$\t$SERVICEDESC$\t$SERVICESTATE$\t$SERVICEEXECUTIONTIME$\t$SERVICELATENCY$\t$SERVICEOUTPUT$\t$SERVICEPERFDATA$
+host_perfdata_file_mode=p
+service_perfdata_file_mode=p
+```
 
-    sudo update-rc.d flapjack-workers defaults
-    sudo update-rc.d flapjack-notifier defaults
+What we're doing here is telling Nagios to generate a line of output for every host and service check into a named pipe. The template lines must be as above so that `flapjack-nagios-receiver` knows what to expect.
 
-Config for the init scripts can be found in `/etc/defaults`.
+All hosts and services (or templates that they use) will need to have process_perf_data enabled on them. (This is a real misnomer, it doesn't mean the performance data will be processed, just that it will be fed to the perfdata output channel, a named pipe in our case.)
 
+Create the named pipe if it doesn't already exist:
 
+    mkfifo -m 0666 /var/cache/nagios3/event_stream.fifo
 
-Testing
--------
 
-Tests are in `spec/` and `features/`.
+Running
+=======
+
+    bin/flapjack [options]
+
+    $ flapjack --help
+    Usage: flapjack [options]
+        -c, --config [PATH]              PATH to the config file to use
+        -d, --[no-]daemonize             Daemonize?
+
+The command line option for daemonize overrides whatever is set in the config file.
+
+flapjack-nagios-receiver
+------------------------
+
+There is a control script that uses [Daemons](http://daemons.rubyforge.org/) to start, stop, restart, show status etc of the flapjack-nagios-receiver process. Options after -- are passed through to flapjack-nagios-receiver. Run it like so:
+
+    flapjack-nagios-receiver-control start -- --config /etc/flapjack/flapjack-config.yaml /path/to/nagios/perfdata.fifo
+    flapjack-nagios-receiver-control status
+    flapjack-nagios-receiver-control restart -- --config /etc/flapjack/flapjack-config.yaml /path/to/nagios/perfdata.fifo
+    flapjack-nagios-receiver-control stop
+
+* Bypassing daemons control script: *
+
+This process needs to be started with the nagios perfdata named pipe attached to its STDIN like so:
+
+    flapjack-nagios-receiver < /var/cache/nagios3/event_stream.fifo
+
+Or the path to the fifo can just be given as an argument, thanks to the magical powers of ARGV:
+
+    flapjack-nagios-receiver /var/cache/nagios3/event_stream.fifo
 
-To run tests:
+Now as nagios feeds check execution results into the perfdata named pipe, flapjack-nagios-receiver will convert them to JSON encoded ruby objects and insert them into the *events* queue.
 
-    $ rake spec
-    $ rake cucumber
+Importing contacts and entities
+-------------------------------
 
+The `flapjack-populator` script provides a mechanism for importing contacts and entities from JSON formatted import files.
+
+    bin/flapjack-populator import-contacts --from tmp/dummy_contacts.json --config /etc/flapjack/flapjack-config.yml
+    bin/flapjack-populator import-entities --from tmp/dummy_entities.json --config /etc/flapjack/flapjack-config.yml
+
+TODO: convert the format documentation into markdown and include in the project
+
+In lieu of actual documentation of the formats there are example files, and example ruby code which generated them, in the tmp directory.
+
+executive
+---------
+
+Flapjack executive processes events from the 'events' list in redis. It does a blocking read on redis so new events are picked off the events list and processed as soon as they created.
+
+When executive decides somebody ought to be notified (for a problem, recovery, or acknowledgement or what-have-you) it looks up contact information and then creates a notification job on one of the notification queues (eg sms_notifications) in rescue, or via the jabber_notifications redis list which is processed by the jabber_gateway.
+
+Resque Workers
+--------------
+
+We're using [Resque](https://github.com/defunkt/resque) to queue email and sms notifications generated by flapjack-executive. The queues are named as follows:
+- email_notifications
+- sms_notifications
+
+There'll be more of these as we add more notification mediums.
+
+Note that using the flapjack-config.yaml file you can have flapjack start the resque workers in-process. Or you can run them standalone with the `rake resque:work` command as follows.
+
+One resque worker that processes both queues (but prioritises SMS above email) can be started as follows:
+
+    QUEUE=sms_notifications,email_notifications VERBOSE=1 be rake resque:work
+
+resque sets the command name so grep'ing ps output for `rake` or `ruby` will NOT find resque processes. Search instead for `resque`. (and remember the 'q').
+
+To background it you can add `BACKGROUND=yes`. Excellent documentation is available in [Resque's README](https://github.com/defunkt/resque/blob/master/README.markdown)
+
+Redis Database Instances
+------------------------
+
+We use the following redis database numbers by convention:
+
+* 0 => production
+* 13 => development
+* 14 => testing
 
 Architecture
 ------------
 
-    -------------------        -------------------
-    | web interface / |        | visualisation / |
-    | dsl / flat file |        | reporting       |
-    -------------------        -------------------
-              |                   |
-               \                 /
-                \               /
-                 ---------------
-                 | persistence |
-                 ---------------
-                        |
-          -------------------------------
-          |                             |
-          |                             |
-          |                             |
-    -------------                  ------------
-    | populator |---          -----| notifier |
-    -------------  |          |    ------------
-                   |          |
-                  --------------
-                  | beanstalkd |
-                  --------------
-                        |
-       -----------------------------------
-       |                |                |
-    ----------      ----------       ----------
-    | worker |      | worker |       | worker |
-    ----------      ----------       ----------
-
-
-- populator determines checks that need to occur, and puts checks onto `jobs` tube
-  - populator fetches jobs from a database 
-  - populator can be swapped out to get checks from a dsl or flat files
-- workers pop a check off the beanstalk, perform check, put result onto `results` tube, 
-  re-add check to `jobs` tube with a delay. 
-- notifier pops results off `results` tube, notifies as necessary
+TODO: insert architecture diagram and notes here
+
+Dependencies
+------------
+
+Apart from a bundle of gems (see flapjack.gemspec for runtime gems, and Gemfile for additional gems required for testing):
+- Ruby >= 1.9
+- Redis >= 2.4.15
 
-Releasing
----------
 
- 1. bump version number + release date in gemspec
- 1. create a new tag with `git tag -a -m "releasing 0.9" 0.9`
- 1. push to github with `git push --tags`
- 1. create a new gem with `rake build`
- 1. push gem to Gemcutter with `rake push`