smart_proxy_monitoring 0.0.1

data/README.md

@@ -0,0 +1,174 @@
+# Smart Proxy - Monitoring
+
+This plug-in adds support for Monitoring to Foreman's Smart Proxy.
+It requires also the Foreman Monitoring plug-in.
+
+# Installation
+
+Please see the Foreman manual for appropriate instructions:
+
+* [Foreman: How to Install a Plugin](http://theforeman.org/manuals/latest/index.html#6.Plugins)
+
+The gem name is `smart_proxy_monitoring`.
+
+RPM users can install the `rubygem-smart_proxy_monitoring` packages.
+
+This plug-in has not been packaged for Debian, yet.
+
+# Configuration
+
+The plug-in requires some configuration on the Monitoring server and the Smart Proxy.
+For now the only supported Monitoring solution is Icinga 2.
+
+## Icinga 2
+
+The Smart Proxy connects to the Icinga 2 API using an API User with password or
+certificate to get Monitoring information. It requires at least Icinga 2 version 2.5.
+
+The Icinga project provides detailed [documentation on Icinga 2](http://docs.icinga.org/icinga2/).
+The required steps for connecting the Smart Proxy and Icinga 2 will be found below.
+
+### Monitoring Server
+
+On the Monitoring Server you have to enable the API and create API User.
+
+For testing the fastest way to setup this will be the following commands.
+
+```
+# icinga2 api setup
+# systemctl restart icinga2.service
+```
+
+This will create the certficates, enable the API feature and create and API User `root` with
+a random password. The configuration of the API User will be located in `/etc/icinga2/conf.d/api-users.conf`.
+
+More detailed instructions:
+
+To enable the API follow the next steps, if the API is already enabled skip this steps
+and start by creating an API User. The API will already be enabled if you use the Icingaweb 2
+Module Director for configuration, Icinga 2 as Agents or in a distributed or high-available
+setup.
+
+Before you can enable the API a CA and a host certificate are required, the instructions
+will help you to setup Icinga 2's own CA. You can also use your Puppet's certificates or
+any other CA.
+
+To create Icinga 2's own CA run:
+
+```
+# icinga2 pki new-ca
+```
+
+Afterwards copy the CA certificate to Icinga 2's pki directory:
+
+```
+# cp /var/lib/icinga2/ca/ca.crt /etc/icinga2/pki/
+```
+
+To create a certificate request for the node run:
+
+```
+# icinga2 pki new-cert --cn $(hostname -f) --key /etc/icinga2/pki/$(hostname -f).key --csr /etc/icinga2/pki/$(hostname -f).csr
+```
+
+And then sign the certficate request to get a certificate by executing:
+
+```
+# icinga2 pki sign-csr --csr /etc/icinga2/pki/$(hostname -f).csr --cert /etc/icinga2/pki/$(hostname -f).crt
+```
+
+With the certificates created and placed in Icinga 2's pki directory you can enable the API feature.
+
+```
+# icinga2 feature enable api
+# systemctl restart icinga2.service
+```
+
+To allow API connections you have to create an API User. You should name him according to the use case,
+so instructions will create an user named `foreman`. 
+
+Password authentication is easier to setup, but certificate-based authentication is more secure.
+
+Password authentication only requires you to create an API User object in a configuration file
+read by Icinga 2.
+
+```
+# vi /etc/icinga2/conf.d/api-users.conf
+object ApiUser "foreman" {
+  password = "foreman"
+  permissions = [ "*" ]
+}
+# systemctl reload icinga2.service
+```
+
+Certificate-based authentication requires the API User object and a signed certificate.
+
+```
+# vi /etc/icinga2/conf.d/api-users.conf
+object ApiUser "foreman" {
+  client_cn = "foreman"
+  permissions = [ "*" ]
+}
+# systemctl reload icinga2.service
+# icinga2 pki new-cert --cn foreman --key /etc/icinga2/pki/foreman.key --csr /etc/icinga2/pki/foreman.csr
+# icinga2 pki sign-csr --csr /etc/icinga2/pki/foreman.csr --cert /etc/icinga2/pki/foreman.crt
+```
+
+### Smart Proxy
+
+Ensure that the Monitoring module is enabled and uses the provider monitoring_icinga2.
+It is the default provider so also no setting for use_provider is fine.
+If you configured hosts in Icinga2 only with hostname instead of FQDN, you can add `:strip_domain` with
+all the parts to strip, e.g. `.localdomain`.
+
+```
+# vi /etc/foreman-proxy/settings.d/monitoring.yaml
+---
+:enabled: true
+:use_provider: monitoring_icinga2
+```
+
+Configure the provider with your server details and the API User information.
+Typically you will have to change the server attribute, copy the CA certificate from the server (located
+in /etc/icinga2/pki/) and provide the authentication details of the API User. If using the IP address
+instead of the FQDN of the server, you will have to set verify_ssl to false.
+
+```
+# vi /etc/foreman-proxy/settings.d/monitoring_icinga2.yaml
+---
+:enabled: true
+:server: icinga2.localdomain
+:api_cacert: /usr/share/foreman-proxy/monitoring/ca.crt
+:api_user: foreman
+:api_usercert: /usr/share/foreman-proxy/monitoring/foreman.crt
+:api_userkey: /usr/share/foreman-proxy/monitoring/foreman.key
+#:api_password: foreman
+:verify_ssl: true
+```
+
+
+# TODO
+
+Monitoring:
+* Add host creation and update
+
+Provider Icinga2:
+* Add endpoint and zone management for Icinga 2 as agent
+
+# Copyright
+
+Copyright (c) 2016 The Foreman developers
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+

data/bundler.d/monitoring.rb

@@ -0,0 +1,2 @@
+gem 'smart_proxy_monitoring'
+gem 'rest-client'

data/lib/smart_proxy_monitoring/configuration_loader.rb

@@ -0,0 +1,8 @@
+module ::Proxy::Monitoring
+  class ConfigurationLoader
+    def load_classes
+      require 'smart_proxy_monitoring/dependency_injection'
+      require 'smart_proxy_monitoring/monitoring_api'
+    end
+  end
+end

data/lib/smart_proxy_monitoring/dependency_injection.rb

@@ -0,0 +1,8 @@
+module Proxy::Monitoring
+  module DependencyInjection
+    include Proxy::DependencyInjection::Accessors
+    def container_instance
+      @container_instance ||= ::Proxy::Plugins.instance.find { |p| p[:name] == :monitoring }[:di_container]
+    end
+  end
+end

data/lib/smart_proxy_monitoring/monitoring_api.rb

@@ -0,0 +1,77 @@
+require 'sinatra'
+require 'smart_proxy_monitoring/monitoring_plugin'
+
+module Proxy::Monitoring
+  class Api < ::Sinatra::Base
+    extend Proxy::Monitoring::DependencyInjection
+    inject_attr :monitoring_provider, :server
+
+    include ::Proxy::Log
+    helpers ::Proxy::Helpers
+    authorize_with_trusted_hosts
+    authorize_with_ssl_client
+
+    delete '/host/:host' do |host|
+      begin
+        validate_dns_name!(host)
+        host = strip_domain(host)
+
+        server.remove_host(host)
+      rescue Proxy::Monitoring::NotFound => e
+        log_halt 404, e
+      rescue Proxy::Monitoring::ConnectionError => e
+        log_halt 503, e
+      rescue Exception => e
+        log_halt 400, e
+      end
+    end
+
+    post '/downtime/host/:host?' do |host|
+      author = params[:author] || 'foreman'
+      comment = params[:comment] || 'triggered by foreman'
+      start_time = params[:start_time] || Time.now.to_i
+      end_time = params[:end_time] || (Time.now.to_i + (24 * 3600))
+
+      begin
+        validate_dns_name!(host)
+        host = strip_domain(host)
+
+        server.set_downtime_host(host, author, comment, start_time, end_time)
+      rescue Proxy::Monitoring::NotFound => e
+        log_halt 404, e
+      rescue Proxy::Monitoring::ConnectionError => e
+        log_halt 503, e
+      rescue Exception => e
+        log_halt 400, e
+      end
+    end
+
+    delete '/downtime/host/:host?' do |host|
+      author = params[:author] || 'foreman'
+      comment = params[:comment] || 'triggered by foreman'
+
+      begin
+        validate_dns_name!(host)
+        host = strip_domain(host)
+
+        server.remove_downtime_host(host, author, comment)
+      rescue Proxy::Monitoring::NotFound => e
+        log_halt 404, e
+      rescue Proxy::Monitoring::ConnectionError => e
+        log_halt 503, e
+      rescue Exception => e
+        log_halt 400, e
+      end
+    end
+
+    def validate_dns_name!(name)
+      raise Proxy::Monitoring::Error.new("Invalid DNS name #{name}") unless name =~ /^([a-zA-Z0-9]([-a-zA-Z0-9]+)?\.?)+$/
+    end
+
+    def strip_domain(name)
+      domain = Proxy::Monitoring::Plugin.settings.strip_domain
+      name.slice!(domain) unless domain.nil?
+      name
+    end
+  end
+end

data/lib/smart_proxy_monitoring/monitoring_http_config.ru

@@ -0,0 +1,5 @@
+require 'smart_proxy_monitoring/monitoring_api'
+
+map '/monitoring' do
+  run Proxy::Monitoring::Api
+end

data/lib/smart_proxy_monitoring/monitoring_plugin.rb

@@ -0,0 +1,19 @@
+require 'smart_proxy_monitoring/version'
+
+module Proxy::Monitoring
+  class NotFound < RuntimeError; end
+  class ConnectionError < RuntimeError; end
+  class Error < RuntimeError; end
+
+  class Plugin < ::Proxy::Plugin
+    plugin 'monitoring', Proxy::Monitoring::VERSION
+
+    uses_provider
+    default_settings use_provider: 'monitoring_icinga2'
+
+    http_rackup_path File.expand_path('monitoring_http_config.ru', File.expand_path('../', __FILE__))
+    https_rackup_path File.expand_path('monitoring_http_config.ru', File.expand_path('../', __FILE__))
+
+    load_classes ::Proxy::Monitoring::ConfigurationLoader
+  end
+end

data/lib/smart_proxy_monitoring/version.rb

@@ -0,0 +1,5 @@
+module Proxy
+  module Monitoring
+    VERSION = '0.0.1'.freeze
+  end
+end

data/lib/smart_proxy_monitoring.rb

@@ -0,0 +1,4 @@
+require 'smart_proxy_monitoring/version'
+require 'smart_proxy_monitoring/configuration_loader'
+require 'smart_proxy_monitoring/monitoring_plugin'
+require 'smart_proxy_monitoring_icinga2'

data/lib/smart_proxy_monitoring_common/monitoring_common.rb

@@ -0,0 +1,6 @@
+module Proxy::Monitoring
+  class Error < RuntimeError; end
+  class NotFound < RuntimeError; end
+
+  class Provider; end
+end

data/lib/smart_proxy_monitoring_icinga2/icinga2_api_observer.rb

@@ -0,0 +1,66 @@
+require 'thread'
+require 'socket'
+require 'json'
+
+module ::Proxy::Monitoring::Icinga2
+  class Icinga2ApiObserver
+    include ::Proxy::Log
+    include ::Proxy::Monitoring::Icinga2::Common
+
+    attr_reader :semaphore
+
+    def initialize(queue)
+      @queue = queue.queue
+      @semaphore = Mutex.new
+    end
+
+    def monitor
+      loop do
+        logger.debug "Connecting to Icinga event monitoring api: #{Icinga2Client.baseurl}."
+
+        ssl_socket = Icinga2Client.events_socket('/events?queue=foreman&types=StateChange&types=AcknowledgementSet&types=AcknowledgementCleared&types=DowntimeTriggered&types=DowntimeRemoved')
+
+        logger.info 'Icinga event api monitoring started.'
+
+        while line = ssl_socket.gets
+          next unless line.chars.first == '{'
+
+          with_event_counter('Icinga2 Event API Monitor') do
+            begin
+              parsed = JSON.parse(line)
+              if @queue.size > 100_000
+                @queue.clear
+                logger.error 'Queue was full. Flushing. Events were lost.'
+              end
+              @queue.push(parsed)
+            rescue JSON::ParserError => e
+              logger.error "Icinga2 Event API Monitor: Malformed JSON: #{e.message}"
+            end
+          end
+
+        end
+        logger.info 'Icinga event api monitoring stopped.'
+      end
+    rescue Errno::ECONNREFUSED => e
+      logger.error "Icinga Event Stream: Connection refused. Retrying in 5 seconds. Reason: #{e.message}"
+      sleep 5
+      retry
+    rescue Exception => e
+      logger.error "Error while monitoring: #{e.message}\n#{e.backtrace.join("\n")}"
+      sleep 1
+      retry
+    ensure
+      ssl_socket.sysclose unless ssl_socket.nil?
+    end
+
+    def start
+      @thread = Thread.new { monitor }
+      @thread.abort_on_exception = true
+      @thread
+    end
+
+    def stop
+      @thread.terminate unless @thread.nil?
+    end
+  end
+end

data/lib/smart_proxy_monitoring_icinga2/icinga2_client.rb

@@ -0,0 +1,127 @@
+require 'json'
+require 'uri'
+require 'rest-client'
+require 'thread'
+require 'socket'
+require 'base64'
+
+module ::Proxy::Monitoring::Icinga2
+  class Icinga2Client
+    class << self
+      def client(request_url)
+        headers = {
+          'Accept' => 'application/json'
+        }
+
+        options = {
+          headers: headers,
+          user: user,
+          ssl_ca_file: cacert,
+          verify_ssl: ssl
+        }
+
+        auth_options = if certificate_request?
+                         {
+                           ssl_client_cert: cert,
+                           ssl_client_key: key
+                         }
+                       else
+                         {
+                           password: password
+                         }
+                       end
+        options.merge!(auth_options)
+
+        RestClient::Resource.new(
+          URI.encode([baseurl, request_url].join('')),
+          options
+        )
+      end
+
+      def events_socket(endpoint)
+        uri = URI.parse([baseurl, endpoint].join(''))
+        socket = TCPSocket.new(uri.host, uri.port)
+
+        ssl_context = OpenSSL::SSL::SSLContext.new
+        ssl_context.ca_file = cacert
+
+        if ssl
+          ssl_context.set_params(verify_mode: OpenSSL::SSL::VERIFY_PEER)
+        else
+          ssl_context.set_params(verify_mode: OpenSSL::SSL::VERIFY_NONE)
+        end
+
+        if certificate_request?
+          ssl_context.cert = cert
+          ssl_context.key = key
+        end
+
+        ssl_socket = OpenSSL::SSL::SSLSocket.new(socket, ssl_context)
+        ssl_socket.sync_close = true
+        ssl_socket.connect
+
+        ssl_socket.write "POST #{uri.request_uri} HTTP/1.1\r\n"
+        ssl_socket.write "Accept: application/json\r\n"
+        unless certificate_request?
+          auth = Base64.encode64("#{user}:#{password}")
+          ssl_socket.write "Authorization: Basic #{auth}"
+        end
+        ssl_socket.write "\r\n"
+
+        ssl_socket
+      end
+
+      def get(url)
+        client(url).get
+      end
+
+      def post(url, data)
+        client(url).post(data)
+      end
+
+      def put(url)
+        client(url).put
+      end
+
+      def delete(url)
+        client(url).delete
+      end
+
+      def cert
+        file = Proxy::Monitoring::Icinga2::Plugin.settings.api_usercert
+        return unless !file.nil? && File.file?(file)
+        OpenSSL::X509::Certificate.new(File.read(file))
+      end
+
+      def key
+        file = Proxy::Monitoring::Icinga2::Plugin.settings.api_userkey
+        return unless !file.nil? && File.file?(file)
+        OpenSSL::PKey::RSA.new(File.read(file))
+      end
+
+      def cacert
+        Proxy::Monitoring::Icinga2::Plugin.settings.api_cacert
+      end
+
+      def user
+        Proxy::Monitoring::Icinga2::Plugin.settings.api_user
+      end
+
+      def password
+        Proxy::Monitoring::Icinga2::Plugin.settings.api_password
+      end
+
+      def ssl
+        Proxy::Monitoring::Icinga2::Plugin.settings.verify_ssl
+      end
+
+      def certificate_request?
+        cert && key
+      end
+
+      def baseurl
+        "https://#{Proxy::Monitoring::Icinga2::Plugin.settings.server}:#{Proxy::Monitoring::Icinga2::Plugin.settings.api_port}/v1"
+      end
+    end
+  end
+end

data/lib/smart_proxy_monitoring_icinga2/icinga2_initial_importer.rb

@@ -0,0 +1,109 @@
+require 'thread'
+require 'socket'
+require 'json'
+
+module ::Proxy::Monitoring::Icinga2
+  class Icinga2InitialImporter
+    include ::Proxy::Log
+
+    def initialize(queue)
+      @queue = queue.queue
+    end
+
+    def monitor
+      logger.debug 'Starting initial icinga import.'
+
+      around_action('Initial Host Import') do
+        import_hosts
+      end
+
+      around_action('Initial Services Import') do
+        import_services
+      end
+
+      around_action('Initial Downtimes Import') do
+        import_downtimes
+      end
+
+      logger.info 'Finished initial icinga import.'
+    rescue Exception => e
+      logger.error "Error during initial import: #{e.message}\n#{e.backtrace}"
+    end
+
+    def import_hosts
+      results = Icinga2Client.get('/objects/hosts?attrs=name&attrs=last_check_result&attrs=acknowledgement')
+      results = JSON.parse(results)
+      results['results'].each do |result|
+        parsed = {
+          host: result['attrs']['name'],
+          result: result['attrs']['last_check_result']['state'],
+          timestamp: result['attrs']['last_check_result']['schedule_end'],
+          acknowledged: (result['attrs']['acknowledgement'] != 0),
+          initial: true,
+          type: '_parsed'
+        }
+        @queue.push(parsed)
+      end
+    end
+
+    def import_services
+      results = Icinga2Client.get('/objects/services?attrs=name&attrs=last_check_result&attrs=acknowledgement&attrs=host_name')
+      results = JSON.parse(results)
+      results['results'].each do |result|
+        parsed = {
+          host: result['attrs']['host_name'],
+          service: result['attrs']['name'],
+          result: result['attrs']['last_check_result']['state'],
+          timestamp: result['attrs']['last_check_result']['schedule_end'],
+          acknowledged: (result['attrs']['acknowledgement'] != 0),
+          initial: true,
+          type: '_parsed'
+        }
+        @queue.push(parsed)
+      end
+    end
+
+    def import_downtimes
+      results = Icinga2Client.get('/objects/downtimes?attrs=host_name&attrs=service_name&attrs=trigger_time')
+      results = JSON.parse(results)
+      results['results'].each do |result|
+        next unless result['attrs']['trigger_time'] != 0
+        parsed = {
+          host: result['attrs']['host_name'],
+          service: result['attrs']['service_name'],
+          downtime: true,
+          initial: true,
+          type: '_parsed'
+        }
+        @queue.push(parsed)
+      end
+    end
+
+    def start
+      @thread = Thread.new { monitor }
+      @thread.abort_on_exception = true
+      @thread
+    end
+
+    def stop
+      @thread.terminate unless @thread.nil?
+    end
+
+    private
+
+    def around_action(task)
+      beginning_time = Time.now
+      logger.info "Starting Task: #{task}."
+      yield
+      end_time = Time.now
+      logger.info "Finished Task: #{task} in #{end_time - beginning_time} seconds."
+    rescue Errno::ECONNREFUSED => e
+      logger.error "Icinga Initial Importer: Connection refused in task #{task}. Reason: #{e.message}"
+      logger.error "Icinga Initial Importer: Restarting #{task} in 5 seconds."
+      sleep 5
+      retry
+    rescue JSON::ParserError => e
+      logger.error "Icinga Initial Importer: Failed to parse JSON: #{e.message}"
+    end
+  end
+end