smart_proxy_reports 0.0.5

data/README.md

@@ -0,0 +1,183 @@
+Smart Proxy Reports
+===================
+
+Transforms configuration and security management reports into Foreman-friendly
+JSON and sends them to a Foreman instance. For more information about Foreman
+JSON report format, visit
+[foreman_host_reports](https://github.com/theforeman/foreman_host_reports).
+
+## Usage
+
+Send a POST HTTP call to `/reports/FORMAT` where FORMAT is one of the following formats.
+
+### Puppet
+
+Accepts Puppet Server YAML format:
+
+* [Example input](test/fixtures/puppet6-foreman-web.yaml)
+* [Example output](test/snapshots/foreman-web.json)
+
+## Development setup
+
+Few words about setting up a dev setup.
+
+### Ansible
+
+Checkoud foreman-ansible-modules and build it via `make` command. Configure
+Ansible collection path to the build directory:
+
+    [defaults]
+    collection_path = /home/lzap/work/foreman-ansible-modules/build
+    callback_whitelist = foreman
+    [callback_foreman]
+    report_type = proxy
+    proxy_url = http://localhost:8000/reports
+    verify_certs = 0
+    client_cert = /home/lzap/DummyX509/client-one.crt
+    client_key = /home/lzap/DummyX509/client-one.key
+
+Configure Foreman Ansible callback with the correct Foreman URL:
+
+Then call Ansible:
+
+    ANSIBLE_LOAD_CALLBACK_PLUGINS=1 ansible localhost -m ping -vvv
+
+## Example data
+
+For testing, there are several example data. Before importing them into Foreman, make sure to have `localhost` smart proxy and also a host named `report.example.com`. It is possible to capture example data via `incoming_save_dir` setting. Name generated files correctly and put them into the `contrib/fixtures` directory. There is a utility to use fixtures for development and testing purposes:
+
+```
+$ contrib/upload-fixture
+Usage:
+  contrib/upload-fixture -h               Display this help message
+  contrib/upload-fixture -u URL           Proxy URL (http://localhost:8000)
+  contrib/upload-fixture -f FILE          Fixture to upload
+  contrib/upload-fixture -a               Upload all fixtures
+
+$ contrib/upload-fixture -a
+contrib/fixtures/ansible-copy-nochange.json: 200
+contrib/fixtures/ansible-copy-success.json: 200
+contrib/fixtures/ansible-package-install-failure.json: 200
+contrib/fixtures/ansible-package-install-nochange.json: 200
+contrib/fixtures/ansible-package-install-success.json: 200
+contrib/fixtures/ansible-package-remove-failure.json: 200
+contrib/fixtures/ansible-package-remove-success.json: 200
+```
+
+### Importing into Foreman directly
+
+To import a report directly info Foreman:
+
+```
+curl -H "Accept:application/json,version=2" -H "Content-Type:application/json" -X POST -d @test/snapshots/foreman-web.json http://localhost:5000/api/v2/host_reports
+```
+
+### Puppet
+
+To install and configure a Puppetserver on EL7, run the following:
+
+```bash
+# Install the server - modify as needed for your platform
+yum -y install https://yum.puppet.com/puppet7-release-el-7.noarch.rpm
+yum -y install puppetserver
+# Correct $PATH in the current shell - happens on start of fresh shells automatically
+source /etc/profile.d/puppet-agent.sh
+# Configure the HTTP report processor
+puppet config set reports store,http
+puppet config set reporturl http://$HOSTNAME:8000/reports/puppet
+# Enable & start the service
+systemctl enable --now puppetserver
+```
+
+If you prefer to use HTTPS, set the different reporturl and configure the CA certificates according to the example below
+```
+# use HTTPS, without Katello the port is 8443, with Katello it's 9090
+puppet config set reporturl https://$HOSTNAME:8443/reports/puppet
+# install the Smart Proxy CA certificate to the Puppet's localcacert store
+## first find the correct pem file
+grep :ssl_ca_file /etc/foreman-proxy/settings.yml
+## find the localcacert puppet storage
+puppet config print --section server localcacert
+## then copy the content of both pem files to each other
+cp /etc/foreman-proxy/ssl_ca.pem /tmp/smart-proxy.pem
+cp /etc/puppetlabs/puppet/ssl/certs/ca.pem /tmp/puppet-ca.pem
+cat /tmp/smart-proxy.pem >> /etc/puppetlabs/puppet/ssl/certs/ca.pem
+cat /tmp/puppet-ca.pem >> /etc/foreman-proxy/ssl_ca.pem
+# restart the services
+systemctl restart puppetserver
+systemctl restart foreman-proxy
+```
+Note that this means that the Puppetserver API will trust client certificates signed by the Smart Proxy CA
+certificate and will be subject to authorization defined in puppet's auth.conf, e.g. a client with the certificate
+of the same cname can get a catalog for such node. That is typically not a bad thing but you need to consider the
+implications in your SSL certificates layout. Similarly the proxy will now trust certificates signed by the
+Puppet CA, however they are still subject to smart proxy trusted hosts authorization.
+
+By default an agent connects to `puppet` which may not resolve. Set it to your hostname:
+
+```bash
+puppet config set server $HOSTNAME
+```
+
+You can manually trigger a puppet run by using `puppet agent -t`. You may need to look at `/var/log/puppetlabs/puppetserver/puppetserver.log` to see errors.
+
+## Status mapping
+
+### Puppet
+
+* changed -> applied
+* corrective_change -> applied
+* failed -> failed
+* failed_to_restart -> failed
+* scheduled -> pending
+* restarted -> other
+* skipped -> other
+* out_of_sync
+* total
+
+### Ansible
+
+* applied -> applied
+* failed -> failed
+* skipped -> other
+* pending -> pending
+
+## Contributing
+
+Fork and send a Pull Request. Thanks!
+
+### Unit tests
+
+To run unit tests:
+
+    bundle exec rake test
+
+There are few snapshot tests which compare input JSON/YAML with snapshot fixtures. When they fail, you are asked to delete those fixtures, re-run tests, review and push the changes into git:
+
+```
+rm test/snapshots/*
+bundle exec rake test
+git diff
+git commit
+```
+
+## License
+
+GNU GPLv3, see LICENSE file for more information.
+
+## Copyright
+
+Copyright (c) 2021 Red Hat, Inc.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.

data/bundler.d/reports.rb

@@ -0,0 +1 @@
+gem "smart_proxy_reports"

data/lib/smart_proxy_reports/ansible_processor.rb

@@ -0,0 +1,149 @@
+# frozen_string_literal: true
+
+module Proxy::Reports
+  class AnsibleProcessor < Processor
+    KEYS_TO_COPY = %w[status check_mode].freeze
+
+    def initialize(data, json_body: true)
+      super(data, json_body: json_body)
+      measure :parse do
+        @data = JSON.parse(data)
+      end
+      @body = {}
+      logger.debug "Processing report #{report_id}"
+      debug_payload("Input", @data)
+    end
+
+    def report_id
+      @data["uuid"] || generated_report_id
+    end
+
+    def process_results
+      @data["results"]&.each do |result|
+        process_facts(result)
+        process_level(result)
+        process_message(result)
+        process_keywords(result)
+      end
+      @data["results"]
+    rescue StandardError => e
+      logger.error "Unable to parse results", e
+      @data["results"]
+    end
+
+    def process
+      measure :process do
+        @body["format"] = "ansible"
+        @body["id"] = report_id
+        @body["host"] = hostname_from_config || @data["host"]
+        @body["proxy"] = Proxy::Reports::Plugin.settings.reported_proxy_hostname
+        @body["reported_at"] = @data["reported_at"]
+        @body["results"] = process_results
+        @body["keywords"] = keywords
+        @body["telemetry"] = telemetry
+        @body["errors"] = errors if errors?
+        KEYS_TO_COPY.each do |key|
+          @body[key] = @data[key]
+        end
+      end
+    end
+
+    def build_report
+      process
+      if debug_payload?
+        logger.debug { JSON.pretty_generate(@body) }
+      end
+      build_report_root(
+        format: "ansible",
+        version: 1,
+        host: @body["host"],
+        reported_at: @body["reported_at"],
+        statuses: process_statuses,
+        proxy: @body["proxy"],
+        body: @body,
+        keywords: @body["keywords"],
+      )
+    end
+
+    def spool_report
+      report_hash = build_report
+      debug_payload("Output", report_hash)
+      payload = measure :format do
+        report_hash.to_json
+      end
+      SpooledHttpClient.instance.spool(report_id, payload)
+    end
+
+    private
+
+    def process_facts(result)
+      # TODO: add fact processing and sending to the fact endpoint
+      result["result"]["ansible_facts"] = {}
+    end
+
+    def process_keywords(result)
+      if result["failed"]
+        add_keywords("HasFailure", "AnsibleTaskFailed:#{result["task"]["action"]}")
+      elsif result["result"]["changed"]
+        add_keywords("HasChange")
+      end
+    end
+
+    def process_level(result)
+      if result["failed"]
+        result["level"] = "err"
+      elsif result["result"]["changed"]
+        result["level"] = "notice"
+      else
+        result["level"] = "info"
+      end
+    end
+
+    def process_message(result)
+      msg = "N/A"
+      return result["friendly_message"] = msg if result["task"].nil? || result["task"]["action"].nil?
+      return result["friendly_message"] = result["result"]["msg"] if result["failed"]
+      result_tree = result["result"]
+      task_tree = result["task"]
+      raise("Report do not contain required 'results/result' element") unless result_tree
+      raise("Report do not contain required 'results/task' element") unless task_tree
+      module_args_tree = result_tree.dig("invocation", "module_args")
+
+      case task_tree["action"]
+      when "ansible.builtin.package", "package"
+        detail = result_tree["results"] || result_tree["msg"] || "No details"
+        msg = "Package(s) #{module_args_tree["name"].join(",")} are #{module_args_tree["state"]}: #{detail}"
+      when "ansible.builtin.template", "template"
+        msg = "Render template #{module_args_tree["_original_basename"]} to #{result_tree["dest"]}"
+      when "ansible.builtin.service", "service"
+        msg = "Service #{result_tree["name"]} is #{result_tree["state"]} and enabled: #{result_tree["enabled"]}"
+      when "ansible.builtin.group", "group"
+        msg = "User group #{result_tree["name"]} is #{result_tree["state"]} with gid: #{result_tree["gid"]}"
+      when "ansible.builtin.user", "user"
+        msg = "User #{result_tree["name"]} is #{result_tree["state"]} with uid: #{result_tree["uid"]}"
+      when "ansible.builtin.cron", "cron"
+        msg = "Cron job: #{module_args_tree["minute"]} #{module_args_tree["hour"]} #{module_args_tree["day"]} #{module_args_tree["month"]} #{module_args_tree["weekday"]} #{module_args_tree["job"]} and disabled: #{module_args_tree["disabled"]}"
+      when "ansible.builtin.copy", "copy"
+        msg = "Copy #{module_args_tree["_original_basename"]} to #{result_tree["dest"]}"
+      when "ansible.builtin.command", "ansible.builtin.shell", "command", "shell"
+        msg = result_tree["stdout_lines"]
+      end
+    rescue StandardError => e
+      logger.debug "Unable to parse result (#{e.message}): #{result.inspect}"
+    ensure
+      result["friendly_message"] = msg
+    end
+
+    def process_statuses
+      {
+        "applied" => @body["status"]["applied"],
+        "failed" => @body["status"]["failed"],
+        "pending" => @body["status"]["pending"] || 0, # It's only present in check mode
+        "other" => @body["status"]["skipped"],
+      }
+    rescue StandardError => e
+      logger.error "Unable to process statuses", e
+      { "applied" => 0, "failed" => 0, "pending" => 0, "other" => 0 }
+    end
+  end
+end

data/lib/smart_proxy_reports/processor.rb

@@ -0,0 +1,103 @@
+# frozen_string_literal: true
+require "pp"
+require "proxy/log"
+
+module Proxy::Reports
+  class Processor
+    include ::Proxy::Log
+
+    def self.new_processor(format, data, json_body: true)
+      case format
+      when "puppet"
+        PuppetProcessor.new(data, json_body: json_body)
+      when "ansible"
+        AnsibleProcessor.new(data, json_body: json_body)
+      else
+        NotImplementedError.new
+      end
+    end
+
+    def initialize(*, json_body: true)
+      @keywords_set = {}
+      @errors = []
+      @json_body = json_body
+    end
+
+    def generated_report_id
+      @generated_report_id ||= SecureRandom.uuid
+    end
+
+    def hostname_from_config
+      @hostname_from_config ||= Proxy::Reports::Plugin.settings.override_hostname
+    end
+
+    def build_report_root(format:, version:, host:, reported_at:, statuses:, proxy:, body:, keywords:)
+      {
+        "host_report" => {
+          "format" => format,
+          "version" => version,
+          "host" => host,
+          "reported_at" => reported_at,
+          "proxy" => proxy,
+          "body" => @json_body ? body.to_json : body,
+          "keywords" => keywords,
+        }.merge(statuses),
+      }
+      # TODO add metric with total time
+    end
+
+    def debug_payload?
+      Proxy::Reports::Plugin.settings.debug_payload
+    end
+
+    def debug_payload(prefix, data)
+      return unless debug_payload?
+      logger.debug { "#{prefix}: #{data.pretty_inspect}" }
+    end
+
+    def add_keywords(*keywords)
+      keywords.each do |keyword|
+        @keywords_set[keyword] = true
+      end
+    end
+
+    def keywords
+      @keywords_set.keys.to_a rescue []
+    end
+
+    attr_reader :errors
+
+    def log_error(message)
+      @errors << message.to_s
+    end
+
+    def errors?
+      @errors&.any?
+    end
+
+    # TODO support multiple metrics and adding total time
+    attr_reader :telemetry
+
+    def measure(metric)
+      t1 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      yield
+    ensure
+      t2 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      @telemetry ||= {}
+      @telemetry[metric.to_s] = (t2 - t1) * 1000
+    end
+
+    def telemetry_as_string
+      result = []
+      telemetry.each do |key, value|
+        result << "#{key}=#{value.round(1)}ms"
+      end
+      result.join(", ")
+    end
+
+    def spool_report
+      super
+      logger.debug "Spooled #{report_id}: #{telemetry_as_string}"
+    end
+  end
+end

data/lib/smart_proxy_reports/puppet_processor.rb

@@ -0,0 +1,142 @@
+# frozen_string_literal: true
+
+module Proxy::Reports
+  class PuppetProcessor < Processor
+    YAML_CLEAN = /!ruby\/object.*$/.freeze
+    KEYS_TO_COPY = %w[report_format puppet_version environment metrics].freeze
+    MAX_EVAL_TIMES = 29
+
+    def initialize(data, json_body: true)
+      super(data, json_body: json_body)
+      measure :parse do
+        if Gem::Version.new(RUBY_VERSION) < Gem::Version.new("2.6.0")
+          # Ruby 2.5 or older does not have permitted_classes argument available
+          @data = YAML.load(data.gsub(YAML_CLEAN, ""))
+        else
+          @data = YAML.safe_load(data.gsub(YAML_CLEAN, ""), permitted_classes: [Symbol, Time, Date])
+        end
+      end
+      raise("No content") unless @data
+      @body = {}
+      @evaluation_times = []
+      logger.debug "Processing report #{report_id}"
+      debug_payload("Input", @data)
+    end
+
+    def report_id
+      @data["transaction_uuid"] || generated_report_id
+    end
+
+    def process_logs
+      logs = []
+      @data["logs"]&.each do |log|
+        logs << [log["level"]&.to_s, log["source"], log["message"]]
+      end
+      logs
+    rescue StandardError => e
+      logger.error "Unable to parse logs", e
+      logs
+    end
+
+    def process_resource_statuses
+      statuses = []
+      @data["resource_statuses"]&.each_pair do |key, value|
+        statuses << key
+        @evaluation_times << [key, value["evaluation_time"]]
+        # failures
+        add_keywords("PuppetResourceFailed:#{key}", "PuppetHasFailure") if value["failed"] || value["failed_to_restart"]
+        value["events"]&.each do |event|
+          add_keywords("PuppetResourceFailed:#{key}", "PuppetHasFailure") if event["status"] == "failed"
+          add_keywords("PuppetHasCorrectiveChange") if event["corrective_change"]
+        end
+        # changes
+        add_keywords("PuppetHasChange") if value["changed"]
+        add_keywords("PuppetHasChange") if value["change_count"] && value["change_count"] > 0
+        # changes
+        add_keywords("PuppetIsOutOfSync") if value["out_of_sync"]
+        add_keywords("PuppetIsOutOfSync") if value["out_of_sync_count"] && value["out_of_sync_count"] > 0
+        # skips
+        add_keywords("PuppetHasSkips") if value["skipped"]
+        # corrective changes
+        add_keywords("PuppetHasCorrectiveChange") if value["corrective_change"]
+      end
+      statuses
+    rescue StandardError => e
+      logger.error "Unable to parse resource_statuses", e
+      statuses
+    end
+
+    def process_evaluation_times
+      @evaluation_times.sort! { |a, b| b[1] <=> a[1] }
+      if @evaluation_times.count > MAX_EVAL_TIMES
+        others = @evaluation_times[MAX_EVAL_TIMES..@evaluation_times.count - 1].sum { |x| x[1] }
+        @evaluation_times = @evaluation_times[0..MAX_EVAL_TIMES - 1]
+        @evaluation_times << ["Others", others] if others > 0.0001
+      end
+      @evaluation_times
+    rescue StandardError => e
+      logger.error "Unable to process evaluation_times", e
+      []
+    end
+
+    def process
+      measure :process do
+        @body["format"] = "puppet"
+        @body["id"] = report_id
+        @body["host"] = hostname_from_config || @data["host"]
+        @body["proxy"] = Proxy::Reports::Plugin.settings.reported_proxy_hostname
+        @body["reported_at"] = @data["time"]
+        KEYS_TO_COPY.each do |key|
+          @body[key] = @data[key]
+        end
+        @body["logs"] = process_logs
+        @body["resource_statuses"] = process_resource_statuses
+        @body["keywords"] = keywords
+        @body["evaluation_times"] = process_evaluation_times
+        @body["telemetry"] = telemetry
+        @body["errors"] = errors if errors?
+      end
+    end
+
+    def build_report
+      process
+      if debug_payload?
+        logger.debug { JSON.pretty_generate(@body) }
+      end
+      build_report_root(
+        format: "puppet",
+        version: 1,
+        host: @body["host"],
+        reported_at: @body["reported_at"],
+        statuses: process_statuses,
+        proxy: @body["proxy"],
+        body: @body,
+        keywords: @body["keywords"],
+      )
+    end
+
+    def spool_report
+      report_hash = build_report
+      debug_payload("Output", report_hash)
+      payload = measure :format do
+        report_hash.to_json
+      end
+      SpooledHttpClient.instance.spool(report_id, payload)
+    end
+
+    private
+
+    def process_statuses
+      stats = @body["metrics"]["resources"]["values"].collect { |s| [s[0], s[2]] }.to_h
+      {
+        "applied" => stats["changed"] + stats["corrective_change"],
+        "failed" => stats["failed"] + stats["failed_to_restart"],
+        "pending" => stats["scheduled"],
+        "other" => stats["restarted"] + stats["skipped"] + stats["out_of_sync"],
+      }
+    rescue StandardError => e
+      logger.error "Unable to process statuses", e
+      { "applied" => 0, "failed" => 0, "pending" => 0, "other" => 0 }
+    end
+  end
+end

data/lib/smart_proxy_reports/reports.rb

@@ -0,0 +1,30 @@
+require "socket"
+
+module Proxy::Reports
+  class PluginConfiguration
+    def load_classes
+      require "smart_proxy_reports/spooled_http_client"
+    end
+
+    def load_dependency_injection_wirings(container, _settings)
+      container.singleton_dependency :reports_spool, -> {
+          SpooledHttpClient.instance.initialize_directory
+        }
+    end
+  end
+
+  class Plugin < ::Proxy::Plugin
+    plugin :reports, Proxy::Reports::VERSION
+
+    default_settings reported_proxy_hostname: Socket.gethostname(),
+      debug_payload: false,
+      spool_dir: "/var/lib/foreman-proxy/reports",
+      keep_reports: false
+
+    rackup_path File.expand_path("reports_http_config.ru", __dir__)
+
+    load_classes PluginConfiguration
+    load_dependency_injection_wirings PluginConfiguration
+    start_services :reports_spool
+  end
+end

data/lib/smart_proxy_reports/reports_api.rb

@@ -0,0 +1,55 @@
+require "sinatra"
+require "yaml"
+require "smart_proxy_reports/reports"
+require "smart_proxy_reports/processor"
+require "smart_proxy_reports/puppet_processor"
+require "smart_proxy_reports/ansible_processor"
+
+module Proxy::Reports
+  class Api < ::Sinatra::Base
+    include ::Proxy::Log
+    include ::Proxy::Util
+    helpers ::Proxy::Helpers
+    authorize_with_trusted_hosts
+    authorize_with_ssl_client
+
+    before do
+      content_type "application/json"
+    end
+
+    def check_content_type(format)
+      request_type = request.env["CONTENT_TYPE"]
+      if format == "puppet"
+        log_halt(415, "Content type must be application/x-yaml, was: #{request_type}") unless request_type.start_with?("application/x-yaml")
+      elsif format == "ansible"
+        log_halt(415, "Content type must be application/json, was: #{request_type}") unless request_type.start_with?("application/json")
+      else
+        log_halt(415, "Unknown format: #{format}")
+      end
+    end
+
+    EXTS = {
+      puppet: "yaml",
+      ansible: "json",
+    }.freeze
+
+    def save_payload(input, format)
+      filename = File.join(Proxy::Reports::Plugin.settings.incoming_save_dir, "#{format}-#{Time.now.to_f}.#{EXTS[format.to_sym]}")
+      File.open(filename, "w") { |f| f.write(input) }
+    end
+
+    post "/:format" do
+      format = params[:format]
+      log_halt(404, "Format argument not specified") unless format
+      check_content_type(format)
+      input = request.body.read
+      save_payload(input, format) if Proxy::Reports::Plugin.settings.incoming_save_dir
+      log_halt(415, "Missing body") if input.empty?
+      json_body = to_bool(params[:json_body], true)
+      processor = Processor.new_processor(format, input, json_body: json_body)
+      processor.spool_report
+    rescue => e
+      log_halt 415, e, "Error during report processing: #{e.message}"
+    end
+  end
+end

data/lib/smart_proxy_reports/reports_http_config.ru

@@ -0,0 +1,10 @@
+require "smart_proxy_reports/reports_api"
+
+map "/reports" do
+  run Proxy::Reports::Api
+end
+
+# TODO: Deprecated, remove in 1.0
+map "/host_reports" do
+  run Proxy::Reports::Api
+end