smart_proxy_host_reports 0.0.1

data/README.md

@@ -0,0 +1,145 @@
+# Smart Proxy Host Reports
+
+Transforms configuration and security management reports into Foreman-friendly
+JSON and sends them to a Foreman instance. For more information about Foreman
+JSON report format, visit
+[foreman_host_reports](https://github.com/theforeman/foreman_host_reports).
+
+## Usage
+
+Send a POST HTTP call to `/host_reports/FORMAT` where FORMAT is one of the following formats.
+
+### Puppet
+
+Accepts Puppet Server YAML format:
+
+* [Example input](test/fixtures/puppet6-foreman-web.yaml)
+* [Example output](test/snapshots/foreman-web.json)
+
+## Development setup
+
+Few words about setting up a dev setup.
+
+### Ansible
+
+Checkoud foreman-ansible-modules and build it via `make` command. Configure
+Ansible collection path to the build directory:
+
+    [defaults]
+    collection_path = /home/lzap/work/foreman-ansible-modules/build
+    callback_whitelist = foreman
+    [callback_foreman]
+    url = http://localhost:8448/host_reports
+    verify_certs = 0
+    client_cert = /home/lzap/DummyX509/client-one.crt
+    client_key = /home/lzap/DummyX509/client-one.key
+
+Configure Foreman Ansible callback with the correct Foreman URL:
+
+Then call Ansible:
+
+    ANSIBLE_LOAD_CALLBACK_PLUGINS=1 ansible localhost -m ping -vvv
+
+## Example data
+
+For testing, there are several example data. Before importing them into Foreman, make sure to have `localhost` smart proxy and also a host named `report.example.com`. It is possible to capture example data via `incoming_save_dir` setting. Name generated files correctly and put them into the `contrib/fixtures` directory. There is a utility to use fixtures for development and testing purposes:
+
+```
+$ contrib/upload-fixture
+Usage:
+  contrib/upload-fixture -h               Display this help message
+  contrib/upload-fixture -u URL           Proxy URL (http://localhost:8448)
+  contrib/upload-fixture -f FILE          Fixture to upload
+  contrib/upload-fixture -a               Upload all fixtures
+
+$ contrib/upload-fixture -a
+contrib/fixtures/ansible-copy-nochange.json: 200
+contrib/fixtures/ansible-copy-success.json: 200
+contrib/fixtures/ansible-package-install-failure.json: 200
+contrib/fixtures/ansible-package-install-nochange.json: 200
+contrib/fixtures/ansible-package-install-success.json: 200
+contrib/fixtures/ansible-package-remove-failure.json: 200
+contrib/fixtures/ansible-package-remove-success.json: 200
+```
+
+### Importing into Foreman directly
+
+To import a report directly info Foreman:
+
+```
+curl -H "Accept:application/json,version=2" -H "Content-Type:application/json" -X POST -d @test/snapshots/foreman-web.json http://localhost:5000/api/v2/host_reports
+```
+
+### Puppet
+
+To install and configure a Puppetserver on EL7, run the following:
+
+```bash
+# Install the server - modify as needed for your platform
+yum -y install https://yum.puppet.com/puppet7-release-el-7.noarch.rpm
+yum -y install puppetserver
+# Correct $PATH in the current shell - happens on start of fresh shells automatically
+source /etc/profile.d/puppet-agent.sh
+# Configure the HTTP report processor
+puppet config set reports store,http
+puppet config set reporturl http://$HOSTNAME:8000/host_reports/puppet
+# Enable & start the service
+systemctl enable --now puppetserver
+```
+
+If you prefer to use HTTPS, set the different reporturl and configure the CA certificates according to the example below
+```
+# use HTTPS, without Katello the port is 8443, with Katello it's 9090
+puppet config set reporturl https://$HOSTNAME:8443/host_reports/puppet
+# install the Smart Proxy CA certificate to the Puppet's localcacert store
+## first find the correct pem file
+grep :ssl_ca_file /etc/foreman-proxy/settings.yml
+## find the localcacert puppet storage
+puppet config print --section server localcacert
+## then copy the content of both pem files to each other
+cp /etc/foreman-proxy/ssl_ca.pem /tmp/smart-proxy.pem
+cp /etc/puppetlabs/puppet/ssl/certs/ca.pem /tmp/puppet-ca.pem
+cat /tmp/smart-proxy.pem >> /etc/puppetlabs/puppet/ssl/certs/ca.pem
+cat /tmp/puppet-ca.pem >> /etc/foreman-proxy/ssl_ca.pem
+# restart the services
+systemctl restart puppetserver
+systemctl restart foreman-proxy
+```
+Note that this means that the Puppetserver API will trust client certificates signed by the Smart Proxy CA
+certificate and will be subject to authorization defined in puppet's auth.conf, e.g. a client with the certificate
+of the same cname can get a catalog for such node. That is typically not a bad thing but you need to consider the
+implications in your SSL certificates layout. Similarly the proxy will now trust certificates signed by the
+Puppet CA, however they are still subject to smart proxy trusted hosts authorization.
+
+By default an agent connects to `puppet` which may not resolve. Set it to your hostname:
+
+```bash
+puppet config set server $HOSTNAME
+```
+
+You can manually trigger a puppet run by using `puppet agent -t`. You may need to look at `/var/log/puppetlabs/puppetserver/puppetserver.log` to see errors.
+
+## Contributing
+
+Fork and send a Pull Request. Thanks!
+
+## License
+
+GNU GPLv3, see LICENSE file for more information.
+
+## Copyright
+
+Copyright (c) 2021 Red Hat, Inc.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.

data/bundler.d/host_reports.rb

@@ -0,0 +1 @@
+gem "smart_proxy_host_reports"

data/lib/smart_proxy_host_reports/ansible_processor.rb

@@ -0,0 +1,135 @@
+# frozen_string_literal: true
+
+class AnsibleProcessor < Processor
+  KEYS_TO_COPY = %w[status check_mode].freeze
+
+  def initialize(data, json_body: true)
+    super(data, json_body: json_body)
+    measure :parse do
+      @data = JSON.parse(data)
+    end
+    @body = {}
+    logger.debug "Processing report #{report_id}"
+    debug_payload("Input", @data)
+  end
+
+  def report_id
+    @data["uuid"] || generated_report_id
+  end
+
+  def process_results
+    @data["results"]&.each do |result|
+      process_facts(result)
+      process_level(result)
+      process_message(result)
+      process_keywords(result)
+    end
+    @data["results"]
+  rescue StandardError => e
+    logger.error "Unable to parse results", e
+    @data["results"]
+  end
+
+  def process
+    measure :process do
+      @body["format"] = "ansible"
+      @body["id"] = report_id
+      @body["host"] = hostname_from_config || @data["host"]
+      @body["proxy"] = Proxy::HostReports::Plugin.settings.reported_proxy_hostname
+      @body["reported_at"] = @data["reported_at"]
+      @body["results"] = process_results
+      @body["keywords"] = keywords
+      @body["telemetry"] = telemetry
+      @body["errors"] = errors if errors?
+      KEYS_TO_COPY.each do |key|
+        @body[key] = @data[key]
+      end
+    end
+  end
+
+  def build_report
+    process
+    if debug_payload?
+      logger.debug { JSON.pretty_generate(@body) }
+    end
+    build_report_root(
+      format: "ansible",
+      version: 1,
+      host: @body["host"],
+      reported_at: @body["reported_at"],
+      status: 0,
+      proxy: @body["proxy"],
+      body: @body,
+      keywords: @body["keywords"],
+    )
+  end
+
+  def spool_report
+    report_hash = build_report
+    debug_payload("Output", report_hash)
+    payload = measure :format do
+      report_hash.to_json
+    end
+    SpooledHttpClient.instance.spool(report_id, payload)
+  end
+
+  private
+
+  def process_facts(result)
+    # TODO: add fact processing and sending to the fact endpoint
+    result["result"]["ansible_facts"] = {}
+  end
+
+  def process_keywords(result)
+    if result["failed"]
+      add_keywords("HasFailure", "AnsibleTaskFailed:#{result["task"]["action"]}")
+    elsif result["result"]["changed"]
+      add_keywords("HasChange")
+    end
+  end
+
+  def process_level(result)
+    if result["failed"]
+      result["level"] = "err"
+    elsif result["result"]["changed"]
+      result["level"] = "notice"
+    else
+      result["level"] = "info"
+    end
+  end
+
+  def process_message(result)
+    msg = "N/A"
+    return result["friendly_message"] = msg if result["task"].nil? || result["task"]["action"].nil?
+    return result["friendly_message"] = result["result"]["msg"] if result["failed"]
+    result_tree = result["result"]
+    task_tree = result["task"]
+    raise("Report do not contain required 'results/result' element") unless result_tree
+    raise("Report do not contain required 'results/task' element") unless task_tree
+    module_args_tree = result_tree.dig("invocation", "module_args")
+
+    case task_tree["action"]
+    when "ansible.builtin.package", "package"
+      detail = result_tree["results"] || result_tree["msg"] || "No details"
+      msg = "Package(s) #{module_args_tree["name"].join(",")} are #{module_args_tree["state"]}: #{detail}"
+    when "ansible.builtin.template", "template"
+      msg = "Render template #{module_args_tree["_original_basename"]} to #{result_tree["dest"]}"
+    when "ansible.builtin.service", "service"
+      msg = "Service #{result_tree["name"]} is #{result_tree["state"]} and enabled: #{result_tree["enabled"]}"
+    when "ansible.builtin.group", "group"
+      msg = "User group #{result_tree["name"]} is #{result_tree["state"]} with gid: #{result_tree["gid"]}"
+    when "ansible.builtin.user", "user"
+      msg = "User #{result_tree["name"]} is #{result_tree["state"]} with uid: #{result_tree["uid"]}"
+    when "ansible.builtin.cron", "cron"
+      msg = "Cron job: #{module_args_tree["minute"]} #{module_args_tree["hour"]} #{module_args_tree["day"]} #{module_args_tree["month"]} #{module_args_tree["weekday"]} #{module_args_tree["job"]} and disabled: #{module_args_tree["disabled"]}"
+    when "ansible.builtin.copy", "copy"
+      msg = "Copy #{module_args_tree["_original_basename"]} to #{result_tree["dest"]}"
+    when "ansible.builtin.command", "ansible.builtin.shell", "command", "shell"
+      msg = result_tree["stdout_lines"]
+    end
+  rescue StandardError => e
+    logger.debug "Unable to parse result (#{e.message}): #{result.inspect}"
+  ensure
+    result["friendly_message"] = msg
+  end
+end

data/lib/smart_proxy_host_reports/host_reports.rb

@@ -0,0 +1,29 @@
+module Proxy::HostReports
+  class PluginConfiguration
+    def load_classes
+      require "smart_proxy_host_reports/spooled_http_client"
+    end
+
+    def load_dependency_injection_wirings(container, _settings)
+      container.singleton_dependency :host_reports_spool, -> {
+          SpooledHttpClient.instance.initialize_directory
+        }
+    end
+  end
+
+  class Plugin < ::Proxy::Plugin
+    plugin :host_reports, Proxy::HostReports::VERSION
+
+    default_settings reported_proxy_hostname: "localhost",
+      debug_payload: false,
+      spool_dir: "/var/lib/foreman-proxy",
+      keep_reports: false
+
+    http_rackup_path File.expand_path("host_reports_http_config.ru", File.expand_path("../", __FILE__))
+    https_rackup_path File.expand_path("host_reports_http_config.ru", File.expand_path("../", __FILE__))
+
+    load_classes PluginConfiguration
+    load_dependency_injection_wirings PluginConfiguration
+    start_services :host_reports_spool
+  end
+end

data/lib/smart_proxy_host_reports/host_reports_api.rb

@@ -0,0 +1,53 @@
+require "sinatra"
+require "yaml"
+require "smart_proxy_host_reports/host_reports"
+require "smart_proxy_host_reports/processor"
+require "smart_proxy_host_reports/puppet_processor"
+require "smart_proxy_host_reports/ansible_processor"
+
+module Proxy::HostReports
+  class Api < ::Sinatra::Base
+    include ::Proxy::Log
+    include ::Proxy::Util
+    helpers ::Proxy::Helpers
+
+    before do
+      content_type "application/json"
+    end
+
+    def check_content_type(format)
+      request_type = request.env["CONTENT_TYPE"]
+      if format == "puppet"
+        log_halt(415, "Content type must be application/x-yaml, was: #{request_type}") unless request_type.start_with?("application/x-yaml")
+      elsif format == "ansible"
+        log_halt(415, "Content type must be application/json, was: #{request_type}") unless request_type.start_with?("application/json")
+      else
+        log_halt(415, "Unknown format: #{format}")
+      end
+    end
+
+    EXTS = {
+      puppet: "yaml",
+      ansible: "json",
+    }.freeze
+
+    def save_payload(input, format)
+      filename = File.join(Proxy::HostReports::Plugin.settings.incoming_save_dir, "#{format}-#{Time.now.to_i}.#{EXTS[format.to_sym]}")
+      File.open(filename, "w") { |f| f.write(input) }
+    end
+
+    post "/:format" do
+      format = params[:format]
+      log_halt(404, "Format argument not specified") unless format
+      check_content_type(format)
+      input = request.body.read
+      save_payload(input, format) if Proxy::HostReports::Plugin.settings.incoming_save_dir
+      log_halt(415, "Missing body") if input.empty?
+      json_body = to_bool(params[:json_body], true)
+      processor = Processor.new_processor(format, input, json_body: json_body)
+      processor.spool_report
+    rescue => e
+      log_halt 415, e, "Error during report processing: #{e.message}"
+    end
+  end
+end

data/lib/smart_proxy_host_reports/host_reports_http_config.ru

@@ -0,0 +1,5 @@
+require "smart_proxy_host_reports/host_reports_api"
+
+map "/host_reports" do
+  run Proxy::HostReports::Api
+end

data/lib/smart_proxy_host_reports/processor.rb

@@ -0,0 +1,104 @@
+# frozen_string_literal: true
+require "pp"
+require "proxy/log"
+
+# TODO: move everything into a module
+
+class Processor
+  include ::Proxy::Log
+
+  def self.new_processor(format, data, json_body: true)
+    case format
+    when "puppet"
+      PuppetProcessor.new(data, json_body: json_body)
+    when "ansible"
+      AnsibleProcessor.new(data, json_body: json_body)
+    else
+      NotImplementedError.new
+    end
+  end
+
+  def initialize(*, json_body: true)
+    @keywords_set = {}
+    @errors = []
+    @json_body = json_body
+  end
+
+  def generated_report_id
+    @generated_report_id ||= SecureRandom.uuid
+  end
+
+  def hostname_from_config
+    @hostname_from_config ||= Proxy::HostReports::Plugin.settings.override_hostname
+  end
+
+  def build_report_root(format:, version:, host:, reported_at:, status:, proxy:, body:, keywords:)
+    {
+      "host_report" => {
+        "format" => format,
+        "version" => version,
+        "host" => host,
+        "reported_at" => reported_at,
+        "status" => status,
+        "proxy" => proxy,
+        "body" => @json_body ? body.to_json : body,
+        "keywords" => keywords,
+      },
+    }
+    # TODO add metric with total time
+  end
+
+  def debug_payload?
+    Proxy::HostReports::Plugin.settings.debug_payload
+  end
+
+  def debug_payload(prefix, data)
+    return unless debug_payload?
+    logger.debug { "#{prefix}: #{data.pretty_inspect}" }
+  end
+
+  def add_keywords(*keywords)
+    keywords.each do |keyword|
+      @keywords_set[keyword] = true
+    end
+  end
+
+  def keywords
+    @keywords_set.keys.to_a rescue []
+  end
+
+  attr_reader :errors
+
+  def log_error(message)
+    @errors << message.to_s
+  end
+
+  def errors?
+    @errors&.any?
+  end
+
+  # TODO support multiple metrics and adding total time
+  attr_reader :telemetry
+
+  def measure(metric)
+    t1 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    yield
+  ensure
+    t2 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    @telemetry ||= {}
+    @telemetry[metric] = (t2 - t1) * 1000
+  end
+
+  def telemetry_as_string
+    result = []
+    telemetry.each do |key, value|
+      result << "#{key}=#{value.round(1)}ms"
+    end
+    result.join(", ")
+  end
+
+  def spool_report
+    super
+    logger.debug "Spooled #{report_id}: #{telemetry_as_string}"
+  end
+end

data/lib/smart_proxy_host_reports/puppet_processor.rb

@@ -0,0 +1,125 @@
+# frozen_string_literal: true
+
+class PuppetProcessor < Processor
+  YAML_CLEAN = /!ruby\/object.*$/.freeze
+  KEYS_TO_COPY = %w[report_format puppet_version environment metrics].freeze
+  MAX_EVAL_TIMES = 29
+
+  def initialize(data, json_body: true)
+    super(data, json_body: json_body)
+    measure :parse do
+      if Gem::Version.new(RUBY_VERSION) < Gem::Version.new("2.6.0")
+        # Ruby 2.5 or older does not have permitted_classes argument available
+        @data = YAML.load(data.gsub(YAML_CLEAN, ""))
+      else
+        @data = YAML.safe_load(data.gsub(YAML_CLEAN, ""), permitted_classes: [Symbol, Time, Date])
+      end
+    end
+    raise("No content") unless @data
+    @body = {}
+    @evaluation_times = []
+    logger.debug "Processing report #{report_id}"
+    debug_payload("Input", @data)
+  end
+
+  def report_id
+    @data["transaction_uuid"] || generated_report_id
+  end
+
+  def process_logs
+    logs = []
+    @data["logs"]&.each do |log|
+      logs << [log["level"]&.to_s, log["source"], log["message"]]
+    end
+    logs
+  rescue StandardError => e
+    logger.error "Unable to parse logs", e
+    logs
+  end
+
+  def process_resource_statuses
+    statuses = []
+    @data["resource_statuses"]&.each_pair do |key, value|
+      statuses << key
+      @evaluation_times << [key, value["evaluation_time"]]
+      # failures
+      add_keywords("PuppetResourceFailed:#{key}", "PuppetHasFailure") if value["failed"] || value["failed_to_restart"]
+      value["events"]&.each do |event|
+        add_keywords("PuppetResourceFailed:#{key}", "PuppetHasFailure") if event["status"] == "failed"
+        add_keywords("PuppetHasCorrectiveChange") if event["corrective_change"]
+      end
+      # changes
+      add_keywords("PuppetHasChange") if value["changed"]
+      add_keywords("PuppetHasChange") if value["change_count"] && value["change_count"] > 0
+      # changes
+      add_keywords("PuppetIsOutOfSync") if value["out_of_sync"]
+      add_keywords("PuppetIsOutOfSync") if value["out_of_sync_count"] && value["out_of_sync_count"] > 0
+      # skips
+      add_keywords("PuppetHasSkips") if value["skipped"]
+      # corrective changes
+      add_keywords("PuppetHasCorrectiveChange") if value["corrective_change"]
+    end
+    statuses
+  rescue StandardError => e
+    logger.error "Unable to parse resource_statuses", e
+    statuses
+  end
+
+  def process_evaluation_times
+    @evaluation_times.sort! { |a, b| b[1] <=> a[1] }
+    if @evaluation_times.count > MAX_EVAL_TIMES
+      others = @evaluation_times[MAX_EVAL_TIMES..@evaluation_times.count - 1].sum { |x| x[1] }
+      @evaluation_times = @evaluation_times[0..MAX_EVAL_TIMES - 1]
+      @evaluation_times << ["Others", others] if others > 0.0001
+    end
+    @evaluation_times
+  rescue StandardError => e
+    logger.error "Unable to process evaluation_times", e
+    []
+  end
+
+  def process
+    measure :process do
+      @body["format"] = "puppet"
+      @body["id"] = report_id
+      @body["host"] = hostname_from_config || @data["host"]
+      @body["proxy"] = Proxy::HostReports::Plugin.settings.reported_proxy_hostname
+      @body["reported_at"] = @data["time"]
+      KEYS_TO_COPY.each do |key|
+        @body[key] = @data[key]
+      end
+      @body["logs"] = process_logs
+      @body["resource_statuses"] = process_resource_statuses
+      @body["keywords"] = keywords
+      @body["evaluation_times"] = process_evaluation_times
+      @body["telemetry"] = telemetry
+      @body["errors"] = errors if errors?
+    end
+  end
+
+  def build_report
+    process
+    if debug_payload?
+      logger.debug { JSON.pretty_generate(@body) }
+    end
+    build_report_root(
+      format: "puppet",
+      version: 1,
+      host: @body["host"],
+      reported_at: @body["reported_at"],
+      status: 0,
+      proxy: @body["proxy"],
+      body: @body,
+      keywords: @body["keywords"],
+    )
+  end
+
+  def spool_report
+    report_hash = build_report
+    debug_payload("Output", report_hash)
+    payload = measure :format do
+      report_hash.to_json
+    end
+    SpooledHttpClient.instance.spool(report_id, payload)
+  end
+end