agentless-catalog-executor 0.9.0

data/developer-docs/docker.md

@@ -0,0 +1,170 @@
+# Docker Docs
+
+## Setup
+
+[Docker-compose installation](https://docs.docker.com/compose/install/) would need to be followed and setup in order to use the ACE containers for development.
+
+
+The ACE compose file is dependent on a Docker network created when launching the Puppetserver and PuppetDB containers within the `spec/` directory, the network will default to `spec_default`, since the containers are built from the `spec/` directory they will be assigned the `<folder>_default` network.
+
+As the ACE container is build outside of the `spec/` directory it would not be able to create the `spec_default` network. This can be created manually through:
+
+```
+docker network create spec_default
+```
+
+Once this is done the ACE container can be launched by executing the following within the root folder:
+
+```
+docker-compose up -d --build
+```
+
+This will take some time as it needs to perform the initial build of fetching the images and running through the build.
+
+Navigate to the `spec/` folder and build the Puppetserver and PuppetDB containers using the same command. The Puppetserver will take some time to start and typically using the following command to verify that it is ready:
+
+```
+docker logs --follow spec_puppetserver_1
+```
+
+Once the Puppetserver is ready, the following message is reported:
+
+```
+2019-03-18 15:42:19,964 INFO  [p.s.m.master-service] Puppet Server has successfully started and is now ready to handle requests
+2019-03-18 15:42:19,965 INFO  [p.s.l.legacy-routes-service] The legacy routing service has successfully started and is now ready to handle requests
+```
+
+On Linux, ensure that you have access to all volumes:
+
+```
+sudo chmod a+rx -R volumes/
+```
+
+At this point it is required to generate certs for the `aceserver`, this can be achieved though:
+
+`docker exec spec_puppet_1 puppetserver ca generate --certname aceserver --subject-alt-names localhost,aceserver,ace_aceserver_1,spec_puppetserver_1,ace_server,puppet_server,spec_aceserver_1,puppetdb,spec_puppetdb_1,0.0.0.0,puppet`
+
+On Linux, ensure that you have access to the newly created files:
+
+```
+sudo chmod a+rx -R volumes/
+```
+
+Reasoning for this is that it makes it easier to ensure that the cert names are consistent across environments.
+
+## Verifying the services
+
+[Postman](https://www.getpostman.com/) is advisable to verify that the endpoints are configured. In order to set up Postman, navigate to Settings > Certificates and add client certificates for hosts `0.0.0.0:8140` and `0.0.0.0:44633` where the CRT file points to `spec/volumes/puppet/ssl/certs/aceserver.pem` and Key file points to `spec/volumes/puppet/ssl/private_keys/aceserver.pem`
+
+*Note*: These cert and key files will only be created when the PuppetServer container has finished initalising.
+
+### PuppetServer /tasks/:module/:task
+
+```
+https://0.0.0.0:8140/puppet/v3/tasks/:module/:task?environment=production
+```
+
+Is the endpoint to get the task metadata from a PuppetServer, i.e.
+
+```
+GET https://0.0.0.0:8140/puppet/v3/tasks/panos/apikey?environment=production
+
+RESPONSE
+{
+    "metadata": {
+        "description": "Retrieve a PAN-OS apikey",
+        "files": [
+            ...
+        ],
+        "parameters": {},
+        "puppet_task_version": 1,
+        "remote": true,
+        "supports_noop": false
+    },
+    "name": "panos::apikey",
+    "files": [
+        ...
+    ]
+}
+```
+
+This can be used to construct the request body that will be used to execute the [ACE `/run_task`](#ace-runtask) endpoint.
+
+### ACE /run_task
+
+```
+POST https://0.0.0.0:44633/run_task
+BODY {
+	"target": {
+		"remote-transport": "panos",
+		"name":"pavm",
+		"hostname": "vvtzckq3vzx995w.delivery.puppetlabs.net",
+		"user": "admin",
+		"password": "admin",
+		"ssl": false
+	},
+	"task": {
+    "metadata": {
+        "description": "Retrieve a PAN-OS apikey",
+        "files": [
+            ...
+        ],
+        "parameters": {},
+        "puppet_task_version": 1,
+        "remote": true,
+        "supports_noop": false
+    },
+    "name": "panos::apikey",
+    "files": [
+        ...
+    ]
+}}
+
+RESPONSE
+{
+    "node": "vvtzckq3vzx995w.delivery.puppetlabs.net",
+    "status": "success",
+    "result": {
+        "apikey": "LUFRPT14MW5xOEo1R09KVlBZNnpnemh0VHRBOWl6TGM9bXcwM3JHUGVhRlNiY0dCR0srNERUQT09"
+    }
+}
+```
+
+Running the containers through Docker does have the benefit that the containers will be a better representation of how the ACE service will work in PE, however for developing and verifying changes it can be considered slow as changes may require the ACE container to be rebuilt which can take some time, an alternative approach for local development is [running the service locally](#running-ace-locally), this way the Puppetserver and PuppetDB containers are only required to be running.
+
+## Running ACE locally
+
+The ACE service can also be ran directly through Puma rather than building the container, this has the benefits of being able to specifying local changes of Bolt within the Gemfile rather than having to make the changes in the container, or committing the changes and rebuilding the containers which can take some time.
+
+When running locally through Puma there is a caveat on the tasks that are being executed and a possible conflict with the version of Ruby on the local installation, where solutions are highlighted in the [incorrect Puppet Ruby version](#incorrect-puppet-ruby-version).
+
+Launching the service locally can be achieved by running the following:
+
+```
+ACE_CONF=config/local.conf bundle exec puma -C config/transport_tasks_config.rb
+```
+
+### Incorrect Puppet Ruby version
+
+The tasks typically used in networking modules have a shebang referencing the Puppet Ruby, there are two approaches to getting around this.
+
+When constructing requests to be sent to ACE, in the target hash the interpreter can be specified, i.e.
+
+```
+{
+   "target":{
+      "remote-transport":"panos",
+      "name":"pavm",
+      "interpreters":{
+         ".rb":"/Users/thomas.franklin/.rbenv/versions/2.5.1/bin/ruby"
+      }
+   },
+   "task": {hash from puppetserver /tasks endpoint}
+}
+```
+
+Although this would need to be included in every request - a 'permanent' solution would be to symlink the Puppet Ruby to the development version of Ruby, i.e.
+
+```
+sudo ln -svf $(bundle exec which ruby) /opt/puppetlabs/puppet/bin/ruby
+```

data/docker-compose.yml

@@ -0,0 +1,12 @@
+version: "3"
+services:
+  aceserver:
+    build: .
+    ports:
+      - "44633:44633"
+    tty: true
+    stdin_open: true
+networks:
+  default:
+    external:
+      name: spec_default

data/lib/ace/config.rb

@@ -0,0 +1,73 @@
+# frozen_string_literal: true
+
+require 'hocon'
+require 'bolt_server/base_config'
+
+module ACE
+  class Config < BoltServer::BaseConfig
+    attr_reader :data
+    def config_keys
+      super + %w[concurrency cache-dir puppet-server-conn-timeout puppet-server-uri ssl-ca-crls]
+    end
+
+    def env_keys
+      super + %w[concurrency puppet-server-conn-timeout puppet-server-uri ssl-ca-crls]
+    end
+
+    def ssl_keys
+      super + %w[ssl-ca-crls]
+    end
+
+    def int_keys
+      %w[concurrency puppet-server-conn-timeout]
+    end
+
+    def defaults
+      super.merge(
+        'port' => 44633,
+        'concurrency' => 10,
+        'cache-dir' => "/opt/puppetlabs/server/data/ace-server/cache",
+        'puppet-server-conn-timeout' => 120,
+        'file-server-conn-timeout' => 120
+      )
+    end
+
+    def required_keys
+      super + %w[puppet-server-uri cache-dir]
+    end
+
+    def service_name
+      'ace-server'
+    end
+
+    def load_env_config
+      env_keys.each do |key|
+        transformed_key = "ACE_#{key.tr('-', '_').upcase}"
+        next unless ENV.key?(transformed_key)
+        @data[key] = if int_keys.include?(key)
+                       ENV[transformed_key].to_i
+                     else
+                       ENV[transformed_key]
+                     end
+      end
+    end
+
+    def validate
+      super
+
+      unless natural?(@data['concurrency'])
+        raise Bolt::ValidationError, "Configured 'concurrency' must be a positive integer"
+      end
+
+      unless natural?(@data['puppet-server-conn-timeout'])
+        raise Bolt::ValidationError, "Configured 'puppet-server-conn-timeout' must be a positive integer"
+      end
+    end
+
+    def make_compatible
+      # This function sets values used by Bolt that behave the same in ACE, but have a different meaning
+      @data['file-server-uri'] = @data['puppet-server-uri']
+      @data['file-server-conn-timeout'] = @data['puppet-server-conn-timeout']
+    end
+  end
+end

data/lib/ace/configurer.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+require 'puppet/configurer'
+
+module ACE
+  class Configurer < Puppet::Configurer
+    # override the configurer to return the facts
+    # related to the transport and the trusted
+    # facts which is passed to the configurer.run
+    def get_facts(options)
+      transport_facts = Puppet::Node::Facts.indirection.find(Puppet[:certname],
+                                                             environment: Puppet[:environment]).values
+      trusted_facts = options[:trusted_facts]
+      { transport_facts: transport_facts, trusted_facts: trusted_facts }
+    end
+  end
+end

data/lib/ace/error.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module ACE
+  class Error < RuntimeError
+    attr_reader :kind, :details, :issue_code, :error_code
+
+    def initialize(msg, kind, details = nil, issue_code = nil)
+      super(msg)
+      @kind = kind
+      @issue_code = issue_code
+      @details = details || {}
+      @error_code ||= 1
+    end
+
+    def msg
+      message
+    end
+
+    def to_h
+      h = { 'kind' => kind,
+            'msg' => message,
+            'details' => details }
+      h['issue_code'] = issue_code if issue_code
+      h
+    end
+
+    def to_json(opts = nil)
+      to_h.to_json(opts)
+    end
+  end
+end

data/lib/ace/fork_util.rb

@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+# English module required for $CHILD_STATUS rather than $?
+require 'English'
+
+module ACE
+  class ForkUtil
+    # Forks and calls a function
+    # It is expected that the function returns a JSON response
+    # Throws an exception if JSON.generate fails to generate
+    def self.isolate
+      reader, writer = IO.pipe
+      pid = fork {
+        success = true
+        begin
+          response = yield
+          writer.puts JSON.generate(response)
+        rescue StandardError => e
+          writer.puts(e)
+          success = false
+        ensure
+          Process.exit! success
+        end
+      }
+      unless pid
+        log "Could not fork"
+        exit 1
+      end
+      writer.close
+      output = reader.read
+      Process.wait(pid)
+      if $CHILD_STATUS != 0
+        raise output
+      else
+        JSON.parse(output)
+      end
+    end
+  end
+end

data/lib/ace/plugin_cache.rb

@@ -0,0 +1,79 @@
+# frozen_string_literal: true
+
+require 'fileutils'
+require 'ace/puppet_util'
+require 'puppet/configurer'
+require 'concurrent'
+require 'ace/fork_util'
+
+module ACE
+  class PluginCache
+    attr_reader :cache_dir_mutex, :cache_dir
+    def initialize(environments_cache_dir)
+      @cache_dir = environments_cache_dir
+      @cache_dir_mutex = Concurrent::ReadWriteLock.new
+    end
+
+    def setup
+      FileUtils.mkdir_p(cache_dir)
+      self
+    end
+
+    def with_synced_libdir(environment, certname, &block)
+      ForkUtil.isolate do
+        ACE::PuppetUtil.isolated_puppet_settings(certname, environment)
+        with_synced_libdir_core(environment, &block)
+      end
+    end
+
+    def with_synced_libdir_core(environment)
+      pool = Puppet::Network::HTTP::Pool.new(Puppet[:http_keepalive_timeout])
+      Puppet.push_context({
+                            http_pool: pool
+                          }, "Isolated HTTP Pool")
+      libdir = sync_core(environment)
+      Puppet.settings[:libdir] = libdir
+      $LOAD_PATH << libdir
+      yield
+    ensure
+      FileUtils.remove_dir(libdir)
+      pool.close
+    end
+
+    # the Puppet[:libdir] will point to a tmp location
+    # where the contents from the pluginsync dest is copied
+    # too.
+    def libdir(plugin_dest)
+      tmpdir = Dir.mktmpdir(['plugins', plugin_dest])
+      cache_dir_mutex.with_write_lock do
+        FileUtils.cp_r(File.join(plugin_dest, '.'), tmpdir)
+        FileUtils.touch(tmpdir)
+      end
+      tmpdir
+    end
+
+    def environment_dir(environment)
+      environment_dir = File.join(cache_dir, environment)
+      cache_dir_mutex.with_write_lock do
+        FileUtils.mkdir_p(environment_dir)
+        FileUtils.touch(environment_dir)
+      end
+      environment_dir
+    end
+
+    # @returns the tmp libdir directory which will be where
+    # Puppet[:libdir] is referenced too
+    def sync_core(environment)
+      env = Puppet::Node::Environment.remote(environment)
+      environments_dir = environment_dir(environment)
+      Puppet[:vardir] = File.join(environments_dir)
+      Puppet[:confdir] = File.join(environments_dir, 'conf')
+      Puppet[:rundir] = File.join(environments_dir, 'run')
+      Puppet[:logdir] = File.join(environments_dir, 'log')
+      Puppet[:codedir] = File.join(environments_dir, 'code')
+      Puppet[:plugindest] = File.join(environments_dir, 'plugins')
+      Puppet::Configurer::PluginHandler.new.download_plugins(env)
+      libdir(File.join(environments_dir, 'plugins'))
+    end
+  end
+end

data/lib/ace/puppet_util.rb

@@ -0,0 +1,57 @@
+# frozen_string_literal: true
+
+require 'openssl'
+
+module ACE
+  class PuppetUtil
+    def self.init_global_settings(ca_cert_path, ca_crls_path, private_key_path, client_cert_path, cachedir, uri)
+      Puppet::Util::Log.destinations.clear
+      Puppet::Util::Log.newdestination(:console)
+      Puppet.settings[:log_level] = 'notice'
+      Puppet.settings[:trace] = true
+      Puppet.settings[:catalog_terminus] = :certless
+      Puppet.settings[:node_terminus] = :memory
+      Puppet.settings[:catalog_cache_terminus] = :json
+      Puppet.settings[:facts_terminus] = :network_device
+      # the following settings are just to make base_context
+      # happy, these will not be the final values,
+      # as per request settings will be set later on
+      # to satisfy multi-environments
+      Puppet.settings[:vardir] = cachedir
+      Puppet.settings[:confdir] = File.join(cachedir, 'conf')
+      Puppet.settings[:rundir] = File.join(cachedir, 'run')
+      Puppet.settings[:logdir] = File.join(cachedir, 'log')
+      Puppet.settings[:codedir] = File.join(cachedir, 'code')
+      Puppet.settings[:plugindest] = File.join(cachedir, 'plugins')
+      Puppet.push_context(Puppet.base_context(Puppet.settings), "Puppet Initialization")
+      # ssl_context will be a persistent context
+      cert_provider = Puppet::X509::CertProvider.new(
+        capath: ca_cert_path,
+        crlpath: ca_crls_path
+      )
+      ssl_context = Puppet::SSL::SSLProvider.new.create_context(
+        cacerts: cert_provider.load_cacerts(required: true),
+        crls: cert_provider.load_crls(required: true),
+        private_key: OpenSSL::PKey::RSA.new(File.read(private_key_path, encoding: 'utf-8')),
+        client_cert: OpenSSL::X509::Certificate.new(File.read(client_cert_path, encoding: 'utf-8'))
+      )
+      Puppet.push_context({
+                            ssl_context: ssl_context,
+                            server: uri.host,
+                            serverport: uri.port
+                          }, "PuppetServer connection information to be used")
+      Puppet.settings.use :main, :agent, :ssl
+      Puppet::Transaction::Report.indirection.terminus_class = :rest
+    end
+
+    def self.isolated_puppet_settings(certname, environment)
+      Puppet.settings[:certname] = certname
+      Puppet.settings[:environment] = environment
+      env = Puppet::Node::Environment.remote(environment)
+      Puppet.push_context({
+                            configured_environment: environment,
+                            loaders: Puppet::Pops::Loaders.new(env)
+                          }, "Isolated settings to be used")
+    end
+  end
+end