cem_acpt 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: d8b8813ddfec58349a3ce7c587dbab4a4eae04d7fa24c0ed87896de41ba3d160
+  data.tar.gz: '099f6ed6a7570e2283a740d89bd3864638ff470c8d9a8456bd3eb64ca8d61217'
+SHA512:
+  metadata.gz: ce3f4d2ec04e379cea66c728da7d9a0c8b54a339b4023d40bf47e787e96aaf91c4f6fcd4c5513e38b3c3459de32bf575ad2efbfd8c4e440fd60a16fee0a4ced7
+  data.tar.gz: 3f442bebaab6df3e75925ccfd4324a9b38d84c649805ef59a9af407ad097ca0154797d916c7039ae810f5e9db182e58ccd345049e3ad6c41b9fda97405e828c1

data/.gitignore

@@ -0,0 +1,11 @@
+/.bundle/
+/.yardoc
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+
+# rspec failure tracking
+.rspec_status

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--color
+--require spec_helper

data/.travis.yml

@@ -0,0 +1,6 @@
+---
+language: ruby
+cache: bundler
+rvm:
+  - 2.7.2
+before_install: gem install bundler -v 2.1.4

data/CODEOWNERS

@@ -0,0 +1 @@
+* @puppetlabs/abide-team

data/Gemfile

@@ -0,0 +1,7 @@
+source "https://rubygems.org"
+
+# Specify your gem's dependencies in cem_acpt.gemspec
+gemspec
+
+gem "rake", "~> 12.0"
+gem "rspec", "~> 3.0"

data/Gemfile.lock

@@ -0,0 +1,68 @@
+PATH
+  remote: .
+  specs:
+    cem_acpt (0.1.0)
+      concurrent-ruby (~> 1.1.9)
+      deep_merge (~> 1.2.2)
+      net-scp (~> 3.0)
+      net-ssh (~> 6.1)
+      puppet-modulebuilder (> 0.0.1)
+      rake (> 0.8)
+      rspec (> 0.1)
+      serverspec (> 0.1)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    concurrent-ruby (1.1.9)
+    deep_merge (1.2.2)
+    diff-lcs (1.5.0)
+    minitar (0.9)
+    multi_json (1.15.0)
+    net-scp (3.0.0)
+      net-ssh (>= 2.6.5, < 7.0.0)
+    net-ssh (6.1.0)
+    net-telnet (0.1.1)
+    pathspec (1.1.3)
+    puppet-modulebuilder (0.3.0)
+      minitar (~> 0.9)
+      pathspec (>= 0.2.1, < 2.0.0)
+    rake (12.3.3)
+    rspec (3.11.0)
+      rspec-core (~> 3.11.0)
+      rspec-expectations (~> 3.11.0)
+      rspec-mocks (~> 3.11.0)
+    rspec-core (3.11.0)
+      rspec-support (~> 3.11.0)
+    rspec-expectations (3.11.0)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.11.0)
+    rspec-its (1.3.0)
+      rspec-core (>= 3.0.0)
+      rspec-expectations (>= 3.0.0)
+    rspec-mocks (3.11.0)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.11.0)
+    rspec-support (3.11.0)
+    serverspec (2.41.8)
+      multi_json
+      rspec (~> 3.0)
+      rspec-its
+      specinfra (~> 2.72)
+    sfl (2.3)
+    specinfra (2.82.2)
+      net-scp
+      net-ssh (>= 2.7)
+      net-telnet (= 0.1.1)
+      sfl
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  cem_acpt!
+  rake (~> 12.0)
+  rspec (~> 3.0)
+
+BUNDLED WITH
+   2.1.4

data/README.md

@@ -0,0 +1,146 @@
+# CemAcpt
+
+CemAcpt is an acceptance testing library / command line application for running acceptance tests against the CEM modules. It is heavily inspired by Puppet Litmus. CemAcpt is fully compatible with Puppet Litmus acceptance tests.
+
+CemAcpt was written to facilitate running a single acceptance test file against one or more test nodes concurrently. This is useful when your module supports a lot of different operating systems and modifies low-level components of those operating systems. For CEM, we manage things such as firewall and bootloader configurations which requires us to test against multiple base images of a single operating system (i.e. RHEL 8 with `iptables` and RHEL 8 with `firewalld`). Additionally, we test our module using both Puppet 6 and Puppet 7. As you can see, we now need to test against four test nodes that are all RHEL 8 with slightly different baseline configs. CemAcpt allows us to do so quickly in parallel.
+
+Major differences between Puppet Litmus and CemAcpt are:
+
+- CemAcpt revolves around mapping a single acceptance test file to one or more test hosts.
+- Test node image names can be dynamically defined by configurable parsing of the acceptance test name.
+  - This allows your acceptance test names to define the node it will run against.
+- CemAcpt has it's own CLI for running provisioning of hosts and the test suite and does not use `rake`.
+- CemAcpt runs everything, including provisioning the test node, creating a temporary manifest on the test node, and running the acceptance test on the test node, in parallel. No matter how many test / host combinations there are, running the test suite will only take as long as the longest running single test / host combo.
+- CemAcpt is configurable via a YAML file.
+
+## Concepts
+
+CemAcpt is underpinned with a few core concepts. These concepts shape the architecture and functionality of CemAcpt.
+
+### Platforms
+
+Platforms are backends that handle provisioning, interacting with, and destroying test nodes. Platforms typically represent something like a cloud provider or hypervisor, but as long as a platform implements the required methods it could do anything.
+
+Platforms are created by creating a file in `lib/cem_acpt/platform`. The name of the platform file is significant as it will be capitalized and used as a classname. PLATFORMS MUST implement the methods stipulated by `lib/cem_acpt/platform/base.rb` in a single module called `Platform` because the platform system will dynamically create the platform class from this module. How the methods are implemented is irrelevant as long as they take the correct input and give the correct output.
+
+Platforms will be passed all configuration options under the top-level key `node_data`. Node configuration should be generic and should apply to all test nodes regardless of the test being ran against them. For test-specific configurations, use `test data`.
+
+See [lib/cem_acpt/platform/gcp.rb](lib/cem_acpt/platform/gcp.rb) for an example of a platform file.
+
+### Tests
+
+Each acceptance test should be specified in the config under the top-level key `tests`. These tests SHOULD NOT be paths and should not contain the suffix `_spec.rb`. For example, if you have an acceptance test `spec/acceptance/our_acceptance_test_spec.rb`, the tests config would look like this:
+
+```yaml
+tests:
+  - our_acceptance_test
+```
+
+Aside from ways of manipulating test data outlined below, tests are typically matched one-to-one with a test node.
+
+### Test data
+
+Test data is a collection of data about a specific test that persists through the entire acceptance test suite lifecycle. Specifically, test data is implemented as an array of hashes with each hash representing a single run through the acceptance test suite lifecycle. What this means is that each item in the test data array is couple with a single test node that is provisioned and a single test run against that node. Test data is used to store values that are test-specific, as opposed to node data which is generic.
+
+Test data can be configured using the top-level key `test_data`. There are several supported options for manipulating test data:
+
+#### for_each
+
+When specified, should be a hash of `key -> Array` pairs. For each of these pairs, a copy of the test data is made for each item in the array, with that item becoming the value of a variable `key`.
+
+Example:
+
+```yaml
+test_data:
+  for_each:
+    collection:
+      - puppet6
+      - puppet7
+
+tests:
+  - our_acceptance_test
+```
+
+In the above config, instead of test data consisting of a single hash, it will have two hashes that have the `collection` variable set to `puppet6` and `puppet7` respectively. This means that there will be two test nodes provisioned and two runs of the test `our_acceptance_test`, one run against each node.
+
+#### vars
+
+Aribitrary key-value pairs that are injected into the test data hashes. Think of these as constants.
+
+#### name_pattern_vars
+
+A Ruby regex pattern that is matched against the test name with the intent of creating variables from named capture groups. See [sample_config.yaml](sample_config.yaml) for an example.
+
+#### vars_post_processing
+
+Rules that allow for processing variables after all other test data rules are ran. See [sample_config.yaml](sample_config.yaml) for an example.
+
+### Image name builder
+
+Much like `name_pattern_vars`, specifying the `image_name_builder` top-level key in the config allows you to manipulate acceptance test names to create a special test data variable called `image_name`. This is helpful for when you have multiple platform base images and want to use the correct image with the correct test. See [sample_config.yaml](sample_config.yaml) for an example.
+
+## Installation
+
+Add this line to your module's Gemfile (usually under `group :development`):
+
+```ruby
+gem 'cem_acpt', require: false
+```
+
+Then, add the following to your `spec_helper_acceptance.rb` (replacing the Puppet Litmus configuration lines):
+
+```ruby
+require 'cem_acpt'
+
+CemAcpt.configure_spec_helper!
+```
+
+To use CemAcpt in your acceptance tests, you must call the method `initialize_test_environment!` in your acceptance test's `describe` block before doing anything else:
+
+```ruby
+describe 'cem_linux CIS Level 1' do
+  initialize_test_environment!
+  ...
+```
+
+## Usage
+
+### RSpec methods
+
+CemAcpt enables you to use all ServerSpec methods in your acceptance tests, and adds three new methods that can be used (these methods are used the same as in Litmus):
+
+- `apply_manifest(manifest, opts = {})`: Applies a Puppet manifest given as a string.
+- `idempotent_apply(manifest, opts = {})`: Applies a Puppet manifest given as a string twice and fails if the second apply reports changes.
+- `run_shell(command, opts = {})`: Runs a shell command against the test node.
+
+### Config file
+
+CemAcpt can use a config file (default path is `./cem_acpt_config.yaml`) to configure it's settings. Below are some of the options available in the config file:
+
+- `tests`: An array of test names you want to run. These are assumed to exist in the path `spec/acceptance`. You should leave off the suffix `_spec.rb` from the test file names.
+- `platform`: The backend platform that acceptance tests will run on. Currently, only `gcp` is supported.
+- `test_data`: Configurations for test data. Allows assigning variables to test data either dynamically or statically.
+- `node_data`: Configurations for nodes created by the platform. The available options are platform-dependent.
+- `image_name_builder`: If this key is specified, platforms will be passed a dynamically generated image name based on the configurations under this key.
+
+### Semantic acceptance test names
+
+With CemAcpt, acceptance test names can have semantic meaning and influence how your acceptance test nodes are provisioned and how test data is created. The semantic meaning of the test names can be configured in the config file.
+
+#### Test data
+
+Test data can
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/cem_acpt.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "cem_acpt"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/cem_acpt.gemspec

@@ -0,0 +1,37 @@
+# frozen_string_literal: true
+
+require_relative 'lib/cem_acpt/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = 'cem_acpt'
+  spec.version       = CemAcpt::VERSION
+  spec.authors       = ['Heston Snodgrass']
+  spec.email         = ['hsnodgrass3@gmail.com']
+
+  spec.summary       = 'CEM Acceptance Tests'
+  spec.description   = 'Litmus-like library focusing on CEM Acceptance Tests'
+  spec.homepage      = 'https://github.com/puppetlabs/cem_acpt'
+  spec.license       = 'MIT'
+  spec.required_ruby_version = Gem::Requirement.new('>= 2.5.0')
+
+  spec.metadata['homepage_uri'] = spec.homepage
+  spec.metadata['source_code_uri'] = 'https://github.com/puppetlabs/cem_acpt'
+  spec.metadata['changelog_uri'] = 'https://github.com/puppetlabs/cem_acpt'
+
+  # Specify which files should be added to the gem when it is released.
+  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+  spec.files = Dir.chdir(File.expand_path('..', __FILE__)) do
+    `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  end
+  spec.bindir = 'exe'
+  spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ['lib']
+  spec.add_dependency 'concurrent-ruby', '~> 1.1.9'
+  spec.add_dependency 'deep_merge', '~> 1.2.2'
+  spec.add_dependency 'net-scp', '~> 3.0'
+  spec.add_dependency 'net-ssh', '~> 6.1'
+  spec.add_dependency 'puppet-modulebuilder', '> 0.0.1'
+  spec.add_dependency 'rake', '> 0.8'
+  spec.add_dependency 'rspec', '> 0.1'
+  spec.add_dependency 'serverspec', '> 0.1'
+end

data/exe/cem_acpt

@@ -0,0 +1,58 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'deep_merge'
+require 'json'
+require 'optparse'
+require 'yaml'
+require 'cem_acpt'
+
+options = {}
+parser = OptionParser.new do |opts|
+  opts.banner = 'Usage: cem_acpt [options]'
+
+  opts.on('-D', '--debug', 'Enable debug logging') do
+    options[:log_level] = 'debug'
+  end
+
+  opts.on('-L', '--log-file FILE', 'Log to FILE') do |file|
+    options[:log_file] = file
+  end
+
+  opts.on('-O', '--options OPTS', 'Set options. Example: -P "param1=value1,param2=value2"') do |o|
+    params = o.split(',').map { |s| s.split('=') }.to_h
+    params.transform_keys(&:to_sym).each do |k, v|
+      options[k] = v
+    end
+  end
+
+  opts.on('-p', '--platform PLATFORM', 'Set platform. Example: -p "gcp"') do |p|
+    options[:platform] = p
+  end
+
+  opts.on('-m', '--module-dir DIR', 'Set module directory. Example: -m "/tmp/module"') do |m|
+    options[:module_dir] = m
+  end
+
+  opts.on('-c', '--config-file FILE', 'Set config file. Example: -c "/tmp/config.yaml"') do |c|
+    options[:config_file] = c
+  end
+
+  opts.on('-I', '--CI', 'Run in CI mode') do
+    options[:CI] = true
+  end
+
+  opts.on('-E', '--no-destroy-nodes', 'Do not destroy nodes') do
+    options[:no_destroy_nodes] = true
+  end
+end
+
+parser.parse!
+options[:module_dir] = Dir.pwd unless options[:module_dir]
+options[:platforms] = ['gcp', 'vmpooler'] unless options[:platform]
+if options.key?(:log_level) && options[:log_level] == 'debug'
+  puts '#################### RUNNING ACCEPTANCE TEST SUITE ####################'
+  puts "Using options from command line: #{options}"
+  puts '#######################################################################'
+end
+CemAcpt.run(options)

data/lib/cem_acpt/bootstrap/bootstrapper.rb

@@ -0,0 +1,206 @@
+# frozen_string_literal: true
+
+require_relative 'operating_system'
+
+# Currently unused in this implementation.
+module CemAcpt::Bootstrap
+  # This class runs bootstrapping commands on the VM after it
+  # is provisioned. This can include installing repos,
+  # installing Puppet, installing other programs,
+  # starting services, and running other commands.
+  class Bootstrapper
+    STATE_ORDER = [:repo, :package, :service, :command, :complete].freeze
+    STATE_ACTIONS = {
+      repo: :run_repo_installs,
+      package: :run_package_installs,
+      service: :run_service_starts,
+      command: :run_commands,
+    }.freeze
+    UNIVERSAL_DEPS = [
+      'git',
+      'curl',
+      'puppet-agent',
+    ].freeze
+    include CemAcpt::Bootstrap::OperatingSystem
+
+    def initialize(instance_name, instance_image, cmd, collection: 'puppet7', repos: [], packages: [], services: [], commands: [])
+      os, major_version = instance_image.split('-')
+      use_os(os)
+      @instance_name = instance_name.split('.').first
+      @cmd_provider = cmd
+      @collection = collection
+      @os = os
+      @major_version = major_version
+      @repos = repos
+      @packages = packages
+      @services = services
+      @commands = commands
+      @logfile = '/var/log/bootstrap.log'
+      @state = 0
+      @retries = 0
+      @results = {}
+    end
+
+    def ruby_version
+      return @ruby_version if defined?(@ruby_version)
+
+      @ruby_version = determine_ruby_version(@collection)
+      @ruby_version
+    end
+
+    def repos
+      @repos.push(puppet_agent_repo).uniq
+    end
+
+    def packages
+      (@packages + UNIVERSAL_DEPS).uniq
+    end
+
+    def services
+      @services.uniq
+    end
+
+    def commands
+      @commands + ruby_install_commands
+    end
+
+    def instance_available?
+      @instance_available ||= test_connection
+    end
+
+    def run
+      raise 'Instance is not available' unless instance_available?
+
+      @results = {}
+      transition_state
+      @results
+    rescue StandardError => e
+      return_results_with_error(e)
+    end
+
+    private
+
+    def return_results_with_error(error)
+      @results[:_error] = {
+        error: error,
+        details: {
+          instance_name: @instance_name,
+          instance_image: @instance_image,
+          collection: @collection,
+          os: @os,
+          major_version: @major_version,
+          repos: repos,
+          packages: packages,
+          services: services,
+          commands: commands,
+          state: @state,
+          retries: @retries,
+        },
+      }
+      @results
+    end
+
+    def execute_state_action
+      state_sym = STATE_ORDER[@state]
+      send(STATE_ACTIONS[state_sym])
+    end
+
+    def transition_state
+      raise @last_error if @results.key?(:_error)
+      return if @state == STATE_ORDER.length - 1
+
+      begin
+        execute_state_action
+      rescue StandardError => e
+        @last_error_time = Time.now
+        @last_errored_state = @state
+        @last_error = e
+        handle_state_error
+      end
+      @state += 1
+      transition_state
+    end
+
+    def run_repo_installs
+      raise 'Instance is not available' unless instance_available?
+
+      run_ssh_command_with_formatting(repo_install_cmd(repos), :repo_installs)
+    end
+
+    def run_package_installs
+      raise 'Instance is not available' unless instance_available?
+
+      run_ssh_command_with_formatting(package_install_cmd(packages), :package_installs)
+    end
+
+    def run_service_starts
+      raise 'Instance is not available' unless instance_available?
+
+      run_ssh_command_with_formatting(service_start_cmd(services), :service_starts)
+    end
+
+    def run_commands
+      raise 'Instance is not available' unless instance_available?
+
+      run_ssh_command_with_formatting(commands.join(';'), :command_runs)
+    end
+
+    def handle_state_error
+      @retries = 0 unless @last_errored_state == @state
+
+      raise @last_error unless @retries < 3
+
+      @retries += 1
+      @state = @last_errored_state
+      transition_state
+    end
+
+    def run_ssh_command_with_formatting(cmd, results_key, log_file: '/tmp/puppet-bootstrap.log')
+      key = results_key.to_sym
+      @results[key] = {}
+      @results[key][:started] = Time.now
+      if cmd.nil? || cmd.empty?
+        @results[key][:results] = 'No command(s) to run'
+        @results[key][:finished] = Time.now
+        return
+      end
+      cmd = "#{cmd} | tee -a #{log_file}"
+      ssh_results = @cmd_provider.ssh(@instance_name, cmd)
+      @results[key][:results] = ssh_results.gsub(%r{\s+}, ' ').split('\n')
+      @results[key][:finished] = Time.now
+    end
+
+    def verify_command(results_key)
+      send("verify_#{results_key}".to_sym)
+    end
+
+    def test_connection
+      return true if @cmd_provider.ssh_ready?(@instance_name)
+
+      raise "Connection test failed for #{@instance_name}"
+    end
+
+    def determine_ruby_version(collection)
+      case collection
+      when 'puppet7'
+        '2.7'
+      when 'puppet6'
+        '2.5'
+      else
+        raise "Cannot determine Ruby version, unsupported collection: #{collection}"
+      end
+    end
+
+    def ruby_install_commands
+      [
+        'curl -sSL https://raw.githubusercontent.com/rvm/rvm/stable/binscripts/rvm-installer -o rvm-installer',
+        'curl -sSL https://raw.githubusercontent.com/rvm/rvm/stable/binscripts/rvm-installer.asc -o rvm-installer.asc',
+        'chmod +x rvm-installer',
+        'gpg --verify rvm-installer.asc rvm-installer && sudo ./rvm-installer',
+        "sudo rvm install #{ruby_version} && sudo rvm use #{ruby_version} --default",
+        'gem install bundler',
+        '[[ -f ./Gemfile ]] && bundle install',
+      ]
+    end
+  end
+end