cem_acpt 0.2.10-universal-java-17 → 0.2.11-universal-java-17

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5d59f3f4432694cd766fc5ca304ade0cad8cb681849181256d8c1ced3ba68438
-  data.tar.gz: eff162b600769222a8d0670e2a528cc389eae0407e35a1d6671c1518248c0480
+  metadata.gz: c97a0886b1604384a7e03f5edec960b594842f69502e5b1fcd4d2a6a18ecb3fc
+  data.tar.gz: 8ad5f879c25f70d0050ec0f7fd24891339b7a80271193d2549e8a6a7224d7727
 SHA512:
-  metadata.gz: a8124eec84379d5737fc4e04327ded0de1da8cb43c724a51cfd892c150823b58b14f9f6f99ec24adff5a7bc69302595e79de34501326c1c6e85fe1b0fa3c99cc
-  data.tar.gz: c6ea839545a9c00779aec17f331a9769a743baf7e05654f25d9efaa0e7edd8490dc528342ba75cdef171dc79328ad0431b204310c53d736f0b8bd2faf71ac6b4
+  metadata.gz: 2279fa0e8ab932d0f02982cf8cacf6aaf4bf8c0a5e709eeb7e3cd0d093a4712f501c175f2a57f64cd0c441e56fa0c51e4238ab1a196d89037f719194981536a1
+  data.tar.gz: 76ca3f9c58e4f1450b3e442b0edc825b68ea5b533796b34e90963630dc219731238128f44dbe1267979199d06efa1a3cef99913318bb3684af663e377ec8b97f

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    cem_acpt (0.2.10-universal-java-17)
+    cem_acpt (0.2.11-universal-java-17)
       concurrent-ruby (>= 1.1, < 2.0)
       deep_merge (>= 1.2, < 2.0)
       ed25519 (>= 1.2, < 2.0)
@@ -68,7 +68,7 @@ GEM
       rspec-its
       specinfra (~> 2.83.1)
     sfl (2.3)
-    specinfra (2.83.2)
+    specinfra (2.83.3)
       net-scp
       net-ssh (>= 2.7)
       net-telnet (= 0.1.1)

data/README.md

@@ -1,8 +1,8 @@
 # CemAcpt
 
-CemAcpt is an acceptance testing library / command line application for running acceptance tests against the CEM modules. It is heavily inspired by Puppet Litmus. CemAcpt is fully compatible with Puppet Litmus acceptance tests.
+CemAcpt is an acceptance testing library / command line application for running acceptance tests for the CEM modules. It is heavily inspired by Puppet Litmus. CemAcpt is fully compatible with Puppet Litmus acceptance tests.
 
-CemAcpt was written to facilitate running a single acceptance test file against one or more test nodes concurrently. This is useful when your module supports a lot of different operating systems and modifies low-level components of those operating systems. For CEM, we manage things such as firewall and bootloader configurations which requires us to test against multiple base images of a single operating system (i.e. RHEL 8 with `iptables` and RHEL 8 with `firewalld`). Additionally, we test our module using both Puppet 6 and Puppet 7. As you can see, we now need to test against four test nodes that are all RHEL 8 with slightly different baseline configs. CemAcpt allows us to do so quickly in parallel.
+CemAcpt was written to facilitate running a single acceptance test file against one or more individual test nodes concurrently, and have each additional acceptance test file do the same. This is useful when your module supports a lot of different operating systems and modifies low-level components of those operating systems. For CEM, we manage things such as firewall and bootloader configurations which requires us to test against multiple base images of a single operating system (i.e. RHEL 8 with `iptables` and RHEL 8 with `firewalld`). Additionally, we test our module using both Puppet 6 and Puppet 7. As you can see, we now need to test against four test nodes that are all RHEL 8 with slightly different baseline configs. CemAcpt allows us to do so quickly in parallel.
 
 Major differences between Puppet Litmus and CemAcpt are:
 
@@ -10,8 +10,72 @@ Major differences between Puppet Litmus and CemAcpt are:
 - Test node image names can be dynamically defined by configurable parsing of the acceptance test name.
   - This allows your acceptance test names to define the node it will run against.
 - CemAcpt has it's own CLI for running provisioning of hosts and the test suite and does not use `rake`.
-- CemAcpt runs everything, including provisioning the test node, creating a temporary manifest on the test node, and running the acceptance test on the test node, in parallel. No matter how many test / host combinations there are, running the test suite will only take as long as the longest running single test / host combo.
+- CemAcpt runs everything, including provisioning the test node(s), creating a temporary manifest on the test node(s), and running the acceptance test on the test node(s), in parallel. No matter how many test / host combinations there are, running the test suite will only take as long as the longest running single test / host combo.
 - CemAcpt is configurable via a YAML file.
+- CemAcpt provides extensive logging, including the ability to save logs to a file and an option to format log output on `stdout` specifically for GitHub Actions.
+
+## Installation
+
+### Command-line usage
+
+Assuming cem_acpt is already configured in your Puppet module, you can run cem_acpt from your command line.
+
+`cem_acpt` require JRuby 9.3.3.0 or greater. To install and use JRuby with `rvm`:
+
+```
+rvm install jruby-9.3.3.0
+rvm use jruby-9.3.3.0
+# If you have a Gemfile.lock already
+rm Gemfile.lock
+bundle install
+jruby --server <jruby opts> -S bundle exec cem_acpt <opts>
+```
+
+> If you do not delete Gemfile.lock before running `bundle install`, you will most likely encounter dependency version errors. Just delete Gemfile.lock and run `bundle install` again to get past these.
+
+### Installing into a Puppet module
+
+Add this line to your module's Gemfile (usually under `group :development`):
+
+```ruby
+gem 'cem_acpt', require: false
+```
+
+Then, add the following to your `spec_helper_acceptance.rb` (replacing the Puppet Litmus configuration lines):
+
+```ruby
+require 'cem_acpt'
+
+CemAcpt.configure_spec_helper!
+```
+
+To use CemAcpt in your acceptance tests, you must call the method `initialize_test_environment!` in your acceptance test's `describe` block before doing anything else:
+
+```ruby
+describe 'cem_linux CIS Level 1' do
+  initialize_test_environment!
+  ...
+```
+
+## Usage
+
+### RSpec methods
+
+CemAcpt enables you to use all ServerSpec methods in your acceptance tests, and adds three new methods that can be used (these methods are used the same as in Litmus):
+
+- `apply_manifest(manifest, opts = {})`: Applies a Puppet manifest given as a string.
+- `idempotent_apply(manifest, opts = {})`: Applies a Puppet manifest given as a string twice and fails if the second apply reports changes.
+- `run_shell(command, opts = {})`: Runs a shell command against the test node.
+
+### Config file
+
+CemAcpt can use a config file (default path is `./cem_acpt_config.yaml`) to configure it's settings. Below are some of the options available in the config file:
+
+- `tests`: An array of test names you want to run. These are assumed to exist in the path `spec/acceptance`. You should leave off the suffix `_spec.rb` from the test file names.
+- `platform`: The backend platform that acceptance tests will run on. Currently, only `gcp` is supported.
+- `test_data`: Configurations for test data. Allows assigning variables to test data either dynamically or statically.
+- `node_data`: Configurations for nodes created by the platform. The available options are platform-dependent.
+- `image_name_builder`: If this key is specified, platforms will be passed a dynamically generated image name based on the configurations under this key.
 
 ## Concepts
 
@@ -79,62 +143,6 @@ Rules that allow for processing variables after all other test data rules are ra
 
 Much like `name_pattern_vars`, specifying the `image_name_builder` top-level key in the config allows you to manipulate acceptance test names to create a special test data variable called `image_name`. This is helpful for when you have multiple platform base images and want to use the correct image with the correct test. See [sample_config.yaml](sample_config.yaml) for an example.
 
-## Installation
-
-### Installing from RubyGems
-
-To install the gem locally to use as a CLI tool:
-
-`gem install cem_acpt --prerelease`
-
-### Installing into a Puppet module
-
-Add this line to your module's Gemfile (usually under `group :development`):
-
-```ruby
-gem 'cem_acpt', require: false
-```
-
-Then, add the following to your `spec_helper_acceptance.rb` (replacing the Puppet Litmus configuration lines):
-
-```ruby
-require 'cem_acpt'
-
-CemAcpt.configure_spec_helper!
-```
-
-To use CemAcpt in your acceptance tests, you must call the method `initialize_test_environment!` in your acceptance test's `describe` block before doing anything else:
-
-```ruby
-describe 'cem_linux CIS Level 1' do
-  initialize_test_environment!
-  ...
-```
-
-## Usage
-
-### RSpec methods
-
-CemAcpt enables you to use all ServerSpec methods in your acceptance tests, and adds three new methods that can be used (these methods are used the same as in Litmus):
-
-- `apply_manifest(manifest, opts = {})`: Applies a Puppet manifest given as a string.
-- `idempotent_apply(manifest, opts = {})`: Applies a Puppet manifest given as a string twice and fails if the second apply reports changes.
-- `run_shell(command, opts = {})`: Runs a shell command against the test node.
-
-### Config file
-
-CemAcpt can use a config file (default path is `./cem_acpt_config.yaml`) to configure it's settings. Below are some of the options available in the config file:
-
-- `tests`: An array of test names you want to run. These are assumed to exist in the path `spec/acceptance`. You should leave off the suffix `_spec.rb` from the test file names.
-- `platform`: The backend platform that acceptance tests will run on. Currently, only `gcp` is supported.
-- `test_data`: Configurations for test data. Allows assigning variables to test data either dynamically or statically.
-- `node_data`: Configurations for nodes created by the platform. The available options are platform-dependent.
-- `image_name_builder`: If this key is specified, platforms will be passed a dynamically generated image name based on the configurations under this key.
-
-### Semantic acceptance test names
-
-With CemAcpt, acceptance test names can have semantic meaning and influence how your acceptance test nodes are provisioned and how test data is created. The semantic meaning of the test names can be configured in the config file.
-
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.

data/lib/cem_acpt/test_runner/run_handler.rb

@@ -105,10 +105,7 @@ module CemAcpt
             if result['examples'].empty? && !result['messages'].empty?
               logger.error(result['messages'].join("\n"))
             else
-              failed = result['examples'].reject { |e| e['status'] == 'passed' }
-              failed.each do |e|
-                logger.error(test_error_msg(runner.node.node_name, e))
-              end
+              handle_example_error_pending_skipped(runner.node.node_name, result['examples'])
             end
           else
             handle_runner_error_results(runner)
@@ -118,6 +115,19 @@ module CemAcpt
         end
       end
 
+      # Handles logging for any failed / skipped / pending examples.
+      def handle_example_error_pending_skipped(node_name, examples)
+        examples.each do |e|
+          next if e['status'] == 'passed'
+
+          if e['status'] == 'pending'
+            logger.info(test_pending_msg(node_name, e))
+          else
+            logger.error(test_error_msg(node_name, e))
+          end
+        end
+      end
+
       # Handles logging the results of the runners that errored.
       def handle_runner_error_results(runner)
         logger.error("SUMMARY: Encountered an error with test #{runner.node.test_data[:test_name]} on node #{runner.node.node_name}")
@@ -136,6 +146,19 @@ module CemAcpt
           "Error: #{err.message}",
           'Backtrace:',
           err.backtrace.join("\n"),
+          "\n",
+        ].join("\n")
+      end
+
+      # Formats a test result for tests that are pending or skipped. Is used for logging.
+      # @param node [String] the name of the node the test ran on
+      # @param result [Hash] the test result to format
+      # @return [String] the formatted test result
+      def test_pending_msg(_, result)
+        [
+          "TEST PENDING / SKIPPED: #{result['full_description']}",
+          "REASON: #{result['pending_message']}",
+          "\n",
         ].join("\n")
       end
 

data/lib/cem_acpt/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module CemAcpt
-  VERSION = '0.2.10'
+  VERSION = '0.2.11'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: cem_acpt
 version: !ruby/object:Gem::Version
-  version: 0.2.10
+  version: 0.2.11
 platform: universal-java-17
 authors:
 - puppetlabs
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2022-07-26 00:00:00.000000000 Z
+date: 2022-08-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement