pdqtest 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 000ec1d235ba0b6ad374ef81ded2be06e0afd599
-  data.tar.gz: be76b9944535f196fba9535bfa2adcf0bc04c4bf
+  metadata.gz: a0ea0994090ec12f98d62f75eb95cdaa98aec982
+  data.tar.gz: 388fec4606165f561b92049ca7bd5a91d2322566
 SHA512:
-  metadata.gz: 60a7f9d3f0aeeac29618081754420221e27ada958726242e3eb55740157efc2347bcc8db7de4026c16fbf24ad567a3a429cb9085b0f5f43147f967de8fc760ee
-  data.tar.gz: 358f66da0df4815fe27218c9907e850d7fa84a1c405c14970ef705099afc876fbd23ab1b9e433aa2d3bef28911723115ec76254fe6f0c66e834e3d5785ab466b
+  metadata.gz: 6e4098facbb04bdd3fb57f0f98200f80829d01fa5d5a166d3d0fb223aa9dcc6351d245cb0e855ba0ff29fdaa46e35a11df2ed62027721693581867e42cd390ca
+  data.tar.gz: 771f55dd4ab0d274f2cf2aa75e2280f9013d6a17cec4a34d705ff8c7029977cd8c9301f207216b425a6ee89e90d4e8575498210e92bce8103ebd5232217d2e60

data/README.md

@@ -6,7 +6,9 @@ PDQTest - Puppet Docker Quick-test - is the quickest and easiest way to test you
 * Linting
 * Syntax
 * RSpec
-* Acceptance (BATS)
+* Acceptance ([BATS](https://github.com/sstephenson/bats))
+
+And can generate code to retrofit testing to a new or existing module, along with skeleton RSpec and acceptance tests to get you started.
 
 PDQTest runs linting, syntax and RSpec tests within the machine it is running from and then loads a docker container to perform acceptance testing.
 
@@ -26,7 +28,7 @@ gem 'pdqtest
 * It's advisable to specify a version number to ensure repeatable builds
 
 ### Puppet modules
-To add PDQTests to a puppet module, run the command:
+To add PDQTests to a new or existing puppet module, run the command:
 ```
 pdqtest init
 ```
@@ -94,6 +96,123 @@ pdqtest --keep-container all
 * Keep the container Running
 * After testing, the container used to run tests will be left running and a message will be printed showing how to enter the container used for testing.  Your code is avaiable at `/cut`
 
+## About acceptance tests
+* Acceptance tests run within a Docker container managed by PDQTest
+* The Docker container breaks all the rules of Docker and runs a full version of Systemd to allow complete testing of Puppet code in the most portable way (basically treat docker as a high speed VM)... This is deliberate - PDQTest exists to get results, not be a perfect Docker app.
+* The test container uses a minimal version of Centos 7, there's no option to change this at present
+* The container will only be started if at least one example file is present and contains the correct magic marker
+* You can get a shell on the docker container if needed:
+  * A representitive system, identical to that which will be used to run the tests `pdqtest shell`
+  * The actual container that was just used on a test run `pdqtest --keep-container all` and run the command indicated on exit
+
+### Test workflow
+1. Scan for all example files under `/examples`.  Files must end in `.pp` and contain the magic marker `#@PDQTest` to be processed
+2. If example files are present, start the docker container
+3. For each example file:
+  * Check for a file at `/spec/acceptance/EXAMPLE_NAME__setup.sh`,   If it exists, run it inside the container.  This is a useful place to install preqrequisites, edit files, install mock services, clean up after other tests, etc.
+  * Check for a file at `/spec/acceptance/EXAMPLE_NAME__before.bats`, If it exists run BATS against it.  This is normally used to check the system state is clean or that any test prerequisites have been setup correctly
+  * Run `puppet apply` on the example file twice
+  * Check for a file at `/spec/acceptance/EXAMPLE_NAME.bats`, If it exists run BATS against it.  This is normally used to check the state of the system after running puppet.  You can do things like check services are running, check files edited correctly, or basically anything that you can write in bash!
+4. Destroy the container unless we were asked to keep it with `--keep-container`
+
+Note: `EXAMPLE_NAME` is the example filename minus the directory and `.pp`, eg the `EXAMPLE_NAME` for `examples/foo.pp` would just be `foo`.
+
+### Example files
+* Found inside `/examples`
+* Regular puppet code
+* Will be executed _twice_ (to check for idempotency) with `puppet apply`
+* _MUST_ contain the magic marker `#@PDQTest` on a line at the top of the file to indicate that this test should be processed by PDQTest
+* Basically the same as a regular puppet [smoke test](https://docs.puppet.com/puppet/latest/tests_smoke.html#module-smoke-testing) but run automatically and with system state tests
+
+### BATS tests
+If you've never seen a BATS test before and have previously been writing server spec code, you'll probably kick youself when you see how simple they are.  Here's an example:
+
+```BASH
+# Tests are really easy! just the exit status of running a command...
+@test "addition using bc" {
+  result="$(ls /)"
+  [ "$?" -eq 0 ]
+}
+```
+
+Tests are all written in BASH and the testcase passes if the stanza returns zero.  This means that basically any Linux/Unix sysadmin is now empowered to write Testcases and do TDD - score!
+
+Since our tests are written in BASH, there are of course all the usual bash quirks such as failing on incorrect spacing, bizzare variable quoting etc, but given the simplicity gained its a worthwhile tradeoff.
+
+Consider the following (real) test:
+
+```BASH
+@test "ClientAliveCountMax" {
+  grep "ClientAliveCountMax 1" /etc/ssh/sshd_config
+}
+```
+
+Here, we have done exactly what it looks like we have done - checked for the value of a particular setting in a text file.  Since our tests our plain old BASH, it means we can do things like test daemons are running with `systemctl`, test ports are open with `netstat` and do complete system tests `wget` and `curl`.  Cool!
+
+#### Worked Example
+Lets say you have two examples, `foo.pp` and `init.pp`.  In this case, the full range of files to create for acceptance testing would look like this:
+```
+mycoolmodule
+├── examples
+│   ├── foo.pp
+│   └── init.pp
+├── ...
+└── spec
+    ├── acceptance
+    │   ├── foo__before.bats
+    │   ├── foo__setup.sh
+    │   ├── foo.bats
+    │   ├── init__before.bats
+    │   ├── init__setup.sh
+    │   └── init.bats
+    ├── ...
+```
+
+* You can put any puppet code you like (includeing an empty file...) in each of the files under `/examples` and it will executed with `puppet apply`
+* If you need to test multiple different things (eg different parameters for a new type and provider), you need to create different testcase for each distinct thing to test.  See below for help generating these
+* You can skip setup or BATS testing before and/or after running `puppet apply` if desired by deleting the appropriate files
+* If you delete all of the files under `spec/acceptance` for a given example, then PDQTest will just check that puppet runs idempotently for your example
+* To disable tests temporarily for a specific example, remove the magic marker `#@PDQTest` from the desired example
+* Nested examples (subdirectories) are not supported at this time
+
+## Real world example
+See https://github.com/GeoffWilliams/puppet-filemagic for a project with lots of acceptance tests.
+
+## Test Generation
+Creating a bunch of files manually is an error prone and tedious operation so PDQTest can generate files and boilerplate code for you so that your up and running in the quickest time possible.
+
+### Skeleton
+* `pdqtest init` will generate a basic set of tests to get you started (tests init.pp)
+
+### RSpec tests
+The skeleton tests created by `pdqtest init` only cover the `init.pp` file which is useful but your likely going to need to support more classes as your modules grow.  PDQTest can generate basic RSpec testcases for each new puppet class that exists in the manifests directory for you:
+
+```shell
+pdqtest generate_rspec
+```
+
+* For every `.pp` file containing a puppet class under `/manifests`, RSpec will be generated to:
+  * Check the catalogue compiles
+  * Check the catalogue contains an instance of the class
+  * This gives developers an easy place to start writing additional RSpec tests for specific behaviour
+  * Its safe to run this command whenever you add a new class, it won't overwrite any existing RSpec testcases
+
+### Acceptance tests
+
+#### Generate boilerplate files for each example found (including those without a magic marker)
+
+```shell
+pdqtest generate_acceptance
+```
+
+#### Generate boilerplate files for one specific example
+
+```shell
+pdqtest generate_acceptance examples/mynewthing.pp
+```
+* Note:  will also create examples/mynewthing.pp if you haven't created it yet
+
+
 ## Development
 
 PRs welcome :)  Please ensure suitable tests cover any new functionality and that all tests are passing before and after your development work:
@@ -110,6 +229,7 @@ You should use pdqtest if you find it increases your productivity and enriches y
 * Don't forget to run `pdqtest setup` before your first `pdqtest` run to download/update the Docker image
 * If you need to access private git repositories, make sure to use `fixtures.yml` not `.fixtures.yml`
 * If you need a private key to access private repositories, set this up for your regular git command/ssh and `pdqtest` will reuse the settings
+* Be sure to annotate the examples you wish to acceptance test with the magic marker comment `#@PDQTest`
 
 ## Support
 This software is not supported by Puppet, Inc.  Use at your own risk.

data/exe/pdqtest

@@ -97,6 +97,25 @@ Escort::App.create do |app|
     end
   end
 
+  app.command :generate_rspec do |command|
+    command.summary "Generate Acceptance test"
+    command.description "Create a testcase in /examples and the corresponding files for acceptance testing"
+    command.options do |opts|
+      opts.opt(:example,
+        'Generate only this example (eg --example examples/newtest.pp)',
+        :long     => '--example',
+        :type     => :string,
+        :default  => nil,
+      )
+    end
+    command.action do |options, arguments|
+      PDQTest::Emoji.disable(options[:global][:options][:disable_emoji])
+      example = options[:global][:commands][:acceptance][:options][:example]
+
+      PDQTest::Skeleton.generate_acceptance(example)
+    end
+  end
+
 
   app.command :shell do |command|
     command.summary "Shell"

data/lib/pdqtest/skeleton.rb

@@ -27,10 +27,13 @@ module PDQTest
 
     def self.install_skeleton(target_file, skeleton, replace=true)
       skeleton_file = Util::resource_path(File.join(SKELETON_DIR, skeleton))
-      if File.exists?(target_file) and replace and should_replace_file(target_file, skeleton_file)
-        # move existing file out of the way
-        FileUtils.mv(target_file, target_file + BACKUP_EXT)
-        install = true
+      install = false
+      if File.exists?(target_file)
+        if replace and should_replace_file(target_file, skeleton_file)
+          # move existing file out of the way
+          FileUtils.mv(target_file, target_file + BACKUP_EXT)
+          install = true
+        end
       else
         install = true
       end
@@ -39,8 +42,8 @@ module PDQTest
       end
     end
 
-    def self.install_example
-      example_file = File.join(EXAMPLES_DIR, 'init.pp')
+    def self.install_example(filename)
+      example_file = File.join(EXAMPLES_DIR, filename)
       if ! File.exists?(example_file)
         template = File.read(Util::resource_path(File.join('templates', 'examples_init.pp.erb')))
         init_pp  = ERB.new(template, nil, '-').result(binding)
@@ -81,18 +84,45 @@ module PDQTest
       # skeleton files if required
       install_skeleton('Rakefile', 'Rakefile')
       install_skeleton(File.join('spec', 'spec_helper.rb'), 'spec_helper.rb')
-      install_skeleton(File.join('spec', 'acceptance', 'init.bats'), 'init.bats', false)
-      install_skeleton(File.join('spec', 'acceptance', 'init__before.bats'), 'init__before.bats', false)
-      install_skeleton(File.join('spec', 'acceptance', 'init__setup.sh'), 'init__setup.sh', false)
       install_skeleton('.travis.yml', 'dot_travis.yml')
       install_skeleton('.gitignore', 'dot_gitignore')
       install_skeleton('.rspec', 'dot_rspec')
       install_skeleton('Makefile', 'Makefile')
 
-      install_example()
+      install_acceptance()
       install_gemfile()
 
       # Make sure there is a Gemfile and we are in it
     end
+
+    def self.install_acceptance(example_file ="init.pp")
+      install_example(File.basename(example_file))
+      example_name = File.basename(example_file).gsub(/\.pp$/, '')
+
+      install_skeleton(File.join('spec', 'acceptance', "#{example_name}.bats"), 'init.bats', false)
+      install_skeleton(File.join('spec', 'acceptance', "#{example_name}__before.bats"), 'init__before.bats', false)
+      install_skeleton(File.join('spec', 'acceptance', "#{example_name}__setup.sh"), 'init__setup.sh', false)
+    end
+
+    # Scan the examples directory and create a set of acceptance tests. If a
+    # specific file is given as `example` then only the listed example will be
+    # processed (and it will be created if missing).  If files already exist, they
+    # will not be touched
+    def self.generate_acceptance(example=nil)
+      examples = []
+      if example
+        # specific file only
+        examples << example
+      else
+        # Each .pp file in /examples (don't worry about magic markers yet, user
+        # will be told on execution if no testscases are present as a reminder to
+        # add it
+        examples += Dir["examples/*.pp"]
+      end
+
+      examples.each { |e|
+        install_acceptance(e)
+      }
+    end
   end
 end

data/lib/pdqtest/version.rb

@@ -1,3 +1,3 @@
 module PDQTest
-  VERSION = "0.2.0"
+  VERSION = "0.3.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: pdqtest
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors:
 - Geoff Williams
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2017-04-04 00:00:00.000000000 Z
+date: 2017-04-06 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler