pdqtest 1.9.9beta7 → 1.9.9beta8

data/doc/pdk.md

@@ -1,7 +1,7 @@
 # PDK Integration
 
-First off, what is PDK? PDK is Puppet's development kit. The bits we are 
-interested in right now are:
+First off, what is PDK? [PDK](https://github.com/puppetlabs/pdk/) is Puppet's 
+development kit. The bits we are interested in right now are:
 * Validation `pdk validate...`
 * Testing `pdk test...`
 * Packaging `pdk build...`
@@ -34,8 +34,8 @@ VMs in order to extract maximum speed. It offers a different approach to
       * Complex OS commands that need to report different state between puppet
         runs can be mocked with Python and a small SQLite database or simple
         text files (eg the `sysctl` command)
-      * Fake different Unix-like Operating Systems like AIX by subverting the 
-        commands that your puppet code would executes. This is particularly 
+      * Fake different Unix-like Operating Systems such as AIX by subverting the 
+        commands that your puppet code would execute. This is particularly 
         useful in this case since there are no AIX desktop virtualisation 
         solutions right now and there probably never will be
       * See [examples](examples.md) for real-world modules using these
@@ -52,7 +52,7 @@ for and you should look towards solutions like Beaker or
 
 ## How does PDQTest work with PDK? Do I have to choose?
 
-No you don't have to choose! PDQTest integrates with PDK by making the minimal
+No, you don't have to choose! PDQTest integrates with PDK by making the minimal
 amount of invasive changes:
 
 ### New projects
@@ -68,12 +68,15 @@ After running `pdqtest init`, run `pdk update` to have PDK process the
 
 ### Existing PDQTest projects
 To make upgrading to PDQTest 2.0 + PDK easy for our existing users, we automate
-the workflow normally carried out by `pdk convert`:
+the workflow normally carried out by `pdk convert`.
 
 We do not include the entire set of files that would have been generated by
 `pdk convert` or `pdk new module`. If you want the full set, you should run 
-one of these commands yourself before `pdqtest init` and they will be left alone
-unless PDK starts writing new files that we are already using.
+one of these commands yourself before `pdqtest init` and the resulting files 
+will be left alone unless PDK starts writing new files that we are already 
+using.
+
+#### Key integration points
 
 **metadata.json**
 
@@ -131,17 +134,22 @@ or supported.
 
 
 ### PDQTest directory structure
-These are the files installed, files marked 🛠 are generated by PDK:
+* 🛠 - File generated by PDK
+* ⚡ - File updated/replaced when you run `pdqtest upgrade`:
 
 ```
-├── bitbucket-pipelines.yml
+├── bitbucket-pipelines.yml ⚡
 ├── Gemfile 🛠
 ├── Gemfile.local
 ├── Gemfile.project
+├── .gitattributes 🛠
 ├── .gitignore 🛠
-├── Makefile
-├── make.ps1
+├── Makefile ⚡
+├── make.ps1 ⚡
 ├── .pdkignore 🛠
+├── .pdqtest
+│   ├── Gemfile ⚡
+│   └── Gemfile.lock
 ├── .puppet-lint.rc
 ├── Rakefile 🛠
 ├── spec
@@ -151,10 +159,10 @@ These are the files installed, files marked 🛠 are generated by PDK:
 │   │   │   └── test.yaml
 │   │   └── hiera.yaml
 │   └── spec_helper.rb 🛠
-└── .travis.yml
+└── .travis.yml ⚡
 ```
 
-The remaining files are specific to PDQTest - notably:
+Notes:
 * `.travis.yml` - Complete test suite for Linux modules
 * `bitbucket-pipelines.yaml` - logical testing only (unit/RSpec) 
 * `.puppet-lint.rc` - Make lint errors test failures, ignore double quotes, etc
@@ -162,119 +170,103 @@ The remaining files are specific to PDQTest - notably:
 * `make.ps1` - Essential launch script for Windows
 * `spec/fixtures/hiera.yaml` - Mock system-wide `hiera.yaml` file
 * `spec/fixtures/hieradata/test.yaml` - Mock system-wide hieradata
-* `Gemfile.project` - Used to enable the PDQTest gem (and any others you need)
+* `Gemfile.project` - Used to enable additional gems in PDK if needed (eg for 
+  RSpec)
 * `Gemfile.local` - Transient file created at runtime by `Makefile` or
-  `make.ps1` to enable the PDQTest gem at runtime. This is a vital integration
-  point, see details below
+  `make.ps1` to enable additional PDK gems see details below
 * `.gitignore` - PDK has a massive list of files to ignore for new projects so
-  these are imported
-
+  these are imported and we add some of our own
+* `.gitattributes` - PDK doesn't process this on `pdk update` so it ignores our
+  `.sync.yml` customisations. This is handy to force LF for teams working on 
+  windows
+* `.pdqtest/Gemfile` - it's **impossible** to share a `Gemfile` with PDK
+  (believe me I tried) Therefore we need our own and it lives here. See details
+  below.
+* `.pdqtest/Gemfile.lock` - corresponding lock for PDQTest
 
 ## Why are the launch scripts essential/How does the PDQTest gem load itself?
+
+### Extra gems for `pdk` command
 PDK provides its own `Gemfile` to select the gems available at runtime. There
 are hooks in this file to load additional configuration from two locations:
 * `~/.gemfile`
 * `YOUR_PROJECT/Gemfile.local`
 
-We support side-by-side installation of all versions of PDQTest so that users
-are not forced to upgrade when they are not ready to, so we need a per-project
-way to load our gem into the bundle. `Gemfile.local` is perfect for this but we
-are not supposed to store permanent data in there according to PDK's generated
-`.gitignore`.
+These are good for loading additional GEMs that run inside the PDK ruby
+environment created by the `pdk` command _only_! PDK modifies it's `Gemfile` at
+runtime according to the target being run and this which makes it unsafe for us
+to load the `pdqtest` gem here, because the bundle will change while it is being
+used. 
 
-The simplest way to workaround this issue is for us to create a symlink from 
-`Gemfile.project` to `Gemfile.local` at runtime and we _must_ use our launch
-scripts to do this since we are unable to use any Ruby code to do this trick
-since we are not yet in the bundle.
+Using `pdk bundle exec` as basically an alias for `bundle exec` did not work as
+PDQTest needs to call the `pdk` command for its lifecycle targets. This caused
+serious errors on Windows.
 
-For this reason, you **must** launch PDQTest using the provided `Makefile` or
-`make.ps1` scripts, at least initially.
+We are not supposed to store permanent data in `Gemfile.local` as its in
+PDK's generated `.gitignore`. The workaround to this is for us to create our own
+per-project `Gemfile` with the gems we would have liked to have put in 
+`Gemfile.local` and copy/symlink as required. 
+
+The main use of this is to load additional gems during the `pdk test unit` phase
+that without customising via `.sync.yml` or a custom template repository 
+(although these should work too/instead).
 
 For further background see: 
 * [PDK-1177](https://tickets.puppetlabs.com/browse/PDK-1177) and 
 * [#50](https://github.com/declarativesystems/pdqtest/issues/50) 
 
-## When developing PDQTest, your version number **must** match a released version
-_This should only affects PDQTest developers - for posterity..._
-
-Due to some strange hackery between `pdk bundle` and `bundle exec` you have one
-more gotcha when it comes to using developing PDQTest and thats that the version
-number of the gem you build *MUST* match a released version of the gem on 
-rubygems.org otherwise `bundle exec` will barf with errors like this:
+### Load PDQTest and supporting gems
+Since PDK updates `Gemfile` during execution, the only sensible option for
+loading PDQTest is to have our own directory containing our own `Gemfile` 
+and `Gemfile.lock`:
+* `.pdqtest/Gemfile`
+* `.pdqtest/Gemfile.lock`
+
+This affords complete separation from PDK and lets us use the provided ruby. The
+only caveat is that we must `cd` into the `.pdqtest` directory before executing
+the `pdqtest` command.
+
+When you run `pdqtest upgrade`, it will update `.pdqtest/Gemfile` with the newer
+version of the `pdqtest` gem.
+
+As ever, the per-project `.pdqtest/Gemfile` means that we support side-by-side 
+installation of all versions of PDQTest while letting PDK take over the main 
+`Gemfile` ensures full PDK compatibility.
+
+### `Makefile` and `make.ps1`
+To support these two scenarios, `Makefile` and `make.ps1` automate the project
+preparation (bundling/symlinking) by providing the following targets:
+* `Gemfile.local`:
+    * Windows: Copy `Gemfile.project` to `Gemfile.local`, replacing any content,
+      then run `pdk bundle` to update PDKs gems
+    * Linux: Symlink `Gemfile.local` to `Gemfile.project`, then run `pdk bundle`
+      to update PDKs gems
+* `pdqtestbundle` - run `bundle install` using system/custom ruby against
+  `.pdqtest/Gemfile`
+* `pdkbundle` - run `pdk bundle install` to re-bundle PDK
+
+The remaining targets all work by jumping into the `.pdqtest` directory and then
+using `.pdqtest/Gemfile` to launch `pdqtest`. This avoids users having to jump
+around the project directories to get work done.
 
-```shell
-$ bundle exec pdqtest
-Could not find hiera-3.4.5 in any of the sources
-Run `bundle install` to install missing gems.
-```
+For this reason, you **must** launch PDQTest using the provided `Makefile` or
+`make.ps1` scripts, at until your familiar with jumping to the `.pdqtest`
+directory to run `bundle exec pdqtest` yourself.
 
-There seems no other way to fix this other than building the development gem 
-with a version matching an existing release. The gem then needs to be installed
-using `gem install ...` and it will then magically start working. Using other
-gem tricks like setting `:path` will appear to work but fail when `bundle exec`
-is called. The other approach of trying to run everything with 
-`pdk bundle exec pdqtest` fails with various errors depending what platform your
-on, presumably because `pdqtest` runs `pdk` command as part of its tests.
-
-The final final caveat of this approach is that the PDQTest gem dependencies are
-pinned at whatever the _real_ release uses, so if any were changed or added they
-need to be temporarily added to `Gemfile.project`
-
-The above works on Linux... For windows there is also the situation where:
-* PDK vendors its own gems that magically appear
-* PDK can only use "published" gems(specs?) it downloads - or appears to
-* when you go to `bundle install` without using PDK you blow up the lock 
-  file
-* When you `bundle exec pdqtest` you get the above error because _something_
-  uses the old one `gemspec`
-* The answer seems to be:
-    1. use `gem 'pdqtest', :path=>'c:/vagrant/pdqtest'` in `Gemfile.local`
-    2. `pdk bundle install`
-    3. `bundle update pdqtest` - yes even though your not supposed to...
-    4. `bundle exec pdqtest` - now it should work... why is beyond me
-* If you still have errors:
-    1. `bundle install`
-    2. `rm Gemfile.lock`
-    3. `pdk bundle install` - working?
-
-Somtimes you just have to upload the whole gem file as a pre-release object:
-* Version set to x.x.xWHATEVER
-* `gem push pdqtest-x.x.xWHATEVER.gem`
-* `gem install pdqtest --pre`
-
-
-If your just want to use PDQTest, you will hopefully never have to worry about
-this... On the other hand, if this all sounds confusing... That's because it is!
 
 ## What happens when I upgrade PDQTest?
-When a new version of PDQTest is released, upgrade by running:
-
-```shell
-gem install pdqtest
-cd /your/project
-pdqtest upgrade
-pdk bundle install
-```
-
-This updates:
-* Project gems in `Gemfile.project` (currently `pdqtest` and `puppet-strings`)
-* Our CI and CLI integrations:
-    * `Makefile`
-    * `make.ps1`
-    * `.travis.yml`
-    * `bitbucket-pipelines.yml`
-    
-The final `pdk bundle install` command upgrades PDK's bundle to the include our
-update and is critical to making the whole process work.
+When PDQTest is upgraded, we update the files above marked ⚡ in your project. 
+This doesn't impact PDK at all with the exception that we take over 
+`.travis.yml` since the PDK one doesn't do what we want it to.
 
 ## What happens when I upgrade PDK?
 PDK operates independently from PDQTest and maintains its own files. Your free
 to run `pdk update` to upgrade the your files to the latest PDK templated ones
-at any time.
+whenever you like.
 
 To protect PDQTest files from alteration (notably `.travis.yml` and
-`bitbucket-pipelines.yml`) we merge them with any existing `.sync.yml` to
-prevent churn.
+`bitbucket-pipelines.yml`) we merge instructions to have PDK leave them alone
+to `.sync.yml` (merging with any existing rules) to prevent churn.
 
 If you have further customisations to PDK controlled files your options are:
 * Use git to revert any change by PDK
@@ -290,27 +282,21 @@ open a [ticket](https://github.com/declarativesystems/pdqtest/issues)
 
 ## How do PDQTest lifecycle tests relate to PDK?
 
-| PDQTest   | PDK                            |
-| ---       | ---                            |
-| `syntax`  | `pdk validate metadata,puppet` |
-| `rspec`   | `pdk test unit`                |
-| `build`   | `pdk build --force`            |    
-
-* Previous `lint` subcommand was removed in PDQTest 2.0 as its now covered by 
-  `syntax`
+| PDQTest   | PDK                                                                  |
+| ---       | ---                                                                  |
+| `logical` | `pdk validate metadata,puppet`, `pdk test unit`                      |
+| `all`     | `pdk validate metadata,puppet`, `pdk test unit`, `pdk build --force` |
+| `fast`    | N/A - run `puppet-lint` and `rake syntax` to finish faster           |
 
 ## What actually happens when I run PDQTest?
 
 `bundle exec pdqtest all` is the default target executed by `make` and 
-`.\make.ps1`. There are a few other targets that let you skip parts of the build
-or run individual parts. The complete run looks like this: 
+`.\make.ps1`. The complete run looks like this: 
 
 1. `pdk validate 'metadata,puppet'`
 2. Install modules listed in the `metadata.json` file using R10K against a 
    temporary `Puppetfile` at `Puppetfile.pdqtest`
 3. Generate a `.fixtures.yml` file based on `metadata.json` 
-   Currently impacted by 
-   [#47](https://github.com/declarativesystems/pdqtest/issues/47)
 4. `pdk test unit`
 5. Run all acceptance tests
 6. `puppet strings generate --format=markdown` to generate `REFERENCE.md`
@@ -320,15 +306,10 @@ At each stage of the process, we output emoji's to keep you informed of
 progress. Any failure prevents running the next phase of testing and lint errors
 are considered failures.
  
-### Technical details (code)
-If anyone knows a better way to do this, I'd love to hear about it:
-[PDQTest lifecycle](https://github.com/declarativesystems/pdqtest/blob/master/exe/pdqtest)
-[PDK Wrapper](https://github.com/declarativesystems/pdqtest/blob/master/lib/pdqtest/pdk.rb)
-
 ## PDQTest is pretty slow!
 This is a side effect of having to shell out to run PDK via system calls. 
-There's really no other way to do this while maintaining full PDK compatiblity 
-(if you know different, please enlighten me).
+There's really no other way to do this while maintaining full PDK compatibility 
+(if you know different, please open a ticket and let me know how).
 
 That said you might just want to run syntax and lint tests and acceptance tests
 as quick as possible, in which case run:

data/doc/puppet_facts.md

@@ -11,11 +11,12 @@ consistent set of facts for both RSpec and acceptance testing, in the file that
 PDK expects them to be in.
 
 ## 🐉 Differences from PDK 🐉
-The above behaviour is not how PDK intends this file to be used as some other 
-mechanism is supposed to be used for acceptance tests (see PDK manual).
+The above behaviour is _not_ how PDK intends this file to be used as some other 
+mechanism is supposed to be used for acceptance tests.
 
 It's pretty rare to actually need `spec/default_facts.yml` unless your doing
-unit testing since you can insert custom facts as part of your test setup.
+unit testing since you can insert custom facts as part of your test setup 
+script (eg `spec/fixtures/init__setup.sh`).
 
 Be aware that PDK sets a few facts to specific values that we will pick up and
 use in your acceptance tests:
@@ -30,23 +31,3 @@ macaddress: "AA:AA:AA:AA:AA:AA"
 You would need to use a forked version of the PDK templates to fix these
 properly, but your using the structured facts anyway so it makes no difference,
 right...?
-
-## Older versions
-PDQTest < 2.0 includes built-in 
-[factsets](https://github.com/declarativesystems/puppet_factset) covering many
-common operating systems, however, sometimes additional facts are needed 
-throughout a module to mock an external fact or custom fact from a different 
-module that is expected to be some known value.  In this case, creating the 
-directory:
-
-```
-spec/merge_facts
-```
-
-Inside the module being tested will result in all JSON files contained being 
-merged into all OS specific factsets used for testing as generated by 
-`pdqtest generate_rspec`.
-
-To put it simply, drop `.json` file(s) in this directory to add them to every
-factset.  These facts will be made automatically avaialable to facter when 
-running acceptance testing

data/doc/puppet_module_dependencies.md

@@ -32,16 +32,11 @@ fixtures:
 ```
 
 It is an error to define the same module in both `metadata.json` and 
-`.fixtures.yml` and the results of doing this are undefined:
+`.fixtures.yml` and the results of doing so are undefined:
 * Always use `metadata.json` for public forge modules and we will update 
   `.fixtures.yml` based on it
 * Modules from git should _only_ be defined in `.fixtures.yml`
 
-## Notes:
-* The file format of `.fixtures.yml` and `fixtures.yml` for specifing git 
-  repositories is identical
-* Previous versions of pdqtest configured the r10k cache via `.r10k.yaml` which
-  caused [#44](https://github.com/declarativesystems/pdqtest/issues/44). To fix
-  this, we now use the default r10k cache dir at `~/.r10k/cache` and don't write
-  `.r10k.yaml` any more. You should remove any `.r10k.yaml` files from your 
-  project unless you need it for something specific.
+### pdk build
+Note that running `pdk build` will remove any modules present under 
+`/spec/fixtures/modules`. Run unit tests to restore them.

data/doc/running_tests.md

@@ -2,8 +2,9 @@
 
 ## Important
 You **must** use the provided launch scripts `Makefile` or `make.ps1` to run
-PDQTest, at least for the first time. See [PDK integration](pdk.md) for more
-information.
+PDQTest, at least until your familiar with how PDQTest works. This is because 
+PDQTest must always be run from the `.pdqtest` directory. See 
+[PDK integration](pdk.md) for more information.
 
 ## Quickstart
 If you just want to run all tests:
@@ -18,41 +19,48 @@ make
 .\make.ps1
 ```
 
-Alternatively, you can choose to run different groups of tests by supplying a
-target from this table:
+## Make targets
+Alternatively, you can choose to run a `Makefile`/`make.ps1` target by supplying
+an argument from this table:
 
-| Target        | Description                                                            | PDK compatible? |
-| ---           | ---                                                                    | ---             |
-| all           | lint, syntax, rspec, acceptance, strings, build                        | yes             |
-| fast          | lint, syntax, acceptance, strings, build                               | no              |
-| shell         | run acceptance tests and print command to get a shell in the container | yes             |
-| shellnopuppet | open a shell in the test container                                     | yes             |
-| logical       | syntax, lint                                                           | yes             |
+| Target          | Description                                                            | PDK compatible? |
+| ---             | ---                                                                    | ---             |
+| `all`           | metadata, syntax, lint, rspec, acceptance, strings, build              | yes             |
+| `fast`          | syntax, lint, rspec, acceptance, strings                               | no              |
+| `acceptance`    | acceptance tests only                                                  | -               |
+| `shell`         | run acceptance tests and print command to get a shell in the container | yes             |
+| `setup`         | download required docker images for this version of PDQTest            | -               |
+| `shellnopuppet` | open a shell in the test container                                     | yes             |
+| `logical`       | metadata, syntax, lint, rspec, docs                                    | yes             |
+| `pdqtestbundle` | install gems needed for PDQTest                                        | -               |
+| `docs`          | generate `REFERENCE.md` using Puppet Strings                           | -               |
+| `Gemfile.local` | symlink/copy `Gemfile.local` from `Gemfile.project` and re-bundle PDK  | yes             |
+| `pdkbundle`     | re-bundle PDK                                                          | yes             |
+| `clean`         | cleanup `/pkg` and `/spec/fixtures/modules`                            | -               |
 
-* PDK compatible means we run the `pdk` command for this part of the lifecycle
+**PDK compatible**
+* yes: we run the `pdk` command for this part of the lifecycle
+* no: we run an alternative command
+* -: There is no corresponding `pdk` command
 
+**Examples**
 
-**Example**
+Test the module as quickly as possible:
 
-**Linux**
+*Linux*
 ```shell
 make fast
 ```
 
-**Windows**
+*Windows*
 ```shell
 .\make.ps1 fast
 ```
 
-## Only run acceptance tests
-```shell
-bundle exec pdqtest acceptance
-```
-
 
 ## See also
 
 PDQTest help:
 ```shell
-bundle exec pdqtest --help
+cd .pdqtest ; bundle exec pdqtest --help
 ```

data/doc/test_generation.md

@@ -23,4 +23,14 @@ This will also create examples/mynewthing.pp if you haven't created it yet.
 
 ## RSpec tests
 PDQTest < 2.0 includes rspec test generation. This functionality is replaced by
-PDK in later versions.
+PDK in later versions:
+
+```shell
+pdk new class
+```
+
+Generates Puppet classes _and_ RSpec tests
+
+## Other files
+You should investigate PDK for generating more boiler plate code for things like
+types/providers, defined resources, etc.

data/doc/tips_and_tricks.md

@@ -1,6 +1,7 @@
 # Tips and tricks
 * You can put any puppet code you like (including an empty file...) in each of 
-  the files under `/examples` and it will executed with `puppet apply`
+  the files under `/examples` and it will executed with `puppet apply` as long
+  as it contains the magic marker `#@PDQTest` or `#@PDQTestWin`
 * If you need to test multiple different things (eg different parameters for a 
   new type and provider), you need to create a different (acceptance) testcase
   for each distinct thing to test
@@ -11,14 +12,16 @@
 * To disable tests temporarily for a specific example, remove the magic marker 
   `#@PDQTest` or `#@PDQTestWin` from the example
 * Nested examples (subdirectories) are not supported at this time
-* Since the all of the `*__setup.sh` scripts are run in the container as root 
-  before executing tests, they can be used to mock _almost anything_ in the 
-  test system:
+* Since all of the `*__setup.sh`/`*__setup.ps1` scripts are run in the container
+  as `root`/`Administrator` before executing tests, they can be used to mock 
+  _almost anything_ in the test system:
   * Replace system binaries to fake network operations
   * Add system binaries to simulate other operating systems such as AIX, 
     Solaris, etc
   * Create/copy files, directories, etc.
   * Install OS packages
   * Install python scripts to mock database servers using SQLite... 😉
-* Files added by PDQTest >= 2.0 have the magic marker: 
+* Files added by PDQTest >= 2.0 that are not originating from PDK are marked: 
   `*File originally created by PDQTest*`
+* See [development](development.md) guide for how to turn on debugging and exit
+  stack traces

data/doc/troubleshooting.md

@@ -5,16 +5,17 @@
   your in and you are using `rvm` its probably because `rvm` overrides `cd` and
   does strange things. You can probably turn this off. Alternatively, use 
   `rbenv`
-* Don't forget to run `pdqtest setup` before your first `pdqtest` run to 
-  download/update the Docker image
+* Don't forget to run `make setup` or `.\make.ps1 setup` before your first 
+  PDQTest run to download/update the Docker image
 * If you need to access private git repositories, make sure to use 
   `.fixtures.yml` not `fixtures.yml` (changed in 2.0.0)
 * If you need a private key to access private repositories, set this up for your
-  regular git command/ssh and `pdqtest` will reuse the settings
+  regular git command/ssh and PDK should reuse the settings
 * Be sure to annotate the examples you wish to acceptance test with the magic
   marker comment `#@PDQTest` or `#@PDQTestWin`
-* Be sure to run `make` or `bundle exec pdqtest all` to download dependencies
-  when running acceptance tests.  Previous versions (re)downloaded modules as
-  required from inside docker but this step has been replaced with a simple
-  symlink to reduce the amount of downloading so the modules must already be
-  present.
+* Be sure to run `make` or `.\make.ps1` to download dependencies when running 
+  acceptance tests.  Previous versions (re)downloaded modules as required from 
+  inside docker but this step has been replaced with a simple symlink to reduce 
+  the amount of downloading so the modules must already be present.
+* Update to the latest PDK before running PDQTest
+* See the [PDQTest 1.x -> 2.x upgrade notes](upgrade_1_2.md) for other gotchas