beaker 3.35.0 → 3.36.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (57) hide show
  1. checksums.yaml +4 -4
  2. data/.gitignore +32 -9
  3. data/CHANGELOG.md +15 -1
  4. data/CONTRIBUTING.md +37 -31
  5. data/DOCUMENTING.md +68 -44
  6. data/README.md +14 -31
  7. data/acceptance/tests/base/dsl/helpers/host_helpers/curl_on_test.rb +3 -3
  8. data/beaker.gemspec +1 -1
  9. data/docs/README.md +5 -12
  10. data/docs/concepts/beaker_libraries.md +2 -2
  11. data/docs/concepts/glossary.md +11 -0
  12. data/docs/concepts/style_guide.md +24 -91
  13. data/docs/concepts/test_tagging.md +3 -10
  14. data/docs/concepts/testing_beaker_itself.md +3 -2
  15. data/docs/concepts/ticket_process.md +17 -45
  16. data/docs/how_to/archive_sut_files.md +7 -33
  17. data/docs/how_to/change_terminal_output_coloring.md +7 -4
  18. data/docs/how_to/confine.md +5 -16
  19. data/docs/how_to/debug_beaker_tests.md +14 -6
  20. data/docs/how_to/hosts/README.md +2 -5
  21. data/docs/how_to/hosts/archlinux.md +4 -2
  22. data/docs/how_to/hosts/cisco.md +9 -30
  23. data/docs/how_to/hosts/eos.md +5 -14
  24. data/docs/how_to/hypervisors/README.md +9 -31
  25. data/docs/how_to/install_puppet.md +16 -51
  26. data/docs/how_to/platform_specific_tag_confines.md +11 -31
  27. data/docs/how_to/preserve_hosts.md +13 -54
  28. data/docs/how_to/rake_tasks.md +1 -4
  29. data/docs/how_to/recipes.md +9 -2
  30. data/docs/how_to/run_in_parallel.md +10 -10
  31. data/docs/how_to/ssh_agent_forwarding.md +8 -13
  32. data/docs/how_to/ssh_connection_preference.md +10 -10
  33. data/docs/how_to/test_arbitrary_beaker_versions.md +5 -12
  34. data/docs/how_to/the_beaker_dsl.md +44 -21
  35. data/docs/how_to/upgrade_from_2_to_3.md +13 -53
  36. data/docs/how_to/use_hocon_helpers.md +10 -28
  37. data/docs/how_to/use_user_password_authentication.md +13 -10
  38. data/docs/how_to/write_a_beaker_test_for_a_module.md +6 -2
  39. data/docs/tutorials/README.md +14 -55
  40. data/docs/tutorials/creating_a_test_environment.md +47 -52
  41. data/docs/tutorials/installation.md +57 -21
  42. data/docs/tutorials/lets_write_a_test.md +8 -1
  43. data/docs/tutorials/quick_start_rake_tasks.md +63 -62
  44. data/docs/tutorials/subcommands.md +19 -32
  45. data/docs/tutorials/test_run.md +14 -16
  46. data/docs/tutorials/test_suites.md +14 -43
  47. data/docs/tutorials/the_command_line.md +14 -2
  48. data/lib/beaker/host/unix/pkg.rb +9 -2
  49. data/lib/beaker/host_prebuilt_steps.rb +1 -1
  50. data/lib/beaker/shared/host_manager.rb +6 -3
  51. data/lib/beaker/version.rb +1 -1
  52. data/spec/beaker/host/unix/pkg_spec.rb +66 -17
  53. data/spec/beaker/host/unix_spec.rb +4 -4
  54. data/spec/beaker/host_prebuilt_steps_spec.rb +10 -6
  55. data/spec/beaker/hypervisor/hypervisor_spec.rb +2 -2
  56. data/spec/beaker/shared/host_manager_spec.rb +8 -0
  57. metadata +9 -2
@@ -3,10 +3,10 @@ require "helpers/test_helper"
3
3
  # construct an appropriate local file URL for curl testing
4
4
  def host_local_url(host, path)
5
5
  if host.is_cygwin?
6
- "file://#{path.gsub('/', '\\\\\\\\')}"
7
- else
8
- "file://#{path}"
6
+ path_result = on(host, "cygpath #{path}")
7
+ path = path_result.raw_output.chomp
9
8
  end
9
+ "file://#{path}"
10
10
  end
11
11
 
12
12
  test_name "dsl::helpers::host_helpers #curl_on" do
data/beaker.gemspec CHANGED
@@ -22,7 +22,7 @@ Gem::Specification.new do |s|
22
22
  # Testing dependencies
23
23
  s.add_development_dependency 'rspec', '~> 3.0'
24
24
  s.add_development_dependency 'rspec-its'
25
- s.add_development_dependency 'fakefs', '~> 0.6'
25
+ s.add_development_dependency 'fakefs', '~> 0.6', '< 0.14.0'
26
26
  s.add_development_dependency 'simplecov'
27
27
  s.add_development_dependency 'rake', '~> 10.0'
28
28
 
data/docs/README.md CHANGED
@@ -4,21 +4,14 @@ Documentation for Beaker can be found in this repository in [the docs/ folder]()
4
4
 
5
5
  ## Table of Contents
6
6
 
7
- - [Tutorials](tutorials) take you by the hand through the steps to setup a
8
- beaker run. Start here if you’re new to Beaker or test development.
9
- - [Concepts](concepts) discuss key topics and concepts at a fairly high
10
- level and provide useful background information and explanation.
11
- - [Rubydocs](http://rubydoc.info/github/puppetlabs/beaker/frames) contains the
12
- technical reference for APIs and other aspects of Beaker. They describe how it
13
- works and how to use it but assume that you have a basic understanding of key concepts.
14
- - [How-to guides](how_to) are recipes. They guide you through the steps
15
- involved in addressing key problems and use-cases. They are more advanced than
16
- tutorials and assume some knowledge of how Beaker works.
7
+ - [Tutorials](tutorials) take you by the hand through the steps to setup a beaker run. Start here if you’re new to Beaker or test development.
8
+ - [Concepts](concepts) discuss key topics and concepts at a fairly high level and provide useful background information and explanation.
9
+ - [Rubydocs](http://rubydoc.info/github/puppetlabs/beaker/frames) contains the technical reference for APIs and other aspects of Beaker. They describe how it works and how to use it but assume that you have a basic understanding of key concepts.
10
+ - [How-to guides](how_to) are recipes. They guide you through the steps involved in addressing key problems and use-cases. They are more advanced than tutorials and assume some knowledge of how Beaker works.
17
11
 
18
12
  ## Other Resources
19
13
 
20
- In addition to the overview above, which matches Beaker's main README docs section,
21
- this doc's README has some links to other outside resources:
14
+ In addition to the overview above, which matches Beaker's main README docs section, this doc's README has some links to other outside resources:
22
15
 
23
16
  * [Latest Gem Release Notes](https://github.com/puppetlabs/beaker/blob/master/HISTORY.md#LATEST)
24
17
  * [Video: Beaker 101 talk at PDXPUG, May 2014](https://www.youtube.com/watch?v=cSyJXTYFXFg)
@@ -1,6 +1,6 @@
1
1
  # Beaker Libraries
2
- Engineering at Puppet Labs has written several libraries that extends the functionality provided
3
- by Beaker.
2
+
3
+ Engineering at Puppet Labs has written several libraries that extends the functionality provided by Beaker.
4
4
 
5
5
  To learn how to create beaker libraries, see the [Beaker-Template](https://github.com/puppetlabs/beaker-template/blob/master/README.md) documentation.
6
6
 
@@ -0,0 +1,11 @@
1
+ # Glossary
2
+
3
+ Most terms used in Beaker documentation should be common. The following documents project jargon which may otherwise be confusing.
4
+
5
+ ## Coordinator
6
+
7
+ The Coordinator is the system on which Beaker itself is run. In many environments this will be a local host, typically a developer's primary machine. Used instead of Master to avoid confusion (see [Roles](/docs/concepts/roles_what_are_they.md)).
8
+
9
+ ## System Under Test
10
+
11
+ A System Under Test (SUT) is one of the systems which is the subject of testing with Beaker. Contrast the [Beaker Coordinator](#coordinator).
@@ -2,31 +2,16 @@
2
2
 
3
3
  ## Scope of this guide
4
4
 
5
- The purpose of this guide is to provide definitions for best practices when
6
- writing Beaker tests, both syntactically and stylistically. This guide will
7
- define and provide examples for preferred test layout and conventions. Common
8
- patterns that are recommended as well as patterns that should be avoided will be
9
- described.
10
-
11
- No style manual can cover every possible circumstance. When a judgement call
12
- becomes necessary, keep in mind the following general ideas:
13
-
14
- 1. **Readability matters**. If you have to choose between two equally effective
15
- alternatives, pick the more readable one. This is, of course, subjective, but if
16
- you can read your own code three months from now, that's a great start. Don't be
17
- clever over readable, unless you have a documented purpose. Use object oriented
18
- programming when things get complex and
19
- [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself).
20
- 2. **Inherit upstream conventions**. Beaker is still ruby, so use the
21
- [ruby community style guide](https://github.com/bbatsov/ruby-style-guide). When
22
- not called out here, use the ruby style guide.
5
+ The purpose of this guide is to provide definitions for best practices when writing Beaker tests, both syntactically and stylistically. This guide will define and provide examples for preferred test layout and conventions. Common patterns that are recommended as well as patterns that should be avoided will be described.
6
+
7
+ No style manual can cover every possible circumstance. When a judgement call becomes necessary, keep in mind the following general ideas:
8
+
9
+ 1. **Readability matters**. If you have to choose between two equally effective alternatives, pick the more readable one. This is, of course, subjective, but if you can read your own code three months from now, that's a great start. Don't be clever over readable, unless you have a documented purpose. Use object oriented programming when things get complex and [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself).
10
+ 2. **Inherit upstream conventions**. Beaker is still ruby, so use the [ruby community style guide](https://github.com/bbatsov/ruby-style-guide). When not called out here, use the ruby style guide.
23
11
 
24
12
  ## Test Naming
25
13
 
26
- Tests should test what they say they test. Test names, both the name of the test
27
- file and the value given to the
28
- [`test_name`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#test_name-instance_method)
29
- function, should provide an accurate indication about the purpose of the test.
14
+ Tests should test what they say they test. Test names, both the name of the test file and the value given to the [`test_name`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#test_name-instance_method) function, should provide an accurate indication about the purpose of the test.
30
15
 
31
16
  The `test_name` function should be the first line in the Beaker test file.
32
17
 
@@ -38,17 +23,9 @@ test_name "file resource: set default modes" do
38
23
 
39
24
  ## Structure Methods Should Use Explicit Blocks
40
25
 
41
- These methods aid in self-documenting your tests, including indention in the
42
- logs. If you don't use explicit blocks, beaker does not know how to properly
43
- indent your test's output.
26
+ These methods aid in self-documenting your tests, including indention in the logs. If you don't use explicit blocks, beaker does not know how to properly indent your test's output.
44
27
 
45
- The most common
46
- [structure methods](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure)
47
- are
48
- [`#test_name`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#test_name-instance_method),
49
- [`#step`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#step-instance_method),
50
- and
51
- [`#teardown`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#teardown-instance_method).
28
+ The most common [structure methods](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure) are [`#test_name`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#test_name-instance_method), [`#step`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#step-instance_method), and [`#teardown`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#teardown-instance_method).
52
29
 
53
30
  **Good:**
54
31
  ```ruby
@@ -67,17 +44,11 @@ on(host, "echo 'do this thang'")
67
44
  - Return the state of the system to the way it was prior to test execution
68
45
  - Put the teardown as early in the test as possible
69
46
 
70
- Teardowns must be used to return the system to the state it was in prior to the
71
- execution or attempted execution of the test. Beaker will gather all teardowns
72
- encountered throughout the execution path of the test. These teardowns will all
73
- be executed when the test exits, even if the test exits early due to a failure
74
- or error.
47
+ Teardowns must be used to return the system to the state it was in prior to the execution or attempted execution of the test. Beaker will gather all teardowns encountered throughout the execution path of the test. These teardowns will all be executed when the test exits, even if the test exits early due to a failure or error.
75
48
 
76
49
  ### Place Teardowns Early
77
50
 
78
- Teardowns can be placed anywhere in the test file or its helpers. The preferred
79
- style is to have a teardown step near the beginning of the test file to show the
80
- reader that the system state will be restored.
51
+ Teardowns can be placed anywhere in the test file or its helpers. The preferred style is to have a teardown step near the beginning of the test file to show the reader that the system state will be restored.
81
52
 
82
53
  **Good:**
83
54
  ```ruby
@@ -93,25 +64,13 @@ test_name 'The source attribute' do
93
64
  end
94
65
  ```
95
66
 
96
- Teardowns are at the mercy of the scoping of the variables necessary to perform
97
- the restoration of the system. This fact means that additional teardown steps
98
- will need to be added within the scope necessary to do their job. Effort should
99
- be taken to make the teardown steps prominent and readable so that it can be
100
- confirmed, via the logs, that the system has been restored.
67
+ Teardowns are at the mercy of the scoping of the variables necessary to perform the restoration of the system. This fact means that additional teardown steps will need to be added within the scope necessary to do their job. Effort should be taken to make the teardown steps prominent and readable so that it can be confirmed, via the logs, that the system has been restored.
101
68
 
102
- Teardown steps registered outside of tests should use
103
- [`#step`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#step-instance_method)
104
- to document and log what they are doing.
69
+ Teardown steps registered outside of tests should use [`#step`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#step-instance_method) to document and log what they are doing.
105
70
 
106
71
  ## Acceptable Exit Codes
107
72
 
108
- When using the Beaker
109
- [`on`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Helpers/HostHelpers#on-instance_method)
110
- method, the default setting is that only an exit code of 0 (zero) will not
111
- trigger an error. When other exit codes are acceptable, the
112
- `:acceptable_exit_codes` key with an array of exit codes should be passed in the
113
- options hash to `#on`. If 0 (zero) is the only acceptable exit code, then the
114
- `:acceptable_exit_codes` symbol must not be used.
73
+ When using the Beaker [`on`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Helpers/HostHelpers#on-instance_method) method, the default setting is that only an exit code of 0 (zero) will not trigger an error. When other exit codes are acceptable, the `:acceptable_exit_codes` key with an array of exit codes should be passed in the options hash to `#on`. If 0 (zero) is the only acceptable exit code, then the `:acceptable_exit_codes` symbol must not be used.
115
74
 
116
75
  **Good:**
117
76
 
@@ -132,13 +91,9 @@ on(host, "rm #{file_to_rm}", :acceptable_exit_codes => [0,1])
132
91
  on(host, "rm #{file_to_rm}", :accept_all_exit_codes => true)
133
92
  ```
134
93
 
135
- In the last case, when any exit code is allowed, one must follow-up with a valid
136
- assertion test.
94
+ In the last case, when any exit code is allowed, one must follow-up with a valid assertion test.
137
95
 
138
- If an exit_code outside of 0 is expected, one must use acceptable_exit_codes so
139
- the test will fail on the proper assertion and not error at that command. Allow
140
- only the minimum expected set of exit codes unless coverage is provided by
141
- subsequent assertions.
96
+ If an exit_code outside of 0 is expected, one must use acceptable_exit_codes so the test will fail on the proper assertion and not error at that command. Allow only the minimum expected set of exit codes unless coverage is provided by subsequent assertions.
142
97
 
143
98
  ## Test Outcomes
144
99
 
@@ -146,10 +101,7 @@ When to use each, and how to format the message:
146
101
 
147
102
  ### Expecting Failure
148
103
 
149
- If your tests are failing due to an "expected failure", you should wrap your
150
- failing assertion in an
151
- [`expect_failure`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#expect_failure-instance_method)
152
- block with an explanatory logging message:
104
+ If your tests are failing due to an "expected failure", you should wrap your failing assertion in an [`expect_failure`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#expect_failure-instance_method) block with an explanatory logging message:
153
105
 
154
106
  ```ruby
155
107
  expect_failure('expected to fail due to PE-1234') do
@@ -157,9 +109,7 @@ expect_failure('expected to fail due to PE-1234') do
157
109
  end
158
110
  ```
159
111
 
160
- Note that `expect_failure` will only trigger from failed assertions. It won't
161
- take care of failed host or `on` commands. To deal with expected failure from an
162
- `on` invocation, you'd want something more like this:
112
+ Note that `expect_failure` will only trigger from failed assertions. It won't take care of failed host or `on` commands. To deal with expected failure from an `on` invocation, you'd want something more like this:
163
113
 
164
114
  ```ruby
165
115
  on(blah, 'blah', :allow_all_exit_codes => true) do |result|
@@ -171,26 +121,15 @@ end
171
121
 
172
122
  ### `fail_test`, `pass_test`, & `skip_test`
173
123
 
174
- These can be used anywhere in a test to exit early. A
175
- [`skip_test`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Outcomes#fail_test-instance_method)
176
- between two assertions, for instance, will run the first assertion, raise an
177
- exception for `skip_test`, run teardown, and then exit as we expect. The same is
178
- true for
179
- [fail_test](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Outcomes#fail_test-instance_method).
124
+ These can be used anywhere in a test to exit early. A [`skip_test`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Outcomes#fail_test-instance_method) between two assertions, for instance, will run the first assertion, raise an exception for `skip_test`, run teardown, and then exit as we expect. The same is true for [fail_test](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Outcomes#fail_test-instance_method).
180
125
 
181
- [`pass_test`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Outcomes#fail_test-instance_method)
182
- is typically not required. When the end of a test is reached without causing an
183
- error (due to bad test code execution, or an unhandled exit code or exception)
184
- or failure (due to assertions), then it passes.
126
+ [`pass_test`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Outcomes#fail_test-instance_method) is typically not required. When the end of a test is reached without causing an error (due to bad test code execution, or an unhandled exit code or exception) or failure (due to assertions), then it passes.
185
127
 
186
- `pass_test` can be used in a situation where one knows a test has passed before
187
- the end of a test under certain circumstances, such as during a loop that has
188
- not yet completed.
128
+ `pass_test` can be used in a situation where one knows a test has passed before the end of a test under certain circumstances, such as during a loop that has not yet completed.
189
129
 
190
130
  ### Skipping Tests
191
131
 
192
- Skipping tests can be used, for instance, when they are temporarily failing or
193
- not yet complete.
132
+ Skipping tests can be used, for instance, when they are temporarily failing or not yet complete.
194
133
 
195
134
  **Good:**
196
135
  ```ruby
@@ -203,14 +142,8 @@ confine :to, :platform => 'solaris:pending'
203
142
 
204
143
  ## Confining
205
144
 
206
- Another way that you can skip or manipulate tests is by confining them to apply
207
- to a subset of the SUTs available for testing. Confining is a complex topic, and
208
- one we don't have the length to get into in the style guide. For an explanation
209
- of confining, as well as the best practices in using it, check out our
210
- [confine doc](../how_to/confine.md).
145
+ Another way that you can skip or manipulate tests is by confining them to apply to a subset of the SUTs available for testing. Confining is a complex topic, and one we don't have the length to get into in the style guide. For an explanation of confining, as well as the best practices in using it, check out our [confine doc](../how_to/confine.md).
211
146
 
212
147
  ## Assertions
213
148
 
214
- Always include a _unique_ error message in your assertion statement. Use strict
215
- asserts whenever possible (e.g. assert_equal, assert_match.
216
- [More info](http://danwin.com/2013/03/ruby-minitest-cheat-sheet/)).
149
+ Always include a _unique_ error message in your assertion statement. Use strict asserts whenever possible (e.g. assert_equal, assert_match. [More info](http://danwin.com/2013/03/ruby-minitest-cheat-sheet/)).
@@ -1,19 +1,13 @@
1
1
  ## What is This?
2
2
 
3
- Beaker test tagging allows you to add tags to tests (using the
4
- [`tag` DSL method](http://www.rubydoc.info/github/puppetlabs/beaker/master/Beaker/DSL/TestTagging#tag-instance_method)),
5
- so that you can include or exclude a specific subset of the tests given for use
6
- in this run. Why would you want to use this? Here are some examples of what
7
- you can do with this functionality:
3
+ Beaker test tagging allows you to add tags to tests (using the [`tag` DSL method](http://www.rubydoc.info/github/puppetlabs/beaker/master/Beaker/DSL/TestTagging#tag-instance_method)), so that you can include or exclude a specific subset of the tests given for use in this run. Why would you want to use this? Here are some examples of what you can do with this functionality:
8
4
 
9
5
  - Run groups of tests separately from the same testing codebase
10
6
  - Declare different actions that should be taken when a test fails
11
7
  - Make new tests go through a provisional process before being considered solid tests
12
8
 
13
9
  ## How Tagging Works
14
-
15
- Add tags to a Beaker test at the beginning, like you would if you were using
16
- confine. Things to stay aware of:
10
+ Add tags to a Beaker test at the beginning, like you would if you were using confine. Things to stay aware of:
17
11
 
18
12
  - A test that is not executed due to a tag will be considered a ‘skipped’ test
19
13
  - Tags are free form strings and will not be subjected to any correctness testing
@@ -53,8 +47,7 @@ Preferred style no-block example:
53
47
 
54
48
  Beaker will raise an error if `--test-tag-and` & `--test-tag-exclude` contain the same tag, however.
55
49
 
56
- Beaker will also raise an error if you use both `--test-tag-or` & `--test-tag-and`, because
57
- it won't be able to determine which order they should be used in.
50
+ Beaker will also raise an error if you use both `--test-tag-or` & `--test-tag-and`, because it won't be able to determine which order they should be used in.
58
51
 
59
52
  ## CLI Examples
60
53
 
@@ -6,8 +6,7 @@ While Beaker provides the testing harness for much of the acceptance testing tha
6
6
 
7
7
  ### Product Coverage
8
8
 
9
- Beaker test coverage covers the LTS PE version, currently 2016.4.0, and the latest released version of PE, currently 2016.5.0.
10
- Since there is only a single major version of Puppet itself currently supported, beaker only run tests on the latest y-release of Puppet 4, currently 4.8.z. This currently resolves to puppet-agent 1.8.x.
9
+ Beaker test coverage covers the LTS PE version, currently 2016.4.0, and the latest released version of PE, currently 2016.5.0. Since there is only a single major version of Puppet itself currently supported, beaker only run tests on the latest y-release of Puppet 4, currently 4.8.z. This currently resolves to puppet-agent 1.8.x.
11
10
 
12
11
  ### Platform Coverage
13
12
 
@@ -16,9 +15,11 @@ The platforms that beaker covers in its regression testing are largely what is s
16
15
  ## Test Suite Phases
17
16
 
18
17
  ### Beaker Spec
18
+
19
19
  The initial step in Beaker's pipeline is to execute spec testing with supported and future rubies; 2.2.5 and 2.3.1.
20
20
 
21
21
  ### Beaker Acceptance
22
+
22
23
  All acceptance tests use actual OS's with beaker installed and use beaker itself to verify that its own methods and classes are working.
23
24
 
24
25
  * The Base tests are tests that do not require puppet be installed on the SUT. This includes much of the DSL and host helpers.
@@ -1,75 +1,49 @@
1
- This doc is here to explain the lifecycle of a Beaker ticket. If you have any
2
- questions about the workflow of a Beaker ticket, or the stage that it's in, you
3
- should be able to answer them here. If not, then please let QE know, and we'll
4
- work to answer your question and update this doc, so that we can do this better
5
- going forward.
1
+ # Ticket Process
6
2
 
7
- Note that a typical Beaker ticket goes through these states. They can sometimes
8
- go in order, but on average, they loop around through the various stages,
9
- can cycle a number of times before becoming resolved, and will sometimes skip
10
- stages as well.
3
+ This doc is here to explain the lifecycle of a Beaker ticket. If you have any questions about the workflow of a Beaker ticket, or the stage that it's in, you should be able to answer them here. If not, then please let QE know, and we'll work to answer your question and update this doc, so that we can do this better going forward.
4
+
5
+ Note that a typical Beaker ticket goes through these states. They can sometimes go in order, but on average, they loop around through the various stages, can cycle a number of times before becoming resolved, and will sometimes skip stages as well.
11
6
 
12
7
  ## Administrivia
13
8
 
14
- - Beaker tickets live in the
15
- [BKR project](https://tickets.puppetlabs.com/issues/?jql=project%20%3D%20BKR)
9
+ - Beaker tickets live in the [BKR project](https://tickets.puppetlabs.com/issues/?jql=project%20%3D%20BKR)
16
10
  - Create a [Jira account](http://tickets.puppetlabs.com) to interact with them
17
11
 
18
12
  ## Pre-Development States
19
13
 
20
- These are the states that a Beaker ticket goes through before it is picked up for
21
- development. Note that this is only for work done by the Beaker team. If there
22
- is something you'd like to contribute to, we would appreciate that you go
23
- through the development states for tracking purposes, but you can skip directly
24
- to the "In Progress" state if you're going to work on a ticket.
14
+ These are the states that a Beaker ticket goes through before it is picked up for development. Note that this is only for work done by the Beaker team. If there is something you'd like to contribute to, we would appreciate that you go through the development states for tracking purposes, but you can skip directly to the "In Progress" state if you're going to work on a ticket.
25
15
 
26
16
  ### Open
27
17
 
28
- The state for newly filed issues. These have not been triaged, and are
29
- waiting to be looked at by the team.
18
+ The state for newly filed issues. These have not been triaged, and are waiting to be looked at by the team.
30
19
 
31
20
  ### Needs Information
32
21
 
33
- The state for blocked issues. These can be blocked by another issue (please link
34
- that issue from the ticket if that's the case), or they can be blocked on info
35
- needed from another party. If this is the case, please assign the ticket to the
36
- person you need info from to move the ticket forward.
22
+ The state for blocked issues. These can be blocked by another issue (please link that issue from the ticket if that's the case), or they can be blocked on info needed from another party. If this is the case, please assign the ticket to the person you need info from to move the ticket forward.
37
23
 
38
24
  ### Accepted
39
25
 
40
- This is the state for when a ticket has been triaged, but hasn't yet been
41
- estimated for work from someone on the Beaker team, or someone ready to
42
- take on the work. Any preliminary information needed to understand the
43
- ticket (before investigation) should be gathered before this point.
26
+ This is the state for when a ticket has been triaged, but hasn't yet been estimated for work from someone on the Beaker team, or someone ready to take on the work. Any preliminary information needed to understand the ticket (before investigation) should be gathered before this point.
44
27
 
45
28
  ### Ready for Engineering
46
29
 
47
- An accepted ticket that has been estimated should be put into the Ready for
48
- Engineering state. This does not necessarily mean that a ticket has been
49
- prioritized against other Beaker work, but prioritization should occur by
50
- the time the issue has been picked up in a sprint.
30
+ An accepted ticket that has been estimated should be put into the Ready for Engineering state. This does not necessarily mean that a ticket has been prioritized against other Beaker work, but prioritization should occur by the time the issue has been picked up in a sprint.
51
31
 
52
32
  ## Development States
53
33
 
54
- Once you're ready to pick up a ticket & work on it, then it should go through
55
- these states.
34
+ Once you're ready to pick up a ticket & work on it, then it should go through these states.
56
35
 
57
36
  ### In Progress
58
37
 
59
- This state is set aside for a Beaker contributor to let others know that
60
- they're currently working on a particular issue.
38
+ This state is set aside for a Beaker contributor to let others know that they're currently working on a particular issue.
61
39
 
62
40
  ### Ready for Merge
63
41
 
64
- Once a Pull Request (PR) is generated for an issue, the contributor should set
65
- the status to Ready for Merge. We do have a github integration setup, so if
66
- you've titled your PR correctly (according to our
67
- [contributor docs](/CONTRIBUTING.md)),
68
- it will be linked from the JIRA ticket.
42
+ Once a Pull Request (PR) is generated for an issue, the contributor should set the status to Ready for Merge. We do have a GitHub integration setup, so if you've titled your PR correctly (according to our [contributor docs](/CONTRIBUTING.md)), it will be linked from the JIRA ticket.
43
+
44
+ In the Beaker project, we leave the assignee as the person who wrote the proposed change, so that they know that they have to keep pushing for their code to be merged.
69
45
 
70
- In the Beaker project, we leave the assignee as the person who wrote the
71
- proposed change, so that they know that they have to keep pushing for their
72
- code to be merged.
46
+ A team member will review the PR. If changes are required, they will be communicated in the PR discussion on GitHub.
73
47
 
74
48
  ### Resolved
75
49
 
@@ -79,6 +53,4 @@ Once your PR is merged, then you can Resolve your ticket.
79
53
 
80
54
  BKR.next
81
55
 
82
- The reason that we do this now and not before is that we use this field to
83
- autogenerate our release notes. We want to make sure that we capture only
84
- work that is _in_ the next release, not work that's _intended_ for it.
56
+ The reason that we do this now and not before is that we use this field to autogenerate our release notes. We want to make sure that we capture only work that is _in_ the next release, not work that's _intended_ for it.
@@ -1,49 +1,23 @@
1
1
  # How to Archive Files from the Systems Under Test (SUTs)
2
2
 
3
- Oftentimes when you're dealing with beaker test development or troubleshooting
4
- a failed acceptance test, you'll need to get information from a SUT. The
5
- traditional way that we've advocated getting information from these machines is
6
- to use our [preserved hosts functionality](preserve_hosts.md).
3
+ Oftentimes when you're dealing with beaker test development or troubleshooting a failed acceptance test, you'll need to get information from a SUT. The traditional way that we've advocated getting information from these machines is to use our [preserved hosts functionality](preserve_hosts.md).
7
4
 
8
- If you're preserving hosts just to SSH in and look at log files, however, this
9
- can be a tedious exercise. Why not just bring the log files to you on the
10
- beaker coordinator? This doc explains exactly how to do that using our
11
- `archive_file_from` Domain-Specific Language (DSL) method
12
- ([method rubydocs](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Helpers/HostHelpers#archive_file_from-instance_method)).
5
+ If you're preserving hosts just to SSH in and look at log files, however, this can be a tedious exercise. Why not just bring the log files to you on the beaker coordinator? This doc explains exactly how to do that using our `archive_file_from` Domain-Specific Language (DSL) method ([method rubydocs](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Helpers/HostHelpers#archive_file_from-instance_method)).
13
6
 
14
7
  # How Do I Use This?
15
8
 
16
- `archive_file_from` is a part of the beaker DSL, so it's available in all test
17
- suites. Just call it from your tests, and it'll execute, pulling any particular
18
- file you need off your SUTs, and dropping it on your beaker coordinator's
19
- file system.
9
+ `archive_file_from` is a part of the beaker DSL, so it's available in all test suites. Just call it from your tests, and it'll execute, pulling any particular file you need off your SUTs, and dropping it on your beaker coordinator's file system.
20
10
 
21
- A common example of a post-suite step to archive files that were created during
22
- a particular test is included in the Rubydocs, referenced above. Path details,
23
- and details of all method arguments are documented there as well. Check it out,
24
- and with the right use, you won't need to preserve hosts at all to debug any
25
- test failures.
11
+ A common example of a post-suite step to archive files that were created during a particular test is included in the Rubydocs, referenced above. Path details, and details of all method arguments are documented there as well. Check it out, and with the right use, you won't need to preserve hosts at all to debug any test failures.
26
12
 
27
13
  # Challenges
28
14
 
29
15
  ## Conditionally Saving Files From SUTs
30
16
 
31
- One thing that people tend to want from this functionality is to only archive
32
- files from SUTs when a beaker run has failed. At this point, beaker does not
33
- have access to other suites from a current one. This means that in practice,
34
- a post-suite (where one would typically put archiving and other post-processes)
35
- will not be able to archive files ONLY IF the test suite has had any failures
36
- or errors.
17
+ One thing that people tend to want from this functionality is to only archive files from SUTs when a beaker run has failed. At this point, beaker does not have access to other suites from a current one. This means that in practice, a post-suite (where one would typically put archiving and other post-processes) will not be able to archive files ONLY IF the test suite has had any failures or errors.
37
18
 
38
- Our suggestion to get the functionality required would be to have beaker always
39
- archive the appropriate files in the post-suite of your tests, but then only
40
- have Jenkins (or your job running system, whatever that may be) conditionally
41
- take them from the beaker coordinator to whatever external archive system you
42
- rely on for later analysis. This can both get you the files that you need from
43
- the SUTs and save on space, as only files that need analysis will be kept.
19
+ Our suggestion to get the functionality required would be to have beaker always archive the appropriate files in the post-suite of your tests, but then only have Jenkins (or your job running system, whatever that may be) conditionally take them from the beaker coordinator to whatever external archive system you rely on for later analysis. This can both get you the files that you need from the SUTs and save on space, as only files that need analysis will be kept.
44
20
 
45
21
  # When Did This Come Out?
46
22
 
47
- `archive_file_from` was originally added to the DSL in beaker
48
- [2.48.0](https://github.com/puppetlabs/beaker/releases/tag/2.48.0), released on
49
- [July 27, 2016](https://github.com/puppetlabs/beaker/blob/master/HISTORY.md#2480---27-jul-2016-47d3aa18).
23
+ `archive_file_from` was originally added to the DSL in beaker [2.48.0](https://github.com/puppetlabs/beaker/releases/tag/2.48.0), released on [July 27, 2016](https://github.com/puppetlabs/beaker/blob/master/HISTORY.md#2480---27-jul-2016-47d3aa18).