beaker 3.35.0 → 3.36.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.gitignore +32 -9
- data/CHANGELOG.md +15 -1
- data/CONTRIBUTING.md +37 -31
- data/DOCUMENTING.md +68 -44
- data/README.md +14 -31
- data/acceptance/tests/base/dsl/helpers/host_helpers/curl_on_test.rb +3 -3
- data/beaker.gemspec +1 -1
- data/docs/README.md +5 -12
- data/docs/concepts/beaker_libraries.md +2 -2
- data/docs/concepts/glossary.md +11 -0
- data/docs/concepts/style_guide.md +24 -91
- data/docs/concepts/test_tagging.md +3 -10
- data/docs/concepts/testing_beaker_itself.md +3 -2
- data/docs/concepts/ticket_process.md +17 -45
- data/docs/how_to/archive_sut_files.md +7 -33
- data/docs/how_to/change_terminal_output_coloring.md +7 -4
- data/docs/how_to/confine.md +5 -16
- data/docs/how_to/debug_beaker_tests.md +14 -6
- data/docs/how_to/hosts/README.md +2 -5
- data/docs/how_to/hosts/archlinux.md +4 -2
- data/docs/how_to/hosts/cisco.md +9 -30
- data/docs/how_to/hosts/eos.md +5 -14
- data/docs/how_to/hypervisors/README.md +9 -31
- data/docs/how_to/install_puppet.md +16 -51
- data/docs/how_to/platform_specific_tag_confines.md +11 -31
- data/docs/how_to/preserve_hosts.md +13 -54
- data/docs/how_to/rake_tasks.md +1 -4
- data/docs/how_to/recipes.md +9 -2
- data/docs/how_to/run_in_parallel.md +10 -10
- data/docs/how_to/ssh_agent_forwarding.md +8 -13
- data/docs/how_to/ssh_connection_preference.md +10 -10
- data/docs/how_to/test_arbitrary_beaker_versions.md +5 -12
- data/docs/how_to/the_beaker_dsl.md +44 -21
- data/docs/how_to/upgrade_from_2_to_3.md +13 -53
- data/docs/how_to/use_hocon_helpers.md +10 -28
- data/docs/how_to/use_user_password_authentication.md +13 -10
- data/docs/how_to/write_a_beaker_test_for_a_module.md +6 -2
- data/docs/tutorials/README.md +14 -55
- data/docs/tutorials/creating_a_test_environment.md +47 -52
- data/docs/tutorials/installation.md +57 -21
- data/docs/tutorials/lets_write_a_test.md +8 -1
- data/docs/tutorials/quick_start_rake_tasks.md +63 -62
- data/docs/tutorials/subcommands.md +19 -32
- data/docs/tutorials/test_run.md +14 -16
- data/docs/tutorials/test_suites.md +14 -43
- data/docs/tutorials/the_command_line.md +14 -2
- data/lib/beaker/host/unix/pkg.rb +9 -2
- data/lib/beaker/host_prebuilt_steps.rb +1 -1
- data/lib/beaker/shared/host_manager.rb +6 -3
- data/lib/beaker/version.rb +1 -1
- data/spec/beaker/host/unix/pkg_spec.rb +66 -17
- data/spec/beaker/host/unix_spec.rb +4 -4
- data/spec/beaker/host_prebuilt_steps_spec.rb +10 -6
- data/spec/beaker/hypervisor/hypervisor_spec.rb +2 -2
- data/spec/beaker/shared/host_manager_spec.rb +8 -0
- metadata +9 -2
@@ -3,10 +3,10 @@ require "helpers/test_helper"
|
|
3
3
|
# construct an appropriate local file URL for curl testing
|
4
4
|
def host_local_url(host, path)
|
5
5
|
if host.is_cygwin?
|
6
|
-
"
|
7
|
-
|
8
|
-
"file://#{path}"
|
6
|
+
path_result = on(host, "cygpath #{path}")
|
7
|
+
path = path_result.raw_output.chomp
|
9
8
|
end
|
9
|
+
"file://#{path}"
|
10
10
|
end
|
11
11
|
|
12
12
|
test_name "dsl::helpers::host_helpers #curl_on" do
|
data/beaker.gemspec
CHANGED
@@ -22,7 +22,7 @@ Gem::Specification.new do |s|
|
|
22
22
|
# Testing dependencies
|
23
23
|
s.add_development_dependency 'rspec', '~> 3.0'
|
24
24
|
s.add_development_dependency 'rspec-its'
|
25
|
-
s.add_development_dependency 'fakefs', '~> 0.6'
|
25
|
+
s.add_development_dependency 'fakefs', '~> 0.6', '< 0.14.0'
|
26
26
|
s.add_development_dependency 'simplecov'
|
27
27
|
s.add_development_dependency 'rake', '~> 10.0'
|
28
28
|
|
data/docs/README.md
CHANGED
@@ -4,21 +4,14 @@ Documentation for Beaker can be found in this repository in [the docs/ folder]()
|
|
4
4
|
|
5
5
|
## Table of Contents
|
6
6
|
|
7
|
-
- [Tutorials](tutorials) take you by the hand through the steps to setup a
|
8
|
-
|
9
|
-
- [
|
10
|
-
|
11
|
-
- [Rubydocs](http://rubydoc.info/github/puppetlabs/beaker/frames) contains the
|
12
|
-
technical reference for APIs and other aspects of Beaker. They describe how it
|
13
|
-
works and how to use it but assume that you have a basic understanding of key concepts.
|
14
|
-
- [How-to guides](how_to) are recipes. They guide you through the steps
|
15
|
-
involved in addressing key problems and use-cases. They are more advanced than
|
16
|
-
tutorials and assume some knowledge of how Beaker works.
|
7
|
+
- [Tutorials](tutorials) take you by the hand through the steps to setup a beaker run. Start here if you’re new to Beaker or test development.
|
8
|
+
- [Concepts](concepts) discuss key topics and concepts at a fairly high level and provide useful background information and explanation.
|
9
|
+
- [Rubydocs](http://rubydoc.info/github/puppetlabs/beaker/frames) contains the technical reference for APIs and other aspects of Beaker. They describe how it works and how to use it but assume that you have a basic understanding of key concepts.
|
10
|
+
- [How-to guides](how_to) are recipes. They guide you through the steps involved in addressing key problems and use-cases. They are more advanced than tutorials and assume some knowledge of how Beaker works.
|
17
11
|
|
18
12
|
## Other Resources
|
19
13
|
|
20
|
-
In addition to the overview above, which matches Beaker's main README docs section,
|
21
|
-
this doc's README has some links to other outside resources:
|
14
|
+
In addition to the overview above, which matches Beaker's main README docs section, this doc's README has some links to other outside resources:
|
22
15
|
|
23
16
|
* [Latest Gem Release Notes](https://github.com/puppetlabs/beaker/blob/master/HISTORY.md#LATEST)
|
24
17
|
* [Video: Beaker 101 talk at PDXPUG, May 2014](https://www.youtube.com/watch?v=cSyJXTYFXFg)
|
@@ -1,6 +1,6 @@
|
|
1
1
|
# Beaker Libraries
|
2
|
-
|
3
|
-
by Beaker.
|
2
|
+
|
3
|
+
Engineering at Puppet Labs has written several libraries that extends the functionality provided by Beaker.
|
4
4
|
|
5
5
|
To learn how to create beaker libraries, see the [Beaker-Template](https://github.com/puppetlabs/beaker-template/blob/master/README.md) documentation.
|
6
6
|
|
@@ -0,0 +1,11 @@
|
|
1
|
+
# Glossary
|
2
|
+
|
3
|
+
Most terms used in Beaker documentation should be common. The following documents project jargon which may otherwise be confusing.
|
4
|
+
|
5
|
+
## Coordinator
|
6
|
+
|
7
|
+
The Coordinator is the system on which Beaker itself is run. In many environments this will be a local host, typically a developer's primary machine. Used instead of Master to avoid confusion (see [Roles](/docs/concepts/roles_what_are_they.md)).
|
8
|
+
|
9
|
+
## System Under Test
|
10
|
+
|
11
|
+
A System Under Test (SUT) is one of the systems which is the subject of testing with Beaker. Contrast the [Beaker Coordinator](#coordinator).
|
@@ -2,31 +2,16 @@
|
|
2
2
|
|
3
3
|
## Scope of this guide
|
4
4
|
|
5
|
-
The purpose of this guide is to provide definitions for best practices when
|
6
|
-
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
No style manual can cover every possible circumstance. When a judgement call
|
12
|
-
becomes necessary, keep in mind the following general ideas:
|
13
|
-
|
14
|
-
1. **Readability matters**. If you have to choose between two equally effective
|
15
|
-
alternatives, pick the more readable one. This is, of course, subjective, but if
|
16
|
-
you can read your own code three months from now, that's a great start. Don't be
|
17
|
-
clever over readable, unless you have a documented purpose. Use object oriented
|
18
|
-
programming when things get complex and
|
19
|
-
[DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself).
|
20
|
-
2. **Inherit upstream conventions**. Beaker is still ruby, so use the
|
21
|
-
[ruby community style guide](https://github.com/bbatsov/ruby-style-guide). When
|
22
|
-
not called out here, use the ruby style guide.
|
5
|
+
The purpose of this guide is to provide definitions for best practices when writing Beaker tests, both syntactically and stylistically. This guide will define and provide examples for preferred test layout and conventions. Common patterns that are recommended as well as patterns that should be avoided will be described.
|
6
|
+
|
7
|
+
No style manual can cover every possible circumstance. When a judgement call becomes necessary, keep in mind the following general ideas:
|
8
|
+
|
9
|
+
1. **Readability matters**. If you have to choose between two equally effective alternatives, pick the more readable one. This is, of course, subjective, but if you can read your own code three months from now, that's a great start. Don't be clever over readable, unless you have a documented purpose. Use object oriented programming when things get complex and [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself).
|
10
|
+
2. **Inherit upstream conventions**. Beaker is still ruby, so use the [ruby community style guide](https://github.com/bbatsov/ruby-style-guide). When not called out here, use the ruby style guide.
|
23
11
|
|
24
12
|
## Test Naming
|
25
13
|
|
26
|
-
Tests should test what they say they test. Test names, both the name of the test
|
27
|
-
file and the value given to the
|
28
|
-
[`test_name`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#test_name-instance_method)
|
29
|
-
function, should provide an accurate indication about the purpose of the test.
|
14
|
+
Tests should test what they say they test. Test names, both the name of the test file and the value given to the [`test_name`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#test_name-instance_method) function, should provide an accurate indication about the purpose of the test.
|
30
15
|
|
31
16
|
The `test_name` function should be the first line in the Beaker test file.
|
32
17
|
|
@@ -38,17 +23,9 @@ test_name "file resource: set default modes" do
|
|
38
23
|
|
39
24
|
## Structure Methods Should Use Explicit Blocks
|
40
25
|
|
41
|
-
These methods aid in self-documenting your tests, including indention in the
|
42
|
-
logs. If you don't use explicit blocks, beaker does not know how to properly
|
43
|
-
indent your test's output.
|
26
|
+
These methods aid in self-documenting your tests, including indention in the logs. If you don't use explicit blocks, beaker does not know how to properly indent your test's output.
|
44
27
|
|
45
|
-
The most common
|
46
|
-
[structure methods](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure)
|
47
|
-
are
|
48
|
-
[`#test_name`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#test_name-instance_method),
|
49
|
-
[`#step`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#step-instance_method),
|
50
|
-
and
|
51
|
-
[`#teardown`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#teardown-instance_method).
|
28
|
+
The most common [structure methods](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure) are [`#test_name`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#test_name-instance_method), [`#step`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#step-instance_method), and [`#teardown`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#teardown-instance_method).
|
52
29
|
|
53
30
|
**Good:**
|
54
31
|
```ruby
|
@@ -67,17 +44,11 @@ on(host, "echo 'do this thang'")
|
|
67
44
|
- Return the state of the system to the way it was prior to test execution
|
68
45
|
- Put the teardown as early in the test as possible
|
69
46
|
|
70
|
-
Teardowns must be used to return the system to the state it was in prior to the
|
71
|
-
execution or attempted execution of the test. Beaker will gather all teardowns
|
72
|
-
encountered throughout the execution path of the test. These teardowns will all
|
73
|
-
be executed when the test exits, even if the test exits early due to a failure
|
74
|
-
or error.
|
47
|
+
Teardowns must be used to return the system to the state it was in prior to the execution or attempted execution of the test. Beaker will gather all teardowns encountered throughout the execution path of the test. These teardowns will all be executed when the test exits, even if the test exits early due to a failure or error.
|
75
48
|
|
76
49
|
### Place Teardowns Early
|
77
50
|
|
78
|
-
Teardowns can be placed anywhere in the test file or its helpers. The preferred
|
79
|
-
style is to have a teardown step near the beginning of the test file to show the
|
80
|
-
reader that the system state will be restored.
|
51
|
+
Teardowns can be placed anywhere in the test file or its helpers. The preferred style is to have a teardown step near the beginning of the test file to show the reader that the system state will be restored.
|
81
52
|
|
82
53
|
**Good:**
|
83
54
|
```ruby
|
@@ -93,25 +64,13 @@ test_name 'The source attribute' do
|
|
93
64
|
end
|
94
65
|
```
|
95
66
|
|
96
|
-
Teardowns are at the mercy of the scoping of the variables necessary to perform
|
97
|
-
the restoration of the system. This fact means that additional teardown steps
|
98
|
-
will need to be added within the scope necessary to do their job. Effort should
|
99
|
-
be taken to make the teardown steps prominent and readable so that it can be
|
100
|
-
confirmed, via the logs, that the system has been restored.
|
67
|
+
Teardowns are at the mercy of the scoping of the variables necessary to perform the restoration of the system. This fact means that additional teardown steps will need to be added within the scope necessary to do their job. Effort should be taken to make the teardown steps prominent and readable so that it can be confirmed, via the logs, that the system has been restored.
|
101
68
|
|
102
|
-
Teardown steps registered outside of tests should use
|
103
|
-
[`#step`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#step-instance_method)
|
104
|
-
to document and log what they are doing.
|
69
|
+
Teardown steps registered outside of tests should use [`#step`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#step-instance_method) to document and log what they are doing.
|
105
70
|
|
106
71
|
## Acceptable Exit Codes
|
107
72
|
|
108
|
-
When using the Beaker
|
109
|
-
[`on`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Helpers/HostHelpers#on-instance_method)
|
110
|
-
method, the default setting is that only an exit code of 0 (zero) will not
|
111
|
-
trigger an error. When other exit codes are acceptable, the
|
112
|
-
`:acceptable_exit_codes` key with an array of exit codes should be passed in the
|
113
|
-
options hash to `#on`. If 0 (zero) is the only acceptable exit code, then the
|
114
|
-
`:acceptable_exit_codes` symbol must not be used.
|
73
|
+
When using the Beaker [`on`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Helpers/HostHelpers#on-instance_method) method, the default setting is that only an exit code of 0 (zero) will not trigger an error. When other exit codes are acceptable, the `:acceptable_exit_codes` key with an array of exit codes should be passed in the options hash to `#on`. If 0 (zero) is the only acceptable exit code, then the `:acceptable_exit_codes` symbol must not be used.
|
115
74
|
|
116
75
|
**Good:**
|
117
76
|
|
@@ -132,13 +91,9 @@ on(host, "rm #{file_to_rm}", :acceptable_exit_codes => [0,1])
|
|
132
91
|
on(host, "rm #{file_to_rm}", :accept_all_exit_codes => true)
|
133
92
|
```
|
134
93
|
|
135
|
-
In the last case, when any exit code is allowed, one must follow-up with a valid
|
136
|
-
assertion test.
|
94
|
+
In the last case, when any exit code is allowed, one must follow-up with a valid assertion test.
|
137
95
|
|
138
|
-
If an exit_code outside of 0 is expected, one must use acceptable_exit_codes so
|
139
|
-
the test will fail on the proper assertion and not error at that command. Allow
|
140
|
-
only the minimum expected set of exit codes unless coverage is provided by
|
141
|
-
subsequent assertions.
|
96
|
+
If an exit_code outside of 0 is expected, one must use acceptable_exit_codes so the test will fail on the proper assertion and not error at that command. Allow only the minimum expected set of exit codes unless coverage is provided by subsequent assertions.
|
142
97
|
|
143
98
|
## Test Outcomes
|
144
99
|
|
@@ -146,10 +101,7 @@ When to use each, and how to format the message:
|
|
146
101
|
|
147
102
|
### Expecting Failure
|
148
103
|
|
149
|
-
If your tests are failing due to an "expected failure", you should wrap your
|
150
|
-
failing assertion in an
|
151
|
-
[`expect_failure`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#expect_failure-instance_method)
|
152
|
-
block with an explanatory logging message:
|
104
|
+
If your tests are failing due to an "expected failure", you should wrap your failing assertion in an [`expect_failure`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Structure#expect_failure-instance_method) block with an explanatory logging message:
|
153
105
|
|
154
106
|
```ruby
|
155
107
|
expect_failure('expected to fail due to PE-1234') do
|
@@ -157,9 +109,7 @@ expect_failure('expected to fail due to PE-1234') do
|
|
157
109
|
end
|
158
110
|
```
|
159
111
|
|
160
|
-
Note that `expect_failure` will only trigger from failed assertions. It won't
|
161
|
-
take care of failed host or `on` commands. To deal with expected failure from an
|
162
|
-
`on` invocation, you'd want something more like this:
|
112
|
+
Note that `expect_failure` will only trigger from failed assertions. It won't take care of failed host or `on` commands. To deal with expected failure from an `on` invocation, you'd want something more like this:
|
163
113
|
|
164
114
|
```ruby
|
165
115
|
on(blah, 'blah', :allow_all_exit_codes => true) do |result|
|
@@ -171,26 +121,15 @@ end
|
|
171
121
|
|
172
122
|
### `fail_test`, `pass_test`, & `skip_test`
|
173
123
|
|
174
|
-
These can be used anywhere in a test to exit early. A
|
175
|
-
[`skip_test`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Outcomes#fail_test-instance_method)
|
176
|
-
between two assertions, for instance, will run the first assertion, raise an
|
177
|
-
exception for `skip_test`, run teardown, and then exit as we expect. The same is
|
178
|
-
true for
|
179
|
-
[fail_test](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Outcomes#fail_test-instance_method).
|
124
|
+
These can be used anywhere in a test to exit early. A [`skip_test`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Outcomes#fail_test-instance_method) between two assertions, for instance, will run the first assertion, raise an exception for `skip_test`, run teardown, and then exit as we expect. The same is true for [fail_test](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Outcomes#fail_test-instance_method).
|
180
125
|
|
181
|
-
[`pass_test`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Outcomes#fail_test-instance_method)
|
182
|
-
is typically not required. When the end of a test is reached without causing an
|
183
|
-
error (due to bad test code execution, or an unhandled exit code or exception)
|
184
|
-
or failure (due to assertions), then it passes.
|
126
|
+
[`pass_test`](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Outcomes#fail_test-instance_method) is typically not required. When the end of a test is reached without causing an error (due to bad test code execution, or an unhandled exit code or exception) or failure (due to assertions), then it passes.
|
185
127
|
|
186
|
-
`pass_test` can be used in a situation where one knows a test has passed before
|
187
|
-
the end of a test under certain circumstances, such as during a loop that has
|
188
|
-
not yet completed.
|
128
|
+
`pass_test` can be used in a situation where one knows a test has passed before the end of a test under certain circumstances, such as during a loop that has not yet completed.
|
189
129
|
|
190
130
|
### Skipping Tests
|
191
131
|
|
192
|
-
Skipping tests can be used, for instance, when they are temporarily failing or
|
193
|
-
not yet complete.
|
132
|
+
Skipping tests can be used, for instance, when they are temporarily failing or not yet complete.
|
194
133
|
|
195
134
|
**Good:**
|
196
135
|
```ruby
|
@@ -203,14 +142,8 @@ confine :to, :platform => 'solaris:pending'
|
|
203
142
|
|
204
143
|
## Confining
|
205
144
|
|
206
|
-
Another way that you can skip or manipulate tests is by confining them to apply
|
207
|
-
to a subset of the SUTs available for testing. Confining is a complex topic, and
|
208
|
-
one we don't have the length to get into in the style guide. For an explanation
|
209
|
-
of confining, as well as the best practices in using it, check out our
|
210
|
-
[confine doc](../how_to/confine.md).
|
145
|
+
Another way that you can skip or manipulate tests is by confining them to apply to a subset of the SUTs available for testing. Confining is a complex topic, and one we don't have the length to get into in the style guide. For an explanation of confining, as well as the best practices in using it, check out our [confine doc](../how_to/confine.md).
|
211
146
|
|
212
147
|
## Assertions
|
213
148
|
|
214
|
-
Always include a _unique_ error message in your assertion statement. Use strict
|
215
|
-
asserts whenever possible (e.g. assert_equal, assert_match.
|
216
|
-
[More info](http://danwin.com/2013/03/ruby-minitest-cheat-sheet/)).
|
149
|
+
Always include a _unique_ error message in your assertion statement. Use strict asserts whenever possible (e.g. assert_equal, assert_match. [More info](http://danwin.com/2013/03/ruby-minitest-cheat-sheet/)).
|
@@ -1,19 +1,13 @@
|
|
1
1
|
## What is This?
|
2
2
|
|
3
|
-
Beaker test tagging allows you to add tags to tests (using the
|
4
|
-
[`tag` DSL method](http://www.rubydoc.info/github/puppetlabs/beaker/master/Beaker/DSL/TestTagging#tag-instance_method)),
|
5
|
-
so that you can include or exclude a specific subset of the tests given for use
|
6
|
-
in this run. Why would you want to use this? Here are some examples of what
|
7
|
-
you can do with this functionality:
|
3
|
+
Beaker test tagging allows you to add tags to tests (using the [`tag` DSL method](http://www.rubydoc.info/github/puppetlabs/beaker/master/Beaker/DSL/TestTagging#tag-instance_method)), so that you can include or exclude a specific subset of the tests given for use in this run. Why would you want to use this? Here are some examples of what you can do with this functionality:
|
8
4
|
|
9
5
|
- Run groups of tests separately from the same testing codebase
|
10
6
|
- Declare different actions that should be taken when a test fails
|
11
7
|
- Make new tests go through a provisional process before being considered solid tests
|
12
8
|
|
13
9
|
## How Tagging Works
|
14
|
-
|
15
|
-
Add tags to a Beaker test at the beginning, like you would if you were using
|
16
|
-
confine. Things to stay aware of:
|
10
|
+
Add tags to a Beaker test at the beginning, like you would if you were using confine. Things to stay aware of:
|
17
11
|
|
18
12
|
- A test that is not executed due to a tag will be considered a ‘skipped’ test
|
19
13
|
- Tags are free form strings and will not be subjected to any correctness testing
|
@@ -53,8 +47,7 @@ Preferred style no-block example:
|
|
53
47
|
|
54
48
|
Beaker will raise an error if `--test-tag-and` & `--test-tag-exclude` contain the same tag, however.
|
55
49
|
|
56
|
-
Beaker will also raise an error if you use both `--test-tag-or` & `--test-tag-and`, because
|
57
|
-
it won't be able to determine which order they should be used in.
|
50
|
+
Beaker will also raise an error if you use both `--test-tag-or` & `--test-tag-and`, because it won't be able to determine which order they should be used in.
|
58
51
|
|
59
52
|
## CLI Examples
|
60
53
|
|
@@ -6,8 +6,7 @@ While Beaker provides the testing harness for much of the acceptance testing tha
|
|
6
6
|
|
7
7
|
### Product Coverage
|
8
8
|
|
9
|
-
Beaker test coverage covers the LTS PE version, currently 2016.4.0, and the latest released version of PE, currently 2016.5.0.
|
10
|
-
Since there is only a single major version of Puppet itself currently supported, beaker only run tests on the latest y-release of Puppet 4, currently 4.8.z. This currently resolves to puppet-agent 1.8.x.
|
9
|
+
Beaker test coverage covers the LTS PE version, currently 2016.4.0, and the latest released version of PE, currently 2016.5.0. Since there is only a single major version of Puppet itself currently supported, beaker only run tests on the latest y-release of Puppet 4, currently 4.8.z. This currently resolves to puppet-agent 1.8.x.
|
11
10
|
|
12
11
|
### Platform Coverage
|
13
12
|
|
@@ -16,9 +15,11 @@ The platforms that beaker covers in its regression testing are largely what is s
|
|
16
15
|
## Test Suite Phases
|
17
16
|
|
18
17
|
### Beaker Spec
|
18
|
+
|
19
19
|
The initial step in Beaker's pipeline is to execute spec testing with supported and future rubies; 2.2.5 and 2.3.1.
|
20
20
|
|
21
21
|
### Beaker Acceptance
|
22
|
+
|
22
23
|
All acceptance tests use actual OS's with beaker installed and use beaker itself to verify that its own methods and classes are working.
|
23
24
|
|
24
25
|
* The Base tests are tests that do not require puppet be installed on the SUT. This includes much of the DSL and host helpers.
|
@@ -1,75 +1,49 @@
|
|
1
|
-
|
2
|
-
questions about the workflow of a Beaker ticket, or the stage that it's in, you
|
3
|
-
should be able to answer them here. If not, then please let QE know, and we'll
|
4
|
-
work to answer your question and update this doc, so that we can do this better
|
5
|
-
going forward.
|
1
|
+
# Ticket Process
|
6
2
|
|
7
|
-
|
8
|
-
|
9
|
-
can cycle a number of times before becoming resolved, and will sometimes skip
|
10
|
-
stages as well.
|
3
|
+
This doc is here to explain the lifecycle of a Beaker ticket. If you have any questions about the workflow of a Beaker ticket, or the stage that it's in, you should be able to answer them here. If not, then please let QE know, and we'll work to answer your question and update this doc, so that we can do this better going forward.
|
4
|
+
|
5
|
+
Note that a typical Beaker ticket goes through these states. They can sometimes go in order, but on average, they loop around through the various stages, can cycle a number of times before becoming resolved, and will sometimes skip stages as well.
|
11
6
|
|
12
7
|
## Administrivia
|
13
8
|
|
14
|
-
- Beaker tickets live in the
|
15
|
-
[BKR project](https://tickets.puppetlabs.com/issues/?jql=project%20%3D%20BKR)
|
9
|
+
- Beaker tickets live in the [BKR project](https://tickets.puppetlabs.com/issues/?jql=project%20%3D%20BKR)
|
16
10
|
- Create a [Jira account](http://tickets.puppetlabs.com) to interact with them
|
17
11
|
|
18
12
|
## Pre-Development States
|
19
13
|
|
20
|
-
These are the states that a Beaker ticket goes through before it is picked up for
|
21
|
-
development. Note that this is only for work done by the Beaker team. If there
|
22
|
-
is something you'd like to contribute to, we would appreciate that you go
|
23
|
-
through the development states for tracking purposes, but you can skip directly
|
24
|
-
to the "In Progress" state if you're going to work on a ticket.
|
14
|
+
These are the states that a Beaker ticket goes through before it is picked up for development. Note that this is only for work done by the Beaker team. If there is something you'd like to contribute to, we would appreciate that you go through the development states for tracking purposes, but you can skip directly to the "In Progress" state if you're going to work on a ticket.
|
25
15
|
|
26
16
|
### Open
|
27
17
|
|
28
|
-
The state for newly filed issues. These have not been triaged, and are
|
29
|
-
waiting to be looked at by the team.
|
18
|
+
The state for newly filed issues. These have not been triaged, and are waiting to be looked at by the team.
|
30
19
|
|
31
20
|
### Needs Information
|
32
21
|
|
33
|
-
The state for blocked issues. These can be blocked by another issue (please link
|
34
|
-
that issue from the ticket if that's the case), or they can be blocked on info
|
35
|
-
needed from another party. If this is the case, please assign the ticket to the
|
36
|
-
person you need info from to move the ticket forward.
|
22
|
+
The state for blocked issues. These can be blocked by another issue (please link that issue from the ticket if that's the case), or they can be blocked on info needed from another party. If this is the case, please assign the ticket to the person you need info from to move the ticket forward.
|
37
23
|
|
38
24
|
### Accepted
|
39
25
|
|
40
|
-
This is the state for when a ticket has been triaged, but hasn't yet been
|
41
|
-
estimated for work from someone on the Beaker team, or someone ready to
|
42
|
-
take on the work. Any preliminary information needed to understand the
|
43
|
-
ticket (before investigation) should be gathered before this point.
|
26
|
+
This is the state for when a ticket has been triaged, but hasn't yet been estimated for work from someone on the Beaker team, or someone ready to take on the work. Any preliminary information needed to understand the ticket (before investigation) should be gathered before this point.
|
44
27
|
|
45
28
|
### Ready for Engineering
|
46
29
|
|
47
|
-
An accepted ticket that has been estimated should be put into the Ready for
|
48
|
-
Engineering state. This does not necessarily mean that a ticket has been
|
49
|
-
prioritized against other Beaker work, but prioritization should occur by
|
50
|
-
the time the issue has been picked up in a sprint.
|
30
|
+
An accepted ticket that has been estimated should be put into the Ready for Engineering state. This does not necessarily mean that a ticket has been prioritized against other Beaker work, but prioritization should occur by the time the issue has been picked up in a sprint.
|
51
31
|
|
52
32
|
## Development States
|
53
33
|
|
54
|
-
Once you're ready to pick up a ticket & work on it, then it should go through
|
55
|
-
these states.
|
34
|
+
Once you're ready to pick up a ticket & work on it, then it should go through these states.
|
56
35
|
|
57
36
|
### In Progress
|
58
37
|
|
59
|
-
This state is set aside for a Beaker contributor to let others know that
|
60
|
-
they're currently working on a particular issue.
|
38
|
+
This state is set aside for a Beaker contributor to let others know that they're currently working on a particular issue.
|
61
39
|
|
62
40
|
### Ready for Merge
|
63
41
|
|
64
|
-
Once a Pull Request (PR) is generated for an issue, the contributor should set
|
65
|
-
|
66
|
-
|
67
|
-
[contributor docs](/CONTRIBUTING.md)),
|
68
|
-
it will be linked from the JIRA ticket.
|
42
|
+
Once a Pull Request (PR) is generated for an issue, the contributor should set the status to Ready for Merge. We do have a GitHub integration setup, so if you've titled your PR correctly (according to our [contributor docs](/CONTRIBUTING.md)), it will be linked from the JIRA ticket.
|
43
|
+
|
44
|
+
In the Beaker project, we leave the assignee as the person who wrote the proposed change, so that they know that they have to keep pushing for their code to be merged.
|
69
45
|
|
70
|
-
|
71
|
-
proposed change, so that they know that they have to keep pushing for their
|
72
|
-
code to be merged.
|
46
|
+
A team member will review the PR. If changes are required, they will be communicated in the PR discussion on GitHub.
|
73
47
|
|
74
48
|
### Resolved
|
75
49
|
|
@@ -79,6 +53,4 @@ Once your PR is merged, then you can Resolve your ticket.
|
|
79
53
|
|
80
54
|
BKR.next
|
81
55
|
|
82
|
-
The reason that we do this now and not before is that we use this field to
|
83
|
-
autogenerate our release notes. We want to make sure that we capture only
|
84
|
-
work that is _in_ the next release, not work that's _intended_ for it.
|
56
|
+
The reason that we do this now and not before is that we use this field to autogenerate our release notes. We want to make sure that we capture only work that is _in_ the next release, not work that's _intended_ for it.
|
@@ -1,49 +1,23 @@
|
|
1
1
|
# How to Archive Files from the Systems Under Test (SUTs)
|
2
2
|
|
3
|
-
Oftentimes when you're dealing with beaker test development or troubleshooting
|
4
|
-
a failed acceptance test, you'll need to get information from a SUT. The
|
5
|
-
traditional way that we've advocated getting information from these machines is
|
6
|
-
to use our [preserved hosts functionality](preserve_hosts.md).
|
3
|
+
Oftentimes when you're dealing with beaker test development or troubleshooting a failed acceptance test, you'll need to get information from a SUT. The traditional way that we've advocated getting information from these machines is to use our [preserved hosts functionality](preserve_hosts.md).
|
7
4
|
|
8
|
-
If you're preserving hosts just to SSH in and look at log files, however, this
|
9
|
-
can be a tedious exercise. Why not just bring the log files to you on the
|
10
|
-
beaker coordinator? This doc explains exactly how to do that using our
|
11
|
-
`archive_file_from` Domain-Specific Language (DSL) method
|
12
|
-
([method rubydocs](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Helpers/HostHelpers#archive_file_from-instance_method)).
|
5
|
+
If you're preserving hosts just to SSH in and look at log files, however, this can be a tedious exercise. Why not just bring the log files to you on the beaker coordinator? This doc explains exactly how to do that using our `archive_file_from` Domain-Specific Language (DSL) method ([method rubydocs](http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Helpers/HostHelpers#archive_file_from-instance_method)).
|
13
6
|
|
14
7
|
# How Do I Use This?
|
15
8
|
|
16
|
-
`archive_file_from` is a part of the beaker DSL, so it's available in all test
|
17
|
-
suites. Just call it from your tests, and it'll execute, pulling any particular
|
18
|
-
file you need off your SUTs, and dropping it on your beaker coordinator's
|
19
|
-
file system.
|
9
|
+
`archive_file_from` is a part of the beaker DSL, so it's available in all test suites. Just call it from your tests, and it'll execute, pulling any particular file you need off your SUTs, and dropping it on your beaker coordinator's file system.
|
20
10
|
|
21
|
-
A common example of a post-suite step to archive files that were created during
|
22
|
-
a particular test is included in the Rubydocs, referenced above. Path details,
|
23
|
-
and details of all method arguments are documented there as well. Check it out,
|
24
|
-
and with the right use, you won't need to preserve hosts at all to debug any
|
25
|
-
test failures.
|
11
|
+
A common example of a post-suite step to archive files that were created during a particular test is included in the Rubydocs, referenced above. Path details, and details of all method arguments are documented there as well. Check it out, and with the right use, you won't need to preserve hosts at all to debug any test failures.
|
26
12
|
|
27
13
|
# Challenges
|
28
14
|
|
29
15
|
## Conditionally Saving Files From SUTs
|
30
16
|
|
31
|
-
One thing that people tend to want from this functionality is to only archive
|
32
|
-
files from SUTs when a beaker run has failed. At this point, beaker does not
|
33
|
-
have access to other suites from a current one. This means that in practice,
|
34
|
-
a post-suite (where one would typically put archiving and other post-processes)
|
35
|
-
will not be able to archive files ONLY IF the test suite has had any failures
|
36
|
-
or errors.
|
17
|
+
One thing that people tend to want from this functionality is to only archive files from SUTs when a beaker run has failed. At this point, beaker does not have access to other suites from a current one. This means that in practice, a post-suite (where one would typically put archiving and other post-processes) will not be able to archive files ONLY IF the test suite has had any failures or errors.
|
37
18
|
|
38
|
-
Our suggestion to get the functionality required would be to have beaker always
|
39
|
-
archive the appropriate files in the post-suite of your tests, but then only
|
40
|
-
have Jenkins (or your job running system, whatever that may be) conditionally
|
41
|
-
take them from the beaker coordinator to whatever external archive system you
|
42
|
-
rely on for later analysis. This can both get you the files that you need from
|
43
|
-
the SUTs and save on space, as only files that need analysis will be kept.
|
19
|
+
Our suggestion to get the functionality required would be to have beaker always archive the appropriate files in the post-suite of your tests, but then only have Jenkins (or your job running system, whatever that may be) conditionally take them from the beaker coordinator to whatever external archive system you rely on for later analysis. This can both get you the files that you need from the SUTs and save on space, as only files that need analysis will be kept.
|
44
20
|
|
45
21
|
# When Did This Come Out?
|
46
22
|
|
47
|
-
`archive_file_from` was originally added to the DSL in beaker
|
48
|
-
[2.48.0](https://github.com/puppetlabs/beaker/releases/tag/2.48.0), released on
|
49
|
-
[July 27, 2016](https://github.com/puppetlabs/beaker/blob/master/HISTORY.md#2480---27-jul-2016-47d3aa18).
|
23
|
+
`archive_file_from` was originally added to the DSL in beaker [2.48.0](https://github.com/puppetlabs/beaker/releases/tag/2.48.0), released on [July 27, 2016](https://github.com/puppetlabs/beaker/blob/master/HISTORY.md#2480---27-jul-2016-47d3aa18).
|