gitlab-qa 2.1.0 → 2.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.gitlab-ci.yml +0 -31
- data/README.md +43 -35
- data/docs/what_tests_can_be_run.md +2 -0
- data/lib/gitlab/qa/runtime/env.rb +2 -0
- data/lib/gitlab/qa/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 27bc2abc8cfebcb0ca9db1d08a42543c07c4a0c324bb247f5017bf021daf126a
|
4
|
+
data.tar.gz: d544281d1b8195ad55432abbc31ead297d64a864e6fd3dbe5caf97cd192e716c
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 2fb60fe3f21ccfd1cfefa59e16ec373ad8dc9b9545ec13eca963f524a36cc7a6f9857734a73aea1f4cd7b224bb2402fd1835e8dde9dac44d6ec7c7763b0d244e
|
7
|
+
data.tar.gz: bccb952d05b6cee766989618f3fa53427db7c069b86e720ceeebd91a098219313650b0534aea1918643749bef65187116d8780155bd4f83e3f7f9edfd44ce4d4
|
data/.gitlab-ci.yml
CHANGED
@@ -3,7 +3,6 @@ services:
|
|
3
3
|
|
4
4
|
stages:
|
5
5
|
- check
|
6
|
-
- sanity
|
7
6
|
- test
|
8
7
|
|
9
8
|
variables:
|
@@ -36,24 +35,6 @@ check:rspec:
|
|
36
35
|
tags:
|
37
36
|
- docker
|
38
37
|
|
39
|
-
.sanity: &sanity
|
40
|
-
stage: sanity
|
41
|
-
image: $TEST_IMAGE
|
42
|
-
only:
|
43
|
-
- schedules
|
44
|
-
tags:
|
45
|
-
- docker
|
46
|
-
|
47
|
-
ce:version:
|
48
|
-
script: bin/qa Test::Sanity::Version CE
|
49
|
-
allow_failure: true
|
50
|
-
<<: *sanity
|
51
|
-
|
52
|
-
ee:version:
|
53
|
-
script: bin/qa Test::Sanity::Version EE
|
54
|
-
allow_failure: true
|
55
|
-
<<: *sanity
|
56
|
-
|
57
38
|
.test: &test
|
58
39
|
stage: test
|
59
40
|
image: $TEST_IMAGE
|
@@ -191,15 +172,3 @@ ee:kubernetes:
|
|
191
172
|
<<: *test
|
192
173
|
<<: *high-capacity
|
193
174
|
<<: *ee-qa
|
194
|
-
|
195
|
-
qa:staging:
|
196
|
-
script:
|
197
|
-
- export GITLAB_USERNAME="gitlab-qa"
|
198
|
-
- export GITLAB_PASSWORD="$GITLAB_QA_PASSWORD"
|
199
|
-
- unset EE_LICENSE
|
200
|
-
- bin/qa Test::Instance::Staging
|
201
|
-
<<: *test
|
202
|
-
retry: 0
|
203
|
-
when: manual
|
204
|
-
except:
|
205
|
-
- triggers
|
data/README.md
CHANGED
@@ -5,17 +5,17 @@
|
|
5
5
|
|
6
6
|
## Definitions
|
7
7
|
|
8
|
-
- **GitLab QA framework**: A framework that allows
|
8
|
+
- **GitLab QA framework**: A framework that allows developers to write end-to-end
|
9
9
|
tests simply and efficiently.
|
10
10
|
Located at [`gitlab-org/gitlab-ce@qa/qa/`][qa-framework].
|
11
|
-
- **GitLab QA instance-level scenarios**: RSpec scenarios that
|
11
|
+
- **GitLab QA instance-level scenarios**: RSpec scenarios that use the
|
12
12
|
GitLab QA framework and Capybara to setup and perform individual end-to-end
|
13
13
|
tests against a live GitLab instance.
|
14
14
|
Located at [`gitlab-org/gitlab-ce@qa/qa/specs/features/`][instance-level-scenarios].
|
15
|
-
- **GitLab QA orchestrator** (this project): An orchestration tool that
|
16
|
-
QA test suites in a simple manner.
|
15
|
+
- **GitLab QA orchestrator** (this project): An orchestration tool that enables
|
16
|
+
running various QA test suites in a simple manner.
|
17
17
|
- **GitLab QA orchestrated scenarios**: Scenarios where containers are started,
|
18
|
-
configured and
|
18
|
+
configured, and execute instance-level scenarios against a running GitLab
|
19
19
|
instance.
|
20
20
|
Located at [`gitlab-org/gitlab-qa@lib/gitlab/qa/scenario/test/`][orchestrated-scenarios].
|
21
21
|
|
@@ -31,6 +31,17 @@ GitLab consists of multiple pieces configured and packaged by
|
|
31
31
|
The purpose of the QA end-to-end test suite is to verify that all pieces
|
32
32
|
integrate well together.
|
33
33
|
|
34
|
+
### Testing changes in merge requests before the merge
|
35
|
+
|
36
|
+
The ultimate goal is to make it possible to run the QA test suite for any
|
37
|
+
merge request, even before merging code into the `master` branch.
|
38
|
+
|
39
|
+
### We can run tests against any instance of GitLab
|
40
|
+
|
41
|
+
GitLab QA is a click-driven, black-box testing tool. We also use it to run
|
42
|
+
tests against the staging environment, and we strive to make it useful for our
|
43
|
+
users as well.
|
44
|
+
|
34
45
|
### GitLab QA tests running in the CI/CD environment
|
35
46
|
|
36
47
|
Manual steps should not be needed to run the QA test suite.
|
@@ -38,46 +49,38 @@ GitLab QA orchestrator is CI/CD environment native, which means that we should
|
|
38
49
|
add new features and tests when we are comfortable with running new code in the
|
39
50
|
CI/CD environment.
|
40
51
|
|
41
|
-
### GitLab QA test
|
52
|
+
### GitLab QA test failures are reproducible locally
|
42
53
|
|
43
54
|
Despite the fact that GitLab QA orchestrator has been built to run in the CI/CD
|
44
55
|
environment, it is really important to make it easy for developers to reproduce
|
45
56
|
test failures locally. It is much easier to debug things locally, than in the
|
46
|
-
CI/CD.
|
57
|
+
CI/CD environment.
|
47
58
|
|
48
59
|
To make it easier to reproduce test failures locally we have published the
|
49
60
|
`gitlab-qa` gem [on rubygems.org](https://rubygems.org/gems/gitlab-qa) and we
|
50
61
|
are using exactly the same approach to run tests in the CI/CD environment.
|
51
62
|
|
52
|
-
It means that using `gitlab-qa` CLI tool,
|
53
|
-
runs GitLab QA test suite, is a reproducible way of running
|
54
|
-
in the CI/CD.
|
63
|
+
It means that using the `gitlab-qa` CLI tool, which orchestrates the test
|
64
|
+
environment and runs the GitLab QA test suite, is a reproducible way of running
|
65
|
+
tests locally and in the CI/CD environment.
|
55
66
|
|
56
|
-
It also means that we cannot have
|
67
|
+
It also means that we cannot have custom code in `.gitlab-ci.yml` to, for
|
57
68
|
example, start new containers / services.
|
58
69
|
|
59
|
-
### Test installation / deployment process too
|
70
|
+
### Test the installation / deployment process too
|
60
71
|
|
61
|
-
We distribute GitLab in a package (like Debian package or a Docker image) so
|
62
|
-
we want to test installation process to ensure that our package is not
|
72
|
+
We distribute GitLab in a package (like a Debian package or a Docker image) so
|
73
|
+
we want to test the installation process to ensure that our package is not
|
74
|
+
broken.
|
63
75
|
|
64
76
|
But we are also working on making GitLab be a cloud native product. This means
|
65
|
-
that using Helm becomes yet another installation / deployment
|
66
|
-
want to test with GitLab QA.
|
77
|
+
that, for example, using Helm becomes yet another installation / deployment
|
78
|
+
process that we want to test with GitLab QA.
|
67
79
|
|
68
|
-
|
69
|
-
important
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
The ultimate goal is to make it possible to run the QA test suite for any
|
74
|
-
merge request, even before merging code into the `master` branch.
|
75
|
-
|
76
|
-
### We can run tests against any instance of GitLab
|
77
|
-
|
78
|
-
GitLab QA is a click-driven, black-box testing tool. We also use it to run
|
79
|
-
tests against the staging, and we strive for making it useful for our users
|
80
|
-
as well.
|
80
|
+
Considering our goal of being able to test all changes in merge requests, it is
|
81
|
+
especially important to be able to test our Kubernetes deployments, as that is
|
82
|
+
essential to scaling our test environments to efficiently handle a large number
|
83
|
+
of tests.
|
81
84
|
|
82
85
|
## Documentation
|
83
86
|
|
@@ -85,15 +88,19 @@ All the documentation can be found under [`docs/`](/docs/README.md).
|
|
85
88
|
|
86
89
|
## How do we use it
|
87
90
|
|
88
|
-
Currently we
|
89
|
-
nightly.
|
91
|
+
Currently, we execute the test suite against GitLab Docker images created by
|
92
|
+
Omnibus nightly via a pipeline in the [nightly](https://gitlab.com/gitlab-org/quality/nightly)
|
93
|
+
project.
|
94
|
+
|
95
|
+
We also execute the test suite nightly against our [staging environment](https://staging.gitlab.com)
|
96
|
+
via a pipeline in the [staging project](https://gitlab.com/gitlab-org/quality/staging).
|
90
97
|
|
91
|
-
|
98
|
+
Finally, we trigger GitLab QA pipelines whenever someone clicks `package-and-qa` manual
|
92
99
|
action in a merge request.
|
93
100
|
|
94
101
|
## How can you use it
|
95
102
|
|
96
|
-
GitLab QA tool is published as a [Ruby Gem](https://rubygems.org/gems/gitlab-qa).
|
103
|
+
The GitLab QA tool is published as a [Ruby Gem](https://rubygems.org/gems/gitlab-qa).
|
97
104
|
|
98
105
|
You can install it with `gem install gitlab-qa`. It will expose a `gitlab-qa`
|
99
106
|
command in your system.
|
@@ -107,10 +114,11 @@ All the scenarios you can run are described in the
|
|
107
114
|
|
108
115
|
### How to add new scenarios
|
109
116
|
|
110
|
-
Scenarios (test cases) and scripts to run them are located in
|
117
|
+
Scenarios (test cases) and scripts to run them are located in the
|
111
118
|
[CE](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa) and
|
112
119
|
[EE](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/qa)
|
113
|
-
repositories under `qa/` directory, so please also check the documentation
|
120
|
+
repositories under the `qa/` directory, so please also check the documentation
|
121
|
+
there.
|
114
122
|
|
115
123
|
## Contributing
|
116
124
|
|
@@ -30,6 +30,8 @@ For more details on the internals, please read the
|
|
30
30
|
|
31
31
|
* `GITLAB_USERNAME` - username to use when signing into GitLab
|
32
32
|
* `GITLAB_PASSWORD` - password to use when signing into GitLab
|
33
|
+
* `GITLAB_FORKER_USERNAME` - username to use for forking a project
|
34
|
+
* `GITLAB_FORKER_PASSWORD` - password to use for forking a project
|
33
35
|
* `GITLAB_USER_TYPE` - type of user to use when signing into GitLab: standard (default), ldap
|
34
36
|
* `GITLAB_LDAP_USERNAME` - LDAP username to use when signing into GitLab
|
35
37
|
* `GITLAB_LDAP_PASSWORD` - LDAP password to use when signing into GitLab
|
@@ -11,6 +11,8 @@ module Gitlab
|
|
11
11
|
'GITLAB_PASSWORD' => :user_password,
|
12
12
|
'GITLAB_LDAP_USERNAME' => :ldap_username,
|
13
13
|
'GITLAB_LDAP_PASSWORD' => :ldap_password,
|
14
|
+
'GITLAB_FORKER_USERNAME' => :forker_username,
|
15
|
+
'GITLAB_FORKER_PASSWORD' => :forker_password,
|
14
16
|
'GITLAB_USER_TYPE' => :user_type,
|
15
17
|
'GITLAB_SANDBOX_NAME' => :gitlab_sandbox_name,
|
16
18
|
'GITLAB_QA_ACCESS_TOKEN' => :qa_access_token,
|
data/lib/gitlab/qa/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: gitlab-qa
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 2.
|
4
|
+
version: 2.2.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Grzegorz Bizon
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2018-
|
11
|
+
date: 2018-08-20 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: climate_control
|