gitlab-qa 7.12.0 → 7.13.3
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.gitlab/ci/rules.gitlab-ci.yml +100 -0
- data/.gitlab-ci.yml +188 -138
- data/docs/running_against_remote_grid.md +39 -10
- data/docs/what_tests_can_be_run.md +60 -22
- data/lib/gitlab/qa/component/preprod.rb +0 -4
- data/lib/gitlab/qa/component/production.rb +0 -4
- data/lib/gitlab/qa/component/release.rb +0 -4
- data/lib/gitlab/qa/component/staging_ref.rb +9 -0
- data/lib/gitlab/qa/report/report_as_issue.rb +1 -1
- data/lib/gitlab/qa/runtime/env.rb +2 -0
- data/lib/gitlab/qa/runtime/omnibus_configurations/license_mode.rb +1 -1
- data/lib/gitlab/qa/scenario/test/instance/staging_ref.rb +18 -0
- data/lib/gitlab/qa/version.rb +1 -1
- data/lib/gitlab/qa.rb +2 -0
- metadata +5 -2
@@ -8,20 +8,49 @@ I.e, if you have a Selenium server set up at http://localhost:4444 or if you hav
|
|
8
8
|
|
9
9
|
| Variable | Description | Default | Example(s) |
|
10
10
|
|---------------------------|----------------------------------------------------------------|----------|--------------------------------|
|
11
|
-
| QA_BROWSER | Browser to run against | "chrome" | "chrome" "firefox"
|
11
|
+
| QA_BROWSER | Browser to run against | "chrome" | "chrome" "firefox" "safari" |
|
12
12
|
| QA_REMOTE_GRID_PROTOCOL | Protocol to use | "http" | "http" "https" |
|
13
13
|
| QA_REMOTE_GRID | Remote grid to run tests against | | "localhost:3000" "provider:80" |
|
14
|
-
| QA_REMOTE_GRID_USERNAME | Username to specify in the remote grid. "USERNAME@provider:80" | |
|
14
|
+
| QA_REMOTE_GRID_USERNAME | Username to specify in the remote grid. "USERNAME@provider:80" | | "gitlab-sl" |
|
15
15
|
| QA_REMOTE_GRID_ACCESS_KEY | Key/Token paired with `QA_REMOTE_GRID_USERNAME` | | |
|
16
|
+
| QA_REMOTE_TUNNEL_ID | Name of the remote tunnel to use | "gitlab-sl_tunnel_id" | |
|
17
|
+
| QA_REMOTE_MOBILE_DEVICE_NAME | Name of mobile device to test against. `QA_BROWSER` must be set to `safari` for iOS devices and `chrome` for Android devices. | | "iPhone 12 Simulator" |
|
16
18
|
|
17
|
-
##
|
19
|
+
## Testing with Sauce Labs
|
18
20
|
|
19
|
-
|
21
|
+
Running directly against an environment like staging is not recommended because test logs expose credentials. Therefore, it is best practice and the default to use a tunnel.
|
20
22
|
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
23
|
+
To install a tunnel, follow these [instructions](https://docs.saucelabs.com/secure-connections/sauce-connect/installation).
|
24
|
+
|
25
|
+
To start the tunnel, copy the run command in **Sauce Labs > Tunnels** and run it in the terminal. You must be logged in to Sauce Labs. Use the credentials in 1Password to log in.
|
26
|
+
|
27
|
+
It is highly recommended to use `GITLAB_QA_ACCESS_TOKEN` to speed up tests and reduce flakiness.
|
28
|
+
|
29
|
+
|
30
|
+
### Run a test in a desktop browser
|
31
|
+
|
32
|
+
While tunnel is running, to test against a local instance in a desktop browser, run:
|
33
|
+
```shell
|
34
|
+
$ QA_BROWSER="safari" \
|
35
|
+
QA_REMOTE_GRID="ondemand.saucelabs.com:80" \
|
36
|
+
QA_REMOTE_GRID_USERNAME="gitlab-sl" \
|
37
|
+
QA_REMOTE_GRID_ACCESS_KEY="<access key found in Sauce Lab account>" \
|
38
|
+
GITLAB_QA_ACCESS_TOKEN="<token>" \
|
39
|
+
gitlab-qa Test::Instance::Any <CE|EE> http://<local_ip>:3000 -- -- <relative_spec_path>
|
27
40
|
```
|
41
|
+
|
42
|
+
### Run a test in a mobile device browser
|
43
|
+
|
44
|
+
`QA_REMOTE_MOBILE_DEVICE_NAME` can be any device name in the [supported browser devices](https://saucelabs.com/platform/supported-browsers-devices) in the Emulators/simulators list, and the latest versions of Android or iOS. You must set `QA_BROWSER` to `safari` for iOS devices and `chrome` for Android devices.```
|
45
|
+
|
46
|
+
```shell
|
47
|
+
$ QA_BROWSER="safari" \
|
48
|
+
QA_REMOTE_MOBILE_DEVICE_NAME="iPhone 12 Simulator" \
|
49
|
+
QA_REMOTE_GRID="ondemand.saucelabs.com:80" \
|
50
|
+
QA_REMOTE_GRID_USERNAME="gitlab-sl" \
|
51
|
+
QA_REMOTE_GRID_ACCESS_KEY="<found in Sauce Lab account>" \
|
52
|
+
GITLAB_QA_ACCESS_TOKEN="<token>" \
|
53
|
+
gitlab-qa Test::Instance::Any <CE|EE> http://<local_ip>:3000 -- -- <relative_spec_path>
|
54
|
+
```
|
55
|
+
|
56
|
+
Results can be watched in real time in Sauce Labs under AUTOMATED > Test Results
|
@@ -1,5 +1,7 @@
|
|
1
1
|
# What tests can be run?
|
2
|
+
|
2
3
|
[[_TOC_]]
|
4
|
+
|
3
5
|
## The two types of QA tests
|
4
6
|
|
5
7
|
First of all, the first thing to choose is whether you want to run orchestrated
|
@@ -245,8 +247,8 @@ instance-level tests against it.
|
|
245
247
|
To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
|
246
248
|
container is spun up and tests are run from it by running the `Test::Instance`
|
247
249
|
scenario (located under
|
248
|
-
[`gitlab-org/gitlab
|
249
|
-
GitLab
|
250
|
+
[`gitlab-org/gitlab/qa/qa/scenario/test/instance.rb`][test-instance] in the
|
251
|
+
GitLab project).
|
250
252
|
|
251
253
|
Example:
|
252
254
|
|
@@ -370,8 +372,8 @@ that LDAP-related features work as expected.
|
|
370
372
|
To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
|
371
373
|
container is spun up and tests are run from it by running the
|
372
374
|
`Test::Integration::LDAPNoTLS` scenario (located under
|
373
|
-
[`gitlab-org/gitlab
|
374
|
-
in the GitLab
|
375
|
+
[`gitlab-org/gitlab/qa/qa/scenario/test/integration/ldap_no_tls.rb`][test-integration-ldap-no-tls]
|
376
|
+
in the GitLab project).
|
375
377
|
|
376
378
|
In EE, both the GitLab standard and LDAP credentials are needed:
|
377
379
|
|
@@ -393,7 +395,7 @@ $ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)
|
|
393
395
|
$ gitlab-qa Test::Integration::LDAPNoTLS EE
|
394
396
|
```
|
395
397
|
|
396
|
-
[test-integration-ldap-no-tls]: https://gitlab.com/gitlab-org/gitlab
|
398
|
+
[test-integration-ldap-no-tls]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/integration/ldap_no_tls.rb
|
397
399
|
|
398
400
|
### `Test::Integration::LDAPTLS CE|EE|<full image address>`
|
399
401
|
|
@@ -411,8 +413,8 @@ that LDAP-related features work as expected.
|
|
411
413
|
To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
|
412
414
|
container is spun up and tests are run from it by running the
|
413
415
|
`Test::Integration::LDAPTLS` scenario (located under
|
414
|
-
[`gitlab-org/gitlab
|
415
|
-
in the GitLab
|
416
|
+
[`gitlab-org/gitlab/qa/qa/scenario/test/integration/ldap_tls.rb`][test-integration-ldap-tls]
|
417
|
+
in the GitLab project).
|
416
418
|
|
417
419
|
In EE, both the GitLab standard and LDAP credentials are needed:
|
418
420
|
|
@@ -434,8 +436,8 @@ $ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)
|
|
434
436
|
$ gitlab-qa Test::Integration::LDAPTLS EE
|
435
437
|
```
|
436
438
|
|
437
|
-
[test-integration-ldap-tls]: https://gitlab.com/gitlab-org/gitlab
|
438
|
-
[test-integration-ldap-tls-certs]: https://gitlab.com/gitlab-org/gitlab
|
439
|
+
[test-integration-ldap-tls]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/integration/ldap_tls.rb
|
440
|
+
[test-integration-ldap-tls-certs]: https://gitlab.com/gitlab-org/gitlab/blob/master/tls_certificates/gitlab
|
439
441
|
|
440
442
|
### `Test::Integration::LDAPNoServer EE|<full image address>`
|
441
443
|
|
@@ -504,9 +506,9 @@ using SAML.
|
|
504
506
|
|
505
507
|
To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
|
506
508
|
container is spun up and tests are run from it by running the
|
507
|
-
`Test::Integration::InstanceSAML` scenario (located under [`gitlab-org/gitlab-ce@qa/qa/scenario/test/integration/instance_saml.rb`][test-integration-instance-saml] in the GitLab
|
509
|
+
`Test::Integration::InstanceSAML` scenario (located under [`gitlab-org/gitlab-ce@qa/qa/scenario/test/integration/instance_saml.rb`][test-integration-instance-saml] in the GitLab project).
|
508
510
|
|
509
|
-
[test-integration-instance-saml]: https://gitlab.com/gitlab-org/gitlab
|
511
|
+
[test-integration-instance-saml]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/integration/instance_saml.rb
|
510
512
|
|
511
513
|
**Required environment variables:**
|
512
514
|
|
@@ -531,8 +533,8 @@ Mattermost server (see `Test::Instance::Image` above).
|
|
531
533
|
To run tests against the GitLab container, a GitLab QA (`gitlab/gitlab-qa`)
|
532
534
|
container is spun up and tests are run from it by running the
|
533
535
|
`Test::Integration::Mattermost` scenario (located under
|
534
|
-
[`gitlab-org/gitlab
|
535
|
-
in the GitLab
|
536
|
+
[`gitlab-org/gitlab/qa/qa/scenario/test/integration/mattermost.rb`][test-integration-mattermost]
|
537
|
+
in the GitLab project).
|
536
538
|
|
537
539
|
**Required environment variables:**
|
538
540
|
|
@@ -549,7 +551,7 @@ $ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)
|
|
549
551
|
$ gitlab-qa Test::Integration::Mattermost EE
|
550
552
|
```
|
551
553
|
|
552
|
-
[test-integration-mattermost]: https://gitlab.com/gitlab-org/gitlab
|
554
|
+
[test-integration-mattermost]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/integration/mattermost.rb
|
553
555
|
|
554
556
|
### `Test::Integration::Packages CE|EE|<full image address>`
|
555
557
|
|
@@ -668,8 +670,8 @@ against it.
|
|
668
670
|
To run tests against the GitLab instance, a GitLab QA (`gitlab/gitlab-qa`)
|
669
671
|
container is spun up and tests are run from it by running the
|
670
672
|
`Test::Instance` scenario (located under
|
671
|
-
[`gitlab-org/gitlab
|
672
|
-
in the GitLab
|
673
|
+
[`gitlab-org/gitlab/qa/qa/scenario/test/instance.rb`][test-instance] in the
|
674
|
+
in the GitLab project).
|
673
675
|
|
674
676
|
Example:
|
675
677
|
|
@@ -695,8 +697,8 @@ works as expected by running tests against it.
|
|
695
697
|
To run tests against the GitLab instance, a GitLab QA (`gitlab/gitlab-qa`)
|
696
698
|
container is spun up and tests are run from it by running the
|
697
699
|
`Test::Instance` scenario (located under
|
698
|
-
[`gitlab-org/gitlab
|
699
|
-
in the GitLab
|
700
|
+
[`gitlab-org/gitlab/qa/qa/scenario/test/instance.rb`][test-instance] in the
|
701
|
+
in the GitLab project).
|
700
702
|
|
701
703
|
**Required environment variables:**
|
702
704
|
|
@@ -723,6 +725,42 @@ $ export GITLAB_PASSWORD="$GITLAB_QA_PASSWORD"
|
|
723
725
|
$ gitlab-qa Test::Instance::Staging
|
724
726
|
```
|
725
727
|
|
728
|
+
### `Test::Instance::StagingRef`
|
729
|
+
|
730
|
+
This scenario tests that the [`Staging Ref`](https://staging-ref.gitlab.com)
|
731
|
+
works as expected by running tests against it.
|
732
|
+
|
733
|
+
To run tests against the GitLab instance, a GitLab QA (`gitlab/gitlab-qa`)
|
734
|
+
container is spun up and tests are run from it by running the
|
735
|
+
`Test::Instance` scenario (located under
|
736
|
+
[`gitlab-org/gitlab/qa/qa/scenario/test/instance.rb`][test-instance] in the
|
737
|
+
in the GitLab project).
|
738
|
+
|
739
|
+
**Required environment variables:**
|
740
|
+
|
741
|
+
- `GITLAB_QA_ACCESS_TOKEN`: A valid personal access token with the `api` scope.
|
742
|
+
This is used to retrieve the version that staging is currently running.
|
743
|
+
Staging Ref QA users credentials can be found in the shared GitLab QA 1Password vault.
|
744
|
+
|
745
|
+
**Optional environment variables:**
|
746
|
+
|
747
|
+
- `GITLAB_QA_DEV_ACCESS_TOKEN`: A valid personal access token for the
|
748
|
+
`gitlab-qa-bot` on `dev.gitlab.org` with the `registry` scope.
|
749
|
+
This is used to pull the QA Docker image from the Omnibus GitLab `dev` Container Registry.
|
750
|
+
If the variable isn't present, the QA image from Docker Hub will be used.
|
751
|
+
This can be found in the shared GitLab QA 1Password vault.
|
752
|
+
|
753
|
+
Example:
|
754
|
+
|
755
|
+
```
|
756
|
+
$ export GITLAB_QA_ACCESS_TOKEN=your_api_access_token
|
757
|
+
$ export GITLAB_QA_DEV_ACCESS_TOKEN=your_dev_registry_access_token
|
758
|
+
$ export GITLAB_USERNAME="gitlab-qa"
|
759
|
+
$ export GITLAB_PASSWORD="$GITLAB_QA_PASSWORD"
|
760
|
+
|
761
|
+
$ gitlab-qa Test::Instance::StagingRef
|
762
|
+
```
|
763
|
+
|
726
764
|
### `Test::Instance::StagingGeo`
|
727
765
|
|
728
766
|
This scenario tests that the Geo staging deployment (with [`staging.gitlab.com`](https://staging.gitlab.com) as the primary site and [`geo.staging.gitlab.com`](https://geo.staging.gitlab.com) as the secondary site) works as expected by running tests tagged `:geo` against it. This is done by spinning up a GitLab QA (`gitlab/gitlab-qa`) container and running the `QA::EE::Scenario::Test::Geo` scenario. Note that the Geo setup steps in the `QA::EE::Scenario::Test::Geo` scenario are skipped when testing a live Geo deployment.
|
@@ -789,8 +827,8 @@ Smoke tests are quick tests that ensure that some basic functionality of GitLab
|
|
789
827
|
To run tests against the GitLab instance, a GitLab QA (`gitlab/gitlab-qa`)
|
790
828
|
container is spun up and tests are run from it by running the
|
791
829
|
`Test::Instance::Smoke` scenario (located under
|
792
|
-
[`gitlab-org/gitlab
|
793
|
-
in the GitLab
|
830
|
+
[`gitlab-org/gitlab/qa/qa/scenario/test/smoke.rb`][smoke-instance] in the
|
831
|
+
in the GitLab project).
|
794
832
|
|
795
833
|
Example:
|
796
834
|
|
@@ -863,5 +901,5 @@ $ gitlab-qa Test::Instance::Geo EE https://primary.gitlab.com https://secondary.
|
|
863
901
|
|
864
902
|
[Back to README.md](../README.md)
|
865
903
|
|
866
|
-
[test-instance]: https://gitlab.com/gitlab-org/gitlab
|
867
|
-
[smoke-instance]: https://gitlab.com/gitlab-org/gitlab
|
904
|
+
[test-instance]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/instance/all.rb
|
905
|
+
[smoke-instance]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/instance/smoke.rb
|
@@ -113,7 +113,7 @@ module Gitlab
|
|
113
113
|
"found:#{pipeline}.gitlab.com"
|
114
114
|
when 'preprod'
|
115
115
|
'found:pre.gitlab.com'
|
116
|
-
when 'staging-orchestrated', 'nightly', QA::Runtime::Env.default_branch
|
116
|
+
when 'staging-orchestrated', 'nightly', QA::Runtime::Env.default_branch, 'staging-ref', 'release'
|
117
117
|
"found:#{pipeline}"
|
118
118
|
else
|
119
119
|
raise "No `found:*` label for the `#{pipeline}` pipeline!"
|
@@ -15,6 +15,8 @@ module Gitlab
|
|
15
15
|
'QA_REMOTE_GRID_USERNAME' => :remote_grid_username,
|
16
16
|
'QA_REMOTE_GRID_ACCESS_KEY' => :remote_grid_access_key,
|
17
17
|
'QA_REMOTE_GRID_PROTOCOL' => :remote_grid_protocol,
|
18
|
+
'QA_REMOTE_MOBILE_DEVICE_NAME' => :remote_mobile_device_name,
|
19
|
+
'QA_REMOTE_TUNNEL_ID' => :remote_tunnel_id,
|
18
20
|
'QA_BROWSER' => :browser,
|
19
21
|
'QA_ADDITIONAL_REPOSITORY_STORAGE' => :qa_additional_repository_storage,
|
20
22
|
'QA_PRAEFECT_REPOSITORY_STORAGE' => :qa_praefect_repository_storage,
|
@@ -7,7 +7,7 @@ module Gitlab
|
|
7
7
|
class LicenseMode < Default
|
8
8
|
def configuration
|
9
9
|
<<~OMNIBUS
|
10
|
-
gitlab_rails['env'] = { 'GITLAB_LICENSE_MODE' => 'test', 'CUSTOMER_PORTAL_URL' => 'https://customers.
|
10
|
+
gitlab_rails['env'] = { 'GITLAB_LICENSE_MODE' => 'test', 'CUSTOMER_PORTAL_URL' => 'https://customers.staging.gitlab.com' }
|
11
11
|
OMNIBUS
|
12
12
|
end
|
13
13
|
end
|
@@ -0,0 +1,18 @@
|
|
1
|
+
module Gitlab
|
2
|
+
module QA
|
3
|
+
module Scenario
|
4
|
+
module Test
|
5
|
+
module Instance
|
6
|
+
##
|
7
|
+
# Run test suite against Staging Ref environment
|
8
|
+
#
|
9
|
+
class StagingRef < DeploymentBase
|
10
|
+
def deployment_component
|
11
|
+
Component::StagingRef
|
12
|
+
end
|
13
|
+
end
|
14
|
+
end
|
15
|
+
end
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
data/lib/gitlab/qa/version.rb
CHANGED
data/lib/gitlab/qa.rb
CHANGED
@@ -34,6 +34,7 @@ module Gitlab
|
|
34
34
|
autoload :RelativeUrl, 'gitlab/qa/scenario/test/instance/relative_url'
|
35
35
|
autoload :RepositoryStorage, 'gitlab/qa/scenario/test/instance/repository_storage'
|
36
36
|
autoload :Staging, 'gitlab/qa/scenario/test/instance/staging'
|
37
|
+
autoload :StagingRef, 'gitlab/qa/scenario/test/instance/staging_ref'
|
37
38
|
autoload :Production, 'gitlab/qa/scenario/test/instance/production'
|
38
39
|
autoload :Smoke, 'gitlab/qa/scenario/test/instance/smoke'
|
39
40
|
autoload :Preprod, 'gitlab/qa/scenario/test/instance/preprod'
|
@@ -87,6 +88,7 @@ module Gitlab
|
|
87
88
|
autoload :SAML, 'gitlab/qa/component/saml'
|
88
89
|
autoload :Specs, 'gitlab/qa/component/specs'
|
89
90
|
autoload :Staging, 'gitlab/qa/component/staging'
|
91
|
+
autoload :StagingRef, 'gitlab/qa/component/staging_ref'
|
90
92
|
autoload :Production, 'gitlab/qa/component/production'
|
91
93
|
autoload :Minio, 'gitlab/qa/component/minio'
|
92
94
|
autoload :Preprod, 'gitlab/qa/component/preprod'
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: gitlab-qa
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 7.
|
4
|
+
version: 7.13.3
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- GitLab Quality
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-11-04 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: climate_control
|
@@ -190,6 +190,7 @@ files:
|
|
190
190
|
- ".dockerignore"
|
191
191
|
- ".gitignore"
|
192
192
|
- ".gitlab-ci.yml"
|
193
|
+
- ".gitlab/ci/rules.gitlab-ci.yml"
|
193
194
|
- ".gitlab/merge_request_templates/Release.md"
|
194
195
|
- ".rspec"
|
195
196
|
- ".rubocop.yml"
|
@@ -240,6 +241,7 @@ files:
|
|
240
241
|
- lib/gitlab/qa/component/saml.rb
|
241
242
|
- lib/gitlab/qa/component/specs.rb
|
242
243
|
- lib/gitlab/qa/component/staging.rb
|
244
|
+
- lib/gitlab/qa/component/staging_ref.rb
|
243
245
|
- lib/gitlab/qa/docker/command.rb
|
244
246
|
- lib/gitlab/qa/docker/engine.rb
|
245
247
|
- lib/gitlab/qa/docker/shellout.rb
|
@@ -287,6 +289,7 @@ files:
|
|
287
289
|
- lib/gitlab/qa/scenario/test/instance/smoke.rb
|
288
290
|
- lib/gitlab/qa/scenario/test/instance/staging.rb
|
289
291
|
- lib/gitlab/qa/scenario/test/instance/staging_geo.rb
|
292
|
+
- lib/gitlab/qa/scenario/test/instance/staging_ref.rb
|
290
293
|
- lib/gitlab/qa/scenario/test/integration/actioncable.rb
|
291
294
|
- lib/gitlab/qa/scenario/test/integration/client_ssl.rb
|
292
295
|
- lib/gitlab/qa/scenario/test/integration/elasticsearch.rb
|