gitlab-qa 7.12.0 → 7.13.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.gitlab/ci/rules.gitlab-ci.yml +100 -0
- data/.gitlab-ci.yml +188 -138
- data/docs/running_against_remote_grid.md +39 -10
- data/docs/what_tests_can_be_run.md +60 -22
- data/lib/gitlab/qa/component/preprod.rb +0 -4
- data/lib/gitlab/qa/component/production.rb +0 -4
- data/lib/gitlab/qa/component/release.rb +0 -4
- data/lib/gitlab/qa/component/staging_ref.rb +9 -0
- data/lib/gitlab/qa/report/report_as_issue.rb +1 -1
- data/lib/gitlab/qa/runtime/env.rb +2 -0
- data/lib/gitlab/qa/runtime/omnibus_configurations/license_mode.rb +1 -1
- data/lib/gitlab/qa/scenario/test/instance/staging_ref.rb +18 -0
- data/lib/gitlab/qa/version.rb +1 -1
- data/lib/gitlab/qa.rb +2 -0
- metadata +5 -2
@@ -8,20 +8,49 @@ I.e, if you have a Selenium server set up at http://localhost:4444 or if you hav
|
|
8
8
|
|
9
9
|
| Variable | Description | Default | Example(s) |
|
10
10
|
|---------------------------|----------------------------------------------------------------|----------|--------------------------------|
|
11
|
-
| QA_BROWSER | Browser to run against | "chrome" | "chrome" "firefox"
|
11
|
+
| QA_BROWSER | Browser to run against | "chrome" | "chrome" "firefox" "safari" |
|
12
12
|
| QA_REMOTE_GRID_PROTOCOL | Protocol to use | "http" | "http" "https" |
|
13
13
|
| QA_REMOTE_GRID | Remote grid to run tests against | | "localhost:3000" "provider:80" |
|
14
|
-
| QA_REMOTE_GRID_USERNAME | Username to specify in the remote grid. "USERNAME@provider:80" | |
|
14
|
+
| QA_REMOTE_GRID_USERNAME | Username to specify in the remote grid. "USERNAME@provider:80" | | "gitlab-sl" |
|
15
15
|
| QA_REMOTE_GRID_ACCESS_KEY | Key/Token paired with `QA_REMOTE_GRID_USERNAME` | | |
|
16
|
+
| QA_REMOTE_TUNNEL_ID | Name of the remote tunnel to use | "gitlab-sl_tunnel_id" | |
|
17
|
+
| QA_REMOTE_MOBILE_DEVICE_NAME | Name of mobile device to test against. `QA_BROWSER` must be set to `safari` for iOS devices and `chrome` for Android devices. | | "iPhone 12 Simulator" |
|
16
18
|
|
17
|
-
##
|
19
|
+
## Testing with Sauce Labs
|
18
20
|
|
19
|
-
|
21
|
+
Running directly against an environment like staging is not recommended because test logs expose credentials. Therefore, it is best practice and the default to use a tunnel.
|
20
22
|
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
23
|
+
To install a tunnel, follow these [instructions](https://docs.saucelabs.com/secure-connections/sauce-connect/installation).
|
24
|
+
|
25
|
+
To start the tunnel, copy the run command in **Sauce Labs > Tunnels** and run it in the terminal. You must be logged in to Sauce Labs. Use the credentials in 1Password to log in.
|
26
|
+
|
27
|
+
It is highly recommended to use `GITLAB_QA_ACCESS_TOKEN` to speed up tests and reduce flakiness.
|
28
|
+
|
29
|
+
|
30
|
+
### Run a test in a desktop browser
|
31
|
+
|
32
|
+
While tunnel is running, to test against a local instance in a desktop browser, run:
|
33
|
+
```shell
|
34
|
+
$ QA_BROWSER="safari" \
|
35
|
+
QA_REMOTE_GRID="ondemand.saucelabs.com:80" \
|
36
|
+
QA_REMOTE_GRID_USERNAME="gitlab-sl" \
|
37
|
+
QA_REMOTE_GRID_ACCESS_KEY="<access key found in Sauce Lab account>" \
|
38
|
+
GITLAB_QA_ACCESS_TOKEN="<token>" \
|
39
|
+
gitlab-qa Test::Instance::Any <CE|EE> http://<local_ip>:3000 -- -- <relative_spec_path>
|
27
40
|
```
|
41
|
+
|
42
|
+
### Run a test in a mobile device browser
|
43
|
+
|
44
|
+
`QA_REMOTE_MOBILE_DEVICE_NAME` can be any device name in the [supported browser devices](https://saucelabs.com/platform/supported-browsers-devices) in the Emulators/simulators list, and the latest versions of Android or iOS. You must set `QA_BROWSER` to `safari` for iOS devices and `chrome` for Android devices.```
|
45
|
+
|
46
|
+
```shell
|
47
|
+
$ QA_BROWSER="safari" \
|
48
|
+
QA_REMOTE_MOBILE_DEVICE_NAME="iPhone 12 Simulator" \
|
49
|
+
QA_REMOTE_GRID="ondemand.saucelabs.com:80" \
|
50
|
+
QA_REMOTE_GRID_USERNAME="gitlab-sl" \
|
51
|
+
QA_REMOTE_GRID_ACCESS_KEY="<found in Sauce Lab account>" \
|
52
|
+
GITLAB_QA_ACCESS_TOKEN="<token>" \
|
53
|
+
gitlab-qa Test::Instance::Any <CE|EE> http://<local_ip>:3000 -- -- <relative_spec_path>
|
54
|
+
```
|
55
|
+
|
56
|
+
Results can be watched in real time in Sauce Labs under AUTOMATED > Test Results
|
@@ -1,5 +1,7 @@
|
|
1
1
|
# What tests can be run?
|
2
|
+
|
2
3
|
[[_TOC_]]
|
4
|
+
|
3
5
|
## The two types of QA tests
|
4
6
|
|
5
7
|
First of all, the first thing to choose is whether you want to run orchestrated
|
@@ -245,8 +247,8 @@ instance-level tests against it.
|
|
245
247
|
To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
|
246
248
|
container is spun up and tests are run from it by running the `Test::Instance`
|
247
249
|
scenario (located under
|
248
|
-
[`gitlab-org/gitlab
|
249
|
-
GitLab
|
250
|
+
[`gitlab-org/gitlab/qa/qa/scenario/test/instance.rb`][test-instance] in the
|
251
|
+
GitLab project).
|
250
252
|
|
251
253
|
Example:
|
252
254
|
|
@@ -370,8 +372,8 @@ that LDAP-related features work as expected.
|
|
370
372
|
To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
|
371
373
|
container is spun up and tests are run from it by running the
|
372
374
|
`Test::Integration::LDAPNoTLS` scenario (located under
|
373
|
-
[`gitlab-org/gitlab
|
374
|
-
in the GitLab
|
375
|
+
[`gitlab-org/gitlab/qa/qa/scenario/test/integration/ldap_no_tls.rb`][test-integration-ldap-no-tls]
|
376
|
+
in the GitLab project).
|
375
377
|
|
376
378
|
In EE, both the GitLab standard and LDAP credentials are needed:
|
377
379
|
|
@@ -393,7 +395,7 @@ $ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)
|
|
393
395
|
$ gitlab-qa Test::Integration::LDAPNoTLS EE
|
394
396
|
```
|
395
397
|
|
396
|
-
[test-integration-ldap-no-tls]: https://gitlab.com/gitlab-org/gitlab
|
398
|
+
[test-integration-ldap-no-tls]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/integration/ldap_no_tls.rb
|
397
399
|
|
398
400
|
### `Test::Integration::LDAPTLS CE|EE|<full image address>`
|
399
401
|
|
@@ -411,8 +413,8 @@ that LDAP-related features work as expected.
|
|
411
413
|
To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
|
412
414
|
container is spun up and tests are run from it by running the
|
413
415
|
`Test::Integration::LDAPTLS` scenario (located under
|
414
|
-
[`gitlab-org/gitlab
|
415
|
-
in the GitLab
|
416
|
+
[`gitlab-org/gitlab/qa/qa/scenario/test/integration/ldap_tls.rb`][test-integration-ldap-tls]
|
417
|
+
in the GitLab project).
|
416
418
|
|
417
419
|
In EE, both the GitLab standard and LDAP credentials are needed:
|
418
420
|
|
@@ -434,8 +436,8 @@ $ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)
|
|
434
436
|
$ gitlab-qa Test::Integration::LDAPTLS EE
|
435
437
|
```
|
436
438
|
|
437
|
-
[test-integration-ldap-tls]: https://gitlab.com/gitlab-org/gitlab
|
438
|
-
[test-integration-ldap-tls-certs]: https://gitlab.com/gitlab-org/gitlab
|
439
|
+
[test-integration-ldap-tls]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/integration/ldap_tls.rb
|
440
|
+
[test-integration-ldap-tls-certs]: https://gitlab.com/gitlab-org/gitlab/blob/master/tls_certificates/gitlab
|
439
441
|
|
440
442
|
### `Test::Integration::LDAPNoServer EE|<full image address>`
|
441
443
|
|
@@ -504,9 +506,9 @@ using SAML.
|
|
504
506
|
|
505
507
|
To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
|
506
508
|
container is spun up and tests are run from it by running the
|
507
|
-
`Test::Integration::InstanceSAML` scenario (located under [`gitlab-org/gitlab-ce@qa/qa/scenario/test/integration/instance_saml.rb`][test-integration-instance-saml] in the GitLab
|
509
|
+
`Test::Integration::InstanceSAML` scenario (located under [`gitlab-org/gitlab-ce@qa/qa/scenario/test/integration/instance_saml.rb`][test-integration-instance-saml] in the GitLab project).
|
508
510
|
|
509
|
-
[test-integration-instance-saml]: https://gitlab.com/gitlab-org/gitlab
|
511
|
+
[test-integration-instance-saml]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/integration/instance_saml.rb
|
510
512
|
|
511
513
|
**Required environment variables:**
|
512
514
|
|
@@ -531,8 +533,8 @@ Mattermost server (see `Test::Instance::Image` above).
|
|
531
533
|
To run tests against the GitLab container, a GitLab QA (`gitlab/gitlab-qa`)
|
532
534
|
container is spun up and tests are run from it by running the
|
533
535
|
`Test::Integration::Mattermost` scenario (located under
|
534
|
-
[`gitlab-org/gitlab
|
535
|
-
in the GitLab
|
536
|
+
[`gitlab-org/gitlab/qa/qa/scenario/test/integration/mattermost.rb`][test-integration-mattermost]
|
537
|
+
in the GitLab project).
|
536
538
|
|
537
539
|
**Required environment variables:**
|
538
540
|
|
@@ -549,7 +551,7 @@ $ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)
|
|
549
551
|
$ gitlab-qa Test::Integration::Mattermost EE
|
550
552
|
```
|
551
553
|
|
552
|
-
[test-integration-mattermost]: https://gitlab.com/gitlab-org/gitlab
|
554
|
+
[test-integration-mattermost]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/integration/mattermost.rb
|
553
555
|
|
554
556
|
### `Test::Integration::Packages CE|EE|<full image address>`
|
555
557
|
|
@@ -668,8 +670,8 @@ against it.
|
|
668
670
|
To run tests against the GitLab instance, a GitLab QA (`gitlab/gitlab-qa`)
|
669
671
|
container is spun up and tests are run from it by running the
|
670
672
|
`Test::Instance` scenario (located under
|
671
|
-
[`gitlab-org/gitlab
|
672
|
-
in the GitLab
|
673
|
+
[`gitlab-org/gitlab/qa/qa/scenario/test/instance.rb`][test-instance] in the
|
674
|
+
in the GitLab project).
|
673
675
|
|
674
676
|
Example:
|
675
677
|
|
@@ -695,8 +697,8 @@ works as expected by running tests against it.
|
|
695
697
|
To run tests against the GitLab instance, a GitLab QA (`gitlab/gitlab-qa`)
|
696
698
|
container is spun up and tests are run from it by running the
|
697
699
|
`Test::Instance` scenario (located under
|
698
|
-
[`gitlab-org/gitlab
|
699
|
-
in the GitLab
|
700
|
+
[`gitlab-org/gitlab/qa/qa/scenario/test/instance.rb`][test-instance] in the
|
701
|
+
in the GitLab project).
|
700
702
|
|
701
703
|
**Required environment variables:**
|
702
704
|
|
@@ -723,6 +725,42 @@ $ export GITLAB_PASSWORD="$GITLAB_QA_PASSWORD"
|
|
723
725
|
$ gitlab-qa Test::Instance::Staging
|
724
726
|
```
|
725
727
|
|
728
|
+
### `Test::Instance::StagingRef`
|
729
|
+
|
730
|
+
This scenario tests that the [`Staging Ref`](https://staging-ref.gitlab.com)
|
731
|
+
works as expected by running tests against it.
|
732
|
+
|
733
|
+
To run tests against the GitLab instance, a GitLab QA (`gitlab/gitlab-qa`)
|
734
|
+
container is spun up and tests are run from it by running the
|
735
|
+
`Test::Instance` scenario (located under
|
736
|
+
[`gitlab-org/gitlab/qa/qa/scenario/test/instance.rb`][test-instance] in the
|
737
|
+
in the GitLab project).
|
738
|
+
|
739
|
+
**Required environment variables:**
|
740
|
+
|
741
|
+
- `GITLAB_QA_ACCESS_TOKEN`: A valid personal access token with the `api` scope.
|
742
|
+
This is used to retrieve the version that staging is currently running.
|
743
|
+
Staging Ref QA users credentials can be found in the shared GitLab QA 1Password vault.
|
744
|
+
|
745
|
+
**Optional environment variables:**
|
746
|
+
|
747
|
+
- `GITLAB_QA_DEV_ACCESS_TOKEN`: A valid personal access token for the
|
748
|
+
`gitlab-qa-bot` on `dev.gitlab.org` with the `registry` scope.
|
749
|
+
This is used to pull the QA Docker image from the Omnibus GitLab `dev` Container Registry.
|
750
|
+
If the variable isn't present, the QA image from Docker Hub will be used.
|
751
|
+
This can be found in the shared GitLab QA 1Password vault.
|
752
|
+
|
753
|
+
Example:
|
754
|
+
|
755
|
+
```
|
756
|
+
$ export GITLAB_QA_ACCESS_TOKEN=your_api_access_token
|
757
|
+
$ export GITLAB_QA_DEV_ACCESS_TOKEN=your_dev_registry_access_token
|
758
|
+
$ export GITLAB_USERNAME="gitlab-qa"
|
759
|
+
$ export GITLAB_PASSWORD="$GITLAB_QA_PASSWORD"
|
760
|
+
|
761
|
+
$ gitlab-qa Test::Instance::StagingRef
|
762
|
+
```
|
763
|
+
|
726
764
|
### `Test::Instance::StagingGeo`
|
727
765
|
|
728
766
|
This scenario tests that the Geo staging deployment (with [`staging.gitlab.com`](https://staging.gitlab.com) as the primary site and [`geo.staging.gitlab.com`](https://geo.staging.gitlab.com) as the secondary site) works as expected by running tests tagged `:geo` against it. This is done by spinning up a GitLab QA (`gitlab/gitlab-qa`) container and running the `QA::EE::Scenario::Test::Geo` scenario. Note that the Geo setup steps in the `QA::EE::Scenario::Test::Geo` scenario are skipped when testing a live Geo deployment.
|
@@ -789,8 +827,8 @@ Smoke tests are quick tests that ensure that some basic functionality of GitLab
|
|
789
827
|
To run tests against the GitLab instance, a GitLab QA (`gitlab/gitlab-qa`)
|
790
828
|
container is spun up and tests are run from it by running the
|
791
829
|
`Test::Instance::Smoke` scenario (located under
|
792
|
-
[`gitlab-org/gitlab
|
793
|
-
in the GitLab
|
830
|
+
[`gitlab-org/gitlab/qa/qa/scenario/test/smoke.rb`][smoke-instance] in the
|
831
|
+
in the GitLab project).
|
794
832
|
|
795
833
|
Example:
|
796
834
|
|
@@ -863,5 +901,5 @@ $ gitlab-qa Test::Instance::Geo EE https://primary.gitlab.com https://secondary.
|
|
863
901
|
|
864
902
|
[Back to README.md](../README.md)
|
865
903
|
|
866
|
-
[test-instance]: https://gitlab.com/gitlab-org/gitlab
|
867
|
-
[smoke-instance]: https://gitlab.com/gitlab-org/gitlab
|
904
|
+
[test-instance]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/instance/all.rb
|
905
|
+
[smoke-instance]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/instance/smoke.rb
|
@@ -113,7 +113,7 @@ module Gitlab
|
|
113
113
|
"found:#{pipeline}.gitlab.com"
|
114
114
|
when 'preprod'
|
115
115
|
'found:pre.gitlab.com'
|
116
|
-
when 'staging-orchestrated', 'nightly', QA::Runtime::Env.default_branch
|
116
|
+
when 'staging-orchestrated', 'nightly', QA::Runtime::Env.default_branch, 'staging-ref', 'release'
|
117
117
|
"found:#{pipeline}"
|
118
118
|
else
|
119
119
|
raise "No `found:*` label for the `#{pipeline}` pipeline!"
|
@@ -15,6 +15,8 @@ module Gitlab
|
|
15
15
|
'QA_REMOTE_GRID_USERNAME' => :remote_grid_username,
|
16
16
|
'QA_REMOTE_GRID_ACCESS_KEY' => :remote_grid_access_key,
|
17
17
|
'QA_REMOTE_GRID_PROTOCOL' => :remote_grid_protocol,
|
18
|
+
'QA_REMOTE_MOBILE_DEVICE_NAME' => :remote_mobile_device_name,
|
19
|
+
'QA_REMOTE_TUNNEL_ID' => :remote_tunnel_id,
|
18
20
|
'QA_BROWSER' => :browser,
|
19
21
|
'QA_ADDITIONAL_REPOSITORY_STORAGE' => :qa_additional_repository_storage,
|
20
22
|
'QA_PRAEFECT_REPOSITORY_STORAGE' => :qa_praefect_repository_storage,
|
@@ -7,7 +7,7 @@ module Gitlab
|
|
7
7
|
class LicenseMode < Default
|
8
8
|
def configuration
|
9
9
|
<<~OMNIBUS
|
10
|
-
gitlab_rails['env'] = { 'GITLAB_LICENSE_MODE' => 'test', 'CUSTOMER_PORTAL_URL' => 'https://customers.
|
10
|
+
gitlab_rails['env'] = { 'GITLAB_LICENSE_MODE' => 'test', 'CUSTOMER_PORTAL_URL' => 'https://customers.staging.gitlab.com' }
|
11
11
|
OMNIBUS
|
12
12
|
end
|
13
13
|
end
|
@@ -0,0 +1,18 @@
|
|
1
|
+
module Gitlab
|
2
|
+
module QA
|
3
|
+
module Scenario
|
4
|
+
module Test
|
5
|
+
module Instance
|
6
|
+
##
|
7
|
+
# Run test suite against Staging Ref environment
|
8
|
+
#
|
9
|
+
class StagingRef < DeploymentBase
|
10
|
+
def deployment_component
|
11
|
+
Component::StagingRef
|
12
|
+
end
|
13
|
+
end
|
14
|
+
end
|
15
|
+
end
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
data/lib/gitlab/qa/version.rb
CHANGED
data/lib/gitlab/qa.rb
CHANGED
@@ -34,6 +34,7 @@ module Gitlab
|
|
34
34
|
autoload :RelativeUrl, 'gitlab/qa/scenario/test/instance/relative_url'
|
35
35
|
autoload :RepositoryStorage, 'gitlab/qa/scenario/test/instance/repository_storage'
|
36
36
|
autoload :Staging, 'gitlab/qa/scenario/test/instance/staging'
|
37
|
+
autoload :StagingRef, 'gitlab/qa/scenario/test/instance/staging_ref'
|
37
38
|
autoload :Production, 'gitlab/qa/scenario/test/instance/production'
|
38
39
|
autoload :Smoke, 'gitlab/qa/scenario/test/instance/smoke'
|
39
40
|
autoload :Preprod, 'gitlab/qa/scenario/test/instance/preprod'
|
@@ -87,6 +88,7 @@ module Gitlab
|
|
87
88
|
autoload :SAML, 'gitlab/qa/component/saml'
|
88
89
|
autoload :Specs, 'gitlab/qa/component/specs'
|
89
90
|
autoload :Staging, 'gitlab/qa/component/staging'
|
91
|
+
autoload :StagingRef, 'gitlab/qa/component/staging_ref'
|
90
92
|
autoload :Production, 'gitlab/qa/component/production'
|
91
93
|
autoload :Minio, 'gitlab/qa/component/minio'
|
92
94
|
autoload :Preprod, 'gitlab/qa/component/preprod'
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: gitlab-qa
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 7.
|
4
|
+
version: 7.13.3
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- GitLab Quality
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-11-04 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: climate_control
|
@@ -190,6 +190,7 @@ files:
|
|
190
190
|
- ".dockerignore"
|
191
191
|
- ".gitignore"
|
192
192
|
- ".gitlab-ci.yml"
|
193
|
+
- ".gitlab/ci/rules.gitlab-ci.yml"
|
193
194
|
- ".gitlab/merge_request_templates/Release.md"
|
194
195
|
- ".rspec"
|
195
196
|
- ".rubocop.yml"
|
@@ -240,6 +241,7 @@ files:
|
|
240
241
|
- lib/gitlab/qa/component/saml.rb
|
241
242
|
- lib/gitlab/qa/component/specs.rb
|
242
243
|
- lib/gitlab/qa/component/staging.rb
|
244
|
+
- lib/gitlab/qa/component/staging_ref.rb
|
243
245
|
- lib/gitlab/qa/docker/command.rb
|
244
246
|
- lib/gitlab/qa/docker/engine.rb
|
245
247
|
- lib/gitlab/qa/docker/shellout.rb
|
@@ -287,6 +289,7 @@ files:
|
|
287
289
|
- lib/gitlab/qa/scenario/test/instance/smoke.rb
|
288
290
|
- lib/gitlab/qa/scenario/test/instance/staging.rb
|
289
291
|
- lib/gitlab/qa/scenario/test/instance/staging_geo.rb
|
292
|
+
- lib/gitlab/qa/scenario/test/instance/staging_ref.rb
|
290
293
|
- lib/gitlab/qa/scenario/test/integration/actioncable.rb
|
291
294
|
- lib/gitlab/qa/scenario/test/integration/client_ssl.rb
|
292
295
|
- lib/gitlab/qa/scenario/test/integration/elasticsearch.rb
|