gitlab-qa 7.12.0 → 7.13.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -8,20 +8,49 @@ I.e, if you have a Selenium server set up at http://localhost:4444 or if you hav
8
8
 
9
9
  | Variable | Description | Default | Example(s) |
10
10
  |---------------------------|----------------------------------------------------------------|----------|--------------------------------|
11
- | QA_BROWSER | Browser to run against | "chrome" | "chrome" "firefox" |
11
+ | QA_BROWSER | Browser to run against | "chrome" | "chrome" "firefox" "safari" |
12
12
  | QA_REMOTE_GRID_PROTOCOL | Protocol to use | "http" | "http" "https" |
13
13
  | QA_REMOTE_GRID | Remote grid to run tests against | | "localhost:3000" "provider:80" |
14
- | QA_REMOTE_GRID_USERNAME | Username to specify in the remote grid. "USERNAME@provider:80" | | |
14
+ | QA_REMOTE_GRID_USERNAME | Username to specify in the remote grid. "USERNAME@provider:80" | | "gitlab-sl" |
15
15
  | QA_REMOTE_GRID_ACCESS_KEY | Key/Token paired with `QA_REMOTE_GRID_USERNAME` | | |
16
+ | QA_REMOTE_TUNNEL_ID | Name of the remote tunnel to use | "gitlab-sl_tunnel_id" | |
17
+ | QA_REMOTE_MOBILE_DEVICE_NAME | Name of mobile device to test against. `QA_BROWSER` must be set to `safari` for iOS devices and `chrome` for Android devices. | | "iPhone 12 Simulator" |
16
18
 
17
- ## Examples
19
+ ## Testing with Sauce Labs
18
20
 
19
- *Run QA Smoke Tests against firefox on SauceLabs*
21
+ Running directly against an environment like staging is not recommended because test logs expose credentials. Therefore, it is best practice and the default to use a tunnel.
20
22
 
21
- ```bash
22
- $ QA_BROWSER=firefox \
23
- QA_REMOTE_GRID=ondemand.saucelabs.com:80 \
24
- QA_REMOTE_GRID_USERNAME=user \
25
- QA_REMOTE_GRID_ACCESS_KEY=privatetoken \
26
- gitlab-qa Test::Instance::All --tags smoke
23
+ To install a tunnel, follow these [instructions](https://docs.saucelabs.com/secure-connections/sauce-connect/installation).
24
+
25
+ To start the tunnel, copy the run command in **Sauce Labs > Tunnels** and run it in the terminal. You must be logged in to Sauce Labs. Use the credentials in 1Password to log in.
26
+
27
+ It is highly recommended to use `GITLAB_QA_ACCESS_TOKEN` to speed up tests and reduce flakiness.
28
+
29
+
30
+ ### Run a test in a desktop browser
31
+
32
+ While tunnel is running, to test against a local instance in a desktop browser, run:
33
+ ```shell
34
+ $ QA_BROWSER="safari" \
35
+ QA_REMOTE_GRID="ondemand.saucelabs.com:80" \
36
+ QA_REMOTE_GRID_USERNAME="gitlab-sl" \
37
+ QA_REMOTE_GRID_ACCESS_KEY="<access key found in Sauce Lab account>" \
38
+ GITLAB_QA_ACCESS_TOKEN="<token>" \
39
+ gitlab-qa Test::Instance::Any <CE|EE> http://<local_ip>:3000 -- -- <relative_spec_path>
27
40
  ```
41
+
42
+ ### Run a test in a mobile device browser
43
+
44
+ `QA_REMOTE_MOBILE_DEVICE_NAME` can be any device name in the [supported browser devices](https://saucelabs.com/platform/supported-browsers-devices) in the Emulators/simulators list, and the latest versions of Android or iOS. You must set `QA_BROWSER` to `safari` for iOS devices and `chrome` for Android devices.```
45
+
46
+ ```shell
47
+ $ QA_BROWSER="safari" \
48
+ QA_REMOTE_MOBILE_DEVICE_NAME="iPhone 12 Simulator" \
49
+ QA_REMOTE_GRID="ondemand.saucelabs.com:80" \
50
+ QA_REMOTE_GRID_USERNAME="gitlab-sl" \
51
+ QA_REMOTE_GRID_ACCESS_KEY="<found in Sauce Lab account>" \
52
+ GITLAB_QA_ACCESS_TOKEN="<token>" \
53
+ gitlab-qa Test::Instance::Any <CE|EE> http://<local_ip>:3000 -- -- <relative_spec_path>
54
+ ```
55
+
56
+ Results can be watched in real time in Sauce Labs under AUTOMATED > Test Results
@@ -1,5 +1,7 @@
1
1
  # What tests can be run?
2
+
2
3
  [[_TOC_]]
4
+
3
5
  ## The two types of QA tests
4
6
 
5
7
  First of all, the first thing to choose is whether you want to run orchestrated
@@ -245,8 +247,8 @@ instance-level tests against it.
245
247
  To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
246
248
  container is spun up and tests are run from it by running the `Test::Instance`
247
249
  scenario (located under
248
- [`gitlab-org/gitlab-ce@qa/qa/scenario/test/instance.rb`][test-instance] in the
249
- GitLab CE project).
250
+ [`gitlab-org/gitlab/qa/qa/scenario/test/instance.rb`][test-instance] in the
251
+ GitLab project).
250
252
 
251
253
  Example:
252
254
 
@@ -370,8 +372,8 @@ that LDAP-related features work as expected.
370
372
  To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
371
373
  container is spun up and tests are run from it by running the
372
374
  `Test::Integration::LDAPNoTLS` scenario (located under
373
- [`gitlab-org/gitlab-ce@qa/qa/scenario/test/integration/ldap_no_tls.rb`][test-integration-ldap-no-tls]
374
- in the GitLab CE project).
375
+ [`gitlab-org/gitlab/qa/qa/scenario/test/integration/ldap_no_tls.rb`][test-integration-ldap-no-tls]
376
+ in the GitLab project).
375
377
 
376
378
  In EE, both the GitLab standard and LDAP credentials are needed:
377
379
 
@@ -393,7 +395,7 @@ $ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)
393
395
  $ gitlab-qa Test::Integration::LDAPNoTLS EE
394
396
  ```
395
397
 
396
- [test-integration-ldap-no-tls]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/qa/qa/scenario/test/integration/ldap_no_tls.rb
398
+ [test-integration-ldap-no-tls]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/integration/ldap_no_tls.rb
397
399
 
398
400
  ### `Test::Integration::LDAPTLS CE|EE|<full image address>`
399
401
 
@@ -411,8 +413,8 @@ that LDAP-related features work as expected.
411
413
  To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
412
414
  container is spun up and tests are run from it by running the
413
415
  `Test::Integration::LDAPTLS` scenario (located under
414
- [`gitlab-org/gitlab-ce@qa/qa/scenario/test/integration/ldap_tls.rb`][test-integration-ldap-tls]
415
- in the GitLab CE project).
416
+ [`gitlab-org/gitlab/qa/qa/scenario/test/integration/ldap_tls.rb`][test-integration-ldap-tls]
417
+ in the GitLab project).
416
418
 
417
419
  In EE, both the GitLab standard and LDAP credentials are needed:
418
420
 
@@ -434,8 +436,8 @@ $ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)
434
436
  $ gitlab-qa Test::Integration::LDAPTLS EE
435
437
  ```
436
438
 
437
- [test-integration-ldap-tls]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/qa/qa/scenario/test/integration/ldap_tls.rb
438
- [test-integration-ldap-tls-certs]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/tls_certificates/gitlab
439
+ [test-integration-ldap-tls]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/integration/ldap_tls.rb
440
+ [test-integration-ldap-tls-certs]: https://gitlab.com/gitlab-org/gitlab/blob/master/tls_certificates/gitlab
439
441
 
440
442
  ### `Test::Integration::LDAPNoServer EE|<full image address>`
441
443
 
@@ -504,9 +506,9 @@ using SAML.
504
506
 
505
507
  To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
506
508
  container is spun up and tests are run from it by running the
507
- `Test::Integration::InstanceSAML` scenario (located under [`gitlab-org/gitlab-ce@qa/qa/scenario/test/integration/instance_saml.rb`][test-integration-instance-saml] in the GitLab CE project).
509
+ `Test::Integration::InstanceSAML` scenario (located under [`gitlab-org/gitlab-ce@qa/qa/scenario/test/integration/instance_saml.rb`][test-integration-instance-saml] in the GitLab project).
508
510
 
509
- [test-integration-instance-saml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/qa/qa/scenario/test/integration/instance_saml.rb
511
+ [test-integration-instance-saml]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/integration/instance_saml.rb
510
512
 
511
513
  **Required environment variables:**
512
514
 
@@ -531,8 +533,8 @@ Mattermost server (see `Test::Instance::Image` above).
531
533
  To run tests against the GitLab container, a GitLab QA (`gitlab/gitlab-qa`)
532
534
  container is spun up and tests are run from it by running the
533
535
  `Test::Integration::Mattermost` scenario (located under
534
- [`gitlab-org/gitlab-ce@qa/qa/scenario/test/integration/mattermost.rb`][test-integration-mattermost]
535
- in the GitLab CE project).
536
+ [`gitlab-org/gitlab/qa/qa/scenario/test/integration/mattermost.rb`][test-integration-mattermost]
537
+ in the GitLab project).
536
538
 
537
539
  **Required environment variables:**
538
540
 
@@ -549,7 +551,7 @@ $ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)
549
551
  $ gitlab-qa Test::Integration::Mattermost EE
550
552
  ```
551
553
 
552
- [test-integration-mattermost]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/qa/qa/scenario/test/integration/mattermost.rb
554
+ [test-integration-mattermost]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/integration/mattermost.rb
553
555
 
554
556
  ### `Test::Integration::Packages CE|EE|<full image address>`
555
557
 
@@ -668,8 +670,8 @@ against it.
668
670
  To run tests against the GitLab instance, a GitLab QA (`gitlab/gitlab-qa`)
669
671
  container is spun up and tests are run from it by running the
670
672
  `Test::Instance` scenario (located under
671
- [`gitlab-org/gitlab-ce@qa/qa/scenario/test/instance.rb`][test-instance] in the
672
- in the GitLab CE project).
673
+ [`gitlab-org/gitlab/qa/qa/scenario/test/instance.rb`][test-instance] in the
674
+ in the GitLab project).
673
675
 
674
676
  Example:
675
677
 
@@ -695,8 +697,8 @@ works as expected by running tests against it.
695
697
  To run tests against the GitLab instance, a GitLab QA (`gitlab/gitlab-qa`)
696
698
  container is spun up and tests are run from it by running the
697
699
  `Test::Instance` scenario (located under
698
- [`gitlab-org/gitlab-ce@qa/qa/scenario/test/instance.rb`][test-instance] in the
699
- in the GitLab CE project).
700
+ [`gitlab-org/gitlab/qa/qa/scenario/test/instance.rb`][test-instance] in the
701
+ in the GitLab project).
700
702
 
701
703
  **Required environment variables:**
702
704
 
@@ -723,6 +725,42 @@ $ export GITLAB_PASSWORD="$GITLAB_QA_PASSWORD"
723
725
  $ gitlab-qa Test::Instance::Staging
724
726
  ```
725
727
 
728
+ ### `Test::Instance::StagingRef`
729
+
730
+ This scenario tests that the [`Staging Ref`](https://staging-ref.gitlab.com)
731
+ works as expected by running tests against it.
732
+
733
+ To run tests against the GitLab instance, a GitLab QA (`gitlab/gitlab-qa`)
734
+ container is spun up and tests are run from it by running the
735
+ `Test::Instance` scenario (located under
736
+ [`gitlab-org/gitlab/qa/qa/scenario/test/instance.rb`][test-instance] in the
737
+ in the GitLab project).
738
+
739
+ **Required environment variables:**
740
+
741
+ - `GITLAB_QA_ACCESS_TOKEN`: A valid personal access token with the `api` scope.
742
+ This is used to retrieve the version that staging is currently running.
743
+ Staging Ref QA users credentials can be found in the shared GitLab QA 1Password vault.
744
+
745
+ **Optional environment variables:**
746
+
747
+ - `GITLAB_QA_DEV_ACCESS_TOKEN`: A valid personal access token for the
748
+ `gitlab-qa-bot` on `dev.gitlab.org` with the `registry` scope.
749
+ This is used to pull the QA Docker image from the Omnibus GitLab `dev` Container Registry.
750
+ If the variable isn't present, the QA image from Docker Hub will be used.
751
+ This can be found in the shared GitLab QA 1Password vault.
752
+
753
+ Example:
754
+
755
+ ```
756
+ $ export GITLAB_QA_ACCESS_TOKEN=your_api_access_token
757
+ $ export GITLAB_QA_DEV_ACCESS_TOKEN=your_dev_registry_access_token
758
+ $ export GITLAB_USERNAME="gitlab-qa"
759
+ $ export GITLAB_PASSWORD="$GITLAB_QA_PASSWORD"
760
+
761
+ $ gitlab-qa Test::Instance::StagingRef
762
+ ```
763
+
726
764
  ### `Test::Instance::StagingGeo`
727
765
 
728
766
  This scenario tests that the Geo staging deployment (with [`staging.gitlab.com`](https://staging.gitlab.com) as the primary site and [`geo.staging.gitlab.com`](https://geo.staging.gitlab.com) as the secondary site) works as expected by running tests tagged `:geo` against it. This is done by spinning up a GitLab QA (`gitlab/gitlab-qa`) container and running the `QA::EE::Scenario::Test::Geo` scenario. Note that the Geo setup steps in the `QA::EE::Scenario::Test::Geo` scenario are skipped when testing a live Geo deployment.
@@ -789,8 +827,8 @@ Smoke tests are quick tests that ensure that some basic functionality of GitLab
789
827
  To run tests against the GitLab instance, a GitLab QA (`gitlab/gitlab-qa`)
790
828
  container is spun up and tests are run from it by running the
791
829
  `Test::Instance::Smoke` scenario (located under
792
- [`gitlab-org/gitlab-ce@qa/qa/scenario/test/smoke.rb`][smoke-instance] in the
793
- in the GitLab CE project).
830
+ [`gitlab-org/gitlab/qa/qa/scenario/test/smoke.rb`][smoke-instance] in the
831
+ in the GitLab project).
794
832
 
795
833
  Example:
796
834
 
@@ -863,5 +901,5 @@ $ gitlab-qa Test::Instance::Geo EE https://primary.gitlab.com https://secondary.
863
901
 
864
902
  [Back to README.md](../README.md)
865
903
 
866
- [test-instance]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/qa/qa/scenario/test/instance/all.rb
867
- [smoke-instance]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/qa/qa/scenario/test/instance/smoke.rb
904
+ [test-instance]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/instance/all.rb
905
+ [smoke-instance]: https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/scenario/test/instance/smoke.rb
@@ -1,7 +1,3 @@
1
- require 'net/http'
2
- require 'json'
3
- require 'uri'
4
-
5
1
  module Gitlab
6
2
  module QA
7
3
  module Component
@@ -1,7 +1,3 @@
1
- require 'net/http'
2
- require 'json'
3
- require 'uri'
4
-
5
1
  module Gitlab
6
2
  module QA
7
3
  module Component
@@ -1,7 +1,3 @@
1
- require 'net/http'
2
- require 'json'
3
- require 'uri'
4
-
5
1
  module Gitlab
6
2
  module QA
7
3
  module Component
@@ -0,0 +1,9 @@
1
+ module Gitlab
2
+ module QA
3
+ module Component
4
+ class StagingRef < Staging
5
+ ADDRESS = 'https://staging-ref.gitlab.com'.freeze
6
+ end
7
+ end
8
+ end
9
+ end
@@ -113,7 +113,7 @@ module Gitlab
113
113
  "found:#{pipeline}.gitlab.com"
114
114
  when 'preprod'
115
115
  'found:pre.gitlab.com'
116
- when 'staging-orchestrated', 'nightly', QA::Runtime::Env.default_branch
116
+ when 'staging-orchestrated', 'nightly', QA::Runtime::Env.default_branch, 'staging-ref', 'release'
117
117
  "found:#{pipeline}"
118
118
  else
119
119
  raise "No `found:*` label for the `#{pipeline}` pipeline!"
@@ -15,6 +15,8 @@ module Gitlab
15
15
  'QA_REMOTE_GRID_USERNAME' => :remote_grid_username,
16
16
  'QA_REMOTE_GRID_ACCESS_KEY' => :remote_grid_access_key,
17
17
  'QA_REMOTE_GRID_PROTOCOL' => :remote_grid_protocol,
18
+ 'QA_REMOTE_MOBILE_DEVICE_NAME' => :remote_mobile_device_name,
19
+ 'QA_REMOTE_TUNNEL_ID' => :remote_tunnel_id,
18
20
  'QA_BROWSER' => :browser,
19
21
  'QA_ADDITIONAL_REPOSITORY_STORAGE' => :qa_additional_repository_storage,
20
22
  'QA_PRAEFECT_REPOSITORY_STORAGE' => :qa_praefect_repository_storage,
@@ -7,7 +7,7 @@ module Gitlab
7
7
  class LicenseMode < Default
8
8
  def configuration
9
9
  <<~OMNIBUS
10
- gitlab_rails['env'] = { 'GITLAB_LICENSE_MODE' => 'test', 'CUSTOMER_PORTAL_URL' => 'https://customers.stg.gitlab.com' }
10
+ gitlab_rails['env'] = { 'GITLAB_LICENSE_MODE' => 'test', 'CUSTOMER_PORTAL_URL' => 'https://customers.staging.gitlab.com' }
11
11
  OMNIBUS
12
12
  end
13
13
  end
@@ -0,0 +1,18 @@
1
+ module Gitlab
2
+ module QA
3
+ module Scenario
4
+ module Test
5
+ module Instance
6
+ ##
7
+ # Run test suite against Staging Ref environment
8
+ #
9
+ class StagingRef < DeploymentBase
10
+ def deployment_component
11
+ Component::StagingRef
12
+ end
13
+ end
14
+ end
15
+ end
16
+ end
17
+ end
18
+ end
@@ -1,5 +1,5 @@
1
1
  module Gitlab
2
2
  module QA
3
- VERSION = '7.12.0'.freeze
3
+ VERSION = '7.13.3'.freeze
4
4
  end
5
5
  end
data/lib/gitlab/qa.rb CHANGED
@@ -34,6 +34,7 @@ module Gitlab
34
34
  autoload :RelativeUrl, 'gitlab/qa/scenario/test/instance/relative_url'
35
35
  autoload :RepositoryStorage, 'gitlab/qa/scenario/test/instance/repository_storage'
36
36
  autoload :Staging, 'gitlab/qa/scenario/test/instance/staging'
37
+ autoload :StagingRef, 'gitlab/qa/scenario/test/instance/staging_ref'
37
38
  autoload :Production, 'gitlab/qa/scenario/test/instance/production'
38
39
  autoload :Smoke, 'gitlab/qa/scenario/test/instance/smoke'
39
40
  autoload :Preprod, 'gitlab/qa/scenario/test/instance/preprod'
@@ -87,6 +88,7 @@ module Gitlab
87
88
  autoload :SAML, 'gitlab/qa/component/saml'
88
89
  autoload :Specs, 'gitlab/qa/component/specs'
89
90
  autoload :Staging, 'gitlab/qa/component/staging'
91
+ autoload :StagingRef, 'gitlab/qa/component/staging_ref'
90
92
  autoload :Production, 'gitlab/qa/component/production'
91
93
  autoload :Minio, 'gitlab/qa/component/minio'
92
94
  autoload :Preprod, 'gitlab/qa/component/preprod'
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: gitlab-qa
3
3
  version: !ruby/object:Gem::Version
4
- version: 7.12.0
4
+ version: 7.13.3
5
5
  platform: ruby
6
6
  authors:
7
7
  - GitLab Quality
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2021-10-26 00:00:00.000000000 Z
11
+ date: 2021-11-04 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: climate_control
@@ -190,6 +190,7 @@ files:
190
190
  - ".dockerignore"
191
191
  - ".gitignore"
192
192
  - ".gitlab-ci.yml"
193
+ - ".gitlab/ci/rules.gitlab-ci.yml"
193
194
  - ".gitlab/merge_request_templates/Release.md"
194
195
  - ".rspec"
195
196
  - ".rubocop.yml"
@@ -240,6 +241,7 @@ files:
240
241
  - lib/gitlab/qa/component/saml.rb
241
242
  - lib/gitlab/qa/component/specs.rb
242
243
  - lib/gitlab/qa/component/staging.rb
244
+ - lib/gitlab/qa/component/staging_ref.rb
243
245
  - lib/gitlab/qa/docker/command.rb
244
246
  - lib/gitlab/qa/docker/engine.rb
245
247
  - lib/gitlab/qa/docker/shellout.rb
@@ -287,6 +289,7 @@ files:
287
289
  - lib/gitlab/qa/scenario/test/instance/smoke.rb
288
290
  - lib/gitlab/qa/scenario/test/instance/staging.rb
289
291
  - lib/gitlab/qa/scenario/test/instance/staging_geo.rb
292
+ - lib/gitlab/qa/scenario/test/instance/staging_ref.rb
290
293
  - lib/gitlab/qa/scenario/test/integration/actioncable.rb
291
294
  - lib/gitlab/qa/scenario/test/integration/client_ssl.rb
292
295
  - lib/gitlab/qa/scenario/test/integration/elasticsearch.rb