gitlab-qa 0.6.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA256:
-  metadata.gz: a621d1f6282fb329b8f8ae089a113b66b5129017b4dc352d9afa3d21ce4a00b4
-  data.tar.gz: 51ee39c46554f5c32869c8ab96e53a1671b79790d1ee954ef1de744501ba32d9
+SHA1:
+  metadata.gz: 2417c38c75bb22753661942c5431959e224d9439
+  data.tar.gz: de43e7bbe95039d2118d66cbbdb4b1b222efe4cb
 SHA512:
-  metadata.gz: be2da63340fc7348cc62698396eb295490f4e8955ca7fe27b50c3a7098827c74261fb4e6b38981346a273a8f9edc04a26783ec5c279f244679cf60d4085e24f1
-  data.tar.gz: f5362769d30beeeca1295783ea201b63ce5e3901bc553aa810863968fd9ea8011eac7fa98481f4cbef1b58fa3b1a1d88feab985d0e6ec24c45ecde1ddfbc37a2
+  metadata.gz: 561abef16e8297daf5ae70ad37e1f6e05be302815cf0c60088b8bb486af1079d57e96f20e3cbc9d833094479e841fd59ea669da2960a25aacc67bcf2b84d6d4d
+  data.tar.gz: 9a8ebd4089627554cb9f45ac5586c3210441b1a428ac68fbe286742e93b3cd02ba2cd4d2e1293e12d2cefc8d2970d31c64bd0668c69948a0ded5d49816825644

data/.gitlab-ci.yml

@@ -118,6 +118,12 @@ ee:update:
   <<: *test
   <<: *high-capacity
 
+ee-previous-to-ce-next:update:
+  script:
+     - bin/qa Test::Omnibus::Update EE CE
+  <<: *test
+  <<: *high-capacity
+
 ce:upgrade:
   script:
     - if [[ $RELEASE ]]; then exit 0; fi

data/README.md

@@ -1,36 +1,51 @@
 [![Gem Version](https://badge.fury.io/rb/gitlab-qa.svg)](https://rubygems.org/gems/gitlab-qa)
 [![build status](https://gitlab.com/gitlab-org/gitlab-qa/badges/master/build.svg)](https://gitlab.com/gitlab-org/gitlab-qa/pipelines)
 
-# GitLab end-to-end tests
+# GitLab QA orchestrator
+
+## Definitions
+
+- **GitLab QA framework**: A framework that allows developer to write end-to-end
+  tests simply and efficiently.
+  Located at [`gitlab-org/gitlab-ce@qa/qa/`][qa-framework].
+- **GitLab QA instance-level scenarios**: RSpec scenarios that are using
+  GitLab QA framework and Capybara to setup and perform individual end-to-end
+  tests against a live GitLab instance.
+  Located at [`gitlab-org/gitlab-ce@qa/qa/specs/features/`][instance-level-scenarios].
+- **GitLab QA orchestrator** (this project): An orchestration tool that allows to run various
+  QA test suites in a simple manner.
+- **GitLab QA orchestrated scenarios**: Scenarios where containers are started,
+  configured and that run instance-level scenarios against a live GitLab
+  instance.
+  Located at [`gitlab-org/gitlab-qa@lib/gitlab/qa/scenario/test/`][orchestrated-scenarios].
+
+[qa-framework]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/qa/qa/
+[instance-level-scenarios]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/qa/qa/specs/features/
+[orchestrated-scenarios]: https://gitlab.com/gitlab-org/gitlab-qa/blob/master/lib/gitlab/qa/scenario/test/
 
-End-to-end test suite that verifies GitLab as a whole.
+## Goals and objectives
 
 GitLab consists of multiple pieces configured and packaged by
 [GitLab Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab).
 
-The purpose of this test suite is to verify that all pieces do integrate well together.
-
-## Architecture
-
-See the [GitLab QA architecture](/docs/architecture.md).
-
-## Goals and objectives
+The purpose of the QA end-to-end test suite is to verify that all pieces
+integrate well together.
 
 ### GitLab QA tests running in the CI/CD environment
 
-GitLab QA is an automated end-to-end testing framework, which means that manual
-steps should not be needed to run GitLab QA test suite. This framework is
-CI/CD environment native, which means that we should add new features and tests
-when we are comfortable with running new code in the CI/CD environment.
+Manual steps should not be needed to run the QA test suite.
+GitLab QA orchestrator is CI/CD environment native, which means that we should
+add new features and tests when we are comfortable with running new code in the
+CI/CD environment.
 
 ### GitLab QA test failure are reproducible locally
 
-Despite the fact that GitLab QA has been built to run in the CI/CD environment,
-it is really important to make it easy for developers to reproduce GitLab QA
+Despite the fact that GitLab QA orchestrator has been built to run in the CI/CD
+environment, it is really important to make it easy for developers to reproduce
 test failures locally. It is much easier to debug things locally, than in the
 CI/CD.
 
-To make it easier to reproduce test failures locally we have published
+To make it easier to reproduce test failures locally we have published the
 `gitlab-qa` gem [on rubygems.org](https://rubygems.org/gems/gitlab-qa) and we
 are using exactly the same approach to run tests in the CI/CD environment.
 
@@ -38,7 +53,7 @@ It means that using `gitlab-qa` CLI tool, that orchestrates test environment and
 runs GitLab QA test suite, is a reproducible way of running tests locally and
 in the CI/CD.
 
-It also means that we can not have a custom code in `.gitlab-ci.yml` to, for
+It also means that we cannot have a custom code in `.gitlab-ci.yml` to, for
 example, start new containers / services.
 
 ### Test installation / deployment process too
@@ -47,7 +62,7 @@ We distribute GitLab in a package (like Debian package or a Docker image) so
 we want to test installation process to ensure that our package is not broken.
 
 But we are also working on making GitLab be a cloud native product. This means
-that using Helm becomes a yet another installation / deployment process that we
+that using Helm becomes yet another installation / deployment process that we
 want to test with GitLab QA.
 
 Keeping in mind that we want to test our Kubernetes deployments is especially
@@ -55,8 +70,8 @@ important with consideration of the need of testing changes in merge requests.
 
 ### Testing changes in merge requests before the merge
 
-The ultimate goal of GitLab QA is to make it possible to test changes in
-merge requests, even before merging a code into the stable / master branch.
+The ultimate goal is to make it possible to run the QA test suite for any
+merge request, even before merging code into the `master` branch.
 
 ### We can run tests against any instance of GitLab
 
@@ -64,12 +79,16 @@ GitLab QA is a click-driven, black-box testing tool. We also use it to run
 tests against the staging, and we strive for making it useful for our users
 as well.
 
+## Documentation
+
+All the documentation can be found under [`docs/`](/docs/README.md).
+
 ## How do we use it
 
 Currently we trigger test suite against GitLab Docker images created by Omnibus
 nightly.
 
-We also trigger GitLab QA pipelines whenever someone clicks `package-qa` manual
+We also trigger GitLab QA pipelines whenever someone clicks `package-and-qa` manual
 action in a merge request.
 
 ## How can you use it
@@ -80,115 +99,19 @@ You can install it with `gem install gitlab-qa`. It will expose a `gitlab-qa`
 command in your system.
 
 If you want to run the scenarios or develop them on Mac OS, please read
-[Mac OS specific documentation](/docs/macos.md) as there are caveats and things
+[Mac OS specific documentation](/docs/run_geo_qa_against_gdk.md) as there are caveats and things
 that may work differently.
 
-1. Run tests against a Docker image with GitLab:
-
-    `gitlab-qa Test::Instance::Image CE|EE|<full image address>`
-
-1. Run tests against a Docker image with GitLab and Mattermost:
-
-    `gitlab-qa Test::Integration::Mattermost CE|EE|<full image address>`
-
-1. Run tests agains a Docker image with GitLab Geo:
-
-    `export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)`
-
-    `gitlab-qa Test::Integration::Geo EE`
-
-1. Test update process between two CE or EE subsequent versions:
-
-    `gitlab-qa Test::Omnibus::Update CE|EE|<full image address>`
-
-1. Test upgrade process from CE to EE:
-
-    `gitlab-qa Test::Omnibus::Upgrade CE|<full CE image address>`
-
-1. Run tests against any existing instance:
-
-    `gitlab-qa Test::Instance::Any CE|EE nightly|latest http://your.instance.gitlab`
-
-    For instance, to run it against https://staging.gitlab.com:
-
-    `GITLAB_USERNAME=your_username GITLAB_PASSWORD=your_password gitlab-qa Test::Instance::Any EE latest https://staging.gitlab.com`
+All the scenarios you can run are described in the
+[What tests can be run?](/docs/what_tests_can_be_run.md) documentation.
 
 ### How to add new scenarios
 
 Scenarios (test cases) and scripts to run them are located in
 [CE](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa) and
 [EE](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/qa)
-repositories under qa/ directory, so please also check the documents there.
-
-## How does it work?
-
-GitLab QA handles a few scenarios:
-
-### `Test::Omnibus::Image CE|EE|<full image address>`
-
-This scenario only tests that a GitLab Docker container can be run.
-
-This spins up a GitLab Docker container based on the given edition or image:
-  - `gitlab/gitlab-ce:nightly` for `CE`
-  - `gitlab/gitlab-ee:nightly` for `EE`
-  - the given custom image for `<full image address>`
-
-### `Test::Omnibus::Upgrade CE|EE|<full image address>`
-
-This scenario tests that:
-
-- the GitLab Docker container works as expected by running tests against it (see
-  `Test::Instance::Image` below)
-- that it can be upgraded to a new (`nightly` or custom image) container
-- that the new GitLab container still works as expected
-
-### `Test::Instance::Image CE|EE|<full image address>`
-
-This scenario tests that the GitLab Docker container works as expected by
-running tests against it.
-
-To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
-container is spun up and tests are run from it by running the `Test::Instance`
-scenario (located under `qa/scenario/test/instance` in the GitLab codebase).
-
-### `Test::Integration::Mattermost CE|EE|<full image address>`
-
-This scenario tests that GitLab instance works as expected when
-enabling the embedded Mattermost server (see `Test::Instance::Image`
-above).
-
-### `Test::Integration::LDAP CE|EE|<full image address>`
-
-This scenario tests that GitLab instance works as expected with an external
-LDAP server. The scenario spins up an OpenLDAP server, seeds users, and
-attempts to LDAP-related tests against a GitLab instance.
-
-In EE, both the GitLab standard and LDAP credentials are needed:
-
-1. The first is used to login as an Admin to enter in the EE license.
-2. The second is used to conduct LDAP-related tasks
-
-### `Test::Instance::Any CE|EE|<full image address>`
-
-This scenario tests that the any GitLab instance works as expected by running
-tests against it (see `Test::Instance::Image` above).
-
-## Supported environment variables
-
-* `GITLAB_USERNAME` - username to use when signing into GitLab
-* `GITLAB_PASSWORD` - password to use when signing into GitLab
-* `GITLAB_USER_TYPE` - type of user to use when signing into GitLab: standard (default), ldap
-* `GITLAB_LDAP_USERNAME` - LDAP username to use when signing into GitLab
-* `GITLAB_LDAP_PASSWORD` - LDAP password to use when signing into GitLab
-* `GITLAB_SANDBOX_NAME` - The sandbox group name test suite is going to use (default: `gitlab-qa-sandbox`)
-* `EE_LICENSE` - Enterprise Edition license
-* `QA_SCREENSHOTS_DIR` - Path to a directory where screenshots for failing tests
-  will be saved (default: `/tmp/gitlab-qa/screenshots`)
-* `QA_LOGS_DIR` - Path to a directory where logs will be saved (default:
-  `/tmp/gitlab-qa/logs`)
-* `DOCKER_HOST` - Docker host to run tests against (default: `http://localhost`)
-* `CHROME_HEADLESS` - when running locally, set to `false` to allow Chrome tests to be visible - watch your tests being run
+repositories under `qa/` directory, so please also check the documentation there.
 
 ## Contributing
 
-Please see the [contribution guidelines](CONTRIBUTING.md)
+Please see the [contribution guidelines](CONTRIBUTING.md).

data/docs/README.md

@@ -1,4 +1,7 @@
 # GitLab QA Documentation
 
 - [Architecture](architecture.md)
+- [How it works](how_it_works.md)
+- [What tests can be run?](what_tests_can_be_run.md)
+- [Run Geo QA tests against your Geo GDK setup](run_geo_qa_against_gdk.md)
 - [Trainings](trainings.md)

data/docs/architecture.md

@@ -3,3 +3,7 @@
 <img src="https://docs.google.com/drawings/d/e/2PACX-1vTBlBcIBFFnQOd9xGQ7-2rij96mP2-ajNd9sQILhSI2D7x-q_8oMuO_GFG8-BWRoQZkVHWdQZTdXF8c/pub?w=855&amp;h=2426">
 
 _[edit diagram (for GitLab team members only)](https://docs.google.com/drawings/d/12mTCF7BU-xaxxS9MmIjkSYnf31RN4wgkkHcXTmS0NoA/edit)_
+
+----
+
+[Back to README.md](../README.md)

data/docs/how_it_works.md

@@ -0,0 +1,37 @@
+# How it works
+
+## Architecture
+
+For a visual architecture explanation, please read the
+[GitLab QA architecture](./architecture.md) documentation.
+
+## What happens when you run `gitlab-qa Test::Instance::Image CE`
+
+1. `bin/qa` calls [`.perform`][instance-image] on the fully qualified constant
+  `Gitlab::QA::Scenario::` + the scenario name, e.g. `QA::Scenario::Test::Instance::Image`
+1. A new `gitlab-ce` container is started with `Component::Gitlab.perform`
+1. Then `Component::Specs.perform` is called, which starts a `gitlab/gitlab-qa`
+  container (from [`gitlab-org/gitlab-ce@qa/Dockerfile`][gitlab-dockerfile])
+  and pass it the instance-level scenario to run (e.g. `Test::Instance` for
+  [`gitlab-org/gitlab-qa@lib/gitlab/qa/scenario/test/instance/image.rb`][instance-image]),
+  then the address of the live instance to be tested, and optional extra arguments.
+1. Within the `gitlab/gitlab-qa` container, these arguments are passed to `bin/test` (since it's
+  the
+  [`ENTRYPOINT` defined at `gitlab-org/gitlab-ce@qa/Dockerfile`][gitlab-dockerfile]),
+  and then to `bin/qa`.
+1. `bin/qa` then calls `.launch!` on the fully qualified constant `QA::Scenario::` +
+  the scenario name, e.g. `QA::Scenario::Test::Instance`, and ultimately calls
+  [`.perform`][instance]
+1. The `.perform` method [saves the instance address for later][instance], then sets up an
+  `RSpec::Core::Runner`, pass it the extra arguments,
+  [configure the `Capybara` browser environment][runner],
+  and [starts the actual `RSpec` run][runner].
+
+[instance-image]: https://gitlab.com/gitlab-org/gitlab-qa/blob/master/lib/gitlab/qa/scenario/test/instance/image.rb
+[gitlab-dockerfile]: https://gitlab.com/gitlab-org/gitlab-ce/blob/60f51cd20af5db8759c31c32a9c45db5b5be2199/qa/Dockerfile
+[instance]: https://gitlab.com/gitlab-org/gitlab-ce/blob/60f51cd20af5db8759c31c32a9c45db5b5be2199/qa/qa/scenario/test/instance.rb
+[runner]: https://gitlab.com/gitlab-org/gitlab-ce/blob/60f51cd20af5db8759c31c32a9c45db5b5be2199/qa/qa/specs/runner.rb
+
+----
+
+[Back to README.md](../README.md)

data/docs/{macos.md → run_geo_qa_against_gdk.md}

@@ -25,10 +25,10 @@ docker-machine and possibly customizing network settings in the virtualization p
 
 Native Docker bundles it's own lightweight virtualization technology that works just like VirtualBox,
 but without requiring manual intervention. This provides less opportunity to customize network between
-docker containers and the host machine, but works out of the box when mapping container ports to ports 
+docker containers and the host machine, but works out of the box when mapping container ports to ports
 on localhost.
 
-The major difference is that it never exposes the network as `bridge` to macOS, and so `--hostname` 
+The major difference is that it never exposes the network as `bridge` to macOS, and so `--hostname`
 and `--network` only work inside docker, it has no effect when trying to access the containers from macOS.
 
 There are people in Docker's forum that claim to be able to [expose the network][Docker Route]
@@ -43,11 +43,10 @@ your DNS and point to the IP/port of `dnsdock` application.
 When using OS X Docker, you need to go to Preferences > Advanced and allocate at least **5.0 GB**,
 otherwise some steps may fail to execute the `chrome-webdriver`.
 
-When using docker-machine, see [this StackOverflow link for increasing memory]
-(https://stackoverflow.com/questions/32834082/how-to-increase-docker-machine-memory-mac/36982696#36982696).
+When using docker-machine, see [this StackOverflow link for increasing memory](https://stackoverflow.com/questions/32834082/how-to-increase-docker-machine-memory-mac/36982696#36982696).
 
 This is required because chrome-webdriver makes use of `/dev/shm` shared memory. The VM will normally use
-~ 3Gb but unless you allocate 5.0 or more some magic numbers may not enable a bigger /dev/shm in the
+~3GB but unless you allocate 5GB or more some magic numbers may not enable a bigger `/dev/shm` in the
 'host' VM that "native" docker runs on.
 
 Please note that while it's possible to run multi-node tests like Geo Primary and Secondary, you can't
@@ -55,7 +54,7 @@ access the machines from your host machine, as they are all exposed as `0.0.0.0:
 of that they don't match the configured VHOSTs in your GitLab installation, so the redirect login
 fails.
 
-It has to do with the lack of `bridge` support from Docker on macOS, also this is also something 
+It has to do with the lack of `bridge` support from Docker on macOS, also this is also something
 Docker Inc [doesn't want to fix][Docker bridge issue].
 
 To see if this limitation is still present, check the [documentation][Docker Networking].
@@ -112,3 +111,6 @@ You should be able to use your navigator and point it to `http://gitlab-primary.
 [Docker bridge issue]: https://github.com/moby/moby/issues/22753#issuecomment-253534261
 [dnsdock]: https://github.com/aacebedo/dnsdock
 
+----
+
+[Back to README.md](../README.md)

data/docs/trainings.md

@@ -6,3 +6,11 @@ A great introduction to the GitLab QA project, its internal structure, how it's
 used, and how to start contributing to it. Check out the
 [video](https://youtu.be/Ym159ATYN_g) and
 [slides](https://docs.google.com/presentation/d/1-3YlYTIBzd2kSjGVYGPq1xqz4O7qtfABhoZRi_PcGRg/edit?usp=sharing).
+
+## GitLab QA at FOSDEM
+
+[Grzegorz's presentation of GitLab QA at FOSDEM 2018](https://youtu.be/arPxzixIl38).
+
+----
+
+[Back to README.md](../README.md)

data/docs/what_tests_can_be_run.md

@@ -0,0 +1,266 @@
+# What tests can be run?
+
+## The two types of QA tests
+
+First of all, the first thing to choose is whether you want to run orchestrated
+tests (various Docker containers are spun up and tests are run against them,
+also from a specific Docker container) or instance-level tests (tests are run
+from your host machine against a live instance: a local GDK installation or a staging/production instance).
+
+Ultimately, orchestrated tests run instance-level tests, the difference being
+that these tests are run from a specific Docker container instead of from your
+host machine.
+
+## Orchestrated tests
+
+Orchestrated tests are run with the `gitlab-qa` binary (from the
+`gitlab-qa` gem), or in the `gitlab-qa` project, with the `bin/qa` binary
+(useful if you're working on the `gitlab-qa` project itself and want to test
+your changes).
+
+These tests spin up Docker containers specifically to run tests against them.
+Orchestrated tests are usually used for features that involve external services
+or complex setup (e.g. LDAP, Geo etc.), or for generic Omnibus checks (ensure
+our Omnibus package works, can be updated / upgraded to EE etc.).
+
+For more details on the internals, please read the
+[How it works](./how_it_works.md) documentation.
+
+## Supported environment variables
+
+* `GITLAB_USERNAME` - username to use when signing into GitLab
+* `GITLAB_PASSWORD` - password to use when signing into GitLab
+* `GITLAB_USER_TYPE` - type of user to use when signing into GitLab: standard (default), ldap
+* `GITLAB_LDAP_USERNAME` - LDAP username to use when signing into GitLab
+* `GITLAB_LDAP_PASSWORD` - LDAP password to use when signing into GitLab
+* `GITLAB_SANDBOX_NAME` - The sandbox group name the test suite is going to use (default: `gitlab-qa-sandbox`)
+* `EE_LICENSE` - Enterprise Edition license
+* `QA_SCREENSHOTS_DIR` - Path to a directory where screenshots for failing tests
+  will be saved (default: `/tmp/gitlab-qa/screenshots`)
+* `QA_LOGS_DIR` - Path to a directory where logs will be saved (default:
+  `/tmp/gitlab-qa/logs`)
+* `DOCKER_HOST` - Docker host to run tests against (default: `http://localhost`)
+* `CHROME_HEADLESS` - when running locally, set to `false` to allow Chrome tests to be visible - watch your tests being run
+
+### `Test::Instance::Image CE|EE|<full image address>`
+
+This tests that a GitLab Docker container works as expected by running
+instance-level tests against it.
+
+To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
+container is spun up and tests are run from it by running the `Test::Instance`
+scenario (located under
+[`gitlab-org/gitlab-ce@qa/qa/scenario/test/instance.rb`][test-instance] in the
+GitLab CE project).
+
+Example:
+
+```
+$ gitlab-qa Test::Instance::Image CE
+```
+
+### `Test::Omnibus::Image CE|EE|<full image address>`
+
+This tests that a GitLab Docker container can start without any error.
+
+This spins up a GitLab Docker container based on the given edition or image:
+
+- `gitlab/gitlab-ce:nightly` for `CE`
+- `gitlab/gitlab-ee:nightly` for `EE`
+- the given custom image for `<full image address>`
+
+Example:
+
+```
+$ gitlab-qa Test::Omnibus::Image CE
+```
+
+### `Test::Omnibus::Update CE|EE|<full image address>`
+
+This tests that:
+
+- the GitLab Docker `latest` container works as expected by running
+  instance-level tests against it (see `Test::Instance::Image` above)
+- it can be updated to a new (`nightly` or `<full image address>`) container
+- the new GitLab container still works as expected by running
+  `Test::Instance::Image` against it
+
+Example:
+
+```
+# Update from gitlab/gitlab-ce:latest to gitlab/gitlab-ce:nightly
+$ gitlab-qa Test::Omnibus::Update CE
+
+# Update from gitlab/gitlab-ee:latest to gitlab/gitlab-ee:nightly
+$ gitlab-qa Test::Omnibus::Update EE
+
+# Update from gitlab/gitlab-ce:latest to gitlab/gitlab-ce:my-custom-tag
+$ gitlab-qa Test::Omnibus::Update gitlab/gitlab-ce:my-custom-tag
+```
+
+### `Test::Omnibus::Upgrade CE|<full image address>`
+
+This tests that:
+
+- the GitLab Docker container works as expected by running instance-level tests
+  against it (see `Test::Instance::Image` above)
+- it can be upgraded to a corresponding EE container
+- the new GitLab container still works as expected by running
+  `Test::Instance::Image` against it
+
+Example:
+
+```
+# Ugrade from gitlab/gitlab-ce:nightly to gitlab/gitlab-ee:nightly
+$ gitlab-qa Test::Omnibus::Upgrade CE
+
+# Ugrade from gitlab/gitlab-ce:my-custom-tag to gitlab/gitlab-ee:my-custom-tag
+$ gitlab-qa Test::Omnibus::Upgrade gitlab/gitlab-ce:my-custom-tag
+```
+
+### `Test::Integration::Geo EE|<full image address>`
+
+This tests that two GitLab Geo instances work as expected.
+
+The scenario spins up a primary and secondary GitLab Geo nodes, and verifies
+that the replications (repository, attachments, project rename etc.) work as
+expected.
+
+To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
+container is spun up and tests are run from it by running the
+`QA::EE::Scenario::Test::Geo` scenario (located under
+[`gitlab-org/gitlab-ee@qa/qa/ee/scenario/test/geo.rb`][test-geo] in the GitLab
+EE project).
+
+**Required environment variables:**
+
+- `EE_LICENSE`: A valid EE license.
+
+Example:
+
+```
+$ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)
+$ gitlab-qa Test::Integration::Geo EE
+```
+
+[test-geo]: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/ee/scenario/test/geo.rb
+
+### `Test::Integration::LDAP CE|EE|<full image address>`
+
+This tests that a GitLab instance works as expected with an external
+LDAP server.
+
+The scenario spins up an OpenLDAP server, seeds users, and verifies
+that LDAP-related features work as expected.
+
+To run tests against the GitLab containers, a GitLab QA (`gitlab/gitlab-qa`)
+container is spun up and tests are run from it by running the
+`Test::Integration::LDAP` scenario (located under
+[`gitlab-org/gitlab-ce@qa/qa/scenario/test/integration/ldap.rb`][test-integration-ldap]
+in the GitLab CE project).
+
+In EE, both the GitLab standard and LDAP credentials are needed:
+
+1. The first is used to login as an Admin to enter in the EE license.
+2. The second is used to conduct LDAP-related tasks
+
+**Required environment variables:**
+
+- [For EE only] `EE_LICENSE`: A valid EE license.
+
+Example:
+
+```
+$ gitlab-qa Test::Integration::LDAP CE
+
+# For EE
+$ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)
+
+$ gitlab-qa Test::Integration::LDAP EE
+```
+
+[test-integration-ldap]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/qa/qa/scenario/test/integration/ldap.rb
+
+### `Test::Integration::Mattermost CE|EE|<full image address>`
+
+This tests that a GitLab instance works as expected when enabling the embedded
+Mattermost server (see `Test::Instance::Image` above).
+
+To run tests against the GitLab container, a GitLab QA (`gitlab/gitlab-qa`)
+container is spun up and tests are run from it by running the
+`Test::Integration::Mattermost` scenario (located under
+[`gitlab-org/gitlab-ce@qa/qa/scenario/test/integration/mattermost.rb`][test-integration-mattermost]
+in the GitLab CE project).
+
+**Required environment variables:**
+
+- [For EE only] `EE_LICENSE`: A valid EE license.
+
+Example:
+
+```
+$ gitlab-qa Test::Integration::Mattermost CE
+
+# For EE
+$ export EE_LICENSE=$(cat /path/to/Geo.gitlab_license)
+
+$ gitlab-qa Test::Integration::Mattermost EE
+```
+
+[test-integration-mattermost]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/qa/qa/scenario/test/integration/mattermost.rb
+
+### `Test::Instance::Any CE|EE|<full image address>:nightly|latest|any_tag http://your.instance.gitlab`
+
+This tests that a live GitLab instance works as expected by running tests
+against it.
+
+To run tests against the GitLab instance, a GitLab QA (`gitlab/gitlab-qa`)
+container is spun up and tests are run from it by running the
+`Test::Instance` scenario (located under
+[`gitlab-org/gitlab-ce@qa/qa/scenario/test/instance.rb`][test-instance] in the
+in the GitLab CE project).
+
+Example:
+
+```
+$ export GITLAB_USERNAME=your_username
+$ export GITLAB_PASSWORD=your_password
+
+# Runs the QA suite for an instance running GitLab CE 10.8.1
+$ gitlab-qa Test::Instance::Any EE:10.8.1-ce https://your.instance.gitlab
+
+# Runs the QA suite for an instance running GitLab EE 10.7.3
+$ gitlab-qa Test::Instance::Any EE:10.7.3-ee https://your.instance.gitlab
+```
+
+### `Test::Instance::Staging`
+
+This scenario tests that the [`staging.gitlab.com`](https://staging.gitlab.com)
+works as expected by running tests against it.
+
+To run tests against the GitLab instance, a GitLab QA (`gitlab/gitlab-qa`)
+container is spun up and tests are run from it by running the
+`Test::Instance` scenario (located under
+[`gitlab-org/gitlab-ce@qa/qa/scenario/test/instance.rb`][test-instance] in the
+in the GitLab CE project).
+
+**Required environment variables:**
+
+- `GITLAB_QA_ACCESS_TOKEN`: A valid personal access token with the `api` scope.
+  This is used to retrieve the version that staging is currently running.
+
+Example:
+
+```
+$ export GITLAB_QA_ACCESS_TOKEN=your_api_access_token
+$ export GITLAB_USERNAME="gitlab-qa"
+$ export GITLAB_PASSWORD="$GITLAB_QA_PASSWORD"
+
+$ gitlab-qa Test::Instance::Staging
+```
+
+[test-instance]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/qa/qa/scenario/test/instance.rb
+
+----
+
+[Back to README.md](../README.md)

data/lib/gitlab/qa/component/specs.rb

@@ -17,7 +17,7 @@ module Gitlab
 
           puts "Running test suite `#{suite}` for #{release.project_name}"
 
-          @docker.run(release.qa_image, release.tag, suite, *args) do |command|
+          @docker.run(release.qa_image, release.qa_tag, suite, *args) do |command|
             command << "-t --rm --net=#{network || 'bridge'}"
 
             variables = Runtime::Env.variables

data/lib/gitlab/qa/release.rb

@@ -68,8 +68,8 @@ module Gitlab
           end
       end
 
-      def edition_tag
-        @edition_tag ||= "#{edition}-#{tag}"
+      def qa_tag
+        tag.sub(/\.([ce]e)/, '-\1').sub(/\.(\d+)\z/, '')
       end
 
       private

data/lib/gitlab/qa/scenario/test/instance/any.rb

@@ -8,15 +8,11 @@ module Gitlab
           # including staging and on-premises installation.
           #
           class Any < Scenario::Template
-            def perform(edition, tag, address)
-              release = Release.new(edition).tap do |r|
-                r.tag = tag
-              end
-
+            def perform(edition_and_tag, address, *rspec_args)
               Component::Specs.perform do |specs|
                 specs.suite = 'Test::Instance'
-                specs.release = release
-                specs.args = [address]
+                specs.release = Release.new(edition_and_tag)
+                specs.args = [address, *rspec_args]
               end
             end
           end

data/lib/gitlab/qa/scenario/test/integration/geo.rb

@@ -27,6 +27,7 @@ module Gitlab
                   postgresql['md5_auth_cidr_addresses'] = ['0.0.0.0/0'];
                   postgresql['max_replication_slots'] = 1;
                   gitlab_rails['db_key_base'] = '4dd58204865eb41bca93bd38131d51cc';
+                  gitlab_rails['monitoring_whitelist'] = ['0.0.0.0/0'];
                   sidekiq['concurrency'] = 2;
                   unicorn['worker_processes'] = 2;
                 OMNIBUS
@@ -41,6 +42,7 @@ module Gitlab
                       gitlab_rails['db_key_base'] = '4dd58204865eb41bca93bd38131d51cc';
                       sidekiq['concurrency'] = 2;
                       unicorn['worker_processes'] = 2;
+                      gitlab_rails['monitoring_whitelist'] = ['0.0.0.0/0'];
                     OMNIBUS
 
                     secondary.act do

data/lib/gitlab/qa/scenario/test/omnibus/update.rb

@@ -7,12 +7,15 @@ module Gitlab
       module Test
         module Omnibus
           class Update < Scenario::Template
-            def perform(next_release)
-              next_release = Release.new(next_release)
+            def perform(previous_edition, next_edition = nil)
+              next_edition ||= previous_edition
+
+              previous_release = Release.new(previous_edition).previous_stable
+              next_release = Release.new(next_edition)
 
               Docker::Volumes.new.with_temporary_volumes do |volumes|
                 Scenario::Test::Instance::Image
-                  .perform(next_release.previous_stable) do |scenario|
+                  .perform(previous_release) do |scenario|
                   scenario.volumes = volumes
                 end
 

data/lib/gitlab/qa/version.rb

@@ -1,5 +1,5 @@
 module Gitlab
   module QA
-    VERSION = '0.6.0'.freeze
+    VERSION = '1.0.0'.freeze
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: gitlab-qa
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 1.0.0
 platform: ruby
 authors:
 - Grzegorz Bizon
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2018-06-11 00:00:00.000000000 Z
+date: 2018-06-29 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: gitlab-styles
@@ -118,8 +118,10 @@ files:
 - bin/setup
 - docs/README.md
 - docs/architecture.md
-- docs/macos.md
+- docs/how_it_works.md
+- docs/run_geo_qa_against_gdk.md
 - docs/trainings.md
+- docs/what_tests_can_be_run.md
 - exe/gitlab-qa
 - fixtures/ldap/tanuki.ldif
 - gitlab-qa.gemspec
@@ -167,7 +169,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.7.7
+rubygems_version: 2.5.2
 signing_key: 
 specification_version: 4
 summary: Integration tests for GitLab