GitLab Quality Department, 3 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Orchestrated end-to-end tests at GitLab - Part 1

Description

In this video, GitLab Senior Software Engineer in Test Sanad Liaquat talks about Orchestrated end-to-end tests and how to run them. He also discusses the code briefly especially the Test Scenarios, the Components and how gitlab-qa calls the Test Scenarios in the /qa directory of the GitLab project.

Deck: https://docs.google.com/presentation/d/15F7kUyHxGuudJoMzqytzw5L9ApiyzQ5aQ0apId7dVo8/edit?usp=sharing

This is the first video in a two part series. The second video can be found here: https://youtu.be/U8zCle_Up6I

A

Hi, my name is sanad, I'm a software engineer in test at gitlab. In this video I talk about orchestrated tests and how to run them. This is a first video in a two-part series where we'll be talking about orchestrated tests and in the second video I'm going to walk you through writing an actual orchestrated entrant test scenario.

A

So, let's get started so I've made some assumptions over here uh while recording this video. um I am assuming that the viewers have some knowledge about the gitlab qa framework and I'm not going to talk about the basics, also um I'm assuming that there's some understanding about how to write end-to-end tests. However, if you came here looking for that information, I have some links for you for where to get started.

A

uh This is the first link I have over here and there's also a link I've put down for a step-by-step tutorial uh for getting you up to speed with writing and doing this.

A

Now, let's talk about what orchestration actually means- and I have a snapshot over here- that I've taken from google.

A

I, like the second definition where it says that orchestration is coordination of elements of situation to produce to produce a desired effect in context of entering tests at gitlab. These elements are docker containers that run different applications um like say, saml, ldap or or or gitlab itself.

A

We have a way of representing these elements in getlab qa project and we represent them with with a component. We have a component class. Each component class represents these elements that talk to each other when, when the tests are run, so a component may or may not have an associated docker container so, for example, the staging component, it only runs a test against the staging environment.

A

uh So if the staging environment is a live environment, so we don't have to spin up a gitlab inside a container for for that, it does not require a container. However, any orchestrated tests, any components using orchestrated tests do spin up one or the other docker containers.

A

I wanted to show some of the existing components we have in gitlab qa project.

A

If we go to lib, gitlab, qa, sorry, gitlab, qa and then the component directory you'll see a bunch of components over here.

A

Let's look at one of the simpler components: male hog, this component is responsible for pulling the mailhawk service and running it inside a docker container.

A

The most interesting method over here that I want to talk about is the instance method. What it does is it inside prepare it pulls the docker image and creates the network if it's not already created um and then starts the docker container right, there's also tear down over here, which stops the stop the docker container and removes it.

A

Let's also look at the git lab gitlab component, and this, uh as you would already have guesses responsible for uh running the gitlab application inside a docker container.

A

um Again, let's look at the instance method here, there's prepare where uh this case we set up uh disable animations by by setting a anonymous bus config, and then it pulls the git lab uh image.

A

That's provided for uh with the tag and also creates a network if it's not already created uh reconfigures waits for the instance to become up and then then, once uh the instance is up.

A

If you need to run any commands inside the docker container. That's where this uh this is. This happens over here uh again, there's a tier down where it would stop and then remove the container.

A

Next, I want to talk about test scenario. A test scenario in gitlab qa project is where the components come and work together at a very minimum. A test scenario would have just the specs component. The specs component is actually responsible for running the test, suite from the git from the qa directory in the gitlab project.

A

uh You need to provide it with the target environment where you want to run the run the test, and it would kick off with us against that environment, and you would also need to provide it with with a scenario that you need to run. There are basically three types of test scenarios. The most basic one would have just one component, uh and that would be the spec specs component.

A

uh The next level is where we would have two components, and then we can also have three or more components we'll be looking at an example for each of these three types of components. Next so test instance, any is our example of the most basic kind of a test scenario.

A

It just has one component there's the specs component: we need to provide it with a suite in this case. We're providing it with test instance. All this is a test scenario that exists in the qa directory inside the gitlab gitlab project.

A

uh We also need to provide it with a release which it will use to pull the appropriate qa image and then create a container from that queue image and run the tests against the address that we provide here. We can also provide our spec arguments, so this is an example of how the command would look like.

A

uh One thing I wanted to point out here is that uh here we are set providing it with ee release, and this would be translated. This will be used to pull the gitlab nightly nightly qa image.

A

Next, let's talk about the architecture diagram.

A

I've pulled it from the documents in gitlab qa project, the wikitlab qa gm will spin up a container using the qa image, and that container would in turn, run the qa scenario that we provided with against the live gitlab instance again. We provided that gitlab instance in the command. Here, let's talk about it, a scenario with two components: an example of such a such a test scenario is test instance image. It's made up of specs component and the gitlab component, and we need to provide it with a git lab release and based on that release.

A

It will pull the get lab image and the gitlab qa image and create docker containers. Out of those. I have a link over here with an example of where test instance image is running on ci.

A

If you look at the code, the gitlab component needs a release, a volume and a network, and this network is shared with network and the release are shared with the specs component. We also need to provide the specs component with with the suite again as previously discussed this week is basically a test scenario inside the qa directory in gitlab project. You would notice that, in contrast to the test instance, any component where we provided the address from the command line over here, the address is provided from the from the gitlab component.

A

So this means that the gitlab component would spin up a docker container running gitlab inside it, and the address would be used for running the tests against that, against that instance, inside the docker container.

A

Let's look at the architecture. The gitlab qa gem would spin up two containers, one based on the gitlab qa image and the other, based on the gitlab image, where the gitlab application would be running. The gitlab qa container would run the qa scenarios against the gitlab instance inside the gitlab container for a test scenario. Example that has three components we'll be looking at test: integration smtp it's made of made up of three components: the specs component, the gitlab component and the email hog.

A

We need to provide it with a release and it would use that release to pull the appropriate qa image the gitlab image and create container containers out of those. It will also be pulling the mail hog image.

A

Let's look at the code so here inside perform. We are spinning up the gitlab docker container and then we are also spinning up the mail hog container and and with the specs component, we are running the tests against the gitlab component.

A

Now, a point to note here is that we are setting certain omnibus configuration for gitlab container, and this is one of the reasons why we have orchestrated tests. It allows us to spin up gitlab with certain instance level settings that we cannot change when we when we are running tests against a live environment such as staging.

A

So we are doing that right here and I'll show you another example later on where we changed the omnibus settings. But but the point I wanted to make here is that this is why orchestrated tests could not be run against live environments.

A

Another thing to note here is the spec suite. We have it's specified as test integration smtp, as opposed to test instance, all in our in all our previous examples.

A

So this means that the tests would call test integration, smtp, test scenario or or the suite inside, of the qa directory in the gitlab code base the architecture here for uh the scenario with three components: it's pretty much self-explanatory.

A

um Let's keep going through it when to write orchestrated tests uh whenever we need instance level settings such as in this example. I have here where, where I am passing omnibus config for setting up github omniock, we will need orchestration also when we need to test integration with external services such as male hog, ldaps ml or elastic elastic search or when, whenever you need multiple instances of gitlab such as geo, you know that you have you'd, be writing orchestrated tests.

A

I also wanted to mention that whenever we add the orchestrated tag in tess, it's just a way to tell our spec that you need to filter those tests out when running against live environments such as staging canary or protection.

A

The reason behind that is that we cannot run orchestrated tests against those environments against live environments all right. Let's run a test scenario with three components:.

A

So this this is an orchestrated test scenario, I'll copy the command here and run it on my terminal.

A

You can see that it's trying to pull the mail hog image, but since I already had that on my local, it started it up in a container.

A

And then the next thing it did was it tried to pull the gitlab nightly image. I also had that on my local, so I just started the container again.

A

We are now waiting for gitlab to become available.

A

We're now running the tests.

A

This is actually executed by these spec.

A

Components there you go, it does fast and we are tearing down the all the containers.

A

If you look at the docker run command that was executed by the spec component, you would notice that we are passing a test scenario onto the gitlab qa image, as well as a target. A gitlab instance run the test against.

A

Let's first look at the docker file that was used to create this. um This image, this gitlab qa image uh that file is in qa directory, so we are in uh the gitlab project, we're going to the qa directory and there's a docker file here. The thing that I want to point out here is the entry point: if you notice that it has been test, so this docker file bundles the entire qa directory into the image and then the entry point point provided has been test. Let's look at bin test inside qa directory.

A

There you go so what bin test does is that it executes the uh the qa script and passes all the arguments to it. What does the qa script do? It uses the first argument passed to it, converts it into a constant and calls it on the qa scenario module.

A

What was the first argument that was passed here? It was the test scenario test integration smtp.

A

Let's look at where that scenario is, it should be under qaqa scenario, test integration, smtp, and what we see here is that the test integration, smtp scenario- is actually a test instance all scenario, except that it is passing a tag, the smtp tag here, so that would mean it would just run all the tests that has the smtp meta on them.

A

Let's search for it.

A

So there's just one test that has the smtp meta, and that is the exact test that we ran right here.

A

That's it for this video in the next video I'm going to walk through the steps of writing into an orchestrated test, take care.