GitLab GitLab Runner Open Office Hours, 15 Apr 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitLab Runner Open Office Hours - April 15th 2020

Description

Senior Backend Enginer, Tomasz, reviews and MR from our wider community that is attempting to fix a bug with how the runner handles autoscaled VMs when using the docker-machine executor

The MR reviewed is https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1549

A

So hello, everybody good morning, good afternoon, good evening, whatever time zone you're in this is the gate, lab runner, open office hours or Wednesday April 15th in those time zones.

A

Our goal here is really just to kind of shine a bit of light on like what we do when we go through a community contribution for the runner project or related projects.

A

If there's any questions, if anyone has any questions about open, em, arms or open issues or and they on grab for, we can prioritize talk about those. Otherwise, what we'll do is we'll have one of our back-end engineers go through a code review of some existing open hours, so at any point neither be chat or speak up, you're, comfortable and and asking questions as you'll note from the recording? No, it's like what this is recorded, we'll post it to our YouTube channel later on once the recordings process.

A

So on the call there's myself I'm the engineering manager for the runner team. We have Steve Tomas, Pedro and Luke Amir, who are back-end engineers on the runner, team and Ray part of our community relations game, and it looks like we've got a couple key members on the call, and so hello welcome, feel free to go in and jump in with any questions you have at any point, and otherwise, unless we're you have anything.

B

A

No cool twice when you get started.

C

Okay, so I was trying to find some some interesting requests that we could review.

C

Our so that we had a discussion in the in the issue about the mesh request, number 2000.

D

C

We have the community contributor with us already I.

A

Don't think so: okay.

C

So maybe let's weigh this one.

A

I'm, just guessing because based on a username but I, think the username is a real name. But.

C

Okay, let me just find.

C

Proper sculpture in zoom, oh I, think this is the one okay I I was looking through. Some met, requests that we have created from the contributors and one of the metrics that they found is related to dr. machine, not scaling and the proposition to change how this is done, which is which is related to one of the one of the problems that we've seen in the past.

C

So, let's, let's maybe take a look, what we have in this Englishman request.

C

Okay, get LeBron, tries to create one machine crates, a lot of machines that was because acquire cause, multiple items in short period of time and machine created by previous calls are not immediately shown visuals and some of next course not properly account number of creating machines. So we are, we are facing here some race condition, problem.

C

Okay, what we, what we have here this is this is quite big file. Let me take a big look on what we have in this file at all documents in provide there.

C

This is this is the main file that implements.

A

C

C

And we have the update machines, methods, ok and what change we have here.

C

The details freight.

C

We are there in the details in the data structure, which is machines, data, ok- and here we are doing call log- why we need to do this lock. Well, we do operate here and details.

C

Okay, machine detailed structure, machine detailed structure, which itself is implemented that way it was implemented.

C

It will be easier for me if I will switch my local code. So let me let me pull the changes from the contributor.

A

There's an excellent terminal background.

C

Okay and branch name is fix, multiplier.

C

Okay, we will have to change. Luckily executors, it's my ID right now. Leo. Are you still? Okay, let's close, what's not important right now, executors poker machine here we can see how much things we have inside of the doctor executors, and for that we need to look on the provider and provider first five-star, the only one.

D

C

It so that's what's well, let's: first, what was change and then we think about. If the change is like it depend what we could improve in it. Here we have the line.

C

What I did I thought? The changes from the contributor of locally.

A

C

I I fetched the remote branch, I added a new remote branch from the from the fork and and just switch it to the branch of this user. So I have now.

A

C

No, it was, it was only to get the source locally, because now in the ID I can I can easily navigate through the code and.

A

C

Don't necessarily need to remember well where everything is created. Okay, so at this moment, I can already say that that this is. This is definitely something that we need to need to change from now. I, don't think we have. We have the code directly fused anywhere in the code anyway, the best we just state without it.

C

Let's try to. Let's try to understand what what the problem we have here in the in the meticulous description we have. We have information that, because of some some time, concurrency problems, the machines that we accounting are not always counted properly and, let's first look we're update machines called job. The Pershing's is called inside.

C

Acquire acquire is called by the club runner before we ask for a job when, when runner finds that within the concurrent and limit settings it has some free slot, it first asks the executor with the acquire call if there is really the capacity on the executor. So in case of doctor machine we can have two options. We either have machines treatment on demand and, in that case, acquire almost always return positive response.

C

Unless we hit the total limit of machines, if we have idle machines setting configure, then the machines are created in the background, and this call will try to figure out if we have any other idle machines waiting for a new job. If we have something like that, and I acquired returns, positive positive response, we we didn't know that we can really ask guitar for a new job, because we have a capacity for this so going quickly through what we have here.

C

A lot machines from what I remember it rates over all of the machines that keep up runner is managing at this moment. No matter it if if there are used ID ones- and here we are calling update machines.

C

Machines machine is data. This is one of the things that we started recently. Refactoring and big refactor for this file is waiting good comments, describing what is happening where from now, let's just work with what we have, okay, so in the update machines, the machine slice is the result of what machine so from what I remember, all machines applicable to certain run or worker in conflict. We have config, ok and what we want to get from. It is the machines data structure.

C

The machine is data which implements some counting, and this is this is what gives us information. How many machines in different states will have for this specific runner, walker? My runner Walker here I mean the thing that we in conflict on file, recognized by the two prices and runners directory, which the fiends defines the bulimia. The connection to get lab lot. Executor is used and in context of this, this met up done, operates.

C

Okay, so sure description is it's just the first few charges from the token here we create an.

D

C

Slice- and we now start iterating over the machine that we've got, we get machine details for each of them. Machine details is taken from that DJ's field. Ok, we mark when it was last seen.

C

Yes, this is, this is the place where we decide whether we should remove a machine or or not, and it is updated. No data is not updated with this place, but if we have the machine news too many times, we did remove it from the machines that we tracked with the details.

C

Okay and finally, we add the details of the machine to the data structure and at this moment, in current implementation, this is all yes, we we don't have this currently and what not the contributor proposes is: firstly, lock to set the lock on the structure, so we can block Inc operation and then we iterate over all details.

C

Then, over all machines.

C

And if the name of the machine is.

C

We exit from it.

C

They continue to exit this loop and continue their the for loop outside of it. So we found we found that the.

C

Details object taken from here is.

C

It's within the machines and then we try to add it again. hmm This really strange part of the code. Let's maybe take a quick look on the test. Maybe the test will be some light. Was was mistrust to resolve.

C

Okay, here is some creation, latency.

C

Number 100.

C

Test machine, okay, test machine.

C

This mocks the doctor machine interface, okay, so this will be the time of the create operation, but nobody will be taken by doctor machine create call, and the next changes are lines in 77.

C

That's not, it will acquire calls and create.

C

You have a test machine provider.

C

Create machine config title count one idle time: five.

C

We try to acquire the machine five times, and this operation should be done very quickly.

C

We expect that we don't have no free machines at this moment and when the machine is created which happens here.

C

C

C

It's the number of all the machines in the idle State. Okay,.

C

And when we start the creation.

C

Machines provider.

C

C

We honest I'm, not sure what exactly the the code here tries to achieve.

C

Yeah, like I, have a feeling that I understand what is the problem. However, I can't see how this how this magic was is fixing it. The test is not fully showing me.

C

What we try to assert here so at this moment at this moment, what we, what we definitely can do with this miss request, is first, we need to ask need to ask for a way to read description of how we can reproduce this work in few steps, step by step. What is happening because there is a clearly have a problem, and he already knows what is the problem, but the description here is not as clear to understand what we should what we should look on.

C

The second thing is that that this construction should be definitely changed. First, is that the usage of the lobe gold label? It's definitely not what we should do. This should be decomposed to to a separate function. Call. This is how we resolve this in all other places.

C

The second thing that I'm not not fully understanding here, is that we call we call that data here twice.

C

First time getting the machine by name from the details, map.

C

It also locks everything so should be a little safer. The concurrency.

C

If we don't have such information for the machine created if acquire, if require what we do with the choir flag here, we don't use a choir flag here, so this is not used at all, so basically using the machine name from the from the sliced path here, and this has the list of machine names hanging by by a specific runner read from the disk for each of such machines.

C

We are either getting the details or creating the base details where the base Diesel's have the time set to now that the idle state and it's the first usage of the machine.

C

And then, then we are passing this to the update machine and I'm here missing somewhere the place where the machines are in fact created, because this is. This is the place that we are looking for.

C

Anyway, if our big machines.

C

Doesn't return arose an error we treated as valid machines, and this slice is not updated later by the code here and if a bleep machine returns some error, so we have too many builds for the machine, but we have too many machines.

C

Don't have too many idle machines. The machine that we now try to handle is is being removed from the list. So at this moment we are removing it from the list of machines handled by the run, but we are adding information about this machine to the counter. At this moment it should have the state, yes removing.

C

So basically, at this moment we say that the machine is idle or in one of the creating states or in the remove state, and then we again iterate over the details again trying to match this with the machines that we did here and then add some of them like.

C

This is not changed because this was passed by the function argument, so we have neither stable list of the machine names. What can be changed is this one, because this could get a new created machine, so it looks like it tries to find a machine that is already on this list, but was not here at the call of this moment, which is again.

C

At least for now, for me, it looks like just waiting for luck that it was not here, but a magnetic moment later, it will be here, but we we are still trying to fight here with some risk conditions, so may be possible that something is not yet created here and will be created after we will remove. We will exit from this machine. This lock will be unlocked.

C

This noose definitely needs some some bigger explanation, so lets me. Let me now try to compose all of this into some questions that we can ask the user.

C

Yeah is there anything else, I should write here.

D

C

Think not understanding what is the problem, the exact problem and fixing. This are the only things that this moment I can I can talk about having having the information, how we can reproduce the problem or how it works. We can. We can then start so looking on what exactly this exchange, try stretching.

B

Maybe the conductor t-shirts related to maybe we can get some.

C

More information, good comment, okay, but I think I think this is the. This is the issue that we we already known from our past discussions that docker machine is just creating a lot of machines at the start, even if the idle count set.

C

Should do a minimal value? Okay, we have some walks here.

C

C

We have conflict on this.

C

Okay, it's not a public installation.

C

You may think I will deduct their tokens in the moment. I do count one idle time: 14400 of the cattle count zero.

C

Okay, so runners should create one machine, nothing more, and we see here that.

C

When you procreate checks, 1 2, 3, 5, 6,.

C

B

C

Of much is created can.

B

This be happening, so the machine is being created up free time. We call a choir right.

C

Not fully, maybe let me let me find the magic was where I was refactoring this, because there we had descriptions. What and when is happening.

C

C

Provider a lot of changes.

C

C

Not machine names- and here we have this redundant machines. This is what was previously done with.

C

No just not their function here.

C

You you choir is calling here update machines, I, think this is what I hear renamed to the.

C

Remove redundant machines and, let's find where is it, remove redundant machines one here? It is.

C

Get machine details is machine, redundant, schedule machine. We know how so at this place here in the update machine is called, which next is which next is handled here. We are not handing creation of machines at all. We are just looking for machines that are not needed anymore because of too many machines too many area machines too many jobs executed on the machine. So this part of code doesn't touch creation at all.

B

C

The creation like this is really hard to find out what is happening where.

C

Because a choir require where we have the choir call here. This is the code that we that we use now here we are here. We are calling create machines which I did not fully sureit's responsible for really creating the machines. I remember there was something strange in the in the calls here, but let's look: what the create machines does.

C

B

C

B

I think is happening yes, so we call acquire every time the runner tries to fetch a job right. Yes,.

C

B

This create machine is just marking my sink machines as needed, right and memory, and, like imagine that the interval is every 5, what every 1. Second right, so a 5 seconds, we ask cudlip api for a job every 1 second, so in that case we end up calling acquire 5 times and 5 seconds and we end up creating an idle machine per acquire. Imagining that the idle machine sucks to 1. So we end up creating 5 machines and a span of 5 seconds, because the first machine hasn't been hadn't finished. Yet.

C

Mm-Hmm, okay, so us for calling the acquire it's called. It's called here process runner is being called here.

C

Which itself acts acts on the runners.

C

Which is being.

C

Triggered here, so the request interval that we have is divided by the number that both of runners sections, that we have in the config file and in each such interval. We are scheduling one call, but should, at the end finish with requesting the job which is here, but before we do this, we check if we have even the capacity and I'm sure not not not here, not request quest rest request request.

C

This is this is dumb down here we first check if we have capacity on the provider and then we try to fetch the job from the plug. So yes, this call is executed. Each check interval divided by the number of runs so going down from from that call, we have the provider require, call which, in case of doctor, machine was closest files in case of doctor machine ends here. So if we don't have the cream machine configuration, we don't do nothing and then for each acquire call.

C

We call the load machines, the load machines, uses machines list and machines list. Remember just this one implementation, it reads the configuration directory and lists all of them. Machine directories that we have because then the directors have exactly the same names as as the machines, so here we get all of the machines existing on this host, and then we are using filter. Machine list function to limit this, to only machines belonging to this Runner section.

C

So at this moment we have the list of machines existing on the disk. This may be machine that is running. This may be machine that is I.

C

Do this may be machine that is already being destroyed or is stuck in removing, or this is machine that is being created, so something's already triggered the machine creation and from some moment it's already visible on the disk, even if it's not yet created in the cloud provider then having this list, we are calling this update machines method where this change is done, and here we are comparing the list taken from the disk with what we have in the machine details, structure in the map and.

C

For the machines that we.

C

For the machine is that, let's, let's think what we what we can have here, we we can get here a machine if it was all the creation was already triggered by.

C

By rather then, at some moment it should be added to the end details. So what we really need to find is is when, and details is being is being fitted, because this is how the Machine creation is ended. So, let's get back from this call here, we call create machines, and this should be done only when the data, so what we, what we got from here, data available is greater or equal of get idle count. So we have already the number of idle machines that we want.

C

We don't create a new machine we exit from this. If the total number of machines is bigger, equal of the limit and the limit itself is not zero, we also exited. In other case, we call the create function.

C

Which then, should somewhere called machine provider machine create machine provisional fails. So at this moment we trigger creation of the machine which is done here and the data machines data structure is the only thing that that can stop us from from doing and then we are logging, logging machine details and we are looking if we have any free machine. We should just check if we have any machine in the idle side and at some place check the state required, so it is reserved for the rather.

C

C

If we have zero idle machines and idle count is non zero, then it means that we don't have don't replace two to execute the job, and this is where the acquire also prevents this. So so, basically, what we are interested in is create machine data available and data to eval. If these numbers are not updated, then it means that we will try to create a new machine.

C

And how the how the change of both could could affect this.

C

Create when the machine is created, where is the details updated? Because everything happens here around the details.

C

Details name equal to delete iteration, iteration.

C

Requesting iteration, so it looks like this is the only place.

C

This is the only place where the M details is updated. I don't see, I, don't see any other any other write to this to this field, because here we have initialization that that creates the create structure and everything. But this one reads from so in details is updated only when we already found a machine on this machine directory and.

C

We call this function.

C

Which is being done in three places.

C

First, one is create, so when we create machine we immediately added to details.

C

Second, one is find free machine.

C

The third one is update machine which is being updated here.

C

Okay, without without drying down growing down how this happens, step-by-step I still don't see how this how this can be resolved the problem, because, even even if we, if we a.

C

Big machine is, is called acquire just after weekly, we read the list of machines from disk, but before we schedule a machine creation, so.

C

With this, this slice is not being updated.

C

C

C

C

A

Just wanted taken all the time. We only have a few minutes left.

C

Mm-Hmm, okay, I will I will redact the comment that he was writing. I, I think I start seeing the problem, however, still having having the help from the customer that sees this life would be really helpful, especially that that that the user wrote that the fix works for his situation and it works. So why I'm not saying fully how how this, how this helps us?

C

It's it's it's being seen in reality, so I will I will I will just update it's a little and and follow follow with this much request. After after all, I.

B

Think this is a perfect example from what we experienced right, like it's just a 5 line change, but if I find change result, it is Oh an hour trying to figure out what's going on so even.

D

B

The change is small. It still does take some time to review, especially when it's part of the tricky parts base.

A

Yeah, it's a great point.

A

Thanks to peer around applause.

A

So ray you, I can't respond to you in the chat for some reason: I can't click on.

A

D

Works out, like the next office hour in May, we have a hackathon also on the 13th and 14th next months. We can coincide at what the hackathon, if that works for people, or we can find another time just a suggestion.

A

I think it works I think during the 13th at the same kind of time, McCourt could be a Wednesday again so yep put that in the calendar, I'll create a new issue, cool.

D

A

Thanks so much yeah.

D

Thanks everybody joining.