GitLab Configure Stage, 2 Nov 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitLab Kubernetes Agent: TunnelRegistry deadlock fix

Description

Discussing the fix for https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues/183.

MR https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/merge_requests/557

A

um Hello, my name is mikhail I'd like to talk about this issue that I fixed recently so you're, seeing timeouts in this functionality where cuba ctl couldn't get reply from the server from cars in 32 seconds or at all yeah. So.

A

Before bumping the deployment to a new version- graham, thank you graham, I took uh the select core routine stack, trace, dump using prof and that helped us a lot to understand. What's you know what's wrong, so you can see this stuff traces here and then I open the merge request, fixing the the deadlock so yeah it turned out. It was a deadlock, and so I want to talk about how I found the problem and how to read stack traces and the how tunnel register works in general to start with.

A

So, first of all, let's open the code and when you are looking at stack traces, the first thing you always need to do is to open the correct version of the code. Otherwise line numbers, don't you know don't align quite often so, especially for a fast moving project or when you update dependencies.

A

So this was uh htc one 14, 5 or c1. So let's check out that version.

A

This um c2 is the version with the fix. This is the version that was buggy and so.

A

Basically, let's first talk about how the tunnel register works.

A

It lives here in the reverse tunnel module in the route because it could be used by something else, not necessarily the server or the agent. It is currently only used by the server but yeah. If we decide to do something else someday, it might be useful for something I.

A

So how does it work tunnel registry implements two interfaces: tana finder find tunnel. This is the method that is called by the cast that needs to find the tunnel to an agent, and the second interface is tunnel handler, which is also called by the cast, but the one that accepted the connection from an agent.

A

So first we register the connection, basically incoming connection from an agent and then some other connection from another cast will try to find the correct tunnel to forward the request to so. This is the the two interfaces that panel registry implements and the way it works is basically, um if you know, if you heard of actor model or actors, then you should understand how what it means how it works.

A

The idea is that you have an independent well, what's called an actor that owns some state and then it processes incoming messages and sends out outgoing messages.

A

So an alternative approach is to have a piece of shared state and then whoever needs to do something with it read or write uh grabs the lock performs the mutations or whatever it is to do, read, write both then releases. The lock and then some some other thread may come come along to do something else and yeah. So with actors, the actor owns the state and then it's a gore routine.

A

That just runs and listens for incoming messages and when it gets a message over a channel, it does something depending on message usually and then it may send out messages or you know, depends on what one wants to so. The difference is, is there are no locks? There is a dedicated go routine and it interacts with the outside world with other goroutines via channels.

A

I like this pattern, a lot because it allows you to.

A

Use contact says they just compose nicer with the blocking cause blocking calls all the channels you can mix in periodic um things that need to happen uh within that uh go routine like you can have a timer that fires every end. Second, to do some garbage collection or some background activity. Even if there are no colors. If you do this via locks, then you need to have a separate.

A

You know thread or go routine, that wakes up every n seconds, grab, grabs local and does whatever whatever it needs to do. Yeah and with with actor you just have a loop. This run runs in separate, go routine it as you can see it.

A

Reads from channels these are all channels. You can see that here we have four channels.

A

We select on all four channels and also we have the context where you know we are taught by the thing that started us that we can stop. We need to stop.

A

If we, if we needed to do some garbage collection, we would have a ticker here that would tick every n seconds, let's say, and then we would have an extra case and we would receive from that theater channel. Do the periodic task and then continue it's it's a pattern that is very nice.

A

So what are the channels we we do so we have. As I shown you here, we have two interfaces that we implement and basically we have requests to register.

A

This is the first one and then request requests to unregister when the context is cancelled and then we have requests to find a tunnel and request to stop looking for a tunnel when context is done. So let's look at the handle tunnel, so yeah, that's where we receive register and register, find and stop looking for, stop finding abort find request.

A

So, let's look at the method.

A

So this is called by. You can see.

A

This is the connect method of the reverse tunnel server and it's followed by.

A

An agent which wants to establish a reverse channel connection- that's authentication here and then delegates to handle and polling stops on turns, and this is a blocking method. So it will.

A

You can see what it does. So that's not important right now for this conversation I just want to like go over it and go over the high level logic without the pairs. So we register the tunnel.

A

Then we either wait for the cancel or for an error to return so to unblock the incoming connection, return whatever needs to be reported in time connection, and then, if the context was cancelled here we haven't registered. So we just returned- and here we have registered, so we need to unregister so that that is what we do. We try to tell the run, go routine that we want to unregister.

A

Alternatively, if has picked up the tunnel already and is trying to send us a reply like what to return. We are reading from this return error channel and here, if we have unregistered the contract, is we still need to return something right? So uh we we read from this from this channel as well.

A

So this is the part that is called by the server server method that is called by an agent and.

A

This is this: is it it's? It's a blocking call that sits there and registers waits for reply or if the context is canceled and registers, and that also waits for reply. That is straightforward, but look how nice, um how nicely it works with context.

A

Imagine using your mutex here and you would need to have another go routine, that waits for context cancel. Then you would need to use broadcasts on youtube to tell to tell the whoever holds the yeah. I don't even know how to work. I don't know it's. I mean, of course everything is possible, but it's just better with channels in this particular case.

A

uh So let's look at the other method, which is called by um it's. Also.

A

This is the router that routes, the request from one cast to the the request: that's coming from a cast to gateway, so routing cost gateway, cost agent, that's the chain, so this code is being executed in the gateway class and the one which has the agent's connection that we that we, it was received by the connect method, and so here we are finding the tunnel and we are using the contacts to stop.

A

Looking for the tunnel where, when the context is cancelled, then once we have found the tunnel we are using the tunnel here forwarding the incoming stream. There is plumbing here. That's let's not talk about this, and once we are done, we are calling done.

A

On the tunnel to so, if you have sent something there, it's sent performed, send or reads on the stream. Then done. Will.

A

Just change the state. There is a simple state machine inside if we haven't done the need to just return new to the return channel so that this method is unblocked and you return something here. Otherwise it will get stuck forever. So you see you it's a deferred function. So even if we have an error here, let's say we still unblock the command.

A

um Then let's look at the frontal implementation.

A

Also, let's not talk about this well, we create the requests to find the tunnel phantom requests and then we send that request to the find request channel. So we are sending to that actor. Go routine that I think of it as.

A

This we have a connector routine, it's like spinning in that for loop. Then we have incoming requests to connect. It's difficult, connect.

A

Connect yeah and then we also have requests to find.

A

To find the tunnels and state is here maintained by by that guarantee, and these requests are sent to us by maybe without the loop but but by uh by other governments. Let's just.

A

uh One one two.

A

um So it's like actors or guru teams talking to each other while channels, so here we are sending sending the find request to them to this one, and then here is a two case select case one case two: we are either waiting for the contacts to be cancelled, this one or for uh tunnel to be.

A

Returned from that main actor quality, and if you got that tunnel from it from the main version, we return it. If we got new because that's possible when.

A

When this thing is shutting down, we just return, cancel.

A

Here, if the context was cancelled, we need to tell that this. This goal routine. That find requests is, we know we are no longer interested in the result and if, if there is a result actually concurrently with this, uh then we get the and we got the tunnel that is not new, then we just you, know closed uh or done it immediately.

A

So this these two methods are very similar.

A

Also, contacts and channels composed very nicely.

A

So, let's look at the run method what it does well, I have shown you so this is the register it handles. The registration then register it handles on registration and find and find the board. So this state is maintained in these maps. We have thanos budget ideas. This is just like a secondary index in the database to make it easier to find tunnels by an agent id, and then we also store find tunnels request by agent.

A

So I yeah, I will not go into details, how this works, but you can imagine how there's nothing super interesting. uh So, let's look at so we have checked out the correct division. Let's look at the.

A

This is the magic quest that fixes the problem. Here I copied the stack traces of the relevant routines, so this is the first one that.

A

We have eight hundred fifteen routines blocked, trying to register so they are calling handle tunnel. So this is all the gore routines that from from agents that are trying to establish a new tunnel connection- and we can see this here by the structure- it's the connect method or we can just copy this and then two times shift paste and just you can go here. He needs to um your favorite.

A

Editor can probably do this as well or you can just navigate to the path, and why- and you see, they are called and let's look at it inside of handling where it actually works. So it's blocked on this select, and that means it's trying to register this tunnel so and here we do not see because it's an aggregated view, people of tells us that the right hundred fifteen routines blocked here. If you look at uh so non-aggregated view uh which it can also give you a few space, there is a query.

A

Your query parameter one aggregates two doesn't aggregate.

A

If you look at individual guarantees, it will show you for how long this particular one has been blocked. So this one doesn't because there are 800 there is no like you can tell how long all of them have, because each of them has a separate lifetime walking profile.

A

So anyway, um they're all blocked trying to register with the tunnel and that's not normal. It should be instantaneous because the actor go routine doesn't do anything else. It just reads from tunnels from channels does in memory, modifications of the steady tones and then those sorry replies via the other channels that are passed like find, request, passes the channel, for example, and also the register requests request, passes a channel to reply to for this particular request.

A

um So it should be instantaneous this that guarantee doesn't perform any ion. It doesn't perform any anything that can block for a long time because it if it blocks it's a bad luck very quickly. So then we also have 773 goroutines that are calling the handofano tunnel. So these are 700 something go routines that are, they were trying to perform a.

A

um Api call and they they got stuck and that's also not how to work. Obviously, you can also just navigate to the line and they are blocked in this case. Actually there they have registered the interesting.

A

Yes, so they're trying to unregister okay, these ones yeah they were they have registered, but then they were trying to unregister, oh okay. This is still the handle tunnel, not fine, so yeah. This are the ones that are trying to unregister, because the contacts have been canceled and they are stuck here. There's no reply return and yours. It also couldn't unregister.

A

And then there should be another one which is defined yeah forward stream on the tunnel. So this is the one where.

A

ah Wait: it's not doing.

A

It's not opening the right file. Okay, now, okay, this is the right yeah, be careful when you are navigating this. Sometimes.

A

So this is the forward stream, which is followed by the incoming the incoming uh request and they all blocked trying to read from.

A

From my goal to performing the proxy, but it's stuck okay yeah: this is a different problem- was fixed in different guarantee, and this is the this is the problematic structures it's only one, because that's the actor holding the state, so it's always only one. Otherwise we would use matrixes and it's having some unregistration of some.

A

A

And what it's trying to do is start trying to return you to.

A

To put putting you into a channel that is then used by the handle.

A

Yeah, this is the one that yeah. This is the channel it's trying to write to its unbuffered, so someone must be reading from it for the right to succeed, but yeah. The only the only case where there is no one reading is.

A

If someone has already read from this channel or if we should be writing to it, and the problem here was that.

A

Yeah, so the problem here was that logic in this one: it's just it should. It should be checking well.

A

Yeah, it should be checking if this particular panel is still registered with that active routine or if it's, if it has been removed. If it hasn't been removed, if it's still registered, then we call done don't block it, and then we remove it from our state in member state and yeah wrong. If condition left to deadlock.

A

Yeah, so I think that's pretty much it.

A

Let me questions.