GitLab Environments Team, 2 Aug 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: GitLab Agent for K8s: Deep Dive into KAS and Agentk connection handling and tunneling requests

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

A

We have a session here where Mikhail and myself briefly go over or briefly, let's see how briefly we'll be I'll, go over how the tunnel Works in Cass and h, k um in a sense that we want to find out like how is the tunnel registered?

A

um How is it on registered and then in the end like how it's used by the modules and yeah I think we start with how it's registered, how it's unregistered and then maybe in the next session, we can find out like how it's then being used for module like the proxying module for kubernetes API calls and yeah I think it probably makes most sense. If you share your screen Nicole. What do you think and then I think like we can?

A

Maybe we can start with, because that's also important, like you know, if we only have let's say two cast instances without any HMK, they need to register themselves as well, so that in the end we can find tunnels right, because only one cast instance has a tunnel to a specific H and K. So whenever there's a request coming from a user, we first need to find the Gateway cast instance um throughout for right.

B

Sounds like you can record the meeting by yourself.

B

You know the things already so this is. There is this document which is yes, quite well high level. It is up to date, but uh on the low level details it's outdated, yeah, but um it's kind of the diagram that was there from the day. One and.

C

B

There can be one or more class instances and there may be a single. Let's say this is Agent num id1 and this is ancient id2. So there may be one replica like in this case and then maybe two replicas one replica again and they may be connected to the same instance like.

C

B

These are these two are connected to different instances and actually One agent. One replica can be connected to multiple cast instances as well, because it's an implementation detail how TCP, uh how grpc manages CP connections and it can automatically PCP connections and the load balancer may send those connections to different custom instances so yeah, basically any number of replicas connected to any number of consequences, or maybe to the same one.

A

Yeah I mean I I, read this document and also there at the at the very bottom of this um there's, also like a more detailed, um so here yeah here we have a graph for a call. But then, if you go further down even.

B

You know this is the sequence diagram exactly.

C

B

Is the routing number and request there's one.

B

Details yeah and then this is kind of it kind of reflects the implementation structure.

B

And also this is request charging document. It was strategic. This is not specific to shaytano.

B

This is how, in it any client in this case it says public API, jrpc server, uh and in this case it's client is Rayos, because this is only exposed to rails, but it can be the kubernetes API proxy. That's part of cast talking to this thing, so it accepts an HTTP connection and within the same program talks to a different server using using grpc, and then all this Machinery is identical.

B

So, for example, we could Implement a jrpc API on this server to I. Don't know, make a certain request to kubernetes and only make it available to rails, and this will this will use the same Machinery as a kubernetes proxy, but we can expose them different way or help any logic on top of that proxy layer.

B

Also, theoretically. Well, not. Theoretically, we could proxy something else, not necessarily uh kubernetes API server. Agent K could talk to a local, well, not local, to anything in in the cluster or outside of the cluster. Maybe even and then we can process that and make it accessible inside of cast and then somehow expose that to rails or even to develop. Somehow you know.

C

B

We could I, don't know, expose a local port on on on some IP inside of the network, make it available to external consumers.

B

If, if we ever need this feature, that such feature will use the same Machinery, so it's basically multiple I would say maybe two logical layers. This reverse panel layer. This is the plumbing, and this document is about that layer. But then there is also another document, and that document is the I'm going to just share access.

B

It's the first feature that you use use the set on the tunnel. They call it set down on this feature, but um it just talks about kind of product side of things and how it's how it works from the user's point of view, and uh now here it's talk, it talks about, uh you know, allow details and API blah blah blah and.

B

Yep, it's basically it it refers you to that other document we just saw so you see this whole feature is built on that underline layer.

C

B

And we can also see this in the module structure for others who are watching this. A more almost all of the functionality in the agent and the agent and class is split into modules, and this is the reverse tunnel module the first document, the lower layer- and this is the communities API proxy that sits on top of this architecture-wise. You know.

B

And if you look at the cost side of things here, um there's a like it looks like a lot of stuff, but it's actually just Plumbing authentication authorization, blah blah blah management, cookies, together and so on. But this is the thing that reads and sends data and receives and responds and then.

B

It's like a generic thing that helps you to serve and bond HP and are turned into advanced RPC and.

B

The area code pipe and we give it uh aligned hrpc client. This is the response, STP response. This is HTTP request and some extra net information. So what does it makes? The request using the client, outbound, client and.

B

Writes it writes it to well, it's a lot of blending again, but it writes the response to the response. Writer HTTP response writer reads: the request from the request object.

C

B

This but but you see, this is the client and what's interesting, is that, let's, let's unwind it like? Where is.

A

It yeah I think that's that's what interests me the most like what you know, this client that is being used from.

B

This requirement is this: is the kubernetes API client and it's we are calling make request so, okay, what what are we talking to? What is the server? This is the plan. What is the server? Let's go there, okay, it's being passed here and it's the factory and it's the grpc connection, clamp connection interface and okay. So where is it connected? What is it connected to it's coming from config, and this is the connection that can be used to send the request to an agent K instance.

B

If you need to provide agent ID and some other things, but, okay, where is it coming from um it's coming from agent internal server in memory connection? What is this.

B

That's the thing: what is this? It's a grpc connection, but it's connected to this string doesn't matter it's just readability Bible, because we use a custom dialer and what is this? It's a listener and dialer. So it's a thing that is basically in an in-memory pipe between uh do not well, it gives you an accept as a listener would and gives you at the context as a diagram would so you have to go routines, one except incoming requests to connect and another one can connect.

B

So you can use this as a pipe between clients and a server in in memory in the same program under the hood it uses the. Where? Is it not pipe standard? Library Notepad, which gives you two connected? Netcon objects.

A

In this netcon object that you connect, one is from the reverse tunnel and the other one is um wow.

B

This is grpc client, the one one netcon is used by the grpc client. Another NETCOM is used by grpc server. You can see it here so listener.

B

It has a dialer, it's it's going here and then the listener is also so we use it here in the client, but- and we use it here- sorry here as a listener.

C

B

Where is the listening code?

B

It's going here and we are starting the server with server using that listener. So, basically, at this client is connected to This Server.

A

Okay, okay and.

B

What what what is This Server? It's an internal in-memory server that um within used to to some.

B

um We should use it somewhere, yeah, okay, we use the would use it for request routing um and this is creating new router. So.

B

um For routing we have as I as I said as it is described in this document, we have multiple cast instances, so the first class that accepts a request.

B

Well, this is actually a connection from Agents, but it's more about this request again. This is an example with rails in mind, because when this was designed, we didn't have say tunnel in mind. I.

A

Mean the sequence diagram below basically shows that no, they basically.

B

Yeah here it's not a user interface or not side job, it's a good love as a plant, but they it's actually. You know cubicity or in NCI, for example, or help or whatever UI, making a request to kubernetes and but again here it's not explosive, realistic. It's exposed us just hrpc endpoint. That does this certain thing yet list of running.

B

A

C

B

A

B

Basically, yeah: what does this router do? uh It's?

B

It has two parts.

B

That accepts two servers: internal server and private API server. So.

B

The router sends requests.

B

Yeah, it's a bit download, so the router registers.

B

It's renders these two on for all the API endpoints Japanese endpoints that we want to expose and we'll see that how this method is used later.

B

We register them on both servers, so internal server, the one that we connect to Via the pipe in memory pipe and we also register the same thing on the private API server, which we haven't talked about yet so prior to API. Server is the server it's visible. Here.

B

Is the server pass talk store, so we have routing costs and Gateway cars. Routing class is the cast that accepts a request from a client which can be that class as well or can be something else doesn't matter. It's the low level, lower layer, lower level layer, so this crackle is something that one stock says and API endpoint on the agent. The whole purpose of this lower level layer is to expose grp scene points on the internal.

B

Oh here it's on this internal aprtc server, as if they are so that the client is calling those and points as if they.

C

B

Client is talking directly to This Server. Actually, so, basically,.

B

This proxy, when it's using this client, it's it, it could basically talk to the agent directly well, it thinks that it talks to the agent directed basically there's no difference from its perspective.

B

It does know that it's actually talking to any memory server, which is then proxied to another cast, and then let's go into the agent. So there's like several hops, it's working with this climate, grpc client, as if it's just working with the agent directly. That's the whole point of this um to hide all this routing complexity.

B

So internal server is what accepts requests to be routed to the agent. Then it sends them to private JPC server on that same Customs or on the other past instance, but conceptually let's, let's just say there are two separate instances, um so we have a routine class and Gateway cast and because we have to name them somehow right so.

C

B

Give them these names and I am a router registers. Handlers on both JPC servers, internal and private grpc server. Then the job of the Handler on an internal server.

B

This route to cast stream handle.

B

The job is draw to another Customs, so its job is to find another cast instance that has a connection from the agent that can serve this particular request.

A

And what does cancer mean? Is it just because, let's assume that I have an agent that has two replicas they're connected to different kind of different casts and both are Gateway cases they're, not the routing casts which Gateway Cass would it does it prefer one of those or does it just connect to the first one it finds or how is that working.

B

B

Let's first we'll get into into that in a second, but let's first finish that higher level kind of explanation trying to do here um so yeah.

C

B

Is the Handler that sends data to sends the request finds and that constant sends the requested, and then this one is the one that accepts that request here and.

B

It's called connection registry here, but now I think it's called. It's called Uh I'm not register something like that. So.

B

Its job is to well, it's not documented. His job is to find a disconnection from an agent.

C

B

On these questions or on itself, you know: do I have this connection yeah and it is looking for it and then once it finds it, it says: I am ready.

B

It sends a response here to this property pass saying: hey I have a connection for you.

A

B

And then this routing pass, it actually has some logic to I can just open it, and this is what you just asked: how how it finds like which cast to talk to this classroom. Forwarder.

B

Wait no first before forwarding it needs to find that ready Tana, so it means it needs to find cast with a tunnel. This is happening here in this method, so.

B

Utano finder, we create the finder and we call find that's how it works.

A

And and this find.

B

B

Finder and we find- and it finds it- that's the explanation so so. First it connects to itself.

B

It connects to.

B

This routing class is also a Gateway class. Yes, so first thing it does: it actually connects to it its own private, JPC server. That's what is this thing does and then well, there's um like if you need automatic here. Basically that goes through different transitions, but it it tries different because instances and it also keeps sporting radius for new URLs to connect to for this agent, ID, okay and then eventually, well, not eventually, but it connects to one and it can immediately reply. I have no ton of this agent ID, but I will keep looking.

B

So we keep that connection open then reconnect to another one. If there is another one and that's that that Customs can say, hey I have it done already. Then we cancel this first connection because we just pick the web any any one. That is ready.

B

Whichever is first, we use that panel because it doesn't matter who is first they're all the same and.

A

You can only occupy one like if, if an agent has a connection to a cast, can this only be occupied by one request or like this crpc then basically like it has multiple TCP connections and you can use the same tunnel multiple times because grpc handles that for you or. How is that working.

C

B

A single agent replica can have two to five hundred logical grpc connections so up to 500 streaming rpcs in flight.

C

B

All right and how many TCP connections are used is up to grpc. Okay, I think it's actually typically just one connection to a single IP but I guess if we had more IPS exposed because the club casket lab.com is behind cloudflare, so you always kept a single IP with the Cloud Player as far as I know, and because of that I guess, because uh JPC just opens a single HP connection because it just doesn't need more connections, but if we had more IPS, maybe it would do more.

B

It may do more for some reason: I don't know, I haven't dug into this. It just works.

A

Okay, but I mean theory is but.

B

Conceptually conceptually yeah, it can have any number of TCP connections and it multiplexes. The in-flight rpcs across allows TCP connections that it has opened.

B

Yeah, so we just use the First cast that responds basically with a matching connection and yeah matching here means.

B

It's it should be somewhere here.

B

Or not here because.

C

B

It's logics at least once yeah, so this is the where we in a separate Gore routine, try to connect, be with back off, to connect to to cast this particular URL and then open a new stream and yeah wait wait for a reply either. It's not no tunnel uh message say this: basically, a way for the gate request to tell this articles that, but he immediately that hey you're asking about this I, don't know uh you're asking if I have a tunnel with this agent ID, but I have no talents right now, but assistant.

A

When the cash is wrong, you get disconnected or.

B

Or that tunnel has been used already or any other reason, but it just says: I have no connection right now at this very second, but once it gets a connection if it ever gets that connection from an agent it will. This is still this. One is still waiting here, so it will immediately say: hey I, have it done already so.

C

B

See it we are waiting for either no tunnel or uh done already. So if it says no tunnel, then we basically.

C

B

Well, we keep waiting right because it can, it can immediately sit down already or it can just say no tunnel, and then it can say done already, so this basically blocks until there is a one.

A

Of the reasons.

B

Yeah not one, but first this or that okay, because you see, if we get done already, we return that Sentinel error value. So here if we see that same thing value, we just know that this was just returned from here, so that this visit uh returns from the call. Basically, it's a way to make it exit that's visible, and then we just fall through because it's unexpected error. We do this on purpose.

C

B

Then we continue because it's all good, we found the tunnel yeah.

A

So it's not really an area.

B

Yeah, it's not really now it's a it's called Poison poisoned pure pattern. Well, we use poison peers, usually used to top sampling, but in this case well it actually is yes,.

A

Because of the visit interface.

B

Yeah, it's just visits until there's an error or it gets in EO, but you want to stop earlier than Eagle. So we have to do something. So we just return an error and then we continue- and here we.

B

Actually, grab a local mutex check that we are the first. If some other girl routine, has already started streaming to a different class instance, then this would be set already if it, if it is that we will return if it's not set, then we still we're still holding in the Box here below we send the connection. uh Send the message saying: I'm starting streaming to you, like routing cost, sends a message here from the internal server to this private server. Saying I'm I'm going to send you the real request, yeah.

B

And uh in that, when this receives it, it actually starts changing data to that to Agent K via that reverse tunnel, because before that it didn't actually use anything, no data has been sent to Agent K. It just said it just took the connection from the pool that establish a tunnel. He took it from the pool it it was. It said, I have one for you, but then this can just abort right because it's it found the connection somewhere else already.

B

So in that case, this would put the tunnel back into the pool it can be later used or it can be used by another cast instruments or even the same. Customs I mean this. The other routing cast instance for.

C

B

Request so yeah panel is like: if it's not used, then it's put back to be used a bit later. Okay, sorry.

B

So yeah the little one decided to wake up um so yeah. The tunnel is put back into the wolf panels that are available for use and can be picked up by another request.

B

But if we let's say we select to start streaming and then we send it to start swimming without an error. I'm gonna reset this fly in this garage and then we release the musics.

B

And then we cancel all the other goal routines that are talking to other cast instances because, as I said, the routing costs can connect to my main kind, Mutual class instances.

C

B

Whichever response first uh yeah, we will use that, but we don't immediately connect to all of them. That would be wasteful. Like imagine, you have I mean we are running 18 instances of caste in production before each request. We connect to all class instances that will be 70 dollars, yeah. Basically, so that's why we have already seen the first place with pass URLs per agent ID so that we look up which class instances have a tunnel right now for this agent age, and then we connect only to well.

B

We always connect to ourselves because that's the most efficient thing this is also using in memory pipe. Not we are by passing the whole was stuck TCP and everything right. It's just memory pipe between when I say pipe I, don't mean Unix parts or.

C

You, like yeah I,.

B

Mean just it's two objects in memory: ones writing another one. Taking that data and doing something with reaching it. You know and doing something else, um so we are connecting all to our to our own private server always, but then we wait for like a state machine here we wait for No tunnel and then it tries next class, so we yeah. This is the cash list of URLs that we know for this agent, ID, yes, and so we we try the next one, and this is where we are starting.

B

This is where we start.

B

Go routines for this URL we will start ever go routing for the zero if we haven't started aggregate for it already, and this thing is what I just showed you. This method that is in a loop tries to connect, because sometimes you can't even connect right and then once you have connected you're waiting here.

C

B

Here we are also boiling uh it's in a separate growth in somewhere here uh yeah, if not satin, bowling we're also starting polling. This that's a separate goroutine and.

C

B

Are pulling this pause radius for URLs for this agent age? We get a list of URLs. We send it through that channel into that Goro team and yeah. It checks if it needs to established a connection or not, then also, if we have.

B

Url- and we also have a watchdog timer here- that if we haven't heard from that class for any reason, yeah.

C

B

10 milliseconds, we are going to.

B

um Where is it this? We are going to see if we need to try another Customs yeah.

C

B

We are going to try it if we have one in the list like this thing will iterates over the Euros latest URL is known to have a tunnel from this for this agent and.

A

B

Is coming from a radius list of URLs for this agent ID.

A

Okay and the red is um like you know connecting through this- is implemented in this tunnel querier, like that's the only point where we yeah yeah here and then the polling, which will populate the urls, where, like for.

B

All the URLs that.

A

We're getting we're sending them to the URLs Channel and then in the for Loop below we yeah. We get them.

B

Out we assign it to this variable this variable, it's just a list of strings with three arrows and we use that variable. You see, we don't Define a new variable here. We are mutating rest area assigning this variable and this closure uses that here and we are looking through it and seeing if we are connected to it already then continue. If we are not connected, we are trying it and return from log.

B

And then reset the timer, and then if that was the very first time we didn't connect to a cast because, for example, we ran out of Euros to try.

C

B

We start polling here because sometimes you have a list of cached urls and you connect to the first one and you you get panel available message.

B

And you don't need to start falling, so it's an optimization. We don't pull if we don't need the power yeah.

A

And this is um we only like this code here is per request that we want to make.

A

So if we so if we want to connect to H and one and we found the cast whatever and then this connection, like you know, response a response from H and K or kubernetes or whatever the module does and then right after we have another request coming in for the same agent, we run this entire code again finding the correct cast, although some stuff may be cached on by redis, because the request before found the cast instance and whatever is that correct or.

B

Almost correct, we do run this all chord again but, as I said, we have the URLs for this agent ID cached. Here, yes inside of the query, so we do not need to go anywhere. We don't talk to radius yeah. We have it in in memory of the program and we just try this urls.

A

Okay, can you shoot casseroles? What does it do.

C

A

Okay, oh yeah, okay, okay, it's.

C

A

A map I thought this is gone: okay, okay,.

B

It's a cache on top of her I, just expiring cash, on top of it. Basically, okay.

A

B

It doesn't cash forever. Of course it it just clashes it there for a very short period of time. I, don't remember how long.

A

And we get new connections because we have a pub sub two red. Is that updates? No.

B

And this this functionality doesn't use Pub sub, it probably should actually there's I think there is a um issue for it. There was a mission for the weekend actually implement this position. Pops up.

A

Okay, so it it was all like it actively pulls.

B

Yeah it pause. Also, this is aggregating query. You see it's uh yeah. This is where it's actually discarding the expired cash so that we don't consume all the memory and.

A

What's the period um period is that like seconds or minutes or.

B

Several minutes: okay, but I, don't remember how many.

A

It's okay, like I, was just curious about five.

B

Five minutes: okay,.

A

And if, like you know, if, if one um connection is ending up here and like, is this expiration date being updated by someone in case, the tunnel is still active or does it always expire? And we always have to like recreate.

B

It well, we have used up the tunnel, there is no more tunnel if we picked it up, sent the request and got the reply and closed everything. That's it I know it's done. The agent establishes a new one. Well, it has up to five hundred thousands of them. Those ensures that it has a at least some number open.

B

So there are always three tunnels unless you have reached 500. Basically, okay in practice, I I got some good routine dumps and there are 70 80.

B

Look at those dumps so.

A

B

Depends on your workload, but I just wanted to point out another thing here. This is aggregating query and it's.

B

B

Surprisingly, what kind of aggregation uh imagine you have as it typically is? How and or maybe Cube still that does Discovery concurrently and that's dozens of concurrent requests on the same agent I did and if we don't have it in the cache, what is going to happen? We are going to start I, don't know 20 polling guarantines and they will eventually find the title and everything but whiteboard already.

B

You know 20 from 20 gold routines. If you can only actually need this data one time right. So that's what it does. It duplicates the data and the polling. Basically that's what this that's, why it's called maybe start boiling and maybe stop boiling because yeah. If we are the first to look for this agent ID, we will start polling and if you are the last, we will stop boiling and once we get data we call the Callback but the polling. Actually you see it gets a listener object for this agent ID.

B

If there is no no one calling already, we create new one added yeah, but if, if there is someone calling already, we just use the same listener and we just add a consumer to the list. Basically here.

B

So this ensures we don't put radius more than once for a single agent ID, but we also should do Pub sub to cut cut some latency.

A

Okay, but yeah I think it's. That part is clear.

B

Well, what I haven't showed you, which is also important, is the agent part of this whole yeah.

A

Exactly like, how is the tunnel established in the first place from an ancient.

B

So so far we saw how we looked at the proxy. Just to recap, uh we looked at the community by proxy, which uses the JPC client JPC client to talk to well, this diagram doesn't apply. It uses the client to talk to internal here. I was imagining that we will use the chair, PC server and the PC client to like real stocks to that class to do make a request, but ins. Instead of that we have an HTTP server here, and this is let's say: Cube CTL.

C

B

Cube CTL back into HTTP server, and that is the HTTP server. It's currently pay proxy and it makes a request to the internal server and use that plant. That thing looks for air cast instance, that has a tunnel with this agent IGN A1. That actually has that service and the method implemented, which is grpc's servicing method yeah. But we only have one for this for this one, which is.

B

Make request it's defined here in RPC.

B

This is the governance API that agent exposes, and it has only one well. This is called service and this is called method, so it's got only one method with that accepts a stream of requests. Well, it's a single request, but it's just contains multiple consists of multiple messages like headers body body and then trailers or an error yeah. So it's a stream of messages and then the stream of responses as well.

B

And so, basically, when we are looking for the agent connection, the tunnel, we are not only looking for the agent age, we are also looking for the service and method, and this filtering is done somewhere. I, don't remember where, but it yeah it stands somewhere in the router. Probably, um why is this important? It is important because suppose you want to add a new new method here to I, don't know or.

B

B

Whatever now, let's say, we want to call a service in kubernetes, not kubernetes API, but the service inside of the cluster. Let's say by name. So we can add an RPC here and we can say it's not HP. It's TCP. We don't care if it's HTTP or anything, and we can expose this using some API.

B

Android piece API or HTTP, socket members or whatever um two rails or two, an external consumer.

B

So we don't know if a particular customer have a great had upgraded to the most recent version and if they have this RPC, so looking for just agent IDs like we are going to send a request, but it will blow up because there's no such service method on this service. Okay,.

A

Interesting yeah.

B

Right so we are looking for an agent that implements service and method.

A

And this is this: is the service like the service information automatically shared by grpc, or is that something that we have to do or Implement ourselves.

B

Yeah, yes, we do that in let's go out. I think now is good time to have a look at the agent side of things.

B

um So this is the module for the reverse tunnel, the agent module for the reverse tunnel. So.

A

This is the basic, like you know, underlying code of opening up a tunnel right.

C

B

So it starts the connection, manager and trans and it does all the job good job. That's your explanation. Well, this is the information that we send agent descriptor and this this. What has the services and methods.

C

B

One on the hrpc server: this is job which are PCU, it has the get service and form method and yeah. We just get all the services and for each service we get a list of methods, and we put all of that into services agent descriptor uh to find somewhere.

A

This is a product that we Define ourselves. It's all: okay, yeah.

B

B

It's a list of services and each service has a name and a list of Methods. At least each method has a name yeah I thought we can add more views here. So that's why it's a separate message. uh This is just a list of strings yeah again, if you later want to attach some net information we can but yeah. This is what is sent with each tunnel to cast and.

B

I now remember where this Victory happens.

A

So in here the max connection is the 500 that we've been talking about as well all right. The max connections is that.

C

Yeah, okay, yeah.

B

Promise from yeah here.

C

B

And we also scale up by 10. So if we have fewer than two idle connections, we go up by 10, not by one but then, and then you we use them, use them use them as soon as we hit one idle connection, we go plus 10. and.

A

They're basically like once they have been used, they are destroyed no longer there and we basically decrement where it's you know, yeah. Basically, yes,.

B

Okay- and this happens in.

B

The Connection Manager.

B

Election manager manager support Connections in their life minus.

C

B

Connections and manages connections.

B

So, first things: first, we start the minimum number why the connections? What do we need- and it's just definitely basically that ensures that there is at least minimum number of Connections in a mobile number minimum number of connections in the idle State yeah and it switches switches them between either active radioactive and then stopped, and this just transitions in that automata. Basically on idle transitions from that to that, and so on and.

A

Why is there, why is there a stopped state.

B

B

uh To detect incorrect code and to detect uh incorrect usage of the API. Well, it's the same thing, but also.

B

Okay, there was some other, some other reason.

C

B

We have yeah it's kind of the duplication of a decrement, because because of instance, there are different code paths that decremented the okay.

B

um And that's that was the best way to do this, to just duplicate it like that. Okay- um and this is the connection manager, but it doesn't know anything about the connection. Apart from it, manages the number of connections yes and it gives a connection which is.

B

B

Connection Factory, yes, we have a connection Factor, it creates connections and we we give the factory two callbacks foreign, active and.

B

Nonidal callbacks on there a Connection Manager to let the manager know if this particular connection is active or either so the manager can account, for you know, what's happening and that new connections or whatever do.

C

B

A

Maybe like a very basic question, but how like? How does a tunnel relate to a connection? Is it it's the same thing right? It's.

B

The same thing: yes, okay but I, don't know it's just from the yeah. It's it's the same thing. Just I, don't know: I use, different terminology, I. Think, okay,.

A

Yeah, no just for it was I.

B

Think there was some Clash initially I use connection the word connection, but then there was some Clash somewhere and I was looking for a new name and then I renamed it Cortana, but then I guess I didn't rename it here.

A

B

I forgot to rename it here or something, but it's the same thing.

A

Okay, now that I I.

B

Mean I think the server code uses tunnel and the agent code uses connection okay,.

A

B

I should switch your name.

B

Oh okay: let's look at that connection, so so it's That's The, Connection Manager. It accepts a factory and that's the fact that it's just a function that creates that connection um and that connection has a run and there's a loop with back off and it drives.

B

And then, if there was no error, it continues immediately. So immediately makes another attemptive connection has successfully processed request or whatever. Oh it's all good. We continue immediately. If there was cancellation or timeout will continue immediately. Anything else. Any other error we back off. So we stop, we don't have our class. Basically, if we get an error, we slow down.

A

So but in this case like we, we said before that, whenever the connection is being where the tunnel has been used, it will be, it is destroyed. Basically, but here it looks to.

C

A

Maybe I'm missing something, but it looks to me like it's just being reused. um Okay,.

B

I know on the server side like conceptually the grpc request um stream is, is done, there's no more stream. There is no matano yeah. So this can this connection. Object is still there. Yes and it's it's gonna, try again so I guess there is a difference, makes this grpc request, which is by the way we can look at the product definition. All this thing, um you see it's quite quite big.

C

B

So the the reverse style this service, that the server exposes the reverse server.

A

Is that what is being called in in the attempt method we saw before? Yes.

B

Yes, this is what we call and we have the request and we have the response. Yeah and the request contains the descriptor and then the header of the request, which is.

B

A

Metadata for the like, like what to call or metadata about the agent itself like I, imagine.

B

This is metadata about the response, and this is metadata. This is message of the response, because it's a reverse funnel.

A

Oh yeah, true, the request is actually yeah. The request is actually the response of the actual okay. True, yes,.

B

And- and this is also reversed.

C

B

Here is actually server. Server responds to this connect by telling the agent what grpc method to call.

A

B

This is in the connection before yeah. This is coming from class and then the message and then when, when the server wants foreign, the connection to close the outbound connection to the average RPC server yeah, we say it sends it's the closet method uh message.

B

This is grpc protocol encoded into Auto messages, basically and reversed.

B

That's right code, reverse Donna, because yeah.

C

B

A way to go server there, but it's actually on the client side, yeah.

A

Makes sense makes sense.

B

There's actually a really old RPC issue, Well everybody's asking for for this very simple as part of grpc implementation.

A

C

A

Sure, if I understood correctly about this, when a tunnel is being uh close, so he said, there's a limit of 500 connections or tunnels, and if you know in the beginning, we have two tunnels, then two are being used. We scale up by 10 and then no more tunnels are being used or all the you know. The connections have been finished.

A

I I before I saw this code. I thought I thought that it will go back to two, but that doesn't seem to be the case right. It will stay up at you know uh whatever it's scaled up to and eventually it will reach 500 and never scale back down. Basically, is that correct.

B

Okay, so connection.

A

So these will just be like logical, grpc tunnels that stay open. It's just like a memory thing Network wise there is. There won't be any activity, there's no like open, like 500 TCP connections. Right. It's like one piece of maybe.

B

500 RPC streams 500, but they may all be on the same PCP connection.

A

B

On maybe one more TCP connections- I, don't know how many interpreci happens. So uh uh what is that 33 in.

C

B

Note how when we run the connection in a Gore routine, so factories used to create the connection. Then we persisted in our yes: Mentor persists it in the map.

B

um We started in a go routine, but then we give it two contexts. Yeah.

C

B

See all context and both context.

B

What's going on it's interesting that the linter is not unhappy here, it's it's fine. Oh, we have a context to use for attempts and the concepts to use for polling. So basically the one that is used for polling is what is the cancel for it?

B

We save it in the in the map. So that's what we can sell when we want to stop the connection. We only cancel polling. We don't conserve this.

A

B

A

Only canceled when we stop entirely or what yeah.

B

This is canceled when the manager wants to.

A

Stop this particular connection.

B

Okay down the number of connections.

A

Yeah, okay, and when does it do this like? When does it scale down.

B

I, don't remember, that's yeah.

B

So why not connection transitions into idle would check? Okay,.

A

If it's not been okay, okay, that.

B

Is it is already in the idle you see on idle? It's already in that in the idle it's current state is Idle. This is transition from active into ILO. We don't do anything well, I mean we don't try to evict it, but if it transitions from idle to idle- and this happens.

B

um On idol is called.

B

um Here so when, when an attempt is done, it is always called even.

C

B

Haven't actually handled anything yeah.

A

B

A

Skip down right, because if, if it's been inaccurate for this, you know we saw the time comparison.

B

Okay makes sense, yeah, so I do to idle transition, it means we are idle and then we are either again and and time has passed already and we cancel polling here. So this is executed. Synchronously here, I didn't differ and that attempt goes out of the loop here continue immediately, but Paul will check the context and we have just canceled it. So this is here continuing the jumps to this Loop and it checks the done so this will immediately exit.

C

B

So this round will terminate so yeah. That's how scale downwards! Okay.

A

Makes sense: okay, yeah. Let's continue with the attempt, then I'm interested in the connect call and like how then we send you know the description and everything right yeah and we basically, this basically is, like you know, establishing a tunnel with.

C

A

Cast like we are being connected to the cast instance that is given us to buy. You know the gitlab load balancer or whatever. um There is.

C

A

Cast specific Logic, for which, like if the local answer is, is bad and we are ending up at the same cast. Every Time cast wouldn't rebalance anything right.

B

No, but it will just send traffic to you yeah why this class instance yeah. It can't do anything. It can't move a TCP connection, but it has to somewhere else. You know.

C

B

So yeah we can agent connects to cast here and sends the descriptor yeah information about services and methods.

C

B

The future we could add more information here if we need to yeah like um not just that we do not have any versioning information, so we cannot make break backwards compatibility, but we can always introduce new methods or extend existing methods in the backwards compatible way yeah, and we can always.

B

It is safe to call a method, and then this all this Machinery will handle the case where the method is not present, nothing will blow up. We will just wait until an agent shows up with with that method, implemented yeah. If the customer is running an old version that doesn't have this method well, they will get them there. They can't do anything but yeah.

C

B

A

B

A

I think I see the the tunnel connection Clash, so I think the tunnel is an established thing on inside of a connection right, at least from what I see here in the attempt it looks like you know, the tunnel is only a tunnel if you've been connected over connection to an HM right. Does that make sense.

B

ah You mean this is establish it I.

A

Think, yes, probably.

A

Yeah, if you have to call this connection, the uh struct here tunnel then like what is yeah.

C

What is this client.

A

Right, it's just it's the tunnel, client or whatever, so but okay um yeah! Then we we have a like a streaming connection there at line 93 again right with.

B

Yes, we just wait for cast to send us the request before and that will contain the information at that stage. We transition to active, because we got a request that we need to proxy, no, not proxy, sent to it's. Also an internal grpc server. How to internal grpc servers, one task and one in agent. They are not the same thing. Obviously it's here right, so we here we we got a request and we need to make that request to that server.

B

Yes, this is where we connect to that internal server. I mean create a new RPC.

A

Yeah, but where do we now in? Do we in this new stream? Do we dispatch to the correct module server or what is this internal server code I'm wondering like? How does it end up then, in the in the kubernetes API module from here.

B

So this is a client connection. Well, as you can see, um do this in terms of your business or it is also in a memory similar to what I showed you Pro, identical to what I showed you in incast. This is the same same thing in a memory pipe between the client and the grpc server and we yeah. We are making an RPC through that pipe.

B

Let's first look at the client side and we can go to the server server on the client side yeah, because.

C

B

Have everything inverted? We need to call the server outside in the client and that's what it is. So here we connect to that server and uh private data.

B

By the tunnel, which is connection to cast into connection to the internal server Yeah.

C

B

Basically, here and- and here like this visit, we haven't talked about it. It's a region from this grpc stream.

C

B

These are callbacks or the mess uh in a one-off field. Let's just look at the concrete.

A

For the the state machine here or the you know, yeah.

B

This it's a state machine recorded in the these are called annotations. I. Think um sorry, let's look at the request we have. First is the descriptor second header. This is everything that the last standard thing.

B

But you can see that they are all in a one-off. This means that only one can be said if, if you craft a request or a response, crafted product message that contains multiple set, the first, the the last one on The Wire, will be the one that will be set. Everything else will be arrested by the.

B

So there's no way multiple will be set, so only one of them will be set.

C

B

We use this pattern in a lot of places in this project. Yeah same with the response, we also use one off with, and the transitions are encoded in here in this options, they're called options: Maybe.

B

So first is first allowed transition from nothing is one or minus one, which is e of.

C

B

Then, from one next allowed is two or three or a of and then from two two is allowed, so you can have multiple messages, then maybe three or maybe or and then after three only e-off is allowed yeah and the same stuff here or similar stuff here. But there's also trailers and.

C

B

Error and B from after trailers and trailers can go after a message order.

B

B

So visitor basically goes through messages a few in that one of and cause these callbacks, depending on which one which field is set in one of so, if it's the first field, which is the request before requested for number one number one, uh this callback is called with that requested before you can also see the types here.

B

Correct types, sound reflection, magic, yeah,.

A

That's really cool yeah.

B

It was really really convenient to have all that I mean the purpose of this whole stream, visitor is to enforce the correct Behavior. So we we Define what messages can can go in what order and in the protofile using this automata, and then it is enforced by that visitor. Basically, when reading from the Stream- and this visitor is used not only here in a lot of places actually and it's it's defined in.

B

Purposely twine automata in this package, the product for it.

C

B

Yeah that defines extensions on fields and extensions on one of this first allowed and yeah it's. You can read the doc here. It's yeah.

A

I've never implemented them, but it looks yeah yeah.

B

It's not very common to use them, but if you look at the value data that we use like this comes from the data, this is this thing for the data defines you have to come up with random High number and use that as the ID okay.

C

B

The specters, your underground number between between this master range so and then the code for it is somewhere here as well. Invisible you can. You can look at all this visit magic using reflection, collection and cause cause the callbacks using reflection.

C

B

And there's a lot of validation in the options somewhere options: yeah, there's a lot of validation here to ensure that function has correct signature and so on and so forth and yeah.

A

Yeah, that's some nice code, nice functionality.

B

Yeah, it was very convenient from the beginning to, instead of in each Place tracking, the state uh of okay. I got this message now that next valid one would be this or that and imagine doing that in in all the places or just trusting the client. That client runs on the customer machine. It can send you garbage and I, don't know, correction code if your code is not defensive, so this is bulletproof defensive.

B

You know code that is written one once tested, then we just use that so protect cast and including from ourselves, because this thing called works. One or two bucks.

C

B

B

Go ahead, I was about to continue just.

A

Yeah explaining this I mean I I, see like I, see the visit pattern here and and how you know the automata is mapping to the callbacks and why we have these products. um What is the stop propagation? There I mean I read the comment but I'm not entirely sure I understood.

B

So if we have the ideas that once we have started handling a request, you don't want um seek term to the agent to start on the you know, abort everything we want this request to finish gracefully. Then we exit.

B

Okay, that's the point: okay,.

A

Can you show this propagate until function.

B

It just starts to go routine that listens to uh to that context, but if, if we were told to stop so if context is canceled, it propagates the cancellation to that internal thing. But if we stop the algorithm stops and it stops propagation, it doesn't cancel. The other context is it's a way to connect to contacts and break the connection? If you want to break the connection between them,.

C

A

Okay, yeah that.

B

Connection is again a connection. Yes,.

A

Yeah, okay and then um so we have. We have this internal server con. um This. Can you go to the new stream.

B

It's just PC Library code.

B

We are just establishing and streaming RPC.

A

How does it map to the correct server I mean the server is implemented in like kubernetes API module right, the proxy module? And yes, yes, okay and then like so here like every like. This is a connection. Okay,.

B

Yes, let's go and look for where this is coming from. uh It's coming from the factory factory got it from here, and here we got it from here internal server connection, and here we got it from there.

B

Is one two three third zero connection? Okay,.

A

That, okay, let's it.

B

Comes from this file up internal server, and that is that con and again you see this the same pattern. We have that listener. That is both a dialer and a listener. We use in the client Construction again pipe uh as the dialer and we use that listener here as a listener, and we use it as a listener for that server and that server is this this server, so we have a client connected to the server via any memory pipe and that server that server we put it in that field. Here, oh, that's track!

B

That's the race where this is going after that um I think. That's the only place it's going to construct modules, just like gone by the way.

B

A

Think I'm missing is we're like we sent the information about the um like the agent descriptor with the service name and the method name. Where does.

B

C

A

Use that where's, the dispatching of this.

B

Or that's where we are going right now: okay,.

C

B

So so that is the server it's going here. You know it's going into the mod agent config server server and that's the description.

C

B

So, okay, where is it used? It's used oops.

B

Click the wrong thing here: um this is the Cuban CPI agent Factory, and that's what it does it registers that server on that server.

C

B

Then, as a Handler, it adds a Handler to that server. So it registers that thing with kubernetes API make request.

A

Okay, yeah, so okay I thought we somehow have to do the dispatching ourselves, but this is actually grpc code like.

B

A

Which is this like register the description that we got and that's basically, it right and yeah yep, okay,.

B

Makes sense and yeah this is just the server that implements that make request.

B

So so conceptually.

B

Communities API uh server.

B

Sorry server proxy in that proxy method,.

B

Before that, no.

B

All right, let's keep that again. Client climbed yeah she's, doing that even CPA client make request here in this file in the server module on the agent. But it's it's calling this fun this method in the agent. Basically, all of that Machinery is to call that function somewhere on a different computer, basically yeah, and what this function does.

B

It also establishes a connection but to the equivalency, Dash server, yeah, yeah, okay and it it does the thing and then that's Japanese yeah, all the piping, there's I, don't know how many pipes between all those servers and clients.

A

Okay, I think I think it's it's clear so far, I think the only major thing that we haven't touched on is how an agent is being registered or persisted to read this, and these you know the this tunnel registry code. I think it is that you know once we connect it to once a connection or tunnel from h. K connects to Cass Cass will persist that in redis and.

A

Yeah I think that would also be interesting to see.

B

So this happens in the reverse panel sure, so the connect and.

B

um In the hand authenticate we authenticate and it's called the handle tunnel.

B

That is implemented on the tunnel registry.

C

B

uh We get the description of the descriptor, sorry I, we get the descriptor.

A

The agent descriptor.

B

That is the agent descriptor again, it's good to wrap messages like this because sensibility, we cannot think so yeah yeah um and then that descriptor goes into that tunnel and then we register Tana.

C

B

ah By the way, this is where we are looking for requests that are looking for a tunnel to this agent. Id find the requests by agent agency. If we have an out an incoming requests where some other classes waiting for a tunnel for this agent engine and this agent ID satisfies that request. It supports the servicing method.

B

Then we just without even going to radius, immediately give it a tunnel to that. Okay.

A

But this is only like optimization if we would wouldn't have this code here, which.

B

A

This and you know yeah, it's.

B

Just a shortcut, optimization Okay, then, if nothing is waiting or whatever, maybe you don't support that method that it wants? We just persisted in memory tunnels by agent ID, it's a map which is the G2.

B

It's a set basically of Thanos, where it's just a pointer to extract each time is a unique pointer. So it's a set. This is too conveniently iterate over the set by agent.

C

B

And then and delete that inner or one operation as well, so the value is just we don't care, then we also have just a list of all the panels and then we also have a buy agent, IG map or or set of order, find requests.

B

C

B

So we just registered in memory of this cast, and then we also need to register the Mac in radius and also a proton register. Does tracker.

A

B

A

The agent tracker then.

B

Tracker registrators, the implementation of the register and the tracker is the thing that can register and register and but also we have a queryer.

A

B

It's a kind of a right interface and read interface. Basically,.

C

I mean because.

B

This uh panel register uses another right and right interface, register and unregister mutative operations and the router uses the read interface when it's looking for us looking for panels, looking focus with the planners, so we have two interfaces implemented by a single thing and it has all the state and memory, but also it has it uses that tracker too.

B

B

To register no, this is.

B

Yeah, this is what we were looking at yeah. This is uh yeah, what's used to register data in radius register tunnel, and this is again Reds. uh So this was done now registry, and this is a registracker.

B

C

B

Again here we have an optimization where so the goal is to have in.

C

B

Urls of cast classes pair agent age, so it's a map from an integer to a set of strings. Basically,.

C

B

So we don't have sets, we have well. Actually it does have sales, but we have ah a hash Implement implemented with the garbage collection and this expiration. So we use that here- and this is where this is, what it is expiring hash and it's a by agent IG we'll we look up. It's actually not a single level hash. It's a two level hash right. So we look up the correct hash. Internet is by agent ID and inside of that hash.

B

We have keys that are strings and that that are pass. Urls and values are empty. We don't need the value we.

C

Just use the keys.

B

We could actually have used enough stats, but we already have expiration implemented for the hash and we use Flash in other places. So we just use the hash here with an empty value, which is the same. Basically as I said, and as an optimization this field. What is it for? It counts. The number of times register was called for this agent ID.

C

B

And we also have our own private.

A

Like concurrently, so we know the amount of tunnels is that.

B

uh Yes, by agent, I did Count yeah yeah, okay, so for this agent idea we don't need to put so we only care about pushing our own private URL, um the API hero into that set right to register this class for this agent ID.

B

So we just only need to put it there. One time doesn't matter how many times we have. We just need to put it there, one time to say: I have tunnels for this agent. Id send me requests and that's why we have that optimization. We it used to actually track every panel in radius. That's we don't need that. It's it's a waste of CPU and network and memory. You just need to register a PC URL or this ID one time and unregister it one time when there's zero. So that's what you're doing incremented!

B

If it's one, it's the first one, let's.

B

The unregister does the opposite of its zero. It actually removes it yeah.

C

B

Just decrements it um and yeah and the real interface, which is which, which is what we saw in the router, remember it has the Euros by agent ID yeah it just scans that hash and for for keys in the hash. Look, it doesn't look at the valley at all. You see this cover doesn't use that.

C

B

B

uh A cast instance, maybe another one because registry from here this whatever is there, we just return to the product, but that's how this thing is implemented and.

B

That's how we track okay, yeah.

A

And this is all implemented in the reverse tunnel module. So there's an agent tracker module. What is.

B

This tunnel and just tab yeah.

A

Directory? Yes, yes, but no! No! There's an agent tracker module like a high level module. What is it that.

B

Person unrelated to Thanos, okay, who should rework this, but what it does it just registers.

B

It's used in a single place in.

A

But what is it for? Is.

B

It for, let me explain, let me explain so when an agent came a replica, iPod starts it well, it always has the get configuration connection, no cost yeah. Each replica has only one get configuration connection, so by counting get configuration we can count the number of connected ports, and this is actually a metric that we use. We also have another method that uh bounce unique agent IDs by using has some radius and reading that from the hash um we have that on the dashboard. So this agent tracker is what registers such Connections in the radius.

B

It also has other other methods here, because we can have connections by agent, ID connections by project ID project ID is the configuration project ID here and then connect the pages like.

B

I, don't remember what that is actually.

B

I, don't remember what what this is for, but when we, when you go to the UI and open the agents list, you see that some are green and summer. Read connect not connected a UI asks rails about connected agents and rails asks cast uh by calling one of his uh via RPC, because we have that server here yeah. So this is called by cast to find find out which one is connected to which one is not. So we need to rework that to send more information and we should stop up using the get connection.

C

B

Call and instead use a dedicated RPC for that there is an issue for that. There's a long discussion on. We have figured out how to do it. Of course, it's not a problem. It's just conceptual way to best put it and configure this out a long time ago, just never implement this. We should do it.

C

B

To cover that from get configuration and also it will work this API by the way, because there's some inefficiency there I don't remember.

C

B

That's what this method? That's! What this module does implements this API for rails, to to ask us about the connected agents.

C

A

Okay and in terms of um like TCP connections or you know, grpc connections, if you know this, this module here is open. Opening. This get configuration call so there's like one Connection open the reverse tunnel. You know it does another one.

B

um And this has nothing to do with the reverse Thanos.

A

Yeah this one not but I mean in general or if the you know, the get Ops um repo is opening a connection to Cass um to fetch updates of objects. Do they use separate like TCP connections, or is this all the same connection um from an agent.

B

This is all up to grpc TCP connection management is completely hidden. Okay,.

A

We use the same like I, don't know, I missed the the name that chairperson uses like the same dialer or the same like you know,.

A

Which is not the connection we were talking about in the reverse: module right. It's like a grpc now.

C

B

Enjoy PC, it's JPC, client cloudcon, so.

A

One agent replica has one crpc connection, which may in many modules may use this like the reverse tunnel, um DHL tracker, whatever okay, yes, okay,.

B

Now it's constructed here in app.

B

Somewhere construct class connection here we go um options, blah blah blah, and then we either wrap it in web sockets or not and surprised our context. That's what establishes that connection yeah and it it returns. Client con.

C

B

Clanton and this thing implements lamp, Corner interface, I think yeah, that's what we that's! What we use.

C

B

A

The reverse tunnel and stuff right, yeah.

C

A

Yeah I remember in the the visit there when we get the request info number thing, your request info, whatever we open it.

C

A

Okay, cool um yeah I learned a lot from this. It was quite.

C

A lot we have covered.

B

If not everything, most of it definitely yeah.

C

B

Going into too much technical details like.

B

A

um Yeah very nice, very nice um I'll check the docs once again to see if anything doesn't match uh to what we've seen in the last. You know, I, don't know how long it was an hour or more um almost two hours um and update the docs if there's anything wrong, um yeah cool.

A

Thank you very much for taking the time.

C

So it's already late for.

A

Me it's, uh we can stop here.

A

C

B

A

Good I'll start recording.