GitLab It's time to Go!, 16 Nov 2022

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: It's time to Go! Episode 6 - Remote Development spike

Description

https://gitlab.com/groups/gitlab-org/-/epics/9138

A

uh So today we decided to pair on where we can go despite to see if we can integrate some support for the file into the agent and and server uh kubernetes agent and server, and we don't know if it will work out or not. But we decided to see what what's possible and what yeah, what we can do.

B

So um I guess I can share my screen started. First, let me show you.

A

So what we have discussed is currently current idea of. What we could do is uh user makes an interaction with the website to start a Dev environment from a Dev file, or maybe they want to start the vs code editor in the browser, and it makes an interaction with the lab API and the desire to start an environment is persisted in the database, and then we have, let's say agent number 10 connecting to the server to cast.

A

Then it's similar to githubs, basically that connection from Agents to server sits there and, like waits for work to be done basically or it's similar to GitHub Runner. That's calling the API and then cast sees that it's agent number 10 and it goes and asks rails periodically. Hey! Is there a should I give it some work or do it? How work for agent number 10, uh and uh eventually there is something in the database in it. That was like yeah. Here is what you should do in some form.

A

We don't know what format will be in, but the the class replies to the no cast processes that they file generates the kubernetes resources from it uh and maybe does some post-processing. If we, whatever you know whatever it needs to do.

C

Yeah clean the editor, for example, by injecting the information about the idea or other things, yeah.

A

And then that stuff is sent as a package to like a fork to the agent.

A

An agent applies it to the cluster and reports back on progress. I guess something like that, then that information can be put back into the database associated with this job record.

A

I, don't know what the terminology should be used here and then on the user interface. The user can see that something is happening. This is the current status. uh Ideally we configure, we are doing the dashboard, and maybe we can have a special support for that kind of information and we can show it there somehow uh yeah. So this is the high level can't wait. The idea that they have at the moment so which are trying to locate this the place in the code base where that can be, can be done.

A

So it will be similar to githubs because it is similar, so I think we should create a new module and that will I don't know.

A

Now that's called the file, it would come rename anything here later, and uh so we will leave the server module and the agent module server module, because that's the thing that you'll be pulling the club API and making the Transformations and then the agent module, because that will be the thing that applies whatever it got from the server to the cluster, so yeah, let's create or we can just copy something like API.

A

This just contains.

A

uh Wait, for example, module.

C

Name, let's call it, um maybe you know yeah.

A

Then we need an agent and server and RPC directories, uh let's copy that from somewhere and then I yeah.

B

C

Will we need an agent uh directly or like? Can we just can use existing functionality, I guess perhaps and.

A

So we wanted to be a separate thing because there's a high chance we'll have a special requirements. You know the special domain business logic that we need to live somewhere and.

C

Even in the ages like, we will definitely have that impact, but things will have pestological Creations too.

A

Yeah but yeah I think it's not the same as pickups I think it will be different because it might need to expose extra ports. It may need to.

A

I, don't know, do extra things.

C

A

That makes sense. Well that's what we are trying to understand anyway. Do we need it or not, I think we should well. We will need it, but uh if not, we will delete it. So let me just clean everything up quickly.

A

um This is stuff that is specific to the thing I copied this code from uh like delete over here or most of the things um server.

A

And that can be your remote just want to have a high level structure, and this will be useful and.

C

Just like talking about the goals, like anything as an example that accesses the kubernetes API like listing the namespace, that's that's all we want to do initiated by the agent right.

A

C

To have a client recall like have this module yeah, well, I'm, sorry, initiated by the server I. Have a server like to be able to enlist the name spaces they give a list of namespaces back in that room illustrate the plumbing that we need and just communicated to the agent and the rest should be good. It's easier to hook up with no questions.

B

C

A

So, uh let's start with the definition of the RPC, because what we need RPC first uh and.

B

C

It's a little less namespaces right, just like what's the namespaces on.

B

C

Is that a good example.

A

Well, I think we can just start like doing what we what we need to do like it's just I mean in the same space as well. uh Just.

C

A question we don't have the file operator and everything configured right, no.

A

No, we don't but uh to understand what we need. We I suggest we start building what we need, rather than building something.

B

A

We I know that we can listen in spaces, but we are trying to discover our Norms. That's what I think we should do. I'd.

C

A

To discover unknown problems that we haven't considered, so if we are building something else, then maybe it's the same in structure, but we will not think about our problem. We will think about time, spaces right um question. The the service should be removed at right, foreign.

C

A

Yeah absolutely thank you so.

B

C

Fine and we can work either way like the goal of the shell and I, had was just to to understand this money like how we would look at that. That's why we wanted to start with something simple: we that would already work, and then we could converge kubernetes, but, however, you think is important to the spider.

A

Let's call it get work, yeah, we're already name it, but for now I get work to do you know and then so we're gonna do that and it will be get.

B

A

A

B

Then it will be.

A

Something like that and he will have.

A

We can look at githubs. Actually, this is the RPC, and this is a stream. Probably here we.

B

A

You can just send an object, and that will be.

A

The request uh we for now we will have an empty empty request and response will have it's.

A

So reply responses a list of yam objects. Let's.

C

Go check, that's the work, we're doing.

B

A

C

A

Yeah, um okay, that's just go like.

C

This you were saying that was the the response right, yeah.

A

C

I mean whatever, however, we communicate the outcome of the the kubernetes operation whenever the way to do that is successor or error. Failure.

A

um So I this is an API that the agent will call on class.

C

A

It- and this is the reply that, because then Spike to the agent and the agent should provision that.

C

So the agent is going to be upholding this. It's.

A

Going to be like long boiling opening that request waiting for a reply, if customers have anything to say, doesn't reply then once it has something, it replies, okay, so that we don't like Paul all the time. So that's probably how we want it to look, and then we should plug this into.

B

Yeah and removed- and this should be I- know that.

A

We need to knowledge I need that as well.

A

uh Awesome! Okay, let's plug this into the product, generator.

B

I think it's here: yes,.

A

And this goes after.

A

I'm just trying to keep them sorted.

C

Do you have to change the uh other all extractor generated.

A

Yeah I'm just plugging into the build so that it generates for the build generates the.

B

Broader definitions and it.

A

Did uh you know here we have that one being for get work request to.

C

Make Command you in making yourself.

A

Regenerate product yep, so we have our RPC uh definition quite simple, and uh now we have uh let's start with the server. Let's clean this up, you just need to copy from.

B

A

uh Well and then.

B

B

We should start replacing this.

A

Doesn't do perfect another package.

A

And this will be exposed on the register agent. Now this is uh agent server. We should register completed so I, don't know how do I do that? I. Don't remember.

B

ah I think it's RPC Microsoft PC. If you see.

A

The RPC that we have just generated.

C

B

This is registering.

C

The the server RPC endpoint for them.

A

To remove this or that's what you want, I think BT server and the implementation will be.

A

A

Our implementation will be.

B

A

I'll create this.

B

A

And I, don't like it to be in separate file, you can't be consistent with other modules, yeah.

A

And this needs to unimplemented.

A

Remote server, that's that's the generated thing as well. Comes from this.

C

File, if you mean unsafe or unimplemented.

A

Thank you why? What is this unsafe thing? I have no.

C

Idea, don't look at it. It's not safe. Okay,.

A

uh So other model and that's the thing we need to implement.

C

So that is just like a placeholder until we have some implementation that fulfills that yeah.

B

Well, the type signature.

C

And contract- or this is the.

A

um Method.Grpc will call.

C

When I mean the the unimplemented dev server and destruct like it,.

A

C

Contains something.

A

Yeah, it just has that unimplemented and grpc, since some version checks that you have it embedded, it will not work okay written there. If you don't have.

A

um Oh, let's just go down, but we don't have anything for.

C

A

um Code, that's our server, let's stop holding for it and we registered with this JPC, sir. It's given to the module from the outer program- and this is the thing that the agent connects to, and so this is the thing that the agent will be calling and basically your business logic goes here, the one that.

C

Points will take the agent. What is the thing? The agent will be calling the.

A

Age of the server this function, okay,.

C

A

This is the Handler of that purposes that the agent will be called um on the agent module okay, and if we need any parameters for the server they they go here and they can be put here and that's it and the module is empty.

B

Doesn't need to do anything so, and this is.

A

Done this is well, scaffolding is done, and.

C

So the part that I'm I'm still missing is how well you know something external to the server like we could figure it out just pass.

C

You know, from from the command line at another, positional parameter to cast that it's like you know, dash dash Foo, and then it takes a value bar like how would we use that you know eventually that will be a listener endpoint from the monolith. You know the monolith, or maybe a JavaScript client calling this, but how do we use that to trigger something to happen here?.

A

Now, for now you can have you can hard code, something here, just return and fix it.

C

And right but I'm looking to learn the plumbing of how well we hook up that for.

A

C

An API endpoint to trigger this every time, medium that our PC endpoint with the server exposes is called. It will make something happen here to pass information to the to make the agent.

A

You in invoke it from the agent side yeah from from that from.

C

The show that's that's the architecture like.

A

C

Yep yep we're going to say here: we want here's, here's a debt file, demo right we're going to pass it to some endpoint, a grpc endpoint that the server explodes and do the work with this, which is transform that and uh what it is, the operator. How do you say it on kubernetes.

A

Well, you can, you cast, exposes an hrpc API to.

A

C

But right a gym is a wrapper around right. We can like yes, so.

A

We can so it would be like.

C

The monolith calling the gem like yeah.

A

But that's not the approach that we want. We want like in the end I think we want the opposite thread fast, like asking rails for something, but so to just to play with it. I think we can just hardcore.

C

It I don't understand that, like how email has for me calling rails for.

A

So it's similar to github's, where agent connects to cast and then cast starts pulling um in case of uh githubs, but in case of uh remove that this this place will have a loop that will be pulling uh around 12..

A

I can't hear you sorry.

C

I hit mute like we have to do polling, there's, there's nothing in this architecture that lets us say we respond to an external event that we will, via that the servers API external.

A

C

A

So the thing is, there are multiple um cast instances. We run mine in production right now, but it will be more and more and more so when you make a call from gitlab you like, which one oh.

C

Okay, cool right, I assumed they were behind. You know Workforce or similar balancers, even like, even if.

A

It's the same time, even it's behind the load balancer, which I think it actually is for that RPC, uh so rails, talks to some load balance a Google Cloud load balancer for that internal port for grpc there. But again the request lands on some replica and the agent may be connected to another replica. So you.

C

Okay, I get it so there's there's.

A

No that's one problem. Another problem is that the agent may not be connected right now in this second, because they have network maintenance or maybe they're, restarting it because they're upgrading it or you know 1 000 data reasons. So your request will come.

C

Out and there's no persistence here so.

B

A

Exactly that's.

C

Why you have to have a pulling architecture.

A

Yeah there is no persistence in the past, but there is persistence in real, so we.

C

Want so Michelle and I were looking for something that didn't exist, that's why, when we did it find it.

A

Foreign is not the best mechanism and we have this problem with githubs and we want to solve it with the we have an issue for that. uh There is a notification mechanism when there is a commit in you know, Italy commits a commit like persists, a commit I think it pushes a notification through some rails mechanisms, or we want to hook into it. So for that we may also want to hook into something like this and.

C

What I was imagining like? We have action table, but there's there's any table. The action. Cable supports websites, but any cable supports um RPC connections as well. So it's like some sort of registry event: okay, here's here's! A registry of you know cast servers I'm going to open, Action, cable connection to it from you know the monolith just like we do for Action table connections to the web clients and say whenever this event happens, then I will invoke a call on this. I can see it via any table.

C

I see there's many Architectural Components that we don't have to make that happen, but.

A

I guess that's how it works. Ideally, yes, we need some sort of mechanism, Central mechanism for just to dispatch notifications, uh but also notification can be missed because whatever and polling is the second mechanism that will catch it.

C

Right, that's assuming you know, a a guaranteed delivery. You know cue somewhere, that is, on the other end right credits.

B

Or whatever yeah.

A

So uh for now we can just have a look that both the server.

B

C

Works, it does the same thing. It's a lot simpler. This.

A

Is, as you see, this is super similar. This is like the object to synchronize for githubs.

C

And right and we were looking at this and then you know, there's just the poll Loop and that's what it does and then in there it calls with the hook to the the.

A

B

A

Is the ah signature doesn't match? Because this is a streaming response and we have a just unary RPC? That's helps.

B

A

B

A

This is the full configuration.

B

A

B

A

And hopefully,.

B

A

And here we can I.

B

Don't know yes.

B

A

You know Define this on that and then.

A

This can be time out if this got tired of boiling. We don't need to return anywhere actually in this case, but also no no response. Otherwise we're done. We didn't want.

B

A

B

A

If there is no area, but that can happen or maybe I don't know, actually in this case we return that response. That's correctly, and here we are supposed to view the response or return and error. Basically.

A

A

Okay- and we also need that boiling configuration.

B

C

See so where did um get up so just pause, Italy, but there's other places in here like foreign instead.

A

um Sorry sorry, what was that again.

C

um So with the github's example, you're pulling Italy, but we all you know we're not going to have our data in our deposit for you. We need to put it somewhere either the relational database or.

B

C

uh I thought I saw some of the other modules that connect to redis. Was there an example of that.

A

C

I mean so we can finish wiring up the agent stuff. First.

A

Yeah like or I just want to provide stuff.

C

For you, here, okay or or the rest API anything well, yes,.

A

Some something like that- um and this will be needed later for now. Let's return.

A

Once you're done, you return done.

A

um This needs to be filled from in the factory here then.

A

Let's look at the agent.

B

B

Be it doesn't use the internal server.

A

A

Okay and then uh doesn't.

B

Need a server! Yes, that doesn't.

A

uh What we do need here is uh top a water. Maybe that's how we can call it.

A

B

A

Would well, we can just call it here for now, but yeah, it's I think it's it's better to separate it and right.

C

So what is the it's.

A

The thing that little worker.

C

And that's the thing that is calling the kubernetes API.

A

It's its job is to establish that it's super similar to the tops. Actually, this is the thing so this it holds various things and then, when it runs, it starts the pipeline of three things. First thing is watching the start uh connects to the server tries to get data back then the second stage decodes the data it got, and then third stage applies it to the cluster right.

C

We saw this, and so the watching is just a blocking call and it whenever.

A

C

Whole return to something is when it will be. You know.

A

A

In helper, that does various things, but in our case we don't need any of that. This is.

C

So the Watcher is implemented as a poll under the covers, as well from the age side. In that right, there online.

A

No, it's well or I. Think!

A

Oh, okay! Yes! Actually! Well, yes, and no I mean yes, because there is a group around it, but also no because each each thing is kind of a long ball. So it connects in a way to reply. It only retries when there's a okay. So it's a long! So it's a long ball yeah, not just.

C

That drawing appall without blocking call in in each iteration yeah.

A

So we use the same thing but much simpler, because this RPC for autopsy is more complex. It it chunks the response into multiple pieces because there may be a lot of files and files can be big, so it cuts them into pieces, but you don't need any buttons or unnecessary, so we will have.

C

I'm gonna ask another question: yeah the so the implementation of the polling in the server RPC. In the point um you know you check the error there and there were the three cases that you could return from.

C

What makes when does making this block like? What is the poll waiting on here.

C

In the no in this on the server side in the five percent, like the poll that we're doing there.

A

C

Or in this in our new one and then they must serve that goal so.

B

C

Line 20 like what is that whole like what causes it to to iterate or or block under the coverage. In our case.

A

So this ball um blocks when this function returns with like continue so this configuration it contains information about back off. So if there are errors slows down and then if there are no errors and we return continue, it continues after waiting for the interval, so.

C

Okay and so in.

B

A successful response.

C

To the rest, API will return nowhere to do like that's where it will be implemented, and then it will just yeah exactly and that will be continued yeah.

A

If, let's now work to do, then this will.

B

C

A

Got it else like send the work.

B

Back, which is the yes, because nothing.

A

And then this then there's a field and then return done.

A

That's the logic that we use here.

C

Perfect I can't get it so that's starting to tie it together. That's at least something for us to start building on and, as you said, we need to definitely write up this architecture that we came up with last night.

A

Something like that.

B

A

C

Discover you well the reason that that process has to happen here is as a libraries we're getting used to do the processing guard. So that can't happen. My real site.

A

So this is the worker and then we we need uh let.

B

Me see, clients I, wish you foreign.

A

A

A

A

And yeah here, I'll go client, get work context.

B

B

A

uh Well, we can just whatever allow the person, okay, I, don't know, and uh it's also work like.

B

B

A

A

um And then, if context was canceled, then we need to exit from the loop.

C

So this this run is going to get invoked from where.

A

uh It will be once we once in above it um from the module.

B

B

B

A

Something like that, maybe so you can also feed configuration into it. If we need configured we probably need configuration, but for now we don't so it just works like this, and then we need to populate it, and we can do it here in the factory uh it needs, so worker is our worker and the worker needs. The client and client will be PC new new.

A

um The client- and this is the config class connection- something like this. This is the connection because that is set up by the outcome program. It's given to the module. You don't need to worry about how it works. It just works.

B

A

That's it, uh you just need to implement it. The worker.

B

A

And you can look at how it's implemented in the tops. This apply the.

C

Workers, that is the work that will be done on the server.

C

A

Right this worker is in the agent it's the thing that applies to manifests.

C

Okay, that that will handle the the talking to the kubernetes API. Yes, is there an example of that is that somewhere in that kubernetes API? Yes,.

A

In the tops in manifest, we have chart, tops or home support and manifestops for.

B

A

Manifest and you can look at this worker to the apply.

C

A

Like one two three, the stages of the pipeline, which is reports and applies, so this is the applied stage and white job. That's.

B

The that's how it applies so, a.

A

Player is a huge you can see how it's constructed in the factory here.

A

Yeah, it's given the clients to the uh to the.

B

A

Basically and inventory, client um yeah just probably do the same thing.

C

A

C

Make our own a new client or.

A

Yeah it should, it should be separate, it should pull it separate and then yeah just you got the you got the work here. The manifests need to parse them, probably similar to what this decode does and then you need to apply them which is apply around you, give it the object, stop light and then apply options how to apply them, and then you need to handle the events like look for.

A

Well, you will see it's a stream of events.

A

They have types like what what's happening, you know and then eventually everything's fine and that's it you you're done for this for this work item, then you can pick up the next one, and another thing to think about is that.

C

So the outlier is what abstracts the kubernetes API interface.

B

C

A

Yes- and this comes from the CLI utils library- from the this cpcli special interest group for CLI from Kimber mantis- and it's what we use to apply manifest to the cluster, it does a lot of things there ordering.

A

um Keeping track of what has been applied and then what on the next situation, needs to be removed because it has been deleted from the desired set of objects and have many other things. So.

C

Yeah all right we got through it, I hope we can watch this video very carefully multiple times, yeah push this to a branch, and then we can take it from here. Just drop the name, the name of the branch in select.

B

On the remote channel.

C

Yes, this is it's very complex, but it does a lot of stuff. A lot of very Necessary Things.

A

C

Perfect really appreciate your help: cool.

A

Well, we actually did out of time for this right, perfect timing, thank you to everybody who watched.