GitLab Environments Team, 16 Mar 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Gitlab Agent development environment setup

Description

Timo and Vlad setting up the GitLab Agent development environment

https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent

https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/developing.md

A

Okay, cool and I'll share my screen now.

A

Okay, do you see the cast documentation? Yes,.

B

A

Awesome um and we're recording already so good. uh Actually, let me do the first thing first, which I should have done before this, but um yeah, let it run in the background and in the meantime, I was reading this.

B

A

From there, the hunger is a good one.

B

So actually, what you can also do is use this um how's. It called basalisk I think which.

A

B

It's like a package manager for bazel or for you know for bazel itself. It's like a launcher for yeah this one. Do you think I should install it I? Did it using this I'm, not sure? What's documented in the agent, it says, Brazil, it says basil.

A

Oh, it's actually Stills, then. Okay,.

B

A

Just clicking on the first link, first of all, right.

B

Okay, yeah yeah I suggest doing this. I also do this um so.

C

B

Don't mess around with other projects in case you have some.

A

A

Suggestion to go here, Brewing stop as a list.

D

A

D

A

How is the update going.

B

This is okay, probably gonna, take a while I guess, I.

A

It used to like, like it used to be much quicker like six five months ago, when I was more active in development.

D

A

Me, like I, don't know one and a half minutes to run the GDK update. Oh.

B

A

It takes like I, don't know five.

B

Ten minutes for me, it's more like 15 minutes to be honest, but yeah and I'm on an M1 I would I mean I guess, depending on your machine yeah, it takes.

A

Forever yeah, so let's go into the GDK yaml away. Oh, it's not sure.

D

D

A

Do we also have I just wonder if we have anything in the example.

A

uh Agent, oh yeah, a lot of stuff actually, um but let's start with just enabled thing.

D

A

Yeah, this is a local, so I. Don't think that anybody seen my token is a problem. um Okay, optional, CI tunnel do I need this for the very first.

B

um Installation I, have it I'm, not sure. um Probably let's do this. Let's do this, it's something we anybody probably want to do pretty early on. Do you have https in your GDK already or nope, okay,.

A

uh Not at all I.

B

I think we have to set this up actually.

A

I wonder if I have engine excellent um okay? Well, this is installing and Unix yep not found and I will need to wait while the Basilisk is installing.

B

Okay, that's still running okay,.

A

While we waiting I guess.

A

What what is this this is my local IP or.

A

B

A

B

Yeah, it's for I think.

D

A

Me, oh okay, I see I, see. Let me copy this I guess so we'll figure it out.

B

Yes and I mean these: are it's the configure, for example, the agent listen address, it's the address where the agent connects to and you can then, if you specify those to some values, when you run the agents on your machine or in a cluster or whatnot, then you can connect to this port um to establish the connection from the agent 2D.

A

Okay, uh do I actually need the host name? Can I just use this? Oh I I need if I want to use it. uh Tell us right, um yes, do I. Why do I need to tell us? Can we just somehow ask.

B

Not sure if you can disable it in chaos, actually I, don't think so.

A

A

I'll check on my console, okay, nice. So we can install from Linux.

A

In the meantime, well,.

B

I guess I mean um it says that TLS is enabled if both the certificate file and the key file are provided.

A

Okay, so if I skip this, it should just work with http right.

B

Yeah we can try.

A

Okay, I want I want to try to skip this. To make this uh simpler, I guess: um okay, we already run on GDK, oh I, wonder if gtk doesn't even download. Actually we can check it right.

A

um Please go away file open, GDK.

A

How the folder should should be called gitlab Agent.

B

Or well it doesn't matter, you can also use the the one provided here. I think there is the gitlab cast no.

A

Oh gitlabcast yeah, that's what I I was asking: how do okay I'm just trying to understand if it's already downloaded or if it.

B

Does probably you need to call reconfigure and then it downloads I, don't know how GDK works here? um I have one which I certainly don't use so I here. No, if you love ksh, and what is this this one yeah, where is oh? No, it's in the services or what's.

A

B

A

B

A

How did they end up there, but I don't need this right, I mean it's.

B

So I think we will. We will start it ourselves, so I I think we don't. We don't need it.

B

It should be here right Gila. If this is sorted alphabetically, it should be gitlab, minus Cass somewhere. So I, don't think that you have the directory.

B

I suggest you just call gilab uh GDK reconfigure and with the with cast agent, enabled.

A

I mean it will do it I'm, just okay: we can actually stop it now. If I recall correctly, update will call reconfigure itself.

A

It just launch I'm, not sure. Actually, if the configure will be enough, another three configure will recreate files, but I'm not sure if it will download the repositors.

B

Yes, I'm actually not sure either.

A

Okay, we'll look ahead while we.

B

So actually I think the oh I.

A

Just realized that half of this is not edited so I'm going to stop this again.

A

So Port is Port important for curve or.

B

No, you can remove it, I think yeah.

D

B

Can just use it your default Port, whatever you have.

A

B

A

Will just remove everything, except for the.

B

For this I think I think these you can remove as well. I think this will be used by GDK to create your cast configuration, but since we're going to use our own actually I.

A

Would like to use the inauguration from the.

B

A

B

So yeah, so let's uh let's leave them then.

A

To wait for it to finish something: okay, let's look ahead again. What's so update, reconfigure restart and it will print that it starts everything uh I can tell logs I can so. This is the port which uh cast a listen song.

B

Right yeah scroll up, then you see it in the example. Yes, yeah, okay yeah. This.

C

B

Cass listens on and that's where your um your agent will connect to okay.

A

And what is this then?.

B

um It's for the API that well, what is it the.

A

Kubernetes numbers.

B

Yeah, it's for the proxy I think the the kubernetes proxy port I think.

A

Okay, honestly.

B

C

A

What does it mean.

D

D

B

Eight one five.

D

B

Maybe it's also the port that gitlab connects to now. It's our now. It's sort of proxies I think I.

D

Don't know I'm.

B

Just checking the API.

A

You can always stop my receipts.

A

D

A

So it prints into the nginx config, which is interesting. What else.

A

There are a lot of things so default.

B

Yeah, it's for it's for the the kubernetes proxy, pretty sure.

A

Oh I should never run uh Ruby Minds, while I'm not I'm, dating something.

A

How is our okay, at least engine exit installs?

A

Oh now we see we installed.

C

A

Okay, that's good! There were some errors, though, that maybe because I restarted it at some point.

B

Well, it says vv159, which looks um which looks weird to me.

B

You see this like updating, git.

A

B

And then there's a double v.

A

I wonder I mean this. Whole thing seems a little strange that it says updating, then installing.

B

Well, it updates the repo and then it just like, builds the agent I. Think that's! What's.

A

B

Can but I think it failed. It failed to clone this 15 9.

A

Yeah, it probably uses Master, then.

B

A

Which can try to see where is it so it's.

A

Oh I just saw it and I lost it.

A

This girls, bunch of stuff.

D

A

A

What is this I wonder?.

B

A

uh What is this file? I was wondering, but yeah. It looks like some kind of built.

B

Yes, it's that's. That's! That's one of the basal configuration files, yeah.

A

All right, so how is the update going a lot of stuff I really want to interrupt it since I changed the conflict somewhere in the middle, but maybe we can let it finish, but.

B

Did you did you change.

A

The I removed a bunch of stuff, if you remember like I, had all of this.

B

A

B

Removed at this before you put this in Ran aboard and I removed it and run it again, I.

A

Didn't abort it that's what I'm saying that I want to? Actually, let's try it I, don't know we'll take time, but yeah now at least it will have the right configuration okay uh educate. So this is just um in the meantime. uh I guess: I need some uh cluster management. You managed you mentioned last time that you origin, Sunrise Tool, which allowed you.

B

B

B

Which is running k3s, which is a lightweight kubernetes distribution in a Docker container.

B

A

You have Docker setup, I have Rancher.

B

Okay, should work I, don't know I'm using Colima, so I don't know what what are you using Colima? It's called.

A

B

Yeah with C, oh.

A

B

D

A

So it is basically the same like Docker desktop, but something yeah.

B

Exactly it's probably also working with Rancher I'm, just being a little bit hesitant because I can't tell for sure, but we can install this and see what works and what doesn't I guess: okay, kites! I! Guess you can just Brew install this k3d! Okay,.

A

Let's do it yeah.

A

Okay, um do I need anything from it like later and any other like dependency or.

B

I, don't think so. No, okay, good.

A

We can return to it later.

A

So does the GDK install the agent from the Basilisk or like? Why do we even have this dependency.

B

I, don't know what she decayed us to be honest, but I'm, just using basil basilisk manually to install different kind of bazel versions without conflicting between projects.

B

And you can just set a bazel version for a certain like directory or your system, yeah, okay and then I as I said: I, don't know what GDK does, but if you run the make files in cast, then it will just use the bazel. You have selected from basilisk and run it pretty much the same as with ASDF. If you know that okay.

A

Yeah yeah, okay,.

B

Same thing, basically,.

A

A

Okay, um I wonder what basal cells is. Is it like a built environment or.

B

Yeah, it's like a build system kind of.

A

B

Use this to to um to manage the the project in terms of like the go things, but it also builds our images and handles dependencies and these kind of things it does a lot and I'm no expert in in it. So yeah.

A

Yeah give me a second close, the window. Yes, it's way too cold.

A

Why is this here.

D

A

It actually looks like it's run. Past runs faster now when it was just when it just has finished some of the operations before um okay, while we're still waiting anything height.

A

Loyalty controls nice, um we're always still waiting, will I need to install anything else uh later on or.

B

um No I think that's it. If you do, you have Cube CTL, yeah tools, yeah I. Think then we're we're good.

A

B

um Do you have you also have the runner installed, uh yeah, yeah, okay, because.

D

A

Been given the works.

B

I think yeah, because what we can do afterwards is we can configure a project to have CI access and then use um yeah use that to connect to the cluster using cast.

A

A

Then I guess just waiting.

B

Yeah, how far is this and k3d is installed already yeah? It was super.

A

B

Good I mean yeah, we can create a cluster here, so what you can do is on k3d.

A

B

um Hi this is so annoying it when I focus the zoom window. I have this bar at the bottom, so I can't see your prompts. Okay.

C

B

It's k3d and then cluster create, and then you give it a name.

A

In cluster noise.

B

Space cluster space.

A

Create uh I don't know, but that's yeah. Okay run this I'm sure. If this is supporting okay.

B

Okay, so yeah we need a Docker socket. How does this work with launcher? If you do Docker PS here? Do you.

A

Oh yeah need to go to Rancher.

A

Into the second output on this window, so.

A

A

Too many things I, don't know.

A

In the uh let me close, it.

B

So this run this launcher desktop has a kubernetes running. Is that.

A

Yeah I found the right window. I guess maybe honestly I didn't touch it for quite a while. So.

B

A

uh Update available restart.

A

I wonder more like does it? Does it work right now.

A

A

Just in the past it was so easy you know, but we're starting on screen.

A

Oh, it even tries to do something with my cluster and the gcp.

B

A

Probably just scanning.

B

Your um Cube config locally in your users directory probably it.

A

Says kubernetes is running not just even yeah.

B

It seems to have some kind of cluster or something.

A

Let's just restart it I don't know in the past, I was able, like I at least needed, to run the uh sorry to run the runner and everything in it. So.

A

The radius is shutting down.

A

What is dashboard.

A

Will restart itself I, wonder and how oh.

D

A

We have everything configured in the mirror, actually G the deterioration figure I, don't think we need it, but.

B

Hey, let's give me a second sure, foreign.

D

D

A

Thank you, I'm.

B

A

So in the in the meantime, I restarted the Rancher okay, my Cube CTO.

B

A

This conflict to like separate file- and it looks like I, managed to create a cluster. Yes,.

B

Exactly so, you can do a cube, CTL um cluster info, for example,.

D

B

Okay, so that you see there the local port and everything you can do, Cube CTL get all minus uppercase a and then you have you know you see everything that's running in the cluster which will be useful afterwards. So here you see your cluster state. So that's good! This.

A

Is it way too much or it's like a normal? No.

B

No, this looks normal. It would be bad if the status of these parts at the top will be like failed or um crashing or whatever. um That would be a problem, but they all seem to run fine. You also see the restarts so they're not like restarting, and it only appears as running I think this. This is good.

A

This one is the restart from now.

B

Yes, but this is a this- is just a job. This is completed. This is okay, it's okay! Oh okay! It's okay!.

A

Okay cool, so what do we do? Next um open jdk I mean open the UI.

B

You restarted everything in you, restart GDK, right.

A

I reconfigured, but maybe didn't restart okay.

B

I mean, theoretically, if we have this and we don't stop Cass I would assume it will just work. I, don't know actually.

A

What do you mean it will just work, is? Does it already know where to connect what.

B

I told you in the beginning is that I usually start cast myself using like peso whatever and now GDK started, cast and I wonder if this now just works even without having https and everything in nginx or if this won't work, I don't know we can try. So this seems to be running. Can you lock this? Can you do GDK tail for the service.

A

This looks good. Yeah works. Fine, yes, now our message is very small. English.

A

Do you have any project we can test like I, don't know any functionality of the kubernetes agent.

B

You mean directly in Cass in ngdk, already yeah.

A

I, don't know no, we I I, don't I, don't mean like directly in the GDK, but more like do.

B

We have any templates, for instance, yeah I, don't know about template, but we could just create a very simple project and I can run you through. What's there and kind of you know, explain you, okay, so new project, then um yeah you can create a new project.

A

B

A

B

Root, namespace doesn't really doesn't matter.

D

B

Yeah, read me: it's okay or.

A

Yeah doesn't matter yes,.

B

Okay cool, please.

C

B

Me yeah, so if you go to infrastructure on the left side and then kubernetes clusters.

B

Yes, you can do this, but now you don't have any clusters any agents configured yet so what we first need is well, if you do this, this is like the old thing, I think: okay, what we want! uh We have to create an agent configuration file and in the beginning we can just have an empty one, but we need to get the path right. So if you go to the web IDE for this project, okay,.

A

A

I wonder if it works. Okay in the GDK I.

B

Also, I use it regularly. It's it's okay, I.

A

May remember that I had some troubles. Okay,.

B

A

Least, this is the old web ID right.

B

Well, yes, yeah, it's a legacy one! So here you can create a new file and the file yeah yes, and the file needs to be in Dot, gitlab, slash agents, and now we need to put the name of the agent. So you can give this a random name. Pretty much I usually use the same name as the cluster I have locally, so that I have like a match, um and it's easier for me to debug, but I guess this is so it's first test in your case. Yeah.

B

Yes and then slap, no, no, that's not the file. So sorry, it's a folder and then in there, okay. So in here we need a file. Okay,.

A

I need to do this again.

A

First test and the file name.

B

And then it's config.yaml and yaml with an a it, won't work without the a for some reason. Okay, now, okay, cool and this I mean we don't need much more to create an agent than that. So you can just commit an empty file.

A

Oh okay, uh as.

D

B

um And then you can go back to this cluster page we've been before.

A

Yeah once I find Project yeah uh infrastructure clusters.

B

Okay and now we need this token there, this.

A

B

Yes, yes, you also see it in the top. You can just copy it. Okay, yeah and you can I store this in a file locally um in my cast folder just for ease of use and then reuse this for this agent, so I don't have, because you only see this now once.

A

Okay, let's actually finally open the project right well, open, let's shoot! Okay!

A

B

One right um yeah in here I have a file which I call I, don't know my agent name.token, okay,.

A

Well so, first first token right.

B

Yeah I mean the file name, doesn't matter cast the agent won't know about this, but.

A

B

A

So, okay- and we just posted here, cool I'm, not sure if it's safe, what to ask a bunch of pieces. Okay,.

B

What's next, okay, um now you can go, you can close this, uh the the model window there and in GDK uh gitlab, okay- and here you see the agent, so it's not connected right, um which is what we expect, because we don't have an agent yet so now you can go to um some terminal inside of the the project we have.

A

um Inside the project they mean I need to clone this one or knowing.

B

So for inside the cast project: okay, where we just save the token and.

D

A

Okay and I guess I can do.

B

A

I mean oh well, too many things called Agent well.

B

I think what we, what we try to do now is we expect actually that Cass is running right in from GDK, but we need an agent to connect to Cass and I. Don't think that GDK is creating an actual agent I, don't think so.

A

What do you mean creating the actual agent.

B

Running an actual agent I think it only runs Cass, so this gitlab agent project you see on gitlab has two components it's cast, which is the kubernetes agent server.

B

This is I, think what you see and well actually I know it's gitlab Kate's agent. This is not an agent I think this.

A

B

Yeah, yes, but we need to start an agent now which connects to cast which uses cluster so whether to make or no usually I mean. Usually we would use some I, don't know Helm and install the agent inside the cluster. But what you can do now is we can start it using bazel and you can run bazel run.

A

B

Like this right, no it's b-a-c-e-l b, a c uh like set. Sorry sorry, oh no I said this no I said like the last like sorrow, set.

A

Let me see uh right: okay, let's see okay,.

B

And now we do slash, slash CMD, slash, H and K.

B

Okay, now, when you run when you press enter, it will compile whatever you need using bazel, and it will run that binary basically, and we need a little bit more than just that. But we can see the errors and like go from there, but we need to specify a few things like where Cass is located.

B

um The pod namespace, um the name of the pod, which are just two environment variables. It requires, and we also need a token um okay saved in the file before.

A

B

So, let's give this a second too.

B

A

Okay, cash address and token file right.

D

A

I assume first address.

B

Yeah, you need to do um I.

C

Can't see it again.

B

A

Me see I I. Let me change this.

B

Oh yeah, if you do this yeah, this will be yeah. This is great! So after the H and K, you need to do two dashes yeah, two dashes and now space, and then the cast address with minus minus cast address. Okay. So so yes, and then here you specify what we put into um what we put into the GDK configuration this one.

B

D

A

I guess: there's Dash.

B

Token file, that's asking exactly, and then you provide the path to the file we created previously. Yep yeah now run this and let's see okay and now this is the environment. Variable I told you we can just um we can just use default like pod, namespace equals default.

B

Foreign and I think we also need a second variable, which is called pod underscore name, and we can call this gitlab agent or something killer. Minus hn I think it doesn't really matter. Yeah.

B

Okay, maybe put an absolute path: I, don't know if it's going to switch direct I, think it's switching directory section. We can just do dollar PVD and then okay well in the command before like.

A

Yeah I think, first no.

D

A

A

Like this right.

B

Yes, probably won't work, yeah.

B

um Yeah, it must be HTTP yeah.

B

No sorry, it's not HTTP. It's grpc colon, slash the grpc interface. There.

A

Yeah, maybe that's why we need.

A

Okay, does that look like it works.

B

Yes, nice at least I'm not seeing any errors, which is good.

A

B

I think what we I mean, what we can do now is we can use github's features. Maybe if you want Okay.

C

B

As a very basic example, we can put the Manifest files into the same project here so we can create. Maybe you go to the web ID again and let's create a folder called, manifests. Maybe.

B

Like this right, manifest yeah maybe create a a file called um Maybe namespace namespace.yaml.

B

A

B

We need we, um you need to copy some namespace uh kubernetes.

D

B

Okay, I can also send you one or you have one: okay, cool um yeah. That's that's good uh yeah, there's a typo in your file name, but it doesn't matter um yeah.

B

So, okay, you can also open the the agent configuration file again because.

C

B

That we actually want to use github's features, I! Guess it's easiest. If I send you a uh which you can use.

A

Is it documented somewhere.

B

Yes, um there is a github's workflow documentation.

B

Oh boy, why doesn't this.

B

Yeah, it's pretty much this, which I just sent you in Slack.

A

By the way, you only see one my screen right, you don't see yeah.

B

Volume one with the gitlab.

B

Okay, so in here, what we basically say is that we don't want to use githubs and we have a manifest project and we give it a project ID in gitlab, which is wrong here. It should be H and test actually and not. Oh.

A

B

B

Cool and then it's in manifests yeah and then every yaml file, basically in that directory and every time we will do a change, the agent would reconcile this configuration. Okay, doesn't.

A

Matter that I removed the a since I didn't use it here. Oh.

B

Yeah exactly yeah, okay,.

A

B

A

B

Can Commit This to to Main and.

A

Oh, why did I do it now? I need to create.

B

Yeah just create a merge request, not too bad.

A

Except every time we need to wait for the webpack.

B

I think you can somehow turn this off right somewhere, but you.

A

Can it will consume more memory, but, yes, you can merge.

B

Okay, now, if you actually switch back to your agent logs, you should see that it reconciles something if it all works.

B

Yes, here you see that it received the namespace kind development and so on.

B

And if you, if you then use Cube CTL, do you actually know like K9s? No, okay,.

C

Can you open it.

B

Can you open a new window here in the terminal yep you can Brew install Brew install K9s?

B

A

Yeah Hunter was telling me but I guess you talked like taught him this wonderful thing. I.

B

Don't know if I told him or yeah, but now we can just run around K9s and we will use the cube context. You have set locally by default, so that's probably the one we have and it's a browser for you kubernetes um cluster and right now you see the parts and you can switch between the objects using colon.

B

So you can press colon and then you can say plot, for example, and then you would see parts or you can say um namespace yeah exactly and now it shows you all the namespaces, and here we see the development namespace which we just created before yeah.

A

Awesome foreign.

B

Actually is using the CI tunnel. If you want to, then you have you know you have both things. Try it out, yeah. Let's do it. Okay,.

B

So um let's go to the web ID again.

B

A

Wonder if I can enable the vs code locally um so do I need to open this one.

B

Wait give me a second.

A

B

Okay, um you need to open well, you can create a gitlab CI yaml file with the contents. I sent you via slack. It's just like a random example, which I just had available.

B

So here what we have is just a job called Cube CTL, but the name doesn't matter, but what basically gitlab does is it provides you a cube, config file and there's a context set for you or you can choose a context which is named after your project, colon agent name. So here on the line you have your cursor, you can say, root, slash, H and test on line five here.

A

Oh okay, yeah, but you already did it for me right.

B

No, no it's wrong. It should be agent test, not H and K minus test without the minus, uh oh yeah yeah, and then the colon in the end was correct. We need to tell it which agents, because you can have multiple agents connected to your project, and here we called it first test: oh yeah, yes, and then the other commands here are just printing out the cube config. You can remove this line actually and then it runs some random Cube, CTL commands.

A

um No nice connection.

B

To your cluster, basically, this job will connect to your kubernetes cluster.

A

B

Yeah, let's try this.

A

D

A

All changes committed, Pipelines running they're, still waiting for the docker image.

A

A

It does work right, I mean.

A

D

They want to do is.

D

A

You cannot can.

B

Is your, is your Runner itself running a Docker container or how is your setup.

A

Yeah I mean normally.

A

uh What do you mean.

B

Like did this runner ever work, do you know yeah? It did work okay, but.

A

It was maybe a while ago, but.

B

The surrounder itself running inside a runner inside a Docker container, no okay, so.

D

B

Your Runner, no okay,.

A

A

I'm just trying to understand how why can I do it from here, but but the runner itself doesn't see it. Oh actually, let's try.

D

A

That is interesting.

D

A

And it looks like it's running and it should run I, don't know properly. Foreign.

A

A

B

I mean I, don't know how your Runner is set up. Probably maybe it's the permissions from the Runner process user that doesn't have access to your VAR run. Docker socket.

A

B

B

Is this socket there actually.

A

B

No and that's the socket it tries to connect to the runner.

A

Yes, this is what I took from the runner locks. So what I'm trying to uh understand this?

A

Why does this work.

B

Yeah, probably it knows how to Call to um to the docker demon or whatever is provided by.

A

B

Yeah, so what is, if you do cat, if you look at the file of the docker config.json file in your home directory, what does it say? Yeah, that's.

A

What I was looking for? I guess.

D

A

Not much okay. What is there.

B

D

B

A

Okay, so this is probably the last right circuit.

A

A

Okay: uh let's try it I guess.

B

I wonder if I, maybe you can just Sim link them I, don't know not that this is a particularly good idea, but.

A

No I thought that maybe.

B

No in the runner yeah, you can configure it there.

A

Actually, uh into.

A

um Actually, I wonder if I'm not the first to person who sees this issue because Rancher is a default thing. We recommend for running Docker images.

A

So it is strange that it does work out of the box. I guess.

A

Set up Runner and you can config file.

A

Okay, so I do have this already.

A

Okay, now we recommend to write to use it very differently. I guess so. This is register.

A

A

Those those document uh Docs were really updated since the last time I tried them.

A

Maybe it's not the best use of our time. I guess I can try to debug it alone. Yeah.

B

A

Yeah and I would like I would expect everything to work right.

B

I think so yeah I mean it looks. It looks good so far and you can have a connection already from cast to h k in the cluster. So that looks that looks good.

A

Yeah thanks for helping me with all of this yeah.

B

A

I'll stop the recording when I find how to do it. Oh yeah! Stop sharing, stop recording nice uh or not. Why.