Layer5 Meshery Development, 14 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: MeshSync Weekly Meeting (July 14, 2020)

Description

Kubernetes sharedInformers, custom controllers, and fingerprinting service meshes.

A

A

Hey good morning, I'm sorry I was uh been here. The whole time I forget didn't know it was past time well, you're, just taking some hey banayak.

B

Yeah headley good morning.

A

I'm putting way too much of an I in the last part of your name.

C

A

It's shinde, oh okay, yeah! Well, no, I mean when I say I think when I say vinayak, I'm saying I'm putting too much of a an I it's more like um vignac, no, not.

B

Right, the first time.

A

Okay, good all right, good man boy, I was talking with uh well with um well, even her first name is, but I was talking with um shreddy and uh she popped my bubble, and she told me like just you know that I don't know how you know half to most of the names that I'm enunciating or about you know, aren't quite there and then she said. Well, you know technically, like her name is more like.

A

Oh man, it's more like it's more like.

D

Shitty I like, like the r the the r is, uh let's see.

E

I see, oh boy, that I don't think I can yeah. I think it's it's.

A

The thing is like I'd, be willing to give it like some healthy practice. It's just that that name in particular it's so dangerous it's! So it's just.

A

Nice well, abhishek is with us um vinayak, whose name I can announce it with confidence. Now, as with to abhishek and vinayak, we were able to kind of ad hoc meet for a little while yesterday um to review some progress that um some initial progress that arip had made and then kind of a second um crack at that that natish has done and and has.

A

Essentially, I think the team has arrived essentially at a skeleton of uh well of uh well the initial structure of what not just mesh sync probably looks like but um and nitisha. Maybe you want to um describe it.

F

Yeah so essentially around the code structuring and how we set up the meshi operator repository so that we can, we can basically treat it like a monorepo for all the components of this meshi kubernetes deployment, which, which kind of include uh the the meshing piece. You have some policy engines and you have a whole bunch of other other components that we'll be building out, but the idea being that we can share share the code with the code across all of these projects.

F

So we have a we've started with the skeleton for mesh sync and adip is going to fill out some some more bits for sinking istio, um but yeah we're building on that, there's going to be certain patterns to use in terms of identifying a mesh. So it's it's it's still. uh It's still alpha, so it'll keep changing so yeah. That's essentially what we have you can take a look at the link uh lee sent.

F

Maybe we can enhance the readme and start describing the directory structure.

A

A

Sense, try to I'm going to try to move um some of our conversation to the bottom of the dock. All right, I'm going to try to help clean up our docs concepts.

A

Yeah and so the to the extent that we ended up using something like oppa as the as a well, maybe as a component or maybe the component for the policy engine um oppa can be, I mean oppo would fit inside there right because because it um is deployable in a couple of different ways, um I thought you could it's been like two years since I earnestly looked at using it, but we had treated it as a library as a package last time that we went to go use it so.

A

A

And then maybe I'll poke, poka adip and see if he's.

A

A

My secret, if I could yeah there, we go.

A

So maniac you had um there's a couple of research items that you were looking into and I think you you made a comment recently.

B

Yeah on some of your findings, you want to talk.

C

A

C

It was around shared informants, uh tries to reduce the number of api calls to the kubernetes api server. That was our main concern right. Are we going to overload the kubernetes master api server with a lot of queries, so the job of shared informer? So, at the end of the day, it's going to be a single watch uh watching on kubernetes api resource api server for the resource updates and multiple controllers, along with our own controller, we'll get the update for that.

F

Yeah, I think that's what we've settled on adip is going to do that for listing crds right now, like uh the cubecard will get crds, so we're gonna start there so that we can look for istio.iocrds to signal that sto is running on the cluster, but yeah. We will stick to our shared index and farmers.

C

Yeah yeah yeah yeah, so we initially had a concern. Is it going to.

A

That makes sense, and so that's um that's essentially the design of kubernetes itself that um and that's the design of a shared informer that I mean yeah. There's.

A

uh uh There were, maybe there were kind of two parts to that conversation. I think one was hey: how do we um efficiently?

A

How do we perform this efficiently in large environments? Another one was uh well or part of that was. How do we, prime the cache very very like very quickly, but while also not overloading kubernetes.

F

How how much does performance matter with respect to mesh sync.

A

uh I um I it might well there's a couple of there's two ways in which I think um it could be an issue. um One is when you think about it in context of of a piece of management software. Look um like here's, a here's!

A

Here's a relatable example is like you're, using a network management system, it's using snmp as a protocol to to look at resource utilization on of a switch or on the device, and you set it to poll once every minute or once every 30 seconds and at some point the management system becomes the issue, because it's asking the device so often like hey how much free memory do you have in your routing tables and turns out, like the the pro the small program that runs to figure, that out takes a little bit of a little bit of resources and so um and concept here.

A

uh So we were just trying to be on guard about something like that, like hey, if we're now granted, that was that's a polling system. This is more eventing and so the the system itself kubernetes, is a bit more in control about how you know how quickly it's going to respond.

A

If there are you know, if there are thousands of objects, we may not be interested in all of them, um but if we're trying, but if we're also trying to design a system in which the the human, the user ends up feeling like they're, getting pretty much a real-time perspective of what's transpiring in the when we're sourcing, you know from here for sourcing some, some metrics from prometheus, maybe some logs from somewhere else and then just updates about objects and their status and deployments and things that yeah, I don't know yeah it's it's a good.

A

It's good for us to have the to take a look.

F

I yeah, but I still think uh shared informers would be fine for the use case because uh you basically in in this use case you're. Just I'm just talking about the mesh sync controller. His job is uh he's just a scanner.

F

So so, even if uh there's a delay in uh the shared and former like a plum milliseconds, it shouldn't be a big deal. Yes,.

A

Good deal yeah, these were, and these were questions that we'd raised up. um I think just before on a call and one of our early calls, I think, would be before you'd um we're on, and it was just uh us acknowledging like okay, hey. How do we and- and I guess in some respects, um to to what vinaya could just how? How you were just describing, how shared informers, work or well is a shared informer with a shared index. Is that the right um or is it sort of one of the same shared informers? Have.

F

They're pretty similar.

A

That, even if there is um like, let's say that the delay goes up to 10, milliseconds or, like you know, we get a we're seeing worse performance than we, otherwise were that's not likely and or that really actually shouldn't be gonna say it shouldn't, be an artifact necessarily of mesh sync, because that's a job that kubernetes is having to do anyway.

A

Now that said, hey if no other inf, now I'm going to use the wrong terminology, but if no one, if there's no need for measuring or kubernetes to be running in former or um running informers, for a bunch of objects that it otherwise wouldn't be um informing on that, then I can see us. You know adding a small overhead.

F

No, I didn't, I don't think I think it's pretty efficient. I I mean it's a very.

F

I guess I shouldn't have raised this point either, because shared informants are pretty performant. Okay, so I mean the other alternate alternative is overloading the api server with watches.

A

Yeah and this one's more so I guess maybe yeah say more, I could see it seems like this. One would be more performant. The shared informers.

F

D

And the resources are just lower, so nice.

A

It sounds like um it's about as good as it probably gets anyway. It sounds like you know. Kubernetes itself has been designed in consideration that there's other systems that are, you need to know what's going on inside of these control loops and that it's designed to inform as many others like in some respects. If there's you know one to ten thousand um uh process and like subscribers for for a particular update on an object that the the overhead there is negligible between 1 to 10, because it's going to send out the same event either way.

A

And so yeah, so I mean I I'm not sure if there's yeah other than like shared memory like so uh another question lee manik.

F

um How what do you think are the mechanisms for identifying a mesh? We we recognize. Crds is one the the other two are not concrete, but we were thinking of searching for deployments and then searching for images to figure out what version we're running.

F

Is there anything else we might use for? So I I everything I say is from a very istio-centric perspective, because I haven't played with a lot of other meshes, but that's something someone has to take on and say this is how we can identify kuma. This is how we can identify console connect.

F

So if there's a common pattern to these things, then we can build out a fingerprint fingerprinting package and an interface that we can reuse or use across all the meshes. But I do want to abstract these things out and you can. You can use the builder pattern to say the first one is fingerprints with crds. The next step is fingerprint with deployments, and the third step is fingerprint with images and that fills out the final object model saying that we know it's present.

F

We know the control plane's running and we know the version something like that, like we have to build a model, an object out of what comes out of it.

A

Is the builder pla when um the builder pattern? Is that a go thing, or is that you just are you just mean like in general, like um it's.

F

It's in general, so it's it's what I was what I pasted on slack sure.

A

You don't have to uh that makes sense. I was just wondering if it was like a specific.

F

Yeah, so it's it's not I mean it is used in go kubernetes. Does this a lot, uh but the alternative is the options pattern, but I feel with options. It's it's not going to be that clear options is great for setting up uh config values or some structure values uh like in addition to the defaults or overriding defaults.

F

uh The builder pattern is essentially where we're saying fingerprint on crds. Whatever result you get fingerprint on um on on deployments images, and we can just chain all of these in in order.

E

A

Well, okay, so let's think about that for a minute. So what makes one mesh different from another, so I mean what there's different deployment models, some of them sidecar proxies, and so, if you're, looking at a service, you might find a you know a proxy in the same pod. All right! That's interesting! That doesn't I mean granted if the proxy's image name is istio proxy right. It's probably you.

F

Know the other thing lee is that it could be that the mesh has not deployed any sidecars yet so you can't assume that their customer resources deployed or there's sidecars injected, so you'll have to base everything off of the control plane, and so this.

A

Is yeah totally um and that specific example that I was giving is?

A

It um is a good thing to examine so that you can answer a different question, which is more or less the thing you just said, which is hey what services are present and of those that are present, which ones are on the mesh which ones are off the mesh and part of the like the concept of, or what you were articulating earlier about sort of tiered discovery is, is like that's one of those subsequent questions that you would kind of ask later in like in some respects it kind of I mean it actually does matter eventually, if there's no control plane and yet there's some proxies that are clearly istio proxies side card to you know it's like hey.

A

You've got rem residual artifacts of a prior deployment. Okay, you have an issue but, and so yeah so so identify so the question then how to uh what? What are the? What? What are the most common.

F

So so, let's, let's focus on istio, I think uh crds is going to be one of them, because all the all the meshes that use custom resources, we would just do a cube, cuddle, get crds and search for the istio io or say for istio. I think once we have identified that istio is the control plane is running. We can kind of integrate the sda cuttle uh ctl code and basically like what sdo ctl does with version that could be used. Actually we could start with that too.

F

We probably don't even need to look at crds. You could just integrate a search for your stereo is basically using your ctr. That's why.

E

F

Reinvent everything right yeah.

A

That's uh that's exactly what I was going to say when you were done not talking about it, yeah yeah. Only unless they're doing something really, you know like really stupid or something, but even at that, like obviously it works for and and the thing about and like the thing is like, even if it is you know, was that um well they're going to guarantee that they're going to sustain that implementation, they're going to make sure that it works from version immersion, so yep.

F

I think yeah we, it might be a good idea to just go with that. Instead of the but yeah, we should still like what adip is doing. uh We we need to. um We need to expose that function so that we can use it for more types other than crds or whatever. So we we have. We have this module that we can use to create shared informers when we need it, it might not be uh urgent.

F

I think I think the first step would be now that we think about is is using using the istio ctl approach, so we can import their package and uh just basically invoke that version command. Yeah.

A

It is, it is yeah, and then we will find like in istio's case specifically because they are putting so much off behind istio ctl, that it is the case that there may be some other or or there are a number of other uh functions. You know commands that they've included that yeah they'll be good leverage. Now the linker d- um hey not dissimilar in that the linker d utility, the command line does quite a bit like they expect that you're installing using that command line as well.

A

um I think the issue with both of these is, I think that there's an ist I mean- and I get well so so there's an istio ctl go client, oh, no, um that's the wrong way! There's a yeah! There's a gold package for for the.

D

For the ctl or for the ctl.

F

Is part of the sto repository, so you can basically just import that code. So if I, if I just look at steel ctl and the cobra commands whatever it's invoking, I could just invoke that version function or if, if they expose one, it's.

E

F

Api- it's probably a rest call to pilot and pilot responds with the version. So when linker d is going to be the same, the cli is talking to the uh linkedin control plane. I forget what it is the api server that they have and that's responding with the version. So I think I think that that might be a good way for things that we know we can. uh We can leverage their own packages where we do not know we invent our own.

A

Here's a here's, a brain tickler, maybe is.

A

I mean just and like some of these questions that I ask, are you know they're in you know I mean some of these can be set aside because, like hey look, that's too tough and so you know note taking we'll move on, but so how do we support um or understanding that that um methodology may change from?

A

And maybe it's like the likelihood so low that doesn't matter but from istio 1.0 to istio 1.7 um do.

A

Yeah, how let's say that it did change three times in those versions, and so we and we wanted to support going back that far.

A

um What mechanisms are there for dealing with importing three different versions of the same package.

F

They are backwards, compatible, that's a guarantee that istio makes with the same versioning.

F

So uh if they don't, we open a bug, because that's that's the breaking feature unless they go to sto 2.0 and then things change, uh but as long as they're in the one dot version uh we we can assume that everything will be supported.

F

um We could we could use the master branch or the last table release and keep upgrading if we have to, uh but I think I think we can assume that it is backwards compatible. They don't get rid of, let's say uh objects like I can't remember, I think, authentication or whatever they had.

A

Like uh or like mixing.

F

A

Yeah well and that's a good uh yes, I like that answer. It does put assigned faith to and let's say that they, okay, like you're, saying they don't um you can go file. We can go ask for them to do something to the extent that they are never going to, and it was a major issue for us and we said we're not going to use like one thing to do is that we fall back to more generic patterns of like doing uh grabbing all the crds looking for ac or something something and then.

A

Which can be written up in kind of the tiered discovery model, but uh but is it pos? This is coming from a place of ignorance. If you can't tell some of these things is, is it possible to import on the same package at different versions like.

F

Yeah good question: no, I should take that back.

C

I think, uh because.

F

Of the study go ahead.

C

You know uh yeah in general, go packages themselves, maintain the version in their path right, so they will be slash, v1, slash so like github.com, slash, kt, slash or co, slash, v1 and then v2 v3, so that same package in the different versions can be implemented or imported. But that's up to the maintainer of the package uh how they are doing that, but.

F

I don't when I I don't think that's possible because of how go mod works. uh Gomar work is gonna pick up the minimum. I forget what the word is compatible package, so if, if you give it 1.5 and 1.2, it's going to pick 1.2 as the version, so that's how go mod works.

F

Here's use this item. This is curious. No that's just uh it's! It's gonna do your go-get for that release version and then, uh basically, in your ghost sum, it's gonna add the checksum or actually yeah. It's gonna add the version that it's using. uh Sometimes it uses the commit charge or the checksums.

F

So yeah, that's what I was referring to is: let's say we, we import, client go and we also import another third-party package which depends on client go so so we could be asking for 1.4, but they depend on 1.2.

F

So since we have to support both uh it'll pick 1.2 because of the same versioning, it's it, it's assumed that it's backward compatible. uh So they would pick whatever I mean it would pick the minimum that's supported across both of them gotcha.

D

A

Okay, uh um is this uh the last time that someone had looked at this? It was either that, like there wasn't enough support for things that we were trying to do or, and so I guess I'm wondering with some of the dates on. I wonder.

A

Maybe some of these libraries don't need to be updated necessarily from release to release okay.

F

Yeah, they probably don't the only thing. They're gonna change is yeah that apis okay. So so previously they used to use product protobufs, and that was becoming a problem. So aspen mesh created the client go that nera just had set up, and I remember contributing to it and then so that we can support it with the operator framework or like building our own operators.

F

So so then they they brought in the aspen mesh client go and made it part of istio client go so now they actually support more native types that have that basically implement all the methods that make it qualify as a custom resource okay, so they might not have every type, but I don't think that's a concern initially for us, okay, but I think they have everything. Now I mean they need to because of the whole operator pattern that they have.

F

They have picked up, but I could be wrong.

A

So um educate me if you would so this we're looking at a client go package for um for interfacing with istio in here the under the apis.

A

Well, I guess is the packet, I guess I'm not under apis client go package inform what what's this? How relevant is this to what we're talking about with.

D

F

It's an informer for their uh their customer resources. I I don't, I don't think we need that, like, like, I said, the custom resources may not be deployed, even though the control plane is running.

F

So I think it's it's better to query the api server using that istio ctl, whatever they use under the hood, and I mean I mean pilot and ask pilot what version are you running? Are you up and running, or whatever information it gives back to us? We we massage that information and kind of uh convert it to our abstract model or object that we want to use across meshes.

A

So sounds sounds good or perfect. Actually, my my question here is more uh just uh one of one of uh learning like uh oh okay, uh just being genuine like hey, hey, we're, hey, I'm just learning about shared informers and we're, and I'm I'm learning more about the anyway. It's just more of like uh what what or what are.

D

They doing here what is this they're doing the same thing: adipose gonna.

F

Do they've just created a package for it so that they can reuse it across uh different places. This is probably for the side cars uh where the sidecars could be watching for resources. I don't know whatever this is, but yeah we're going to take a similar model where, uh where we were thinking that you would have pkg in this case, they have informers, we would we would have a similar directory and blah blah blah.

F

I mean this is just uh they made it modular so that uh they can follow the drive principle and they don't have to basically reinvent everything.

F

So, nice, okay, but if you're, if you're interested in learning about shared information, there's a lot of uh articles around it and I'm sure when I knows some where they they give you the the diagram like it's illustrated on how things are watched and added to the cache and how events are created.

F

So so I I think you can, if you just search for shared informers, kubernetes it'll talk about it.

A

I'll get a better okay, nice and then so for the most part, like the purpose of probably that code that we were just looking at and kind of the client go in general was more, so is more so to facilitate the istio operator.

F

Yeah or any other operator that depends on sto like when, when I wrote an operator at a firm that wanted to uh basically list virtual services, we used client go because client go had the virtual service, dot list or dot watch or dot get option gotcha, because it implemented that regular interface.

F

That kubernetes asks customer resources or their types to implement.

A

So there's some so to your ask, there's a little there's, maybe there's a little bit of thought to be given. That is not a critical need at the moment, but a little bit of thought to be given to um a builder pattern for achieving. Potentially, you know composite fingerprints.

A

You know the initial use case is for identifying the fact that a mesh is deployed, although that builder, to your point, like you know that builder pattern probably isn't totally necessary for identifying the control or pro probably not based on some things that should just be available, but but maybe in the future about answering a bunch of a lot more intel or deriving a lot more intelligent um insights that, like you, know, um yeah.

E

A

It's work, you know, you're running two different you've got multiple versions of the proxy of the proxy running in a given match or whatever those things are. But okay.

A

um Other things that we need to so.

A

Okay, there's a couple of things to two things that might end up going by the wayside and one that I would think would be the first to go, and that's the there's two things in measuring today that kind of achieve similar synchronization with kubernetes. But the architecture is not good and so, and this actually probably needs to be turned into a user story uh to see if mesh sync can just achieve it for us or see how it achieves it.

A

And that's the notion that every time you go to run a performance test from measuring that there'll be a there's the need to to snapshot uh what the environment, what the infrastructure looks like at that point in time, um so that that can be snapshotted and persisted along with the results of the performance test, and so there's a a small set of uh go that will go over and use the kubernetes client to you know. Take it do an assessment of the environment.

A

Gonna do I think we looked at this another time, but but basically what I'm saying is like this would hopefully go away in in the face of mesh sync yeah.

E

Most likely, and so if.

F

We could use that package and we'll package everything in here, so that the adapters can use it or measure you can.

A

Use it nice and then yeah, then you, you just highlighted the second area that um will hopefully be positively impacted by mesh sync um and that being the architecture of the atta. Well one the architecture of the adapters which which maybe the architectures that they they still are.

A

You know separately deployable adapters.

A

But that are you know much more leveraging mesh thing or that it's possible that those adapters morph into a component inside of the mastery operator and we're just kind of talking out loud since we've got.

A

Vinayak and abhishek here, because I think of and our abhishek had to go but natasha. The thing is that that I think the path that arip is on um and and that you're kind of tying off right now does make a lot of sense. uh What what asks do we have of vanayak is what I'm trying to explore.

F

So I'm I'm curious about so linker d and sto, we're kind of saying that have a similar approach because they have their own client, so I'm guessing they have their own.

D

Packages because it's all go go repositories; no, let that yeah linker d doesn't, though, is the.

A

Near as we can tell, I think, um asuko's been uh trying to dig into using to leveraging just that and has been coming up dry so leveraging.

D

The link or dcli's code.

A

Right, um no yeah, it's not that he can't go take is that they didn't have it um what's the right way to make my um they didn't. Have it.

D

They don't have a client go, they do have packages.

F

No, no, no, we don't need a client, though. So client, though, is only meant for uh for um let's say solutions that use custom resources. Linker d does not use custom resources. uh The the thought is that we we basically want to dissect their client, their cli tools code and see what kind of api request it makes. So, I'm pretty sure they must have abstracted a lot of that.

F

So it's basically diving into uh what the code flow is for linker d, cli and everything's written in go other than the data plane, so it should be pretty easy to follow through and say. This is how we can import their package and make the api call to the uh the api gateway or server for linker d, and I can try drawing some diagrams around that, uh but for for vinayak I'm I'm I'm curious. If we can start looking at uh something like kumar or con or console, is it is it a similar approach?

F

Do we have clies for them? Are they written in go, uh but we linker d and sdo are the common ones and there's there's a lot known about these guys, but what about the the ones like kumai and mesh so just curious like what their repositories look like? Is there any code that we could import and use and what is their way of identifying like their their version command if they have a cli and what it does so so that way?

F

Either we find a common pattern or we say that each one has a unique way of querying.

F

Maybe we'll have some tiered approaches where we say: look for crds or look for images or look for deployments, but that's not our our primary uh method of fingerprinting a mesh. The primary should be that hey. Can I talk to the control plane? Is there a control plane up and running if the the response is like timed out, there's no control plane. If there's a response coming back, then there is a control plane.

F

Rather than reinventing, things is what I'm this is gonna uh this good. This is gonna help us speed up the messing development. I know we were discussing crds and other ways of listing, but this just came to my mind that we don't need to totally yeah.

A

It makes a bunch of sense to me. um Have you is this? The right excuse that you need to uh put your put your hands on uh lanker d and console.

C

Yeah, so what what uh so console is one thing, what the second one should be, the kuma or which one.

F

So when I let's do this, uh let's tackle linker d and istio at the same time, uh so me or adeep could start tackling istio. You could start tackling linker d and look at the code on how we can do. uh We can query the their control plane and ask for the version or get any other details we need about that mesh.

F

uh So that way we can divide and conquer, and do these two primary service meshes uh that we want to support and then, if someone else jumps in or we have time we'll start looking at puma and mesh and console if there's a third person, maybe a deeper me, jumps into console and starts looking at that as well, so we're basically tackling each of these meshes in parallel, so I'll try to get the abstractions and the interfaces ready so that all we have to do is focus on the business logic and then just implement those interfaces.

C

Sure sure so, uh I'm just reiterating the uh my understanding. We need to look for how these uh individual uh mesh clies are able to identify or talk and connect with their own control planes. And can we leverage that same logic in our uh scene to fingerprint the meshes right.

F

Yep yep, uh it would be ideal that we can import their go packages like go get them. If we don't have that we'll have to basically mimic the api calls that they make.

C

Sure makes sense, yeah.

F

I don't know if there are certificates and everything. I highly doubt that these uh cli tools have too many certificates uh that they exchange but yeah. We can see what happens so yeah, it's it's it's research work and then uh we can basically see how we want to do this.

C

A

Gents nice does that mean we're we're ending early.

F

Yeah yeah, I think we have. We have things to think about so I'll. So I I guess the I I don't know I'm trying to think out loud how we want to divide responsibility. So we discussed when I takes on linker d adi probably could uh take on istio, and uh I can. I can initially do the architecture of like or just the layout the skeleton code, the interfaces and how you want to structure repo and then jump on to a third mesh.

F

If, if console, is the higher priority, we'll jump on console and I'll start doing the same thing that deep and maniac are doing so that we can just move from mesh to mesh and uh discussion, we will need to have, is what's our what's the mesh sync model object or struct that we want to use to the abstraction that that tells us about the mesh, so it just needs to tell us. Is it up and running what version? Is it running any other information that we need to know which is common across meshes?

F

They don't have to be unique to a mesh.

A

Number of proxies number of some reason: yeah.

F

Some some configuration elements that we need to know about the control plane and then uh maybe the like in esther's case, is the sidecar config map. So we want to figure out what that what the sidecar envoy, bootstrap config looks like or the commands that they're on or whatever.

A

Nice, okay, um oh this is good. uh Did I get any other thoughts.

C

Oh, I think um you should go with this path.

A

As you, if you think about it, um if you can um uh I'll message you his his name, um asuko in uh will be interested in some of your findings. I think on linker d in part, because you were trying to find a more intelligent way of provisioning, linker d, which would also have to do with like leveraging the package inside of its cli to provision it so.

C

Okay, uh so can you just select me his name, so I need to refer to him or.

A

uh Or rather just uh uh be social and include him in your findings or and yeah.

C

Yeah undisputed.

A

Nice all right uh natish uh same time next week.

D

A

And we're on we're on a roll we've formed stormed normed. We might start performing here, quick, yeah, yeah cool, all right, yeah. Thank you, gentlemen. Yep. Okay,.

E

C