Cloud Foundry CF on K8s Forum, 14 Dec 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CF on K8s Working Group Call Recording 14th Dec 2022

Description

2022-12-14
[DW/KB] Add HorizonalPodAutoscalers to the Helm chart (based on the template generated by “helm create”)
Agreed to create a chore for this to be implemented for the API.
Controllers/Webhooks probably need more scaling/perf testing to make it worth investigating.
Webhooks might be interesting to do for High Availability but likely not necessary for performance reasons.
This doesn’t affect the image merging story as webhooks are already part of the same images as controllers, so we can postpone this decision.
[KB/DW] Add tests for the Helm chart
Conditional logic is starting to grow, might warrant testing those conditionals.
Agreed to do some focused testing on the conditional logic.
[GC] v0.5
App specific repository story should be wrapping up this week.
Will make a v0.5 release after that story is completed.

A

Hello, hello, the recording is my progress. Indeed,.

B

The same it's been much more interesting than the other one I just left. Oh.

C

B

Yeah, that's good, there's not.

A

D

C

How many aspects for how many answers.

B

They have my things done that I think they think the m stands for monologue. That's all that seems to be happening.

B

C

Anonymous monologues Anonymous pretty.

B

Much something like that. Oh.

C

Interesting, that's fun.

B

Oh Anonymous monologues always or something, but yes,.

C

Oh you caught me monologuing.

C

Hello, hello, hey Karen, hey Tim, oh.

E

B

So we're not gonna have Yogi and dunnell this evening. This morning today,.

C

At this point, yes uh yeah, Bird, Rock ping, me too um so I guess I will facilitate. Today, um let's see how many more people we get.

C

And uh go from there.

C

We'll give it another couple minutes you have thoughts, please feel free to throw them on the dock.

C

If you don't have access to the dock, I put the link in chat.

B

Please and show your own access to the doctor before helping others. That's right.

B

C

People to gather a little bit, you got things to talk about, throw them on the dock.

C

Here, I'll give it until 35 before we get started.

F

And anyone else we have seen before.

C

Julie likes to show up sometimes depends I, don't know. There's Eric.

B

G

F

I have two zooms open: oh outside I, don't know how he managed to do that. No I, don't know we I won't close any of the two, because it's not gonna be kicked out if I close the wrong one. It's fair.

C

All righty well we'll get started. I'll do my best uh impersonation of Bird Rock! uh Welcome to the CF fund, kubernetes working Group Forum! uh Well, let me paste the uh Doc in chat again.

A

There you go I'll, be your facilitator today, let's kick it off with the first topic.

C

That would be I, think Dave and Kieran.

B

Yeah, so we were just taking a look at the helm: chart uh there's a troll in the backlog to see what learnings we can take from Helm create, and one of the things that we saw was that it templates out a horizontal pod or a scalar, and we were amusing about whether or not that would be nice to actually have in our chart.

F

What is it it's like a standard, kubernetes thing to get our parts to Auto scale. Yeah you get deployments to auto skill, I, guess: okay, interesting memories,.

E

Threshold and it'll spawn more when it hits that or drop back again.

B

uh It might be nice to have for the API deployment.

F

Yeah I wonder how how much we want this to be configurable by users.

B

Yeah, it does put some entries into the values file to control that, in terms of being able to set the limits, and that kind of thing.

E

And enable it and disable it I suppose right.

B

Yes, you can turn it on and off completely.

F

That looks like a good idea. It's the only part that we might want to scale, because the controllers can't be scaled right.

G

F

That means they can, but just for availability yeah only one runs at any time.

H

Yeah, the controllers could technically be scaled. You can scale the web hooks by scaling them horizontally, but I. Don't think what like the the metrics like actually impact the web. Hooks correctly so like, like a controller, might have a ton of memory, but that's because it's doing controller things and there's a reflective.

B

H

Web Hook meeting scaled.

F

And also we're going to merge them all into one container right right, that's going to be tricky to yeah. If we want to scale that, but the whole point of merging them is that we believe we don't need to scale them.

F

Other people might want to have multiple instances for resiliency right.

H

Yeah, that's not something they'd want this thing messing with most likely.

F

Yeah and definitely not the horizontal thing, but the for the API yeah we should we could enable it with some with some values to control it, and some good defaults just sounds like a good idea. Why not? Okay.

H

It's mostly just a convenience tank, though right, because, like nothing, stopping someone from making this on their their own correct.

G

F

Right because the I got the deployment name, all you need is the deployment name which is fixed, so they could do it themselves. I guess.

H

Yeah I mean I, don't have a problem like a convenience thing sounds nice as long as you can disable it.

E

Yeah, just talking about scaling web hooks, I see kpac separate their controller and their web hook um deployment. So you could scale them so you could scale the web.

H

E

Should we be doing that, but then, if we do that, we're gonna have two two things that are going to be blowing up in memory potentially.

H

Yeah and I, don't I mean it's just like a theoretical thing: I, don't think we have any like it. It might be nice to have two con Two controller instances. Just for ha but.

G

H

I haven't seen anything that implicates the web hooks as being like a performance problem. I think it's more of just a if one. If you only have one and it goes down, then that's a problem, but.

F

I guess it depends on how much because it's they are kind of uh of a bottleneck for the system. Right, like everything goes through them, potentially so like if they slow down. It is kind of a problem because it's going to slow down everything, but also if I I, don't know. If our webbooks don't do much. There's no problem.

E

Yeah some scale performance testing before we can sensibly say anything about it. Yeah.

F

E

F

It no, we don't.

H

But for the sake of this, with the way they are now they're- probably not something that should be managed by in HPA foreign.

C

We're thinking it might be, it's a good idea to do it for the apis. It sounds like we're. Gonna make a chore for that, but as far as the controllers, it doesn't make a whole lot of sense.

F

For controllers there's more on any thing and for web books it might be necessary, but maybe it's something that we think it I. Guess it gets it it. It's part of all those things that are more for, like okay. We we want to actually get this ready to be to be run in production, and we want to take care of all those things that are like very productiony.

F

While at the moment we're more caring about other stuff, I guess.

H

B

Think if we do think that the web Hooks could benefit from it, then splitting the amount into a separate deployment- maybe does make sense at that point. But we, like Karen, said we're going to do a lot more scale testing before we get to that point. But.

F

At the moment, at.

H

The moment gone, oh.

F

H

Was just gonna say that the web hooks don't really do like if the web hooks are a bottleneck because of like CPU or anything I think that's the problem with the web hooks themselves.

D

B

H

They only would be scaled for AJ reasons right now, because they're just doing like web requests and like superficial validations like there there shouldn't be anything like too intensive on them.

B

Yeah, it's more just the fact that if, if that part is down because it's being rolled, then you can't create or update resources, yeah.

H

So there might be an argument that we should have. The controllers have two instances by default for for a more production like.

B

H

They're all using um the Lee system, I believe so right, so I think it's safe.

B

That's where we Scout the API to two right.

H

F

I was thinking that, like the image merging work like at the moment, our webbooks are inside the controller's image. Aren't they or there's, maybe they're spread up spread.

E

Around they're, in all the controller components generally.

F

They're, all in the controllers.

E

In each of the control, so everything apart from API is a controller. Isn't it and mostly.

F

Yeah, but we have like this, we have the controllers component, which is like all the controllers except the Builder Runner and task Runner, but it does the do they build a runner and task Runner come with webbooks.

E

E

F

What I was thinking is like splitting out the web books into their own images is still orthogonal to merging the images that we have right now like it's not like. If, if we merge, if we do the image merging and then tomorrow we decide okay, let's actually split up the web books, provided that makes sense. It's that still some that's not undoing what we've done before because like if we're splitting out different things that work together already. Okay,.

H

Yeah and- and there might be valid reasons like kpac- doesn't use controller, runtime or any of the queue builder stuff, but the reason they're with the controllers is because Cube Builder gives you helpers for making them and has a lot of nice libraries and stuff which doesn't prevent you from splitting it out, but like I I'm, not sure if this is the case for sure, but if they somehow are using the same Informer cache as the controllers. That's also a benefit to to keep them together.

F

Yeah I mean we can think I'm happy with the fact that we can postpone that decision to when we are more informed about whether it's needed or not, because I was starting to think. Okay, we, the the merging, is coming on as the next thing. We're gonna do we're gonna work on so do I have to think again about that and think about the boundaries again, but it looks like no. We can go on with the merge and splitting out. Webworks might be a thing in of the future, which is kind of orthogonal.

H

Yeah right, we.

F

Don't have to worry now.

C

All right sounds like we may have wind down anyone notes. I took, does that seem sound reasonable to everybody is I think in particularly once we add I.

H

Think I'm gonna add a note for what Giuseppe just said correct.

H

Do you mind yeah, like or or deceptive yeah.

F

G

F

Maybe I'll add a link as well, if I find.

F

Yeah there you go.

B

Should we move on well.

C

Yep sounds good. Let's move on to the next one hi.

B

um This was just we were discussing, I think that stand up this week, the possibility of having basically unit tests for the output of the helm chart with different configurations, there's a fair amount of logic in there and it felt like it might be good to have tests for that.

B

And if there's anything too controversial there.

F

Yeah, so the idea is to basically just call him template and then assert on the yaml.

E

D

E

About the tools there's some project that uses Korea I've seen they do similar sort of testing using go, but then you've got things like bat tests as well. You.

C

E

To do it in Bash.

C

You help me understand what we're catching with that I'm, just trying to think of like there's.

E

Holiday conditionals in the template, so if you specify a certain value, then you get certain bits. I think a new case we introduced was, if you specify container registry secret and whatever new things called eks.

E

Both of those at the same time, that should be a an error at.

G

E

Moment with you get a required error. If you don't specify either of them, um yeah I mean it can. It can make them a bit more robust by making sure we put in all those making making the making it fail properly. For things like that,.

B

E

Sure required things are required: yeah yeah.

F

I mean, but the it looks like the conditional logic is getting more and more complex in there because, like we have uh mutually exclusive properties and stuff like that, and so it's like this property can only be specified if this is not specified.

F

This can be specified or leave this, or this is already is specified to and stuff like that. So you want to test out the combos so that you, so that's what I would focus on this, so instant, not assert on the full year more but more like okay. If, if these values are in then I know for a fact that this XPath or whatever is not Json path or whatever is not there right and this is etc, etc.

F

In terms of tools, honestly Ginkgo has that g exact stuff that is pretty good at running, commands and AD capturing output, plus it has this Json matches. So maybe that's what we should do if we can convert the AMOLED into Json.

H

F

You can tell kubernetes you can, no, maybe you can tell hell to split Json, no I, don't know.

E

It has yaml matches as well, but the case.

F

E

Yeah, it's the.

C

Way, yeah like.

E

You need to be able to select certain blocks, so you know here's my deployment with this name and and then drill down from there, because you're going to get lots and lots of yaml documents all joined together.

F

Yeah but I would like to be able to say you know if I search for this mat like a grab, if I, if I, if I search for this path.

F

So if there's a deployment with this name, fine, otherwise fail. Something like that. I.

E

Don't think it works within multiple documents, so does it well, if you concatenate it as Jason, it wouldn't be valid Json. It would just be a series of Json documents, so you'd have to zoom in on one first and then you can start doing that trick.

F

Well, I guess you could just split by the divisor and and do a 90 kind of assertion that says: okay, there's at least one document that matches this path.

F

Like one off, uh what is it called I? Don't know.

F

But I feel like the the testing toolkit that we have on go is the way to go, because it's just much more powerful than.

F

On the other hand, you could use if you use like more of a bash based thing: I guess you can use yq, which is pretty powerful for the yamo manipulation. But now it's good for the rest for assertions. I, don't know.

E

You can also use yq inside of go, it seems and then.

F

Oh yeah, you can see exact, yq, I, guess if you want to oh, you can just pipe. It's probably gonna work, uh no pipes. No, you can't! You can wire them up somehow. Well, no! We we.

D

I was uh not really paying attention, but I'm just bringing this meeting into Focus. There is a yq lib you can use. uh We use it in um some of our test code for the confusion.

G

D

E

D

E

Then go if you get the right one, isn't it.

D

F

D

Has underlying um interfaces you can use to formulate a query.

F

D

I can send you I think it's.

E

Is it Mike Farrah's yq.

D

Yeah yeah and there's examples in our um has it after config testing and.

E

Practice yq lib.

F

Neat yeah: that could be the way.

D

Yeah, let me let me see if I can I think uh I wrote a little helper function that allows you to just um post a yq query. Oh uh put.

F

D

Link to it inside in the chat.

F

Yeah, because that's pretty much what we need it's like, okay again, it would have to deal with the concatenated documents thing, but I think that's feasible. uh We can make like a meta matcher that.

G

F

As long as matches, oh yeah, even better yeah, it's.

D

Just a stream, so it should work so I mean in our test we're already using it to like make assertions about the total calculated configuration for Taz adapters.

F

Nice, nice, nice, that's perfect, because the yq docs only talk about the binary, so there might be a risk that it's not actually supposed to be used as a library, yeah.

D

F

Think that's a.

D

That's a real risk, I rationalize that because we were only using it in testing, but.

D

It also looks pretty stable.

F

Yeah that that's what matters that like they don't keep breaking it, because I think we we used to have a singing Garden that uses that used the conco silai fly as a library to do start to search for Flakes and stuff, and it broke every single time because the flying the internal interfaces in Flight kept changing and it broke every single time we had to like fix. The compilation, error push. So if we get into that state, then we'll have to reconsider I guess until then yeah we can.

F

We can take that risk because it's just tests.

C

But aside from implementation details, it sounds like we're all in agreement that we should do some testing for the helm generation. So yeah.

F

At least I mean at least the for those like complex conditionals I feel like it will give us a lot more confidence, because now it's like we're. Writing all this complex logic and we're like oh yeah. It's going to work you test it manually but like that's, it can give you so much more peace of mind. If you just test it.

H

Yeah, we ended up doing it for a lot of our boss releases, for whenever we added a condition to add a.

D

H

Test for it and it was helpful.

F

Yeah I feel like Helm, should have some facility, or there should be like a project that allows people to do this more easily.

C

Nice anything else you want to capture on this one.

C

I think there isn't it: um let's move on to the last one just set the 0.5.

F

Yeah, so we're we're waiting on this last story to be finished, uh which is the one about the app specific container container repositories and it's been dragging along for a bit. So.

F

I, don't I couldn't get them to give me too much. One update the pair is working on it, but apparently they managed to make it work by the end of the day, although they still have some doubts about something I don't know. So the hope is to be done with that. Maybe tomorrow and then ship zero five, the docs are already there. There might be changes to the docs, maybe needed, because they are introducing some Helm value to specify the AC specific ECR specific stuff and then uh once that's done, I will do the say.

F

The usual thing of like triggering CI to generate the turbo. Try to install manually following the instructions, I guess I'll try I should try on both Google and Amazon really and then. If it's done then it works. Then I'll hit the button and publish the release.

C

C

Any discussion on that are we just overall, like okay, cool works, all right, there's nothing else. We've got more time if we want to use it any other topics. Anybody wants to talk about.

C

I'll go with them now we'll end this one early, then. Thank you so much appreciate it. uh I'll catch some of you later and uh see you later.