Cloud Foundry CF on K8s Forum, 9 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cloud Foundry on Kubernetes Working group forum call recording: 9 Aug 2022

Description

[MR] We’re no longer publishing images to the “latest” tags on dockerhub: https://hub.docker.com/layers/korifi-kpack-image-builder/cloudfoundry/korifi-kpack-image-builder/latest/images/sha256-3bb807a16b60775bcbf01aab247db372dfb67fff96acbbf9a4b74b4b9fc23379?context=explore
https://hub.docker.com/r/cloudfoundry/korifi-kpack-image-builder/tags
Dev images have tags like: “dev-0.3.0-4a782d757ee67c2d896e9b97da58cbc4d83827fc”
[RI] Congrats on the 0.2.0 release. Please help publish an update to the CFF blog.
Release notes: https://github.com/cloudfoundry/korifi/releases
Ram to start a Google doc to start collab on this
[RI] Draft of a handout intended to be distributed widely at KubeCon. Request review.
[CC] Cloud Foundry Day
[CC] Korifi training
[GC] Cleaning up task pods + task logging

A

Cool all right, um I dropped a link to the doc um I think everyone's got it, but as as new folks join, we'll drop it some more and feel free to keep adding stuff. But um I want to start with Matt's. First topic.

B

Yeah, um so yesterday, I noticed that we we have a latest tag for all of our images, but um we're not updating it. So the example here is like the kpac image, Builder I think our most recent latest tag is from two months ago.

B

um So I just thought we should talk about this. It's a little surprising, I think um none of us really ever deploy from latest. We usually build the image locally or in the release. We package I think a a log file or we we put it through cable. So this isn't a problem, but if you were to ever to check out a branch or a tag and try to deploy without building the images, it would be broken.

A

um Yeah this actually came up like a month or two ago. I I was whining about it, because I actually used latest and um I know like uh Danny, and some other folks made some changes so that now uh like there there is an image.

A

That's pushed for each like each merge domain and you can use those but I think we decided that we would update latest with the reference to like the actual, like release so like for 0.2 latest, would point to that, and then we went to 0.3 latest would point to that and I'm wondering if maybe we just don't have that Automation in place or something.

B

Yeah, um we definitely haven't published. Up-To-Date latest 410.2 went out so that maybe that's just missing.

C

Yeah because I would expect latest to be latest release, as opposed to latest like commit, but it needs to be latest release, which means every time I. We click the button on the release pipeline. It should also update the latest. You should also push latest- probably it's not doing it or it's bugged. I, don't know.

B

Why not just delete latest.

C

Not fair, it's it's a convention, pretty much everyone has it I I mean it's I agree that it's probably not going to be used a lot, because if you use the releases, then the releases points to specific tags and if you're, building local local copy. Also, you don't want latest right. You want whatever we could call like. Whatever we push for latest commit, which is not latest, but it could be called I, don't know, Dev or something which doesn't exist.

B

You're currently using div number.

C

Of the upcoming release, followed by the show the commit for those images, but like it's not it's not a tag that you that is tracking right. It's it's a new tag. Every time, yeah.

B

All of the yaml files also point to latest by default, and we haven't changed that so I think we want them to point to something valid. Regardless of what the tag is.

C

But if we update like fixing so that latest points to latest release, wouldn't fix that, because you you're pointing to zero two, but your code is way ahead of zero. Two, if you see what I mean so you're still you, you would be deploying zero two every time, but using the ammos that are not from zero to which is potentially broken. That's true, so which this is.

C

Why, like I, made an explore for the distribution stuff and part as part of that I I specified that we should, as we solve this distribution issue, also come up with a better flow for deploying local, your local version, so that uh when you deploy local, it actually deploys images that are exactly built from the commit you're deploying either they're pre-built, because we pre-build them anyway or you build them locally, with K build or something.

C

But you should be pushing like they should be the build from the same, commit you're, pushing so I think now we we should fix latest because we should but I think the yamos will be broken anyway and I. Don't think there is an easy fix.

B

Yeah, we might also then need a note in the readme to just explain the state of the images.

C

Yeah, maybe in the hacking file, so people, beware that, like, if you deploy these CMOS you're, deploying latest release images using not the latest release yeah mods. So it's probably not going to work or be weird in who knows which ways.

C

Because this installing the install instructions are about deploying the released turbo, so those are not affected. Those are fine but like for the hacking, local hacking kind of use case, then yeah that is broken.

C

And I guess we don't? We don't realize because we build the images locally with cable, then push them or Silo them in kind, and we never really use those images to get pushed to get to docker.

B

Yeah we I don't even noticed it because we're we've added the new image to the repo with the um the job test, Runner, um so yeah I, guess it's. What I'm hearing is I should make maybe a chore for a feature to um to update the hacking file and also figure out how to publish latest whenever we cut a release.

C

Yeah, those are at least two chores yeah and then the I remember. We even tried to get rid of those images pushed to every commit to main, but then we realized that we actually needed them in some cases and we bring we brought them back so once we have them. Maybe it would be nice to have like an easy way for people to deploy local like unless you've just committed.

C

There should be an image for the commit you've checked out right right, because CI does that and then we come in so when we would be nice for to use those so that people don't have to rebuild for nothing if they're already built and available in Docker hub.

C

But it's weird because you also want to support the use case of people who've just committed something and want to run it so I, don't know.

A

Yeah I think you mentioned cabled I'm wondering if, if they actually are identical, then they'd have the same digest and maybe I'm not sure if Dr Hub would give them the same digest, though, so it might not work out.

C

Which means like cable might be clear enough to say: oh, this is already built in Docker not going to do anything just use it. Otherwise it's going to build and try to push it.

A

Yeah I, don't know um I I think you still might have to tell cable about the the tag you expect for it to be able to do that, though,.

C

A

C

We figure this out as part of the distribution explore because yeah it's it's a gap that we have right. It's something that is currently you know is has probably never been working. Basically.

A

uh Cool um I don't know Matt, uh were you planning on making those chores or yeah.

B

I'll make those chores, okay, cool.

A

All right, um all right, let's move on um got a topic from ROM about the O2 release.

D

Yeah I saw that uh release was cut. um I was just wondering if we could do like a blog post based on the highlights of the release, some of the release, notes and some other information that folks are willing to share. This is intended to be published on the cff blog.

D

So if someone wants to volunteer themselves or volunteer another person, um either ways um ping me on slack or something and we can take it from there.

C

Is it about writing the post from scratch, or is it about like reviewing.

D

um So if you can give me like a few notes, I'll draft the post myself um and then probably do like a quick review and publish it. Okay, so all I'm looking for is some notes. um If you're a bit lazy to type those notes out, I'm happy to hop on a call.

C

um We have the release notes, but I don't know. If those are enough so I do. Can you write down the release notes but like I, don't.

D

C

Yes, the details.

D

But if there's a little more in terms of context and um in terms of design decisions or something that would add some meat uh to the post itself, um that would be good too.

A

Yeah, um if you want to like start a Google doc or something and share it on slack I'm sure some me and others here would be happy to like start filling that out with you too,.

C

Yeah, it would be nice as well like just do it by Google Docs, because it's super nice.

D

Got it I'll do that thanks.

A

All right uh cool: uh let's move on to the next one, um a draft of a handout for kubecon.

D

Yes, um so Chris and I are going to be standing at a booth for all three days of kubecon.

D

um So by way of a little background, um the cloud Foundry Foundation is a silver sponsor at kubecon, so we have a booth um that needs to be staffed and one of the two projects that are going to be like a big feature. There is corifi and the other is pecator.

D

We are going this route because we feel that it's going to be the most relevant project um that the cloud Foundry Foundation has for folks, that are, you know, expected to come to kubecon and they want to hear stuff about kubernetes. Obviously, so one of the things that we are preparing is a handout about query fee, and this is intent to be intended to be like a single page handout. So like two sides of a page, um so I came up with like a draft for what could go on that.

D

um What doesn't repeat things from our website- and you know just some important things to highlight about the features of the project um so I made a draft of that and um I think people can access it because I see like a few Anonymous animals there. But um the idea is, if folks can take a look at this um cough up.

D

Some ideas about you know if there's other things that we should represent uh instead or if there's any changes that you'd like to see in the general draft, in terms of content copy, um whatever right so open to all kinds of feedback. So we have a few weeks to get this ready, but the sooner we nailed down the content and the sooner we get it out to visual designers um the better. So obviously you can tell from my amazing visual design skill set.

D

It's going to need a lot of help in terms of layout and other things. So if I could get some feedback on that doc, I'd really appreciate it.

A

Cool, thank you and I, see um Eric pointed out in in chat. uh Could you give us comment access to this as well.

A

Right now, everybody can view it and click, but can't comment or anything.

D

Yeah I have updated the doc. Sorry about that.

A

Yeah no worries um great. uh So, let's, um let's move on um something from Chris uh Cloud, Foundry, Day, Yeah,.

E

So, basically, just sticking with kubecon. uh Hopefully uh it won't just be RAM and I uh hanging out of that booth for three days. Hopefully some of you will join us there as well. We'll follow up about that too, when there's there'll be a sign up sheet for times, um and hopefully some of you can join us at the booth, um but a reminder that if you haven't registered for kubecon but you're coming, uh the price jumps up 300 on Thursday so register today or tomorrow.

E

um So that's basically the big news about 500 day right now, looking forward to it um and then next item I wanted to chat with you briefly about um potential training program for karifi.

E

uh It might be a little premature, but I was chatting with Steve Greenberg who helps us with a lot of the uh LF uh Cloud, Foundry training and certification stuff, and um at some point, um maybe later this year, maybe next year, we'll probably want to have some kind of offering it might not be a full training course or something like that, but but a smaller course or something that's just a tutorial training on Curry feet.

E

um I, don't exactly know whether the target audience there would be operators or developers, but I figured it's about time. To start a conversation about that, so um yeah I was hoping to maybe set up a call with a couple of you all I.

E

Don't think we need to have this discussion at the working group meeting, but um with people who are interested uh like to set up a call with with a few of you and Steve and RAM and I, and we could start talking about what that might look like uh sometime down the line. So.

C

Interesting yeah I wasn't expecting it but yeah. It makes sense kind.

E

Of yeah I mean yeah, we, the tutorial um on the website or a proper like LF training, module on griefy I. Think at some point we'll want to do one or both of those. So um you know sometime in the next month, maybe I'll I'll set up a call um I'm, not sure who, amongst you, wants to be involved with that, but uh but I'll I can reach out in the slack channel. So I'll probably set that up sometime later this month,.

A

Cool sounds good um uh unless there's any more for that, one I think we can move on to Giuseppe's topic.

C

Yeah so I started a thread on on slack, but just to gauge opinions in person about this uh task cleaning up thing. So the I had the story ready to be picked up about basically setting up the jobs that we create every time.

C

Someone runs a task uh to be automatically deleted in a configurable range of time, which is in the order of seconds, though, like the setting isn't the number is in seconds, so it could be many seconds but still seconds, uh and we were discussing this and it kind of occurred to me that maybe we don't need to do it. uh I created the story just because I really needed it and we kind of took for granted that it was needed.

C

But then reasoning about it uh and plus I've done some reading about a garbage collection for some like leaking issues that have occurred recently and it looks like pods do get garbage collected by kubernetes automatically. There is a threshold value that can be configured at the cluster level and if a part is terminated, either succeeded or failed. It will be deleted by the time the threshold is reached and think which means we don't necessarily delete these pods ourselves.

C

They're just stick around until it's there's too many and then they'll be deleted automatically, and a pro of keeping them around is that. Currently, our logging strategy is based on just fetching logs from the pods, which means if we carry on the strategy of deleting them after five seconds of after completion, then basically I mean I.

C

Don't think we expose task clocks yet but like even if you want to look at those logs via Cube cattle or something they've begun almost instantly after the task is terminated and that's maybe not great, because in like real world use cases, people almost always want to look at the output of their tasks, just to make sure they went through fine or whatever.

C

So I was thinking we kind of catch two bits with one stone. If we keep them around, we don't have to do the extra work to to clean them up and the logs will be available for as long as possible without impacting the cluster, because again they get garbage collected.

C

This is kind of assuming the garbage collection is cheap, slash almost free, but it might not be the case, so there might be an argument for we never want to reach garbage collection, because at that point the garbage collector, like will cause like performance stress to the to the cluster and I.

C

Don't know anything about this, so I've seen a cluster garbage collecting, but I wanted to hear your opinions, because it does sound like a like a good trade-off to me uh or on top of this tasks themselves, which in turn own task workloads without which own job which owned pods those get deleted after 30 days, that's standard, CF Behavior, which means these things will not stick around forever. They just stick around for many days uh and those 30 days.

C

We could they've been arguments about shrinking that time span, but like default CF, that's that's 30 days, uh so yeah discuss.

A

um I I posted this on slack.

A

um One of my worries about relying on the garbage collector is just like I think that would work perfectly fine if we knew we were the only thing running on the cluster, but it makes me worried about other things that are like using pot like short-lived pods for things um like, if someone's using tecton or something else on the cluster I'm wondering if our task pods were like overwhelm that and I, don't know, I guess I'm wondering if there's like uh I I, fully understand like five seconds after completion.

A

It's like way too soon, but I'm wondering if there's like a middle ground or something um like even even like a day, because I'm I'm thinking um in in the case where there's not a ton of stuff running on the cluster, then people's task pods will just like build up for a long time and when they look at the a namespace will just be a bunch of dead pods in there that doesn't sound great either.

C

Yeah I guess it's yeah it's! It will be a bit noisy but, like also I, don't know how many tasks people run like in general.

A

A

um It varies widely from like none to like I've seen CF for VMS, like users spawn one like every second, which I don't think our system will be able to support anyway, but I've I've been on, like um like support issues where people that 30 days, they ended up with like a million task rows in their in their database, and it like would bring Cloud controllers to a halt. So, like I I, think we can. um We can't really estimate or like how people are going to use it too.

A

Well, so it might be worth putting some other guards in place there I I do know, though, that you're like, is it you're, you're restricted like 200 or something? Is there a limited.

C

Number I heard that number, but I, don't I, don't know if that number is the threshold, because the docs, the dogs mention a threshold which will trigger garbage collection whatever. If 200 is the number that is Will basically cause the node to stop accepting pods, then I would hope that the threshold is a bit earlier than that. So yeah you never reach you never saturate the node, but I, don't know and I guess.

C

That number could also vary a lot from cluster to Cluster I, don't know if it's a hard limit I have the number of like 200 or 200, and something as like a kind of generally accepted limit of pots per node, but yeah I I see that also like, if you have other things in your cluster that are like task task key in nature like one of PODS and those also keep them around relying on garbage collection, then you might reach that threshold relatively soon and then I I, don't know how it decides which pods to delete.

C

When it's time like that's, there's, probably some some weird heuristic, uh which means yeah, we might, we might be the cause of other parts being deleted actually by garbage collection right. If by luck, our pods get kept and other pods get deleted. But again, if, if you don't want that to happen, I don't even know. If you can prevent that.

C

Actually, yeah I don't know yeah, so yeah there might be a bit or not a bit annoying, but you should no one should rely on those parts sticking around, though, like once it's terminated, you should kind of assume it's dead and we do it's just it's. If they stick around for as long as possible, it's nice for logs, that's all that's all, but maybe deleting them after I don't know.

C

12 hours is good enough because, like you, you you're interested in those logs probably drops dramatically like after, like you need them, definitely need them as soon as the thing is run or while it's running even but then as soon as it's terminated. Your interest for for looking at them is probably a lot lower and it probably converges to zero very quickly. So maybe just a few hours is enough or even I, don't know, minutes I, don't know.

A

Like I I think like ours is reasonable or or even like a day or something it's like, because log cash isn't like infinite. It's it's bound by how much RAM you can give for for keeping those logs around so um I I think ncf in a in a busy environment. They generally fall off after a day or two or so, but.

C

Okay, so what I'm hearing is, it might be wise to still implement the behavior. Just don't do like Irene, because five seconds is Extreme and kind of makes logs completely useless, but maybe default to I, don't know an hour or two or 12, and then people can make it configurable so that people can change.

B

C

B

Still feels really fragile, relying on pods for locks I mean in the app case. If you've got something, that's restarting, you can access the previous pods logs and the current ones.

D

B

If it's gone and restarted twice in a second, that's your log history now.

A

Yeah I mean that's, I I think there's like work. We can do with documentation here and stuff, but like our our log strategy right now is like we want to support logs, that you can stream out drink, push or like type CF logs and see them. But if people want durable like log support, um they should inject like a fluent bit sidecar or something and send them elsewhere, I I. Otherwise, um like that's.

B

Just a shame that wouldn't be integrated in the CF log come on, though, isn't it.

A

Yeah but barring um like the law, cash support in CF for Kates was never robust, like that. That was broken on its own, so like barring, like significant investment into getting all that working like I I. Don't think there is a good solution versus like I I. Also, don't think people really in practice rely on CF logs, like returning a ton of Vlogs versus sending them like somewhere, better so like, even even on like CF or VMS, like people like will set up like syslog drains and everything to to send.

C

A

To to somewhere.

C

Yeah because uh uh logger gate has that feature right. You can yeah.

A

And and the kubernetes like analog would be an agent that would send them somewhere.

C

So something like uh fluently that kind of stuff.

A

Uh-Huh so like I I, imagine we could like like teach people like hey, you could like set up like a injection web hooker or something to throw those on, but um trying to rebuild the like CF logging infrastructure doesn't sound like a viable like thing to do.

A

um Yeah like like some of the problems with it in CF for Kates, were um I know they never got it working in a in like a not just HJ, but normally it runs like as a clustered setup and the log caches can kind of like like Shard logs and and replicate them across each other, and they they never got that working. So they just had like basically a single thing that would store logs in in memory until it ran out and then would crash or whatever.

C

A

Ideally, it would drop stuff on the floor, but it was like operationally not perfect, so.

C

A

That's kind of like none of us, and that was with people that had logger Gator and logging expertise like working on it and like I, don't know.

C

Yeah I guess we could. If we really wanted to do this properly, I guess we could think of some sort of to think of a similar pattern to what we've done so far for concurrent implementations. So we could have a I, don't know generic interface for logs, which by default it does what we're doing now and then I. Don't know someone can Implement a fluent debugged implementation.

C

I, don't know how that would look like at all, but if we really want to be sophisticated and like maybe that could allow you to integrate even like these very powerful setups still with to and they would still work with CF logs but yeah, it's probably not the highest priority like if you, if you have fluently working, you probably have a way of looking at those logs and yeah. It would be nice to have to see our vlogs well.

A

Yeah but I like what you said like it goes along with like what Karen was advocating for like earlier, with the interface being like a log, Source or or something address, so that the Sham will be like. However, these get there, we will try to read from that, but.

C

Yeah I would still have to implement I mean with the default. Behavior is already there we'll have to extract it, I guess, and then someone really wants fluency to be integrated. They could Implement an equivalent for fluency or whatever yeah.

A

And- and that's really wanted to wanting it to be integrated with the CF logs command versus if just doing the sidecar is odd and having them, send it to um splunker or wherever you want them to go.

C

C

Okay, so I'll bring back that story and think of a sensible default I, don't know how many seconds is 12 hours.

C

Forty three thousand and two hundred in case you wanted to know.

A

All right, then, um yeah I think we're at the bottom uh feel free to add more stuff, otherwise um I don't know, I think we can call this.

A

Cool I see some waving all.

B

A

See y'all in two weeks: I guess.

D

Thanks Tim thanks everyone, bye-bye.