Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Apps Updates - Maciej Szulik, Red Hat; Janet Kuo, Google; Kenneth Owens, Brex

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Kubernetes SIG Apps Updates - Maciej Szulik, Red Hat; Janet Kuo, Google; Kenneth Owens, Brex

SIG Apps is the special interest group covering deploying and operating applications in Kubernetes with a focus on the application developer and application operator experience. In this session the SIG Apps leads will provide an overview of what we’ve accomplished over the past year, including API promotions, controller improvements, leadership changes, subprojects status etc. They will also share the work that is being planned for the upcoming releases. The session will conclude with an open discussion and Q&A.

A

Okay good morning, once again, uh hello, everyone good morning, good evening good afternoon, depending on where you are welcome, everyone on site and everyone watching us virtually. My name is mache and I'll, be your host today, um unfortunately janet and ken who are our co-chairs for six cli, uh sick apps.

A

Sorry, I'm in the mood from like literally from 24 hours ago, when I was presenting for 6 eli, janet and ken could not be here, but they are also uh leading the sig apps with me ensuring that the organizational side of things is slowly moving forward, as well as the leading the technical direction for sig apps, uh where you can find us well, kubernetes has a slack sig apps, that's where we hang out there's a mailing list. If you have any questions, problems issues feel free to reach out to us.

A

There are always bi-weekly meetings uh happening at 6 p.m. European time, the other time zones are also mentioned. On this slide, you can feel free to add any topics to the agenda. If you feel like you, have something important to talk to us, whether that will be for pull reviews, you have some specific issues that you want to discuss, don't be afraid to reach us and ask any kind of questions, or if you want to be a contributor, that's even more welcome feel free to to ask anything.

A

uh What are what is that the sig apps does so, primarily, we are responsible for ensuring that you can deploy and operate your application on any kubernetes conformant cluster. I link the charter and our annual report, which goes into details. What actually we did, but on on the following: a couple of slides I'll: try to highlight the major initiatives that we've been ongoing for the past two releases, since the last update that we had during kubecon, in los angeles, as well as what we are going to do over the next uh couple releases.

A

So, most importantly, the ga features. These are that you can actually safely use in production and we fully support you to do it ttl after finish.

A

So it's a nice little controller which allows your jobs, because a job is meant to run to a completion and then it's done, uh which basically means the resources hold by a job, even that, if that will be just uh the um the storage in ncd, they will be held in your ad cd forever, that particular small controller, which has been in a beta for quite a while, has been promoted to ga uh thanks seahil.

A

If I remember correctly um you just, there is a tiny, little spec field at a job where you specify how long it has to live after the job has completed, after which the job will be uh removed. So you can think about it like a garbage collector for jobs, the new other two additions are coming from our friends from the wardrobe batch, which I'll talk about in a minute what they do, the first one introduces index jobs.

A

What are index jobs so normally, when you run a job, it will ensure that it will run this many parts for your particular task. I don't know 5 10, whatever you specify index job is like stateful stateful set in a job in a job world and that every single pod will have their own index. So if it fails a particular index with a number um attached to it will be respawned for this particular. So you can do some, for example, embarrassing parallel jobs or um address particular pot within a job with uh headless service.

A

um Another one which was also coming from work group, as I mentioned, is a tiny little improvement to a job api where you can actually suspend a job.

A

This will be useful for if you have an external controller, managing your jobs, when you suspend a running job, uh the controller will remove all the currently running parts um and then upon on suspension, you will be able to recreate whatever was left there to be done. It's kind of similar to what cron jobs allows, because cron jump also has a notion of pausing, but in case of a cron job, the pause means that there are no jobs triggered until uh until we remove the suspension.

A

uh On to the beta features uh again quite a lot of the the beta features and um the work around the job api has been coming from the workgroup batch and the first one is probably something that I've been trying to address since very early days. So if you work ever with jobs, the way jobs currently work is it will spin up this many pots, but the way it is calculating which pots completed for a job. It requires the pots to be on the cluster, which is fine.

A

If your job is running, I don't know like 5 10 up to 100. Let's say pots, that's fine, but if you run a thousand pots or more, those pots lingering around in your cluster are eating serious numbers of your resources, which is a pain.

A

So what aldol did we started implementing a finalizer on the pods that are being created by the job controller and as we are removing them we're also removing the finalizer and we're accounting the numbers properly, so that we can track, and we don't require the pots to be sitting in a quite a while that feature actually- and I will mention that yesterday during his presentation that has been going back and forth several times, there has been multiple issues with it. We struggle with it.

A

So if you have something like that feel free to ping us with all the issues before we reach ga, which will probably happening in a release or two feel free to reach us, let us know drop on on our slack channel on on or on one of our meetings and we're happy to hear from you uh another one uh for stateful sets is actually ensuring that when you are adding new pods to your uh stateful set, you can specify how long?

A

What is the minimal required time for the pod that it has to be ready before we will be able to serve traffic. Something similar already exists for deployments and daemon sets stateful set was uh was not accounted for it, so we want to align all the controllers with it as well.

A

Another tiny little addition from the worker batch, which exposes the information, how many ready pots in a job there are in the status finally uh for the job. So, as I mentioned before, we have an option to suspend a job and by default, jobs do not allow modify that many fields. Only parallelism can be modified.

A

We figure out that for certain use cases you would like to be able to modify the scheduling primitives for a pot in a in a job for that we allow modifying those scheduling directives in a job, but would have one that has not been unsuspended, so you can create a job suspended by default and on before you actually start running the job. You can still modify its scheduling directives, which is handy for for the batch use.

A

Cases on to alpha features, there's a quite a few of them moxon available, which, basically, if you've ever worked with a stateful set and you're rolling, newer version of state full sets. You know that we are trying to roll one by one.

A

um There are some use cases where we would actually be okay running more than one and the max and available actually allows that, so you can specify that we will be running more uh more uh pods within a stateful set towards the newer version. Similarly, for stateful sets mark added an option to decide. What's going on with your pvcs by default, um all the pvcs were deleted when, when stateful said it was either scaled down or removed.

A

Currently, for both of these operations, you can specify that it can be either removed or we will retain the pvc for a little bit longer, so you can disconnect uh the lifetime of a stateful set from uh from the lifetime of the storage backing your application.

A

uh Another that we that we've been talking for quite a while now we finally had some initial idea how we want to implement, and we managed to put together a kubernetes enhancement proposal, which is a document that describes how we want to implement. A particular feature is consolidated workload, controllers statuses, that's something that if you ever work with kubernetes, especially if you're a newcomer and if you play with let's say deployments, you think you know how the controllers work and then you switch over to daemon sets.

A

For example, you actually have to learn how the daemon set presents its statuses, its progress and you so we're trying to as much as possible in line the statuses with all the controllers, so that, once you learn one controller, you will be able to roughly figure out what's going on with your controller across the board, and on top of that, you will be able to also write higher level controllers on top of the uh on top of the existing controllers.

A

That's something that probably will take uh significantly longer than we would wish, because there's a lot of controllers, there are very various edge cases that we're trying to somehow figure out in some in some common ground.

A

Another one that we recently closed is a pot healthy policy for pot disruption budgets. So, during one of the discussions a couple months back, there happened to be an issue with how we are counting pots for a pod disruption budget.

A

Up until now, we are only we are equally counting running and ready pots in the same way or pending pods.

A

There is no way for you to say that only running pots should be accountable for a pdb, so the pod healthy policy actually introduces a little more wiggle room into how pdb's are counting or which pots are being counted for pot disruption, budget and finally, a feature that has been requested multiple times since almost the early days when we introduced the cron jobs or even when it was called scheduler jobs, which is a time support time zone support.

A

um That is a feature that that we rejected a couple years back for multiple reasons, including one that would require all the kubernetes distributions to include uh time zone database.

A

Since the addition of the time zone database into the golang in the default library, we can now safely say that any conformant kubernetes can properly support time zones, although there are some fallouts out of just the fact that we added that uh particular because we tried to put a very strong validation around the time zone um names and it turned out that mac os um with its uh file name handling where you can have various cup sizes lower and upper case, will work equally fails our unit tests, so one of the requirements for beta graduation for crop for time zone in cronjob will be figuring out how to properly handle the time zone.

A

Issue on max primarily.

A

Okay, so what are we going to do over the next couple of releases, so primarily, as I mentioned, we would like to uh get to completion all the current alpha and beta features. If from the beta or alpha features, there is something that you are interested in. You want to help, or you actually want to test, and you can provide features, feel free to do so drop on our slack drop on our mailing list and information that you are having issues. What are the problems?

A

um Feedback is super important for us to ensure that before we graduate something to general availability, that the future is rock stall is rock solid, stable and then there are no uh major issues.

A

um On top of that, there is a a general sentiment towards pushing stuff to ga, or eventually uh they will be removed entirely from kubernetes. uh There is a separate enhancement proposal that has been merged. I think last year were to prevent perma-beta states of certain features. This is to ensure that we are slowly but moving forward and something that has been pushed to a certain level but hasn't progressed to a general availability should actually be removed.

A

So we want to make sure that the current features were trying not to introduce too many features at any single point in time, but rather focus on completing the ones that we already have in progress.

A

Secondly, there is an enhancement proposal about increasing the overall reliability of the kubernetes project or the products that are based on on kubernetes project, and there has been uh sent a letter sent by the majority of the contributors from the kubernetes project, where we are increasing the overall acceptance criteria for any uh poor requests, either adding new features or fixing certain uh areas.

A

So if your pr or your issue will will get some pushback, it's not a personal thing, we're trying to ensure that everything is properly tested. Everything is properly verified before accepting that to core.

A

That also means that if you will be fixing certain area that did not have a test before it might be pushed back and you will be asked to add some tests, uh even though there were none before thankfully, in the majority of the controllers, both the units and integration and e2e testing is pretty good, but we still have certain areas like something that I I've been pinged on last week, where for stateful sets the tests, the unit tests that we actually had we're just verifying that we are throwing an error and but without actually looking into what the error was, and it turned out that because of the misconfiguration of the test, because it was lacking, one always required field, the test will always fail and everyone were expecting.

A

Oh if it's a failure, so it was failing always so it was always giving you a false positive.

A

um The author reached out to me he I started looking at it, and I asked him to to do something similar to what we already have in the batch group, where we are actually introspecting each error for the field names and possibly also the type of the error. So nowadays the stateful sets, if jordan or when jordan approves it uh we'll have a little bit more um appropriate test cases.

A

So there are those edge cases where nobody actually was was aware about them, and um it's pretty nice that someone actually was able to uh to find it. So if you're into this, uh you are more than uh welcome to join us help us with any kind of contributions, if you're interested with either improving our documentation for all the controllers or any kind of specific controllers.

A

If you want to help with uh project management because you're a project me project manager, we are more than happy to to have people help us in various different places, whether that will be managing the issues managing the open pull requests.

A

We still have problems with having proper contributions or, most importantly, reviewers and then approvers in the area. I'm fully aware that getting the going through the contributions ladder in in the sea gaps area is quite complicated because the the initial knowledge needed to ensure that every piece that we're changing and then controllers is not trivial, and it does require quite some time to uh to look at one controller or several uh for me personally, even every single time. At this point in time, if I'm looking at certain pr's, I I'm getting different messages. On slack.

A

Have I ever looked at prs and stuff like that, every single pr that I'm looking from the controllers area, I usually will double check in the running cluster to ensure that it doesn't break something the scope of potential problems when modifying controllers is pretty big.

A

So we need to ensure that anyone who can and who does approve prs in this area is 100 sure, and it's not only about a particular feature, but it also has um a reasonable uh sense in the general direction that we're trying to go with the majority of the controllers. So if you are waiting for some pr reviews in the controllers area, I I apologize. It might take some time, but always try to ping us on slack, even if it has to be a couple of times.

A

I I've been notoriously responding to people a couple weeks or even months after they pinged me if there are times like kubecon and my slack just bubbles up with tons of notification.

A

Unfortunately, I barely track my github notification at this point in time, because it's impossible um so every time I will review apr, I will ask the contributor to to ping me on slag when it gets addressed when he addresses my comments and so that we can move certain features faster with that, oh yeah. I almost forgot so a couple months back.

A

I think it was uh february um a group of people from our scheduling, mostly, but not only uh figure out that they would want to formulate a work group within the kubernetes project that is primarily devoted to improving the overall batch interface.

A

So if you're thinking about, I don't know, um high uh high performance computing, lots of any kind of data, manipulation, uh aiml or even ci flows, so stuff like kubeflow uh or spark.

A

If you are running tools like that and there's quite a few of them that grow around the entire kubernetes core, because over the past couple of years by now, we said that we will provide certain basic features, but we want to see what the community builds around the core kubernetes um so aldo reach out to to us in the segaps reach out to folks, from six scheduling and sig note, and together we're trying to get a group of people that are interested in and we're trying to push certain features forward to improve the overall experience.

A

We are meeting every other thursday at 4 p.m. uh European time, of course, there is a mailing list and a slack channel devoted specifically for us, uh like you've, already saw- and I did mention that a couple of times all those work around index jobs uh the job tracking api uh job suspension.

A

These are contributions done primarily by the folks from the from the batch work group. So if that's something that you're interested in feel free to drop us a note, there was a session yesterday where aldo was introducing what the word group batch is and their use cases.

A

uh I was hoping that it'll be the other way around, but unfortunately it happened on yesterday, but I put a link, so you can probably- and I've heard that the cncf is slowly uploading- the videos from kubecon already, so you should be able to check the recording uh within a days hours. Probably I think, with that, I'm open to questions shall you have any okay.

B

A

B

The first question is since reviewers need so much background. Are there any mentoring, opportunities.

A

Right, that's! That's a very question. That's a very good question. Thank you. We've started a mentoring cohort in combination with six cli, since I'm leading both of the sig apps and six cli efforts where the entry barrier for six cli is significantly lower than for segaps, and they are very frequently overlapping.

A

um I'm hoping that after cubecon, I already spoke with paris. I've seen her earlier today. Here uh we will be slowing slowly putting something together where we will be having private slack channels and we'll be having a group of people that we will be mentoring towards getting to the reviewer and eventually approvers.

B

And my second question is: what is your biggest need on sig apps.

A

Contributors, I would say probably like with all the rest of the people that were speaking either earlier today or even earlier this week. Pretty much every single, open source project is struggling, we're not any different with that, even going through the open prs, even if you don't have the reviewer or approver label. That doesn't mean that you can't review prs. You can't double check um the issues that are being reported by different users, any kind of input like that, where oh I looked at it and I was able to reproduce.

A

Oftentimes users will report a problem which is a very narrow or edge case. That is hard to reproduce and it's hard for us to be able to verify every single edge case, because if there's I don't know, 50 or even more issues open and I have to spend half an hour or an hour per every issue, not to mention similar amount of prs that I need to look where every single pr is roughly the same uh amount of time, sometimes even longer.

A

It's it doesn't scale and there's not that many of us so every kind of input where oh I looked at it and even better if you can uh double check it or ensure that it's still a problem with the latest version. Oftentimes people will report issues that happened in earlier version of kubernetes and something that might have has been fixed by a different change or is not a problem anymore.

A

Any kind of information and input from all of you, where you will just say, oh yeah, I can easily reproduce that on the latest kind cluster, for example- or I did looked at it in the latest kind- and it it doesn't appear to be a problem anymore or any kind of input like that is more than welcome.

A

Okay, I don't see any other questions I'll be here around for a little bit and we have a sick meet and greet in slightly less than an hour at one. I don't know where that is yet, but I'll figure it out, because I'll check my scat in a bit uh I'll, be there for as well. So if you want to talk about sig, apps or six cli, I'm more than happy to do so. Thank you very much.