Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 12 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: “Why Can’t Kubernetes Devs Just Add This New Feature? Seems So Easy!” - Ricardo Katz & Carlos Panato

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

“Why Can’t Kubernetes Devs Just Add This New Feature? Seems So Easy!” - Understanding the Feature Lifecycle In Kubernetes - Ricardo Katz, VMware & Carlos Panato, Chainguard

Speakers: Carlos Panato, Ricardo Katz
You started using Kubernetes. You are doing great! Then you figure out "hey if kubectl has colors probably it would be helpful for other people!" You go ahead, and open an issue in the Kubernetes repository, to figure out that there's another issue opened since 2016 about this! And with a lot of discussions! Yeah, it happens! But why does it take so long? Is this really necessary? In this talk, we will present to you how a new feature gets into Kubernetes and, most importantly, why it takes so long! We are going to pass through some simple scenarios to understand what is this review process, what problems were caught in some real life feature requests reviews, and also other examples of features that were promoted and now became a problem, trying to understand why they reached this status! Join us and learn a bit more about the enhancement process of Kubernetes!

A

Okay, uh welcome everybody uh thanks for attending this talk, my name is Carlos panatu I work at chain guard and I. Also I'm Tech lead on the Sig release and kubernetes and I maintain other projects in the cloud Community. As.

B

Well, hey welcome everyone. My name is Ricardo I work for VMware I am one of the Ingress in Gen X maintainers as well, and you can find me around in ccla, sometimes and yeah. You can find me around messing with code breaking breaking things.

A

With today's talk, we're gonna like try to understand like the future life cycle in kubernetes and how we like people, can open features and like do implementing to the whole life cycle. uh Before we start like. uh We have three questions. The first one is who here uh is here for the first time that doesn't know anything about the caps and the future life cycle can just for curiosity, okay and uh who are here that have features ideas that want to know how to push those to the through their life cycle.

A

Okay, okay and the final one who here open already like one or more features ideas, and it's stuck in between it's like.

B

Okay, because think that we are jerks yeah. Okay, thank you. Everybody.

A

The we're gonna go like go over some examples and then we're gonna like explain the the how the cap works in the real life, for example.

A

This is the one feature idea that is like a kind of let's say, easier way to roll out config Maps, the future ideas like a red open till like a 2016 and uh like there is no conclusions or things yet like this feature is still open and it's not yet implemented like, for example, we have like a more than a thousand people like saying Jesus is a good think to have, but there is no concrete, like outcome from this idea here.

A

Another one is like a simple startup in ordering, like I still open, like from like eight the 2018.

A

There is a lot of questions and a lot of like comments and people like, should we use a Readiness gate like some other new filtering field like to give ideas, there's a lot of discussions, a lot of closet issues around these ideas here, another one that we like it's a dual stack for IPv6 was up in 2018 and it was closed like uh last year that took like uh yeah just three years to get like the full thing.

A

The full story like uh roll out and everybody agreed on those things sometimes takes less sometimes sometimes takes more time, sometimes like never end right. For example, Ricardo opened one idea like for the printing HTTP life liveness check, and this idea didn't like went through and they just closed the the idea they.

B

Closed my issue in my face yeah then maintainers.

A

Okay, there's a lot of modern examples. You can go to the enhancements uh repo and then in the orders, and then you can take a look, but why, like some features, are like just refused or not moving forward, for example? Thus, the first question: if you want to open the feature ideas like this, it's like solves a wide problem, or a very specific, like very Central and very specific use case, for maybe for your company or maybe for your use case. The other one is like it's bring or solve a security concern.

A

That's like gonna fix any security issues or maybe close the door for possible issues in the future. This is fixed or solve a performance. Concern like there is any performance issues in the in the cluster in the kubernetes itself that this feature is gonna like fix or help to improve a little bit- and uh this is this feature gonna be a breaking change like for something like in the future, like remember that there's no breaking change allowed in the ga when you move forward to ga- and this was discussed before- like uh what's the conclusions.

A

What's the the other discussions like people are talking about those things or not? Well, just a brand new idea.

A

Okay, then, let's meet the cap. Like you should know, people know that this is Stephen gustos. He was one of the person behind the implementation of the cap itself and then we like we like to make jokes and memes with Stephen as well.

A

Okay, what's uh cat means cap means, like kubernetes enhancement proposal, it's a way to propose a communicate and coordinate the new efforts in kubernetes Project itself. It's it's. This uh caps are still in uh in beta, but it's a mandatory for our enhancements, starting the release. 114., you can scan this care code that go to the the page in the GitHub repo that have all the explanation for what is the cap? What's demanded our Fields we're gonna cover this uh try to cover a little bit in this 30 minutes, Chicago? Okay, that's.

B

It me yeah, okay, thank you. So, uh first of all we do discussions everywhere, so we have what we call in kubernetes, a seed which is a special interest group, and when you want to add something in in kubernetes, you need to go to some Sig meeting and say: hey I, want to add some new feature. Then the seek may say yeah we agree, or you can get stuck like for six months, bring in this discussion to every meeting and say hey.

B

We want to implement this new admin, Network policy and yeah, hey, Andrew and and get like stuck in the discussions. So there are some things that you need to understand when you, when you want to bring that to the community like. Is this thing an enhancement? Does this need like a blog post? Does it require more sigs? Is this just something for Network or like network security, and maybe the API or whatever? Does it redesign something?

B

It's a big effort in part impact user experience and so on and mostly like people can complain or notice about it right. So if someone can complain or notes about it, probably it's a cat, that's not like a bug fix or something like that. Then we discuss and we discuss a lot. So this is an example that I brought here, because uh that's something that I was close to some time, which is the clusterscope admin Network policy that wasn't called admin. Network policy, it's 21 uh who who actually every time, wondered like? Why?

B

Don't we have like a network policy that the admin manages instead of like each link space here, I'm, not sure? Okay, we have one two okay, so uh it's the idea is actually me as an admin. I want to create some firewall rules, each that can allow my pods to go to each DNS and so on. So it looks just like 670 comments to get this thing merged. It was pretty fast right, so it took like one year and a half just people discussing how this could uh be created or not.

B

Then you go to the alpha implementation. Then you discuss again and you go to the bad implementation and before reaching GA a feature can always be rejected. So it happened uh that you say yeah. We want to implement this, and then um you know what we don't want this anymore. We think that's going to have some impact, so it's kind of frustrating but yeah it happens, and then we discuss a lot more right. So the conclusion is actually yeah. It takes some times because we discuss a lot. Thank you.

B

Folks, no I mean it's a it's kind of part of the process. Yeah.

A

The adjuster one quick comment like when uh the discussion happened, like, uh as you can see in this diagram here like before the alpha implementation, there's a large discussions and then, when the feature like that, people agree like let's move forwards. Let's implement this. This feature it goes to the alpha implementation and then it's gonna like after that gets merged.

A

It's gonna like roll out to the the release, and then it's gonna be like a a little bit like a running in the cluster like getting released for a few releases, and then the discussion, like people start discussing about like how to move this to Alpha to better and then there's a lot of other discussions involved with it. This is going to make require any change in the API we're going to require any like Breaking Chains in this Alpha to better it's allowed to have breaking change.

A

That's not a big problem when you move from beta to GA, then is like another conversation and there's a lot of other process around that. To start like, say: okay, we need to stabilize in beta we're going to stabilize the API and all the stuff. Most likely in beta you're not going to have like any breaking change only if you move from better one to better two, but most of the time like people try to avoid having like API changes and then, when you like.

A

That's a lot of other discussions involved in that and then you move to J and then you like. Finally, uh it's like uh in GA and then it takes a few release to get all this process. You.

B

Do like five lines of code in three months discussing something at least three months: I'm gonna, be I'm. Being really nice seems like day job. So uh how can we get this feature in kubernetes? It's like. Usually. This is something that we see a lot like some folks. They just say: hey I need this thing. They open an issue and, like yeah, keep smoking the developers every month like hey. Is this ready, hey?

B

My boss is asking me: is this ready and then you have some folks that they just say: maybe we can Fork the project and if you for the project, you are going to have some problems, because maintaining kubernetes is not that easy, like I, think the code, it's just like a huge code base, and uh actually the idea is discussion with the community.

B

It's the way to to get that uh I decided to bring some real life examples to you, so you can see actually how this goes and uh and uh what can go wrong actually, because me and Carlos, just saying yeah so we'll just take time, because we discuss a lot like you folks, you people that kubernetes you like to talk. That's not that case.

B

So uh we have the first case, which is you figure out a way of implementing some feature and then you'll say: okay, whatever I'm going to implement this this way, this may see the right way of doing that, and then you figure out during the discussions that it may work right. But probably it's not going to work on the right way, so uh I I started implementing my first cap in kubernetes. Was this what we call Network policy part range, which was just a new field like everybody was like Hey?

B

Can you just add this new field in the API like? If you have the starting part and like you may have a new field calling start to end right and uh yeah I mean it's a really common case.

B

So the first idea was uh in the network policy specification just having a new field called Port range and you can do from into and the exceptions pretty cool right. So that's easy.

B

The thing is that why you're discussing and saying yeah we can go through there uh network was like um so if my network plugin doesn't know about that, you are just going to open all of the fire rules for everyone. So is this gonna work, yeah sure it's the pigeon. It's working right, my network policy, it's covering all of the range of the parts, so it's working the way that I. So uh we we based on that discussion.

B

We reached the conclusion that maybe we should start simple and just adding a new field which is like the end part. So at least, if something fails for the network plugin, it will interpret this way without, like the field here right, so you don't have in in the network plugin it wouldn't implement. It wouldn't interpret uh this network part as like anything so was the right solution. Maybe, but I was going to have some problems.

B

The second one second case is when it has this great idea in 2019, I was like on the left side when I started, implementing stuff like yeah, pretty I'm, still pretty, but a bit more fat right now and uh and then in 2022. You are still discussing uh admin, Network policies or any other new feature, so some features they take time right.

B

So one of the examples was like adding colors to cubecto and I. Want you all to speak with me? That's Cube, CTL team Hawking is not here right. So that's a cube CTL! Thank you. So there is an issue since 2018 about that and the problem was: can we add colors to cube CTL output like describe get pods? Sometimes you do like Cube CTL get pods and you want to see in red if something is in Crash loopback and in green.

B

Something is running right cool, so you have some workarounds for that already, but I don't want to install and have a lot of workarounds like a cute color plugin I want to have this inside and then you start having discussions with the community. Like can the cars in practice scripts that rely on Cube, CTL output, so imagine you are like using Cube, CTO and GitHub actions and like the red color actually can be interpreted as delete. Everything in your cluster right can happen. Can someone write themes for outputs like I like Dracula?

B

Probably someone likes other team like Dark Knight, something like that send what about autonomic people I think it's colorblind, actually the the right yeah, but like people that cannot make differences between red and green. So you need to think on the accessibility of that right and finally, and which this one is the most difficult in kubernetes today. It's like is the maintainer really into long-term, maintain the solution, because sometimes someone just jump into the repo and say hey.

B

This is the new feature: I can go through there and then like okay, the future is fine, but when you have some bugs on that who's going to maintain that right, so it's it's kind of a discussion that does it. Is it worth to have this, so there is an example on that we have a work in progress. Eddie is, as an example, is taking care of that for a new qbrc file and uh was like hey. Is this the best solution for that problem?

B

No, but it actually generated a new discussion on how can we make Cube CTL configurable for the users right so cool.

B

Okay, the third case is when actually when something gets cheaper, so it's like after four years discussing- and you say um you start implementing- after all, all of those discussions- I, don't think it's gonna work, but it works.

B

uh Who uses this or less in the cluster? Who knows who? Who knows who doesn't know? What is this realized? Actually I can do some Okay cool, so this Rolex is is like a Docker image that doesn't have anything other than your program. Whatever you want to run right, so it was made for that. You don't have curve, we don't have the wget, you don't have bash, you don't have anything there. So it's kind of hard to debug.

B

You can do Cube, City I would say get into that and uh it's gonna say: hey I, don't have shell scripts here, I don't have a show. I, don't have a show interpreter nothing. So, uh since 2019 there was a discussion on how can we rely on those really secure images without anything and still I need to the book? The bug that those stuffs right I need to see if the network is working.

B

Thank you. So this is a really hard feature, because, besides saying like yeah, it's just like, can we create a new Cube city of the book program? You have a lot of Machinery under under the hood like you need to implement a way that the cubelet knows that it needs to attach a new container into that pod and create a new network, and so on other things, so it got merged, but it got merged like uh where is it? I can see it but yeah.

B

It started on 116 like 2017 and it got stable in 125. So that's that's! Just nine releases each releases, it's like three months or four yeah yeah. You may you make the calculation I'm bad at that. So that's a four years, maybe okay, the last one and I like this one, because it's a who does red hat installation things here, who have red hat I, mean red hat basic distributions who has ever installed some red hat-based distribution. Linux?

B

Who does this as the first step.

B

Me too yeah, so Red Hat Linux distributions based uh that's not I'm, not talking about red hat, but I mean this basic feature has something called SC Linux that was developed by NSA, and it's really really really hard to maintain and it's enabled by default in Red Hat distribution.

B

So the idea is that you have some security on that right and the first thing that people goes and do is like hey, I'm, just going to disable this thing, because I don't want to manage security in my redhead installation whatever, and in kubernetes we did something similar, which it was pod security policies which is not enabled by default, but some admins they enable by default and when they do that.

B

The easier thing to do is actually disable that in your owning space, so I do that. A lot of people I know that do that you just don't need to tell well right, like I.

C

B

See my manager here- oh he's there. Okay, sorry, I, don't do that in our production clusters, so kubernetes is not secured by default, and I wanted to repeat this to yourself like kubernetes, it's not secured by default right.

B

So the problem was. We need to establish policies that block my pots to run within security configurations. I, don't want to any part running with host Network running as root and other things right, but PSP. It's it's really hard to configure in our workflow. It depends off a bunch of knowledge. It depends on the user and you as a user. Probably you don't want to know all of the permissions that you need. You just want to know that you can run in a secure way or not right.

B

So that's a at least that's what I expect if I try to do something that I'm not supposed to okay I'm, not supposed to run this as rude so I'm just going to drop this configuration and still you as a user, you can change the permissions with Posse security policies, so you can get for yourself a better permission right.

B

So I don't think it's a good idea from the security perspective that I can change my own permissions, so it ends up being easier just to bypass like I said Linux, and uh particularly the policy uh even having a lot of discussions.

B

It was deprecated uh I, guess in 1, 20 121 yeah and remove it on 125 right, so it never reached a ga which is really important to know, because we cannot deprecate anything that reaches GA unless you have another GA, that's compatible to the previous before, but it was better and even in better, the community decided that yeah. This is too hard, let's replace to something else right. So even after all of the discussions and all of those Cycles in better, we say that.

B

Okay, let's block this, uh so there is a new feature called security admission, it's configured on a namespace basis. uh It applies to all of the workloads and the community decided to make it simpler right.

B

So you cannot configure a lot of stuff on that, but at least you may have some baselines on security, so it was like yeah after all of those discussions, but we think that we can just go through this, not not leaving the user a lot of choices, but at least giving them something better that we had before and allowing them to be a bit more secure.

B

Okay, you wanna, you can finish, we can do it together. You can think it yeah how much time we have. Okay I was pretty fast sorry, so uh some conclusions that we have first of all features they are hard right so uh and I've been in your spot for a while in kubernetes as well. Thinking like hey. This is easy, like I, don't know how to code, but maybe it's just like three lines of code like stop being a jerk, come on folks like they are hard.

B

We need to discuss a lot and uh kubernetes code base. It's huge right, so that's something really big and we cannot fast everything in kubernetes today. uh A new feature needs like to you need to test all of the scenarios. If something new breaks, something old and based on the combinations you may end up with each test running like for maybe one week, so it's hard to test all of the things. uh If something breaks people are gonna met, it are gonna, be mad at us right so hey.

B

Why did you implemented this call that broke my workload? It was working fine, so I cannot migrate anymore right. So we need to be extra careful when changing something. Unless sometimes we just don't, have people willing to implement and follow the whole process and yeah. It's tiring. It pays its price, but it is style. So uh I am like I I can see a bit some of uh kubernetes, maintainers or members here and I can speak for myself, but I know that the other ones they know as well like.

B

Sometimes it's just tiring, like you start implementing, and you need to discuss a lot and you have your day job and not not. All all of us are paid to work on the Upstream, so yeah like people need to understand that and if you are willing to have some new feature, please join us like we need people to help us on on the discussions and it's fun. Actually, the discussions are great. You make a lot of jokes during the meetings.

B

Some takeaways don't give up right. So kubernetes needs ideas. uh We are here actually to tell people why it takes time, but still we cannot know everything that everybody needs and if we are missing something, that's uh that's a really required feature. The execution, sometimes it's hard, but it pays its price. Take a look into past enhancements, we'll have some QR codes uh and we are going to upload the newer version of this presentation to schedule the other one. It's a bit like it's already deprecated it didn't reach a ga. Ga is now yeah.

B

Maybe this is GAE I'm Gonna See. By like the questions after yeah explore the features trick new one track, new ones that are in Alpha. We need feedbacks even if they are alpha and you want to jump in. Like say: hey I saw this as an alpha I have this test. Cluster I can Implement on this test. Cluster I don't need to keep an SLA, so I can just add this. In my cluster and my users, they are not going to complain. You can do that like as well production clusters.

B

Just don't tell your boss, don't be shy, bringing new ideas. There is no right or wrong. So maybe you have this new idea and you say yeah. This is stupid, I'm not bringing bring this. You can call me on slack. You can call Carlos on slack. You can call a lot of us and it's like if, if you don't want to speak public publicly, but please bring the ideas if you think it's worth to implement it, raise the issue bring to the seeks as well and understand the reasoning.

B

uh If someone says no, we are not going to implement that as well. That's not because we are jerks. That's because probably we had some discussion on past and saw that maybe this is going to bring us more problem than than solution and kubernetes code, but we are all nice people, 99 of us, not Carlos, and not only of called kubernetes is made, so you can propose improvements in dogs, feedbacks feedback in your own experience as you're as a contributor as well.

B

If you start contributing- and you think it's too much harder to contribute, we need feedbacks for all of those things so just feel free to reach us on slide. You are around that. We take some. We think sometimes some time to answer, but at least we try to answer everybody. Okay and uh I.

A

I want to speak something else. I have just have one I, remember just one another example that that doesn't touch the kubernetes code base itself, but we are planning to move uh part of the inclusive naming what group we are proposing to change the kubernetes kubernetes KK, the the GitHub repository for the master Branch to the main branch like becoming not not calling Master but main. We are discussing this to just to have an idea that doesn't touch code base.

A

We are discussing this size uh last year and we didn't Implement yet because that have gone affect not only the entire kubernetes organization but going to affect Downstream users that people consume kubernetes and build kubernetes on their own. Like we, we send a survey. Last year we got like a few persons from different organizations. Replying we decide to not move forward in that time, because the we just got like 10 answers and I think we.

A

We have much more organizations that rely on the building on the kubernetes on their their own infrastructure, and then we are like just to say this is like a simple renaming, but it's not that simple and inside the kubernetes organization we like need to scan the entire uh repositories, jobs, Pro, everything to know who is relying on the master branch and the name of the like to change. We did a lot of work. This is like just a a normal rename, but in our kubernetes organization, that is a huge stuff. Is not that easy in.

B

Github is just a button yeah right, that's not that way. Okay, folks, I, think that's all for now. Thank you, everyone, and we have time.

A

We have a question yeah. If you have.

A

B

B

Karen is with the microphone. If someone wants to make some questions, Andrew cannot make questions.

D

Thanks um I was just wondering what your thoughts would be about, having a either a product manager or product management sort of principles included in this um type of process and whether you think that would work wouldn't work or how or how that might be best utilized.

D

A

Me see if I understood, the question is like including project managers and.

D

Not not a project product, yeah yeah, so.

A

I'm, a product manager yeah. We have some persons involved with that program and a product manager in the community yeah like, for example, uh in the single release. We have Lowry and she is working like to create a vision and create a roadmap, and we defined all this stuff for at least for the Sig release itself. But that impacts the other six as well, then, like I would like to see more uh product managers and stuff like running and helping the the other six I. Think that's nice.

A

It's a nice thing to have, but uh again like we need those people like to jump in and show up and participate. Yeah.

D

So the best way is just to jump into a Sig and just yep see what you can do from there correct sounds good. Thank.

B

E

Great talk guys that was really awesome. um I have a question on you had a slide. That said, the the flow chart with discuss, discuss, discuss, discuss right in a constant feedback loop. Do you think there's a way to, or do you think, there's a better way to update that process, to make it more exciting for contributors who want to get involved in like we're contributors right, we want to write code, we don't want to sit and write docs. Do you foresee some optimizations to that process to the cut process that could help.

B

You want to answer okay, uh so I am someone that actually don't that's not that I, don't like the kept process, but I got burdened by that as well as you Andrew yeah.

C

Right so we've.

B

Been uh dealing with a lot of caps, and sometimes we just get tired I think that one thing that actually could make caps more uh exciting.

B

It's actually removing a bit of like the discussions being something that uh that formal and maybe having more opportunities between the contributors together together and do whatever they think they they want and then show the result right. So, uh but but that's not about the process, it's more about like how people they need to deal with that right. So, hey I, don't want to keep writing these documentations And discussing my whole life like for one one year and a half, maybe I just want to make my proof.

B

My point so I think it's more about like the contributors to showing how this would work like they're, changing their own mindset and just going ahead and proposing to to like to detect leads like hey. We did this and we've seen a lot of this happening in c network recently right so people just say: hey. We did this. If you don't want to do, if you don't want to use that, we are going to make our own product like kaping. Is one of the things?

B

Maybe it's a way to go on that right, so.

E

B

E

Agree. Thank you.

C

Is is the cap program if there is a program measuring the actual, like Tech time versus cycle time versus like interactions because like at some point, let's say it doesn't look like it's going to take less time to get features in it's going to take more time so in the manufacturing industry, this was figured out so like they decided like what is the the lead time Tech time and so like when you do process analysis like this has been figured out to understand where actually the time is being spent.

C

Is it just waiting is waiting for someone to comment? Is he waiting for someone to write code? Is it during writing? Code is waiting for someone to approve right, so those I think folks will need to get a little bit more deterministic into identify like where is the actual wait time in each of those process, because it might be three years, but it might be three months of work and everything else is waiting right. So do you see anything being done in that sense?.

B

C

Answer is no, it's fine.

A

Yeah, like we need to think that that a lot of people that is like a tech lead and uh like involved in the community, they are doing the free time right and uh sometimes that free time I only have my free time to review a cap or take a look or make a comment. We're going to be like in two or three weeks and uh like on Sundays yeah on my Sunday and I'm like I'm.

A

Not want to do that in my son I'm going to do the next one like I, think we should track more and uh have more better metrics on those things and then maybe starting like as we like, maybe have like some product managers that can track and push things, but uh I think we need still remember. That's like most most of the time is people are doing like in the free type, not in the company base. There's a lot of people. This there's not company based back-end workloads,.

B

Okay, anyone else, okay,.

A

Thank you folks, thanks so much.