Kubernetes Kubernetes Contributor Summit - NA (Detroit) 2022, 2 Dec 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: KCSNA22 - Gateway API Building a k8s API with CRDs

Description

Speaker: Rob Scott

Gateway API is the next generation set of Kubernetes networking APIs, covering load balancing, routing, mesh, and more. It's already been implemented by more than a dozen projects. In this talk we want to take some time to share our experience building this API, specifically focusing on the CRD experience. We'll cover some of the benefits and drawbacks we've found when using CRDs compared with developing APIs directly in k/k. This will include our concept of release channels, our versioning and compatibility guidelines, and our experiences with validation.

A

All right, I think we're we're good to get going here. um Yeah welcome everyone. Thank you for joining us uh today, I'm going to be talking about building a kubernetes API with crds. My name is Rob, as you may have guessed, uh this is going to be all about Gateway API and the lessons we learned along the way. Gateway API is one of the first uh official kubernetes apis, that's trying to be built with crds and we are learning a lot along the way.

A

uh If you just were at the admin, Network policy talk, you notice that it's another API being built with crds and work. You know just a little bit ahead of admit, admin, Network policy and what we're doing and trying to make the path just a little bit smoother. So this talk will go through some of what we learned along the way. uh So, first off uh how many of you are at least aware of Gateway API anyone, okay, great, uh how many of you have used Ingress, API or know what Ingress API is okay.

A

So uh what I think of with Gateway API is that it's really the The Logical next evolution of the Ingress API and load balancing apis in kubernetes. So uh really we built this API to be expressive and extensible. You may be familiar with Ingress API and realize that it's a very, very basic and simple API. With Gateway API. We tried to build a lot more functionality into the API directly. So all the implementations don't need to rely so heavily on annotations custom resources.

A

Etc trying to build this directly into the API itself started with a role oriented resource model. So you see we have three main resources or types of resources. In Gateway API we have a Gateway class, that's really similar to an Ingress class resource. We have gateways those generally map to load, balancers or entry points into your cluster and then finally, we have routes and I've highlighted HTTP route here, but there's TCP route, UDP route routes for any protocol. They exist. Our newest route is grpc route, but more and more are coming now.

A

Most of this talk is not going to be focused on Gateway API itself, but the high level idea of using crds to build an official kubernetes API, how that's gone and things that maybe could have been better. So why would we build an official kubernetes API with crds I? Think a key idea here is that we want to stop expanding the core of kubernetes. Kubernetes itself is an ex a very large project. That's only getting bigger if there are pieces that we can pull out of core and instead build on.

A

As you know, little extensions that you can add on that's something we should try and do and that's a goal with see with Gateway API using crds. We want this to be a plug-in model. We realize that although a huge set of users want load balancing configuration in their kubernetes cluster, not quite everyone needs it, so it doesn't need to be core.

A

It allows you as a cluster admin to choose which apis which bits of functionality you want to allow in your cluster uh and when crds are released or updated, they're immediately available. So anyone using a kubernetes cluster- that's older than 116., okay good. If, if you were I'd love to talk to you, but it sounds like we're good. So that means you can install this API in your cluster today tomorrow. Whenever and it works, you don't have to wait for the next version of kubernetes to roll out it's available.

A

Whenever you want to try it out, it enables this completely independent release cycle. We're not our releases. Gateway API releases are not tied to kubernetes versions they're just tied to when we think a release is ready to go, um and it's really good for long-term collaboration with a large set of contributors. We have some of the maintainers in this room, but we've got a great set of maintainers working on Gateway API, and these numbers are probably slightly off.

A

Tim could probably correct me on the Ingress API number, but my guess is we had somewhere in the ballpark of five contributors over the lifetime of Ingress API. So far with Gateway API. We have a hundred contributors to the API so far and that number is only growing so working with crds working in a separate sub-project has really enabled us to get a lot of new contributors involved in the API design and development process. Within kubernetes, but, as you might guess, crds are certainly not perfect. They have their flaws.

A

uh First off crds can lead to a fragmented ecosystem. Some of the risks we're dealing with here are that, as you start to see more and more of these apis, these official kubernetes apis developed outside of tree. It could become easy to miss them. For example, it could be difficult to find the full set of kubernetes apis if they're not all defined in the same place. We could end up duplicating a lot of the work.

A

For example, we kind of were Trailblazers with what we were doing with building out our docks building out our infrastructure and admin Network policy.

A

I know you had to do plenty of new work to try and make that work and if everyone's doing that same set of work, that's not great for the ecosystem as a whole, uh then because crds are completely untied with the kubernetes release cycle, there are going to be some Cloud providers, some cluster providers that install the crds for you manage them for you, but they may do it in a slightly different way. So a risk is that this experience makes kubernetes just a little more inconsistent depending on your provider.

A

There's different versioning terminology: there's the risk that you know. We introduced some Gateway API specific versioning terminology, there's a risk that other projects could choose slightly different ways to version their crds, uh and you know if we keep on with this pattern, there's a risk that we develop this massive maintenance burden that becomes a problem for the ecosystem as a whole.

A

So these are all scary things, but I do think this. This is worth it. So we need to find ways to mitigate these kinds of problems and that's what I'll try and work through and and before I forget crds do break in unique ways. uh Maybe some of you have experienced that before and had fun with crd's breaking and it's never a fun day again. We need to find ways to make this experience a little bit better.

A

uh I've had the experience of working both on crd based apis, like Gateway, API and Upstream apis, like endpoint slice, Ingress Etc, uh and there are some things that I still miss from working on Upstream apis defined in the kubernetes repo itself. We had really great guaranteed expressive validation, so you knew that your validation rules were for sure going to get executed and you could write just regular old go code to do the validation. You don't really have that with crds.

A

You might if you created a web hook, but then you can't guarantee that it actually ran.

A

Then you know uh inside Upstream kubernetes. We have better support for conversion between API versions. That's a little Messier inside crds and although I I really like that this API the Gateway API enables us to go outside of the bounds of kubernetes versioning, it does make things just maybe potentially a little more inconsistent if different cluster providers are managing and installing these apis at different times um and then I just miss the wonderful existing infrastructure we had in Upstream kubernetes already.

A

But with that there's some things, we've learned along the way that I think may be helpful to anyone. That's either looking at building a crd, based, API uh or is trying to consume a crd based, API or just understand the design process and thought process as we're working on this um Tim recently was had a session where we talked about all the things that we messed up in kubernetes apis and there's a long list. I'm sure I'm sure we're all familiar with things that we wish could be better in a kubernetes, API or two I.

A

Think one of the key problems for any API is that it is hard to get feedback when it's still helpful. So, let's, let's play this this life cycle around for a second. uh You start you build out an alpha API, you think. Okay, this is this is for sure what we want, but let's get some feedback and to get feedback.

A

You need to have something that actually implements that API and when you have something that implements that API or some things that Implement that API they're increasingly not wanting to change because they already have something that implements the API. You started with so the more things that implement the API the harder it is to change anything about that API. But you need more things to implement the API to get feedback. It is a vicious cycle. I've summarized it here, maybe a little bit easier.

A

You need meaningful usage to get meaningful feedback, but when you have meaningful feedback, breaking changes are really painful. It's it's an awful cycle this. This applies to any kind of kubernetes API development. I I have some things that I think can help with this. But this is just a fundamental problem that we struggle with throughout kubernetes today. uh You know we really want to build quality apis, but we need that feedback and to get that feedback we need some usage.

A

So uh one of the things we've really been trying to do in Gateway API is encourage that earlier feedback. So we really want to take that feedback loop that we have here and make it as fast as possible. So when you're, you know when you're, seeing in slack or at kubecon like talks like this asking for feedback about an API, please please come join us tell us what we got wrong before it's too late.

A

Please, uh you know we do what we can, but if you have just a few spare minutes, we are very, very biased towards being implementers of an API, and so we have these thoughts in our mind of well. This is what makes sense as an implementation, but if you're somebody who might want to use the API in the future, if you can come to us and say hey, this is an awful ux. It's better to know that sooner well. We can still make meaningful changes before we get to the breaking changes.

A

Our painful stage of that life cycle.

A

um Similarly, if, if there are similar apis, we can get feedback from them, I am very familiar with the mistakes we made in the Ingress API, uh and so you know this is really the next generation of that API and so we're at least trying to learn from those mistakes and try to build better on top of the feedback we already got from previous apis, similar with service type load balancer.

A

This is also the next generation of that concept so trying to learn from the mistakes we made there and at least do better here and finally, we really need to clearly document the stability of each release. So we need to accept that sometimes things will break, because if we want the best final API, we need to be willing to break things when we realize that what we released initially is not quite right.

A

Okay, lesson: two crd upgrades are really really easy to get wrong uh boy. I have struggled with this. One uh I took a while to try and understand this reaching out to a bunch of people in API Machinery to try and understand exactly how we should be doing this because again we're working with crds. This is a less documented workflow, especially when you have multiple implementations of the API and Gateway API.

A

Just like Ingress has a lot of implementations, I think we have 15 or so now, so we need to make sure our upgrade process works. Well, even when you have you know, multiple implementations in the same cluster, so let's walk through some key vocabulary. Here, that's worth understanding so for each API version. In a crd, you can Define one the storage, so whether this API version should be used as storage or not only one API version can be set as storage version.

A

Then two served should this API version actually be usable to the users and then three deprecated. It is this API version deprecated. Something like that. I want to throw a warning on and say warn the user that hey this is going to be removed in the future.

A

So those three attributes require some very specific sequencing when you're going to go through this versioning process and again uh this is my best, my best idea of the safest way to do this transition. So let's go through a four-step release process, starting with release one or introducing the first Alpha version of our API.

A

Then in version two we say: okay, we that we did everything perfect we're going to take that API version and move it to Beta. So in that release we Mark the Alpha version as deprecated, and then we introduce V1 beta1 and just let it exist. It's not storage version. It's not deprecated! It's just it's just there, then in version three, we don't serve uh alpha one, but it's still there just for conversion. It's still deprecated and it's no longer the storage version, storage, Version, Moves to Beta, 1.

A

and then finally, four releases in we can remove Alpha One. So this is a very, very fine art of going through this upgrade process and trying not to break users along the way. But again, this is not obvious to most people that are actually implementing crds. There may be faster ways to do this, but from what I can tell this is the safest way to make this transition.

A

um These are things that again are not obvious when you're creating crds, but we're trying to make that path just a little bit smoother for people that are building out crd based apis, so a four-step process uh hoping to get something like this more clearly documented.

A

Now, storage version is a really key thing. You notice how we waited until the third release to set storage to True here. That was pretty important in this whole thing, when you set the storage version, that means what what is being served in what is being stored in FCD.

A

So if you set the storage version and then you somehow roll back, and so something is used, you know stored as V1 beta1 and you roll back to an earlier version of the crd that doesn't know about V1 beta1.

A

The best I can tell is things will break I tried to ask a few people, and no one really knew how it would break it. Just that it would not be good so again have to be very careful when you bump the storage version here there is a thing called Kube storage version migrator, which can automatically bump the storage version of your resources for you, but it is not installed by default in most kubernetes clusters and I.

A

Don't know that it's completely perfect, but it seems to be the best thing that exists today to get around this specific issue all right. Third lesson there crds are great and they enable a lot of good benefits, but sometimes the extra layers on top of crd versioning become very, very helpful.

A

So with Gateway API we had you know the base idea of crds. But how do you turn this into a project into a meaningful release? So we had three goals. We wanted to allow resources to graduate to a stable, API version, because you know the core of those resources were stable, but just like Upstream kubernetes. We wanted to still be able to add capabilities and features to those apis in the future, for example HTTP route.

A

It has a very core set of capabilities that are already included, but we know there's still a few things that we'd like to add sometime in the future, but we don't need to wait for that to be complete before we bring the core resource all the way through, then we wanted to provide a well-defined release bundle. So every time we release Gateway API, we want to include everything: a set of conformance tests, a web hook and a crd and attach some kind of semantic version to that.

A

And then finally, we wanted to provide that kind of experimental release track. So the idea that again, there's this stable set of apis that you can use that's your default, but if you want to be one of the the best best people for the community and actually provide feedback early on our experimental features, you can install an experimental set of our apis and give us that feedback so that it's still early enough to make a difference. So experimental release track was very important to us in Gateway API.

A

So within Gateway API, we really had two new version constructs one was a bundle version. That's kind of that semantic version We introduced within Gateway API, so every release of Gateway API has what's called a bundle version. So 0.5.1 I think is our latest release, we're about to release 0.6.0, and that includes crds conformance tests and a web and a validating web hook.

A

The more the newest concept here, and probably the most important one is that of release channels. So the idea that there's an experimental track uh that you know instead of feature Gates that you might find in Upstream kubernetes. Instead, we just give you a different set of crds, so we have our standard set of crds and then a superset which is all the standard things plus some experimental things layered. On top that we include in our experimental set of crds.

A

So whatever release you can choose, I want the standard version or I want the experimental one. So if you choose the experimental one, that's great, please don't use it in production, but again it's really great to get that early feedback, so to go just a little bit. Further release channels are really just experimental, Channel and standard channel. So experimental is this superset of everything that's inside standard, Channel, plus a few experimental things along the way.

A

So when we're adding a new field to the API, regardless of what resource it is, it always starts an experimental channel of the API everything that's new, no matter how confident we are in it or not. It starts an experimental and then, if it meets our graduation criteria, it may graduate to the standard release Channel or if it doesn't meet our graduation criteria. If we have some good feedback, that's telling us this isn't the right path, we'll either drop it from the API or make changes and try again in the next release.

A

Resource life cycle is very similar. Resources start in Alpha. If they meet the graduation criteria, they go to Beta. If not, they can either be removed from the API or we can try again all right. So final lesson of the day. There are lots of resources to learn from here. I'd summarize some of the top lessons we've gotten just from talking to people in API, Machinery architecture, Etc, is always always always have clearly bounded validation for every field.

A

If you do not add a Max size, a max length, they some kind of basic validation, you will regret it forever. I promise you. It is awful, uh always bound your validation and make sure that you're, starting with tighter validation, it's much easier to loosen it than to tighten. It retroactively at basically impossible to tighten it in a backwards compatible way.

A

So within Gateway API we've done everything we can to start with a very small set of things that we allow and if the needle, if somebody really needs a new feature or new value to be supported, we can allow for it at that point. But we start with a very small set, a very small length, Etc and then grow from there.

A

How renaming Fields is awful? I recommend not doing it ever very painful, so, especially with crds. You don't have a whole lot of great conversion mechanism here. So I just recommend get your name right, the first time, hopefully, and then don't leave any room for any kind of ambiguity in your API spec.

A

You may write something and say: oh well, it's very clear that you know what I said is interpreted this way, but you would be surprised how many different people interpret that very unexpected ways, so one be as clear as you possibly can in your API spec, but two. If there's more than one implementation of your API, you really really need conformance tests and you need them early.

A

One of the things that I think sets Gateway API apart is we start with conformance test while we were still in Alpha and we have a very significant list of conformance tests. I've lost track of how much we have still not perfect, but it's a you know. Conformance tests are the one thing that don't have any ambiguity: either you pass or you fail, there's there's no room for well I'm, not sure if the spec means this or that this is what the conformance test says and then finally uh clearly document Your Design decisions.

A

Many of you may be familiar with the kubernetes enhancement process. That's really just a set of design. Docs that explain every major feature why we developed it Alternatives we considered uh anything else about it. We have the same thing in Gateway, it's called Gateway enhancement proposal. It's very slightly different, maybe a little bit simpler. But again it's been invaluable to be able to go back in time and explain hey. This is why we made that decision. We did consider that alternative you're proposing now, but this is why we chose against it.

A

uh These slides are uploaded. I recognize these links on their own are not that helpful, but these slides are uploaded on sked. If you want to go through these links, but again, I think Googling, kubernetes, API conventions and API change guidelines will take you exactly where you want to go as well. The key thing is that API conventions are the starting point. If you're ever going to design your own kubernetes API, well, any API that uses kubernetes highly recommend understanding the API conventions.

A

People much smarter than myself have contributed to them and have really helped me understand how to make an API convention API that avoids many of the mistakes we've made in the past. I'd say many of the many of the things in API conventions are designed to avoid mistakes. We previously made then API change guidelines.

A

These are mostly focused on Upstream apis, but most of the concepts apply to crds and then finally, I want to call out Gateway API versioning guidelines, because we as Gateway API, it looks like I'm just about out of time, so uh we as Gateway API versioning guidelines, because these are the first set of versioning guidelines inside kubernetes that are tied to crds and there's just a little bit of difference in what it means for crds and then cigar, Sig, API machinery and sigoth are amazing. Resources uh attend one of their meetings, ask and slack.

A

They will help answer any and all of your questions highly recommend that uh thank you to everyone in those six that has helped us with Gateway API. So much um I've got a wish list. I don't think I have time to go over all of it. Okay sure I'll go with that. um So you know what I would really love here is just a base set of infrastructure and patterns that work for every project that was coming after us right.

A

We've tried to learn all of this as we went, but man if everyone has to reinvent the same wheel again and again, it's not great for the community as a whole. So if we can take these lessons and turn it into a base set of infrastructure patterns, recommendations that can help making these crd based apis just a little bit easier to work with that'd be a great first step. Then more centralized, Discovery and docs for these official crd based apis.

A

That's going to be huge for adoption of these apis and then some of our API conventions and change guidelines are built with some assumptions about apis being an upstream. That may not always be true as we're building more and more crd based apis.

A

uh And finally, you know crds can feel like the wild west if there was some way to guarantee that you know as a controller that the crd I'm looking at has all the validation that I expect, and it has the schema that I expect. That would also be very helpful uh final thing. Thank you for all your time. Here uh we have lots of ways to get involved in Gateway API. We have way too many meetings, uh weekly meetings on Monday for the project as a whole. Then gamma is our mesh specific sub-project.

A

We we also have weekly meetings for that on Tuesdays and I really really want to emphasize. If you're interested in crd based apis any of this stuff, please come check us out. All our information is up here. We want contributors from all backgrounds and perspectives, uh really value the different uh perspectives that people can bring. So please again come join us we'd love to hear from you thanks.