Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 12 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: API Evolution With CRDs: Best Practices For Authoring & Fuzz Test... James Munnelly & Andrea Tosatto

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

API Evolution With CRDs: Best Practices For Authoring & Fuzz Testing APIs - James Munnelly & Andrea Tosatto, Apple

CustomResourceDefinitions are prolific in Kubernetes. With so many new projects being prototyped, developed and released into the ecosystem, it's essential to ensure you're designing your APIs in a scalable, well tested way. Projects like Kubebuilder and the Operator Framework provide plentiful tooling to keep you on 'the straight and narrow', but some fundamental issues still trip up even the most experienced Kubernetes engineers. In this talk, we'll go over strategies & tooling that can help ensure you are successful when authoring and evolving your APIs, including:

* How to ensure your APIs follow best practices/standards (and linting OpenAPI schemas!)
* Fuzz & integration testing API version conversion functions to ensure platform stability in your clusters
* Automatically validating your OpenAPI schemas are up to date for your type definitions
* Best practices in 2022 for writing validations & mutations
* Ensuring your OpenAPI schema changes are backwards compatible within an API version

James has helped bring various API groups to stability across projects in the ecosystem and both he and Andrea have experience with unique problems that are found when projects reach a certain scale & maturity, having helped take CRDs through a full alpha to beta to GA deprecation cycle.

A

Cool um thanks very much for coming everyone um we're going to get started pretty promptly, because we've got a lot of content to get through and uh everyone's got quite a schedule today. I'm sure. um So thank you for coming. This is a talk on API Evolution, with crds, so um best practices for authoring, crds, fuzz testing, all sorts and like how to maintain uh how to maintain your CDs, so I'm, James, Munley I'm, a field engineer at Apple. um It kind of means, I do all sorts- and this is my colleague here hi.

B

Hi, hello, everyone, um and that was after work in essery at Apple as well uh with James. We both work on the kubernetes platform at Apple, and this talk is really like about, like our journey, like internally, in like building ordering and and also like advising partner teams on how to extend API using series, and the Germany with crd is actually like start really like with the authoring part right is it's like any API like when you get like to design it and implement it and think like how to do things.

B

The first challenge is really like about the modeling um in in the context of kubernetes. There are a lot of examples right in the community. The community is quite big uh at times too many. It can be overwhelming like to find like the right information, the best practices best practices, something that to me is also quite funny. Is that often like, when, with Riker, we used to look into the standard Library as a reference on best practices? But kubernetes has been here very long and not always like the core.

B

Resources are the best representation, because there are some trade-off and the community evolve their understanding on the API. So maybe not always the core resources are the right place to look at on how to implement things um and also like there are also other challenges related to the design of the crds right. In. Do we model everything in one single resource? Do we use more and actually like kubernetes, give us like this object?

B

Ref uh field oops object field which help us like to backup crds, but at the same time, really increase the complexity of of controllers, because now you need to start like handling into the code like this, this references and as well, create cognitive or over for the users, but generally like one one best practice that we learn is that it's it's really worth like breaking down like the model into more uh resources itself and a good example.

B

If you think about it is like the certificate resource, for instance, like you, have water source, the 101 certificate or as well as well like as, for instance, like external Seekers, the map light to one concept, and then you can build the resources like that group all together these entities into, for instance, a concept like a store.

A

So yeah I think there's as Andrea says, once you kind of think about what your concepts are, and you think about kind of like where, uh where you have all these different concepts and how they relate to each other, you then obviously get onto probably the probably the thing you actually do. First, which is actually writing a schema um and writing a schema can be quite difficult.

A

It takes time to evolve it you make mistakes as you go um and yeah I think one of the first things that you then need to uh sorry as you're as you're doing this, uh the API convention stock, which I think most of you probably I'd, hope have seen. um You should definitely go check out.

A

It's really exhaustive, and this is kind of like the reference manual for best practices for like what uh what what are kubernetes feeling API is like and that you know building it like this makes it feel more natural people understand more um kind of like what, uh like people more naturally understand the resources and so on. I think uh one of the things I always say here is really kind of focus on talking to the community. About these things, because it's very very hard as Andreas says, there's a lot of resources out there.

A

um People make mistakes as they go, though V1 apis still have problems in them. Pods have been around for years, um so I think reaching out to the community is really a big big first step.

A

So writing a complete schema. It's required for V1 crds now, and we should stop talking about V1 beta1 crds, because it's you know it's the thing of the past. We should be looking forward. It enables really useful things. Cube CTL, explain who knows Cube CTL, explain here or has used it before yeah, so that that it's a really nice CLI um you just Cube CTL, explain your resource name and it'll actually go and explain to you all of the different fields that you can uh specify. Your descriptions are pulled through there.

A

You can uh display kind of like what the valid values are. All of these sorts of things. You can make sure that you've actually got a full complete schema by looking at the conditions on your crd. So a cube CTL describe on your actual crd resource, we'll show you those conditions, and once you do, this you've got far better validation.

A

The API server can uh kind of actually evaluate all of your resources, as they're, submitted by your end users, and make sure that they're valid that they, you know, don't have any typos in there, which is actually a really common problem. We see people even capitalization or just the wrong word. If users don't know that, there's an error there, they end up pushing these things out and they might make assumptions about how their application or their resource is, going to work that just don't hold true um and yeah.

A

This, as it says, matches the behavior of other core built-in types which I think for me at least.

A

We're good, um yeah I think that's a really important piece there, because people kind of become familiar with kubernetes, it's already complex. We want to really reduce that cognitive overload and overhead.

A

So what can we do with open, API schemas? First of all, as it says, we can actually uh perform all this validation defaulting without expensive network round trips to web hooks validation. Web hooks mutation, workbooks, which we'll talk about in a bit um you've, got quite a lot of flexibility. Here, things like max length on strings, it seems really like small, but if you don't specify a max length for these things, you can have on a completely unbounded amount of data up until the xcd limit, which can become really problematic same with lists items.

A

We've seen, people accidentally insert 10 000 list items into a huge list, and that can be really problematic too for the API server, especially when you look at managed fields and some of these more advanced server-side of play, stuff minimums maximums just helps users once again, regex is for Strings.

A

We can do all sorts there too enum values to actually make sure that the values that are used, that your users specify are valid and make sense, and that makes your controller code Far Far simpler to write because you're not having to deal with all of these kind of other uh deal with values that are unknown. uh Cel who's heard of Cel here as well, yeah common expression language.

A

um So this is a out of Google, I, believe and um originally, um and it it allows you to do some really really advanced stuff with your crds and far more uh a far more validation. So here you can see um we can do like comparators. We can say one field has to be greater or less than another, and things like that. We can check to make sure that certain values are within certain lists and so on.

A

You've even got set operations in there and making sure that things are immutable, so self equals equals old self. Make sure that we can't change anything so defaulting uh yeah.

A

If you have a resource, someone submits it if you think uh think a pod, sorry a deployment or something when you create that, if you don't specify the number of replicas, we can default that to one, because you probably do want that to not be zero like it's that kind of intent. When a user reads that object back, they can then see that they've got replicas one. If we don't have defaulting, it might be unclear to users when they actually go to inspect their resources.

A

What the actual, uh what What's actually going on there image pool policy is another example. You know, if you didn't know- and you didn't just happen to know- new users are really not going to know what the behavior is going to be um doing defaulting in your schema, as opposed to in web Hooks, and so on means that you can. uh These things can actually be applied on read operations as well.

A

So if you have resources that were created previously, you can then still make sure defaults are applied um and yeah just one other little tip for kind of making apis that don't confuse people um having fields that default based on the value of another field can lead to really kind of tricky confusing cases for people, because they might submit one thing which then changes there and then you have like server-side, apply kicks in another user. Another entity starts setting things and then that changes stuff up and it gets really confusing.

A

So just as a best practice, keep your defaults simple as well. So that leads on to a pretty pretty chunky topic, which is versioning and conversion which I'm going to try and take us all through today. Who knows about how I can't really do one to ten with the whole crowd? But how do you feel about versioning and conversion? Have you looked into it before everyone raise your hands? If you have yeah, okay, cool, so versioning is really really fundamental principle in kubernetes you've seen V1 beta one we've seen.

A

V1, you've probably suffered the pain of these apis being deprecated before as well and removed, and had to deal with all of your users going through it, but at its core, an API version is an API is a endpoint that gives some kind of guarantee that if you go and submit a resource today to this endpoint, it will continue to work in future.

A

We won't change the language, the schema and the shape of that resource, um the difference between kind of Alpha, Beta GA, here, um Alpha no rules at all, so I like I, which we have had to really kind of strongly discourage people, relying and leaning on to these Alpha apis Beta. Similarly, we might not change this schema in a breaking way.

A

However, like that resource can go away too, we can still remove that within a few releases and like stating that up front with your users makes it a lot clearer and, to be honest, again, be careful with beta apis, because a migration where you're actually removing something can be super painful, because users come to rely on certain functionality and features.

A

um We haven't seen a ga API go away. Yet, as far as I'm aware, we haven't done Cube kubernetes 2.0, but you know crds. We can move a lot quicker. You know, you've got your own project, so yeah, possibly we'll see that at some point has anyone ever removed a ga API from there thing cool nice, that's good, I, guess ish.

A

um So yeah conversion enables the API server to kind of, uh and in fact, I've got a nice diagram here. It enables the API server to help coordinate different clients that are working at different versions. So here you see nhcd it's hard to me to point up in xdd. All objects are stored in some API version. So in this instance here we've got the storage version set to V1 beta1. That means, as a user submits something they could speak.

A

V1 V1, Alpha, One, V1 beta1, whatever the API server will handle converting that into the storage version. V1 beta1 persist that into xcd and then from there.

A

If any other client asks for it in a different version or anything like that it can, then it can handle that translation.

A

You can see here in internal in like internal core types. We have the idea of like an internal version and a hub which is uses the Hub. So if we write conversions between our external versions that users use and this internal One controller runtime is similar except it just gets away, get rid of the concept it gets rid of the concept of um of the internal version instead just uses one of those external ones as its hub, which is kind of a little bit less to think about and deal with so yeah.

A

What can we do with crds here, because the API server obviously pars on a lot of this code, and we need to actually make sure that we're um we? We can't compile code into our API servers when we're using crds. So, first of all, nothing at all. um We can keep everything V1 Alpha One or we could call it V1 from the start and be really careful, I think that's, probably a pipe dream. Really um no op conversion I know in the Gateway API they've been really trying to push for.

A

Let's call this V1 alpha one, but uh you know let's try and treat it like V1 and go for a no-op conversion. So we don't actually do any conversion, we'll see how that goes. They haven't got to V1 yet, but um you know I think it's not bad. It's not a bad way to do it and it really forces you to think about your API up front too conversion web who's used conversion, Web book or built one who loved it.

A

Yeah um conversion workbooks Opera introduce a lot of operational risk. It's on the hot path for read operations from your API server, so that means things like uh well yeah. First of all, we need to actually make sure they're, reliable and running all the time, but it really is an extension of your control plane. So if you've got a team that manages the control plane, they really need to know about the conversion workbooks that are running because that can like breach slos. It can breach everything. It can really.

A

You know, bring a cluster down, um so yeah great escape hatch. If you do need to do something, but in the cert management project we had a conversion web hook for a number of releases, I'm really glad to say we don't have one anymore. We have V1 resources and I'm really reluctant to actually ever introduce it again. To be perfectly honest: um it's critical, API, server, dependency, let's say and also I think a lot of companies and a lot of teams.

A

Don't actually have a lot of experience running these and they don't fully know what those risks are. um Yeah, so yeah key takeaways try and make sure that you're like keeping this for as little time as possible. I'd say really really push through with your conversions like if you I keep it One release, we made a mistake of having conversion web hook for about seven releases and uh yeah. It's it's a lot of pain and also I would say.

A

Consider publishing, V1 only variants of your crds too for those that actually don't need all of that Legacy new users, for example. They can just start with V1 today, if they're picking up your project.

A

um And yeah just quickly before I hand it back over to uh Andrea. There's been some really interesting work in the kcp project, um which some of you may heard of it's uh talking about using common expression, language again, Cel to actually Define your conversions in a declarative way on your resources. So you can there. The API server can handle the conversion without calling out to web Hooks and it avoids a lot of these complex kind of operational risks.

A

So it's really interesting, like it's very experimental, as you can see it's in a pull request, not in a like merged Branch, but uh it's definitely worth checking out and uh yeah over 2008 to talk about validation and mutation.

B

Yeah, thank you James, so well, I mean conversion is one aspect of the resources here, this life cycle, but as well like where he was like to deal with validating and mutating workbooks.

B

um They generally like validating, where books are generally used like to apply additional constraints which cannot be expressed in the in open API in the open, API specs for crds, like, for instance like, even if we really advise not to do that, but some the state of some Fields might be valid depending on what other, like properties are set on the crds. There is no good way like to do that with the schema itself.

B

So at times this is like something that um people need to resort to and yeah and but generally like the way we really like would like to pitch. The idea of validating with books is real, like as a way to do policy enforcement um on on the life cycle and what the user and the properties of this year these resources that the user are supposed to be using. It's really powerful because in the context of web books- and we will be seeing this later- you also know information about the users like the namespaces.

B

The service account, and things like that that you can really Implement use leverage to implement like policies based on who is the client that is requesting that is making the specific creation of the crd at the same time like mutating, are are also, like very flexible, mean to apply additional changes on Create and update, and the real, like advantage of this compared to um to to the conversion.

B

Web book is also like that they are not executed or read operations so like in thermal scalability, depending on the resource very often like there are less Creations than that reads, so they even if they have like some of the operational burden of that is common to all of their books generally like there are more okay way of dealing like with mutation in general.

B

um If we look at webbooks in general, like a way more like a good example of when this can be used, is, for instance like to ensure that, for instance, specific platforms depending on implementation, that all the parts have a resource class set right, actually a priority class set. This is a typical example of where this can be used right, and this is something that is not enforced in the schema of the Pod object, but, for instance, you can enforce like specifically to your platform.

B

There are very nice uh Community tools that allow you to implement like web books in in both validating and mutating in your cluster OPI kiberno are two of example of projects, and the other thing that I believe like that is really powerful of webbooks, is that they really like combine with the warning API.

B

They allow you to provide like early feedback to users, on the reason why, for instance, like a certain request was like forbidden and and again like push left, the feedback to the user, rather are the relying, for instance, on specific conditions like ready or errors and having a validation after the resources created.

B

Of course like, depending like on, for which objects we Define these web books like they might come like with operational burden, especially regarding the scalability, let's think about like our book for pod objects like this would be essentially like consulted every time like that, a bot get mutated. So it's pretty like the challenge which web books is essential is that they are.

B

They sit on the back of your API server and, depending on the configuration the API server might be dealing like with forwarding a lot of requests there, so they affect the end-to-end like experience of the users with your API.

B

There are also like some challenges, like especially with series that are not that major when we kind of like are still building the schema, because we are learning about the resource or things like that that are related like to the life cycle management of the web books themselves right. So um it's really important to upgrade the webbooks before the crd get upgraded, and at times this can be also like an operational challenge.

B

um One thing that we want to mention, which is really interesting, I believe, is this cap, which is essentially like about again using cell in order to move admission into the API.

B

um We definitely think this is like an interesting feature, because it allows essentially like to directly evaluate these rules and perform these validation and mutation operation in the API, without requiring like any extra component and again like dealing with the scalability aspect, the latency and the operational management of this additional component.

B

So back to Jameson the testing methodologies.

A

Cool so yeah we're going to talk a little bit of testing like I, say um the one thing I say to begin with with all of this is really think about your API server testing versus your controller testing, so testing your apis and then separately testing your controllers, they're, obviously linked here, but it's really important and I think we often don't actually see enough testing in apis um controllers kind of is where people go and then that's that so I'm going to go through kind of the different strategies for this unit.

A

Testing integration, testing, end-to-end testing, with some examples. um So on the API server side unit testing um is anyone ever in a like use. The round round trip stuff yeah. You can see one cool two, um so yeah. Basically, this will generate completely random values for, like of your objects, convert from one version to another version and then make sure that they, the object's still the same.

A

So it makes sure that it's completely like round trip with round tripable to use the word that it is, but um it it makes sure that your conversions are correct. Basically, and that's really really important, because if it's not, then, if a user reads in one API version, uh they might start getting different data and especially with these conversion web hooks or any any kind of conversion.

A

To be honest, that can lead to a really confusing state for your cluster and for your client and that can really kind of start to break things again, and it will be subtle and you won't notice it and it'll be horrible.

A

um Schema. Fuzz testing is a way to actually ensure that your schema is correct. So that's taking your like go type that you've defined, generating random values and then applying your jet. Your schema to that to that type and to that generated instance of your type to make sure that we're not losing any data again.

A

So that makes sure that's effectively applying things like the pruning that the API server does and validation there to make sure that we are, we have a complete schema that actually does account for all the different fields in your go type, um perhaps more obvious here. Web hooks. That's firmly an API server-side concern unit testing, make sure your validations and mutations have tests, I think they're, possibly a bit simpler for us to think about. We can you know it's more traditional thing here, but make sure we're doing things right.

A

um Another one I really like is writing a corpus of actual valid resources, and possibly some invalid ones too, as we evolve things and make changes, you know projects expand. New people come on board. It can be easy to accidentally make these mistakes and just having that that's kind of like a safety net. There can really just give you a bit more confidence that we're not accidentally gonna. You know change something and cause existing resources that users have already created to suddenly become invalid integration testing.

A

So I I think integration testing here, for um we can make sure that basically things work, I, think a lot of a lot of the tests that we write for the API server side can be encompassed into integration tests, um basically making sure that we can create a resource in one version and then read it in another, really simple stuff but like that, will catch the majority I. Think of the issues that you tend to find defaulting, make sure things are set, validation, make sure it says.

A

No, if you said something wrong, really really kind of trivial stuff, but it it's important, because we don't do this that this is you know this is the whole point of the API is to do these things.

A

um An e2e testing, I tend to think e2b tests for the API side is more like. If I create this, you know webhook deployment if I go and create this crd. Does it work? You know, have I specified the right ports on my thing, so that's far higher level, more operational sort of stuff but yeah and then yeah things like our back too. We've got quite a few things to consider, then, on the controller side,.

B

Yeah on the controller side like um we are normally like, we do like go test if you use like, for instance, client, go and controller runtime, you might be used like to interact with a fake client. We normally like do testing starting from the reconcile function, which is a function that is unlink all the object. Events.

B

um One of the caveats that we found like with fake lion is, for instance, that any I we believe it's something that is very important like to do is that it's very hard like to simulate error condition of the API. What if the API is returning you for tonight go away? How does your controller behave? What, if is returning you an error like on an update operation? This is something that is actually pretty hard like to model with a fake client.

B

We often end up mocking the kubernetes client in order to handle all this potential error condition, but it's really an important aspect to make sure about aim force and that's the correctness of the behavior of the controller, which might lead also like to scenarios in which, for instance, controllers are stuck. They cannot progress. The reconciliation not eventually like they back off too often and and therefore, like you, don't see like the operation getting to an end or worse, they don't back.

A

Off at all, yeah, exactly yeah, which is a real problem. Yeah.

B

And actually like something that is interesting about controller runtime is really testing. The result like the result is what we do expect right. Do we want to bake off in this case? Do we want to return an error so that, like controller runtime, is handling for us? The back of? Do we want to have a fixed retry window? What do we want to do?

B

This is actually something very interesting, I think to model as we in order to build robust controller on top of the API and and while library that helps with that I think is really like amp test, which is essentially a kind of evolution of uh the fake client in in the kubernetes, go libraries which really allows you to spin up like a real API server and an etcd and as part of the execution of your your test, and it's really powerful because allow you like to test all these aspects that exists in the API that otherwise would be in the API server.

B

That otherwise would be very hard like to test. And if you look at some of the things that are happening like in the community, you can sell or web books or things like that. It's really allowing you to glue all of these things together and test the behavior of the controller with all this element not only like with the API resources themselves, as they get created.

B

Without all these add-ons, um really like it's real Highway- and this is something pretty incredible and and I definitely I think this is my- must do for all the controller like being able to test against API without using like fake clients like testing the real behavior of the API, as well as because the API evolves. There are new version of the API server that come along and it's important to test the upgrade path, and this is the right way to do that.

B

Not only updating the version of the libraries but as well testing the real API server Behavior, um the last. The last thing, of course again about controller easy testing, as James said, like E3 testing is mostly like a way to validate the behavior like of the controller in a in a cluster, like with all of the other controllers that they might be interacting with coordinating with it's mostly like.

B

If, if I put like my SRE on, it's mostly like for us away to make sure that everything still works, as we do expect that we don't break like the expectation of our users uh when they create the resources and the controller get to handle them more than a real way to kind of like test in the development uh life cycle, like the controller Behavior itself, it's mostly like something more targeted to the later feedback, which is the operational feedback and and also as James pointed out.

B

This is where we test our expectation about airbag permissions and interaction with other resources.

B

um One thing that is important, especially like for our project we often like kind of like provide instant examples to the users on how they should be using like or they are expected to use the resources. It's important to keep testing them like making the documentation always like actual- and this is something where again italies- can have a lot so few closing thoughts. uh We are about a time.

B

um Yeah definitely rely on the schema as much as possible and I like to say this again rely on the scheme as much as possible, because it's the simplest things you can do and without all the complexity and operational burden of additional extensions like web books um and regarding web books really be mindful of the impact operationally and the cost of running them. In the platform they might easily affect the customer experience they might be hard to maintain, are to evolve and, and so really like again rely on the scheme as much as possible.

B

um There is really like a lot of. There has been a very rapid revolution in the last years in the community, about testing methodologies and most of the code that out there use fake client and test is now a thing uh really like revise your testing and make sure to use them test in order to cover more and learn of the aspect of the of the controller interaction with the API and and as well like again. This is another m test is not covering make sure to test.

B

Also the error condition and the behavior of your controllers. In case of Errors back off 500, whatever error that the API is returning and test again is not giving you any real helpers today to test like all of these error conditions. So this is where you need to do extra work, maybe mocking the client, but do it because it's important to ensure the resiliency of the controller and your code in production.

B

Thank you, foreign.