Kubernetes Kubernetes Contributor Summit - NA (San Diego) 2019, 26 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Contributor Summit NA 2019: Don't Build a New API, Build a CRD

Description

Implementing a new API for Kubernetes? Chances are, you should start building it as a CRD instead of using a built-in API type. Come learn why to use a CRD in Kubernetes, see how the newest CRD features are making it easier to match feature parity with built-in APIs, and situations when you might still need a built-in API type (for now).

Event link: https://events19.linuxfoundation.org/events/kubernetes-contributor-summit-north-america-2019/
Session link: https://sched.co/VvN7

A

I think I know a number of you,, but for those of you who don't know me,.

A

I'm solly ross., I'm a, um,, one of the lead maintainers of kubebuilder and a software engineer on gke.

A

And my mission is to make, uh,, writing, kubernetes extensions and hopefully also, uh,.

A

Builtin kubernetes types, uh,, easier,, less arcane., alright,.

A

So, who here has added an api version or an api to kubernete's or made a change.

A

To an existing api version? all right,.

A

So for those of you who haven't,, I put together a quick list based on my own personal experiences and the current.

A

Documentation and we'll just maybe, let's take a look at it. Quickly.

A

Uh, so as you can see, there's a lot of steps, here.

A

Uh,, I'm not going to go over all of them, uh,, but there's a bunch of different things to do. um, and most of it involves changing lots of manual. Go changes in addition to your.

A

Api, changing lots of build scripts, et cetera. um,.

A

So what are some of the problems? Here? well,.

A

There's a really long, uh, api review, latency or change making latency, because you have to make your.

A

Changes,, you have to run all your build scripts,.

A

You have to wait for stuff to compile..

A

It's really hard to get that quick, feedback.

A

It's also hard to test out your api changes on different versions of kubernetes.

A

To send it to people. um,, you can't decouple it from kubernetes versions, cause it's built into the source.

A

Code. um, and then there's also this problem of not having any decorative information.

A

Attached to your api., so when we write validation and defaulting for builtin types,,.

A

We do that as go code., and this is incredibly flexible.

A

It lets us basically do arbitrary logic in validation and defaulting.

B

Sometimes this lets us do really bad things that we re later regret in.

A

Validation and defaulting. um,, but it's very flexible.

A

Unfortunately, this doesn't translate to our open api specification and this.

A

Might seem fine to us as developers of kubernete's,.

A

But it's not great for our users, because users,.

A

It turns out like using fancy tooling that reads the open api spec.

A

They like doing stuff like having client side validation.

A

They like having ids that provide type hinting and they like having.

A

Auto-Generated user interfaces that have nice layouts for their types.

A

Um, and without, with this all this manually written, go code,.

A

All they see in our open api spec is like optional and the raw type like integer.

A

String.- and this is like okay for for ids and forms,.

A

But it's not the greatest and we can do a lot. Better.

A

We can do a lot, a lot better.. So what's the solution?.

A

So I think the solution is crds. um,.

A

Building a api with the crd is a lot fewer steps.

A

So you have some go types,.

A

You add some special comments. That indicates stuff, like validation and faulting.

A

You generate some clients if you need generated clients.

A

This is common for belt, end api types and then you test out your crd,.

A

You generate your yammel from the.

A

So what does this bring us?.

A

It brings us a lot faster, iteration time..

A

Basically, the only compilation here is compiling our go types into crd, ammo.

A

And this is really pretty fast. we're only dealing with like one set of types,.

A

So it's, it's in the order of magnitude, of,.

A

I can run this on editor save, right, like same as go format, maybe a little bit longer,.

A

So this also lets us install quickly,.

A

Test out changes because we don't have to recompile the kubernetes api server.

A

It lets us, if we want to decouple changes from kubernetes releases,.

A

We can release with kubernetes,, but we can also say, oh hey,,.

A

I want to try this on a newer version of kubernetes or an older version of kubernetes.. I want to release updates to this type in between and such as disgust.

A

It gives us a decorative information,, all of our crd tooling,, almost all of it,.

A

All the cool stuff we can do is built on top of builtin primitives and open api.

A

Or kubernetes specific vendor extensions,, but it means all that information is there.

A

If the user is tooling can read open api validation,.

A

It can see suddenly that this field, that's an integer, has a minimum of zero or.

A

This field, that's a string, that's not just a string., that's actually an enum that can have say failure, policy,,.

A

Ignore or fail, right? like now that information is there and instead of just showing a string,.

A

The, the,, a ui, could hypothetically show a down,.

A

Which is a lot better experience for our users. and then finally,.

A

From a kind of project selfish perspective, as someone who helps out with some of.

A

This tooling, um, and on behalf of the people who write the easy,.

A

Easy extension mechanisms as a project,, we should be dogfooding our own tooling.

A

if we can't build kubernetes on top of kubernetes,.

A

How do we expect our users to build good extension, experiences, right?.

A

Like we should be able to use what we build to build new features, in.

A

And that will ensure that our users can also at least have a minimum base.

A

We can build kubernetes with it., so you should be able to build something: cold, too., all right,.

A

So some of you may be thinking by now,, those of you more familiar with crds might be thinking,.

A

I do not want to write open api by hand.

A

This is a miserable experience.. I have done it., it's not great..

A

You get to write, rod,, jason or yammel, and it's highly nested and really not great..

A

So the good thing here, is, uh, we have tooling inside the kubernetes project.

A

That means that you don't have to write open api by hand.

A

So project called controller tools.. It's part of the queue builder sub project of sig, api, machinery., uh,.

A

And it lets us, right, basically go idl for lack of a better term,.

A

Similar to what you do for the builtin types,, but we get to add these additional comments. Like, you know,.

A

If you've ever been in the belting go types and you've seen slash, slash plus.

A

Optional, right?, we get to add stuff like that. That lets us add more flexible stuff that gets.

A

Translated into the open, api.

A

So let's talk about some of the things that crds can do and we're where we're.

A

Good with crvs and they, we've reached parody with builtin api apis and maybe where you might fall back.

A

Into it, fall back to a, built an api.

A

So, let's talk about validation and defaulting.

A

So we actually have pretty robust validation and defaulting from open api specs.

A

Um, open api is validation, is pretty flexible. uh,.

A

It lets us write everything from basic type information to things like minimum,.

A

Maximum, uh, even regular expression, and this special value called format,.

A

Which lets us express kind of builtin primitives for validation.

A

Things that the api server knows how to validate.

A

But we have a hard time expressing declaratively, so maybe, like date, time,.

A

Right?, like it's infamously,, it's pretty complicated to validate like a string as a property, time. um,.

A

So we, we say this is a date time and then the api server knows how to validate it. uh,.

A

If you need really complex, say, cross field, validation,.

A

We can also fall back to validating web hooks.

A

These, let us execute arbitrary, go code to do validation.

A

But they do sue out of process, right?, so we run a pod on the cluster or we own our process. Alongside the api server.

A

And we register something that says whenever a request comes in for this,.

A

Go out and call this code. it's,, and so that mirrors what we have.

A

For builtin api types., that's basically the equivalent of builtin api types, validation.

A

So if we need something beyond what open api can provide,.

A

We can fall back to that. um,, so here's my first call to action of the presentation., though there are some primitives that are, should be possible to validate from open api hypothetically,.

A

But are really hard., it turns out properly validating ip addresses for ipv, four versus ipv,.

A

Six dns names, et cetera.

A

Maybe possible using a regular expression,, really annoying kind of gets gnarly.

A

You don't want to write it out., but the thing about these is: we find them all over kubernete's and kind of all.

A

Over distributed, systems, right?, like what,, what part of kubernete's doesn't talk about? Dns names,.

A

It's all over, uh,, all of our names,, our dns, et cetera.,.

A

So we should be providing primitives as,.

A

As the people who build kubernetes to make this easier.

A

This is something we've run into when adding new kubernetes api apis and.

A

Considering whether or not to make them crds., so we should start thinking of, can we extend the format field?.

A

Can we add additional acceptable formats for fields that are useful to.

A

A

all right. uh,, so immutability and defaulting, getting onto the second half of that, right.

A

Immutability is coming soon.

A

right now, that's one spot where you might need to fall back to a built in api,.

A

But, uh,, it should be coming around one 18 ish.

A

Ish followed. there's a cap,. You can follow it., there's a link in the slides,.

A

Which I will show you, the url for afterwards. um,.

A

Defaulting our defaulting, uh, is pretty powerful in,.

A

Uh, open api., we can specify arbitrary type hierarchies, including like slices and objects.

A

So you can see here, I've got an example, um,.

A

That just ads should be able to see,. Have I skipped ahead?, it's slide.

A

No, I'm good. okay, cool. uh,.

A

We've got an example that has a object that has a subfield that's defaulted and.

A

We've just specified it there. uh,.

A

We can also specify a complex,.

A

Um, behavior around merging.. So if we need to say like,.

A

This list is actually a map,, which is a common thing. We do in kubernetes,.

A

We can do that too.

A

So conversion, uh,.

A

We can do conversion in crds now that used to be a thing that required to build.

A

An api,, but now we have conversion with web hooks. um,.

A

They basically look very similar to what I talked about with validation.

A

Webhooks. you say call out to this go code..

A

It will do some conversion between different versions. um,.

A

And then we can mark a certain version, as the storage version just like in.

A

Builtin, api apis. and that's the version that gets persisted to at cd.

A

Uh,, this is just an example.. You can see of what the configuration looks. Like.

A

It's not too hard to write and it's not too different from what you've expect.

A

From builtin api. all right.

A

Cross field- defaulting. just don't do it., it's not worth it.

A

Don't, don't,, don't try. alright. uh,.

A

So let's talk about some of the things where like,, maybe we might start getting into shaky ground in terms of like crds versus.

A

Built in api apis., so the first is, um, embedding a built in api type into a crd.

A

Now this is possible with some caveats, right?.

A

As previously mentioned at the beginning of the presentation.

A

Remember I was like,, we don't have decorative validation, information on our builtin types.

A

Well, that comes back to bite us when we go to embed stuff into crds.

A

So a really, really common case for both our users and for us is embedding pods back into uh,.

A

Types. so the thing about this, as like the stereotypical like deployment replica set.

A

Replication controller case where we want to say this thing creates pods.

A

So you don't get,, you get the basic validation, but you don't get any of the more complex stuff.

A

Like, you know,, this field is an enm, and sometimes this is acceptable,.

A

But a lot of times you want any more than your user ahead of time. um,.

A

And so for this, you either need to fall back to a builtin api.

A

You need to use a validating web hook that calls out to like does a dry run.

A

Against the api server or some such. um,.

A

Now this is another case where we could make this experience a lot better and.

A

There's a few different ways: we could do it., we could add references and our open api schema.. So you could just say, hey,.

A

Api server,, this entire tree is just a pod spec.

A

Use your belt and definition of pod spec., we could add declarative validation to all of our belt and tights,.

A

Which is really nice for users for the aforementioned reasons, and also solves.

A

This issue. there's a few different ways..

A

This is something else we need to work. On.

A

A

Yeah, so sub resources,, sorry about this., my uh,.

A

I can't actually see the slides on my computer. all right,.

A

So sub resources, what we have scale and status support. um,.

A

That's a lot of what you need.. You can specify a spec path for instance,.

A

For scale, spec in status- and you got slash scale.

A

For arbitrary sub resources. uh, well, first of all,.

A

Make sure you go get through api review and they say it's okay to add a new sub.

A

Resource cause like chances are, you might not need it., but if you do,.

A

You're going to have to go back to a builtin api..

A

Now we've seen this request on occasion,, but it's clear how often it comes up..

A

We may need to eventually add this to crds,, but it might turn out that we don't need a sub resources,.

A

New sub resources, as much as we think we do.

A

there's not a ton of them in kubernetes.. Most of them are kind of weird ones,.

A

Like all of the ones like logs for instance, and all that., most of the rest are stuff like skeleton status.

A

So here's another one where, uh,.

A

We actually need to fall back to a belt and api. high read, read,.

A

Write performance., so this came up recently in the review for a type called endpoint slice,.

A

Which is a cap about reworking the way and points working. kubernete's.

A

Uh,, if you need to read and write a lot of objects, real quickly,.

A

You're, probably gonna need to built an api because we don't have prototype buffs.

A

Support. um,, and it turns out for those of you who are not familiar,.

A

Encoding and decoding jason and go is really slow, like really,.

A

Really slow compared to other stuff., um, and this,, so this is fine for a lot of types,, especially stuff, like deployment where you've got a bunch of kind of biggish.

A

Objects that don't get updated a ton.

A

But for stuff like end points where you're writing all the time and writing.

A

Health checks and stuff, like that, you're, going to encounter some issues.

A

We could add protos support to crds.. This is kind of a,.

A

We'd like to do it eventually and less of it's coming in one 18,.

A

But we're going to need to eventually get there. alright.

A

Field, selectors hint,, you probably don't actually need fields selectors. Unless you are pod and it is.

A

The future and everything is a crd, which would be a glorious future indeed. uh,.

A

And everyone drives hover cars.

A

So the last kind of set of things are kind of the bootstrap problems.

A

A

If you need your types, need to be present for the api server to boot,.

A

You might need to fall back to a builtin, api. uh,.

A

We currently are a little bit sketchy on how to deal with the bootstrapped.

A

Problem for crds. um,, it might be possible to work around it.

A

If you have any brilliant ideas for how to solve it. uh, come talk to us.

A

We'd love to get people involved in that. uh,.

A

Kind of similarly, uh,.

A

If there needed to be installed in cluster boot in order for, uh,.

A

Say the node, just start: pods, uh,.

A

Which is another problem that has come up in the past,, you may need to fall back to a belt tin, api.

A

This is a problem kind of that has been encountered by a couple of storage and.

A

Sig node related projects, uh,, where they eventually moved to belton api. for this reason.

A

We'd like to solve this of one eventually as well, and have kind of a canonical.

A

Path for this to support this kind of thing., but it's not quite there yet.

A

Right. so I've talked a lot about, oh, sorry.

A

A

Uh, so some of it is solved.

A

The main cases that I know have have been solved, by,.

A

Um,, the alpha versions were crds.

A

And then, when we moved to beta,, we moved to a builtin api cause. We weren't quite sure how our users were going.

A

To deal with it. um,, so just about your,.

A

Your comment was like really good timing cause, I'm just about to talk about it.

A

Uh, yeah,, so the two major cases of this that I know of, um,,.

A

Were related to storage and pod. uh, there's,.

A

There's last two uh, there. um,.

A

And there were mostly around, as far as I know, like users expect these things to.

A

Be present on the cluster when they start, uh,, like runtime class,, they like,.

A

Oh, I need to start pods with my runtime class.. It needs to be there.

A

Like maybe, I'm expecting to run a pod in order to run a webhook to support.

A

Runtime class. cause uh,. I think that was before that some of the declarative, validation and defaulting.

A

Was as robust as it is, now. um,.

A

And so like, there was kind of a chicken and egg problem there and they solved.

A

It, solved it by when they moved to beta,, they moved to belton api.

A

So they still got some of the nice benefits that I talked about around rapid.

A

Iteration, um, from during their alpha phase,, which is cool. and that's,.

A

That's a path you can take,, even if you anticipate you're, going to have this bootstrap problem. um,.

A

But it would be awesome to solve this eventually.

A

I think it's kind of been sitting in, uh,.

A

Some of the sig cluster lifecycle, discussions. um,, there's not,.

A

There hasn't been a pressing use case since these two moves that I know of. um,.

A

But they've,, I think been thinking of it on and off and we've had some proposals in the past.

A

I've read a little bit about it. um,, but if you're interested in taking up and championing this work, uh,.

A

Definitely talk to sig cluster lifecycle, and you can also talk to me as well.

A

Uh,, but there are cases that have used this successfully, uh,, that aren't,.

A

Didn't have bootstrap problems. so, um, volume,, the volume snapshot controller is currently using this, um,.

A

That's in the kubernetes csi org. um, and if you, you know,, if you go on the,.

A

The uh, volume snapshot page on the community's docs,.

A

It's like this is a crd type.. You need to install it and then you can use it.

A

Um, cluster api is using this, um,. All of cluster api is built on top of this.

A

And then there's, there's a few other projects,, but these are the kind of prominent ones. and provided I thought, two good examples of ones that are using it. Now.

A

And two good examples of ones that had the bootstrap problem,.

A

Right?, so if you happen to fall asleep during any of this and just woke up in summary,.

A

Crds provide fast iteration time,, a nice decorative features for users,.

A

They're also important., it's also important to dog food,, our own stuff. um,.

A

But we still have some problems,, especially around some of the advanced corner cases., but the key takeaway,.

A

I think this is when I was looking at this,.

A

All of these problems are solvable.. We can,.

A

It's not like, none of these are like fundamentally like we need to rearchitect.

A

Kubernete's entirely to solve this problem, right? like,, it's all,.

A

It's all things that we can solve and we should,.

A

Because if we're hitting these, our users, they're going to,.

A

All right, any questions?.

B

Sub resources?.

A

Oh yeah. feel free.. You should take it offline. uh, well,. I mean,. If you have,,.

A

If you have a question,, I like,, we can take it offline if it goes too long,.

A

But feel free to ask right, now.

B

Um, yeah, so,, so two well known, right, sub resources other than s scale, uh,,.

B

Exac and in logs, right? and so for,.

B

For example, in the cla, in the context of cluster api, having something like logs for a machine object could be useful. not,.

B

I'm not, I'm not saying that necessarily so,, but I can imagine, um, you know,.

B

Use case there,, uh, and maybe other sort of sub answers that maybe we haven't even thought of and.

B

One one way: or, or,.

B

Or maybe even two to implement logs and executives in a different way. right.

B

And one way, maybe to support that is to have, um,.

B

You know, redirect the, the request to a different, um,.

B

Do different service, uh,, where you know, the api servers.

B

Then just responsible for,, you know,, receiving the requests and matching the sub resource with some kind of,.

B

Mmm. yeah. some, some logic that generates a, another url., right., and, and.

B

What, um, yeah. are there other,? Are there discussions around this? um, you know,.

B

Do other people see a need for this?, you know, what, what,? What do you think as,.

B

As,, as far as like the possibilities?.

A

I, um, I've seen some discussion around it. Before.

A

I think this is mainly from what I've seen in the case of like a cost cost.

A

Benefit analysis and doing like the, the,.

A

The common low hanging fruit, first. uh, there are,.

A

I have seen people asking for s for stuff, custom logs before logs., like you, said, logs are kind of one of the weird sub resources.. So like quick, summary,.

A

Sub resources and kubernetes usually fall into two categories, projections,.

A

Which take data in a base resource and transform them.

A

So this is things like scale and status., uh, and um,.

A

I think, like there's a binding sub resource at one point that did weird things.

A

Um, and then there's a stream sub resources, uh,.

A

Which do one or two directional streaming.

A

So this is things like logs exec attach a where you're actually like,.

A

Not acting like a normal crowd resource in kubernetes., uh,.

A

And so the, the things like that aren't projections are kind of like funky to handle.

A

And there's only been a few calls for them that I know of.

A

Um,- and so I think it's just mainly been like,- there's like more pressing things to work, with. um,, that being said,.

A

Talk to sig api machinery., if you have ideas, you know, put together,.

A

Talk to them, first, put together a design. um, yeah.

B

Okay. so he kept wanted, uh, yeah. talk to talk to them. First though.

A

But like most of these, things are like,.

A

We can solve them., we just have to,, you know,.

A

We need to put in the effort and we need to make sure we're designing a solution.

A

That isn't going to cause us trouble in the future.

B

All right. thank you., solly. yup,, no problem.