Kubernetes Reliability & Testing Resources, 27 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Contributor Summit NA 2019: Live API review

Description

This will be a live API review, going through a real PR and showing how it's done. It will cover API norms, less-well-known conventions, rationales, validation, defaulting, and other important API concepts.

This is an opportunity to learn how to make your API review PRs go through faster and easier, with fewer revisions. It's also a great way to see how to do API reviews, in order to start down the path of becoming an API reviewer yourself. Every SIG needs to have active API reviewers to make development smoother and faster, so why not you?

HOW TO PREPARE BEFORE THE WORKSHOP

In order to make the best use of our limited time, please prepare ahead of time.

Reading:

Being familiar with the API conventions and API changes documents would help you get the most out of this workshop.

Laptop and build environment:

A working kubernetes build/test environment is only required if you want to try out API tests and code generation on your own during the workshop. In that case, you should have a laptop capable of building and running basic Kubernetes binaries, with the following software installed:

Go 1.13 (This is a change from the originally advertised Go 1.12)
Docker
git
make
kubernetes/kubernetes GitHub repo

Event link: https://events19.linuxfoundation.org/events/kubernetes-contributor-summit-north-america-2019/
Session link: https://sched.co/Vv6Y

A

All right., are you ready to get started? yes., yes., awesome., um,.

A

My name is jordan liggett.. I have been working on the project for a while and have seen a lot of good.

A

Things and bad things done in apis and I've been responsible for a fair number.

A

Of both of them. uh,, so don't take the things that I say today as like. Coming down from on high and.

A

Like I know, what's right and you're all doing it wrong.

A

like a lot of the things I'm calling out as this doesn't work, well,.

A

The reason I know that is because I did the same thing. uh,.

A

But I thought we would start by sort of looking at, uh,.

A

The path that the api follows inside the api, server, uh,.

A

To kind of look at the different phases so that, as you're thinking through.

A

Designing an api or changing an api or reviewing an api,.

A

You have these pieces in mind., so you can ask yourself like,.

A

Is this doing the right thing in each one of these phases?.

A

Um,, and so this is something that, uh,, stefan put together when we were going through some of the custom resource work,.

A

But it, it lays out pretty clearly the different phases that we, uh,.

A

That we have in the server. and so, uh,.

A

You can kind of ignore all this stuff and just pay attention to what happens inside the server here.. So I'm going to request comes in,.

A

The first thing we do is decode the request., so we're taking an array of bytes off the wire and bring it into an object.

A

And so for apis that are built into the api server, where we have go types.

A

Defined, we're decoding into those go types.

A

And so any schema errors that you run into are going to get caught at that.

A

Point., if you tried to put a number into a string or a object into a,, an array,.

A

It's going to get caught at that point. uh,, the next step, is, uh,.

A

What we call defaulting. so when you submit [inaudible] of an object,.

A

If you don't specify some fields, there's an opportunity for the server to set defaults for those fields.

A

And so you see this in places like replica counts,, uh,.

A

In some of the early workload objects,, if you submit a deployment or a replica set, and you don't tell it how many pods.

A

You want, it'll say, hmm,. You probably want one. uh,.

A

So there things like that,, there's a chance to apply to faults.

A

Then, for types that are built into the api, server,.

A

We go through what's called a conversion phase, because the server can serve.

A

Multiple versions of the same resource., it can serve like a v one or if you one beta one or alpha one, whatever.,.

A

It's better for it to only deal with one version in memory.

A

And so whatever version you submit on the wire or whatever version, it reads from.

A

Etcd,, it always does a conversion into the,. We call it the internal version.

A

This is only in memory and that lets all of the later logic deal with only one.

A

Version of an object. and then when we get ready, serialize it either back out to etcdto, store it or.

A

Back out and the response to send it back to the user., we do a reverse conversion back into a serialized version.

A

And so, if you've ever gotten confused, like you, wanted to add a field and you're like, wait, why,, why are there like six copies of the deployment type?, which one do I change?.

A

The answer is actually all of them. and some of them live in the api package,.

A

The k8s.io's api repo,, those are the external ones, with serialized double jason tags and things.

A

And then there's always one internal copy.

A

That's why that distinction exists. all right,.

A

So we go through conversion. uh,.

A

Now we have the opportunity to do validation. and, like I said,.

A

This always operates on a unified, uh, type.

A

So, no matter if you started with like if you wanted deployment or if you one.

A

Beta one deployment,, whatever you started, with, uh, you're, always validating the internal type.

A

And then once it passes, validation and we say, yup,.

A

This is a request, we're willing to accept,.

A

Then the server handles conversion back into the storage version and persistent.

A

And so, when you're reviewing an api change, in addition to just kind of the user.

A

Experience of does this api make sense? um,.

A

The three areas you want to pay careful attention to are conversion,.

A

Defaulting and validation. uh,.

A

It's not enough to just look at the code, that's actually in the pr, because sometimes we miss one of these.

A

and so, if you're, only looking at the code, that's in the pr, there might not even be validation, added.

A

and if you don't have these categories in your mind as ones to look for, uh,.

A

We will release an api that isn't validated and that's bad news.

A

Um,, once we release an api change, once we say cut v, one 17 or whatever and start accepting data into fields,.

A

It is very difficult to strengthen or tighten validation in the future.

A

Uh, for a couple of reasons. one,, it will break potentially api clients that are currently submitting api requests.

A

And being very happy and successful., uh, and secondly,.

A

Old clients may have already submitted data, and it's already living in etcd.

A

And if you tighten validation,, you have just invalidated. Potentially data, that's already living in etcd and.

A

That causes all kinds of problems. If you try to update those existing objects, later. so decode,.

A

Conversion, defaulting validation.

A

So I thought what we would do is walk through a couple of api changes.

A

One is a pull request, there's actually like code written.

A

And then the other is a proposal. uh, and just talk through some of the considerations,.

A

The questions that we ask during these reviews.

A

And the goal of this is to actually give some feedback to the people who are.

A

Doing these apis, but also, um,.

A

Give you questions that you can ask yourself when you're, making a pr or.

A

Reviewing a pr. and the more of these that we can sort of disseminate into a knowledge that the.

A

Whole community has, the higher quality prs we're going to be able to produce,.

A

Right? if,, if we don't just have five or six people who know these things and know to ask.

A

These things,, but we have 50, uh, that's,.

A

That's really gonna help our community. so with that,.

A

I'm going to dive in and I will say, if there's a question,, please feel free to like raise your hand,. Shout out.

A

I'll, try to repeat the question for the recording..

A

The goal of this is to clarify things..

A

If there are things in our existing conventions, documentation that are not.

A

Clear, we want to improve that., so yeah, raise your hand.

A

Uh, this,, this should be interactive.. I don't expect me just talking about these two to take the whole 90 minutes.

A

So we should have time for that. all right., so the,.

A

The first pull request that I wanted to look: at, um,.

A

Is adding a time to live field into pod object.

A

And so it's helpful to orient ourselves to kind of understand the context of.

A

This., like what,, what is this? uh,? What is this doing?? Why do we need it?.

A

Is it accomplishing its goal? uh, and so there's, uh,.

A

An existing concept of time to live on jobs.

A

And the, the motivation behind this was, if you create a job, and it creates some pods and.

A

Those pods finish doing what they were. They were supposed to do.

A

And then they just sit there living forever., you know, they've,.

A

They've completed, they're just sitting there as an api objects.

A

If you run a bunch of jobs over and over and over,, which you can do from a cron job, uh,.

A

You will actually fill up at cd, eventually, right? uh,. The space isn't free.

A

And so having hundreds or thousands or hundreds of thousands or millions of pods.

A

Sitting around is not actually helpful. uh,, so you might say, well,.

A

Why don't we just delete it as soon as the pod's done? and the answer is,.

A

It's actually helpful to have around for a little while, right? we,.

A

It's helpful to see the status. like,. Did it complete?? Was it successful?.

A

If it failed,, I might want to go look at the logs., so it's helpful to have it around for a little while.. How long is long enough?.

A

Uh, and the answer: is it depends, right??

A

Maybe this thing you want to live for a day or a week,, or maybe you've got some requirement that it live for a certain period of time.

A

For auditability. um, on the other hand,, you might not care. you're like, yeah,.

A

Deleted after a minute., and so the first question that we ask ourselves when we're doing an api review,.

A

Is does this api need to exist at all?, um,.

A

And it's kind of a counterintuitive question,.

A

But sometimes the best api is new api. uh,.

A

And so really thinking carefully about do we need this knob?.

A

Communities has a lot of knobs, um,.

A

Lot of levers and a lot of them have to be set just right.

A

And so a good first question is,. Can we accomplish what we're trying to accomplish, uh,.

A

Without pushing that sort of complexity into the api.

A

In this case, uh,, this was a pretty low level thing., and so there were legitimate use cases for short, ttls and legitimate use cases.

A

For long ttls. and so that's,.

A

And the surface area that we needed to express that is very small..

A

It's just a single field that says: how long should this live after it's complete?, and so this is a good scenario where an api addition makes sense.

A

It doesn't make the api a lot more complex and it satisfies a wide variety of.

A

Use cases., so that's the background for this feature in general.

A

This was already added to jobs.

A

So on a job you can specify how long it should live after it. Completes.

A

And so the proposal also talked about doing this in pods as future work.

A

And so that's what this pr is doing., it's completing that design in the pod object.

A

Yes, no,, it's fine.

A

Right., so the question was: when do we decide when to use annotations versus using like.

A

Putting actual fields on the object? uh,.

A

Generally for functionality that we consider to be inherent or essential to the.

A

Object. we like it to be a field fields, are more strongly validated.

A

They show up in the open api, schema, they're, introspective by tools, uh,.

A

And annotations are great for sort of extensibility use cases or niche use cases.

A

Where, um,, not everyone or not, the majority of people will have a particular use. Case. uh,.

A

But like for, for jobs,, most people don't want their ftd instance filling up with garbage and crashing.

A

Their cluster. uh, and so that's a reasonable thing to actually build into the job controller and a.

A

Reasonable thing to represent in the object.

A

All right,, so I'm just going to jump through, jump into the pull request and kind of talk about what I look for. First kind of.

A

High level overview and what can help reviewers help you get your polls reviewed.

A

Faster. uh,, I don't like click baity titles, but like five tips to get your pull request.

A

Reviewed faster. I, I'd click on that., um,.

A

And so right away, you want to structure your pull requests to make it easy for.

A

The reviewer to see the core change, that's being made.

A

And so, if we look kind of,, there are several commits which can be good and it looks like some of the later.

A

Ones were generated, changes and that's good. that's,, that's helpful..

A

If you take,, we have a lot of generated code.

A

if you put those in their own commit,, that's helpful., generally., um,,.

A

The first commit looks like it has all of the implementation and api changes.

A

And so on, the one hand, this is good,. We can actually see the implementation.

A

I can zoom that in a little bit. how's that?.

A

Is that more visible? yes. okay. um,.

A

We can actually see that we have a working implementation accompanying this.

A

Change., that's always good. uh, on the other hand,.

A

It does make it a little bit tricky to kind of tease apart like what,.

A

What are the api changes and what are the implementation changes., and so it's often helpful to have one commit with api changes,.

A

One commit with implementation changes and then one commit that does all the.

A

Generated uh, files. um, but,, but this is fine.

A

We can kind of jump to the, the api packages and, and see what's going on.

A

Uh,, something we do see sometimes is things that are wanting to make an api change.

A

And the implementation isn't done, yet., uh, and more and more, we are asking that we not merge api changes until we have a.

A

Working implementation. um, for a few reasons.

A

There are a variety of things that can come out of that..

A

The worst is that the apa api change goes in,. We release,.

A

Then we go write the implementation and realize the api doesn't actually give us.

A

What we need to do the implementation. uh, and now we've releaetcdan api, and we have to support it, uh,.

A

Or at least not break it. um, that we don't want to do. That. uh,.

A

That's kind of one extreme on the other is, uh,.

A

Just confidence that it's gonna work well in terms of scalability or usability,.

A

Sometimes you can have an api that works, but it's awkward and if you had been.

A

Working on the implementation,, you would have like structured it slightly differently to make it easier to.

A

Understand or use. um,, so it's,.

A

It's fine to put things in one pull request as long as they're kind of in.

A

Different commits.: it's also fine to have one pull request for the api change and then start a.

A

Second, pull request that kind of bases, on top of the first one and adds.

A

Implementation. uh,: we just want confidence that we have the data we need and the data flow works.

A

And is usable and is all the, all the problems we're going to run into during implementation are understood.

A

So this has the implementation and the api change in one, uh,, one commit.

A

So that's okay. uh,.

A

So, let's start by looking at the, uh,.

A

Types that go file., so this is a,. This is a built in api.

A

If this was a custom resource api,, we would probably be looking at the possibly the goat types if it was using cube.

A

Builder to generate, uh,, but also the actual crd manifest.

A

Like what's the schema, that's defined. uh,, but in this case, it's a built in api. uh,. So we look at the types of dot go file.

A

A

Make that a little bit more readable.

A

There we go. okay. so again,.

A

Api surface that this is adding is small for really fundamental types like pods.

A

Or pod spec.- that's good.. We don't want to make big changes. um,.

A

If you're needing to make big changes to a really fundamental type,.

A

It's worth asking yourself if this is the right approach,.

A

If there's a less disruptive way to accomplish what you want.

A

And so this is a mirroring, the same field that was added to jobs.

A

So there are a few things going on here that are interesting to look at.

A

The first is that this is changing an existing type.

A

And so, whenever you're changing something that exists,.

A

You always have to ask yourself, what are existing clients gonna make of this thing?.

A

And what about all the existing data? That's already persisted?, you know,.

A

There are clusters out there that have hundreds of thousands of pods that.

A

Already exist., what are they going to do with this field?, and so the first thing that you have to keep in mind is that any addition to an.

A

Existing object has to be optional..

A

You can't require a field in an api that people are already using a new field.

A

And an api that people are already using., and so we can see that, uh,.

A

This is marked as optional, uh,.

A

In the open api spec., this actually makes it so that this field is not marked as required.

A

So that's good. um,. It's also a pointer.

A

So in the api conventions we talk about like optional and required fields. um,.

A

There are some sort of weird go type things that bleed through our api.

A

Serialization., and one of them is that if this wasn't a pointer,.

A

We wouldn't be able to tell the difference between the user passing literally a.

A

Zero or not having the field at all. and so if,.

A

If there's a, a field that you're going to want to default or you're going to want to be able.

A

To tell if the user specified it, uh,.

A

Then you probably want it to be a pointer for things like strings and numbers.

A

Otherwise, we, we can't tell the information, isn't in the go: type. yeah.

A

Right., so the question was: does it have to be optional?.

A

Absolutely. or just optional before defaulting., is it possible to have a new field that gets defaulted and then is treated as.

A

Required by validation? you can do that. uh,.

A

That's typically what we do when we have multiple versions of an api.

A

We'll introduce a new field, um,, it will be defaulted in the old version and then treat it as required by.

A

Validation., the important thing here is that these optional attributes feed into the open.

A

Api schema. and so from the schemas perspective,.

A

It has to be considered optional., otherwise things that do client-side validation can choke and refuse to even.

A

Send the object to the server, because there they'll say the scheme said it was.

A

Required- and you didn't give it to me., so I'm not going to even talk to the server and give the server a chance to apply defaults.. So yes,.

A

In validation, we can end defaulting,, we can sort of add a default and then require it in validation,.

A

But from the scheme as perspective is treated as optional.

A

All right, a couple of other things to talk through here. um,.

A

The pro buff aspects, uh,.

A

Will actually get generated for you if you just add the field,.

A

Uh, and define the jason information and then run the,.

A

The make update script that generates everything.

A

It will allocate a prodo tag number for you and generate this for you.

A

So, uh,, I see some people like hand, copying these and trying to understand like what do.

A

All these things, mean? uh,, you generally don't need to do. That. the,.

A

The generation scripts will add the pro decorations for you.

A

Um, we do have linters in cii that will make sure that this field name matches the.

A

Go field name., so if,. If you do fat finger,, these are type of these or not follow. Conventions like lower casing,.

A

The jason field- there,, there are lenders that will catch these., um,.

A

When you're reviewing,, it's sometimes a good idea to double check.

A

There aren't lenders for everything. we're working on that.

A

All right. some other things,, the conventions talk about,.

A

Uh,, how you represent different types of data., and so this is representing a duration, right?.

A

This is saying: how long do I want this thing to live? uh,.

A

And so you might think, oh, duration. cool co has a type for that.

A

We'll just use time dot, duration and it does strings here.

A

Lization and it's really cool. um,, we don't actually recommend using time duration for rest apis because it makes.

A

The go duration, parsing part of our api.

A

And that's actually sort of ugly to implement in other languages.

A

Like we try not to make other languages implement,.

A

Go compatible versions of things., and so rather than externalizing time, duration, parsing,.

A

We put units in the field name and so you'll see like such and such seconds and.

A

Then it's just a regular integer or something. milliseconds. microseconds.

A

I think we have something like that in one place., uh, and this,.

A

This is keeping in mind that not all of our clients are go. Clients., um,.

A

We don't want to make it overly burdensome to parse and produce,.

A

Uh, api requests in other languages.

A

All right, so reading through the documentation on this, uh,.

A

Limits the lifetime of a pod that is finished in it defines what that means.

A

Uh, [inaudible] it's set, then this long after the job. all right,.

A

So this is copy paste., so this is,.

A

Uh, first comment, uh,.

A

A

Of job instead of pod, [inaudible].

A

And one thing we try not to do is drown people in comments.

A

So if there's something that applies in multiple places, just note at once,.

A

Be kind, don't make the same comment like a thousand times.

A

Um, [inaudible] all right,, so the rest of this looks like a copy paste of jobs. Stuff.

A

So we'll assume that that'll get updated.

A

Uh, [inaudible]. all right,, so now we're talking about alpha level, and,.

A

And feature specific, uh, field., so this is,. This is an interesting point.

A

Again,, when we're making changes to an existing api,.

A

You have to think about how existing clients are. Gonna react to that.

A

And it may seem counterintuitive,, but one of those clients is actually the api server. um,.

A

And so, if you think like in terms of time when this field is introduced,,.

A

Uh, say the api server is at one 16, and this field doesn't exist. and then,.

A

Uh, api server upgrades to one 17 and now the fields does exist.

A

And so now you can persist this data in its ed.. So that's cool., um,.

A

And you might think, well,, I don't see a problem. There. well,.

A

What happens if you have more than one api server running at the same time,.

A

Against the same ftd, storage.

A

And so this is how you run highly available api servers.

A

You have more than one and you load balance between them and when you upgrade,.

A

You only upgrade one at a time, right?.

A

So imagine you have three api servers that don't know about this field and you.

A

Upgrade one of them., so one api server knows about this field and you make a request and you submit.

A

Data to that api server and it says, cool ttl seconds have to finish. yep,.

A

I know about that: field, I'll, validate it, I'll persist it..

A

Now that data is in etcd, well you're,.

A

You still have two older api servers that don't know about this, and requests are.

A

Getting load balanced between them. and so what? If another client talks to another server, says, hey,.

A

Let me get this pod, look at it. and then like set a label or set status or something.

A

When this data and etcd that knows about this new field gets decoded into a go.

A

Type by an old api server,, this field, disappears.

A

The old api servers go type, doesn't know how to represent this, and so it drops.

A

The data and if it writes back to etcd,, you now have data loss.

A

This is really nasty. uh, for a long time. This was just sort of a,.

A

Yeah, that that happens., maybe don't use new fields right, away,.

A

Kind of a kind of a thing. um,.

A

And we weren't super happy with that., where we ended up after a while was uh,.

A

The idea of feature gating fields., so when you first introduce a field to an existing type,.

A

It has to start out as alpha level.

A

And that means we document it,. It says this field is alpha. Level.

A

We won't accept data into this field unless this alpha level feature is turned.

A

On. that does a couple of things for us. one,.

A

It lets us get feedback from people.

A

people who want to use. This can enable the alpha feature and actually use it, and.

A

Try it out and say, hey, it melted. My server. like it doesn't scale.

A

What are you people, doing? uh,? So that's good..

A

We want that feedback before we enable it by default., but the other thing that it lets us do is roll out the change over multiple.

A

Releases., so in the first release, we market, alpha.

A

Now all of the servers at that level have this field and their go type.

A

They can round trip the data from etcd,.

A

But in validation we will not let new data into this field.

A

So the apr servers know about the field.. They can roundtrip the data,.

A

But they won't accept data in that field, in new objects.

A

So we're not going to lose data if there's,, if there's data already in etcd,.

A

We will read it and write it back and not drop it., but we're not going to let new data into the field. At least one release. Later.

A

We can promote that feature and start accepting data into the field.

A

And so that guarantees that by the time we start allowing data in,.

A

We know that at least one back level of the api server understands and can.

A

Preserve that data., and so this feeds into our supported, upgrade patterns.

A

This is one of the reasons why skip level upgrades aren't a good idea, because.

A

The development processes that say we roll out changes one release at a time.

A

Start to get violated. if you start running like api servers that are more than one release, apart, uh,.

A

The old ones can start dropping data that the new ones just enabled.

A

All right,, so this is documenting. That field is awful. Level., that's good.,.

A

It's documenting the feature: gate that, uh, is related. To.

A

It does. that's good., um,, so yeah, so other than the, uh,.

A

Other than the dock change about the job, stuff,.

A

The type actually looks pretty good., so let's look at the other types. dot.

A

Go file. remember: there are two in this case.

A

A

Alright,, so that's a good question., so this changed the pod spec type that lives under k8s.io api.

A

But unless it's in one of the other commits,, which is possible,.

A

Let me jump to the full change.

A

I'm not seeing a corresponding.

A

Change in the internal pod, spec.

A

So this also needs to be added.

A

And I kind of forgot to check and see what the ci tests we're doing.

A

We have round tripping tests that will go from internal, external,.

A

And then back., and so if that field didn't get added,.

A

The tests won't, let you merge in that state,, but it can be very confusing.

A

Um, and then, once you add a field internally and externally,, that's when all of the automatic conversion, uh,.

A

Generation stuff will kick in and say, oh,. I see this same field exists here and here,. Let me write the,.

A

The conversion function automatically for you.

A

All right?, so if you think through the three things we want it to look at,, we want it to look at a conversion..

A

I don't think we're going to have conversion if the field wasn't added.

A

Internally, uh, defaulting. uh,, so.

A

It doesn't, there's no default documented here. uh,.

A

But let's look and see if they added defaulting., it doesn't look like they did.

A

A

Yeah. okay. okay., so we're not, we're not defaulting this field. uh,.

A

And we document, uh, what will happen in that case,? So that's fine.

A

Uh, and so then the last question is validation.

A

And you'll notice, I'm just searching., we use defaults. dot., go and validation, dot.

A

Go and type stock go as conventions. um, if you're, adding exists,.

A

Adding to existing things, you're, probably gonna, add functions in those existing files..

A

It doesn't look like any validation was added., and so this is kind of what I was talking. About.. If you didn't know, those were three phases to be paying attention. To,.

A

You wouldn't necessarily look at this and say,. Oh yeah,,.

A

Like something is obviously wrong with validation., the thing that's wrong is that there is none. uh,, and so, let's think about like,.

A

What could you do if this wasn't validated? well,?

A

You could put a negative number in a,. You could put a zero number in that probably, yeah,. Maybe zero makes sense.

A

Maybe it doesn't. it's something. We should at least think about, right?.

A

And decide what we want to do. There. and so,.

A

Uh,, we'll just leave a kind of a breadcrumb there to kind of trigger the discussion.

A

About what we should validate,.

B

A

Should at least validate non negative numbers.

A

I know this is so exciting.

A

The field is set to zero.

A

A

This is good.. I should have like 75 people looking over my shoulder.

A

Very helpful. all right,.

A

Let me look at my outline and see if there's anything else, I wanted to talk.

A

About. uh,, so,.

A

So I said if we release without validation, it's really hard to change. again,.

A

Anytime you're changing an api,. You have to ask yourself, how are old clients and new clients gonna respond to this?.

A

How our old api servers and new api servers are going to respond to this?.

A

And so the tightening of validation or the loosening of validation, uh,.

A

You run into problems,, especially around api server, skew.

A

So before it was an issue about dropping data., but you can imagine if,.

A

Say we've forgotten we releaetcdthis without validating anything and then a.

A

Release later we're like, oh, negative numbers don't make sense,.

A

We should definitely stop that.. Well. Now you can get in a situation where the new api server will forbid. You.

A

From setting a negative number,, but the old api server would have allowed it.

A

And so you can have existing data, that's set to negative one. uh,.

A

And some of our deletion paths actually require you to update the object.

A

So I don't know if you're familiar with finalizers.

A

finalizers are a breadcrumb. You can put on an object that says when someone.

A

Wants to delete this before you, let it be taken out of at cd. um,.

A

Let all the controllers that own these finalizer tokens do stuff.

A

And when they're done, they can take their finalizer name out of the list.

A

And so that's done with an update., and so if you have bad data and that cd and the api server won't, let you update.

A

The object because the existing data is bad,, then you can't. and,.

A

And it has a finalizer on it and you can't even actually update it to remove the.

A

Finalizer to delete the thing that has the bad data., it's no good.

A

Uh,, tightening validation is kind of a landmine.

A

Don't get in that situation by having good validation from the beginning.

A

A

All right., I think that was most of what I wanted to talk through on that one.

A

Are there questions about this change in particular, or things that occurred to.

A

You, while we were going over this before I kind of move on to the next one. yes.

A

A

So if validation had existed.

A

It would be under this file, and that was a field on the pod spec. I believe.

A

Am I in the wrong file?.

A

I apologize. this is what happens when I don't have my computer.

A

There's a validation, dot go file in it., our methods like validate pod and they start at the top level object, and you can.

A

Kind of walk, through, they'll, usually call sub functions for like validate pod spec.

A

And so it's usually a function like that, where we're doing existing validation.

A

And you would add a new stanza that would look at this field and say,.

A

If it's not null, make sure it's zero or,.

A

A

W wish I had a better example to show you.

A

A

Validation runs on the internal types.. That's what I was doing. Wrong.

A

A

So it looks like this. so it,.

A

This is walking through all of the fields in the pod, spec and uh,.

A

Validating them. and then when it finds a problem, uh,.

A

We put structured data about the error., so we have different error. Types,.

A

Whether a field was required or field was invalid or a field like a,.

A

An interim field had an unsupported value..

A

We have a structured type that actually tells you what the valid options are.

A

Um,, so digging into this kind of gives you a good idea of the breadth of the types.

A

Of validation checks we do. uh,, so there was a question about. Could you make a field optional in the schema set.

A

A default and then require it in validation.

A

and if you think, through the pipeline that actually works, right? like we read,.

A

We decode from what the user sent us,. Then we apply defaults.

A

So in defaulting we could say if this field is null,, set it to 60.

A

I don't think we would do that. in this case. We want to preserve existing behavior, but we could, if we decided.

A

Nope, 60 seconds make sense for everybody who didn't have an opinion, uh,.

A

We could apply that default and then, by the time that happens before you get to.

A

Validation, by the time you get to validation, we can be confident that there is a value.

A

Because if these are provided one,, then we have one and if they didn't provide one, then we defaulted one.

A

And so defaulting happens before validation., so validation could say like if it's no market required,.

A

It's kind of a no op case,. You would never actually see that case if we defaulted, the field.

A

But belt and suspenders are nice sometimes.

A

Actually,, let me pull up defaulting, so you can see what that looks like as well.

A

There it is. so [inaudible].

A

Validation happens on the internal types defaulting is per version.

A

So defaulting happens on the external types., and so you can see,.

A

Uh,, basically, you get handed an object and you can do things like say if this field.

A

Is null or zero length, then populate it with some,.

A

Some data. uh, this is actually,. This is actually a really bad example. don't,.

A

Don't do this., don't default, uh, one field baetcdon, another field.

A

Um, [inaudible], this particular example cauetcdus lots of problems where, if you create,, if you created a replication controller- and you put some labels on the,.

A

On the pod spec,, we would default the selector to match the labels and the pod spec,.

A

Which sounds like it's really helpful and nice., but then, if you updated your replication controller to change those labels,.

A

We wouldn't change the selector as well. and so you'd get an error. Saying,.

A

No, your selector has to match your labels and your like selector..

A

I didn't even tell you about a selector.. What are you doing??

A

I'm defaulting is generally better if it's stainless,.

A

If it,, if it is only saying like if this field is, no,, it should be 60 seconds,.

A

Not if this field is knoll and this other field is a thing,.

A

Then derive this field from that field, like that that causes problems on update.

A

When the user changes the other field but doesn't know about the one you.

A

Helpfully defaulted from it.

A

All right. any other questions before we, yep.

A

Good question. are there rules about loosening, validation? yes.

A

Loosening validation from an api servers perspective is actually the same as.

A

Tightening it in reverse. So when you tighten it,.

A

The next generation of api server will prevent you from doing things that the.

A

Current generation lets you do, and so that can cause problems trying to update.

A

Or delete bad data in the next generation., if you loosen, then the n minus one version of the api server will choke on.

A

Updates to data that the next generation will let in. and so you,.

A

It's the same problem just in reverse., and so, if you,, if you absolutely have to tighten or loosen,.

A

You have to roll it out over multiple releases, where you tolerate the data that.

A

The other api server version would consider invalid for at least one release.

A

Um,, that's just for the api server.! That's like the bare minimum that we have to do to not get ourselves in trouble.

A

Where we can't update objects in general., when you're, loosening validation,.

A

You have to think of kind of every client. That's going to interact with your api.

A

Object. um,, so a couple of examples. uh,.

A

Config maps, uh,, the key values, are very restricted.. It's just alphanumeric.

A

Characters- and there was a request to let config maps represent like multiple.

A

Levels of directory hierarchy by putting slashes in the name.

A

And if you narrowly think about just the cubit who's, taking the data from the.

A

Config map and then sort of making files on disc to put into a pod,.

A

If you only consider the cubit as the client,.

A

Then that could actually work. like after you updated the cubelets to new enough versions to understand that.

A

Uh, but if you think more broadly about, oh, config maps are kind of a generic key value. Data structure.

A

Our other clients, reading config maps and extracting them to files on disk.

A

Yes,, probably so in some cases definitely so and so would allowing a slash character.

A

What problems would that cause for all the clients that are reasonably operating.

A

Against this api. so there that was a case where there was a desire to do that.

A

It was fitting us fixing a particular use case and we could fix up the cubit.

A

Given enough releases., but to actually make that work safely across all clients was pretty prohibitive.

A

Um,, that's kind of an extreme example,.

A

But you always have to consider the case like what clients are using this api.

A

And how will they react to this change when you're wanting to change something.

A

All right. other questions?- these are good.

A

Oh yeah,, that's a good point.. I did not check if that was done. Correctly.

A

Okay. and it wasn't,, this is so helpful.

A

I recommend to everyone, have an audience checking after them. um,.

A

Let me see if I have,- it's not represented in this slide.

A

Ah, so there's actually a phase in the pipeline, uh, after we've decoded,.

A

After we've defaulted, after we've converted to internal,, there's a,.

A

We call it, uh,, prepare for create or prepare for update. um,.

A

It's basically just a hook point where the server can do stuff.

A

Uh, different implementations do different things with that.

A

But one of the things we can do is enforce this feature. Gate that says, if you.

A

Submit an object, that's using an alpha field and the alpha feature isn't turned.

A

On, that point that before create, prepare for create, uh,.

A

Prepare for update point is where we go through the object and say,.

A

This feature is off drop, date, drop date. Out of this field,, don't allow it in.

A

Um,, and so I can show you what that looks, like.

A

Uh,, where would be a good one to go?, probably pods.

A

So all of the builtin resources have what's called a strategy,.

A

Which gives you sort of lifecycle points on the object? Handling.

A

So you see, prepare for create, and it's given the,.

A

The object that is being created.- this is the internal pod. After it's been defaulted and converted to internal.

A

Um, but before validation, uh,.

A

And then you have the prepare for update and it's given both the old and the new.

A

Object., so this is when you're updating a pod and it can see, well what,.

A

What was the existing object in etcd and what's the new one that they're wanting.

A

To persist?, mmm., and so in both cases, call this helper method,.

A

Drop disabled pod fields and we pass in the old and the new object when you're.

A

Creating than the old one is no,, we don't have an old object.

A

And you look at that implementation.

A

Drop disabled fields., uh,.

A

So here we've drilled downs, we have the old pod spec and the new pod spec.

A

And so this is,, this is what it looks like generally.

A

So we're going to say if this feature is not enabled and.

A

They didn't already have this field set in the old object,.

A

Then drop date out of this field in the new object.

A

And so the feature enablement piece is pretty clear,.

A

Like you, don't have the feature on this piece took us.

A

Three or four years to figure out that we needed to do. um,.

A

This is what lets you avoid: data loss issues when you're running an n minus one.

A

Api server., so if you think about this, uh,.

A

The features off,, that's cool., you go to create a new object and you've set this field. um,.

A

We don't have an old object., and so if we look at what this does,.

A

If the old one is nil,, then we say no, we weren't,.

A

We weren't already using this field. uh,, so we should go through and drop the data..

A

If the old one already had the data in it,, then we want to preserve it.

A

We don't want to drop it, because that means either they enabled the feature,.

A

Persisted stuff and then turn off the feature,, which is not ideal.

A

But sometimes you got to turn off a feature to stop the bleeding..

A

We don't want it to add data loss on top of that.

A

or you might be in a skew case where there's a newer api server. That's allowing.

A

Data into that field, because the future's graduated to beta and as the old alpha.

A

Server, we're like, okay,, so somebody allowed it in.

A

we're just gonna. Let that same data be persistent., uh,.

A

In case you're getting lost,, this is actually laid out in the api changes. Doc.

A

So I think it was linked to from the session description. uh,, so it talks about how do you make changes to an api and it walks through,.

A

You know, adding a field and associating a feature gate with it and validating it and how.

A

To drop data from it from one release. so.

A

Again, [inaudible], I made all the mistakes that these are protecting. Against.

A

Yeah. all right, before we move on other questions?.

B

A

So the other,, the other thing I thought we would look at- is a proposal.

A

So there's no code associated with it yet,, but it's a proposal to,.

A

Um, allow changing the resources that a particular pod requests, in.

A

So for all of kubernete's,, once you create a pod,.

A

It's frozen. pretty much. we,.

A

There are a couple of things we allow to change in a pod after you create it.

A

But, uh, all the,.

A

All the things that deal with resources, like a pod saying,.

A

I want this much cpu or this much memory, that's locked in place,, uh,,.

A

And that makes it really easy for some things.

A

so create a pod, and then the scheduler can say, oh,.

A

Well, this pod wants two gigs of memory..

A

Let me find a node that has two gigs of memory: free and it'll match,.

A

Make the pod the node. uh,, and so that makes that pretty easy to reason. About.

A

And for a long time, the answer was, well,. What? If I need more memory? well,.

A

Change your deployment, spec and roll out a new revision of the deployment,.

A

Right? uh, and that, that works. okay, that works fine for some things,.

A

But sometimes the restart cost is very high,. Right? imagine,.

A

You can imagine an application that takes 10 minutes to warm up or 30 minutes to.

A

Warm up and fill its cash, or you have like data gravity issues, where, uh,,.

A

I just got all my data to this node and you're going to want me to, you know,.

A

Rev the deployment and it's going to end up over there., I'm going to have to pull all my data over there.. It can be expensive to restart.

A

and so there's a desire to allow, uh,.

A

A feedback loop that updates in place the resources that a particular pod.

A

Requests. and so.

A

Today, uh, the,, the resources are in the pod spec,.

A

So it's just a map of like resource type to amount or quantity.

A

And the assumption of all clients today is whatever is in the spec, is what the.

A

Pod has. uh, and so the scheduler,, if there's not sufficient resources for that,.

A

It will refuse to assign it to a node. and once it gets assigned to a node,.

A

If the node says, oh, I'm out of resources, uh,, I forget, if it does feed back onto the pod or reports pressure,.

A

I forget the exact flow,, but there's only one field in the pod that says,.

A

What resources does this pod have?, and so this is proposing, uh,.

A

Adding a, a second one. uh,.

A

And so just take a minute and kind of read through the description to try to.

A

Understand like what,, what's happening, here. uh,.

A

So they're wanting to add a field into each container that says: resources.

A

Allocated. and so.

A

Existing resources field would be the user specified source of truth.

A

And it would become mutable apparently.

A

And then the resources allocated would be the thing that kind of acknowledges.

A

That the resources the user requested are allowed or available or accepted.

A

So it looks like there was an alternative considered where it was instead of two.

A

Fields in the spec,, like the resources I want, and the resources that have been allocated. uh,.

A

There was a discussion about splitting it between spec and status,.

A

Where the user says, and the spec,, this is what I want.

A

and in the status something says yes or no,.

A

Or like this is what's allocated., so that's an interesting sort of side, road.

A

One of the,, one of the principles of the spec status split, is that status should be.

A

Derivable or reconstructible or replayable.

A

And so, if you,, if you're, trying to decide whether a given field or concept should go in the.

A

Spec or the status,, sometimes it's helpful to kind of do the thought. Experiment, if,.

A

If my status got erased,, could I rebuild that or re-report that or the controller that made all the.

A

Original decisions, could it like replay and make those same decisions again?.

A

And so in this case,, the answer's, not really., if you put it in status,,.

A

Then the scheduler would lose information if the status was thrown away.

A

And so these are,, this is more of an example of two different actors collaborating on the.

A

Specification of the object., and so there are other examples of this, right?.

A

Like a deployment or a replica set., the user says here,.

A

Here's the app I want to run.

A

and then you can hook up like an auto scaler that monitors usage and says,.

A

You said you wanted this to be at 75% cpu, but it's higher than that..

A

So I need to increase the replica count., and so you have an autoscaler that goes. In, changes the spec of the rep of the replica set.

A

So we have examples of different actors: sort of collaboratively working on the.

A

Specification of an object. and actually the server side apply. Work is designed to make that even easier by.

A

Associating which fields in an object are controlled by which actors.

A

So I, as a user, would submit my deployment and it would say,. Oh yeah,.

A

Well, this user is managing all of these fields,.

A

But the autoscaler controller is managing the replicas field.

A

And so that's intended to start letting us give warnings. if I,.

A

As the user went and like tried to stop the replicants field,, it would say, hey,.

A

Just so you know, you're, changing a field, that's managed by the auto scaler.

A

like,. Are you sure you want to do this? like you can force your change, in,.

A

But you might be scaling your like nicely auto scaled tuned hundred replica.

A

Thing back down to one., like are you sure?,.

A

And so surface I'd apply, managed field, tracking is,.

A

Is allowed to make it easier for things to collaborate on an object without.

A

Stopping on each other accidentally. all right., so I that makes sense to me.

A

the argument that we couldn't reconstruct this, that this really is sort of a second phase.. The user asked for x,.

A

The scheduler or the node collaboratively determine that x is available,.

A

Reflect that back into the spec. that seems to make sense.

A

Uh, the questions that I would have.

A

Are around defaulting and other clients.

A

So again, these,, these sort of phases of api processing are helpful to keep in mind.

A

If you have existing objects,.

A

Uh,, what would this new fields default? To?.

A

Would it default to nothing or would it default to match the current value of.

A

The resources field?, so that would be a question I would have.

A

Mmm., the second question would be around existing clients.

A

A

It's helpful to think of clients, not just as the ones who are making requests to.

A

The api,, but also any, any client that is having this data sent to them.

A

And so admission web hooks are actually a client of the api.

A

We ran into this with the ephemeral container work that was done.

A

One of the things we're seeing a lot of is use of admission web hooks to make.

A

Policy decisions, right?, I want to,, I want to put a web in and every object that comes in,.

A

I want to run through some policy decisions and say, yes,.

A

You can run as route or no,. You can't run privilege pods or yes, you can use nfs volumes or whatever.

A

And so those admission webhooks,.

A

They're written at a point in time.

A

and if we add capabilities to api objects,.

A

We have to think about what would that do to someone who made policy decisions.

A

Against the api as, as it existed in the past. and so the,.

A

The case of ephemeral containers, someone might not want to let any pods run as root., and so anytime they see a pod,. They go through all the init containers and they go through all the containers and they say,. If you've got your run as root,, then I forbid you,.

A

That's cool,, that's great., but then we add a femoral containers and now there's a new,.

A

Like a new field with a list of containers which can run as root.

A

And anything that was written as a policy decision against the old api doesn't.

A

Even know about this new field. and so.

A

When there are cases like that,, we often will require that access to the new field be partitioned off and.

A

Separately, authorized., and so that's what this is proposing. For.

A

This resources field is saying we want to control access to this new field and.

A

Only give it to things like the scheduler or the node..

A

We don't want to let just any user be like,. I definitely want two gigabytes.

A

And also I allocate two gigabytes..

A

I hopefully decided that I should be allowed to do. This. um,.

A

And so that,, that's what this is proposing,, having access to this field b via a different endpoint,.

A

Authorized specifically for that. uh,.

A

It doesn't exactly protect, uh, existing clients,.

A

Uh, because you know, they,, they might still want to know that this field exists,, but it does guarantee that no one who currently can mess with pods, uh,.

A

Could set this new field. you'd have to do it specifically via a different endpoint,.

A

Which is specifically authorized., so that's why that's proposing that.

A

Um,, that seems, okay. it,.

A

It's a little fiddly to have a lot of different sub resources to address lots of.

A

Different fields on an object., so I might press on that a little bit more and see if there's a different way to.

A

Authorize that, rather than having a new endpoint.

A

A

A

I'm trying to think what else occurred to me reading through this one.

A

All right,, so this is talking about a resize policy.

A

So the container can indicate what it wants to happen when the different.

A

Resources, change. um,.

A

The options are sort of a map of the resource, type to a policy or a list of.

A

Named, uh, policies. um,.

A

For a long time, we've done this, uh,, pretty much everywhere.

A

A

Experience has shown this is harder to use than it should be. um,.

A

The upside is it is,, uh,, very deterministic.,.

A

You don't get languages trying to reorder things in maps..

A

The downside is it's a pain to use. and if we had it to do over again,.

A

We probably wouldn't. um,.

A

We haven't really worked through new fields. uh,.

A

I think brian made some comment on the v to abi saying the named lists were a.

A

Mistake and he wouldn't do it- again. um,.

A

This may be the first one that I've seen come up since that comment came up,, so I'm not sure what we'll do with this.

A

A

Yeah. so, uh,, so one thing to point: out,, the one on the left is simpler. uh,.

A

It doesn't give you any room: for, uh,,.

A

Expansion., so this is just saying: there's, there's a single policy like scaler. If there was some thing,.

A

Some additional bit of data that you needed for different policies,, like, uh,.

A

I want to restart the container,, but only if it,.

A

But only one container at a time. or like only at this rate or like. If you wanted to have parameters associated.

A

With that,, the structure on the left doesn't give you any place to put that data.. So the,.

A

The key value that the map structure is probably okay,.

A

But the value, if it was an actual object, could give you a place to put.

A

Parameters, if you wanted that in the future,.

A

You have to fight against like the making everything more complicated than it.

A

Should be., but at the same time,, knowing that we have existing rollout and uh, upgrade,.

A

Uh, strategies and some of them have for amateurs associated with them,.

A

Like needing to have parameters seems like a thing we might need.

A

So it'd be interesting to at least think through whether that should be a map of.

A

Like a resource type to update strategy that could take parameters in the.

A

A

Um, the,, so, in addition to actually being able to express the,.

A

The pod resource requests and allocations in the api,, the rest api,.

A

There's changes on the backend needed to communicate this to the thing running.

A

The container. um,, so this is detailing some of the container runtime api changes.

A

This is a grpc, uh, api that the kubelets speaks to the container runtime.

A

And I'm actually going to defer this to the node folks, because I don't know that.

A

Much about this ari api. uh,: this is one of the places it's helpful to have people with expertise in.

A

Different areas., so I would,. I would kick this over to tim, who just walked in,.

A

But I won't make him talk, now, uh, and, and see what they thought about. This.

A

Um, something that I don't think we've done as much with the grpc apis is um,.

A

Sort of versioning and maturing them., so the cri api is actually still technically labeled, as alpha, uh,.

A

Even though it's been around for years and years and everybody's using it.

A

Um,, so something that would be interesting to work through is how we actually get.

A

That to be called v one, and if there are changes we need to make or should make or if it's just a relabeling exercise.

A

So yeah, those were the,. Those were the main thoughts I had about that.

A

the argument for it being in spec made sense. the.

A

uh Huh. questions about defaulting., if that's been thought through that'd, be good to talk. Through. um,.

A

And the sub resource, the,.

A

I think it would be helpful to see what it would look like to enforce access to.

A

That fueled,, not via a sub resource. um,.

A

Things like crds, don't let you do arbitrary sub resources. uh,.

A

And so it's helpful to ask ourselves like if we were writing, if this was a crd,.

A

Like what would we do with this?? Would this be just a policy decision that we would hook in to say only nodes and.

A

Schedulers can touch this or people you authorize, can touch this field rather.

A

Than sort of splitting it up into a million little sub resources.

A

Yeah. so, uh, the question was,. Our sub resources is the preferred way of controlling fine grained access.

A

It's not actually that fine grained.. Typically, we have like spec and status and that's it., and so it's like a whole sub tree of a, of a object, um,.

A

Using them to shard off. like,, if you want to touch this field,.

A

You have to talk to this end point.. If you want to touch this specific field, talk to that end, point., it gets really fiddly,, especially with things in spec.

A

Like it makes it so that you can't actually pull an object and then recreate it.

A

With a single request. um,, that may be what we want in this case, because if the allocation is supposed to,.

A

Uh, be done by a scheduler or a node that has knowledge of like, what's actually.

A

Available,, that may be what we want. uh,, but I wasn't sure, based on this,.

A

I see a hand,, hey, tim,.

A

B

A

Do we uh, one sec.? Do we have a second mic that we can.

A

A

A

I don't either., it's only two ways to find out.

A

No,, you should come up here and talking to this.

C

Sorry,, I didn't mean to take over.. I just wanted to point out that, like when I look at a pez,.

C

When I think about sub resources,, I think of the sub resource as a a way of taking the main resource and granting.

C

Access to some smaller part of it,, which is counter to the general, get ops flow but is useful for things like.

C

Controllers, where you're going to imperatively be changing things, dynamically.

C

It's not a hundred percent true. all right, and there are different ways like we don't have a lot of consistency. Here,.

C

But in general, that's why I shy away from sub resources, unless I can think of it.

C

As that sort of imperative flow, like setting number of replicas on a replica.

C

Set. right. generally not what you're going to check in to get. If you have an autoscaler.

C

Right,, but you don't want to give your auto scaler full access to modify everything and.

C

So you get a nice break down there. that was my point.

A

Yeah., I think daniel said we have a few different styles. one,.

A

One style is the status style of sub resource, where the full object has returned.

A

And it's just a window to only limit you to updating the status.

A

A sub tree a,, another style, is sort of a polymorphic sub resource.

A

So the scale sub resource, uh, adapt lots of different objects to a consistent scale object so that you can.

A

Point an auto scaler at the scale, sub resource.

A

and whether it's a deployment or a replica set or a daymond set name, is that.

A

Scale. no, not damon, set, stateful, set. um,, it can,.

A

Those can sort of expose their replicas field as a scale object that the.

A

Autoscaler can say, how many do you have and what pods do you select and how many pods should you.

A

Have. um,, so that's a,, a polymorphic sub resource.

A

And then we have a bunch of weird, awful sub resources like exec and attach,.

A

Uh, that you'll notice are not present in crds, because we don't really consider them.

A

Best practices. uh,, there is a desire to sort of let you hook, um,.

A

Endpoints or pointers to end points onto resources and that that may happen in.

A

The future,, but it probably won't, look exactly like the current sub resources do that route.

A

Everything through the api, server. uh, that's,- that's not really something we want to do with crds., it's more likely that we would do redirect or links or some sort of very.

A

Targeted aggregation, um,, instead of building it into the api server.

A

All right., I think we have about 15 minutes left.

A

I figured there would be questions about these or just other things,.

A

Things that are unclear, things you've always wanted to know, things that you said.. Is this the way it is because it's good or because it's just the way it is.

A

So I figured we'd leave a fair amount of time for questions and answers.

A

A

You don't want to go on tape., that's fine., okay., uh,,.

A

A

Okay., I'm going to try to repeat that for the recording. uh,.

A

Going back to the question of sub resource versus policy, versus,.

A

Um,, an early iteration., this was done as an annotation, and one of the problems was that anyone could.

A

Just change the annotation.. It was very hard to put policy around it. uh,.

A

I think tim's point about is this something you would check in and expect it to.

A

Be able to create in one, you know, in one shot and have this field be populated, um,.

A

Is probably a good one,, like rule of thumb, to ask., and so if this field is talking about reflecting like what's actually allocated.

A

In the current state,, it's probably not something you would check in or expected to transfer from one.

A

Cluster to another., so it's possible that it would be okay to do.

A

Uh, um,, I'm not,, like I said,, I'm, not a huge fan of like lots of random sub resources,.

A

So it'd be worth thinking about if there was an alternative,, but there may not be. um,.

A

Okay. I see., so the proposal was to default it to match the current field.

A

A

I see. so,, so the,. The goal would be to keep the current field,, the user intent,.

A

And have that replicated into what is quote unquote, accepted.

A

And today it's a single shot,, like whatever the user says they want, they get,.

A

If quota allows it. uh, and this would let us, um,.

A

Be able to see in the api if they had changed their intent from when the pod was.

A

Originally submitted. I see. yeah.

A

A

A

okay. yeah. so,, so that could make sense..

A

I think, it'd be interesting to work through all of the cases, like when you submit a new object, like what do we do if there's data in the field,, what do we do if there's data not in the field,?

A

What do we do if there's data in the field that doesn't match? um,, so it,.

A

Like some of that has been worked through already. okay.

A

Uh, oh,, I remembered something else. I wanted to talk about. uh,.

A

So in some cases like pod specs,, so pods are the,.

A

Probably the oldest object that we have in the api.. They have lots of history,.

A

Uh, lots of validation, lots of oddities. um,.

A

Because we don't allow you to change a pod spec on update.

A

Um,, when we're adding a new field like this,, we have to think about, again,.

A

Ask yourself the question like what is this gonna do to all existing clients?.

A

And so we've, this is talking about adding a new field to the api.

A

Once that field has data in it,.

A

If we just leave validation, the way it is, that new field is gonna, be subject to the same immutability requirements on.

A

Update., so if I get a pod and then write the same pod, back,.

A

If we don't change validation as part of implementing this, by default,.

A

If I change that field validation will complain at me and say, nope,.

A

You can't change that field., uh, and so.

A

That's,, that's fine.. You probably don't want users directly changing this new field on update.

A

But think about a client that is reading the existing api into a go type.

A

When they read it into a go type,, they will drop this new fields. Data.

A

And then, when they write that back,, they will submit a pod update that doesn't have that field present, because they,.

A

You know,, they were built against the one 16 level of the typed client.

A

And so we don't want to break that client.

A

And so we need to think about cases like if I see an update to come in and the.

A

Old object has this resources field set and the new object does not have the.

A

Resources field or a resources allocated field set.

A

This could be an old client that didn't know about the field and dropped it.

A

It could be a new client that knew about the field and intentionally dropped. It.

A

And we can't really tell the difference., and so in a case like that,.

A

We air on the side of not breaking the old client., and so what we would probably do is copy the object or the data from the old.

A

Object to the new object.

A

Yeah. um, so yeah, asking yourself, how,.

A

How will this affect existing clients from an existing client's, perspective?.

A

What things can I do today that will succeed? and with this change,.

A

Will those things still succeed? uh,, so we,.

A

We as we're improving things and making changes and adding capabilities,.

A

Uh,, it can't be at the cost of our existing clients., so.

A

I saw walter's hand,, you had a question. oh,.

B

So you mentioned the better to have 50 reviewers in the five for those of us,.

A

We'd like to help you with that, beyond going and watching this current.

A

Presentation. thank you.. What would be the next steps?.

A

A

We worked last release on identifying reviewers per sig,.

A

So we started doing that.. If you go. Look in the, uh,.

A

The top level owner alias file, um, each sig has a stanza that has a list of names. uh,.

A

Some of the sigs that did a lot of api work in the last few releases,.

A

Like say, windows and six storage. um, really got several people involved and they did a good job of sort of stepping.

A

Up and doing first pass reviews on things. uh,.

A

I would take a look there at the particular areas that you're interested in.

A

And if there are names, there, then get up with them..

A

Some of them have already gone through shadowing, uh,.

A

Api reviews on particular changes. if there aren't names, there, uh,,.

A

Talk to the people who are running that,, that sig or that project and say,.

A

I'm interested, I'd like to learn more. I'd like to be around for things that are going on and start to pick up. First.

A

Past reviews., but I would start there, pick an area that you're interested in and talk to the people running. That and.

A

Take a look at the owner, eylea smile to see who is currently sort of on the hook.

A

For shadowing this., all right,.

A

I'll give you a few minutes back.. I think it's about lunchtime, and, uh,.

A

Thank you, all. I'll, be around., come up, tim's, there.

A

Daniel's around.