Kubernetes Kubernetes Contributor Summit - NA (San Diego) 2019, 26 Nov 2019

Previous Meeting

⏯

youtube image

►

From YouTube: Summit NA 2019: Raising the Bar - Production Readiness in K8s

Description

In SIG-arch, we're building a new process to ensure features are ready for production before being promoted to GA. This will add new gates to promotion from alpha to beta and from beta to GA. The presentation will enable you to be prepared and help you land your feature on time.

Event link: https://events19.linuxfoundation.org/events/kubernetes-contributor-summit-north-america-2019/
Session link: https://sched.co/VvMK

A

All right., thank you. Everybody, today,, I'm john belamaric.

A

I'll be talking about production, readiness,, so,,.

A

What is it? so,? This is a new subproject that we've started.

A

In sig architecture, and the goal,.

A

This is,, it's a project focused on production, readiness,.

A

Which means ensuring that features are ready.

A

As they go through the stages of alpha, beta, ga,, there's different criteria for what we consider production ready.

A

I'll, get into that in, a little more in just a minute.

A

As I said,, the goals of the subproject are to define these criteria.

A

We've also, are planning to build a,.

A

A review gate and process for features.

A

As they move through this., this is really a program that we're just starting., it's kind of in the initial pilot phases,.

A

It's not,. It's not like a new requirement.

A

That's coming really soon,, but our goal is that at some point this will be a new set of reviews,.

A

Sort of like we have to do api reviews sometimes, that we'll have to do production readiness review for different features as they move forward.

A

To make it actually happen, we're trying to avoid any kind of new tooling, and use existing prow tooling to try and simplify it.

A

As much as possible.

A

Another thing, another non-goal is to define these things.

A

In the context of different ways of deploying kubernetes,.

A

So, some people deploy on-prem, some people deploy in the cloud,. Some people deploy on the edge, and the kind of things that we wanna talk about in production. Readiness aren't really gonna get into those different deployment. Models.

A

we're just gonna focus very closely on the sort of universal things that are necessary to make sure the quality is high.

A

So, why?, why are we doing this, and why are we doing it? Now?.

A

Kubernetes has been around now for a few years, and it's grown dramatically, and,.

A

It's grown organically,, meaning we've had people adding things that they see.

A

As needed, functionality and needed features, over time,.

A

But it's an open source project that has, it's somewhat chaotic,, as you might expect of any open source, project, so,.

A

We did introduce this concept of keps, and,.

A

As paris showed, earlier, keps are doing very well,.

A

Everybody's using them,, they're great,, but they're actually written by the people, writing the code, usually, and they're reviewed by the people who are interested.

A

In writing the code, and there's, the people who actually run clusters in production.

A

Don't necessarily see those or review them early enough.

A

Or ever, and,, so that means that those people.

A

Aren't necessarily represented, and we get oversights, things that really would make their lives easier.

A

That get missed or left out, or not looked at.

A

So,, that's really the goal or the motivation for this,.

A

For this project.

A

So,, what do we mean by production, ready?.

A

Really it comes down to these, these core things.

A

So observable. so, we,. It means that you deploy some feature out. There,.

A

Say, some new resource type, and it's got a controller and it actuates some behavior,.

A

And we wanna make sure that if we roll this out,.

A

Then it's actually working.. We want metrics to see that there aren't failures,.

A

We want to know that people are using it.

A

if you're operating a cluster and nobody's using some feature, then, you'd like to know that, or if a lot of people are 'cause, it can affect how you do your upgrades and things like that. so,. We also want to document.

A

What reasonable slos should be expected? From,, that's service level. Objectives, should be expected from a given feature.

A

It needs to be supportable,, there needs to be a playbook, and I don't, the keps today sort of serve as a playbook,, but they don't have sections like,. How do you troubleshoot this feature?? What are the important logs to look at if something's going wrong,? How do you even know if something's going wrong??

A

What are the failure modes,?

A

So all of that needs to be documented,. So that's part of production ready.. It's not always a thing. You need different in the code,.

A

All of this may be there,. It's just,. It's not documented,.

A

It's not ready for people to take out there and use.

A

And be able to support. and,, we also want to examine whether it's gonna have some unexpected impact.

A

On scalability,, so there's a set of questions that come out of this.

A

That try to surface any problems that might be, might occur.

A

Finally,, we want to know that it can be disabled, when it's a pre-ga, feature, so you're properly implementing your feature gates and that, that when you actually turn it off, like say it's been running in production for a week and then you find it's causing problems,. You turn it off, that that's not gonna cause its own problems, by turning it off.. So somebody has to actually ask those questions and examine that, and even for things that are post-ga,.

A

It's a more,. We don't require a future gate for that, once it's a ga the feature gate's on and removed, but,.

A

You should be able to roll back, right?. You should be able to roll back to an earlier version if you're causing massive damage. Somehow.

A

To your service in production., so we wanna make sure that the kind of,.

A

The kind of changes that are implemented when you deploy this feature, don't, aren't irreversible.

A

In the sense of not being able to roll back.

A

So what are we doing as part of this subproject?

A

well, we're developing,? The first thing: we're doing, we're developing a questionnaire,, a production, readiness questionnaire,, it's basically,. If you've read the sutty book or something.

A

From google,, or it's similar to things done internally.

A

At google and other places, is sort of like,. Ask these questions of the people. Writing the software.. How do you,? How do you do those,?

A

How do you troubleshoot this in a live situation,.

A

And what should we look? For?

A

we're also developing a review process,.

A

To try to enforce that. so the idea here is,, I think I talk about it. Later,.

A

The idea here is to trade our sets of reviews based upon graduation from different phases.

A

We're doing some, in order to make sure that this is effective and useful,, and that we ask the right questions,. We are doing some, some sort of field, research, around,.

A

What are the kinds of people, problems people have experienced in production??

A

Have we seen cascading failures,?

A

Have we seen other kinds of,?

A

What's the pain for operators right now,, so that we can make sure that whatever we actually,.

A

The questions we ask actually surface the kind of issues that people have experienced in real life.

A

we're also doing a postmortem of existing features, so,.

A

One that people have brought up is cron jobs,, so cron job's been stuck in beta for a long time and there were a lot of issues with cron job in production,.

A

And it's a big question: of, okay,.

A

Those, some of those were caused by bugs, so,. Would this process even surface that? maybe,? Maybe we can't surface everything, but let's at least see what we can do,.

A

So we wanna look at all these other features and experiences people've had in the past and see if we're asking the right questions to surface and avoid those kinda problems.

A

Finally,, we do understand that more review gates is painful,.

A

And so we want to keep it as lightweight as possible.

A

When, speaking of review gates,, when does all this start?

A

so, we've actually started this process in a pilot phase in 1.17.

A

we have a handful of keps that were sort of voluntarily going through with, the authors, having them answer a set of questions..

A

It's very much just a pilot, phase.

A

originally,. I had hoped that we would be able to move faster.

A

And get something in 1.18 that, that has some teeth,.

A

But I don't think that's realistic at this point, so likely we'll pilot through 1.18 and then,.

A

I would hope to target in 1.19 that we have some kind of,.

A

At least manual process,, if not any tooling.,.

A

A

How does it work? today,? As I said, it's a pilot,.

A

So that means that if you have a kep or a feature that you're either new, wanting to do it alpha,.

A

Or you're wanting to advance from alpha to beta or to ga,.

A

Then we would ask that, if you're,, if you're willing to be a guinea pig,, you come to us at this production readiness review, team,.

A

And we would work with you to answer the questions.

A

In the questionnaire,, which is, I'll have a reference to it, later, and you can just answer those directly in the kep.

A

From there, we would help review your answers to that question, make sure that they're, hopefully provide feedback.

A

That would direct you to maybe do things a little bit differently.

A

in the future,. We want to have tooling that will trigger on a pr, that's graduating,, that's either creating a new alpha api or graduating an api from alpha to beta or maybe ga,.

A

And the idea is that, similar to how,.

A

Even,, actually, we do, in conformance,.

A

One of the ways we do it in conformance is you,.

A

There's a text file where a line is added,.

A

And only a certain set of people have the right.

A

To approve that, and so, we're thinking a similar kind of tooling,, but part of the pilot program is to figure out exactly how we wanna do that.

A

Again, intended to be lightweight.

A

We don't want a sort of centralized production readiness, review team.

A

That everything has to go through. we, with api reviews. We have to do that, and we know that that, that can become a bottleneck,.

A

So as much as possible, as we have each of these sections in the questionnaire,.

A

And I can probably bring it up. Later, those,.

A

We'd, like individual sigs, to probably take responsibility.

A

For individual sections, there, so, sig scalability, for example, has a set of questions that they've presented, and, as the reviews would happen,.

A

They would be the ones that would approve those specific answers in any given kep.

A

So,, how can you help this process move forward,.

A

So that we can have a better kubernetes in the future,, that's more supportable and makes operators' lives easier?.

A

The first thing you could do is volunteer your experiences., as I mentioned, we're,. Part of this is understanding and surveying the landscape.

A

To see what kind of pain people've been through,, so if you have horror stories or not-so-horrible stories,.

A

But that are real operational challenges,.

A

Talk to vallery, here, or jorge, or in,.

A

They're, both working on our sort of field, research.

A

Of this team, and I'm sure they would love to hear from you.

A

There'll be a survey that goes out, they're, putting together, like a google doc,, a google form.

A

That will go out to probably the dev mailing list,.

A

And maybe or the users mailing list,.

A

So if you see that and you've got something to contribute, add to it,, but if you wanna do more, we're also working on a more one-on-one interview, type of format where we can get more in-depth, so,.

A

That should be over the next month or two,.

A

I would say, I would guess, they'll have something there for that..

A

You could also volunteer to run your feature through the pilot process..

A

The questionnaire is out there on the community site. Already.

A

you can go in, and all you need to do is copy. That.

A

Into your kep, and answer the questions right there in your kep, and submit a pr and tag, one of us.

A

I guess I should probably set up a github team for it.

A

We also are looking for feedback on that questionnaire,.

A

So anything you can contribute, there would be great.

A

Really,, that's it.. Here's some references, oops.

A

actually,, that's great.!.

A

Here's some, references on that slide,, I don't know how, if I can scroll, yeah,, but to give you an example, here's. The kind of questions we're talking about.

A

That we would be asking kep authors to answer.

A

The goal, really, is that this shouldn't take long.

A

This should be like, if you're writing this feature,. You probably either know answers to these things,.

A

Or you should know answers to these things,, so if it does take long, it's actually doing its job to make you think about it, but, nonetheless, in general, they shouldn't,.

A

Most of them should be pretty easy to answer.

A

How do I go back,? I can't even see my.

A

So that's really it, any, next, it would just be q&a.

A

(Muffled speaking) [announcer], you guys have any questions,. I have the mic,, so raise your hand, nice and high, and I will bring it to you.

A

All right, so you're all gonna go right after this, and you're gonna go and start answering these questions for your keps, right? awesome., okay, well,. If there's no questions, then,, it's a short session.

A

Right, thank you., (audience applauds).