Kubernetes Kubernetes Contributor Summit - NA (San Diego) 2019, 26 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Contributor Summit NA 2019: Precisely Defining Kubernetes The Path of Conformance

Description

The conformance subproject of SIG Architecture defines the APIs and features that need to be available across all different Kubernetes implementations. This session will discuss how the conformance process works, how to help us ensure workloads and tooling are portable, and how to make sure your feature is ready to promote to GA.

Event link: https://events19.linuxfoundation.org/events/kubernetes-contributor-summit-north-america-2019/
Session link: https://sched.co/XClW

A

I'm so glad you're here.. Thank you for coming.

A

I want to take us on a journey through space and time and and take a look at the.

A

A

And how, as a community, we can actually have global reach and impact.

A

This photo that you see on the screen right now was taken in 1960s.

A

It's one of the first broadcast from space.

A

It's a television broadcast and it's the first time we see clouds.

A

This was taken a bit later., it's a pretty famous picture.

A

It's called pale blue dot.. You might not be able to see it on your screen,.

A

But in that red band on the right about halfway down the screen,.

A

One third of that pixel is earth.

A

It was taken by voyager one.

A

This is probably the most famous selfie in the world.

A

This is all of us.. It's taken in 1972.

A

The picture's called blue marble.

A

And it was taken by the crew of apolo 17.

A

We go forward in how we develop our technology and what we can do with our.

A

Knowing the surface area of the earth and the cloud coverage.

A

There in this is 2001, we now have many satellites and we can see the whole earth.

A

Within 24 hours., they combine those images together in a project called blue marble. Two from.

A

A

This is a day, a single day with all of the clouds in place.

A

This was done in 2002. We were able to remove the clouds and see the land and the.

A

Ice underneath and the scenes so we can remove the ice and actually see an.

A

Actual image of the earth.

A

But what's important in all of this is the people,.

A

The people that cover the surface area of the earth,.

A

And that is why the cncf is all about building sustainable ecosystems, so that.

A

All of the things that we do together can run anywhere on the planet.

A

The linux foundation, cncf is also about fostering communities.

A

The kubernetes community is an excellent example of one of the communities that.

A

Cncf fosters and through that community we're able to take and benefit from the.

A

Work that we're doing together so that we'll have production, workloads.

A

Worldwide., this is the why we have conformance.

A

It's so that what we do can run everywhere,.

A

So I'm going to go back to the beginning of these pictures to kind of do a parallel of our journey on conformance and where we can go from here.

A

This first,, this first photograph, is representative.

A

When we first got a glimpse of what what we were looking at in the surface area,.

A

Of kubernetes,, we were just looking at the log files from running the ede test suite and that.

A

Was done by ken ameche.

A

That work showed us that we were lacking a bit and testing all of the things.

A

That people could do with kubernetes., we stepped back.

A

We stepped back a bit and said, well, what else?? What are the data that we have?.

A

We noticed that we had our audit logs that were getting generated,.

A

So we began to analyze those audit logs and publish what we found to a static.

A

Website by hand and [inaudible]., I think our coverage at that point was somewhere around 14%.

A

And one dot 12, we started to get a really good picture of what was happening.

A

Because we were getting a glimpse of actual test, coverage.

A

What happened in one dot 12? Is we added support in two areas.

A

One was user agents were getting recorded by our api server and the audit logs.

A

So we went and, and could look in the audit log and say who is talking to us today?.

A

And then we asked the ede framework when it communicates,.

A

Which tests are you working on?

A

and that allowed us to quickly say of all of the surface area of kubernetes,?.

A

Which tests are hitting, which end points, particularly which ones have we flagged for conformance.

A

With one dot. 13., we were able to start seeing the changes over time because we could load the.

A

Audit logs from our rci into a series of for comparisons.

A

Um, we set up api snoop,, that's a cncf,.

A

That io and that allowed us to,.

A

And I'll show you that in a bit, um, to compare our coverage over time.

A

It's hard to see all of this, if you have all of the data and being able to query.

A

That. sorry,, you can't read that slide. roadway with dynamically querying. What's there. what we did.

A

instead of having all the audit logs, just in jason form,, we created the database. So we could ask and answer questions that we weren't.

A

Sure of yet. and that's critical to answering,.

A

Asking the right questions and getting the right answers with one dot. 15. um,.

A

It's been a really fun journey on that release, because we had api apis that were.

A

Getting promoted to ga that did not yet have conformance tests.

A

And we have a policy where everything that flows into general available must.

A

Have conformance tests.

A

And this allowed us to identify precisely where our gaps in coverage were and to.

A

Assist that team to do so.

A

One dot 16, um, gave us a new metrics of api surface and coverage.

A

And, uh,, the next part is, uh, what,. What can we do?.

A

Now we've got this increased ability to understand where we're lacking in.

A

Coverage,, but unless each of our sigs represents,.

A

Finds the api apis that they're responsible for and help us get to that 100%,.

A

Um,, it's not gonna happen by itself.. So what we've done in this last release is make it so that you can actually run.

A

Api snoop in your own cluster.

A

And as your applications run or as your test suites run,.

A

You can see, did I hit things that are yet that are not yet covered by our.

A

Conformance suite will what I'm doing run everywhere on the planet.

A

A

The api snoop interface is up at apa, snoop,, that's cncf,,.

A

That io and if you want to run it on your own cluster,.

A

You can look at our get hub repo or reach out to me.

A

I'd love to meet with each of the sigs to help them help you get to 100%, so that.

A

All of the areas of functionality that your group brings to the world has tested.

A

So that everybody can be super confident that their workloads will run.

A

If you want to get in contact with the k8s conformance team,,.

A

We're on slack, uh, and we also have a get hub repo at cncf, slash, k8s, conformance.,.

A

Is there any questions, or would you like to see aps snoop?.

A

It's going to be a bit tricky for me, because I can't see that screen.

A

A

A

And then I'll hit this.

A

And I might come sit up there with you.

A

A

So this is back at one 12 and we can even go back a little bit further and look.

A

At a one 11, if we look at one 11, you'll notice, when it loads that there was zero.

A

Conformance test and zero test heads.

A

Welcome to demo. um,.

A

What happened in one 12 was we added the ability to see what tests were hitting.

A

It., so we can say whether they were conformance or not..

A

There's one 11 loading zero, but we didn't know we had 963 end points,.

A

Which was nice. but conformance stuff has to be generally available,, which means no alpha,.

A

No beta. so we're just going to focus on generally available cause. I want a green, safe.

A

Place to grow ideas all over the world.

A

So if we dropped down and we'd go from one 10 to one,, sorry,.

A

One 11 to one 12: you can see. We now have 373 stable end points.

A

And we're at 20% coverage for conformance test.

A

And if we dropped down- and these are um, cii buckets from our,.

A

Uh, used to be a release blocking job.

A

So this is our end: points, uh, coverage going up and I'll go ahead and skip to the most recent stuff so that we.

A

Can see where are we now and where do we go from here.

A

I'm super happy to say that we have a hundred percent coverage for stable.

A

Admission registration and uh, this one over here,.

A

I believe this one is 100% for stable api extensions are areas that we might.

A

Lack a bit in this yellow one is a storage.

A

We don't write a lot of storage tests yet because we don't have a framework in.

A

Place for that., we need to get there at some point.

A

Another one is auto scaling.

A

We don't have a lot of auto scaling because it depends on uh, some,.

A

A non available, the metric, server. uh,, so we,.

A

We can't hit that, but if we go over here, it's for example, to stable apps.

A

I'll just focus on stable apps for a minute., stable apps is at 26 point. yes.

A

It's only 27% covered.

A

We have a lot of ground there and we have specific end points that we should be.

A

Able to hit. um,- and this is part of our workflow- is identifying- who knows best about these.

A

Particular um, operations, and how can we work with them to ensure that there is a well-written.

A

Conformance test, so that that capability is available to everyone in the world.

A

This is the ui for api snoop that we have currently,.

A

But we also have an end cluster version that you can compare what you're running.

A

Against this coverage., so you can make that 100% for your sig and your functionality.

A

I encourage you to.

A

Spend some time looking at our gaps and helping fill them in so the whole world.

A

Can run kubernetes clouds locally and informally.

A

A

Any questions? oh, go for it.

A

That's a very good question. um,, so the definition of how we interact with kubernetes is actually.

A

Define all of our code.

B

To the dragger endpoint api.

B

Swagger, lagger.

B

Operations. there are 1400.

B

B

B

I believe it's important to do both because if we don't get them,.

B

Other people look at them,, we made the mistake of saying,.

B

Well, I forget the exact api that was there,, but it was,.

B

It was pr number four.

C

It's really early and said,. Please add this.

C

And then we adopt the right, a test,.

C

Something that's generally available and untested., and someone said, wait,.

C

That's old news.: we don't need to use it, but just deprecate it.

C

So I went to my deprecation and they said, no, wait, we're using that.

C

And so when we,, once we put things through, that ga process is really hard..

C

So what I believe it was,.

A

It's your,, it's hard., it's like, we don't need the desk,, just the crud..

A

We need the accurate, meaningful task.

C

And so part of what john bell and mark is doing,.

C

We're working together to create this, um,, the behaviors.

C

So what is it that we do with proven eddies and making sure that we do that,.

A

That that's able to be done everywhere., so writing meaningful tests.

A

What we're doing by measuring operations is giving us this base level,.

A

But to help define that what we've done is we've turned the swagger.

A

Great. cool., what do we think coverage is according to this api definition and the ability to.

A

Query the entire set of anything that happened to the api server and come up.

A

With our own definitions of the denominator of what is it, we want to,.

A

What's our goal? and then our numerator is: how far are we along that journey.

A

Um, one of the other ways we looked at measuring coverage as a whole is by doing so.

A

By looking at the fields for each and every one of our kinds, slash object, times.,.

A

And I think when we looked at the full enumeration of that,, it's 77,000 because it gets really deep and recursive and it's,.

A

It's, it's great,, but I'm using both of those as just a baseline of increasing.

A

So once we do fill out those branches,, we just want to make sure we don't start pulling back testing.

A

So what we'd like to do going forwards to kept open right, now,.

A

To take the numerator, the denominator, as we have them defined, and make sure that if somebody submits a.

A

New piece of code to kubernete's that it doesn't decrease our numerator or.

A

Increase the denominator at the same time. as long as everything increases,.

A

We went up into the right. right., any other questions? yep.

A

Excellent question. so I'll repeat that for the mike.: what's the difference between a test and a conformance test and how can api.

A

Snoop tell the difference?, so when I mentioned back in one dot 12 that we started up,.

A

We updated the ede framework to actually submit the full string of the test that.

A

It was currently running as the user agent., so it'd be like your barber,.

A

Your browser saying, hey,, I'm on this version of software, and this is the person coming into all this..

A

It's like cookies, via the user agent data.

A

We also edit a patch to the kubernetes api server, where the audit logging.

A

Included this user agent data..

A

So when we have this database of all of the ongoing interactions, including.

A

Running it live in cluster and being an audit end point, now you can say,.

A

I just want to see the stuff related to this test.

A

And the difference between a test and a conformance test is two. one,.

A

It has little brackets around the word conformance and that there's cii in place.

A

To make sure that can't happen unless it's in a file called conformance dot text.

A

And the test conformance test data folder and the owners on that file is like.

A

Three john, you're, one who are the other two? clayton.

A

Yeah, clayton, michelin, brian. um,.

A

So that's what it means to have conformance cause. They have a uh,.

A

That's what it conformance is versus a conformance test.

A

That's an excellent question.

A

when it actually test that feature or functionality and it gets approval,.

A

What we've tried to do in the conformance working group is make sure that the.

A

People are proving what the test does is by the experts in that field, defining.

A

What should be conformance is for the sig architecture to do so.

A

We connect sig architecture with the experts in all the different sigs and make.

A

Sure that this definition of what kubernetes is, is well defined and tested.

A

So to be a conformance test,, it needs both.

A

An example of a test that would never become a conformance test is something.

A

That tests something cloud providers, specific. uh,.

A

Cause this is,: we are expecting this to run on every single,.

A

It's the requirement., it needs the you to get the badge.

A

and we want everybody to get the badge.. It needs to do what kubernetes says on the box.. It can't do something that,.

A

That, uh,, akgs or gks, or take your all cases, and, and,.

A

But it's gotta be for everybody..

A

So that's why I'm doing this as I want to do the things that make what we do.

A

Together, available for the whole world,, not just some of us.

A

Thank you for your time. uh,. If you want some help, uh,.

A

Running and observing the behavior of your applications and seeing if what.

A

You're doing can be done all over the world, feel free to reach out to me.

A

I'm hippie hacker and I'm with [inaudible].