Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Conformance Testing the Kubernetes API: Tooling that Makes the D... Stephen Heywood & Caleb Woodbine

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Conformance Testing the Kubernetes API: Tooling that Makes the Difference - Stephen Heywood & Caleb Woodbine, ii.nz

The Certified Kubernetes Conformance Program enables organizations to have confidence that their workloads will interoperate from one installation of Kubernetes to the next. In 2018, to ensure that organization can have the assurance that their workloads can move between vendors the CNCF turned the spotlight on improving the conformance test coverage of the Kubernetes API. This effort has brought us from ~20% of endpoints conformance tested in 1.15 to over 75% by 1.22. Most of the success can be attributed to the tooling developed to streamline the process of locating new endpoints to test before creating a PR to the Kubernetes repo. The tooling helps with identifying the current gaps in conformance, exploring endpoints, writing tests and verifying the test results. In this presentation Stephen will walk us through the tooling that will help us test the remaining 25% of the Kubernetes API.

A

Hello and welcome to conformance testing the kubernetes api tooling. That makes the difference. I'm caleb hello,.

B

I'm stephen we're with ii, a group in new zealand, with a focus on cooperative coding, pairing is sharing for us. You can find us at ii.nz.

A

Who is iii, I has made up of the following humans, myself and stephen, who are your speakers? Brenda who makes ii function, hippie who's, the founder of ii, rian who's, the project manager and zack who's, the data wizard? So let's talk about conformance and its tooling. What is kubernetes conformance.

B

A program by the cncf to ensure that kubernetes is the same everywhere with it. You get stable, apis, no vendor lock-in and portability of workloads. When I run my workloads, I expect them to run the same wherever no matter the vendor.

B

The general current conformance information about the program, its vendors and distributions check out cncf dot, io ck.

A

Pulling that makes the difference. We define and verify conformance through tests today, we'll show some tooling that ii uses to help improve kubernetes conformance coverage. The primary tools we'll focus on are api, snip and pedo sharing data.

A

First, we have the api snip suite. The suite is responsible for collecting and processing the data that makes up the definition of conformance, thus allowing community members to generate the data that displays the results for conformance progress and also build out. Tests are three components that we'll walk through.

A

First is snoop db, which is a postgres database, that processes the kubernetes, open api spec and any etowedo log that you wish to make it ingest.

A

It runs as a job or it runs, live in your cluster for querying, and it also statically renders that data, which is then later picked up by web, which we'll talk about soon next is audit logger. This is useful for live test. Writing. It is a shim that sits between the api server and snip db.

A

It captures logs using api server audit sync and audit policy to point to itself, and it's also what we use for our live. Conformance testing development.

A

Finally, we have web, which you might be most familiar with it's available at apisnoop.cncf.io.

A

This is what picks up that statically rendered data and displays it as some very pretty dashboards and uh useful statistic: information for sharing with the community. Next we have peer.sharing.io.

B

Over the last year, iii has built an open source development environment that runs in kurdman kubernetes.

B

We think it's pretty cool to be working inside of a cluster here, given its name is about peer programming with pierre you get in a cluster with any classes, any repos that you wish to work with in our case today, we'll be working with kubernetes and api snoop.

A

Thanks steven now, let's take a look at pay.sharing.io.

A

Here you can see the home page you're greeted with two buttons after you sign in the first one is to list all instances, and also this button to create a new one.

A

I'm going to go to this one to create a new instance which will eventually give me a cluster here, I'm going to input the kubernetes repo, because, while we're writing on we're, writing kubernetes tests and also the cncf api snip repo, so that we have live test writing enhancements, I'm also going to write api snip ticket writing, which is a repo that ii uses for all about ticket writing for for conformance tests. I'm also going to invite steven as a guest so that we can both pair on it normally now I'd hit launch.

A

But since we already have an instance prepared, let's jump to that one going to the instance after clicking this list, you'll be greeted with a page to display a whole lot of information about it, such as this name, the creator, the time when it was created, who it was shared with and various repos that it's loaded up with on the right hand, side of the page you'll see some beneath the metadata you'll see the sites that are available right here. These are ingress endpoints that are exposed in the cluster.

A

There's several ways to connect. You can connect through such things as a coupe config through ssh or even through the web. We have a pre-prepared window up to connect to the pair session right now, so we're going to switch over to that one once in uh you'll be greeted with a a teammate uh session.

A

Teammate is quite similar to tmux if you're already familiar with it, teammate makes those shared terminals uh makes those multiplex terminals shed uh with this cluster. You can see that we're actually running the latest version, which is 1.22.1 as of today.

A

If I run hostname you'll see that this is the hostname for where we are right now, which happens to match this pod, which is the first of this list, so you can see that we're actually working inside of a pod, that's pretty cool.

A

It's called humex, which is an ubuntu base with teammate emacs and a whole lot of utilities that io uses. You can also see that the uh you can also see the ingresses that are available.

A

This list matches the the one for the instance page. Also you have uh dns access to your instance. You can manage any record that is, at your instance name and beneath.

A

What's also special is that you have full tls across all of the your pair host and subdomains.

A

You also have a web server that is running on dub dub dub, with this you're able to to read and write files into the public html folder. This will allow you to expose any files that you wish and yeah we'll look more into this later. It comes in handy, let's take a list of all of the pods that are running right now, as you can see at the top. You have snoop db and audit logo. Those are the components from earlier and we're going to use them again very soon.

A

Why don't you tell us about emacs and orc mode? Stephen? I think that there's some very interesting things that we can do with that.

B

Yes, caleb, there's a ton of things that we can learn from uh emacs and org mode. Here's a basic document that looks a little similar to markdown, but with zmx we got some interesting superpowers that we get with org mode. So inside of codeblocks, not only can we uh document right, we can both document the actual code, but also the results from them. So here we've got the current working directory and here's the current pods.

B

So this lines up with what caleb showed a little bit earlier, the ability to record both the content and the results is uh pretty powerful and it helps us to keep on track with. What's current and the next things on our checklist like moving.

B

B

This document here this is a starting template that we use when we're looking at starting a new issuer around the potential endpoints that, for conformance it's been helped to structure the discussions that we've had with the conformance group and conformance meetings, as well as with sig architecture.

B

It's been very, very helpful down here when we're looking at testing for new endpoints, we can use api snow to ask it. What is currently untested, and by running this code block, we can start to document the process around untested endpoints, as well as, what's the resources around them, these voice resources once we decide that they need to be tested as part of the conformance process, we'll start to look at documenting them and then use the outline here as a way to start to structure how we're going to go about testing this new resource.

B

Once we've got that in place, we would then update this basic go test test that is currently working around pods to match the new resource. So instead of testing and creating a number of pods and listing them before deleting them, we would end up switching this out for another resource. That's going to be part of conformance in the near future.

B

We can look.

B

We can track how the.

B

Odds are interact during this uh tears.

B

B

If we run it, we're going to see the resources being not only started, but also terminated this lines up with the results that we've been given up here,.

B

We can talk with uh and check with how api snoop has seen those recent endpoints been hit, and now I'll see a sequel query here. The sql queries showing the create delete and the listing these endpoints have already been tested.

B

So, unfortunately, this follow-up query will query what is currently going to be a change in test coverage, but at this stage there's only 402 endpoints, showing full coverage at the moment. Caleb. What happens uh once we've got this file done great.

A

Question so we're actually able to export it into different formats to use for different things. For example, we can export it as various formats such as odt, markdown, html or github player markdown, along with the many more that there are I'm going to export it as html.

A

So I can view it on the web now that it's exported I'm going to go to this folder and then copy that html file into the the public html folder from here we're able to navigate back to pair.sharing.io once on pair.sharing.io we're able to go to the dub dub dub sub domain and view any files on it. So I'm going to go to there and then I'm going to go to the the file that I put in there, which is called mock template.html.

A

Now I can view it in reasonably nice rendered html format. This will enable quick review for anybody, no matter what device that they're on uh yeah we we use this occasionally and another thing that we can do. If I jump back to emacs is, if I go to the org file, and then I export it again, I'm going to this time export to github, flavored markdown and then from there I'm going to once again go back to copy it to the public html and then return to payout.

A

Sharing that I owe and go to that sub domain and then go this time to markdown from here. What I'm able to do is I'm able to create a github ticket out of this, which I paste the contents of that into this box.

A

You can see right here that the ticket is rendered and it shows all of the steps of. Why do we need it?

A

What it is, how we achieve it and so forth and proving it from here we're able to bring this up into a discussion at the conformance meetings, if not beforehand, we we would also be committing this and pushing it to a branch for further review, but I wonder what, if there's an existing endpoint, which just needs to be upgraded to conformance and stable? What can you tell me about that? Stephen.

B

uh Once we've got a test, that's actually part of the kubernetes code base, we'd be looking at um discovering where it is in the code base, and I've got an example here and the diamond set. This endpoint here has got a name.

B

A description inside of the name uh quotes here at the top, and with that um set of information, I can.

B

Go back to a awk file, I've got.

B

Org file here that's got the diamond set status and I'm going to run it at the moment and no endpoints have been had at the moment, but based on the.

B

Based on that description, that's in the top there we can go across to.

B

Here and put that information into a variable and then look at running a genko uh focus test using the e2e test, binary.

B

This is testing the diamond set status and, using some watches, to check that various events have happened. Well, once it's passed, I can go back to the query that we've got here on the block and when I rerun it, I can find a number of endpoints have been hid. This helps us to clarify with the conformance group and the appropriate sig.

B

In this case it was sub apps that we've worked with uh with a number of uh diamond set um endpoints and this last release and we're able to confirm that everything's looking great so, okay once I finish all my work, what would I be thinking of doing next.

A

Yeah, of course, so once we're finished doing our work, we we end up, pushing it away and making sure it's committed and saved, and then we make our way to pear.sharing.org once more, and then we can hit this big red button, which ends up deleting our instance.

A

This will uh delete our cluster and uh mean that we'll have a fresh one for the next day.

B

Why is it important to have a fresh instance every day.

A

Because we want to make sure that our development environments are not pets. This will increase automation and knowledge of how to work on the project and will also keep our configurations up to date and in sync and shared with everyone.

A

So that's pear.

A

I would like to give a special thanks to all the conformance contributors who have uh and community who have helped us get to 77 conformance coverage, which is really awesome. I'd also like to thank anyone. Who's contributed to make any endpoints stable or generally available.

B

Here are some useful links that you may wish to check out. uh The top three are reloading: they around the conformance testing program, uh ways to help out and also what's required, to make sure that your cluster is certified for a particular release of kubernetes.

B

The last one further enhances the discussion around testing and how to find endpoints and just clarifying the process that we've shown in part here today.

A

Next, if you'd like to get in contact, please reach us via the kubernetes slack on the kate's conformance channel. That's where you'll find the kubernetes conformance working group there's also the google group and email that you can send emails to which is on screen right. There.

B

We'd like to give a special shout out to a number of open source projects that have helped not only the uh test writing process become a lot easier, but also uh for presenting this talk. Reveal.Js is really awesome and we're able to drive it through org mode.

A

Thank you, those who attended live and those who have decided to watch later. We hope to hear from you and we hope that this talk has helped you. Let's move on to some live q, a.