Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Contributing to Kubernetes Conformance Coverage - Zach Mandeville & Caleb Woodbine, ii.coop

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Contributing to Kubernetes Conformance Coverage - Zach Mandeville & Caleb Woodbine, ii.coop - https://ii.coop/

In this session we'll walk through the Certified Kubernetes program (cncf.io/ck) followed by a deep-dive into the tooling developed for identification, removal and prevention of gaps in certification test coverage.In the intro, we will cover the steps required to become a CNCF Certified Kubernetes provider.We will also deep dive into the APISnoop technologies supporting conformance, specifically:- Identifying Gaps (apisnoop.cncf.io) Closing Gaps (apisnoop audit sink + test writing w/ humacs.org) Preventing Gaps (release blocking w/ prow.k8s.io) Verifying cloud provider submissions (prow.cncf.io)

A

Hi and welcome to contributing to kubernetes conformance coverage, I'm zach and I'm caleb and we are from ii, and today we are going to be talking about what is conformance, why it's valuable and the tools that we can use to help achieve it.

A

So to start a bit about ii, we are a group of technical folk here in aotearoa, new zealand, with a focus on cooperative coding. We believe that pairing is sharing and we pair in everything we do, including these presentations, and you can find out more about us at ii.coop.

A

The people that make up iii uh we're a small of a powerful bunch consisting of hippie hacker, who is the founder and sort of vision of ii, myself and caleb, who you have met as well as stephen, who is our resident test writer rhian? Who is our project manager and helps us, keep us on track and focused brno? Who does a lot of the automation and proud jobs that we, some of which we'll be seeing later on and brenda who ensures that all of us can actually function?

A

And now, let's talk about kubernetes conformance? What is conformance.

B

Yeah kubernetes conformance is a program that enables every vendor to support the required apis, as do the open source community versions. It's valuable to have consistency across the platform wherever it's run,.

A

And why is conformance important good question.

B

It enables portability of workloads, freedom from vendor lock-in and stable apis. It also allows consistency across providers really. I expect my workloads to run the same, regardless of where which, whichever vendor it is or wherever it is.

A

There's a good background and rationale for conformance at cncf dot. Io ck here you'll also see some nicely detailed steps on how vendors can be certified as kate's conformant. Right now there are about 67 certified distributions, the full list you can see at landscape.cncf.io and then click on the certified case link on the left. But this list is great. That means that you can have consistent unsurprising fully conforming behavior across a breadth of providers.

A

These vendors are certified and added to the list that we just saw through a transparent and open process on the cades conformance repo on github. We will cover this process in the presentation shortly.

A

This process is enabled or is able to happen because in kubernetes the conformance is defined through the api and a test suite that allows for tools to be built that fit in within existing kubernetes workflows. Two great examples of that are sono buoy and api snoop.

B

How do I certify my distribution, I use sonaboy sonoboy is a command line utility that interfaces with your cluster. It deploys conformance suites that allow you to generate data off of it to submit in your pr.

B

You can install it through the first command and then run a generic suite against the your your cluster with the second command.

A

Soda buoy will then run the full e to e test suite on your cluster and, as it's running, you can invoke sono buoy logs to see what is happening. What you'll see is the tests that are that are running at that time and their status and a nice big wall of text.

A

You can also invoke sono buoy status, which will give you a good summarized result that will show you when suddenly is complete and whether you passed here in this example, while we did pass that are complete you'll see that the test count was one, so we skipped a whole bunch of ede tests in our results.

A

This will become important later. If we try to submit this pr.

B

At the end of the run, you'll end up with several files these these represent the complete results of the test suite. You must include these in your pr decades, conformance now that you have the results, let's go and fork the cate's conformance repo and then we'll add the files in the pr.

A

And while the complete instructions for how to do this, you can find on that. Repo.

A

Here is a good example that just shows that a pr was open for talus from their own to the cncf case, conformance repo and their pr is going to include the junit and ede.log files coming from that sono buoy job, along with a readme showing how to reproduce the results and a product.yaml, giving more information about talus.

B

Once the pr has merged you've just got yourself a certified kubernetes distribution and.

A

Not only that you have a sweet logo, I like it and everything it represents, so that was talking about verifying the conformance coverage for a vendor. Now we can talk about improving conformance coverage and a great tool for that is api snoop.

A

So the reason behind this for conformers to have value and to have meaning, then the api must be reliable and consistent and a way to ensure that consistency is through conformance tests.

A

Aps noob is intended to help with all aspects of test, writing and test coverage for kubernetes, from identifying the gaps in coverage to closing those gaps with tests and preventing new gaps from happening.

A

Api snoop is powered by a database that we call snoop db. This is simply a postgres database that is filled with the api schema from kubernetes swagger json, as well as audit events from the ci jobs running the full ede suites on clusters.

A

You can also deploy api snoop to your own cluster, at which point it will log the live audit events as they happen. This can become useful for test writing to get immediate results to ensure that the tests are hitting what you expect.

A

So, let's start by talking about identifying gaps and for that we can use apisnoop.cncf.io on this website. The data in snoop db, showing the current coverage, is visualized as an explorable graph. So we can zoom in to the section of the api of kubernetes that we are interested in. For example, we wanted to see stable and core this api is then color coded. The gray area is showing the end points that are untested.

A

The light area are showing those that are tested, but not conformance tested, and the solid colored area is showing those that are with conformance tests and then there's a nice little summary here on the side. So right now, there's almost 60 coverage on the stable core part of kubernetes, as you zoom into the various sections, you'll also get a shareable link, so you can give this to others to help a conversation or use it as a quick way to return to the part of kubernetes you're interested in along with that.

A

We have a nice conformance progress page here that is showing the history of the conformance coverage.

A

Here we can see the increase in the kubernetes api and the total number of endpoints, as well as that increase in coverage with a cool increased velocity happening, which is great to see now going back to the presentation after identifying, we want to close these gaps with tests and we can do that with test writing for here.

A

uh Let's go ahead and do a demo of how iii writes tests and our our flow for going through it for to start with.

A

When we are writing tests, we start at sharing.io here we can create clusters on the fly and deploy them with applications um to get started, so they are configured to be built to make pairing really easy set up with teammate and and so on. So we can share a session easily and in this case here is a cluster that we had set up beforehand. That is already has api snoop and our ticket writing repo deployed to it from here. We can then log into that one of the pods with just a simple ssh command.

A

Now, let's go ahead and check in on where we've logged in- and here we'll do a pairing session, so kayla will be the driver I'll be the navigator, and what we're seeing is uh our text editor, which is a highly customized version of emacs and an org file within it? That is showing the flow for test writing.

A

Basically, what we do is we create a mock ticket beforehand or a mock test beforehand. I should say where we identify something identify an endpoint. We want to test, write it a version that could maybe hit it verify that it does and then submit that as a ticket. So we can get early feedback on on how it's written based on that feedback, we can then convert it into an actual ede test which we would submit as a pr let's go ahead and see that in a bit more detail.

A

First, we would want to identify an untested feature and we can do that with api snoop here as the database that is connected to this document and we're looking at all the untested, stable endpoints or at least 25 of them.

A

Let's go ahead and filter that even further and let's say we want to just look at ones that are part of the deployment kind.

A

So here is one untested endpoint that could maybe be good to write something for, and maybe we want a bit more detail about it like it's path. We'll just add that, to the query run it again, and now we have the endpoint and its path, which will help us later.

A

We can document it as much as needed following the api reference and then go ahead and write a mock test for it. Now, in the interest of time, we won't write a brand new test. Instead, we'll go through one that we've already written that has actually already been merged. As a ede test, the start of it, we would create an outline showing our intent in this case, we're going to do a life cycle test going through the creation, the patching, the getting the listing of a deployment and then the deletion.

A

So here is the function itself, the mock test, written into the document, and we can actually run.

B

A

And as we run it, if we switch over to run our coop cuddle, we can see that it is firing on our cluster going through what we expected to do.

A

But it is also going to be summarizing the results back into the document, so we can verify that it did what we intended.

A

Now we were trying to target specific endpoints and we want to see if it did do that which we can do again using api snoop. So because this is connected, uh apisbeep is connected to the cluster and is grabbing all the audit events, as they happen. The test that we just wrote had a user agent attached to it for live test. Writing we can filter to that user agent. So we can only see the endpoints hit by this test, which we'll do in this query.

A

There we can see that we hit a number of deployment endpoints as we intended and we can get the projected change in coverage.

A

This will get us a nice point value to see the number of new endpoints we'd hit if this test were to be merged, but because we're using an existing test, it's showing that it's there would be no change in number as we'd expect, but when this was completed, we would then export this entire document into github flavored markdown, which we can then copy and open up as a ticket in the kubernetes repo, and actually we can go ahead and switch back over to the browser, because this entire flow that you saw was being used to create this ticket that we see here and, as you see, the org file we moved through became the body of this github ticket.

A

B

Ticket was then.

A

Converted or used as a basis for a pr for an actual ede test that was then merged and is a part of the conformance suite.

A

Thank you for that.

A

So that was our demo for our test training. Our one way to do test writing once we are starting to close those gaps. We also want to prevent them from happening absolutely.

B

So yeah we can, we can prevent gaps as well as fill them in order to prevent them. We uh we need to establish a healthy baseline. What that means is any endpoint that's promoted to ga must have a conformance test. All certified clusters will ex will be expected to pass all conformance tests.

B

The process is very automated, and so it should be easy to follow.

B

On testgrid.catester io. We see that the jobs that help automate the process of the kids community. This includes testing. Here we see two jobs, conformance audit and conformance scade. These are both owned by sig arch.

A

So in our earlier example, with sono buoy, we ran an incomplete test suite now. If we were to submit that as a pr, including those results, we'd want to quickly know if those results were actually the entire suite.

A

Otherwise it would not be be valid.

A

We can automate that auditing of the results with the conformance audit job, and so what it would do is let's say that we did submit it as a pr which we'll see here when we submitted it, the job would run and then through a friendly cncfci bot would tell us that this conformance request did not include all of the required tests for that version of kubernetes, and it gives us the first test that it could find that did not pass or was not included to help us narrow it down.

A

It also adds a label of required tests missing, so this will make it easier to filter out or find quickly and you'll see. That is a part of a number of labels, each of them coming from different processes or jobs all intended to make it easier to manage and track the various amount of activity happening through the the various github repos.

B

The next is our conformance gate. This uses api snip under the hood it alerts when endpoints are promoted to ga without conformance tests.

B

There is a description shown in the yaml right here and, ultimately, it's intended to be a release blocking job the job manifest has email for alerts. When alerted, we will be able to tell sig release of an untested endpoint. This endpoint will need a test or it will be not included in the next release.

A

So, in summary, to improve the kubernetes coverage, we want to identify any gaps using api, snoop and snoop db, close those gaps through the writing and promotion of tests and prevent new ones from forming with release blocking jobs, as we said, conformance ensures that every vendor's version of kubernetes supports the required apis, as do the open source community versions.

A

There is a strong ecosystem, around conformance, supporting all levels of involvement, with kubernetes from vendors to users to developers. Thanks for being a part of this presentation, should we go with the q, a? I think we shall see you there.