Kubernetes Kubernetes Contributor Summit - NA (San Diego) 2019, 26 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Contributor Summit NA 2019: TestGrid and You

Description

Event link: https://events19.linuxfoundation.org/events/kubernetes-contributor-summit-north-america-2019/
Session link: https://sched.co/VvNY

A

Okay. everyone here for the testgrid and you talk, just to check.

B

If you're interested in the new contributor one,, the rooms got switched and you want to be over there.

B

B

Uh, yeah. all right. yeah, sorry,. I got started late,.

A

But welcome. um, this assessor to you. uh,, I am michelle.. This is shawn.

A

We both work at google., we're on sig testing and we work on test grit. um,.

A

So welcome. and I hope, you're a contributor.. Someone has been good, so far. yeah.

A

All right, so just to get started, what is test grid? um,.

A

Test grade is a highly configurable interactive dashboard for viewing your test.

A

Results in a grid., it is pretty self descriptive. um, or at least I think, so.

A

But uh, he want to get a look at the code and some of the guides we have. um,.

A

It is on github.com google cloud platform, slash test, grid. uh,.

A

We used to have it in test infra,, but it's now been moved into its shiny new home, where it can grow all to itself.

A

And not interfere with other things, intestine from um,.

A

And internet permitting. we,, I will also show a little bit of how it works.

A

Okay, cool. yeah,, so basic overview., I'm going to go fast,.

A

But this is the dashboard index..

A

This basically shows off a bunch of the organization..

A

You can see that this has the test results for a lot of the sigs, um,.

A

Including our very own sig testing down here. um, cic testing has a dashboard group for all of its stuff with a lot of dashboards.

A

Because we run tests on lots of parts of tests in fra., um,.

A

Similarly, there's a lot of other uh, sigs that also have the results. Here. um,.

A

Or some other projects like some stuff under google,, red hat,, um,.

A

And more, if you also want to join. um,, but for now I'm going to go, find a particular,.

A

Let me go find a particular thing that I was looking at.

A

Handy given the number of dashboards there on test grid.

A

All right. um,, so let's say that I was looking at this and I want to debug..

A

I can see that there's a lot of, uh, passes and failures happening here that I kind of want to look into a little bit.

A

More. um,- maybe I'm used to looking at this by alphabetically, so I can sort by name.

A

Um,- and you can kind of see like there is a lot of results here., so I decided,.

A

Eh,, let's go back.

A

mmm. we'll go ahead and do by failures: again,, which is default.

A

Mmm. we'll go ahead and crunch it, so I can see results of for more days. um,.

A

So now we can see like last couple: days, um,- and maybe I want to see if this happening a lot. turns out. Like does look like it's been kind of a pattern, so it doesn't look.

A

Like anything, that's cropped up. Super recently., there's likely just some flaky tests that are causing these kinds of results or.

A

Maybe something with the environment. um,, but let's say I want to look at seeing if maybe some of these are correlated.

A

With particular functions, um, or tags., we can see that these tests are tagged with, like sig, apa, machinery,, six storage,, six scheduling, et cetera., I'm going to hit group,.

A

Which is mass. um,, and this one is basically just, uh,.

A

Once you apply the red x,, anything that has the same string will be grouped together under one row.

A

Alright., so we can see now that it's grouped with all of the things with the same tag.

A

Um,- and we can kind of see that there's like a lot of these- are doing fine.

A

Most of these are due to some of the tests. under some of these specifically,.

A

We can expand out to see exactly which ones six ela.

A

It turns out, it does have a lot of tests.

A

A

and if I want,, I can even see which ones have been only failing or sorry,.

A

Which ones have had at least one failing result for the entire thing.

A

My bad on that.

A

So, yeah. um,. This basically shows off a lot of like how this might be useful for debugging,.

A

But there's also a lot of other features.. So when we say highly configurable and interactive,, there's a lot of stuff that we have brought to external. And we are also hoping to do more as we continue to open source..

A

Mmm. and I'll go ahead and hand to sean. yeah.

B

Can everyone hear me?, okay, perfect.

B

so clearly this is incredible and amazing and everyone wants to use it.

B

So how do you do that? well,? We have a,.

B

We have a read me, but effectively what you have,.

B

What we have is a yammel configuration that describes. I want these dashboards,.

B

I want these dashboard groups in these dashboard tabs and I'm going to point to.

B

This j unit information,, which shows what my logs are.

B

So you just say: like,. I want this test and here's where the logs are.

B

Please draw the pretty graph.: can you turn that off? getting feedback,, I think.

B

Okay. um, that,.

B

Yeah,, that's all you need., a yammel configuration and j unit logs somewhere.

B

Usually in cloud storage- and you can have the pretty graph,.

B

But if you are using prow,, which is an open source, cict system, that the entire kubernetes community uses,.

B

You need neither of those things because prabal do it for you.

B

Test grid can read prow annotations with a particular program.

B

We have called configurator. so instead of specifying. I want this dashboard and this tab and this job,.

B

You just write directly on the job.. This job should have tests,.

B

Put it on this dashboard, email, this person, when it breaks.

B

And it's very quick and very easy, and it's right next to the job. in addition,.

B

You don't have to worry about where the logs actually are because of the pod.

B

Utility that goes with your little test with prowl automatically uploads all of.

B

Your logs to gcs or yeah.

B

B

And then, once this uploaded,, you have your results and so proud does both of those things for you.

B

In addition,, if you're, using an instance of proud that, isn't the kubernetes instance of proud,, we have another utility called transfigure that lets you take the configuration.

B

Of your instance of prow and pass it along so that it can be used on that pretty.

B

Dashboard that we had earlier.

B

So that's configuring, it,, that's getting it up and running.

B

Perhaps you're more interested in how it works., maybe you want to help contribute, thank you.

B

But it works primarily through the magic of protocol buffers.

B

Protocol buffers are a serialized data, format.

B

They're, like xml or jason, they're, not self descriptive.,.

B

They need to have something that goes with it. That describes how it works.

B

So if you look into our repository, you'll see files that indian dot prodo that describes all of the data structures.

B

We use and we use these data structures as an intermediate layer.

B

So we have some component that creates some data and puts it into a protocol.

B

Buffer in cloud storage. and then another component picks that up and kind of passes it along down the.

B

Chain., and by using it as an integration layer like this,,.

B

You can kind of keep things separate.

B

Let me show you what I mean.. We have cloud storage,.

B

It holds data in protos.

B

We have modules which are asynchronously reading and writing these products.

B

The contract files or the output of configurator,.

B

If you're using prow gets read in from get hub.

B

Then that goes into cloud storage.

B

Then another piece takes it from cloud storage and actually goes and gets thej.

B

Unit information, so it gets the actual results,. What passed,.

B

What succeeded historical data,, that kind of thing., and then it writes it back to cloud storage and a different prodo.

B

This one's the state, prodo and then the front end reads the state, prodo and.

B

Actually displays the pretty graph.

B

and these three pieces are all working continuously and asynchronously, so they.

B

Don't have to wait for each other necessarily.

B

And that gives you the kind of availability that we saw.

B

We ranted on convention wifi and it didn't break.. It was great.

B

A

Yeah,, so that's the core thing is basically taking like config, updating,.

A

Storing, and then displaying on the front end., but we have a bunch of other modules that do some of the other things around test grid as well.. So for instance, we have the summarizer, um,.

A

Which takes the state dot prodo all of the, uh,.

A

Updated test results from your tests and then writes a summary prodo. um,.

A

And you can see that it produces, uh, things that we were entering the front end like here., so it has a list of the tests that are failing..

A

It has how well the dashboard isn't doing with recent runs. um,, whether it's like continuous failures or like some spotty failures or completely.

A

Green. um,, and it has a couple of other things like when did it last update,.

A

When the test last run.

B

And is there a latest like run that was completely green in a column.

A

Um,, we have a few other ones as well., so alerter for instance,.

A

We'll take the summary prodo and send you email alerts whenever new failures.

A

Occur. um, an entomologist will grab, uh,.

A

Issues that are related to failing tests and it can associate those on the.

A

Table. um,, so that you can see links to bugs.

A

Um,, theoretically, you could also do other things given the protos here..

A

So let's say you wanted to say,. Instead of just emailing for alerts,.

A

Auto open an issue that has some information about failing, tests., um,.

A

One thing you could do is take the summary prodo,, which has a list of the failing tests for each dashboard, tab. um,.

A

Use the github api to open an issue against like a configured repository for.

A

Instance. um, and then update something called issue: state dot., provo,.

A

Where we store all of the tests, uh, bugs and how they are linked to test grid.

A

So, um,, with that you get a, your failing, tests,.

A

You get information from what we already gathered in summary, on failing tests.

A

That you could add to an issue and that gets linked back to tests and audit or.

A

End test grid., pretty sweet., um, yeah,, so that's a basic overview of how it works..

A

We have some more docs on the, uh,, repost wall,, which will be continually updating, um,.

A

In terms of what we're doing in the immediate future and then into next year.,.

A

Right now, a lot of what we're doing is because this was an internal google.

A

Project for so long, there's some things that, like kind of diverge between the internal and external.

A

Versions that we are working on cleaning, up., um,.

A

And sean has also been working a lot to make this a lot easier to get onboarded.

A

With so, you can use it, um, with your test. Results.

A

You can use it from different repos and not just like, uh,, putting everything manually, into, um,, that keith's repo, uh, things like that.

A

Um, in the next year, where we're basically charging ahead on continuing to open.

A

Source,, depending on like shifting priorities as we find that things are going to be.

A

More impactful. um,, but we also have a issue. That's open in google cloud platform test grid issue.

A

51. um,, and that has just things that you want to list,.

A

Like use cases that you would find useful., you want to use test grid, for,, um,,.

A

Your own thing., you want to see some new cool stuff, feel free to list it there.

A

Um,, if anything, you remember from last year,, there was an issue open in test in fro that has similar things.

A

I've linked that issue. feel free to like bump up some of your stuff. If you find that, like it's still relevant. um, as we continue working and yeah,.

A

If you want to help thanks a bunch in advance., um,,.

A

There's a bunch of ways that you can help us and they're, not all just trying to build a auto bug for instance., so I'm just using test grade is really cool.

A

Uh, and then letting us know how it works. um, or if you need any improvements,.

A

Um,, you cannot get test cards set, up,, uh, from other repos and then have those,.

A

Uh, put into the kate's instance, um, until we get a more open source and more instances that you can run on. so,.

A

Um, feel free to help set up test grid. um,.

A

He find things that are like wrong or missing, or you have like some cool.

A

Features that you think would be a austin add test, grid.

A

feel free to file issues against us in the new repo.. We will, uh, take a look at that and use that to help prioritize our work as well. um,.

A

And feel free to add code, uh,, the reposts kind of in a work in progress state as we are getting started.

A

Open-Sourcing it. but, um, definitely feel free to like run a quick design bias in the sig testing channel.

A

And uh,, we can help you with how it's best to go forward on that. um,.

A

Especially if we can help prioritize like getting this piece of code, out,.

A

We'll definitely make it much easier for you to develop that and we think it'd be really awesome. see. Ya. thanks, everyone. um,.

A

Left a link to the repo and casey,, you want to say it: again? um,.

A

And sean, and I as well as other people who know about test grade, are hanging out.

A

In sec, testing channel usually., so yeah,.

A

If you have any questions, feel free to ask.

A

Um,, in which case [inaudible].

B

Yeah,, if there aren't any questions, we'll be hanging, out.

B

If you come up with something or want to talk about test, grid, uh,.

B

Other than that, that's, that's, yeah.

B

A

We have some stickers at the front as well. If you want some for tusker pro and.

A

Uh, sean,: do you want to talk about those sick, testing, intro, talk as well for one, is that happening?.

B

If I spoke about prow and you don't know what pro is or have questions about.

B

Prow that talk is on wednesday at 10 35, um,.

B

Where I will go over proud and that cic d system.

B

And then maybe that slide will make a little bit more sense.

A

Um,, I think it kinda depends on exactly what you want to do. uh,.

A

Like test grade and proud play nicely together., so he wants to do something: like,, uh, sean,, your first work was kind of making those play together. Better.

A

And like katherine,, you've also done stuff on like pro annotations starting at all..

A

So it's a lot easier to configure it with proud. um,. So things like that,.

A

Like knowing proud does help definitely., but you could also get started with like, um,.

A

You're, a user of test grid- and you see some stuff that, like you, think tusker.

A

Could do better. like for instance, um,? Another thing I know if we go back to summaries is like this is some amount of.

A

Info, but like there's, definitely more that you could collect. um,.

A

Especially if it's like expensive to create and store.

A

Cause that's the main like benefit that we have as storing things asynchronously.

A

Is like you can grab it very quickly and you can do like as long as you want on.

A

The back end. um, like gathering those results., so you want like a super summary with like something over the entire dashboard,.

A

Complicated statistics, or something like that could work.

B

So, um, test grade is very interesting in that it helps to visualize tests,.

B

But just test grid does not deal with actually executing your tests.

B

It reads your test results and tries to make sense of them.

B

Usually, for use cases where you have insane amounts of test results like.

B

Kubernetes, there's a lot of data for kubernetes.

B

Those tests are run with prow, but whatever a cic d solution, you use to run your.

B

Tests is independent of test grid..

B

So long as your results, wind up in a j unit format, test grid will just read.

B

A

I know for instance,, there are some people who host right, train it for matt and I believe, use jenkins.

A

To do so. so yeah, exactly., so the test screen won't handle running it,.

A

But it should handle like if it's in that format it works.

A

A

All right, thanks. I'll, um, and feel free to come find us later. If you have any other questions or as noted,.

A

Like where we have some docs on the repo, we'll be working on getting those more detailed as well with some of the stuff about the architecture and whatnot.

A