Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Who What How: Understanding Kubernetes Development through DevStats - Josh Berkus & Łukasz Gryglicki

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)

Who What How: Understanding Kubernetes Development through DevStats - Josh Berkus, Red Hat Inc. & Łukasz Gryglicki, CNCF (Any Skill Level)

Did you know that Kubernetes has its own analytics gold mine about who contributed, when they did it, and how? DevStats.k8s.io offers a data smorgasbord which slices up Kubernetes contribution activities. This isn't jus for data geeks, though: SIG leads, team managers, project and community managers, and even regular contributors can derive useful insights about how development is doing and where they might need to pitch in through the stats. In this talk, we will cover the charts and data offered by DevStats with an eye towards finding insights for you to take action on. We'll also discuss how you can request additional charts or develop them yourself.

About Josh
Josh Berkus works with the Kubernetes community on behalf of Red Hat Inc. He contributes to sig-release and sig-contribex primarily, and organizes events. In his spare time, he likes to deploy databases, particularly PostgreSQL, on top of Kubernetes and OpenShift. Josh lives in Portland, OR.

About Łukasz
Lukas is a software developer with CNCF. He is the primary developer behind DevStats.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Welcome everybody to the dev stat session that too loud, that's good, okay! Well, maybe the deficit session. We are going to be talking about the system that is watching all of your development activity in kubernetes. How many people here are kubernetes contributors, yeah I got a whole bunch, so the so you're going to see your own stats here, although mostly not individually, the these are mostly aggregate stats.

A

A

We're gonna have a bunch of people talking and exploring various parts of kubernetes development through statistics, I'm josh burkas Lucas is here who's that lead developer in dev stats, and we have LC Philips and Arin, quicken Berger and Tim pepper, who are going to be showing you different, charts and graphs that they use, perhaps more importantly than people's company affiliations, is their Sigma filiation z', because that's the whole reason why we need these charts is in order to answer questions that the various SIG's and other groups have, for example, one of the things that I and several other people are involved with one of them's that that LC co-leads is contributor experience and they need to monitor community health I'm.

A

Looking at how are we doing in terms of people contributing to kubernetes and how the results aren't everything else, I'm on the release team, the release team needs to try to anticipate blockers before they delay a release by looking at development stats. The steering committee wants to look at some of the stats about how developers are doing things in order to get an idea how to get an idea. How policy changes and changes to the way that we do things would affect our developers and the folks at the CN CF one to advertise?

A

How well kubernetes is doing and how big and how fast it is moving as a project. So we have all of these purposes for which we want developer stats and some of the questions that we want to answer our questions like, for example, who is contributing to kubernetes like what companies are they from what regions are they from what types of contributions are different people making and that sort of thing who's doing the most code.

A

Reviews is another question that comes up, particularly because we do shoutouts and other things to people who are doing code reviews because that's really important, which SIG's have the most issues which employers are contributing, because you know companies who join the CNC, F and start contributing kubernetes make big noises about stuff, and some of them contribute a whole lot to kubernetes and some of them, maybe don't so, and then we also want to answer some.

A

What questions some content questions like what's going on right now and development activity, or what is the label that we've got a whole bunch of PRS hung on or how many issues are being filed against each repository, and all of this is time-based as well. So we didn't want to answer other questions like for community health.

A

We want to look at hey when doesn't issue first get a response from a real person on average, or how long does it take to merge a PR or when our busy periods as opposed to our slow periods and do those match up with our expectations?

A

Now, if you've used github from other projects and that sort of thing you know, github has stats right, there's some stats available and github there's a few more available through the API and that sort of thing and they've got. You know some stuff Ohia. This many commits suddenly merge this many pull requests and that sort of thing and that's more or less we're using it first and you still see some people reporting the github stats, but these are really not all that useful for a variety of reasons.

A

The first being that there aren't a lot of stats available period right we've got commits, maybe issues. We've got this one stack called traffic, that's kind of a grab bag for everything that anybody does.

A

We can't report in things based on labels and we use labeling of PRS and issues extensively to categorize them I in terms of contributor activity, in terms of like picking out individuals, github will only show us the top contributors and we can't actually get out all the contributor to characterize them and then the time scales with the default github stats are just really not very configurable, as in you can get like a week or a month back for a limited amount of time, and that's about it.

A

So really not enough data for us to answer any of the questions that we want to answer further kubernetes, as a project makes this even more difficult, because we have some peculiarities combined compared to like a small team github project, for example, most of our traffic on all of our reposes from bots. But we don't want to count that, because it's automated traffic, everything gets merged and stuff doesn't get committed directly to the repo. So this is not supposed to be the end-all.

A

So we have this weird thing where we need to aggregate groups of repos and bind repositories together in order to see them in a way that represents the different sort of development areas of the project.

A

So last year the cloud native computing foundation funded a project I hired Lucas to work on it started July of last year in order to develop a platform for looking at a much richer and more in-depth set of stats and development activity on github and presenting it in better visualizations. Now this is still a work in progress.

A

We are constantly changing around and adding new graphs and consolidating graphs and renaming things be as we find out, which things are actually useful, but we're going to show you a few right now that people have used in their particular SIG's and in their in their work as examples of where you can use these developer stats to see things in the kubernetes project. So, first of all, LC Phillips from contribute experience to talk about PR time to engagement.

B

Okay, test great alright, so this first graph that we're going to talk about is pier time to engagement and what this essentially is is how long does it take for somebody who isn't the author of the PR?

B

How long does it take for that PR to get any engagement, and so we can use this graph in a couple of different ways. So right now we're looking at the across all of the repositories across all time as well, and so we can actually drill this down to specific repository, specific repository groups and why this graph is useful.

B

Is it can tell you how responsive people are to the poor request being made in individual repositories, and so we can go down here and we can see the breakdown the 85th percentile the median, and if we scroll down a little bit Josh, could you scroll down for me the 15th percentile as well, and so we can see across all time these stats over here. So we can see for the 15th percentile.

B

Our average time is just a little under an hour, which I think is actually pretty great, and so we can see that over time how responsive were being ok well,.

A

Thank you. Actually, let me stick to mine net double it. Thank you very much and by the way, if any of you actually have laptops out- and you want to follow along, this is all live, and so you can't actually follow along. What you want to do is go to KS, dev stats, that's C and CF that IO sorry I, don't have any way to enlarge the URL there, but look up kubernetes dev stats is the other way is to search on them, and then you scroll down to the list of charts.

A

We are still in the process of cleaning up this list of charts in terms of having very informative names for things and consolidating some charts and that sort of thing, but we've done a lot of that already and so then you can actually look at some of these and manipulate the parameters of them yourself, including time scales, repositories and other things.

A

So to show you another useful chart.

A

Aaron from cig testing is going to talk about bots commands, hi.

C

I'm Aaron Crick and burgers so yeah. Although steering committee was next, my name I'm here more in a capacity as a cig testing lead.

C

So this graph shows all of the usages of slash bot commands that people can use to interact with our automation throughout the project. I gave a talk earlier this morning talking about how we enable like people who aren't members of the project to add labels to help triage issues. People who are members of the project can be assigned to can /lg TM to signify that they've reviewed a PR, approved it so on and so forth.

C

So I wanted to show a couple things that this graph helps helps me learn in terms of our people actually using our stuff. Are we actually doing a good job of communicating this stuff, so here I'm looking at every single bot can't command ever and it's sorted by most usage? So we can tell that most people, instead of like using github drop-down box to assign people like just typing, slash a sign and then a person's name. It's by far the most popular command.

C

The next one CC is for a requesting review from people, so it's mainly about like delegating between people right.

C

The next most interesting thing to me is we issued a command called retest or we instituted a command called retest about the same time last year, where, if any of the 12 tests that runs against your pull, request failed for whatever reason, and he wanted to rerun those tests to see if it was a true failure or just like an environmental flake.

C

Instead of selecting each individual test, you could just hit slash retest and it would run all the tests that had failed. Initially, there's this really large spike because it turned out to be hugely useful and very popular over time. It's gone down because we now have a bot that looks for pull requests that have been approved and reviewed by humans, so they have the appropriate labels, they're, just not passing tests, and so we can have the bot spam slash retest.

C

It's also a really good proxy for like what's the relative stability of the project over time. As it goes up, that's probably a sign that there's either more traffic or that our tests are flaky, and we should do something about that.

C

Finally, these little dashed red lines here, maybe it's tough to see indicate when a release of kubernetes was cut and if you look at these huge spikes before these are times when people issued the slash approved command. So slash approve is what somebody who has, like the final say on code uses to say. Yes, this code is valid. It's doing the right thing in the right way and you can kind of tell when code freezes happening throughout our release process, because everybody's rushing to get those approvals in before code freeze happens.

C

It's kind of telling me also that there's a lull afterwards, you can see here the nice cube con dip from last year, where it'd be great, if we could find a way to make our release process less sort of herky-jerky a little smoother instead of having people constantly rushed for this, but anyway, those are all sorts of insights. I can view- and this is the sort of stuff that I can't get by looking at git I- can't get it by looking at github.

C

We really needed some specialized scraping of github comments to see how people are interacting with our automation,.

C

A

A

Shrink this a little bit okay! So next up we're going to have Tim pepper with issues open closed by saying you know, this is from your work on the release. Team, I.

D

A

D

So there's the the shareable Earl I, don't expect any of you to copy that down. But if you're looking at the issues open and closed, specifically, you can, if you're looking at the issues open and closed by sig, you can tunnel in specifically to feature type and this this kind of follows on to what what Aaron was showing. So even though sig VMware was next to my name, I'm, actually here thinking more with a sig release and also a bit of contributor experience perspective.

D

So the the red lines are a little easier to see. Maybe here that these are the the release, points and Aaron was talking about the phases where code freeze happens, but I'm curious about when feature freeze happen, so issues themselves fall into two course: categories, there's bugs and there's features and we're talking on the release team about shortening the period of feature freeze to allow more smooth development and maybe get some some of the less herky-jerky that Aaron mentioned. But I was curious. Well, what's our volume of feature open and close?

D

Does it have a normal sort of s-curve sort of feel or what? And- and this is what I see in the the the red arrows that are sort of hand drawn on there are the period are the point at time of the feature freeze, so that got me wondering well, what is this other stuff is? Is it late features or people kind of gaming?

D

The system are we not noticing what's going on so then I went to looking instead of at the seven day, moving average just to the day perspective and comparing just the opens, and the close, you actually see that more or less we kind of have a steady state. People are kind of always doing feature development and to me this makes sense. This is open source like we, we get an idea. We work on it now.

D

The the big question that I'd like to see and I to open a PR against Lucas on this to you to get an enhancement is in the middle of this. Could we overlay the actual label or milestone, and do we see that these spikes are people getting ideas that are going to be in the next release, or are they late things in the current release and then, with that little bit of additional information?

D

I think we could do a little better risk management on the release team and understand whether we're missing something here or whether it is actually safe to shrink the window down some so I was just something. I was looking at.

A

A

Okay and the last one is mine, and this is from I've, been on the release team for several cycles. This one was actually one that I developed that well I developed by submitting an issue to Lucas I around 1.9, because I was tracking the various PRS for 11.9, and there were a lot of PRS that were not receiving attention from sig note, specifically, and so I was like.

A

Okay, so is, is this that the signaled leads are just a wall or is it that sig node is getting a whole ton of stuff thrown at them and they can't keep up, and for that matter, could we detect this in the future before it becomes a crisis during code? Freeze time, you know to say who is heavily loaded now? This is this in a couple of times.

A

This is, you know, peer workload and push sig as a graph, and you can look at sort of the general graph and let's not do last six months here, let's just say the last 30 days, and so you can actually the sort of a general graph and you can kind of see what's going on in general, because you know in terms of of sort of PR activity and we have the sort of slow growth of open PRS.

A

So by the way this one, particularly because we're looking at workload per sig, we specifically are looking at all and pr's and not just recently open PRS, because some old PRS are actually the biggest work generators because they've been open for a long time because they represent things that are hard to get right. The.

A

So you know- and this gives us an idea of general, but really what we want to know is which cigs right now, our hot spots, which ones are general problems and which ones are likely to be problems. And so for that we actually have a numerical chart, which tells us a little bit more so like, for example, and like I said, everything here has configurable time scales and different things so for general, but when I was actually looking at this for 1.9, let's go come on, let's scroll down.

A

So let's look at this in the 1.9 development period, and we look at this in the 1.9 development period we can see which SIG's actually have a lot of open PRS going on now. One of my goals with this is to create what people in so I come from the world of like hardware, performance networks, metrics, etc prior to working with kubernetes database engineer and that sort of thing and so I was trying to construct here.

A

What's called a second-order metric, which is well, we synthesized several raw numbers in order to give us sort of an intelligent assessment of a condition by putting those numbers together. So in this case we're putting several things together, we're putting the number of PRS, which is this first column together.

A

Putting the number of PRS together and here we're multiplying the number of PRS by a factor according to the size label that supplied by the bot. That's an estimate of how big that PR is, and then we are dividing that by the number of active reviewers. So you can get an idea of what the sort of workload is per reviewer.

A

In that sake, because a cig that has a moderate number of PRS, but only has one currently active reviewer is obviously going to be more heavily loaded than one that, even as a pr's, if they have ten active reviewers- and you could see during this period for the one nine period that indeed sig note was really busy.

A

Although was not actually the busiest sig for the whole one nine cycle, the, and so it was a combination of a busyness and leads not being responsive and and so I harassed, the leads to be a little bit more responsive and things got so got a bit better, the, but you know for then we can actually then go and look at this say for the last ten days period, and there is a problem of this.

A

You know I told, was still under development and there's a problem with this, in that I things that have very low numbers, produce weird factors for relative PR work love because they don't actually have a lot of PRS, but they have very few reviewers, because it's there's small work law in general, so ignore the ones that have one columns and look at the ones that have more moderate amounts.

A

So I'm looking here and, for example, if I'm looking for what is potentially going to be a problem sig in terms of workflow for one eleven one of the first things I'm gonna look at here- is AP I-- machinery, for example, but also potentially 6000i, because even though they don't have as many PRS, they also don't have as many approvers.

A

So with that, so that's a few of the things that we explored. So every one of these sort of charts within dev staffs is there to answer certain questions. Some of them are a little bit more general like we have a general compiled one of the github stats by various indexes, because that data is available and somebody might want it someday. But a lot of the more sophisticated charts are there to answer a particular question around kubernetes development.

A

So having explored the charts a little, then I we want to actually talk a little bit about how they actually work partly to help you understand the data that's being displayed, so Lucas do I, want to come up, and today talk about dev stats, the project, because of course it is an open-source project as well. Okay,.

E

So I will just tell a few details about implementation. Here's a github page CN CF des tartes, it's on Apache with a license and it's write written mostly lingo and a bunch of shell scripts.

E

This is a big data flow picture. Actually, the kubernetes is not the only project that uses def starts every every CN CF project has its own dashboards. So there are, there are free data sources, github are Chiefs git and github API and the remaining part which are Postgres database inference. Database and graph are now it's actually duplicated twenty times, because every single CN CF project has its own instance of this.

E

Well, we choose github our Chiefs instead of some other alternatives like bigquery or, for example, github, web rocks or other such things. The advantages of github arches are that we can have full historical arches for every event from the past and every such event contains its state from the past not now github. Our chips is free with no rate limits and it's available on github, our chief org.

E

We are, they are releasing every single event that happened on github during the given hour. All the events are parked into multi JSON array, and this, which is zipped. We are just downloading this every hour. Filter are non queue, Burnett's events or, for example, known parameters for primitives and so on.

E

We are putting all the results in the Postgres database when we can, where, where we have a full historical state of every single event.

E

Next, we need a data source gate. It's used for one single purpose when we are defining repository groups that are something like projects, for example, kubernetes can have few reports test. Infra can contains other reports and so on. If we are also analyzing files modified in the given git commit, so we can define a repository group as being a set of github repos and additionally, some paths and files from the other APIs. So this is why we are using it just to get a list of modified, add related files from each commit here exactly.

E

This is like, for example, we can define something like cube admin which is cube, edenian, repo, +, CMD, cube ATM from Cuba Metis kubernetes, for example, and finally, we are using github API source.

E

1 may may ask that we can use this for everything, but it's rate limited. We have 5000 points per hour, which is a way too low to get data. We need. We can't get the past state of the objects from github puppy and also it's quite slow, generating all data for kubernetes for 4 years of activity takes about two hours.

E

While my estimate to get this data using github api is about three months and more because we are ill, because every single request must be made as a HTTP code to the web service and that requires sending JSON and receiving JSON and it's basically slow. Why do we need github API?

E

There is a problem specific for kubernetes, because if anybody wants to add a label that it can be done, it cannot be done directly. The one must issue comment to the bot like label or so on, and in that case the bot our ink level. So the problem is that when I bought arts level or modifies, for example, milestone, there is no github event generated in this process.

E

So when we are so, the result is that we have a final state from the issue with the old label set and all milestone before the boat applied the changes and, for example, few dashboards that were analyzing data for the releases like 1.9 and so on. We're getting wrong state of the objects because we will have the correct set of the labels or milestone only after the next github activity, which may happen, for example, after the month.

E

So the final state is wrong is wrong and we are using github epoch only to query the recently modified issues and peers from the last two hours, for example, and only to get a final state of the label set and about the milestone of those issues.

E

Databases, as I said, PostgreSQL holds a copy of our historical github data in flux. Db is used for solely for display time series, so it's it's a data source for graph honor that is finally used for displaying data.

E

We are using two databases because calculating Postgres metrics is very complex for some of the metrics, for example, can take, or even more so we are processing all the metrics every hour and saving final results to the influx database time series and those influence time. Studies are very simple: they only contain data data that need to be displayed by the graph owner, so the final process is fast and doesn't require any expensive computation.

E

Finally, we are using graph on ax. In reality are we are using 20 graph on instances that are selected by the Apache reverse proxy, based on the server name? So you can use. Kate's death starts in CF IO and you will get K Q Burnett fana, but you can use, for example, Prometheus that starts in CF IO, and then you will get a Prometheus Gahanna and so on.

E

Also, we've developed a smart tool to mass import, Jason's and modified them. Because, though, is that doing this from from the UI is just not applicable, slow and cannot be automated?

E

If you want to contribute to the DEF starts, you can create issues the issues and peers on the github repo, but if you want to create new charts and requests, there is another way so.

A

I also want to come up and explain how to request new charts.

B

Okay, so you have a great idea for a new chart or another request, so go to that link right there and I'm sure Josh will have his slides posted as well. So what this is the process, so you create a card in the github project with your proposal with that information, and then you wait for the dev stat subcommittee to review it. We meet every week.

B

It should be pretty fast if it's approved, then I'll be moved into the approved column and then it'll be made into an issue and assigned to Lucas and if it isn't there'll be a thorough explanation of why it wasn't approved in your card and that's the project and that's the process. Okay,.

A

Well, thank you. So that's a little bit of a tour of dev stats and information about kubernetes development. We have about five minutes for questions which could include both questions about dev stats and the architecture and the contents, or you could just log kubernetes development information. Questions at me and I can try to dig through dev stats to see if we can answer those.

A

Waitwhat see the session no nope, there isn't a deep side. So no I can't is the answer. What have to do a lot of upselling? Yes,.

A

The question was: can you take us through an instance? We use the stands for solid questing. You remember we did. We did a little bit of that earlier, like I was showing, for example, the business about PR workload and trying to figure out which, which SIG's were heavily loaded, so that the release team could target those and Tim showed us looking at what we were looking at in terms of issue backlog and answering those questions and.

E

Actually, a few numbers from the keynotes: it came from this system so.

A

Everyone so I'll answer for me. What we do in response to that information was, for example, on 1.9 we were looking at here. We go. Let's go back to the actual time zone for the 1.9 development, because you know I realized that, like I had I had realized that sig note was probably heavily loaded, but it had not occurred to me that sig API machinery was heavily loaded little and sig CLI and I. Actually in that case, actually jumped into those two SIG's and I said by the way.

A

You know you guys have a lot of open PRS right now, you know, do you need help and in the case of 6000 I, the answer was actually yes and so I got one of the kubernetes founders to help them out with clearing out PRS, so yeah the well.

A

You said right now we're sort of in the transition, because we're developing this or transition of moving, this from say, sort of first-order stats that are like a metric to sort of second-order stats that are more directly actionable, as we figure out what those are you want to give okay.

A

Yes, you want to answer this.

C

So computing second-order stats is really. It would be great if we could do that within Griffin itself right. So one of the limitations of the influx DB data sources is that I can pretty much only get time series from it. I can't like add to time series together or subtract them or do any kind of comparison whatsoever.

C

Graphite, I totally can I'm not sure if Graphite's the right thing here but I'm wondering if there's some other data source we could use instead of in flux, DB. That would allow us to do some of those second-order things well,.

E

That there is such a data source I was I, was taking into account. The Prometheus and I actually have a green light to do some initial research about how it will behave. Do we have everything we need and probably, if that will work for us, we will just add Prometheus as a time series as an alternate source of data.

A

More questions we'll take one more ok. Well, like I said this is live it's available online to anybody no login required. So please explore the stats yourself and maybe ask some questions based on what you see. So thanks for coming.