Kubernetes Community, 12 May 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Community Meeting 20160512

Description

We have PUBLIC and RECORDED weekly video meetings every Thursday at 10am US Pacific Time.

https://docs.google.com/document/d/1VQDIAB0OqiSjIHI8AWMvSdceWhnz56jNpZrLs6o7NJY

Navops Demo; SIG Apps; SIG Scale, SIG OpenStack updates; merge queue challenges; 1.3 feature update

A

Good morning, all this is Cooper Nettie's communities or thursday may 12. As I say every time this will be recorded and posted on issued. So don't say anything. You wouldn't want your mother to know you're saying today on our agenda. The first thing up is a demo from the team at unova. So Rob you want to tell us a little bit about what you're going to demo and your partner in crime.

A

B

Hi, it's rob the lone gem here with the camera I'll just quickly clip on video. So you know we're real yeah we're going to demonstrate our navbox launch bottle. Show you how we build and looks like things out of on-premise or in the cloud and you're going to show you a quick type of girls, but just like.

B

You will just show my screen, sir. Yes.

A

That would be great if you can share.

A

I needed Rick and that stopped our echo. I unmuted you Robert, okay,.

B

Excellent, thank you so just quickly, I won't go into a full-on advertisement, but just so you know who we are: we've been around for many years and kind of a different space. The HPC space Cena has been building super large clusters for real big oil companies, life sciences, organizations for many years we've been orchestrated workloads on that. We brought our technology over to the kind of Cooper Nettie's container space over the last year, and we've been rebranding and repackaging these solutions for four containers.

B

So what we wanted to talk about and I won't get into the the customers too much here want to talk about the net loss launch product. We came up with it last fall and it's a very easy way to build a cluster. So you I, based solution, gives you a full atomic uber Nettie's flannelette, see dr. cluster, all prepackaged on premises in bare metal or you can build the cloud or Google Cloud and Amazon for that or you can even build a hybrid cloud.

B

We're going to show you how we can do that auto scale as well.

B

Black em son did some testing on our product. A little while back and they saw that the benefit really comes down to a command line approach where you have been a 200 steps to install a whole cluster. This is a simple UI and I'll just jump down and show you that really quickly, I'm not going to take a lot of time and xterra gave us a 10-minute limit and I believe we can achieve that.

B

So you can see Google Cloud running we've got no VM instances will go over the nav ops launch screen, simply UI for DevOps launch to install a master node. That's.

A

C

Being the wrong train, I think.

A

We yeah, we are seeing the wrong screen from you all we're. Seeing is why.

B

A

D

B

I know what I did wrong, I got all excited and they jump to.

B

There we go there, we go. That was my fault. Okay. Let me start really quickly back at the beginning, so here's the google compute engine and you can see there's nothing running. I've just started up a master node, and I think you can tell see that now and if I go on refresh the screen on google compute you'll see how easy it was to create a fully probation master node and that little delay. We had probably allowed me enough time to actually get it started up and running.

B

There is, and you can see the master node up and running it's a little easier than the 200 steps required.

E

B

Day, work in the hood and again we'll go over to compute engine and you'll, see that spinning up cameron, while that's happening I'll just share my whoops I'll, show the slides. Again. You can talk about the architecture yeah.

B

You can see that okay.

F

So right, so we kind of skipped over the actual installation of nav ops launched, which is really just done from a script on the internet, which pulls down a set of containers which set of images and then launches some containers that run on a on a doctor engine. Typically in your your local data center, on your laptop in Rob's case. Here we fold down several containers, the first one that you saw there was the web UI we all kind of a control container that holds the complete provisioning solution.

F

This has a lot of existing technology that we had developed over the last several years for provisioning cluster resources, bolt-on bare metal dimelo on local hypervisors or in the areas cloud providers. As we have that container, then we also have a container that hosts an operating system repository for actually doing local governmental installations. In that case, it's really just a.m. since we're using fedora core atomic.

F

It's posting a HTTP server hook thing nos tree repository that is then I served up through through you know: pixie kick-starts, that's the bare metal of general architecture when running in what Rob showing here get some on the cloud. So you only need local containers, the UI and the kind of orchestration solution or abridging solution, and it goes and it talks directly to GC key and creates the resources assess them out through the cloud in it. The aku, benetti's installation and uses the Cuban agents that are provided as part of fedora core.

B

Cool I'm just going to quickly share down.

B

Video, this is the demo we showed at Google Cloud Print, we're going to just tell him camera just going through how this works. What we've done is we've commissioned the local cluster near it was running on a trade show booth and we were spinning out essentially instances in the cloud as the workload went up not like yeah.

F

So, what's going on right here before we get started, we have the for the for windows. So what would has happened here? Is we've actually set up in a box launch installation with a single master, a single worker on a local brew? Nettie's cluster we've launched several thousands on top of that local infrastructure. You see their little test application. The upper left, which is, as some demo clients that's generating some loading up their demo server.

F

We have in the right that cash showing kind of the status of the replication, colors and possible system I'm in the lower left. We have the the graph on ax from from the google reporting system, trilling up from keith start showing the current load in the cluster and on the right. We have the kind of current state of the gcp project that we're going to be bursting into that's a load increases against our application. The one pot of the one instance running there is our VPN actually really bridging a local.

C

Or will not work anyone.

A

Who is not talking? Can you guys mute because we're getting a lot of feedback yeah.

F

So are experiencing our local final network to the gcp cluster network. So with that we'll get started here. So the first thing I'm going to do is scale up in this movie. I'm scaling up the number of demo clients which this is going to increase the load and we're going to have to scale up in Coober Nettie's the amount of replication for pot instances to handle this. So this is automatically detected by our you know. Through our rule engine I, we we can also use the horizontal pod autoscaler in this case as well.

F

So you can see right there we have to we need to. We want to have eight, and there are six pending. Is we don't have enough resources? And you can see that there with the red message there isn't enough cpu on our single worker to be able to handle all replicas else to move your mouse.

F

So now you can see we are pause. The engine automatically launch snarks for notes there in a gcp work or one work or two to handle the additional loads required to meet the eighth, the eighth pending demo or the eight actual demo application pods. So after a little bit of time there, those startup and those are running they'll immediately join our cluster and we'll see that over there and refine in just a second that the available capacity has increased. We can see that now we're handling. You know more per second, this demo hood.

F

Actually the first instance got available a little bit faster, so we'll see another stair step here in a second where, when the second gcp instance becomes available, so there you go the second one started up now you can see all 8 of our pods are up and running and handling are increased. Our increased load of our web application on this is all possible. You know, because we support all these multiple provisioning technologies inside of our nav ops product, so scaling it back down, um you know, will reduce the load.

F

Our policy engine will detect all of this and will actually start scaling down the scaling down the cluster. Since we don't have, we don't need all these extra replication controller pods. Are these extra pods running to handle our or you could slow?

F

So there we go. Those have been scaled down and I scroll this time, you'll see where we're shutting down those instances and they'll be down and completed here in just a second, so I think Rob. This is always.

B

Yeah, that's all we wanted to show. We just want to show people how we can do hybrid cloud bursting and provisioning of clusters, and now we appreciate you taking the time. If there's any questions we can, we could feel them now. Yeah.

A

We've got time for one or two questions. If anybody has any.

A

All right, no, we can. We can always follow up in I presume email or on slack with you guys of.

B

Course, and will be a toss con, so we hope to see some of you. There.

A

Fantastic thanks.

B

Particular time look but a short stuff. What yeah.

A

Happy to all right, so next up on the list is a little update on the community repo and the wiki move. I, don't know if either Aaron or Matt are on the line, but I am happy to give a little update if either of them is not on the line Aaron or Matt.

A

No okay, so we have been working to set up a new, a new way to handle the cover titties wiki. In order to make it easier to update you wouldn't need actual commit and merge privileges on the Cooper Nettie's main repository in order to make updates. So we have started moving the content out to a new community repo under the Cooper Nettie's organization. So I put in a couple of links in this agenda item Matt Farina has made the first pull request against it and gotten us sort of stubbed out.

A

We want to set up a directory for every special interest group so that you can have your own space to share content right up any information share notes and we can start moving away from what makes for a less than transparent starting point for new users coming into the community. So right now, there's a lot of Google Doc shenanigans in order to get somebody to see what the agendas look like or what what might be being discussed in a particular special interest group.

A

So as as we start bringing in new special interest groups and as we have time, I want to start migrating more of the content around our different meetings and special interest groups, including this one into the community repo. So we will start putting agenda agenda notes up into the community repo there, as well as sending them out to the mailing list, but no longer posting them on the blog for cooper Nettie's. So that work is continuing forward.

A

Aaron was very patient in waiting, for he was the one that created the first full record of the first issue about fixing it. So you didn't have to have merger choir or merge permissions in order to make updates to the wiki, which way we can hopefully get more people engaged with and working on, content out on the web.

A

All right next up on your.

G

Question, how cool is that? How does that mean moving content out of the github site and into the wiki, so.

A

Not documentation, not the so there is, if you go to github com, Cooper Nettie's, Cooper, Nidhi's wiki, there is content there that is really hard to update anything that is in coover, midis or sorry. Github com, Cooper, Nettie's, Cooper, Nettie's github do stays where it is, so that's the documentation site. So we're not looking to move that around. This is just the ad hoc cross communication stuff that has been happening in github com, Cooper, Nettie, scooper, Nettie's wiki.

A

That answer the question. Yep.

H

A

Fantastic alright. So let's move on to sig reporting, so Bob I believe you were going to tell us what success. What is going on with sig scale, I.

I

Would be happy to do so? Let me give me a moment here. While I share my screen.

A

And while you're sharing your screen, I'm going to pitch again that I could use some note-taking help because I I'm Anna Jack uppal of other things on the backside trying to make sure this all goes smoothly. My notes are erratic. At best.

I

H

I

See this now we can great okay, so we actually spent our meeting this morning going through some of these slides as a group just to make sure that we were all kind of on the same page. So we just jump ahead here and I Sarah I sent the link to you as well. So if you want to post it somewhere, you can do that, but the link to this presentation is also in the the six scale meeting notes doc.

I

It always seems like a good idea to repeat some of these things. Just when we meet money on Thursdays, here's the link, so you can. You can follow your nose through the through the normal path, starting with the Google search. Even if you want to this.

A

Is a perfect example of the github com, Cooper Nidhi's to burnet ease, wiki links.

I

Okay, great so, okay, yeah, so anyway else feels like good to do that, all right. So, of course, as you might expect, there's been a lot of discussion around one to three, but there's also been fair. We've been having a fair bit of discussion about kind of the future.

I

Beyond that, really the the community discussion has been kind of along three tracks, which is some performance goals, both one doc three and you know somewhat beyond a set of I'll, say obvious and interesting improvements, but the communities complete enthusiastic about getting done here and then the general topic of data sharing.

I

So okay, so discussion on performance goals, so this has actually probably taken the most has gotten the most energy. For you know a number reasons: one is that I think everyone's eating to avoid I'll call the marketing mistakes. You know confusing kind of short term testing efforts with the overall project goals.

I

So what we refer to, as the hundred note means so we're trying to be very careful about how we talk about these I think there's been a pretty healthy discussion around the need to support different goals and employment patterns. I think the specific one has been the one that I would really want. Everyone to keep in mind is that there are some different deployment patterns here and I think.

I

Certainly, the Google team is very eloquent about explaining that smaller clusters are arranged in the availability zones is a great way to manage to make sure you can round lots, of course, with high availability, but I think there are also some other use cases where larger single clusters, which perhaps may have different availability requirements, are still a requirement. So one of the really great discussions we've been having is about making sure we can get the kind of community goals arranged to support. Both of these there's been a pretty good discussion around you know.

I

Is there some natural size limit? We're, like you know it's just kind of not really that useful to go beyond, and you start looking at some of the stats around the largest clusters in the world. That's probably that may not be effortless pending. You know millions of course kinds of setups. So so that's a that. There's an ongoing, healthy debate about that and just thought. I would mention that there we have had a fair bit of discussion around wide around. Why variation in note sizes have some some companies.

I

Some efforts are oriented towards really large nodes. That's great there's been work going on around us in the note group and then so shout out to the nodes sake and then also federated testing continues to be a hot topic and sig testing and overall, the importance around being able to characterize performance on a bunch of different environments continues to be important and I have one other comment on that: a.

B

I

Bit later here I I'd say we have a very interactive group, I'm kind of walking through this a little bit fast, but there are an awful lot of people here who are part of the sig performance group. Sig scaling group that I welcome you to than to make a comment as I go here. Joe did you have a? Are you on that? You want to make make a comment here. I.

F

Am on um you know, I think you're handling it great I mean you. No no, no comments here. I think it's just there's it's not quite as linear as I think we originally expected in terms of previous goals together, but a lot of ins and outs, so I think the slide capture set great.

F

I

I I, guess I think one of our goals for the presentation today was to make sure that there was some context around this discussion. So as a follow up I'm sure some of you will want to go and look at some other docs and dig into some of the numbers. But this is some of the context around the house. Numbers are being discussed, okay, so kind of two tracks here. What one is two tracks within the track? So so in terms of just discussion of performance goals, one thing I think everyone should be aware.

I

Is that there's a of a shift in terms of how we talk about how we characterize clustered performance mid-stride, so Clinton has been kind of leading the charge on this quinton wojtek and some of the other other Google folks and I think there's some stuff that they are not quite ready to publish here. But this is kind of like the snapshot here, which is to move to a course for cluster pods per core and then pods per second is kind of the one of the ways to look at it in the. But this is work.

I

That's been going on kind of in the course of the 123 work, so for 13. The sort of previous gen way of looking at cluster performance is still the way things are gonna get talked about, which is nodes, pods per notes, maximum number of pods and still still all in the context of the current Layton sees.

I

So I will, I will pause here and see if any one of the other folks wants to make a comment up. 13.

I

And if not, I will move on hey.

D

Bob a quick question: this is probably obvious that I and I've just missed it, but just in terms of how many containers per pod are we or what types of containers are going in those parts. Well,.

I

Again, I'll leave this open for other comments, but in terms of working on control, this is really largely around control, plane kinds of work, so we've been using, for example, the pause, containers and scratch containers. So the goal here has really been that discussion here is really about control, plane performance, not about total cluster throughput and, of course, you quickly get into the specifics of.

I

What exactly are your nodes to be able to talk about that and I think I've covered this in previous meetings, but there's a there's, a node emulator that gets used for a lot of those. A lot of this work right: coo coo, a crook yeah.

F

Honestly, in terms of containers per pod, I, don't think that um those are numbers that that people have been looking at from the control plane. I'm, not sure it makes that big, a difference, I mean if you had 100 containers per pod. It would probably start you know changing things, because the control plane would have to track more data. But my assumption here is that we're talking, you know the two three you know for the most containers working again.

I

I think a lot of this work has really been going on in the node in the note zig, and there are certainly issues that I'm not probably the most qualified to talk about in terms of kind of maximum numbers of containers per node. And this is you know around dr. demon performance or you know other things so.

I

All right shall we move on great okay, so just thought I would throw up here. So this is some of the. So this is on the track of interesting, interesting work. That's being done to to actually move the performance ball ahead in the 13, so the one that six work actually got done pretty early I there's been a bunch of work around control, plane, optimization and network overhead I. Put some PR thing, PR notes here, put some names up here.

I

You of course tend to see the same date, the same names over and over again HTTP to protobuf support. We text on the ton of work on the scheduler infrastructure, I. Think the Red Hat folks as well had worked a lot on the realist and recent optimizations.

I

It is, and I have another comment on this in a minute. Some of these things there's kind of a natural point to go to to the single PR, where you can kind of follow your notes from there. Some of these are pretty large sets of PRS and I. Guess, if you're specifically interested a couple of the names here would be the folks that go poke at and slacked asked some questions, certainly worth mentioning at CD. 30 has been a pretty big, pretty big effort.

I

Chorus guys were talking to the meeting this morning that ETS like this project. This part of the project feel really well et, is passing they're working on some upgraded testing tools, but it's something like they were feeling pretty pretty positive about how this part about how this was going and again I'll pause. If a positive someone wants to jump in with a comment, oh.

A

If this has actually gotten into the tracking documents, just as a as an aside- and it doesn't have to be answered it just, we should make sure that this information is track that attract with what mike is working on in parallel. It's.

I

Mostly, there I think that the issues around kind of pulling together or pretty this bird set of PRS that all kind of encapsulate the work is your art makes it hard and I'm going to lob I'm going to make another. You know lobbying plea for trying to be a little bit more organized about that from the future. They're continuing.

A

To try to work to find a good news.

I

I know I, know, I, think everyone kind of knows the case, but it at least is a good point to show the impact of that so and then on the data sharing track. Certainly I mean it's critical for us to be able to exchange in the exchange data, about performance and about cluster configurations, and so there's still work to be done. There I think Marik has been making some good progress on a on a performance, monitoring, dashboard, I.

I

Think at the meeting this morning he was I, don't know feeling like it was maybe a little bit early, perhaps to be showing us off, but I think it's good great progress in this direction, so I was going to essentially insisted hey. This is great to see this kind of progress, so Sarah might be I guess, I would lobby for a quick demo at some future community meeting of the dashboard yep.

H

I

H

I

I think that was all about all I had here just kind of wider community topics that come up a lot. In the sake.

I

How, and when did we release our gonna, be in an hour LTS model, he's kind of understanding. The future of release management will really helped organize the work, especially because some of these scalability oriented topics are kind of long running projects. It would really help us organize the work if we had had some of that nailed down and then just again the topics around it's kind of hard to follow. Some of these work efforts.

I

If you're not super deeply involved in them- and you know I, I think the the kinds of things that have come up and discussion around you know being diligent about using the labels having single PRS, that kind of covered the epic or the effort or whatever, and you know, release wranglers.

I

Maybe maybe there's some ways to apply some new roles here and I think that's really kind of the some of the work fits Mike's been doing with trying to collect all of the pr's into the red sheets from the cigs kind of falls into this category. So it's great to see that going on.

I

That's really all I have by.

A

Way of an update thanks, Bob and I'm sorry I'm going to cut you off and see.

H

All them Thanks.

I

A

That people reach out to you or joe if they have questions about cig scale, which also happens at nine a.m. on thursday specific time yep.

I

Great, thank you. Yeah.

A

You're most welcome. Thank you for joining. We are going to need to go a little bit quicker through because we also have another last-minute add to the agenda. So I want to introduce Michelle nirali, who is leading up an effort to start a cig that is focused on user experience in the application developer. So Michelle, do you want to talk a little bit about that? Maybe two to five minutes? Hopefully we won't go into a long discussion or will spin that out if we need to sure.

J

Yeah, so my name's Michelle I'm on the home team at dais and I also want to introduce Matt Farina he's on the advanced technology group at HP, and we just want to introduce a new sig or possible new sig hold the gaps, rather it's more of a transformation of sick config. So some of us who have been following say, config and sig. Big data got together there recently to talk about some concerns we had so on the sick and fig side.

J

It seemed like there was a broad range of topics that were being discussed and perhaps the name didn't match the scope. The sig. There was a lot of overlap, also between topics discussed and that's again. Other said some community meeting and participation was dwindling as well, and on this a big data side, they were looking also to perhaps transform into a more general sig for running applications. So basically a long story short.

J

There were about five or six of us on this coal and what we realized was that what we all really actually were interested in doing was creating a cig where we could focus on running applications in communities and how to make that a fantastic experience, especially from the application, developer and DevOps perspective.

J

So there's a lot of attention currently on focusing and focus on building scaling and operating to béarnaise and that's awesome, but we also wanted a place where we could focus on how users define manage and run applications internet ease, especially for people who are just getting started so we're think about creating this app or the sig called sick. Apps will probably going to do weekly meetings wednesday morning at 9am. The first one will be this wednesday and we're going to have someone from openshift named prashant.

J

I do a demo on headset I'm, so I hope you join and not, and I are available for any questions or concerns or any ideas about interesting topics. Matt, do you have anything to add I.

C

Don't I think you did a wonderful job.

A

Awesome and Michelle was hedging about the where we're moving this forward. We think we are going to go ahead and move this forward.

A

I think she's been waiting for for me to answer yes and get her a bunch of specific logistical things, but that will all happen in time for a wednesday meeting as she suggested, and we can go ahead and then out that on Cooper Nettie's dev, when that happens as well as I'm going to pitch here, the Cooper nuku brunetti stash users mailing list which, on those of you who are developing and who are interested in how users are using Cooper Nettie's that we would love to get people seeded into there, so that people can so that we, as people who have experience with Cooper Nettie's, can answer questions when we send users to this mailing list.

A

It's it's no fun to send the user to a mailing list and then say: oh yeah, but there's nobody here that can answer your questions. So we need to get that seeded with people that are interested in and able to answer questions about how Cooper nidhi's works already.

A

But I love the user focus. I think this is super awesome and it is. It is that time in our evolution so also on this list for the sig update with sig cluster ops, it looks like there's a remote architecture or sorry, not a remote. A reference architecture conversation happening today at 1pm Rob did you have anything more than just that announcement.

K

No I'd like to encourage people to come well.

A

Yes, obviously,.

K

No we've been doing demos for a while. This is we're going to get back to some work today.

A

Fantastic well thanks for sharing the timing on that. So I did say that we had a new agenda item that was pressing and urgent, and so I will add eric paris or let him take a stage here to talk about the merge q, yeah.

C

So I guess I want to not name names but call a lot of people out and discuss what we think we can do to make things better. The merge two over the last couple days has been lucky to merge. 10 real PRS, there's currently 54 real PR Zin, the Q I considered a real PR if it needs e2e test.

C

So the right things are going. If you created a new PR today, it wouldn't get merged for about a week, which seems pretty unacceptable. What I've noticed in the last two days is an extremely large number of Google employees who are adding priority labels to their PRS for whatever reason to jump in front of the queue, and this doesn't seem manageable. When we have so many people backed up that arbitrary features. Kind of are leaping to the front.

C

I know that we have these 13 release features as well, but I'm not sure what the like best way to attack this. In case people wonder the q could run at about 50 PR zayday if it went full speed for 24 hours. um Maybe it's a little lower right now, seeing some tests are taking longer may be 44p hours a day, so if we had no flakes the Q would be able to run and drain by end of day tomorrow, I'd like to start a discussion on how to make this better.

C

My gamification thought is that you're only allowed to up a priority from p 3 2, p 2. If you fix a flake, the second flake, you can go from p2 to p1 and if you fix a third flake, you can go to p0 like that way. You're incentivized! If you want your PR to go first, to make the system work better for everybody else, I'd like to have this discussion off list, but everybody should start thinking about a stop arbitrarily labeling your features, so they get in front of everybody else's feet, cheers and B.

C

How do we make this better for everybody and I'll shut up and let other people discuss and I know we don't have a lot of time to discuss. We.

A

Have a little bit- and this is a really important thing, but including I'm, seeing little golf claps in the in the chat, so I think figuring a way to make this work. I love your gamification idea, I, don't know how we can enforce that. Maybe there's is. We certainly can suggest this as or suggest this and engage and encourage this as a community I don't know.

A

Perhaps there is someone, maybe a build cup of the day or some of the other team that can jump in and suggest if this is, if this is possible, if we can enforce it or if there's an easy way that we can make this better as a community experience, if.

C

There's general agreement I regularly get queried by people at Red Hat for anybody doesn't know, that's where I am about what the priority is in the key you and why things have left to the top, so I would have no problem if the community likes the idea of putting your issue fixed, like such-and-such upping priority, nah I would have no problem looking for things with priority that didn't have that and yelling at people at least for the next couple weeks, while we're crunching towards 13 I, don't know if this has to be a enforced good thing if we might all be able to just kind of create pressure on ourselves to follow the system.

C

L

Quick question: do you have any recommendation on when it would be okay to up the priority, I think like there's anything.

C

L

C

A bug should jump in my mind: okay,.

M

Well, I, don't know the end. The end I think if it fixes a flake that's frequently encountered, then that should be like a p0 p1. But if it's like any there's a ton of us working on features that we're trying to get into cube, 13 and so I, don't think any new feature should necessarily immediately Pumped higher than the normal p3 priority, because it's it's really annoying to everybody else. But I.

M

The only thing I see right now, that's worth moving up higher is things that fix the broken test infrastructure or things that fix like a p0 flight for a few one, flake yeah.

L

I definitely buy that what about things that are touch a lot of code and you keep the developer, keeps getting hit in rebase hell like? Is that just like a number.

M

Of us get hit in rebase, hello and I. Think there are some I know like I know scheduled job when and recently and I can get that. That would be difficult so like if your PR like grows beyond like a certain size. Maybe I could reason on that. But I haven't seen that as the metric that people are using lately to you. Well.

C

Priority I kind of like my gamification solution to that problem, because, if you're in rebates hell, you might want to spend some of your time chasing a flake and which case you would move up the queue and help you get out of your rebased cycle. So.

N

It kind of self-interest.

C

Help everybody idea yeah.

A

Joe also has a comment: he's been very graciously raises hand, although I can't see everybody's video, so that won't be effective as a queuing mechanism. I think.

F

You know having a default priority for any work that people want to do. I think there's also room for another priority above that for p0 p1 work that actually has gone through, essentially, as we start to develop the more formal process of getting stuff into a release where you actually have a wide agreement where you're like you know that stuff is sort of on the track, and you know the community has sort of put effort as a hole into that thing.

F

Getting that thing in there at a higher priority I think makes sense, then also be.

N

F

That that isn't necessarily can do a release. I'm late on that, like fixing the flakes and letting the the queue flow freely, is definitely the highest priority. You.

M

End up getting like cigs competing with each other right so like if the notes sake see something that's a high priority. Get in. You know then you're going to end up competing with like I, don't know pick another cig, be a TI machinery, cig and everybody's going to say that their stuff is is p0 p1. So then priority ends up being not thing. Well,.

F

Know what I would say is that this is stuff that cuts across the project. That is actually you know part of the release planning cycle where essentially, it's being trapped on a spreadsheet, there's a document in an owner to actually back that thing up right. So so in terms of release priorities, this has already been sort of longitudinally identified as being critical to the release versus something that's just opportunistic I was hoping to sort of sneak this in you know, type of thing, I.

A

Think the key point is cleaning up the submit queue and the flakes so that we don't have to have these trade-offs and that's one of the reasons I like Eric's idea of instant incenting people in order to fix flakes and push PRS through and I know that we still get into the more work to get to the more work and suddenly we're yak shaving. But but it is. But it is a reasonable approach to this.

C

Yeah I know there are some people in chat talking about the overall health. It is, I guess, like our worst current like test, has about an eighty percent success rate which, if our overall q had an eighty percent success rate. That would be wonderful right.

E

C

Only thing that, over the last this is I think day and a half I'm, not sure how long this has been up. Oh no, two days it has worked, is a go build. That is the only thing that hasn't flaked in two days, so there's lots of them in there. It compiles ship it yeah. Well.

C

So, even though it might seem like that one's like doesn't matter, it blocks everything for an hour and a whole lot of those different one. Things blocking for an hour mean we're making no progress right.

A

And this is not. This is a line of single points of failure and, in some cases or a set of single points of failure, I.

A

Let's, let's see if we can encourage better behavior and certainly work on on flakes, is it possible for the broader community who is doing future work at this point to actually meaningfully contribute against flakes and see how this is is happening? We had a lot of discussion about this a couple of months ago and we were trying we Google we're trying to make it more transparent and easy for people to help on that and I'm certain that we can help.

A

We can be helpful in helping people learn how to do this better, but we need to make sure that we push this forward, something that's.

M

Not obvious to me is like how you can tell which lake is the worst flake.

L

There's actually a little bit of work going on. One of the test engineers at Google is looking at higher highlighting flake of the day. Basically, what was the most single flaky tests in the last 24 hours in hopes that if we can fix that one for two weeks running, it helps in a large way. He did some basic analysis of like looking back a few weeks.

L

What would cause the most problem- and it did turn out to be just a few things, causing lots of problems, at least for this one pretty looked at so hopefully we'll get something like that soon, and that can help direct our efforts to the most high value places.

A

Okay, certainly a thing that we will continue to have to pay attention to and fix and make better and thank you Eric for bringing up bringing it up and pushing everyone toward better behavior on this I will keep a line item to talk about this again next week, but I'm sure we can do this also offline on mailing list and in slack probably.

C

Thank you, yeah.

A

Happy to all right, we have about 15 minutes left to go through the 1.3 feature, update which to be fair, work. I, don't know that we need to spend 15 minutes on it.

A

We just want to make sure that we are touching on the the key features and I want to follow up also with Mike and the work that he started last week for those of you who weren't here during the Asia friendly time zone meeting last week, mike is looking for more help from the cig leads in order to get sort of a dashboard e overview.

A

So we'll get there after going through a couple of the quick feature or couple quickly through a couple of the features that had been listed is p0, so TJ do you want to go up first and then Mike? We can follow up and take a look at your documents again.

L

Sure that works for me um yeah, so I'll start off I.

L

Actually, each you know I've been giving updates on sort of these p0 blocker 1.3 issues features, we've talked about, I've, been gathering, notes and reading them and having a note, taker, write them and the things I figured I, just paste them directly into the dock and say don't take or some time so so I am basically reading off the community meeting notes here so for the for the four highest priority sort of issues that we're tracking the sort of TL DR status is people are working hard.

L

People are on track for feature complete, which is now seven business days away, but they're tight on that more or less so, I guess business as expected. Although we've done a better job of sort of scoping and moving resources around and asking for help early so that they're not going to blow past feature complete, which is which is great so the first off is uber Nettie's.

L

Basically, designs are complete, the majority of code is is under code review and there are a handful of PRS remaining, including the Indian tests on, but people from from several different companies are contributing their to move that forward, and so that one is is on track.

L

Pet sets same same story more or less probably a little bit further along. This is the staple application support feature. The last few PRS are in review and once those get in the Indian test will will shortly follow those. So hopefully we will get to writing the Indian tests late this weekend next week in terms of scaling, the 1.3 goals is a 2000 node clusters working and passing the sls.

L

You heard bob just give a long overview about how scaling is going and the gist of it is that 2000 node clusters are working on cue park, which is sort of a denote emulator that it helps test the scheduler and the density test is working so on those two thousand nodes were able to schedule 60,000 pods, actually with a with a lower latency than we were four.

L

ah Even the 1000 note clusters in 1.2, so those the improvements that have happened in material and look great, and the next step is to use real clusters, obviously to validate those those sort of emulated nests.

L

And then the last item is about distributed testing. So this one is a little behind. Perhaps it's hard to hold this to the feature complete date in some sense, since it's a test, dashboard and not actual code within communities, but the dashboard for aggregating all individual test results, including from from outside companies other than Google, is in flight and the estimate for that is 527, which is two weeks away.

L

So hopefully we will have that out and working in about two weeks and at that point, there'll be a much nicer view of seeing exactly which test is failing and looking over time and being able to sort by sort of most flaky and all of those things, as well as enabling other companies to upload their results and get those all addressing with you.

A

Awesome, thank you um like. Are you going to go quickly through the features that you had listed in here? I? Don't think we need to go through features regularly, but it is good, especially as we're coming up against the end of the 1.3.

N

Yes, sir, so I mean the mustard kind of spread shades we shared with sig leads I would appreciate if TJ can go through it and quickly throughout so I compiled it out of the least out of the documents shared by difference in groups in the beginning of 13 I, think some of the items should be thrown out of there because they tracked by Google internally.

N

So if TJ can go through the list and throws them out, that would be helpful, general observation, so between chorus and redheads, a number of items which Appy once and a blocked on lgtm. Even though implementation is already there, it seems like proposal was so GTM. Can we go through the cycle a little bit where somebody else asks for the comment, and then we go back um and given that there's seven business days remaining, it would be nice to unblock them.

N

So if folks, who are maintained is convincing fatal dose, the next day would be great. Can.

A

We collectively agree that the part that the person that is identified as the owner based on the the list of different sub component owners, is the last person to say yes and as they say yes, then we're good is that at this point, are we are we tight enough on the definition of owners and that list that we can do that at this point, that'll.

N

Be great from our perspective.

A

Not hearing any nose or any concerns, let me chase chase that as a as it to do and Joe actually also in the Y.

N

Yeah, that's up to TJ, basically right on her and afraid. I know TJ trucks internally. If he can merge the lists, I just don't have enough kind of righties ability, so the least mostly reflects right now chorus and redhead and some other external contributions. Yeah.

A

N

A

Be great if we could get a better view across the whole community, and that includes Google is just another member of the community yeah.

N

That would be awesome. I keep asking community I get kind of community. A cig leads to review it and make it accurate. So my general observations are 28 items right now there is 10, p, 2 and all of them I in progress or waiting for final lgtm. So it's I don't know it's cutting pretty tight in my world ah and then yeah. So the dog has quick update on the current status. So rocket Metis is pretty close. If somebody can help this, the advisor at least accelerator g LG TMZ's, it would would be helpful.

N

We are basically reflector in c advisor to be somewhat less dr., specific and then Cooper natives. We already covered a plus h, cd3, we already convert in six scale and then the self-hosted cube letters, an example of when it has been a GTM tied and stuck again. Code is already from our side.

A

Awesome, thank you so, against.

N

Please signify senses the wrong link first time, but they sent a corrected run. Please review help clean up the least I think it will be in benefit for room I.

A

Would could not agree more and thank you very much to you, Mike having joined chorus and tokoro us for sharing your time with us in getting this sort of wrangled from an external perspective external to Google perspective, but a broader community perspective type.

N

Listen alright.

A

We have two minutes for one more cig update if you're still on here, because you wanted to jump back and give us an update from the sig OpenStack yeah.

O

I'm still here so we have a brave like, like early wrote, an important update from sig OpenStack. We have released our first OpenStack provider for kebab scripts. I will share the link in chat right now and I will edit to the mitten admitting gender assu are. We have released the first full set of scripts to deploy your communities cluster on OpenStack. It is based on heat and some souls tech stuff. So now it's done. Listen that you need to deploy communities, cluster or nuttin strike is only open state mesquite and they skip up strips.

O

So we go fantastic.

A

Thank you so much II that was, and that was even a super, quick update, so fantastic to Menace.

O

A

Minutes you did, you did great, so I have one further notice, which is we will do in Asia friendly timing again on jun 2nd. So if you are working with people who live and exist in in the asian time zones or if you're, traveling, today's and time zone will do, will test this again on jun 2nd, but we had exactly one person in the asia time zones.

A

On the last meeting, we don't see a substantial uptick in that then I think I'll double back in July and keep this at 10am until we see more requests for Asia friendly. If anyone who did present today wants to go, take a peek and at the notes and add a little bit more, that would be super helpful I'll, send these out tomorrow and then otherwise. We've got a like two minutes three minutes. If anyone wants to bring up any specific time zones, I, don't.

K

Have a toughie I'm.

A

Sorry, my I'm reading chat as I'm talking and that's never a good idea. If anyone has any specific topics, I had.

K

A question about oz, con o thing: you said that you were going to have sort of ask the experts. It's our place, to sign up for that I wrangled, two of my guys in to send.

A

Me an email that would be super helpful support.

K

A

We are having an a Google booth that is showing off all of the open source projects and within Google cloud or the google clouds contributing to and cooper. Nettie's is going to have the presence there doing open office hours, and then we also have a cooper, Middies hackathon that is happening on thursday at Aulis con. It is actually the odds con contribute, hackathon and one of the days is sponsored one of the rooms. One of the days is sponsored by Cooper Nettie's in the cloud native compute foundation.

A

So if people are interested in helping on either of those, please send me an email or find me on slack or you know, send me a carrier pigeon or something all right. Listen.

K

You want to note by carrier pigeon, not just the pigeon.

A

Preferentially I said: you'd send me a carrier pigeon that contains some information, all right right or a tweet I'm.

K

Not gonna go anywhere close to that yeah.

A

Anyway, everybody have a great week and we will, I will see you from the hackathon or just before the hackathon next week at oz, con.

B

A

Have a great week.