Kubernetes SIG Architecture, 11 Jun 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 20200611 SIG Architecture Community Meeting

Description

GMT20200611 180502 SIG Archit 640x360

A

All right, hello, everyone- this is the um bi-weekly code organization, sub-project meeting under sig architecture um and today is june 11th 2020.. um So we have a pretty light agenda. The first one is something that that been raised. I think I'm missing context on that, but I'll let them just walk through it.

B

Yeah so before ben starts, complaining I'll give a quick overview. So uh what happened was uh we were iterating over uh cia, so c advisor? We usually pull it uh just before a code freeze uh typically, and we make a tag on c advisor and then we pull it um towards the end just before the code freeze. But recently what we've been doing is like do a periodic update of c advisor.

B

You know, because we were trying some things out and there were major changes that were going in, so we were trying hard to test it uh because we don't have adequate test coverage in c advisor itself. uh We have only a few uh operating systems and uh combinations of environments where things run, so we try to make it more frequent.

B

um See advisor lacks enough hands to test in other uh things as well. So you know so the best way to do it is just throw it into the mix and kubernetes, and then let people go while um trying things out on everywhere that they usually try kubernetes on. So that was our way of uh opening up c advisor to more things. But what happened was um when the k log request came in the request was for structured logging, so we ended up updating a bunch of repositories to a newer version of k.

B

Log and c advisor was one and as a result, what happens is if there is a regression we find in k in c advisor.

B

We can't go back to a previous version of c advisor so that that was the loop where um c advisor got stuck in because we found that we broke. We did indeed break some scenarios where uh cpu was not being found, and you know some other assorted problems. So um what has happened? Is we basically broke kind um from master uh running against uh kubernetes master? uh That was the main main thing and there's a bunch of people who were waiting for the fixes and then we have.

B

We haven't been able to uh move to newer c advisor because uh for one thing uh the cpu fix landed yesterday or there is probably one pr still in flight. um So that's where we are currently.

B

uh So we couldn't go back to a previous version of c advisor which used to work because you know we updated k-log um in kk. So that's that's the whole story. uh Ben. Do you want to add.

C

I would add um so: we've discovered it in kind, but it turns out that it's just all kubernetes of any sort is broken on any system with uh hyper-threading disabled. uh It was also broken on arm, though I believe we did get that patch in um now, though, we're running into okay, um so we can't roll backwards because of the k log thing, but we also still can't roll forward because of all of the secret v2 stuff, and um I think grpc incompatibilities, um so I'm not sure there's any one individual problem here.

B

C

On the whole, we got into a point where we had a known bug. We really should have been able to say: okay revert to the previous behavior.

C

We've got this new shiny, pneuma detection stuff, but it doesn't work on all systems yet, and we can tell from the bug fixes we weren't really fully aware of the file system layout that we were trying to read through nc advisor, so the stuff was not stable, yet it wasn't ready and we couldn't roll it back and we are still unable to roll forward and it's been over a month. um We should not let our dependencies get into that state.

C

I don't know um how we can coordinate that better, um but we've got a number of these changes going on that are like. Oh, you have to roll forward. You have to do this new, like we're changing the lib container interface, we're changing grpc, we're changing k log.

C

um I think there's also something of a testing issue here that we missed some of these, um but uh it didn't take us very long to detect it. It's just we were like okay. Well, we want to get this big sweeping change in, so we don't want to roll back, um but we never managed to roll forward and we still haven't the fix is in c advisor, but looking at I was attempting to update that today and um it looks like we've got a few more organization messes that are the new problem with that yeah.

B

uh Plus things got more complicated because we got sidetracked with the container d run c problem as well, um so that that's a whole another thread. We need to unscrew um so yeah and but one thing one bright light here is uh at least we know that we caught the problem earlier, uh I would have.

B

It would have been even worse if we had done what we used to do before, where we cut c advisor towards the end and actually shipping it, um because uh not many people might have had a chance to try at that point so that that's the flip side of it.

C

Well and we're lucky there that we I mean so we had a member of the istio team that tends to try to test with kubernetes head on their corporate workstation at google, where hyper threading is disabled and they caught this while, but it took us a bit to pin down because it it creates a strange failure mode where you have zero cpus and nothing schedules. um I I think my promise, I don't understand why we got in a situation where that wasn't a simple.

C

Like revert of this completely new behavior of reading cpus, I mean we're using a completely different linux mechanism to detect them before we were reading um proc, cpu info and now we're reading this whole virtual file system. That um has different directories for different nodes and things. um I understand the intention to pick up the new network. I don't understand why we weren't able to uh like, even if we couldn't you know, revert k log. Why yeah? How did we get into a state where we kept just trying to roll forward for this long.

B

So I filed a reward um and david uh uh said: let's: let's try to move forward. um Okay katrina was doing a bunch.

C

Of work for this, like I think, the rolling port is reasonable, but surely we have to have some kind of cut off point where you know, if reverting is going to take an entire release, then it's our roll forward is going to take an entire release. It's not a rule for it anymore.

B

uh Yeah at that point, we didn't have enough information that we have now to say that yeah it would have otherwise I would have pushed for a reward. uh I did push for a reward, but then right doing it, we didn't.

C

B

Information at that point for.

C

At least the projects we own do we have a policy on how long we will try to roll forward on a breaking bug like this before you know rolling something back.

B

uh Typically, you know, I must do a knee jerk and roll things back, but this was like c advisor which it does, which is not a community project uh as well. I.

D

Think it's directly proportionate to like how strong the signal is that something is broken. So the.

B

D

That we didn't have a test that was failing or a test job that was failing either as a pre-submit or as a release blocking job like we had the report, so we knew it was a real issue, but it wasn't like a light that was green is now red and we need to get to green again.

D

um It's a much stronger case, like those are the things that usually spawn a like. We need. We need to roll this back like it was good on day three. It was bad on day five, let's roll back to day three like those are the ones.

C

I think we need something to cover the situation, because, like the fact that we somehow managed to break these particular cpu platforms doesn't seem like something we can scale uh testing of, like. I don't think we're gonna run uh like special c advisor tests on every cpu, with every kernel configuration with hyper setting on and off, and um I I think we need a way to say. Okay, we don't have a ci signal here, but we we know this is broken. Okay,.

B

So, let's uh two things here right, one is leave the c advisor as a special project, so you know that it has a different set of rules. The other one is g, linux uh ben. I don't really care about g linux, but if there's something else which is in public um that uh that we broke, then I would like to add a ci job for it. We I mean so we I mean so.

C

We didn't take it linux, but the bug has nothing to do with g linux.

B

It doesn't matter if there is somebody else we are breaking. We add uh a ci job for that system. We do have that uh fedora core os, uh but we don't have upstream ci there's nobody in the release.

C

I don't think it's reasonable to try to have ci across every cpu to begin with, with every kernel configuration if we have a user reporting that this is broken, we need to fix it and c advisor may not officially be in project, but in practice the maintainers are.

B

Right and kubernetes has room to roll back right, but again this is not come at a release boundary. We are not trying to make a release and rolling back, and uh you know this is we haven't even reached code fees, so I don't think it's a fair thing to. If this had happened after code freeze, then I would say: okay, you gotta dial things back. We haven't reached code freeze, yet, oh, we still don't have a platform right. uh So what we haven't reached code free, so we haven't really broken anything.

B

Yet we haven't broken anybody yet anyway, but yes in general. I think that we should be more careful, but at this point we didn't even have a choice uh of going back to rolling back all the five projects back in uh to k log v, one either. um So so that's actually what I would like to talk.

D

About more, I think, I think, the implications for sig node, like there are some things that we should take to take note on test coverage and management of c advisor and, like I think those are good conversations to have the k-log v2.

D

Rollout is what I'm more interested in here, because that I think that has lessons we can take uh like putting ourselves in a position where we have no way to go forward and no way to go backward um is really problematic, and so thinking about how we, how we did the v2 rollout and if there are ways we could have done that differently, that would have preserved our options.

B

We did have an option jordan, which we still do, which is go back to the old c advisor and use the k log b, one plus v, two, that might uh you know the hack that we had for making sure that we don't lose the logs. So that is still on the table. I just don't wanna. Do it? That's all.

B

I do have to wait for c advisor to be fully fixed.

B

C

B

See there's fixed but we're unable to pick it up uh when was it fixed yesterday, ben.

D

Right yeah, we know like the the known issues are fixed, but the fragile detection is still the thing being used, and so it's entirely possible. We will discover some.

B

D

Like pushing a concrete implementation into a bunch of dependencies and then trying to synchronously update, all of those dependencies seems like it was an own goal like.

D

I think, the the way I would have maybe approached it is um switching to a logging, interface and saying, like let's have an interface, that we can connect up to cloud v1 or to catalog v2, and that way we're not we're not trying to coordinate like five or six downstreams.

D

All at once and like once, it's in rolling back one means rolling back everything like uh uh yeah. um There aren't a lot of dependencies that we have where we control we control the diamond dependency and then like try to propagate it out. uh This is the main one I'm aware of, um but at least being aware of the situation and trying to trying to avoid getting the same situation in the future would be good. What were you gonna say david.

E

uh I was gonna say that I find the idea of having ci jobs for the cpu configurations or whatever it is here, um various like linux, flavors to see a reference to arm. If we want those things to work, I think having ci jobs for them is pretty reasonable.

E

uh I I can appreciate that something was found to be broken.

E

It's broken on my laptop, it's not something that is easy to to objectively consider and say, like it was broken on his laptop, so it reverted. I I understand, like you know sure different people are different and like maybe this particular person, we have a lot of confidence in, but I'm I do struggle with with having this dependency. If we hadn't owned all the projects using k log, we wouldn't have tried to update the dependency this way. Would we right like we would have jordan's on mu, but I'm confident.

D

E

D

E

Don't own them all.

D

We went out to all of them and like contribute, we we contributed changes to them. Okay, for our own benefit,.

C

D

C

Cpu detection thing: it's it's a serious regression that it doesn't work on some potential platform and I still disagree. I do not think it is reasonable for us to try to fund and maintain ci for every possible like cpu out there, but I mean we do support them. They were functioning perfectly fine before this change. uh The way that we detected things before was pretty robust, we're just counting the processes sirs and uh the c uh proc cpu info. We switch to reading this under-documented virtual file system and it still probably has bugs.

D

Were you saying that the entire arm architecture was broken.

C

That is the impression I got. I don't have access to testing that either.

D

Like if, if we release an arm version, it seems like there should be a test that tests against arm. Like david, I tend to agree like every last cpu oddity and, like I don't think, that's reasonable, like it's an infinite number of combinations, but like an entire architecture. That seems egregious to me if that's.

C

Was actually what happened it? It's that and also cpu hot plug, which is I mean debatable. Yeah.

D

um I think that conversation is probably probably belongs more to sig node than to us. I think our concerns should be around like how what are we doing in the way we manage dependencies and.

D

Especially ones that are highly referenced, and can we improve how we make transitions there so that we don't put the project in a place? They can't recover from usually.

C

Something that I think I'm missing here is why we couldn't have done the v1 v2 k. Log shift thing temporarily.

C

uh This is enough of a problem or because, like mechanically we're unable to do that,.

B

So the problem was, we didn't want to go back to those five six projects where we updated from b1 to v2 to roll them all back, and then uh you know including c advisor and then go forward uh with the reward to see.

C

Why do we need to roll all of them back simultaneously if we have some kind, if we have some kind of one to two shim, why do we need to roll back all of them.

D

I I think ben is asking like where we are right now going forward with c advisor means pulling in a run c version. That's like mid development um like even picking up the new version of c advisor with the fix doesn't put us in a great place. um Our if we rolled c advisor back to the last known good version, then that would be before c advisor picked up k, log b2 and we could shim okay like v1, to k log b2. That would.

B

Yeah that that's fine with me, but it's not our call right. It's signals.

C

uh I mean it seems like it's been our call about the uh thing, I'm not talking about changing c advisor itself, I'm talking about rolling back the c advisor dependency yeah. I.

D

I think when the c advisor thing was going on, we were saying like we can't pick up the new fix, because this is in progress, um and so, if there's a way for them to pick up the new fix, I I think some of the signature folks would have rolled it back and shimmed it. So if we we should give them the information about how to do that. So they can decide.

D

If that's what they want to do, and then we should consider ways to improve uh both the rollout of things like catalog v2, and if there are different approaches like an interface based approach, that would mean we would not have such tight coupling between all these dependencies right.

B

And the other thing is, I just feel like going and ripping out kellogg from all the, though all those dependencies, so we don't have to deal with them.

B

It's just use uh basic logging stuff uh from golang.

D

Because we already shim that yes yeah, that seems like it either basic go logging or the interface. That seems better to me, but I haven't thought about it. That much.

B

Again, the motivation for the keylog v2 came from the structured logging, otherwise we won't have even done you know. So. Many changes in k log that we ended up with.

C

uh um There is an interesting point there about the architectures. We don't test- that's probably not a discussion for this meeting, but if we are going to say that that is a much bigger call to make- and there are many architectures for which we've never had any testing that we've released today and we do, we do normally pull in fixes for uh such as s390x powerpc.

C

So in general we have pretty limited testing, for we definitely don't have windows, 32-bit testing.

B

The um that we tell there is it is, as is it may it may work, it may not work, we are not guaranteeing that it works other than fd64. I think that's the approach we've taken to these bugs uh I'm not talking about bugs, but this is a general statement of support for those architectures right. uh It's it's use at your own risk.

B

We are not. That is why, when the risk folks came, we told them look. You got to do a cap which thoroughly writes down all the all the all the things that they'll have to do.

C

I I again, I actually don't think that was super reasonable, given that we continue to ship power, pc and s390x. Okay,.

B

So I'm trying to differentiate between.

B

Picking up fixes and the general support statement that we are making about it right.

C

B

C

Mean we do ship those platforms which we don't test and we do fixes for them and we do do support for them again. It is I'm not sure why these lines.

B

Seem kind of uh we have that's a balance right ben. We can't say no uh to some of these platforms because because otherwise they they'll never be able to make progress.

C

So sure but okay, then I mean why are we applying this differently to these two different platforms? It.

B

Is not different, what is different, we it's, it may not work. It may not work right, that's what we are telling them and if it doesn't work we'll find a way to move forward right, but the difference with risk v. Is we blocked them from even building?

B

Well, I don't. I wasn't my call, but I would have preferred to tolerate a risk, but then they did have another problem then, which stopped, which is uh they had some changes in golang which weren't in the version of golang that we were using, so it wouldn't have worked anyway at the initial.

C

Pr, but yes, this is probably discussion for sig note, but I I yeah.

D

I think it's super strange that we release for architectures that we don't have tests for and that we distinguish between like well we'll release this a build for this architecture with no tests, but we're not going to release for that architecture because we can't guarantee that it works and we don't have tests like that.

D

We should test what we release. If, if someone wants to contribute a fix that allows kubernetes to build on a platform um like and doesn't add, complexity or add problems like, I don't have a problem with that, but changing our release, scripts and being like all right. Here's the release for you, know web assembly and we're like. Does it work like?

D

If so, if someone the thing with releasing, it is if we release one version and someone consumes it and it works they're like great. You guys, support this architecture and then we release the next version and maybe something broke, and we don't know because we don't have any tests and they upgrade and they're like look. I was working and you broke me and and that's a reasonable thing for them to say and saying like. Well, I don't know we just built it. We don't know if it works.

B

D

B

Telling, which is exactly what we are saying for g linux right uh it used to work. It doesn't work anymore right, but there's nothing.

C

Esoteric about that platform, it's just intel architecture yeah, I wouldn't overtrain. On g linux. I would in particular linux about having hyper threading disabled.

B

C

Like that's a pretty standard mitigation.

D

Especially given vulnerabilities recent vulnerabilities like disabled, hyper threading is a thing that a lot of linux servers are doing, especially virtualized, like shared environment servers, so that that doesn't seem like the same issue as here's, an entire architecture that we we've never actually even run or tried.

D

So, uh but I agree, these are questions for signaled and or sig release. Not really us. Our place is like how are we helping the project be able to pick up fixes or roll things back or roll things forward? uh Are there things that we're doing that are causing problems or recommendations? We can make so dense?

D

If you want to like jump into the c advisor issue and say like if you want to roll c advisor back to the last known good level, even though that means c advisor would be logging to k log b1, here's how you can do that and like temporarily shim k, log v1 to v2 like doing that for the c advisor logs, the volume of logs coming from c advisor, like that, I don't know that doesn't seem like it would be that problematic and and give sig node the information.

D

So they can make the decision like do we want to roll back while we work out whatever's going on in c advisor.

D

If, if you want to.

B

Yeah I can. I can do that john for sure, yeah.

D

uh I will probably copy the notes, maybe with some explanation, uh into a thread and start that with sig release and ignored and say like this happened, it's not great.

D

This seems inconsistent like what what are we doing here.

B

Yeah so uh can can I can we spend a few more minutes on uh the moving forward bit uh just in case that's uh something that we could entertain as well, uh so so for moving forward um the issue right now is um we need to pick up a newer ecd um because of uh one of the dependencies fails. Otherwise, one of the one of the things fails otherwise, so we need a newer dependency of hcd and grpc to get picked up.

B

If we are able to do that quickly, then we can rev up c advisor. So that's that's what uh I I spent some time yesterday and that's that's where I ended up. Don't.

C

We also wind up with the lib container and run c yeah mid-range.

D

The uh where are we now bullet rolling forward to the currently fixed version of the advisor.

B

Yeah, so on the on the uh run c thing, uh I push back majorly on uh updating, run c to an intermediate sha um in c advisor, and then uh they came back. They looked at it, they came back and they said oh uh you're, not using c groups v2 in kubernetes. So it's not going to affect you and you know they they add like a set and then in the end it wasn't my call because the change was going into c advisor, not kkk. Sure.

B

Here, I'm saying.

C

That uh isn't that also going to.

B

C

You have a problem with the roll forward.

B

uh Yeah, so that's what the pr that I pointed you to uh from gcp. That updates.

E

B

The c advisor and run c, and then I told gcp that that's not enough. We need to pull in a newer, xcd and grpc.

C

Right, and that also has the lib container changes that again change the interface we depend on and are not actually just secret v2 changes so.

D

Yes, what it sounds like to me is: there are a lot of issues that need to be resolved before we can confidently pick up a new c advisor. um Yes, so yeah. If we, we should discuss this with signate, but my recommendation would definitely be go back to last known good ski advisor.

B

uh So, uh ben and jordan, between the two of you, can you talk to david? Please yeah, because that's that's where this needs to get worked on a little bit more.

C

I'm I'm not sure if this was what jordan was referring to earlier, but I think we should actually go back to sig release outside of this meeting and talk some more about the architectures we're building yeah, there's some development that we've been overlooking that we have a bunch of builds that we just have no coverage on of any sort.

B

uh We have tracking issues for um some of them, and people have been promising stuff and they haven't been showing up. So if we want to remove architectures, then we need to start the conversation and to get rid of them uh on the same side. um You know windows right.

B

What do we test on windows? Why do we even ship windows right? So that's if you.

A

Want to open up that can.

B

Of worms uh yeah, I'm happy to uh you know, start that.

C

Too one of them would be. We have some very limited testing of the clusters, but I believe all the client stuff. We don't- and I think we still ship the client for like 32-bit or something yeah.

B

The hci shim has been a thorn in my back for a while, so I I want to I- I don't want to deal with that anymore either.

B

In the same way, uh we somebody brought this to my attention that looked dims. uh We don't even do any conformance testing for windows notes like. Why is that even uh something that we are allowing as a project right? Why do we have an architecture where you can't run conformance tests uh with only windows, only uh nodes in a kubernetes? uh You know.

A

D

As as fascinating as it is for us to talk about this, I don't, I don't think the right people are on this call.

B

I'm just waiting because you know ben gave me.

D

D

I think we should start that conversation with, like sig release for the build stuff stick arch conformance for the conformance. That's like they're good questions right. Why did they get.

B

C

I agree: that's been another area that uh you know. Oh the tests are coming and we're working on it and it's they're not there exactly.

B

That was a good session ben thank.

C

You how manageable it is to get a ci with hyper threading off. um I'm I'm not sure how I'm not sure, if that's a thing that any provider lets you toggle and I don't know if we have access to any that have it off. But I'll look around sounds good.

B

You've done reviews.

A

I'm just writing.

B

A

Action items on the agenda.

B

A

Yeah, I could do that after the call we don't have to um wait around is.

B

There anything.

A

Else that needs to be discussed or.

D

um This is the big one.

A

Yeah, okay, should we get in 20 minutes early then give folks some time back.

B

A

D

B

Okay thanks a lot: okay, bye.