Cloud Foundry CF on K8s Forum, 8 Mar 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CF on K8s Working Group Forum Call 8th March 2022

Description

Topics
RBAC-related flakes in envtest tests
The test setup is creating RoleBinding in a namespace immediately after the namespace is created, so it isn’t related to HNC propagation (controllers aren’t running anyway)
Does it take a significant amount of time for K8s to process the RoleBinding and apply it?
See this PR for related context.
[JH] also, flakes in e2e tests run locally
[TD] is it just a symptom of a lack of strong read consistency
https://etcd.io/docs/v3.3/learning/api_guarantees/
https://discuss.kubernetes.io/t/question-is-the-kubernetes-api-strongly-consistent/7861
As with all distributed systems, it is impossible for etcd to ensure strict consistency. etcd does not guarantee that it will return to a read the “most recent” value (as measured by a wall clock when a request is completed) available on any cluster member.
[PC] Progress on the name?
[GC] Losing approver status?
Process for maintaining maintainer status?
What is the process for offboarding maintainers?
[GC] Brand new roadmap board!
[GS] We started the concourse track of chores

A

All right now I've got the recording on so uh thanks everyone for showing up again to the working group form. uh The document is up and I will once again link it in the chat for everyone to drop topics in. If you don't feel like dropping a topic directly in there or cannot, then you can drop it in the chat and I will add it for you.

A

But let's switch over to that looks like we have a couple topics, a few topics right now and we're going to have a few minutes. Add some more if we'd like to give it a couple more minutes.

A

All right: well, while people continue to have things, we can probably get started with our first topic, which has a bunch of details, so this should be fun and meaning r back related flakes and m test tests.

A

um So we have a bunch of notes. Test setup is creating role binding in a namespace immediately after the name space is created, so it isn't related to hnc propagation controllers aren't running anyway.

A

um Does it take a significant amount of time for kates to process the role binding and apply it? There's a pr and then some additional notes about flakes and edd tests, renault glee and some notes about uh read consistency, so anyone that has thoughts go for it.

B

I think maybe this original topic was added by dave, who isn't here. uh I know that he put in the pr that's to essentially do a weight loop. I think that's the pr that's linked, there's I think dave's vr there, which kind of does a little bit of waiting around after we make stuff in there.

B

C

I guess from our perspective I can explain a little bit a thought process. We noticed this pr, uh so basically the pr introduced adventure list in a couple of uh helpers for like creating roles and other things, I'm not sure it's just if it was just for also.

C

Basically the idea is you send a create request to kubernetes using uh gates, client and then there is eventually loop trying to get the thing uh for a certain period, and this, like the intent of such a change in our heads, was uh kind of implying that the operation in case is not synchronous which felt wrong.

C

In our opinion, because like when you send a request to case it returns 200, which is pretty much okay and then you should be immediately able to get the thing and I think like if, if the case serve the case, api is following the http spec. It should be returning 202 with the polling location header if, uh if their intent was to be asynchronous, so we did a quick experiment yesterday and today, basically reverted that commit uh but replaced the eventually with a plane.

C

Get like plane expect get this thing, and we thought this should fail sooner or later. If the eventually is needed right, because you create the object, then immediately try to get it and we run this in a loop.

C

We were able to run it for a lot of times, like I don't know like 50 runs or even more. We hit different flakes. Sometimes the end testing is crashing because sports cannot be allocated, but that's not related, and sometimes we got the 401 402.

C

I don't remember with like a forbidden status, but it never failed to get things immediately after creation, so this experiment kind of suggests that this eventually is not really needed, but also uncovers some sort of uh eventualness like eventual consistency after you create the object, so the object's pretty much there, but it's not in effect immediately, and we think this probably uncovers like two different problems like first of all. Our suspicion, if they eventually is needed, is one thing and another thing. What do we do?

C

Is this eventually consistency with this eventual consistency like do we need to worry about this in production codes and actual handlers, because users might create roles and then they won't be authorized immediately. So uh that's another. I I think resolvable question, so that's our that that's what's been concerning us. I guess that's! That's the reason for this.

A

C

Kind of thread in the agenda.

D

uh What was your um your cluster set up in these tests because, like I guess the fact is, it is eventually consistent um like kubernetes, the kubernetes api will accept rights, but, um depending on how many fcd nodes you have and like how many kubernetes api servers you have and which nodes they're connected to like the reads will like um will take some time to actually reflect that, and I think this.

E

D

E

So it has two lcds.

D

One api server so since that's all local like I would expect it to be very fast but like in a real cluster. I would expect these problems to be worse, probably just because of like networks and propagation between.

E

The lcds and stuff, so the interesting thing was we never saw get off the create fail, so that was 100 consistent, but we did see the forbidden flake.

E

So even though we successfully got the role, the role wasn't applied when we tried to do something that required it so the because the problems happening within the api server so.

D

E

And eventually polling for our client to see a thing doesn't seem to make any difference at all to it.

D

Yeah, I wonder.

E

D

The get after create is because it the client's doing something like client-side with like knowing it should.

E

Exist, I don't.

D

E

Though so I looked into the code a bit even if you've got caching on the um create, avoids the cache and the get reads from the cache, so that waits for a watch to populate the cache um but yeah. We don't have a cache. Yet.

F

Is there a chance that, like you've, write, let's say you have two lcds and you create the role and it ends up in etsy one? Then you get the raw you get it from etcd1, but then, when it's time to use the role for our black purposes, it looks for it in that cd2 or something like that.

F

F

uh It's really weird that you can get the role like what else could need would need to happen if the role is there and what the I I would assume they are back. Code looks for roles in lcd like what's what's the extra step that is needed for the role to be effective, as we say like.

F

I also it's also terrible that we need to know so many like we need to. We need to actually be aware of so many details of how kubernetes works. It would be nice, we just could not, but I guess this is the world we live in.

C

Yeah, I guess this is a possibility like we're thinking that maybe they're all binding, because it's a resource, maybe it gets reconciled by something. But then I was like wait. uh uh F test doesn't run any reconcilers, so maybe you're right, maybe just reads it. I'm not sure. I guess we need to check in the code.

C

I guess this could be an explanation.

D

Yeah I mean I don't. I don't fully claim to understand the internals of what etcd is doing. It's just like on their own docs. They they commit to saying, like you'll, receive events in the correct order, but um you'll there's no guarantee that you'll receive the most like recent, like version of something or event.

C

Yeah we know about this. We found this as well, but we were reasoning uh that in end test, we expected to have only one instance of it cd, which actually proved wrong. uh I might I must say so, maybe maybe that's actually, maybe that is what's happening in the end. I'm still not sure, and I mean what I don't get is why didn't like. I would expect that one of these uh expect statements with the get straight get right after grade would fail like why.

C

Why would why would that never fail and we'll get the uh the problems only later yeah.

D

That's a good question, maybe.

C

F

We're just very lucky- or maybe there.

C

F

Maybe there is some something special about gets that doesn't apply to you know our the other outback, but maybe getting the resources not the same as using the resource for our bug purposes.

C

I think uh we also try to expand the experiment to the e2es like to do it against the real gates to see how it behaves, but I don't think we finished the experiment properly. So maybe we should just run it a bit more to see if we can get more understanding, because I mean I mean in the end. We can leave this eventually there, but it will still be good to know uh what's happening and where the weights and the polling should be and where.

E

Eventually, it's fixed to test it and not the symptom that you could get if you're just scripting against the api.

C

E

I mean it it if we did that thing where we wrapped our clients with a retry layer that would fix it. But then we also found there's a case where that probably wouldn't be a good thing to do so.

E

That tactic we're using for listing resources across the cluster, where we're just going for the namespaces that have any old binding in them, and then we do a list in each one of those if you're forbidden in any of those and you've got a retry loop going on that lasts, like a few seconds, then multiply that by the number of namespaces, where you're getting forbidden and then you've got a pretty badly behaving or performing request.

E

So it might that sort of wrapping needs to be opt-in, for you need to know when to do it or not, which is not straightforward.

D

uh I mean yeah, I feel like even just special casing the role in points and like organ space and stuff, like might just be a significant win because, like even if those end points are slower. It's not like. I guess what I'm saying is like. I feel like the sleep and retry for those particular endpoints, isn't the end of the world because it's not like people are like doing those frequently or they're in the critical path once you've like added a user and set stuff up, it's just in our tests.

D

They are because we're doing that all the time, but in in real life it wouldn't be.

E

We've we think we can't wait for the role to appear in a get. We have to wait for the role to be applied to a user that has that role. I will not apply it be effective for somebody that has that role, which is a tricky trickier thing to wait for.

B

Yeah, I guess the other wrinkle, I would add, is I'm not sure if it's related or not, but I've been noticing a weird failure. When I run locally against kind where we create an org and then we create a space in that work. Yeah.

B

In spite of the fact that we, in in various versions of the code, had two different loops waiting for the user to have permission to list things in the org and also waiting for a role binding to appear in the test code are still seeing 422 errors creating the space, which means that, like we get through creating the org, we wait for the role binding to appear like a couple different ways, and then we go to create the space, and some part of the code is thinking that the word doesn't exist.

B

So you know it's a different user. I think in that, like namespace reconciler, like it's all using the admin client. So I don't know if it's related to that, but but we're still seeing some some strange flakes.

B

There, with with things not being consistent in a way that is hard to fathom.

B

Now it seems to pass in ci, so I don't know if this is just a local thing, but I've seen it crop over a couple times running those kindness.

A

Yeah, this is pretty interesting. I'd be curious to see what else you all turn up, since you said you're going to continue taking a little bit.

A

um I do think it's worth under trying to understand the source of these things rather than just trying to band-aid them, since it has implications of just how we set up tests in general, not just the role binding stuff, so yeah. Hopefully we can get some more information on this.

A

At some point,.

A

Any more thoughts.

B

A

Shall we let us noodle about it and move on to the next topic.

A

All right sounds like we're in a noodle and move on. uh There is a question about progress on the name.

G

Yeah I put this on here. I was just curious. What we were at- I don't know. I see chris strong here if anybody has a update or anything.

H

I reached out to uh to lf legal and I think them again. I haven't heard that just yet um so I yeah I'm, I I will make sure to talk to them sometime this week and see what kind of trade marketing deals we need to do. For that.

H

You know I did a quick uh domain search and it looks like the obvious ones aren't taken, but we can do a subdomain. I think that's fine top level uh but yeah. I certainly hope um you know to be able to reach out later this week and say this name looks good or not.

A

Awesome, thank you.

A

Cool that'll be uh a good day once we have a name that rolls off the tongue like deep roast. Yes, beef roast by frosting, we'll have to cut a special vegetarian.

A

Dear um cool, so there's another topic: that's losing approver status. Question mark.

F

Yes, I was thinking about this because uh we had uh a teammate from sap leave uh relatively recently mario, and I noticed that he's still in the approvers list. So my first instinct was okay, he's left sap, so maybe I should pr to get get rid of him in that list. We off-boarded him from everywhere else, but then I thought actually, I think the status is not connected to the company necessarily and he still qualifies as an approver like so like. What do I do uh does so does?

F

Does it mean that we get to keep our approval status like until we die or like? Is that I guess that we, I guess we don't have a process for losing status, so maybe I should just keep it there until we figure it out. I guess.

B

I would think his cla would need to be resigned right because he'll have like some sort of corporate cla.

F

Yeah, I don't know about that. Yeah.

H

But that's I mean that's kind of a minor, he could sign a or they could sign a um an individual cla for the new employer to sign a cla, but he they would have to sign an individual cla like on their next pull request. But I don't think that would necessarily affect the privilege status.

F

That means his company has, since he's resigned. His company is gonna. I will have to take him out from its own from their own, like companies like it's kind of, I think on them.

C

Right, I think this happens. Kind already saw this in a pr. I had to eventually remove him as an author of one of the companies because it wouldn't get through, but I I.

F

C

If we take, I mean the question is still valid like if it wasn't him like somebody else who just changed companies but continued working in the same area, then they'll resign, the cla and they'll be able to do it again. So I guess we'll still have to come up with something like. Maybe after a certain period, the status is lost or maybe just not lost.

C

Yes, it's up to everyone to decide.

A

I think the general thing is, we don't have a system right now for managing the approvers maintainers of.

C

A

Since it's still relatively new, so we should probably talk about that in general and come up with a system and criteria. For you know if you want to maintain yeah.

C

A

Your maintainer status, then, what kinds of duties do you have with the project, et cetera um and what's the process for off-boarding someone if they are not planning on continuing to work on the project.

I

Yeah I mean a bunch of that should be kind of um we always have pretty definitive guidance across the cff. I just dropped a link into the approver section um from the rules description, but I think we're even what's in here, isn't particularly clear about losing those roles like what you would need to do to maintain that, once um you're no longer actively contributing, uh I think that um we've maybe intentionally left that vague.

I

I would expect that you know if maybe uh it's a just show of judgment um from any approver. If they're no longer active in an area, you know they can eventually opt themselves out of that role because they're not um performing those duties anymore, but you know technically. I think right now that it's up to the individual person with that role.

F

I mean it does say that you are expected to be active. Basically, so, like I don't know how, after how long you decide that the person is not active anymore, we could conventionally decide it's a year six months, whatever.

I

Yeah, I mean one issue I see with uh see if on kate's um work in particular is that it's all changing so fast. So if you, if you're gone for three months and you come back, then your technical context is going to be pretty stale, so you probably shouldn't be an approver. But again maybe that's just worth the conversation yeah, let's tweet, we know we know his github handle. We have some way of getting in touch with him.

F

Yeah yeah, definitely we, I guess I'm gonna wait a little bit and then I don't expect him to be interested in contributing. So I just wait a little bit and get in touch with them and tell them hey. I think I'm going to remove you. What do you think about it.

A

Cool, that's a a good topic of discussion, glad this came up um just for our future purposes.

A

All right! uh Moving forward brand new roadmap board with a nice link in there.

F

Yeah, I created uh a board to act as the roadmap, so uh I'm basically trying to introduce the concept of epics on the open source project as well. So what I've done is.

F

I have created issues that are tagged as epic and they each each epic issue also has another label, which is the label associated with all the stories in that epic and and I've assigned each epic issue to the epic anchor and if it looks funny it's because I'm using the new github project beta thing, which looks pretty neat, but it's different from what you used to you can switch to a more.

F

If you click on the arrow next to the roadmap, tab name, you can switch back to a board view if you prefer it uh yeah, just a heads up that it's there nowadays, so like kind of gives everyone um a clearer idea of what we're working on. What are the next things? We're gonna work on I've kind of established this uh life cycle for an epic, so you know first, first thing an epic should be proposed.

F

Then, once it's accepted the issue stories should be created for it, then, once the stories are ready, we can it's ready to be work on worked on and then we work on it and then it's done so uh yeah, ideally each we would start each of each each effort with a proposal which we can link in the epic issue, so everything should kind of work out today I was at the toc like the meeting before this one and actually uh there's a proposal to standardize uh the project, like the board format for all projects in the foundation.

F

So there's a so that every project is kind of expected to have a board similar to what we already had with like the project status like with individual issues and pr's, etc, and then I wrote them up board for higher level stuff. I did this completely.

F

I was ignoring everything that that thing went but like we are likely to be pre-lined up lined up.

F

I think the idea is eventually for the for the foundation to have some automation in place that, when every every working group gets these boards automatically created for them, uh but I think we should be able to just keep our ones as long as they are close enough to what they what's ex, what they're expected to be um so yeah the I guess the the hopefully like, as we come up with new epics, we they're gonna, show up here and people who have like a long term idea of where, where we had it, which is, was not possible to get just by looking at the board, because the board is like full of details and pretty chaotic.

F

This one is like a nice bird's eye view.

A

Awesome yeah thanks for doing that. um I know. Definitely some people will be happy about having a decluttered view, so you can see where things are going and I think we're actually getting to a point where this is feasible, because you know a couple months ago it was just sort of tons of things were getting pulled out of the pile and done so it was pretty chaotic to even project forward. So this, I think, is coming in at the right time.

F

Yeah, I I tried to reverse engineer the existing epics so like. If you see, please go and look at the epics and uh oh yeah, so the link is in that in the document, but I'm also gonna link it in the chat, and I was saying if you notice any story is out of place or missing or whatever like. Please go on and fix it, because I did it today.

F

um Just I did I did it this morning. I think I've I have all of them, but if anything is missing or if you think uh there are other epics that should show up here then go on and add them. The format should be relatively straightforward.

A

Awesome yeah I'll definitely take a look if anyone else wants to as well, um but it's definitely worth getting this aligned and keeping it updated for everyone to see.

F

Yeah, I really like the new github thing. uh We might eventually migrate the other board to this. It's a bit more flexible. It's got more features, you can add custom fields to issues. You can group issues by those custom fields, you can add workflows to automate transactions etc, but at the moment they don't provide a migration tool and I don't want to add every story by hand.

F

So I just wait until I guess they will force us into it or I guess, if, with one day I'm feeling particularly keen, I will do it manually or maybe I can script it, but there's no straightforward way that I can see. Unfortunately,.

A

See if they have an api for it.

F

They definitely have an api, so maybe I can whip something up, but.

A

F

Know that I'm gonna do that and the next day a nice button is gonna, come out that does it and that's.

A

The way of things.

F

I'm going to wait, I think, for a bit.

A

Cool um yeah well, we'll see. Obviously it shouldn't just be you pushing that forward. So um I'll definitely help you out with that and anyone else that wants to join in.

F

Yeah, I feel as long as ever like as each uh epic anchor or try conquer, oh 404. Let me see, maybe I need to to make it public or something anyway. um I was saying as long as each epic anchor makes sure that, like their epic looks, okay, like all stories are there and everything is labeled properly. Then we should be good, actually manage access, I'm the only um sorry I'll figure out access so that it's public like it looks like. Am I the only one who can see it? No, like you've, you're, seeing it.

F

A

See it, um but if I open maybe.

F

Only members of the org, then, how do I this thing is very better. Unfortunately, so that's.

A

F

Yeah, it's private okay, manage.

F

Public boom try again, it should now be visible to everyone.

A

F

A

A

Awesome, well, that's all good! I like this. It's also something we can uh refer to in the coming weeks as a quick check-in for where things are.

F

Yeah, I could not use milestones for individual epics, because, uh if you add an issue to a milestone, then you have to you can only live in one milestone at the time. Basically, so I decided to keep milestones for releases and use labels for epics, which means it's harder to to to see the progress of individual epic.

F

Although you kind of see it, if you open a label, it will show you a number of closed and open stories. It should give you an idea, but like the nice progress bar is only for milestones, so we're kind of stuck with uh this like there's, unfortunately, milestones are by design uh exclusive. You know one one test can only live in one milestone at a time. That was the only compromise other than that. I'm pretty happy with this.

F

A

Well, we have looks like we have one more topic. If we are don't have any more comments about the roadmap, but definitely check it out cool. uh It is a topic about. We started the concourse track of chores.

C

And just wanted to say that we decided to do it like we've been prioritizing some chores that will make our uh pr turnaround faster, like optimization of the end-to-end tests, etc and, like it looks like uh the lack of uh traditional so say, concourse pipeline uh is introducing lots of pain points for us and also after the the kill, the health check that we had. Recently, uh we decided to just uh move it forward like we wanted to run a concourse with a vault, but now we decided to just run something uh no matter.

C

If we're going to use volta or anything, we just want to move the e-twist to to conquers really quickly so that we can offload the github, the github action thing, the checks. We believe that the checks will be even.

F

C

Quick this way which will improve our turnaround and it has other added benefits like we have a tool to like uh investigate flakes. We have, uh we will be running, deploying and running against a real cluster which we think would be really beneficial because kind might be behaving differently in some cases.

C

So I just wanted to make sure you all know that's happening, and we will make sure that all the secrets so that we can operate and look at this thing are shared with the whole round.

A

Cool yeah, I think it makes sense historical context being we just tried to use github actions to use all the built-ins and that works really well to a point, but not so awesome for the end-to-end tests. We found out in a few other cases so offloading that into a sort of a better arena is probably a good idea, um but we can still keep like the unit tests and all that and get up actions, because they're really simple to run so.

C

Yeah we'll do this incrementally and we'll see what makes sense after we get to the next uh stable point.

A

Cool awesome: well, thanks for uh getting started on that, it'll definitely alleviate.

I

A

Of the pain that we've been encountering so cool uh any last minute thoughts, topics, comments, concerns doesn't sound like it sounds like everyone's either tired or stated with the information they've gotten. So let's sort of wrap this up again, thanks for everyone. That's uh here today we love seeing people here and obviously we've gotten some good chats out of this, so uh just keep on showing up.

A

If you have thoughts that you want to share or questions, hopefully we'll have that new name in the next roughly two weeks uh we'll see, and then uh we can use the new roadmap board to do a quick check-in in two weeks to just see kind of where things are, which should be a fun new topic of discussion, so yeah once again, thanks everyone for coming and we'll see you in two weeks.