Kubernetes kops Office Hours, 28 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes kops office hours 20200828

Description

Recording of the kops office hours meeting held on 20200828

A

Hello, everybody today is friday, august 28th. I am your. uh This is the cops office hours. I am your moderator, facilitator, justin, santa barbara. I work at google a reminder this meeting is being recorded, will be put on the internet on youtube and please be mindful of our code of conduct, which uh boils down to being a good person. We do have a fairly full agenda. I pasted the link to the agenda in the chat in zoom here. um Please do feel free to add your name uh to the agenda.

A

I will replace it in case. Anyone doesn't doesn't have it. uh Please do feel free to add your name to the agenda, um particularly if you want people to correlate you when watching the video currently you're back from watching the video uh and because we have so much on the agenda.

A

Please do add um your topics at the correct point, which isn't necessarily the end but uh so like if wherever it makes sense and we will endeavor to get through all of them, um but if we don't have them in there in the minutes in the meeting notes, it will be hard to do so all right. So with that and then.

B

I suggest it looks like uh some uh open discussion items are going in the action items section so make sure it's in the right section.

A

Please, yes, uh yes, we we have a review action items from last time section. I can also make some adjustments or try to make some adjustments on the fly. If I make a mistake, please call me on it, but I uh we do have some action items for review from last time. uh Hackman. I believe you bravely volunteered last time to attempt a release and you are going to update us on the uh status of such said release.

C

So, first of all, uh sorry, if I didn't wait for everyone to lgdm, as the procedure says, but being an alpha, I don't think that's really that of much of a problem, um so a short update, short version managed to do build image promotion, even the github draft. It's ready with the release note and everything um thanks.

C

um So there are three things that didn't work um well, two are related to cope uh and the ds3, your s3 private repo that we use.

D

Well, it's fair enough. Yes, the legacy legacy buckets and legacy. uh Yes, the legacy locations. How about that.

C

um Okay, so those those ones are well. Obviously, I don't have permissions to write there, um but there is still the artifact promotion uh that uh probably you also have the credentials for it being the project owner or something uh working at google.

A

It it's actually because I've been working on this on the binary promoter. It is not yet the non-image promoter, it is not yet there and yes, I do have the it's not yet automated, and I am, I currently have permission to promote it manually. I will lose that permission uh once we get it working in an automated fashion, but yes, I will. I will do this.

C

I noticed a few things: the major ones, uh one there is no tooling section in the release document. So if someone new starts it's like, where is the cluster image promoter?

C

For example, there are a few minor other issues with tools, except for the cluster image promoter which doesn't work on mac, which, for me, was a problem, doesn't build.

A

Please do open an issue if you haven't already done so on that topic.

C

I didn't want to open it until we discussed, I will do it after. So I wasn't. I didn't know if it was intentional not to support mac or.

A

It certainly wouldn't have been a deliberate uh decision to not support mac. uh It may have been that we that I I guess this is linus- uh uh that there's some reason why it's hard to support the mac, but uh I suspect it's just that he doesn't use a mac.

C

Yeah, I think there are some libraries, cis unix, something didn't dig too much, because I really wanted to actually finish the release, but other than that. I think someone that does it once probably takes one hour or two hours to find whatever binaries need to be installed.

C

But after that should be a pretty easy task to to go for them, uh and there was one more issue with image promoter. I don't think the ammo that's generated. It's actually correct.

C

The way it's done in the tooling, it adds a new section with duplicating the names so instead of merging with the previous content as a me and peter saw, that is, for the other.

A

That's I uh so I it currently we have it grouped by release, which I think is the right thing to do. Otherwise, it'll become very difficult to review. I think it works if it doesn't work, it works.

C

It works so just that. I'm not sure that if it's meant to work like this or it's just by.

A

Fluke, it's a it's actually, a really good topic.

A

uh If you, if you generate the manifest for the for the non-container images, you'll see that there are different files for each release, which is sort of like a much, in my opinion, clearer way of doing things and that's sort of so it's sort of like a question of how should we use this, and I think that's a great uh issue to open again in that repo and say like look: here's how we're using it and like does this make sense, type thing and that sort of thing, okay, so lower.

E

A

For that cool and on the topic of the uh the requirements, uh one of the things we could do is we could create a subdirectory with its own go mod and uh we could basically reference the the tools as long as they're go tools and then run them from there. So we would have a known version of the tools as well without polluting the main. The main.

C

Goal by the way, there is one tool written by you uh with me yeah and uh you have to merge one of your pr's there, minus src.

A

I probably can't merge it, but yes, I someone has to prove it, but yes, I will make sure it gets merged. Yes,.

C

Okay- and I don't know- I thought it was in your repo- I don't remember uh anyway. um The feeling was that it's doable so with minimal effort and.

C

Because the image promotion works, uh I already did the change in cops to use the images from the official repo. So for the next release we can skip uh pushing to cope dot, io copaio docker.

A

Hub that could be great yes, because, like the uh the reason, these uh the reason that these still ex so we got to get rid of yeah. The reason these still exist is because uh we haven't yet moved the canonical location, in particular from the cubot v2 bucket um that relies on the binary promoter, but yes doing the image. One would be great, um so it's done.

C

My question was: do we want to do it also for 1 18, considering the docker hub.

C

Money not money, because my understanding is that you actually have to be authenticated. You, as a user, doesn't matter if it's a paid repo or something.

C

So when you pull the images you have to be a paid user or something to not hit the rate limiting it's not per bucket, let's say.

A

Yeah and that's even if even if I were to pay for copio yeah, so there's no way, there's no way out of it. Basically, it is by moving it to oh, yes,.

D

I mean yes, it's correct. Yes, yes, okay,.

C

So people would so there is the small limit where you can pull or unauthenticated.

C

There is a bigger limit where you can authenticate with a free account which is 200 pulls, which is tiny because it's per um I say.

C

F

C

No, it's not per image, but it's per. I.

G

Can find the word for it is now per image, but it counts. Two two of the each image counts for two things, because it's it's done against the request against a certain endpoint. um There is, I believe, a workaround for open source projects, but you have to actively get in touch with them. I can find out from someone else. I know the email address to get in touch.

C

So I I heard something that it could be per layer, not.

G

As they've now clarified it, um I I will find a link and.

C

Okay, but in any case, if you want to pull more so you have a bigger clustered and want to do a rolling update, then you would have to somehow authenticate and pull images like that. Otherwise, you will hit their rate limiting.

C

A

C

Moving unclear.

A

On the back portion, I don't know if anyone else feels strongly either way. Sorry, I said sorry, I definitely like agree on the uh moving 119.. I'm I'm not clear on whether we should backboard. I don't know if other people have a strong opinion.

A

We can, we can think about a bit.

E

We can wait to see what happens. It was just a suggestion.

E

I think sorry, I I think if, if we, if we're, extending the um backwards support and also if there is probably a lot of people that won't be able to do to move to 119 and if they have a large cluster and they do a running restart for any reason, a security batch or whatever they might actually get into troubles. For that.

E

So that's one thought.

A

Yeah agreed, I I think the unknown is the whether dockerhub is going to have some like what guy is pointing out about whether dockerhub is going to have some magic flag. They can flip for open source.

C

Projects and if you can get someone from there to talk to you.

C

That's a challenge times with them.

D

I'm not gonna, say.

A

D

uh Should we continue on.

C

C

A

Thing there is another sorry, uh the release artifacts in github, it sounds like was a proposal. You had.

C

The list of artifacts in github is quite big and, from my point of view, it's unnecessary to link directly images or even node up. So my proposal would be to just have all the cops binaries with hashes plus an offline package that someone can download and contains everything. So it will be much less messy.

A

That seems like a good idea. One of the tricks we're using is actually to use github as one of our mirrors.

C

Doesn't work at the moment?

C

Okay doesn't work because github does not allow you to have.

C

Paths so we have binaries that are right, now are per architecture and os whatever and.

A

Yeah, sorry it should it should work, because we have special logic to detect github in our mirror logic and rewrite the paths whose underscores that's how much we wanted this. How much I wanted this yeah, then I guess I messed it up. Okay or like it wasn't, it has to be published, of course. So it's it's sort of you never know. If it's going to work until it's until it's up there.

C

ah Okay, anyway, because, uh as I did, the changes for arm 64., I never looked at this logic part. So I might have broken it a bit, but I think we only need it for node up so shouldn't be and proto cube.

A

Right, yes, I think so I I think it would be nice if we had a second like in the long term. It would be nice if, if uh artifacts.kate said, I o was bulletproof and reliable uh and we didn't have to do this and we could have clean github releases, but that's why we have these binaries and they get to release right now. It's a second mirror.

C

Okay, uh I will think about it.

A

uh You thank you. You have the next item as well. uh I'm sorry, I want to keep us moving along here, say about moving images from kopio to kate's gcrdia, which it sounds like you yeah. We already discussed it yeah.

C

um I said that I will create an issue about handling the graceful recreation of lcd manager volumes.

C

We discussed that if the volumes fail, you have to do a lot of magic to to get it to recover right- or I am sorry.

A

A

Sorry, I didn't mean to apologize.

C

um So last time we discussed that uh lcd manager cannot handle volume failures in one of the.

C

Cluster members, so I create you, said to create an issue in lcd manager, and I created it uh also, I think um at some point we will have to cut a new version. There is one fix for um open stack and there will be also the multi-arch, and maybe this one. If someone manages to work on it, if not, but anyway, we need the three four nine hcd.

A

Right, yes, it's some version, uh I seem to recall there are more fixes going in every day, uh but yes, anyway, thank you for creating the issue. I will. I have not apologize. I have not yet had a chance to look at it. uh We should update to the latest at cd. As I recall, in upstream kubernetes, there was some debate about like whether we were we were waiting for a fix that just landed or something like that to do with permissions file. Permissions.

A

Okay, no problem, but yes- and I think that would be that'll be great and it sounds like uh rodrigo. You have a related. Oh, I guess these aren't necessarily action items from last time, but I think it's okay rodrigo. Do you want to talk about uh ncd manager on.

H

Stretch, sorry about that, I probably put it in the ring section: uh it's mostly about uh moving uh entity manager out of stretch.

H

It's no longer supported, so I know we're using the hypercube uh image before uh that is discontinued. uh There's no update on it. uh I just created, I started using debian base, which is what kk uses and installed the required mount and other packages on there. So uh yeah.

A

That sounds great, uh and did you send a oh cool? This is pr, yes, perfect! Yeah! That's wonderful!

A

uh I'll! Take a look at this, but it sounds perfect, thank you um and it shouldn't. As far as I know, it shouldn't matter it matter to anyone, because it's not it's a container.

C

I also think that if we want we can move to uh this trollers, it's pretty much the same thing I did for protocube just that people would have mount the right anyway. They have to mount the right volumes to use it, but my thought is that this trollers would work for lcd manager also.

A

Can't remember they have to make fs. Do we have to make of us? Do we have to make make uh mkfs xc4 and is it industrious? uh No, we mount. Oh, the house, one yeah.

C

The host one, so it works in any case in case of systemd, or something it's better to mount the host one to have the same version than to use. What's on some image.

A

Let's start with what uh rodriguez said I propose, and then let's see whether we should go to distress. I think that makes sense. Okay, that's.

A

Okay, thank you. um Eric on the gce test, jobs or eric computer looks like.

F

Well, this is just a quick update uh from last week, uh based on the discussion we had uh there. We did. We were having issues with the service accounts that the gce nodes were running as uh they didn't have access to a state store bucket uh or any any storage uh that so we were having trouble granting the permissions uh so justin suggested to create that bucket inside the project uh and manage to manage that as part of the lifecycle of cubetest. So we got a couple prs in uh actually got some changes.

F

We actually got changes into cubetest, which was awesome, um and um we are waiting for some final like thumbs up and um some tests to. Let's see if hopefully that works, uh if it doesn't then I'll put another patch in that will just set the permissions on the bucket. uh I was trying to avoid doing that, but otherwise we should be. We think we may be able to get this going again for real, so I don't need any anything else to add on that. That's all right!

F

A

That's awesome, thank you both. uh I don't know if I can do permissions on that, I can do the the lgtms, but I will.

F

A

F

Yeah, no we're it's we're not waiting on. uh It's not you in this case uh it's in test and for, uh but uh I'm not sure, you're relentless. I can probably.

A

Ping, some people that that might have permission and say you've been pleased, they've been pretty responsive for us, so, okay, cool, yeah, they're, actually pretty pretty good, we'll let you know.

F

If, uh if he runs issues, okay, awesome well, thank you for doing that.

A

Okay, uh let's move into the open discussion section uh first up uh hackman talking about uh our friend calico v2 and ncd2.

C

Yeah um kind of bumped into that uh this week uh there was a bug where, because of some old change there, it was using an older when you selected calico, when you created a cluster, it was using.

C

Fcd31 something I think, or three two something, um and it's not okay. There is a lot of logic there in validation, that's quite confusing, and it's only there to keep.

C

A version of calico for hcd 2, 4 1.11 for kubernetes 1.11 and that one is already deprecated in 1 19. I would just like to clean this up so that we don't have any more confusions because of v2 v3.

C

It would affect probably a very small set of people that would like to use at cdv2 with kubernetes.

C

Well, that would like to use cops 119 to manage their hcd2 cluster.

A

uh Specifically yeah, I don't know if I I don't know, if there's anyone in that camp, anyone that is still doing so, I I don't think it's unreasonable at this time to say like we should not backport it but to say, like cups, 119 does not support this. I think. That's uh I don't know we haven't actually deprecated it. I have we deprecated calico v2, technically.

B

Well, we deprecated the cops. Sorry, no kubernetes won 11 in 19, so we'll be going in 120.

F

What's the maximum version of kubernetes and calico v2 supports, maybe we can drop support calculab2 when we drop support for that kubernetes.

I

A

I I think peter suggested like if calico v2, as far as I know, if calico v2 uh stops working with, say, kubernetes one is only supported after kubernetes 114 or something when we drop kubernetes 114, we could obviously drop calico v2.

C

You know it's just one 111, that's it the manifest stops there. So there is a manifest from from kubernetes 112 that uses only v3. There is no v2, for here. Usually running cops are in calico itself in cops, so, but we have a fixed calico manifest in our repo. So if you want to move to kubernetes 112, you have to switch from v2 to v3.

B

Yeah and so we've deprecated kubernetes 111 in uh 119., and so it will be gone in 120.

C

Well, I wanted to do it sooner because I think it's a very small, a very narrow.

C

Spy of people- I don't know who would want, and anyone that really wants to do that- can do it with cops 118, for example,.

C

It seems too much trouble to keep the v3 validations around.

A

uh I think one of the things we should talk about is whether we want to start the 120 branch. I do 119. uh like cut the 119 branch.

A

um If we feel like we can do it in 120, maybe that maybe that keeps lets us keep our um our deprecation policy, but I I normally have the person arguing for deprecation like like. We should be like careful and do this, but in this case I feel like it's so so on the edge that I I don't know. If it's I don't know, if it's doing anyone a service to encourage them to keep running category two.

B

On the other hand, the benefits just a little bit of complexity, which, when we ranch it's gone so.

A

Yeah, so if we, if why don't we, why don't we prioritize doing one cutting the 119 branch so that master becomes 120 and then we can we'll get.

C

A

Let's pre-announce it pre-announce it in 119 and then.

C

It's not, we don't have to pronounce it because we dropped the whole 111 and 112 version. So that's already you cannot. Anyway, you run.

A

It yeah it's just like it's a it's just a over communication type thing like just telling people that, if you are doing this, then like things, are going to go wrong. Yeah.

C

uh I will let the release note in 118. one okay, cool or 119, but yeah.

A

Thank you thanks. uh I don't know if olay is here, uh he has a he's. Not are you, gonna represent this cluster auto scaler.

C

A

C

Why not uh so he did the pr it's a nice idea to take care of cluster auto scaler, because it's also kubernetes version related. So it's not a generic add-on like nginx ingress, let's say.

C

So would be nice to keep it in line with with cops versions.

C

And you think that it would go very well in add-ons in the new add-on. I.

A

I I I I'm not opposed to the pr uh I'm opposed to the reasoning. I think we can achieve that uh with cluster add-ons, but but uh I'm opposed I'm not a sorry, I'm not opposed to the pr itself. uh I don't know if anyone else has any views on this. This pr.

C

B

If we it's in review cycle, I've got review comments on it, so it's got an unrelated change in there.

A

Yeah, a more like more like more, like uh you know, bigger picture like I'm sure that oh I will address those review things, but uh yeah.

C

This was just generic if we want it or not, and what the plan is, because I think adding a new add-on is something that we need to discuss in the call.

C

So I also think that if the add-on manager will work as we want it, probably we will move it there and together with core dns and whatever else we can move to make things simpler.

A

Yeah the the the argument, the argument that I would make there is uh we have enough add-ons in cop's tree already that uh we have to have a migration path that works well and so having the having the add-on in the tree doesn't cost us anything and it it helps us like figure out what that manifest, should look like and get people using it. So I don't wanna, I don't wanna block. I don't wanna block entry add-ons on a hopefully imminent cluster add-on solution.

B

I mean it needs configuration, so it goes adds. Cluster spec adds to the api.

A

Yeah, we have to figure out how to make that work, but yes in the in the bright new cluster add-ons future, so but yeah so like it doesn't hurt us to have the requirements to discover the requirements in uh entry.

C

Okay uh and switch for it with enable disable disable by default.

A

Right, uh at least for this beginning, when we introduce it yes, it sounds like we've been like trending towards uh making it more uh required, but I I don't want to introduce it and make it enabled by default.

C

I don't think it will work with exactly every cloud we support, so it would be best to just leave people to say, hey, I want to enable it or anyway we can discuss this at a later time, cool.

A

uh So I have the next topic, which is the discussion, a general discussion of uh cluster add-on operators. I propose that we defer that till later, because there looks like there's some sort of potentially better defined topic, so shall I jump unless anyone disagrees, I'm going to jump to mosh about gce support.

I

Yeah um so there's an issue that we've been tracking and twitter has been really helpful, with um addressing issues that we've been finding, so we're trying to set up a production ready cluster on gce, that is private, so with the bastion internal load, balancer and everything.

I

um I was just curious because I noticed even in the documentation that gces isn't really a first-class citizen yet and cops. We need to enable, like the feature flag and everything. So I was just curious if we even have any sort of plan around it to promote it to uh to be officially supported.

F

We uh I I definitely would like to see it fully supported uh that was partially my goal before coronavirus hit and.

I

So, unfortunately, as far as like go knowledge goes, I don't have that much to actually dive into the code and like find the bugs. I can help with any way I can. uh Testing is something I'm willing to do um so with I mean well, we have an account on gce and we're willing to try and help out with anything we can so at least from that area.

I

F

Definitely we are super happy to have more people using it. I think that's one of the biggest things, but what I, when I meant testing I was saying our automated end-to-end testing right now has been broken for gc, uh so that will hopefully help like the idea is get that.

F

Your situation.

A

Eric you're, I think, you're breaking up at least for me, uh judging by some other faces, I think so too. You might turn off your video um because it sounds like bandwidth or something maybe.

F

Sorry not sure what that is much better: um okay in any event, um yeah, so we're working on getting the testing correct first, uh so that we can create a situation that looks like your private networking uh so that we can have that captured in the end to end test, so that we don't break so that we get cops working for gce uh and then once we have that one. You can make sure that.

I

We lost you again um and I guess my last question about it would be so. I mentioned an issue there 8626, which has been open for a while and we just reopened it. Actually it was closed. um Do we want to create a new issue and like centralize, all the open issues we can see there or do we just want to use this one and continue moving forward like, but.

F

Maybe we can, um we can transfer this stuff out of that uh anything. That's uh like any active action items out of that and make a new issue. Is that so reasonable.

A

Yeah, I'm not sure if anybody can hear me yeah, we can hear you, I think, um yeah, it's reasonable. I think yeah everyone's workflows are different and whatever yeah, if you're. If you have some diamonds, look at this, then if that would help you then that's great. um I also want to make gcp work better uh with cops. So uh obviously, uh so I'm it's just a matter of finding all the time but, um as I recall, like the big blockers are uh figuring out uh the things which are harder to change.

A

So, in other words, should we do the ip aliases, which are um uh the adios equivalent? Is the aws vpc cni? Should we do that as the primary mechanism, or should we use uh the routes thing? I think we should use ip aliases, but that's like that's the sort of thing which we have to figure out, which would be a bit like, I think, even with gke.

A

There is no way to switch a cluster and that's what we want to figure out before we make it uh like not feature flagged anything that would require you to re-build your cluster. We want to decide and get a good decision on if you have insight into whether we should use what gk equals ib aliases or whether we should use.

A

I think what we're still doing today, which is um the the custom routes that would be very helpful.

F

Yeah we, we should definitely steer ourselves towards going the the alias range. The secondary ranges that sort of more this more um native, gke or gce networking um so yeah once we, I think, the the the biggest the primary obstacle to getting this getting that flag off is getting testing working and getting like a green test.

F

Once we can do that, we can start thinking about moving that out of the alpha flag and then, as soon as that happens, we can we'll we can really dig in on this and get this this networking stuff ready for you. uh It's a goal of mine too, so we're in this we're on it.

F

Thank you and one more thing on that is, I think there we could benefit from additional testing beyond what we're going to be able to do in cube test, and so we have that other issue that I brought up last week about extending our and test coverage to cover like rolling updates or upgrades and stuff, and so if we could support those tests on gce, that would definitely help in getting it out of nothing.

A

Okay, thank you. uh I I think I know this number well and I have written it down, but I john do you want to yeah? Well, it didn't.

B

Want to take too long, but you know this has come up on past meetings and I'm trying to figure out how to get this thing landed. So.

A

I I don't know why I haven't taken a look at the apollo dress. uh Okay, I it's not that I I just have not looked at it. I apologize. I don't there's no reason for my not having done so. I don't know if anyone else has any.

A

B

uh Okay, the next item, um so I uh on 9653, which was uh getting nodes bootstrapped with cops controller, I'm wondering what would help other cloud provider implementers to uh to take on a similar refactor in their cloud.

B

Do they want separate yeah? Do we should we file an issue for a cloud provider? One issue for all.

A

I can take a look at gce that would be fun uh unless anyone else wants to okay, the likely suspects.

A

The other ones were gonna, be harder. Well, buildings were gonna, be harder to do in a uh fully secure way. I they will likely rely on ip or something like that. I would guess.

B

Yeah, if they don't, if they don't have a mechanism, then they they might be off better off staying with the bucket, but.

A

This is 9653. Yes, this is uh the the this more secure bootstrapping of worker nodes, using uh cloud assist or cloud specific functionality yeah. I.

F

Can yeah I'll try to take a look at that also.

B

uh Yeah the other one is uh surging.

B

Would other cloud providers be interested in adding support for rolling update surges.

A

Yes, what's required for that.

B

uh You, basically just you, have to uh find some way to tag your instances as detached and somehow detach them.

B

I can definitely help point out what is needed, but I need someone who knows the cloud to actually implement it.

B

So maybe an issue would help. I.

A

Issue probably help, I think uh doing it for an issue just doing what you described would saying what you just would probably help a lot, and then we can, if we do one of them first, we can link that pr to that, so that people have a reference of what it looks like. Even if the details will be a little bit different on each cloud.

B

Is uh gce the uh interested party or is there somebody else? Some other cloud want to take it off.

A

I think the openstack people don't our time our times aren't incompatible from uh with this one. So uh I would guess google.

F

Has people that know it and are essentially interested in using it so.

B

Okay and the other, what I must said is: uh what timing do we want for branching and uh cutting uh beta for 119.

A

Has you already added this, or did you add this when I said it, I had already added it. Oh look at you, I've already. I've already said. I might think that we should think about it soon. I don't know if other people think we should.

B

Do like, uh I think our goal is to cut up a beta soon after the release of kubernetes. So are we close enough to caught up that? We want to follow that goal.

C

But again I don't think beta and cutting should be at the same time, necessarily so.

A

That's true: it's it's about um it's about the volume of uh it's about it's about, having a way to keep merging changes that are riskier and we don't necessarily want to delay the 119 uh so like, for example, if we were to introduce cluster api which uh like, if I were to, if I were to somehow get get that into shape uh or the better example the cluster add-ons pr right. This is a like.

A

It's not going to uh it's not going to ship in the 119 time frame so to introduce it now is probably just additional risk like if we wanted to clear that and merge it somewhere, it's nice to have the the two branches so that we can say. Yes, we can merge this it's in a great state, but we don't want to do like take a a schedule risk on the 119 release.

A

So that's the reason. Probably.

B

The better way to structure this is: what do we expect to get into 119 that isn't in already.

F

I'm hoping to have the certificate issue fixed for 119, which I, which is next on the agenda.

C

But that involves an lb right, so I don't really know if it's okay to merge an lb, so we should discuss if it is or not okay to merge it.

A

We can talk about that next. I think. That's certainly one that we should it's it's a it's a it's a candidate for 119 right, like yeah, and what other.

C

Oh, we can add, also peter's pr, so they are dependent.

B

Yeah we need some fix for the uh yeah for the uh the insert.

B

So, okay, I think that's that's our 119 blocker.

A

That's not true.

B

That's not a lot of.

A

Stuff right I mean, but.

C

We have also arm 64.. There are two lcd manager and also pushing multi-arch manifests for the cops images. Are we blocked on multi-arch?

C

I don't know. If we want to announce arm 64 support, then probably this should be done for hcd manager. It was pretty easy to do it manually um for cops. It's um I don't know.

C

I looked at the possibilities again and my question is which authentication would be supported in the cloud thing that pushes the images we have, something that promotes the images to kubernetes staging right.

A

Builds them and writes them here. Yes,.

B

Let me propose that we uh shoot to try to branch sometime after the next meeting, all right, let's evaluate the next meeting, whether we're at a branch. It sounds like.

C

We're not ready to branch. Now, from my point of view, the arm 64 can be announced also for worker nodes. For now, and we can branch the beta, let's say just before the next meeting. uh Multi-Arch images can be done easily at some point and merged, but we have to know that at cd manager, it's anyway a blocker, and we would have to do something about it and also don't forget about hcd manager and docker, because even if our images are on kubernetes registry at cd would still have to be pulled from the regular docker.

C

Do you ncd manager or yes, lcd manager.

A

Yeah, the right, I'm I'm trying to get that merged into uh the ncdadm uh kubernetes six project, uh I'm having a la protracted battle with the uh cla bot rodrigo. Thank you for updating your email. I don't think it's you anymore uh and yes, the so that's that's one path. There may be others.

C

um Okay, that would be cool.

F

B

So I propose targeting the branch after the next meeting we'll check in the next thing see if we're.

A

And people can think about what additional features we need: 119, okay, okay, which brings us on to nicely onto peter's uh your topic around the acm certificate and basic removal of basic growth.

F

So we had a couple ideas for this last week: uh one involved creating a second uh listener on the elb uh that does not have the certificate on it, so it's only layer four and it and it uh targets the same port. On the instance same api server uh authenticated port.

F

uh It turns out there's there's a limitation with eobs, where you can't have an ssl. You can't have two listeners targeting the same port using different protocols.

F

I'm not sure why that's a restriction, but it is so if we wanted to. If we believe the dual listeners uh option is the best way to solve this, then I think we would need to be using nlbs and there is a work in progress, pr to add support for it, but it's fairly out of date. So it's going to need some updates and it's still a significant change. So it'll require some review, but the author said they're hoping to get it done soon.

F

But if we believe that this is the solution, then uh this I think this should be considered a blocker for 119.

F

Unless anyone has any other ideas for how to solve the certificate.

B

I'm thinking the in the other pr targeting the internal dns name might have to do. uh The only other thing I can think of is we bring back support for static tokens.

B

And give them a static token, instead of a basic.

B

Authentication.

A

Okay, they're considering.

D

I I don't like it.

A

But it's a yeah, the other, uh the other one I was thinking of is we have the? uh What is it? We have? The uh the health check uh cube uh coupe api health check, um so we have that one. Maybe that could open another port.

A

uh I don't know that. Wasn't the intent of that of that part of that sidecar. So I'm not wild about it, um but that's another option like or by some means open, a second port.

A

And the obvious means is that that sidecar, okay where's, where are we discussing, is there an issue for this that we're discussing it on or how much issues.

F

It's in, I believe, the first link in my notepad.

A

9756, okay,.

A

All right: well, let's, let's uh I guess it's evaluate those options and I agree that it is very likely a blocker because it's I think people have used this functionality uh and it is a yeah. It's a regression. It's a behavior that used to work and goes.

A

Away, okay, uh anything we want to add on that, or should we carry on and talk about on the issue? Okay, simone hacktoberfest.

E

Yeah hi everybody a bit of a lighter topic. I was wondering oktoberfest is coming in a month and something- and I think it could be a good opportunity to get more attention to the project and attract more contributors.

E

There's a bit of things that the project could do like having labels, and I mean maybe improving the contributors nd and with like links to the development uh documentation.

E

I think I think it could be a good opportunity to to actually gather more contributors and we could be advertised in the cube in the cops users or dev, and I guess a lot of people can use this as a with their company to justify spending some time on it.

A

That sounds cool. I don't think we're ever going to regret that uh sort of the investment in the contributor experience uh and it's nice to have a motivating thing like oktoberfest, uh like octoberfest, starts in october. Right sorry does happen.

E

Okay, I believe so I think he goes off for a month and the goal is to get five pr's opening projects, so I mean even if it's documentation- or uh I mean it's important well, every part is important. um I could do I mean I'm offering to do an effort to look into what it means and say we could do these things to prepare for it.

E

I think maybe I can even go through issues and guess which ones could be a good starter issues, but I'm not sure I'm the most appropriate person to do it.

A

I mean, I guess it would take.

E

Some some time from you guys.

A

I think it's a good idea to at least look into it. I think, in my opinion, I think, yeah. I think it's great. If you want to organize it. I think like trying to corral people to do these things is is would always be appreciated, and I I do worry a little bit that we're going to have like you know not enough review bandwidth if we have a thousand pr's that get opened, but I maybe maybe we can see- I I don't know who organizes who organizes oktoberfest.

A

It's digitalocean, okay, yeah. If we could get, maybe we could persuade them that uh they should give credits for people that want to hack on do support for cops, and these swag.

E

Everywhere for for whoever gets so, there is a some kind of um price for for participating.

E

I'll try to come back next meeting with something I mean whatever I looked at and maybe I don't know an idea of what or what how much effort it would be. Yeah.

A

The other thing that makes it particularly attractive right now, in my opinion, is uh with kubecon going virtual. There aren't that many like fun things and I feel like kubecon- is also going a little bit more corporate. So like this sort of fun thing is not as uh not as does is not the norm anymore and so like this would be complementary to kubecon how about that uh in terms of audience and and fun. uh So I'm I'm particularly enthusiastic about this sort of thing, but yeah. I don't.

A

I don't know a lot of details, but I I think it'd be fun as long as we can make sure that doesn't swamp our limited bandwidth, and I think that would be. I think it's good, like the we might not have time to talk about it, but the google summer of code program, I think, was- was great in this regard, um so this could be like a fun version of that sounds good. Thank you awesome! So yeah. Let's definitely talk about that next time, please and uh uh okay, so we have five minutes left.

A

uh Let's talk so the release plan for the upcoming two weeks. I will do the um I will finalize the the release of the of the alpha three which needed some like the pieces I needed to do um and then it sounds like there's a proposal for alpha four.

A

uh I don't know if someone wants to volunteer to do that again or someone wants to volunteer a big. No is that you're, muted cyprian that may be deliberate, you're swearing.

C

No, no, I wouldn't do that. I actually like this, so um I'm offering to do it. If someone else wants to do it to try it, I don't mind and I can help, because I already did it freshly. So if there are any takers I can help. If not, I can do it.

A

Don't all talk at once, if the? uh If, if someone is keen to do it, then please uh contact clipboard.

C

It's okay, I plan to do it closer to our meeting so in closer to the next two weeks, so there is still time to add stuff to it.

A

Okay, thank you and I don't think we need any other releases. As far as I know, if we do please we need they were not copied from last time. ah That's my mistake, then, okay, so we need a like. I want to change something.

A

Yes, so do you want? Let me copy, I can I got it. Oh sorry, go ahead, you go ahead. I apologize yeah. Let me talk a little bit about uh the google summer code and add-ons in the three minutes we have left. Then, if that's okay, uh so yeah, so we uh um samtoshi.

A

Oh, yes, the time zones are incompatible uh but uh has done amazing work in google summer of code this summer, uh and uh so the final pieces are to get some of that work integrated into the various uh providers um like a lot of it really is a good fit for for cops in my opinion- and I want to give a big shout out to john who's done excellent work, highlighting the places where it might not be the perfect fit for all users, and I, I think think we are in a better place.

A

Well, we're certainly in a better place. I think we're actually in a good place, uh like it felt like on the more recent there's still stuff to do, but it felt like, in the more recent reviews, we've uh like sometimes she's, made changes to split out our back permissions in a good way.

A

I think we're talking about ways to um basically like reduce the ability of the operator further, possibly in a way that is uh like applicable to more than just uh more than just um add-on operators, but in general, like the very notions of kubernetes security, I'd, say.

I

A

Terms of like can I restrict the images which we pull or which I run in my cluster, and can I restrict the ability of uh users to use service accounts that happen to be available in the cluster which are not specific to add-ons, but are, I think, add-ons highlights them? I would ask people to follow john's example and uh like comment on the not only on the code but on the structural issues as well of the of the of the well there's a work in progress pr for core dns. I think you know we.

A

Obviously I will fix the code issues, but the the bigger sort of picture like issues are are important to fix as well and santoshi is, is able to make changes to the add-ons to address those structural issues.

F

A

uh Yeah and john, I don't know if there's anything you want to call out in particular as remaining concerns.

B

uh Not here so I'll, okay I'll take another pass.

A

I mean, I think I think the the one I'm definitely aware of is like um restricting the images that can run in a cluster. I don't know if anyone has a solution for that that they are running today.

A

um It feels like it would be a good thing to uh to introduce generally like the idea of image policy or something like that.

A

But I haven't seen one in open source, which is a little bizarre, so maybe no one cares.

C

One blocker for 119 would be the helper for authentication. Oh sorry, let me see the pr.

A

Are you two, my addons, are you joining 119.

C

Sorry about 119, okay,.

A

Cool sorry, I was just uh yes, I was just yes. I thought that no it's fine! I was just your pr. uh The cube cuddle. Yes, the general authentication. We've definitely done a lot of churn on that and like looking at where we are and whether um whether we are happy to proceed with it.

A

Okay, well, we are at time, uh if there's nothing else,.

F

uh One more thing very briefly: uh this came up a couple times during this meeting about time zones. I'm wondering.

I

F

It would be reasonable to consider uh adjusting or adding another meeting, that's more friendly to european time zones if anyone's open to that.

A

We can certainly do a.

A

An additional meeting as it were like an optional one, um ah and it looks like uh I think, it's a great thing to discuss. uh Okay, we could do and we could have an issue and we could try to policy. People would show up or we could do a doodle poll, uh which sometimes is an indication of whether people would would show up.

A

I I worry that we sort of split people and like people would have to show up to both anyway type thing, so it doesn't necessarily actually help it's sort of a challenge, but it would be nice to like be able to introduce new people, there's also uh asia, there's europe and there's asia, and I feel like we exclude both and those are basically incompatible time zones with each other.

E

Tonight, oh just oh sorry,.

F

Go ahead uh I'll.

E

F

That we can discuss it more but go ahead.

E

I just think that the day itself is problematic. I mean in europe it's like the end of friday, where you're generally going to have you know some kind of activity.

E

If it was thursday I mean another day would be, I think, just better, even if the time zone the time would be the same, I mean that's just one side of it.

A

That's a good point uh that country goes uh feedback that, uh yes, the this it's not necessarily particularly particularly friendly, timely time hours for california, either or west coast, either.

A

All right, uh and so peter you're gonna open an issue. You said okay, yeah and we can do some sort of poll or something it sounds like that might be you because it might be surprising with everyone sort of disrupted globally all right. Well, if there's uh no, the final call for items, and otherwise we will call time.

A

All right: well, I wish everyone a very happy two weeks and looks like we've got some busy weeks of or busy two weeks of development to do and see everyone in two weeks. Thank you.