Kubernetes kops Office Hours, 21 Jun 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: kubernetes kops office hours 20190621

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

Hello: everyone: it is Friday June 21st. This is cops office hours. I, am your moderator facilitator, just in Santa, Barbara I work at Google a reminder. This meeting is being recorded. What we put on the Internet, please be mindful apply for our kind of conduct. I put a link to the agenda in the chat. There are a few things on it, but we should be able to get through it without too much trouble, but please do put any other items on there so that we can be sure to get to them.

A

If so, I want those so that people reviewing the video can sort of see what's happening. If you want put your name in the attendees list, that is wonderful, otherwise I suggest we jump right in Chris. It looks like you have the first two items.

B

Pretty much so so, just what's reminder like there are lots of requests, which is amazing, but everyone is free to review them like we shouldn't even encourage people get into the orc, so everyone is interested feel free to really like makes me more safe to say this is like improvable and I don't like, and the only review before I go then, which is not how it supposed to be anyway, so help us which helps you do this potentially in the future yourself, and also in that regard.

B

Remember we wanted to like you am I building and everything at UC, NC of infrastructure I also know there is like initiatives to move over to use more Python and go and do tooling itself just because of its like Google Ads itself, like 4,000 lines of code for just dash, which is pretty bad, but just I was interested in how far we are here, just because of sacral nerve ability, which Justin mentioned in the agenda as well.

B

Just you know like it would be good of that to push on pull request away to really just trigger a rebuild, which is easier. If people get sick and everything.

A

Yes, thank you. Yes, so there is. uh There is a working group, Cates infra, which should host these things. I would say it's not yet ready to do that yet, but I am trying to get the ducks in a row in preparation for that, so the script, building a new images I think it can be done done more or less unattended. Now so, once we forget, the mechanics I think we could. We could actually get there.

A

They the big thing from that one of you is going to be figuring out where to how to securely store AWS credentials. But that is a problem that the working group kicked in for has to solve anyway, for a bunch of other things, so for like storage of secrets, so that that will happen eventually, but I'm also going to try to get better at pushing images, maybe even trying to go into a schedule like a weekly type thing. The there are separately, though, is another thing happening, which is. There is a sub-project.

A

Our proposal for sub project, a cig cluster lifecycle to basically do more image building.

A

So previously there were some people that were very resistant to this, but they appear to be okay with it now, so people are jumping on the opportunity to actually like make this a sub project and have it be a sort of community-wide thing and not just AWS and I'm, going to paste a link to that we're not just it was some gcpd I should say: I'm gonna place a link to that in the under your thing, under your under your comment in the in the agenda, but it is Coronas enhancements, pull request, one one, zero, two, so right now that sub project is pretty focused on taking an existing ami and adding the kubernetes components to it.

A

I think we should try to align our work with that, though they're also not tackling the actual publication of the image so they're just tackling building a tool that would add files to an ami, but I think this is a first step towards a bigger effort in my mind. So hopefully we can get cube, deploy integrated into that effort and yes, no great progress yet, but lots of things happening I would I. Would summarize I.

B

Mean just the reminder, so I tend to forget the things I just remember today, after the vulnerability so good to have it on the agenda, at least yes,.

A

And I can try to get it to the point where I basically build a new ami every every week, sort of more of like or whatever it would be, but like more of a schedule thing.

A

Because, probably to get it happening and working group kids info, we have to figure out how to do it first and so. I should practice figure it out and do that I mean.

B

You will at some point anyways, because if I start, you will be the person building the summary of the week by ant. Well.

A

The first step would be double you do it every week and then the next step would get automate it. Yes, oh yeah, I'm, gonna, I'm gonna rearrange they Jen's a little bit guy. If that's alright, to like put your thing after the other. Given we talked about releases I'm, gonna move your CNI thing down just sure after I guess everything, oh well, I hope they're all rusted agendas.

A

Do you mind? Do you mind if I do it? Last or after the release discussion uh can do yeah? You know I'm a time conflict. All right, great I mean I'm leaving after but yeah. Okay. What we can? Let's talk about it now then see you now see and I scalability for 112. It looks like you have a good suggestion guide. One. Take it away.

A

C

For context, sky Tanner have sort of moved from as we're calling them artists and clusters to corpse, and part of that was there's there's already a pull request that Alex raised around trying to get cool fights about the BGP route. Reflection were now spending up some more clusters that would quite like to have a scale boss or existing clusters, but don't need the fill way of BGP route reflectors and for that calico heavily recommend and using tyfa to register on the API server and I know because of the weight.

C

A local sort of when this switched over to recommending a CB is the data store and then back to the API server over the course of the three sort of release cycle, and that's why typhoon was previously there and they're not there and then is their back and albeit set to zero replicas. So I was hoping and to basically allow users the flexibility to set up tyfa in front of their api server as if they want to start scaling above 100 nodes.

C

um I, don't know the full history of that CNI then cops though, which is why I was hoping to run it by someone and see if it sounds like a bad idea.

A

Sounds like a great idea to me: I guess: I only concern would be the should we actually be encouraging the BGP route reflector instead like what it, what how would you compare the two for someone doesn't.

C

The the BGP route, reflectors, there's, there's a lot more complexity there in terms of I. Think different people have different opinions about whether they should so the trouble is. You can configure each node to be a reflector through and annotations in the CR D, and that there's there's different opinions from what I've seen as to whether or not you should be doing that on your masters or somewhere else, and so there's a lot more complexity there in terms of an opinionated way of there's, not one accepted way of doing it.

C

I don't think, am I supposed to you know one nice opinionated wave, whereas with typhus it's far simpler to to just go. This is the best way to do it. You can adjust how many reps, because you want that's, that's easy enough. Great, okay,.

A

That sounds good then, and it seems like looking at the Calico ducts. They they push tyfa and they don't really make recommend that well, at least in me,.

C

Yeah they're hit in a way for advanced.

A

Users, I think, is the feeling: okay, so I guess the taifa sounds great and I. Think. If are you in a are you proposing to send a PR? Is that yeah.

C

Yeah, it was basically just I joined to double-check that it sounded sensible before I start dating the work. That's great I'm, more than happy.

A

This sounds great I, don't know whether we should automagically change it, but I think probably the easiest step would be first step like expose the functionality and then in a separate PR. We could at the behavior because, if probably should change the default, if I don't know it's complicated, but yes, we yeah, let's start with the exposing the functionality and then I, don't see any reason that I don't see any reason not to get that it yeah. That sounds great as.

D

Part of this, would you mind updating the dot? Well, it's like mention. You know that you know when you're scaling here a few, the things you might run into a calcio specific and you know, here's how we're proposing yeah.

C

And yeah, if already in a one pill, request that the dog stop date there by the date to start out well put in a few more along with thanks.

A

Wonderful, thank you.

A

Okay, so I have the next two items it looks like, so they are basically informational things that we did a 112 to release, which I think is good, because it only has one bug fixing it around eni reclamation with launch templates if I recall correctly I, possibly only with terraform I, don't recall, but pretty well, relatively minor fix, but important for people that are hitting it, and we did beta two of 113 and alpha 3 of 114, and there is a new, am I going into the Alpha Channel right now, hopefully for to address the TCP sac panic vulnerability, which should yeah so that hopefully, will get them into stable soon, and everyone will be recommended that update.

A

So look for that, probably at the beginning of next week. I guess you also can just update your your kernel on the instances like it's the Debian update, if you're using that image.

A

Okay next on the agenda is Ryan. Are you here and mixed instance, policies? I am.

E

I just want the I posted this issue yesterday, that's something we hit when you're using mixed instance, policies with cops, something we plan for so we mixed instance policies. So it's not familiar, usually I want to run 40% demand, 60 percent spot and it tries to keep that balance, which is great, except that kubernetes doesn't really know about the changes that EWS mates. So, for instance, you still scale down a couple nodes and now you're, don't don't have the right mix.

E

So AWS will then scale up and scale down a note out of erased G on you, because it happens through the load balancer or through dollars. Going group there's no drain on the node, so you end up with everything getting sometimes which was really bad. If core DNS happens to be on that node, which is how we actually noticed the issue so there's two solutions: the first solution and what we actually implemented was to turn on instance, protection on the ASG automatic instance, protection on the SG.

E

That will stop a to be us from doing the scale and rebalancing so it just stops out. The downside goes. You can get out of the ratio that you wanted right. It's only gonna. Well, you know do not remix when you scale up not during stick not scale down to that.

E

So that's the downside there, but it's a simple change and it just works and we vetted the issues, but we voted feted both these solutions with AWS support as well, and then the other solution would be to implement lifecycle hooks on ESG and then use something like logic monitor. Has a project already to do.

E

Interception of those lifecycle hooks drain the nodes, so either one of those solutions would be better than we are today. I think that it looks like my comment down there: instant using work on the PR for the first burst solution which I look at that's what we deploy minutes good for that, but I just want to bring back this I have.

F

A comment on this: this is the same thing we see with regular aSG's right because the rib area, she is also blue, cross bone rebalancing. So if the autoscaler terminates something it will price autoscaler, so you just turned off the sub process the easily balance. So this is something similar. We do this internally and I. Think Ops has that okay, yeah.

D

I actually think for me, yeah I've done that be the suspend process on an instance group. So that's one way you can fix that aspect, but that's just suspend like we do reap suspend rebalance if you're across multiple regions. That way it doesn't try to spin up new things. But this is this. The I first I was like this is an interesting edge case and, frankly, I didn't know. The autoscaler had added extensions group support which that was great to see. So now we can move to that, but I it does I.

D

Once you said that you knew the mix of spot non spot. It kind of makes sense to have this for me in my opinion, and it's it's an easy enough flag for us to just have available to the user change in.

B

The same case, I can also recommend AWS in the labs, as wonders posit Ori, when I tell you to like they wrote a script, it's just watching the the instance metadata and waiting for the termination to all the drain or you to be able to write on it. This is what we used with ECS you just like automatically terminate the nose to drain it before it actually gets shut down. So if you use spot instances, that's potentially something for cops as well to just tell the AWS API tikkun IDs api, hey!

B

This is going to happen in two minutes clean it up beforehand that you don't run into problems later yeah.

E

This is, this is slightly different than the spot spot getting killed. This student trained perfectly risk enduring on demands as well, which don't get that notification that same notification, yeah.

B

I mean a combination of both looking good yeah yeah.

D

F

Question of the first one: should we just set that as the default for everyone, because everyone's gonna keep hitting the same issue, because it sounds like we all hit the same issue with the AZ rebalancing.

F

Let's make this as easy as we can for users and give them the option to turn it off versus turn it on that.

A

Sounds good if we we at least could do that with a. What do we care, mindful of like whether it's a breaking considered a breaking change or a significant change that we would probably want to do on a dotto release, so you can definitely get into 130 No and trip rave a release. Note if we're talking about like putting into the next 112 release. We should be more careful, but if it's bad enough, we can do that. I would.

B

Think about something like if th geez there don't automatically like activate something new, but if you create a new one, maybe added like I, not a big fan of just changing things that already exist and for people that don't know this is going to change.

A

Yes, that is yes. What we tend to do is if we don't have a field, we can change the default like we can add the field and change the default interpretation of the field as it were.

A

So if previously the default was true, we can change it, so the default is false. It's it definitely requires a release. Note and I agree with you. It's not ideal.

A

The other trick is, as you say, to like just when creating a new cluster to do that and till I change, the value with which we create new instance groups. I, guess in this case- and we also could put supporting 2cups upgrade to like recommend these things. I, don't know whether anyone actually runs Cox upgrade once they get into like a certain level, but it maybe we could stop encouraging people to run cuss cups, upgrade it without actually applying it. So you can sort of see those things yeah.

D

I mean I agree that we should probably do the right thing by default, but also I, gotta, think more about the potential consequences of that so I think at minimum I almost have the code done right, I, don't know if you guys had had done any of it in a cop's. You know way.

E

And so we we hacked in a patch into the terraform after cops generated it just to get it out the door because it was impacting fraud. So.

D

I had it in the cops way almost complete and I was just going to leave it the normal way that we contribute things like this, where it's off by default, and then we can enable it and either in a separate PR or we can have a discussion and I can you know patch the default in the PR once we have it. So why don't? We does anyone feel disagree that that go with go ahead with that for now cool all right.

D

Thanks for bringing this up right and making sure that we all know it is a lot of us are gonna run into it or half yeah.

E

Yeah you're all welcome. We tested a instance per phones really early.

A

Yeah, thank you. Next is a Ryan. You have some I think I reference to your PR about channel updates, which I certainly have on my backlog is hoping to get to this weekend, but I.

E

Just wanted I just want to poke the berries okay.

A

Consider me poked yes, I I just want to think about it. Yes, it looks great I would just need to like really think about it very carefully, but yeah. It looks good cuz, it look yeah, it's gonna. Four people don't know this should support changes that are non version. Changes make sure that they are applied correctly.

A

So, for example, if in calico we have a field to enable the yeah to enabled life I guess and that changes the manifest, but not the version of the manifest Ryan's fix will like basically hash the manifest and say this is a different manifest. Even though has the same version because it's still calico like oh, whatever version bran, it is a different configuration, so that will actually be a good fix that otherwise would make those. So that makes those have changes a little bit harder to apply.

A

Next Mike and Rodrigo asking about 113 zero yeah.

D

That's pretty much it. You know we're a couple: betas in and Justin cut another one. Last night, so I mean it became easily. Once we have the beta, then we should wait another you know a week or so, but I'm not aware of any major blockers. I was more just curious. If other people are and especially students, 115 was cut, you know we should push 113 sooner.

A

Yeah I think the other thing I guess I, just don't know whether we want any of the PRS that are open, like, for example, Ryan's one, rather that we should consider that a blocker for one 13-0, whether we should put that into 114 I, don't know how people feel can we live with it for another release, I mean.

E

We've got a half an hour, deploy system to get around. This I mean we've. We just stripped annotations off to make it worse to upgrade. So we can certainly live. Another release like to have a bigger concern with getting the PR Mike's working on or the PR Mike said. He would do, for instance, profiles for other Sudak you put in. That seems like a bigger issue.

A

Or beat me to one okay, that's great, but yeah yup right! So ok! So let's try to do one 13-0 sort of Midland next week. I. Do you think we should get? Actually the image didn't change? So it's not a big deal, but we should get the TCP sac one out of the Alpha Channel and then yeah. Let's do that and then, at the same time, I will also cut there for 115 zero alpha one and we can bump one 14-0 to beta and look at us we're almost caught up we're getting there.

A

So now that the manual release process is going so wonderfully rich. We go to your next up.

F

Amelia I was just wondering what the status of the automative really processes. I know. You put a lot of work into it. I know you're very busy. I, look like I'm gonna have some cycles over the next couple months and so I'd love to help out that.

A

Would be wonderful, I think, let's think so, the pieces are we need to do a build with that tag release we need a way to trigger that bat, and then we need to promote the artifacts in an automated way, and there are two there are: there are three classes of artifacts: there are the images which go to I think they go to docker hub right now, thankfully, that's almost that's the only one I think there are and that, and that one also is doing best in terms of the promoter.

A

There are binary artifacts which go to s3 buckets and those are that is a work in progress in the working group. Capes in working group case infra to do those, but we actually have a PR and we might want to get that working and then there's binary artifacts which go to github, which I think we want to keep, and we don't really have a strong plan for how to do that.

A

Yet I think honestly, if we got if we got the, if we got the tag, if we got a tag build going in prowl, so that we could do a tag and out would come the golden wineries. That would be a great first step maiden, will still promote them manually, while we're still gonna support them manually regardless. But the process will change the that. Then, then we don't worry as much about like how do we know the tag is valid.

A

Well, good, the in other words, we want a second layer versus like someone who github account got hacked versus yes, yeah. The the process of promotion will looks like being.

A

We will do another pull request, stating the DNS controllers, the energy image so again, controller 116 zero has this Shah and we basically send that to a send a pull request to Yama file and that the committing of that will trigger the from image promoter to bet, and we can do the same thing for for the binary as well. Okay, let me start with the tag. Release that'll be great. Yes, we I mean you know.

A

We have like continuous builds already, so hopefully it should be a matter of figuring out how to get that continuous, build to not build the CI build.

D

A

D

You thank you Street is we have someone in the room right now who is just on the 115 Chris.

B

So he might be a.

D

B

Do it but I'm also volunteering to help rebuilding, but at his release, tooling or in the discussions with them as well for 116?

B

So let me tell this well I'm gonna know, but as I was like again 4,000 lines of fresh I'm working with a lock-down Google Cloud account and they built a bunker image, push it to Google container registry, pull it again, clone the release channel and then push to github using a encrypted secret in Google Cloud, and so this is how kubernetes does it so I think we are very fancy already in comparison to the mainline you.

A

Know if they've run that in Google Cloud builder, if they run it in prowl, it's.

B

It's Google cloud build okay, I've been in branch management like it's, like I can take four thousand lines of code thing. You are unable to run this by yourself, like I spent like almost a day to try to build to Banaras by myself. My own Google Cloud account with the same tool link. It is impossible.

A

That's reassuring. The the big advantage of Google cloud build over prowl is be, there's always concerns about the security of prowl. So we pour I. Don't want to do the actual binary promotion from prowl, but I mean we can probably figure it out, but anyway, like today. That's pretty why they're using Google topo. But yes, then the secrets thing is a whole nother thing.

A

Alright, but okay, first step tax and then we yeah, okay, cool yeah, that promoter process is never gonna, be easy run from another account, but yeah we'll see all right Daniel. You are next up with a meta question about sorry, our CID effort, yeah.

G

I'm trying to understand what yeah, what what the goal is, is it to allow cops to the store stake in CR or is there you know some sort of cops controller coming or that's an idea and and I also I guess maybe a related question. What what is the cops API server and is it? Is it usable today or you know what yeah I I don't know? Sorry, it's kind of a.

A

Newbie question: it's not a it's, not any of the questions. Everything to update on this so cops server, which is actually a couple API server. Yes, what the the goal of that is.

A

If you have a kubernetes cluster, you should be able to use the kubernetes api s-- to manipulate the cops to store and manipulate your cop subjects instead of sort of having to ping from ping pong through s3.

A

It should allow better. You know our back rules and things of that nature and yes, and we can have a controller. That applies these things rather than having to run that binary on your machine or in your CI system.

A

The project I guess, kicked off I guess years ago. Now, at the time it was aggregated, I created. Api servers were the thing. A great API servers are no longer the cool thing and the writing is on the wall, and so CRTs are now the new cool thing, the new hotness and they are a much more a much easier interface for users to use. You don't need a separate API server. You don't need any of this. You just register your CDs, so we cups already moved to like real API types.

A

A long time ago like we did even before Cup server, but they are real API types. I think we changed the API group so that they can be registered as CR DS. Today you can, and so yes, we, we will support controllers from there and there are two goals. The short-term goal is to enable cluster API, so I don't think it's all merged, but on the, if all the PRS go in, we should be able to given an existing cops.

A

Cluster create a new instance group, CRD instance in that cluster, only on GCP today, but create it, and it will like basically spawn nodes from there without having to run the cop CLI. So it's it's. uh It's introducing cluster it guy into cops, not for masters not for new cluster bring up, but just for that and then the sort of longer roadmap is full management of of cops clusters. Optionally from that direction as well, so you can still use the the CLI tool.

A

But if you want to manage you know a large number of clusters, you can and you have one cluster first, you can like create those CRD objects in management cluster and have other clusters spawn off.

G

Okay, thank you, I, don't know if there's time for a quick follow-up on that, it.

E

G

I, if I wanted to right, you know sort of the the the dumbest cops controller possible today, something that would I mean is now. It sounds like to serve to support the full range of cops operations. Today, I still need to invoke cops as a as a CLI is that is that correct.

A

You do I believe today, ish, you can change your state store to point to a API server, but it might not work. But that's where we're going.

G

Well, I, okay, so so the state the state might be in CRS, but I still need to invoke. The cops feel lied to to actually oh my.

A

The dumbest one possible would be to effectively watch those watch, those objects and we trigger the same logic. The really demonstrate would be exact, cops right right so can we can presumably like there is a separation of the CLI rapper to the commands that it runs. So you could like, in your controller, start to get smarter and I. Think that's what we've started to do in I'll, see see you on the PR or the I know. There's Mercia, but I'll see you on the PR, where we do that.

A

First, watching of a of a cops instance group and create a machine deployment from there, I see.

G

Okay but but but but today is the dumbest thing possible. That would work would be a executive copy, yeah.

A

It's tits and it's so little a little fussy, because we still write some keys to the s3. Well, the the machines that come up still source their keys from a B s3 bucket by default.

A

So you would still, if you wanted to do it from scratch. You'd still have to like generate keys to the s3 bucket, and this is what the cluster base or VFS cluster base I think there's a field there, which is supposed to enable that to be split but you're at the edge of what might or may not work. At this point, okay,.

G

But but if I, if I yes, so if I wanted to write a dump controller that just you know, looks yeah, looks at looks at CRS, execs cops and I would go, maybe have to store some.

A

State in s3 yeah be the one which would definitely work but might be a little bit distasteful, but as a proof of concept watch, those CRS cops call cops, replace, which then effectively writes them to s3 and then call or then exec cops.

A

B

Wasn't even our PR to actually like make the commands available as a package to implement them into your own code. I, don't know the status. Is that but I I know I've seen it there.

A

Is there is anybody I thought they also were there I thought there was a separation already so like the the CLI commands already call into a package, and there is more logic than there should be in those CLI commands, but you can already do most of that. But, yes, I, think. The problem with that PR was I was I was very happy to see that PR accept because it moved everything at once. It like immediately went into rebase hell and we could never get out of rebase health I. Just.

B

Remembered being there but I didn't like it, I remember that as well yeah you want to do it move one at a time.

A

A

We definitely I mean it would be great, but yes, I, think if you don't want to end up in rebase, hell move one at a time, so yeah Danielle. If you want to like consume one of them, then I would move just cutting it. I.

D

Was going to stay- and you know commenting on this further, the CRB and all that stuff. We did talk about this acute Connie, you it so if you guys haven't seen, I'll actually put our slides in the zoom channel and on there, but yeah. If you guys want to take a look, we went through some of the CRD stuff and how that fits in with cluster yeah.

A

And it looks like a quick follow up or quicker follow-up and Daniel about what happens when cops crashes in the middle of writing state or plant industry sources, uh-huh yeah, the answer is, you can be left in a partial state. You shouldn't neither s3 nor GCS, nor the coop API server will let you write like half a file but and we're sort of careful not to be too weird. But, yes, you certainly will be in an odd state and you should run you should wrap your computer run. You should reconcile again.

A

You should run cops, update, but cops updaters, idempotent or supposed to be idempotent survive difficult choices, not unimportant. So the answer is, you should run it twice. You should run it again to get it from a weird half state into a full state.

G

So sorry, I I think I'm, it's not you. It is I, it is I'd opponent here.

A

You you, it is idempotent, so you can run it twice. It's just like if you get halfway through, if we're doing like a sequence of updates and we get halfway through honestly, it's not clear what the what where you where you are and I guess it depends on the nature of the change you're making and how disruptive it is, but for for normal changes, it's okay, but yeah, like I'm, showing of an example when it would like be not good.

A

Like that, for example, if if we supported changing CNI modes, which we don't, we don't really and you switched from one which used like the V excellent one which used IP IP, we have to make a supporting firewall change at the time.

A

So if we, depending on the order, we do that like until both are done, it's not gonna work, but at the same time, until both are done, it's not gonna. It's not gonna work. Your nodes are it'll talk to each other until that's fully rolled out, you're gonna be weird to say anyway, it's it's better to so it's better to finish it, and then we also have the rolling update command, which will like.

A

We currently only changed the the specification on the a diverse, auto scaling group, and you have to do a rolling update to get it actually apply to the instances and same story there. It should be idempotent if it doesn't complete, you should trigger it again to make sure everyone reconciles, but that should actually fit pretty well I think into the controller loop.

A

Anything a controller is better at that than then people who might forget to rerun it on error.

G

Okay, so so, if look mostly idempotent, but you know there, there might be, as always, some some edge cases, sure I.

A

Like about impotency is a bug partial trend like if you're interrupted halfway through the it should be okay. But we don't test every partial scenario and then.

C

A

Sorry this one say something: sorry buddy spoke over someone. No okay. Would you think oh cool I'm Mike, it looks like you put the CR DS in cluster effect. You can talk Lincoln. There was anything you want to add their work. No I was just having that on cool. Thank you for doing that. The last item on our agenda ryan, 149 city manager, to as measured for two notes, yeah.

E

I just wanted to call the that's, been an issue. That's been sitting on that city manager for a while and I. Didn't want us to forget about it for 1.14. Would you put a version of its need? It is now supported in 1.14. You can expose metric sports to the nodes.

A

Will be a big that'll, be great! Thank you for attention to that one. Yes, we also have to pass on those PRS.

F

Since we're talking about a TV manager, I have a question for you Justin. Would you be against bundling in SUV's manager, CTL on to the SUV manager, image I asked.

D

F

Because I want a way to be able to rapidly remediate an issue boost and right now, I'd have to go and build that binary and find out where it is and keep the same version and whatnot so I've been meaning to put a PR and I just haven't gotten around to it. But I checked I, checked in with you before I do so yeah.

A

I think that would be wonderful to do. We we talked about putting it into the EDD manager, the container image we did about having it as a github download as well. Oh good, I, don't know if there's any value in having it separate. I can't imagine there's that much value, because anyway, I think, though, was one one or either both of those as a start would be good.

A

The if you want together. That's wonderful! Thank you. Otherwise it's definitely on my list but yeah. If you do it, that's wonderful what I created on faster and then you can run it from your local machine and point at the the same @cd state. So I guess we'll call it, which is an s3 bucket yep, and it shouldn't be that important to match the exact versions. But it's pretty good idea. Yeah.

F

Okay, perfect I think Ryan has a great talk on it. That's waiting as a PR cool.

A

I will look at those PRS. Thank you for that nudge.

F

B

F

Question in a previous topic so allow back I believe Gumbel had I, don't know if we have anything else in there. We.

E

F

A while back Gumbel posted a PR to do faster, cluster, rollouts or rotations I believe we kind of put that off, until instance, life cycle. Yes, we.

A

F

We're not going to completely be able to adopt instance, life cycle for everything, especially not for existing clusters and things like AWS and whatnot. Do we want to maybe revisit that PR or something like it.

A

Yes, so we is his life cycle, you mean like cluster API, yeah, closer yeah, sorry, yeah, okay, yes, I do hope we do get there. But yes, as you say, it's it's not imminent. It's not like happening this in 115 as it were, I woke up. Some 15 I think we did change the timings so that it's there's there's fewer. Like big synthetic delays. You know that's gonna, be more to teen as well that the big change I think would be to support parallel parallel rolling. You.

A

I think if we did I think if we did that, maybe we could like scope down the change a little bit to try to do some of some of these things, because if I recall correctly that the the reason why the PRF and gamble became so large was around the edge cases around what happens when you are interrupted when I was in, you are interrupted effectively. Mm-Hmm.

F

Oh trying to yeah.

A

F

A

Beer for that so I think I think if we, my money gut, feels if we, if we, if we did, if we terminated more than one instance in parallel, mm-hm or actually that the first thing to do is for people to have more than one instance group to run those instance groups in parallel and then the second one would be to to do more than one node in our instance group. At the same time, I think.

F

Maybe start out with multiple instance groups. Yes, that one feels much safer.

A

We have an item being live at it. Daniel you wanna.

G

Question this might not be relevant or any more I, don't know, I, probably everybody's already on b3, but I'm just curious for the v2 to v3 migration. Did you stop all right and wait for the rap indices to converge before starting the migration there? There is a there there's an issue, I guess, with with keys that have that have Leah that leases? Yes, sir, if you, if you don't, do it there, there yeah there's some there's some some corruption that.

A

G

A

So we avoided that Oh a not so, basically, the the long and short of it is STD to death. Three migration does not work. If you have multiple, if you have AJ, if you have more than one note, just doesn't work and we did not use the this approach also is has problems the one that described in this, and so we did a different approach where we effectively. We do stock all rights, and we do that by moving sed onto a different port because it seemed he doesn't support that.

A

We then moving the whole cluster onto a different port. So basically no one knows where to reach it, because they're all configured to talk to different port and then we dump the backup and we restore the backup when it is at cd32 XE d3 when it is compatible. When is snapshot and restore compatible. We do that, but when it is not ie be at cv-22, sp3 migration will restore it into a temporary at C, D, D, D to cluster and then load every single key and restore it.

A

I can write them into ed, CD 3 manually and which is basically what that migrate stripped is doing, and then we restart the API servers and then we also bounce all the clients bounce as well. So that should be how we get around this I think, but it also helps that we do this as part of an update. So we also typically bounce all the nodes shortly after so I think that's how we own the other thing is it's not because it's not a live upgrade.

A

We we blew all the API servers have a hard disconnect. So that's how I think we get around this, but we can talk more about this I guess somewhere. If you want to to worry about it, but you know my plan for stdm convergence is that we try not to do this in at CD ADM. We don't. We just never encourage any form of Ed CD migration. That is not that is of this nature. Ever again, I, don't know you feel about that. Yeah yeah.

G

Thanks for the explanation and sorry to have you to.

A

Make your relay now I mean so the hope is that we basically like with that CD manager. We get him on sat on 73 and then I I. Don't want to put this coat in this video I want to like, let's, let's just not ever do this.

G

Agreed I, don't I, don't think it's it's it's I! I! Don't think it's that well understood because it does. You know, there's this TTL issue and there's there's this minor note in the STD docs. But it's I, it's it's really! You you're sort of on your own I think to figure things out and.

A

Yeah I think I think she tells us only events, though right. Oh no, it's leases as well. Oh.

G

Yeah AJ right, you have, you, have the right, the scheduler API, your scheduler and controller manager and whatever else I mean I, suppose you know it could extend to just about any class of applications. That's doing leader election through three at PD leases.

A

Yes, and so the good news is because it's uh because it's a lease, everyone loses their these anyway, during the upgrade so great but yeah. So that's I think that's, basically why we get rid of right. We get around it, but yeah the whole. The whole thing is not the best, but but in terms of how we do it to get to a consistent like it looked like they tried to do it on. No, they suppress it anyway. The way we do it is we basically bring the old data into the new cluster.

A

The new cluster is already a che or like multi-member, and then we write data into it. So, if we believe xv-3 is stable, then nothing and nothing bad can happen to the sed three data, because we are just interacting with as a client got it. Thank you. Thank you. There was a little if you believe it to be stable thing, but yes, all right.

A

Ryan you asked: should we cherry pick those faster rolling updates, we're gonna call time a couple ribs? Should we cherry pick those faster League updates to 113 and 114 I? Don't know uh I will say I did that I seems to be working fine for me. I got some pushback on the PR when it was first merged around like how do we know this is safe, so I also we've.

E

Been running it with the one we have those same timings set in our boys, scripts for with 1.12 and we've had no issues.

A

Well, why don't we I I can't say yes, because I felt like I got it in by saying we're, not gonna cherry pick it so, but I think 113 is probably I'd say. The answer is probably no, because we want to get that release if people want to tree picket 114 I would be supportive of that. But it's not like it's it's.

A

How do other people feel.

A

Okay, one chord seems fine, okay, well yeah, if you train, if you didn't traffic to 114, I think it sounds I think it I think it's good. We can always. We can always revert if we have to, but but logic it should be sound. The logic seems pretty solid, so right.

A

All right, if there are no other items,.

A

Then uh mm-hmm excuse me, then I wish everyone a very happy weekend and I will see everybody in two weeks.