Kubernetes SIG Cluster Lifecycle, 1 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Cluster Lifecycle - etcdadm 20210201

Description

etcdadm office hours

Agenda: https://docs.google.com/document/d/1b_J0oBvi9lL0gsPgTOrCw1Zlx3e7BYEuXnB3d2S15pA/edit#heading=h.e59q52mi9zxu

A

All right welcome to the etsy adm office hours. This is february 1st 2021..

A

uh Let me share the the doc link in the in the chat. uh I see uh nate taylor uh nate, I think uh you reached out. uh There was a there's, an issue that we talked about. um It's great to see you in the uh in the meeting. I don't know if you wanted to talk about that issue. um I I I don't I don't. I don't have anything uh specific in the agenda. I don't know just in a few.

A

If you have anything um so yeah nate, do you wanna you wanna, take it away, you wanna, introduce yourself or uh yeah.

B

Sure um nate taylor uh very amateur go program like not very good, but I can read it and make my own sense and sometimes add things um looking to use std adm uh with my company in order to help with the deployment of our on-prem kubernetes clusters.

B

First thing that I had ran into was the difficulty of getting packages.

B

One option is just pushing out a tarball, and I was was hoping to try and use package management and so submitted a pr to ignore uh trying to grab a package if it already found it.

B

Then daniel pointed out to me that uh it handles cleanup too, which I had missed when I originally started looking at everything, and so after talking um it, it seems like phases could be a nice way to be able to handle things skipping. The cleanup phase skipping the fetch phase, things of of that nature.

B

So there's there's an open issue for it.

B

I am definitely interested in helping with it. um My biggest concern is is is times uh I have a hard time finding free time for this kind of stuff, and with my my lack of experience, I imagine it would take me quite a while to implement phases, especially when we take a look at cube abm and the systems that they use to implement them.

B

It's definitely more complex than anything else. I've done in go.

A

Thanks thanks for uh yeah describing the the issue so yeah, I remember we we, we talked, I think in in slack or on the issue. Also, I'm gonna, I'm gonna, pull up that. um I think there was a pr I'll I'll pull up and put in the doc in a minute um yeah we talked about faces. I I wonder uh if you know maybe there's something uh something some smaller change that could you know that that that would be um that would sort of unblock this um justin? Do you?

A

Do you have any thoughts like right now, ncdm is um pulling or the cli right, the cli it. It uses a dedicated cache uh and it looks to that cache to a folder, a specific folder. I think under var cache at cd adm for a for a tarball that corresponds to the ncd version that you want to use if it doesn't find the tarball it attempts to download it from, I believe, github uh lcd releases, and then it writes that the tarball in that directory- and so then, once once it's there, then it it takes.

A

You know it extracts the artifacts from there puts them into opt, um uh opted cdm, uh and so I think uh yeah it doesn't. It doesn't support sort of uh a user, a user provided at cd, um but it could uh because even even today, at cddm it doesn't, it doesn't really rely on the location of the file right it it it. It uses it for a particular location. uh You know to run ncd, but but I believe that um I think this might happen actually during install.

A

It will actually run at cd to verify that the version that it outputs is the one that it's looking for right. So uh it's conceivable that a user could use or could could yeah could place their own uh lcd.

A

um You know and and tell it and where, where to find it um and ncdm would be fully capable of checking. Okay can't is this is it you know is this? Is this the binary that I expect and and that's it um but yeah, but there's that issue of uh of of cleanup um yeah and uh in.

A

In in in theory, does that does that sound like? um Is there something yeah? Is there some smaller change? uh You know other than uh phases that will um that will sort of unblock that I think faces right. Like nate was saying right. We, it would be a way to um allow for uh for for cleanup, um I'm sorry, I'm a little yeah. I I I'm kind of cloudy on on what the cleanup issue is.

B

So, um a little bit uh more, I had to pull up the code, real, quick to remind myself when you do and in it it does some things to to make sure that it doesn't need to clean up an existing situation, um an existing deployment, uh but when it when it gets to the point of looking to get uh unpack the cash and everything like that, uh what it does is it looks to see if the cache already exists. Grab it.

B

If not, and then unpacks it into the install directories that you that you can specify the default is, is an opt bin. um It does not um that that logic is not um skipped if the std uh binary, the std command is, is already available.

B

um One thing that I had done is is in a temporary environment, where I was testing around. um I had compiled uh what I had submitted as a pull request, and I have I specify the ins folder when I do the in it to point to where the package management installs the binaries uh when it comes time to for a cleanup.

B

uh What happens is that scd-80m attempts to clean up? But if you don't specify installer, it just looks for the default, and so it skips cleaning up the binaries which has worked for my needs. But obviously that's not that's not a real solution, um but that that's just some some further information about it.

C

uh One thing which might be helpful would be uh to differentiate between like the install and uh instances of a service, so, for example, like kubernetes at least historically has run uh separate lcd clusters from main events, and those can be on the same machine, um and so we can sort of imagine that there could be uh a trivially like three uh lcd installations on a single machine. There could be like the system, one that you installed using like packages or whatever right, like the like.

C

If there's a debian package for for etcd like maybe install it and bring up lcd for something completely different outside of kubernetes for the sake of argument, um and then I might want to run etsy adm to bring up a main cluster, I'm gonna run it again to bring up an events cluster, and so now, when I, when I go to to remove, say the main cluster, I don't want to delete the events cluster and I don't want to delete the software that I installed or that I have cached somewhere.

C

um So I don't. I don't think that uh in general it'd be cdm should be deleting uh the binaries um unless it knows that they are specific to a an instance right. So one like. So what would that mean in practice so like, for example, underneath atdm? We could have opt cdadm main and then bin, which would hold like the current version, and we would literally copy them from the cache and then we would know that they were owned.

C

But if they were in user bin, for example, we would we would not be in a delete or if we would not be deleting them, and if we were getting them straight from the cache. We would not be deleting them, because we don't know if someone else is using them. um Similarly like. If we have to uninstall a system d unit file, we would not. We would only install our system d unit file, uh we wouldn't uninstall.

C

Like events, we wouldn't uninstall the system, one unless you had actively pointed it to uh like to the shared like the primary one, that the system installs or to I don't know some flag to say, like just clean up everything.

C

um I don't know if that helps, so I like, I like what you did with uh changing the binary paths. I think I'd argue in general, we shouldn't be deleting the binaries unless we know that they are in a location. That's specific to our instance of ncdm.

C

Does station have instances today, daniel or not.

A

C

uh The the the notion that there might be more than one yeah, I see what specifically adm like maine and events, for example,.

C

A

So I think maybe I do not quite understand.

C

uh So, for example, uh I need to look at exactly how what I specify on it. At the end, it's been a while.

A

But yeah it is. It assumes that there's just one uh that it manages for exactly one fcd process and that ncd process is uh is itself. uh You know, wrapped in a system in a in a single std, systemd service.

C

Yeah, so we could, we could add the notion of uh well. If we want to continue to I I don't know whether the upstream project is going to continue to recommend or whether even still recommends splitting main and events into several sed clusters, um and we could. We could certainly add that notion if, if they do assuming they still do, we could add the notion of uh like everything default, but just like a second instance called events right or a second instance called psyllium actually is the other use case.

C

uh Psyllium the cni provider uh or network magic thing uh also needs or can in some modes uh you can provide it with an std cluster. So there there can be actually three instances of uh lcd running on a control, plane node in certain configurations.

A

It's it sounds. It sounds to me like what you are saying is that the the the problem that nate has raised, uh which is of hey, uh you know a different. You know different, uh let's say different different uh source, for let's say the lcd binary and, and you know avoiding cleanup if you, if it's adm, if ncdm didn't didn't install this, it sounds like what you're saying is that that is a specific instance of maybe the more general problem of hey. There might be multiple instances of ncd on a machine.

A

That's that's a valid use case. Ncdm should support that if it did support that it would automatically or yeah it would necessarily support this. This use case of um of um well, maybe not necessarily, but it would be, it would be a step in you know in in that direction of supporting um different uh yeah, different or multiple lcd, binaries.

A

D

Prakash here I just want to uh give comments.

D

Can I so my understanding was that kubernetes only deals one cluster at a time, and if it is one cluster, it will be only one etc. At a time, however, the responsibility of that goes to cluster api if they want to have multiple clusters and manage it. So it's so is it not that it should belong to the cluster api to invoke those extra hdds rather than doing internally.

C

uh I can I can take this, like certainly the uh the originally. There was one lcd backing at kubernetes, one instance of ncd backing a kubernetes cluster and then I think for performance reasons back in the ecb two days, uh the default configuration or the configuration in the open source, kubernetes communities project, uh the cube up configuration uh start split that out into two.

C

This is like a very long long time ago, split that onto two lcd clusters, and the idea is that the event objects which are sort of higher traffic uh or like higher throughputs would go in and actually like, don't matter. uh um In other words, if you lose uh and that's if you lose an event object in kubernetes, it really shouldn't matter they only last.

C

I don't know like an hour anyway, then they get garbage collected. um Those would go into a separate lcd cluster, uh and so they were every every kubernetes cluster that was brought up in this configuration uh and then the ones which copied that- um and there were strong reasons to copy it, because that was sort of the tested configuration uh those cl. Those kubernetes clusters would have two lcd clusters backing them.

C

They would have one for event, objects and one for everything else and the kubernetes api server, for this reason supports a flag or a set of flags on the command line, where you can say like these kinds of objects, or maybe this lcd path goes to this scd cluster. So there's no guarantee um in kubernetes, for example, that all the objects in your kubernetes cluster come from the same lcd cluster, and so uh the most visible effect of this is.

C

If you look at the resource version on events, it will not be in the same sequence as the resource version elsewhere. It doesn't matter it probably doesn't matter, but uh like there's, no guarantee that across kind on two different kinds, the resource version will be the same because they're not necessarily the same as cd cluster, and you can actually observe that in most configurations or in a lot of configurations, I should say if you look at event objects.

C

uh I do agree that whatever tool is setting up, um the cluster should also be setting up etcd in general, unless you want to do an external lcd cluster, but in general the the tool, whether it's cluster, api or k, ops or kind or whatever it is, should do that. I think what we'd like here is that that all of those tools should use scd-adm to actually do the heavy lifting of setting up ncd.

D

Okay, it makes sense, then.

C

And yeah, like I think I like the way you expressed it daniel like if we, if we address the broader issue, then we uh like nate's nate's, um the the challenge that nate has encountered. It can be seen as an instance or related to that. I think I think that what's different is uh an instance of etsy of an fcd cluster doesn't necessarily own the binaries. It is using in any case.

B

I do believe that um working on support supporting multiple instances would in effect solve the use case that I was running into, um whereas instead of on doing it in it, you know just immediately going and unpacking the cash. It makes sense to look to see if it's already done, in which case that covers my use that case and and the cleanup as well.

B

You know ensuring that we're not always clearing up all binaries that that, so I I believe that it's it's a much better use case to work towards than my own, um and it would solve my own. At the same time,.

A

For um I'm, just I'm trying to recall so kube adm doesn't really have the same.

A

I guess issue because kuwaitium while it it needs to deploy control, plane components, but it does that by delegating to kublet right, it drops manifests that have uh image uh tags and then kubla says okay, I need to pull these, um and so it's cool adm is not really installing or uh or cleaning up for that matter uh you, you could have you yeah. You could use the same. Kube adm multiple times, give it different kubernetes versions, and then you know you'll end up with with uh multiple or with container images.

A

For you know that span these different kubernetes versions. um So I'm just I'm wondering yeah how you know can we have that? Can we have that same sort of simple experience um with ncdm?

A

I yeah, I don't know off the top of my head or or you know, or do we need something like an etsy datium install uh and then it's the idiom in it. uh For example, will um uh you know- maybe maybe maybe maybe or maybe we have we require?

A

I don't know, then I guess that breaks that breaks cli. um You know compatibility uh if we require then two separate steps.

B

I'd say it's fine too. Oh right! Sorry, oh no! No! No! No! Go ahead! I think nato protection was told yeah.

B

I I was gonna state that qb adm has the concept of a cluster um and their configuration files are great for being able to deploy multiple clusters using cube, adm onto the same nodes um and if, if fcd adm had the same concept of a of a cluster, I think that you know, and it's not a cluster in this case it would be an instance, but I I you could call it cluster, but it could accomplish the same sort of goal and help with the logic on on what to process against and what to clean up.

C

Yeah and we don't need to- uh we can continue to have scd-adm init, do the default, uh whatever we define the default to be so, for example, uh like default to an instance that we call main, which might just not have a suffix or what?

C

If, if we do suffixes right, you know default to the standard ports default to the standard version and default to like downloading from github, and then we can have options to say like use the os packages or use this, this lcd version or use this um suffix to differentiate in this port type thing. um That's that's fine. I think we can debate, which should be the default right. Should it be os packages or should it be downloaded, but I um I I think that's uh I I don't think it needs to.

C

I think we can continue to have the the simple experience without uh ruining the the the capability to have more complicated options. Okay,.

A

um All right, I I I guess maybe we can like sketch a little more on on on a um on a github issue, but that seems that I I just know what you know. I really like what you said. I think yeah tell me if I'm, if I miss uh misquoting but like separate in it from install or you know, is that.

C

Is separate from separated from ownership, I think like just because an instance doesn't necessarily own uh the binaries, but but it like, if it, if it, uh if it downloaded them, it probably should delete them, and so therefore, unless they're in the cache, so for example, one way to do that would be.

C

We download it to a shared cache which is shared across all instances.

C

We expand those files into a directory that is specific to the instance, so that then the instances are isolated and you know if the cache gets deleted. It's not a problem or if the cache you know some copy gets updated in the shared cache. It's not a problem, um and so therefore, if if if there are binaries in my instance that I am deleting right now and I delete that instance, I should delete that copy of the of the binaries. So I should clean those up.

C

um Obviously, if I'm using the shared os binaries, I should not delete them, because um I don't own those unless I copied the shared os binaries into my instance in which, because I should delete them right. So it's like it's this notion of like do I own this thing is it is it? Is it definitely mine or is it not mine, and how do these things get cleaned up and that's what I'd be thinking about.

A

I see and it sounds like we could use a location as a as a proxy for whether whether we own it or not,.

C

Yeah I mean like the that's one way. I think the other way is uh yes, I think I think, actually in a systemd unit file, we can put additional inform like metadata. I think we can put any sections we want in there. Don't quote me on that, but uh yeah I I would. uh I would probably lean on um establishing our own, like subtree and saying that, like the things in this sub tree are owned by this instance.

C

That to me feels straightforward to reason about.

A

Okay, uh uh yeah: well, I'm just wondering how so so, let's say: if I've got two instances, then um you know if I run etwm reset. That sort of that applies to kind of the the default, the the instance that was deployed in in the default fashion uh and then how do we? How do we indicate or is that just something that we you know we should talk about, and you know the github issue, but I don't know if you have any ideas off the top of your head.

A

You know: how do we, how do we um yeah? How do we distinguish between these? These instances.

B

One of the the things that that I really like the idea of um is kind of following along with cubadm's config file, but maybe taking it a step further uh for ease of use, for instance, if somebody wanted to just use the default method and just do an xcdm in it, um my my thinking is that we could have if, if it's not if a config file is not used to in it in an instance that we could drop one with the default settings uh somewhere, either in sed etsy, std80m or wherever, um and then actions against that instance can be, um those can be specified.

B

I want to act access against this, this config file and that config file would help it know. Okay, I I have the binaries. I need to clean them up. This is their location um and we could do the same thing even if the defaults aren't called. Even if it's just commands on the command line, we could drop a config file with the commands that were used that overrode the defaults as well.

C

I, like that approach a lot and then I think, like I can see there being a. I suppose I had two instances, one of which was the I. Maybe this is what you're getting at now, one of which was the like unnamed instance, the default one and one of which was events, and I wanted to uh reset scd events, uh but I forgot to pass it in.

C

I don't want to like accidentally do that, but um what we could or like I could see that that's a big foot gun uh what we could do is we could recommend to people that they uh provide explicitly named if they have multiple instances. They provide explicitly named instances so that, like the default, will be some other.

C

You know some other instance that, but if you wanted to have two, you should probably call them primary and uh events or primary and other or whatever it is, rather than relying on one of them being the unnamed one or the default one, and one of them being when you have to remember to pass the flag on, because if you forget to pass the flag, you don't want to do thanks.

B

Yeah we, you could have uh restrictions on, on name of instance, based off of config files that exist so that we know that it's unique and then for for things like the the reset call. um If, if no arguments are passed in like a warning, this will be delete. The default cluster are, you sure, pass in a config file? If not a warning of that nature,.

A

Okay- let's um uh I I guess I I don't know if that uh that issue you opened an issue nate for implement phases for the internet and reset command, as maybe maybe it's a. This is a yeah. This is a new issue and we can brainstorm a little bit more, um but yeah uh this seems it seems reasonable.

A

I'd like to um yeah, I mean I I'd like to also hear you know if anybody else, um because I I guess that you know maybe to play devil's havoc a little bit right, the we we don't want to add complexity unless there is um sort of demand that justifies it. uh Not so so, just apart from something being useful on its own- and I agree that this is this is useful.

A

You know as an idea, but I also uh it would it would help if we uh yeah, if, if if we saw that um it would yeah that it would, you know it would be used um because it would. I think it would add complexity so um anyway. Well, uh uh who wants to who wants to create uh nate? Do you want to create an issue or should I create an issue, and then we can kind of go back and forth on um on?

A

uh You know on on where to go, but I think yeah I think we can. I think we.

B

Can make progress.

A

B

Yeah I can, I can go ahead and create an issue. Thank you.

C

A

See apart from that, uh or actually I guess I should ask, is you know, is that is that sort of wrap up that topic.

B

A

Yeah, okay, um I didn't have anything uh apart from that there was um oh. There was a um another fixed pr.

A

A

To the uh to the, I think, the hack or I guess this is lcd manager um and yeah, I don't know justin, did you have any I'll take that one? I.

C

C

I don't know if it matters for bash, I presume it doesn't matter for bash. Otherwise none of our scripts would work. um The uh there is a cons. I have a vague concern around the I'll have to make sure that our copy, so we have scripts that check that the correct headers are on top right. um I don't I'll have to see what's gone wrong like I'll see, what's called what's happening here.

C

um My guess is that the reason why these are bashy and the reason why the bash is on line 15 instead of line one is because we copied them from somewhere else, and I will have to go and see where we copied them from and find out why they did that, and uh it might just be that's uh yeah, but uh this seems like a reasonable pr. I can. I can certainly take this one, particularly if you want to look at uh 163.

A

A

ah Okay, yes, I will do that.

A

C

Take one one: eight, seven.

A

A

Okay, oh and once okay got it, it was blocked on 162, which now merged. Okay. Sorry, I totally missed this.

C

You know right now: it's fine.

A

Okay, um yeah: let's, uh let's follow up on um on the sort of the multiple instance idea in in an issue, and I I mean I think yeah. I think I think that's that's a.

A

That's a it's a nice. It sounds like a nice direction to go uh also just yeah, just just I guess, architecturally also just to um to separate management of fcd from the idea of managing the the binaries themselves right, because etsy, adm and theory all it needs is fcd the binary of a particular of the version that the user wants and where that lives really is not uh is not relevant to managing uh ncd.

A

um As long as long as that, binary doesn't get, doesn't get you know, sort of moved or deleted because because ncdm right now it does right, it does create the systemd service and that that needs to point to the to the binary. But as long as that binary's there um interesting all right, yeah, let's, uh let's, let's think about that, a little bit more um okay, I that that's that's it.

D

God so nate are you modifying 179 now uh to from phase to the existing? What we discussed or is it still open, 179.

B

I would say still open um and I was going to open a new issue for for supporting multiple instances.

D

So will that be separate or will it be merged.

B

um I I would say that it's separate, um I think phases are still useful, um and so I I would, I would personally vote for for keeping that open as something separate, um but supporting multiple issues is, is a is a good goal and and probably easier to implement off the bat than implementing faces.

B

Okay, good, um one thing that I I would like to to ask about as well. um There was a a pull request that was opened a little while back for adding renewed certificate functionality um and it looks like it was closed, the sale and, and the last comment on it I'll go ahead and put it in the chat. um The last comment on it was uh that somebody was gonna, go ahead and and check it.

B

I believe that the original contributor felt like they had addressed everything, um and so I was hoping to bring that up and see see what we need to do to to move that forward.

B

And then we can talk about it in slack as well. It doesn't need to necessarily take up our time here, but something I figured I would bring up.

A

uh Yeah, yes, uh I think there was. uh There was one outstanding question about.

A

You know whether whether kube api server would need to be, I think, restarted after the rotation uh and it wasn't. It wasn't clear.

B

It doesn't, um and I can I can comment on the pr if that's helpful, okay.

A

Yeah- um and I I mean- I think that let me see the pr itself- I don't think there are any conflicts, so we should be able to. um I actually I mean I don't know what the uh do we need to do. We need to reach out to the pie timer because he did. I guess he he closed it.

B

It was closed for being uh rotten. ah Okay, okay,.

A

Okay, excellent, okay, yeah, so then yeah we can. We can reopen it. uh I I'll I'll ping him and you know see if he has any if he wants to make any updates. um Okay.

A

Thanks for bringing that up.

B

Yeah, um unfortunately, I I work in a slow environment and so being able to handle like upgrades and everything is, is not easy, and so we we. We have automation that renews all of our certificates every week to make sure that we're always good. And so I went to add scd-80m to that and saw this and- and I I did pull in the pr and mess around with it um into my fork and and play with it, and it's looking good so, but I definitely don't want to be running anything. That's not upstream.

B

A

All right yeah, I will. um I will look at this yeah because I agree it's it's. It's it's good functionality. um I'm gonna double check to see. You know if kube adm has sort of changed uh its approach, maybe there's some something that we can learn, but otherwise yeah. I think this was in a pretty good state. There was just that question of you know: do we need to document uh a you know caveat um for for users, but it sounds like f.

A

You can, if you can add that info to the pr that that no could be bi server uh will be able to sort yeah just continue. um Okay, it'll pick up the rotated certificate. That would be. I would appreciate that. Okay, I I've reopened that and uh I'll I'll keep an eye on this thanks yeah, absolutely thanks for bringing it up.

A

All right, I don't have anything else.

A

So maybe in the uh in the in the clustered life cycle meeting tradition, uh I will say going once going twice all right uh thanks everybody.

A

Thank you uh for uh yeah uh for attending for uh yeah, sharing the um sharing your ideas and feedback, and uh we will see you hopefully next time around in a couple weeks. Sure thanks have a good one. Thanks.

A