Kubernetes SIG Cluster Lifecycle, 21 Mar 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Cluster Lifecycle 20180321 - Cluster API

Description

Meeting Notes: https://docs.google.com/document/d/16ils69KImmE94RlmzjWDrkmFZysgB2J4lGnYMRN89WM/edit#heading=h.dqa4ycwj7avf

Highlights:
- Plans for the new repository (owners, migrating code, migrating issues)
- MachineClasses
- Breaking the machine controller's dependency on Google Actuator
- MachineDeployment status

A

Hello and welcome to the March 21st edition of the cluster API working group meeting. That's part of sig cluster lifecycle. The first couple of it in items today are from Chris Nova, who I believe is not quite here yet, but I think we can go through them in her absence. So the first one is owners for the new repository.

A

So those referring to the new github repository that's been set up in the kubernetes savings organization, which is new and, as last I checked, only had like three organization numbers which is frustrating because it means if you're on the organization. You can't do things like use the bots to assign or close issues or add labels, as I found out to my chagrin and yeah. So we need to figure out who is going to be owners of that repository.

A

So after how we get added to the org, it looked like she got add to the or because she was able to close an issue. But it's not clear to me how other people are supposed to get added to that. Org I haven't seen sort of like a new members document. The same way, there is for the normal kubernetes org yeah right now. It's just Brian grant Michelle or Allie and Aaron frickin Berger are the only people in the committee SIG's org at all.

A

So yes, so we need to figure out owners for the repository and figure how that's going to work, but we also need to figure how to get people added to or does anybody wanna call happen to have any information I.

B

Did think we were trying to use bots to avoid org permissions entirely so we'd be the owners thing rather than the get them or Commission's right.

A

But you can't use the bots if you're, not a person in the org, so we have to be in the org, even if we have no explicit permissions like you're, not nesting in a group within the org --that has extra rights access to the repo or anything like that. Just.

C

To clarify on that, you can use the bot if you're, not in New York. The bot can't do certain commands to users that aren't in the work like an on organ or can't own, an issue which is what you tried to do. I'll try.

A

To close it and told me, I, couldn't close it so I tried to assign to myself so I could close it and I couldn't assign it to myself either.

C

Yeah only authors and organ close and only organ immers can be assigned to it. It's it's. It's not. You can't use the bot if you're, not a member they're, just okay,.

A

I mean we bought commands.

C

A

Need to be able to do it like issue management in that repository, which we can't do right now, because nobody has access to those issues right, yes, I would go and argue that author of each issue, like triage that issue that's kind of ridiculous. Yes,.

C

I would argue, though, that being an organ member to manage issues for the organization is a decent requirement like there are other commands like rerun tests, or something that you don't have to be an organ member for are.

A

You sure, because I definitely seen pull requests coming to Caretti is where it says. Someone from the org has to write okay to test before we'll run tests on their PR. Yes, once it's okay to test you're allowed to rerun the test, though okay. That means that right now, if any of us sent a pull request, no tests would be able to be run against it either.

C

Yes, I should put members in New York, but for like general contributors who want to come from outside they, they have a lower privilege level. Access to the bot commands is all I'm saying, so we should definitely fill up this list so that we have rights. We want.

A

Okay, since Chris isn't here, I'm gonna assign an action item to her to follow up on this in terms of owners for the new repository. Does anyone have a suggestion or recommendation on how we, if people to be owners of the repository, the only thing that I think s it so far was when Aaron was going through and setting up owners files for all the existing repositories he set it up. He set up owners aliases, so that sig leads would own all of the sig repos.

A

So, like you know, you know, Lucas and Luke are owners of the the current cube deploy repository, even though they're not actively working on cube, deploy as sort of an escape hatch. You know to help manage manage that if there's turnover, we need to add other people, so that was the only thing that I knew that we should definitely see that presumably people that are active contributors, we should also be feasting on their Styles.

A

Also, does anyone have like a good sort of process for figuring out who those people are or nominating people I, don't know. If there's a anything in kubernetes we can. We can steal off up for that.

B

There there used to be a ladder, I think with fairly strict definitions. I think it's been very much relaxed, and now we can do whatever we want. It does make it awkward. If someone like you know this sort of self submit a PR, it can be sort of like not having guidelines, makes it harder in my opinion, but we should probably make everyone that has got a PR merged reviewer about that it's a starting one and then the approvers I, don't know how you want to do that.

A

That seems like a reasonable suggestion.

A

Obviously, which illusion look up, opt out up being or viewers too, if they merge the PR, but they don't want to get you know bureaus assigned to them by the bots they might want. They might not want to be a reviewer.

D

It would also be nice to have broader presentation across the various organizations that we all work for.

A

Yeah I think that's why why Chris is trying to figure out how we pick people to be being the owners files right.

A

Okay, so next thing was a migration plan. From the current cube deploy a repository for the pieces, we want to pull out that our cloud or environment independence to put them into the repository following the discussion week or two ago, Chris, opened an issue to talk about how we should do. This, there's been a little bit of discussion on that issue.

A

Sort of coming up to my suggestion that you know we can keep debating this, but we should probably just move forward with what we think is reasonable now and then refine later to make forward progress and Chris asks. Should we just open a PR or write a proposal and I put a comment in the dock yesterday and I was reading the notes for this meeting. Basically saying: if it's simple, would you just write a PR?

A

If it's complicated, we should make a puzzle, so I would propose that we just make a PR and if it seems controversial or too complicated, then we should go back and do a proposal. The try novel wait as much red tape as possible.

A

Does anyone believe that it's going to be more complicated than just moving a bunch of files.

C

Yeah I think it'll be slightly more complicated. um Part of it depends I guess on one of the agenda items of whether the change I have our view of whether we think that's very way to go forward, because until something like that happens, you'll be pulling in clouds specific code as well, or else it won't build, and then also we have there's some testing for stuff to turn up. But it that's gonna be some beckon.

C

So if it doesn't have to happen at the same time in the migration, but I don't want you changes to go into the repo without that being turned on so at least get build and test status. So no one breaks the build, but other than that I don't see any complications.

A

Okay I mean that sounds straightforward enough to me that we probably don't need to write the design not for it unless, as we go down the agenda item, that your proposal down there for more strictly breaking out the the cloud flyer stuff is really controversial- need to do that differently. One.

E

One commenter feelings, because all it you.

B

E

You'll be using the case of a pewter. I.

E

Think it will be yeah it'll, be some trouble and I think not something we cannot fix.

E

A

Can you clarify what you mean by so it expects the directory structure, but presumably would have that same directory structure in the new repo as well. I saw.

E

The idea was, it doesn't resemble only food API. Oh.

E

A

Yeah so I think that my proposal was now. We don't bother trying to separate those out unless people need us to, because it's not in.

E

That case, if we do, it is simply the No.

A

Yes, I think that the arguments for splitting apart today, where that people wanted to be able to build other controllers outside of the main code and I, think people that are doing that are likely to want to be able to pull in some of the other code in addition to just API. So they can have reusable libraries for doing watches and you know have the machines that you know controller and so forth.

A

I think that my argument for having the API separate was more that client-side tool and it wants to depend upon the API and not implement the API. They don't care about. Having the controller implementations, they probably maybe want client implementations like they might want to client go above once, but they don't want the machine controller write code, we've entered into their repo and I. Think that's where we might see some demand for splitting them up, but again we're not we're not there yet. So we shouldn't complicate our lives until we need to.

A

So Chris's next agenda, item related to the repository migration is about issues and whether we should migrate existing issues and she was wondering if she should just groom issues in the new issue tracker. I pinged rodrigo about this yesterday, since he's been also been doing. A lot of issue back logging and grooming and his take, which I tend to agree with, is that any issues that are specific to the api we should put in the repo with the api issues that are specific to the the code.

A

It's in that repo we should move over and that issues that are, you know specific to the google implementation. That's in cuba play right now, which you just leave alone, but generally the issue should live in the same repos. That code yeah.

C

I believe we have a like a new label for GCP specific ones, so we need to make sure that that list is accurate and then you should be able to copy most others out or make sure the cluster API tag is on label the issues we care about and just worry about. Those.

A

Rights and I know Chris was gonna, groom the backlog, and this is probably good opportunity, as we move them over to sort of do that grooming and make sure that the right things are tagged for stable, API and offer Peter huh growth or to marketing. You know milestones going forward.

A

Alright, so another commissary, so the next one, it's machine class, so that we we've talked a number of times in the past, about having machine classes and being able to you know basically have sort of a template if you will just to stamp out new sets of machines or new machine deployments, referencing based on sort of simple string name, so that when you are saying give me a new machine, I just I want you know a small machine or whatever that generic name is.

A

You don't have to specify all the provider config, so I link to a PR here where I split that come by the provider config into a struct, while also changing it from a string to a runtime rocket engine.

A

Now that there's a struct there there's a hook where we can plug in reference to a machine class instead of directly embedding the provider config in the runtime brach extension field, which is what we do today and so I'm actively working on some prototype code to do that and I have a meeting with sod from sick storage right after this meeting to talk about storage classes and make sure I'm not going down the wrong path.

A

The main one like I said one of the main goals for doing this is to make sure that it will work with auto scaling, because the the way that the code is written right now, the autoscaler doesn't really have any hooks to plug into to figure out. You know if I scale, this machine set, how much capacity am I going to get by adding one node or adding five nodes. How much is going to be reserved by you know no allocatable?

A

It does have the hooks right now to figure out labels and tanks, which is sort of the other half of the picture that it needs so I think the most important fields to put in the Machine class itself, our capacity and allocatable, because those are the things the autoscaler needs to function correctly, I'm trying to kind of keep everything else in the cloud provider specific reference from the machine class, so that the machine class itself can be relatively small and sort of serve.

A

The purpose for that that piece of automation so again that that code is being rigged and prototypes and tested at the moment, and hopefully, I'll have an update or a PR. Listen out before next week's meeting questions comments, concerns.

F

I think really, when I joined there was a discussion happening that at each layer we should add the machine class right. So in this situation, when consensus I think at that time was, it should be at all the layers. There should be a flexibility to edit all the days. It's just verifying. We what's the yes.

A

That's a great question, so I went back. I was actually not at that meeting, but I went back from a does. Me knows: I think that seemed like a reasonable answer and so the the way that the provider can figure out broken out. It's broken out at the lowest level at the Machine level, so the Machine level you'll be able to either reference a class or inline provider.

A

Config machine sets use a machine spec as their template, so they inherit that behavior of being able to reference out or inline, and the Machine deployments I think will inherit that also so yeah will be at all three layers. Okay,.

F

That's great and from the auto scaling perspective, so not very sure so correct. We are actually currently trying to integrate cluster scanner with the machine controller manager, which was previously known as neutral, and let me change the name to motion so what we have understood that cluster waterskin is exposing a certain interface if they call it a cloud in the cloud provider interface, and there are certain methods that we need to implement for different providers and I was checking out how AWS is doing and so more or less.

F

What I could understand is that cluster or descaler, currently just manipulates in the case of AWS. Let's say the the target size is of the esc, so it's like interplay scale, auto scaling groups right. So in a way what we are doing this we are just plugging in our machine deployment at the moment, so it's like autoscaler will just play around with the replicas field of the machine deployment modify the different whenever it's that's how it should just end this ooh, what sure hoe machine plus yeah.

A

So let me go into that a little bit, so we talked with with Marcin who's. One of the the tech leads for Sega auto-scaling, and he was telling us so they have this this cloud provider interface, but that that's not particularly flexible, because it means that that basically new code, that wants to be built into a lot of scaler has to be put in tree and want to sort of the pushes with kubernetes is to make things more composable and to not have to put all of the code in the same location.

A

In particular, you know they had someone submit PRS and added an azure implementation, but then nobody was supporting it, so they actually had to take it out of their repo, whereas if, if they had a more composable way of building autoscaler than the azure code could live somewhere else, and somebody else could provide support and updates without without having to send them pea arts, and so they were looking at basically, instead of extending that that cloud provider interface of getting rid of it entirely and saying, instead of saying, I'm gonna call out to Amazon to scale SG by one I will use the machines API to scale the machines that are a machine deployment by one.

A

But the thing that they they do with with Ming Zhan, GCP and aSG's on AWS is that they actually call out to the final provider and say you know what size machine is this? How many cores does that have how much memory does it have they at least on GCP? They read from the metadata about what labels and tanks are gonna, be on the machine and how much capacity we've reserved through node allocatable.

A

For that machine and all of those different bits get plugged into the scheduling algorithm that they run to figure out, if I add a new node, will it actually allow the pending pod to be scheduled, and so, from their point of view, they're having to do like, like they have some special code that I broke two weeks ago on GCP that reads: specific values out of the metadata that we from the cluster lifecycle point of view, think that was an internal interface that nobody should be depending upon, but they are depending upon it, and so the machine class gives them an actual interface that they can.

A

They can depend upon, or we expose this data in a standard way across different implementations, and that means that you don't actually have to go change. The autoscaler code, if you want to add support for digitalocean, all you have to do is add the Machine controller for digitalocean set up some machine classes that expose. You know that the values of the autoscaler needs and the autoscaler should just work and that allows autoscaler to basically rebase instead of having cloud provider code.

A

They should eventually be able to delete all their cloud provider code, just talk to the the machines, API and work anywhere that the machines API works, and so that's we're talking about them on how they can get to that eventualities. So, instead of us plugging into the cloud provider interface that they have right now and saying we're gonna add a cluster API hook into your cloud provider.

A

We want to flip it on its head and say the autoscaler will call out to us and we will provide sufficient information to drive their algorithms and then anywhere that the cluster api is implemented. You can run out of scaling with no extra work tuning.

F

It sounds good, so.

A

Yeah, so the the main goal of the machine classes, from my point of view, is to to write the prototype code, verify that it looks like it serves some of the use cases we've talked about in this meeting and then go back to Sugano, scaling and say this is what we've got. We think that it will work for you. Can you verify that it's good enough right and that's to need sort of the bar for it sort of passing muster? Is we know we want it to drive, auto scaling?

A

If you can't drive auto scaling, it's not sufficiently descriptive.

F

So the developments from the auto scaling groups right so that should come only after we go stable with our at least machine set or motion deployments right, yeah.

A

And so Marcin has been been pestering me for the last couple of months. Saying: when are you gonna have a stable API? When can I start writing some code? Like you know, when's, it's gonna stop changing right, so they are they're very excited to start trying to rebase their code against the machines.

A

A

G

Existing customers of the coaster autoscaler without without the machine encoding, so yeah.

A

I mean I'm sure they're, certainly not good I, just like completely repair all the old code day.

A

One great I think it'll be a gradual migration, but it will allow them to support new environments without having to add new code themselves and as we take existing environments and move them over they'll be able to sort of get rid of some of the old code and and maybe the project will fork and we'll say, here's one where if you still want to talk to mixer at G's or whatever you can, you can use this this code and and if you were just want to use the machines API, you can use some different code, maybe maybe I'll end up forking I, don't know what their plans are for for maintenance of the old stuff, but I yeah they're.

A

Definitely not gonna gonna. Do a knife switch and drop all the old support when you diddly, but I, think it gives them a clean path forward to to saying this is great. We no longer have to talk to any infrastructure. We don't have to worry about vendor and cloud libraries, and all we have to do is talk to minetti's api. Is, you know, top to bottom and will work in any Carreras environment, so I think that's pretty appealing to them as to they can get themselves out of the business of talking to underlining for treasure.

H

Question whose machine Colossus, how it will handle will like special needs. For example, if mushroom provides device like accelerator or something like GPU of PG, something like white.

A

So you can you can do that today in in the provider config right, so in that provider config struct, you can define your schema, so you can express things like a GPU or accelerator.

A

So I think the the tricky thing is how how we express that in a generic enough way that the autoscaler can be driven off a bit or, if you say, I'm, trying to settle this pod. It requires an accelerator. How does the autoscaler know from the machine class that creating a machine of this type will give me an accelerator and I think the the way that we are the way that I'm looking at doing the classes?

A

There is, let's see if I can find it, there's something called a resource list in the core, v1 api's, and so you can basically say what are the resources that I need? I think we might be able to leverage that as part of the capacity of your machine class, and so the machine class will basically say like here.

A

Are the resources I provide, whether that's memory, disk CPU, GPU, accelerator, etc and that's a list and then the autoscaler can match up against that list of resources when it's trying to figure out what will be provided from the other languages?

A

H

About not so vyas resources like network cards and, for example, like nominals, like.

C

I think so for special resources like this you're gonna have pods that want to use them, and so the scheduler has to be informed about how to make sure to match up that pod with the right machine, and so we should really just take a key from. However, the scheduler is doing that constraint, matching and represent it the same way, whether it be a resource list or if it's a special one-off label or something like that, I think we should just match. However, the scheduler represents that constraint. Yeah.

A

That's a good point because the the cluster autoscaler, like literally vendors in the scheduler code and runs the scheduling algorithm and so what it does is. It says I'm gonna, pretend to add a new node and I'm gonna. Try to remodel that node as best that I can based on what I know about the platform and then I'm gonna run the scheduler room against it. So, as Chris was saying, if the scheduling algorithm doesn't take these things into account, then the autoscaler is also not gonna.

A

Take them into account, and so it's not something I need to represent and, in particular, I think that things like the network interfaces is not something that's represented in the scheduler at least not today, whereas I think I think GPUs are either directly or indirectly represented they're, either a direct resource or they're indirectly represented via labels, because most people who are gonna create machines that have GPUs are likely to label those machines, so they can make sure their GP workloads are pinned.

A

They might even take them to make sure that they're not wasting those machines on workloads that aren't taking advantage of the GPUs and so based on the labels and paints, which are the primary we're closed steering mechanisms. I think that is often enough to inform the scheduler in the autoscaler to do the right thing.

H

And my water was just what, when we are describing my machine, it should be flexible enough to accommodate all kind of this extended resources, even though it's not used by after scalars yeah.

A

And that's where I want to talk to a lot of Spiller folks, and maybe I should talk to the scheduling folks too, about this.

A

My understanding, or my hope, maybe is that the resource list is sufficiently expressive that it allows new resources to be added in the future that will sort of automatically be able to be used in the machines API as they get added to the core of kubernetes, and that that that is what would be driving the scheduling decisions, the cluster auto scaling decisions and then you'd be able to represent that that machine's API. So that's definitely something I need to validate that assumption.

I

Since we're talking about resources- and we, what was it discussed at some point to map or somehow have the some piece of code map the resources to what the kind of a provider config would look like automatically, maybe as part of this API to the provider itself, that way, the user would just generically specify CPU Ram, you know whatever disk sizing and whatnot and the the providers would know what to provision versus having to figure out manually how to map. You know AWS instance, types to sizes and things like that. Yeah.

A

That's something that we talked about pretty early on at least internally I'm, not sure if that was something we talked about in this meeting publicly was basically being able to say, I, want a machine with these resources and then allowing the underlying cloud provider to sort of do a best fit match. So, if you said, I want three and a half CPUs and a gig of memory.

A

You know that might not be possible to do on all platforms, but you might round that up to four CPUs- and you might say you know AWS if I have four CPUs only supports eight gigs in memory because they want to have the right ratio. So if you ask for three and a half CPUs and Giga memory, I'll get before CPUs and eight gigs of memory, that's the best fit for your request and I'll provide that machine for you. It's not something.

A

That's currently modeled in the machines API and was not sort of the primary goal for the machine classes, but it might be. Something we want to talk about is how we would sort of model a here's, a set of requested resources and then allowing the underlying implementation is to deliver sort of their best fit for those resources.

A

Fair enough I think that's a separate discussion from classes, but maybe maybe there's a way to have that the classes solve both of those.

I

Yep one one thing: I guess in emboss the way we've implemented this is we still kept the API to be you know, except the provider. You know specific config, but we also added the API for the providers to eventually be able to convert the generic input into something more specific for them. So if actually keeping the the basic API the same, but also kind of a extending as an additional feature for this kind of a conversion.

A

Yeah that makes sense, I, think that's sort of the goal with with the machine classes in some sense is, if we said you know we're gonna create you know small, medium and large machine types.

A

Then the providers you know could map those to what they believe is small, medium or large in their environment, and that makes certain things a little bit more portable because you save my machines that config, it just says: I want small machines on with five small machines, but I think you know is a little bit of a leaky abstraction, because if you're gonna move from you know Google to Amazon or to Asher, then you know you have to hope that you know small machine is about the same number of cores and has about the same overheads in terms of node allocatable and some of these other other features.

A

Otherwise, it's not going to be the right size for your cluster anymore right, but but maybe something like auto scaling fixes that, where you say like you know, I have this template I want a bunch of small machines, sure exactly what they look like, because the autoscaler will kick in and give you the right number based on my workload. So we do have that that sort of thing the.

I

Machines classes typically are populated by by I, guess you know an operator or by some more automated process that maybe even shipped as part of the you know, the the provider yeah.

A

That's a good question so the way that it works for storage classes is the provider concede some storage classes where they say you know we're gonna set the default storage class on TCP, which uses standard, p d--'s, but then off. Then cluster operators can go, add more storage classes and they can reference those and they can even change the default. So if machine classes, what I was was sort of picturing in my head was you know when you you launch your? You know, Google machine controller.

A

It might automatically create some number of machine classes that it thinks are sort of common sizes of machines that people would want to use and registers those automatically. But end-users can then always create their own machine classes and rep rep in a map. Those two arbitrary other. You know underlying resources, but those those machine classes would only exist in clusters where they have, they have created them, so their workflow for spinning up a new cluster might be create a new cluster.

A

Now add all my custom machine classes now spin up my machine sets because I have specific ways that I want to configure. My groups of machines that make sense yeah yeah. It's.

D

A

Of a combination of both rates, it's definitely going to be configurable by cluster operators, and it will likely we will likely want to seed at least some default machine classes based on the environment through run again based on which machine controllers you're running projects, you know, say if machine classes don't exist, go ahead and set some default ones for able to use I, see that makes sense.

A

This has been a great discussion, anything else, time. Machine classes.

A

All right, Chris, Chris, Rossiter next.

C

Sorry I was zoning out. What's mine, oh yes, the PR! Yes, so I've gotten some feedback that there have been some hesitations about um modifying the generated code, but it does break the dependency I was wondering what is the overall consensus on this I have there's some more work to be done with like building the new docker images and changing at least the GCP deployer to deploy properly with the supply to part controller, but I was just wondering it modifies to generated files that are only specific to the controller.

C

It doesn't change anything about any degenerated API code or client code, which I think is the high turn stuff for for generating code. Like you add a field or you change a field or something like that. That will need to be regenerated a lot, so I was wondering what everyone's feedback, or if they got a chance to look at what everyone's feedback wasn't. Whether we should move forward with that.

G

mmm I had to look at it, so one problem I mean I mean right now. If you look at the brightest, we have there, the not the container network interface you can do container world with an interface and I was thinking. Maybe approach like this.

G

This is the best way to go, so you can have you know, for example, in one pot you can have the action, the Machine controller, in into one part of the motion, control manager that encompasses like the in the deployment, the asset and induction machine and in the another container you can have only a very small piece of code that who implements like this: she runtime interface, all or whatever. So by breaking dependencies.

G

It would be easier, I think, because I thought, with this project, you were suggesting everyone has to write their own controller and in most cases, people simply want to create a machine not to reprimand the entire controller.

C

Everyone has to implement their own main for a controller, but if you trying to navigate my change right now, but if you look at my controller main, it's basically 67 lines of code that mainly reuses libraries, but only make sure make sure to use the Machine controller library with the Google interface plugged into it. So this interface has to be written for whatever provider you're on whether it's AWS or Asher, or anything like that.

C

So there's no getting around writing that, but the glue code to actually plug them together is not that much more and it also allows you to have flags per machine controller. So we don't have things overriding like. We want to add specific config options to the GC machine controller that I'm sure AWS erasure would not care about. We said this gives us an entry point to also do more customization for specific provider controllers as well.

C

You're right most people just want to create a machine. This change should not affect that in any way they just have. Instead of deploying an API server at CD server and controller manager pod, they have an API server at CD. Menator controller manager and machine controller pod- it's just it just adds another pod that hopefully someone else writes for them and maintains, and the API interface as far as like the cluster API should be exactly the same.

G

Okay, well I'll. Try to summarize my tics might process into like mmm comment and then I'll just send it.

A

To get a preview of your thought process partner, you thumbs up thumbs down or kind of on the fence about description, I.

G

Mean yeah I mean after writing, several controls interning in it ICP I mean it's. There is lot of boilerplate that you have to write in order to get like a fully functional controller. We have to write like, for example, leader, election signal handling and all those kinds of things, and if those things are already implemented you to dislike the controller manger already, and you only have to care about simply creating the machines and and without attending any reconsideration, hooks or critic informers and all those kinds of things. So it it's easier.

G

C

I think leader, election and stuff like that, could be built into the controller library. If you want I, don't think it's there now, but it's something that we should add to the generic controller library, I. Think I, don't think that should be brought out to the glue code. Part of it and the only the only thing the controller library doesn't do as part of the controller right now,.

B

C

Called the machine actuator interface, which is creating deleting basically a crud interface to the underlying cloud API, the machine controller library still has like informer interaction and Reconciliation loops, and all that the one thing that's I had to do here, that is boilerplate is I parse, a config file to into a struct and I handle like a little startup heiress, and then I basically create new shared informers, so that I can pass the shared informers into the machine control over library and that's the the sheraton former and the Google machine actuator into the library and that's it and my main motivation was for doing.

C

This was twofold: one: breaking out the control libraries dependency on Google stuff and also making it easier to add custom configuration to the Google machine actuator, which is, which is something we want to do and not pollute the overall library just plumbing through Google. Only options.

D

Do we foresee these controllers being in in separately, phones in the future yeah.

C

Yes, this I think this is actually one of the main things we want to accomplish before we move to the new repo, so we don't have to move all the Google specific code to like we want that to be devoid of any interaction with the GCE API yeah.

A

And Chris, nobody explicitly said: I'm not going to have any Google code to the new repo and as it is today, if we move code without any google hook in the new repo. As far as we mentioned earlier, that code will not compile at all and so it'll be really hard for someone else to build a controller because I'll be vendor again code that is compiled.

D

So I looked at the PR and I was surprised that the that the generator code is generating any controller bits that you have to change. This is I mean it seems. Like you CLE, you could say you don't want the controller and degenerated it some way right, but a I, it's the plus one for me, yeah.

C

By default, the API server builder generates a lot of code. It generates a deep copy code for the API types conversion code. It generates client.

B

Code, including.

C

Informer's, which is very useful, and it generates a controller kind of like stamped out template that does nothing and the one thing you have to modify is like for the machine type.

C

It generated a machine controller framework that had a machine and pole struct that you were supposed to fill out, and so it basically gave you a controller framework that you could build on for the machine controller and what we did was we built a machine imple we built on machinable for a reconciliation loop and for interacting with the API types we left a lot of the common code in, but when it came to actually doing crud operations on the cloud we put that behind an interface called a machine actuator, and then we wrote a Google machine actuator and satisfied that interface, and then we put a hard dependency on the machine actuator through a factory function, which is what this change is breaking, is breaking that dependency to a factory function.

D

Just yeah, so if you've run the generator again, will it overwrite your changes.

C

If you rerun the controller generation, it will overwrite these changes and then you'll get a build failure because it won't build.

D

So would it make sense just to leave the stub controller in there and have it do nothing, and you just you know, have the separate controllers actually, unfortunately,.

C

The reason I had to modify the generated code was I'm. Modified I had to modify the machine mple in our the struct that it stamped out, and it's initialization now takes another parameter that the generated code does not call. It tries to call with one less parameter, so the generated code would no longer build, even if, if I, if I.

D

C

It it wouldn't build.

C

So yeah the main the main change was instead of the Machine Apple, calling out to a factory function to get its actuator just have the actuator passed in.

G

I have a question so when you're talking about custom, flux and etc to the to the Google, for example, controller. Are you referring to flack specifically to the controller or that that we relate more or less to the actuator stuff.

C

That relates more or less the actuator one of the the biggest flag is just gonna, be a config flag. That's gonna be a parcel config, we're gonna back it by a config map and right now we're thinking about having like supported bundles like if they request, like kubernetes 1.10 with a say on a boon. This is the actual GC image, and this is the actual installation setup script for that, whereas we can now with that, we could support like a 1.9 setup method and 1.10 setup method.

C

If they're like slightly different things that need to happen for that, it's just future proofing for one having flexibility with versions, also allowing users if they want to change how its installed slightly or even bring their own OS like if they want to create a SUSE, a Linux image and wants us a handle want to own how to set up a node for that they can specify it through this config map, so something that might be useful to other people.

C

But it's also going to have very GC specific stuff, like here's, a GCE base image to use like a api path to it. Yeah.

A

Presumably also use those flags they need to plumb through, like authentication authorization bits into the actuator as well. Yeah.

C

One thing that we might want to do is like diverse, like we want to have a custom certificate signer and not have the Machine controller, have access to the cluster priming key we want to make. It may have an API endpoint to defer the signing to some other trusted source and interface to back that with Google key store or something like that. But.

G

The specific project view will happen in the actuator or in the the the code that surround the the control. If it's not there, I.

C

Think it would happen in the actuator. Anything we feel is count enough for everyone. We would probably want to promote upstream- and there may be some things like that- that we want to experiment with it. First on Google and once we like get the use cases down and see that okay, we made a mistake here, but we can change it and then get a decent use case. Now. We'd probably bring it to the community and say: would this be useful in the generic library.

A

Because the generic library is calling out to the actually to do crud right, create this machine Billy's. This machine update this machine and the actuator is the thing that's actually talking to the underlying cloud, and so that's where you need to configure that actuator to do that correctly.

G

Yeah great one was concern is with this approach: you're only you, we are only limited to go lonk. Basically, yes,.

A

If you want to reuse the library as you're limited to go lanes, you can always write your own controller that you run instead of this controller.

A

That's gonna be I, guess if we followed your approach of a sidecar container with with specific hooks, then you could reuse the controller logic and that interface becomes a basically RPC between processes. Right.

G

Yeah, this is my original idea, like Gojira PC, or something like like the container runtime, don't contain your network interface yeah.

A

Yeah big gaps, I see of that is both CNI and CRI, and CSI are going through very long, convoluted sort of standardization processes which I don't think we want to do right now and so I think doing it in process now and if we find we need to break that out later. That seems like a task we get to account later, which is what we did with container runtime right, like initially, the culet had built in direct support for docker, and then they decided we want to break this apart.

A

Let's now take on the you know the the large task of breaking it apart. This is probably a better approach, so we can't really get something done in the short term. Yeah.

C

And at least this in process binding, isn't forced upon you, you get to choose, what's binded that questions yeah great yeah, so there there are concerns for the modifying of the generated code. One person may be volunteered to update the API. A server builder I'll respond that this may be something we want to go through if they wish to upstream the ability to customize this you're in good a bit.

C

So we so it's not an issue, but from what I'm gathering this sounds like a thumbs up if I Martin, you want to put your feedback on there before we move forward with I still have a lot of things to do. I'm, basically, writing make files and changing I. Basically, soft write make files and change the G speed employer to have a different deployment to account for the extra pod, but other than that.

A

Yeah I think this would be great to get in so that we can unblock migrating and code to the new repo to once we figure out. Oh yeah, all right. We got 10 minutes left, there's one final agenda item, which is also from Chris Nova who's on here, which is asking for an update on machine deployment.

A

So I don't know. If anyone here can provide an update on machine deployments, there have been some initial design dachshunds around I, think mostly from Messam from our side and then obviously you know the SVP folks have the machine deployer in their machine controller manager as well as sort of a base that we can. We can build on top of I, don't know if there's any concrete updates that people can provide on on getting that upstream.

F

So we have sorry I have not been able to check the architecture for the machine deployment, but one hint on that that we recently found while integrating or the scalar that we will require from the mushroom deployment. So basically autoscaler expects that under the machines which are under a certain machine deployment, so we should be able to delete a specific machine and at the same point in time we should be able to decrease the number of replicas of the machine deployment as well right. So this two things should happen in parallel.

F

So if we are having a standard approach of just scaling down blindly, then internally we will we may end up. Initially we end up doing the things like the machines with running pending and terminated. Make you high priority to the terminated machines, two scales in town first, so that's where new logic would require what we need.

F

We can have a new annotation for a priority, so you just basically assign the priorities to the to the machine objects and we expect either autoscaler or certain other component to actually put a priority annotation on the machine object, and then machine deployment takes third priority number as an input and then tries to delete those machine object first before the other machine objects. So that might that just one hits of that we found while integration of the scanner.

F

A

That's a great point, and that's actually something that was kind of extension to us as one of the requirements is I want to be able to shrink the set by by a particular single or particular set of machines and and funnily enough, one of the gke customer actually asked for that feature on deployments inside of communities as well or they said if I have a deployment or communities replica set. I want to be able to shrink that by a particular one also like they said as a feature request.

A

I want to be able to delete that specific replica and you know not having not have a race condition where I'm trying to delete it and shrink it and having and fighting with controller and and see gaps does not get an implement of that. So we might be, might be threading some new ground there, but think that that sort of feature requests has also been asked for in kubernetes itself managing pauses. So it seemed unreasonable to just build that directly in.

J

Is there time now when the first machine, like the type definition is available and.

A

So I think I, don't I, don't think anybody is actively working on trying to put a PR together for the types for machine to Planet. Yet okay, they are, they should let us know now so we're not duplicating effort, but I, don't I, don't know if anybody has assigned to do that quite yet, and B I, don't think anybody's, actually I think it's sort of in everybody's head, as as for the next step after machine sets, but I haven't seen anything pop up. That lets me think something's doing for yet more.

F

Or less will be replicating the types from the actual deployment itself right, just calling it machine I think.

A

For the most part, any cool, as with machine sets, will need to be careful to make sure that all of the fields apply in the same way which, in in machines in this case, I believe that they did and then like. As you mentioned, sort of making sure that these these other use cases that aren't yet implemented for deployments or replica sets that we have a way to express those in our API, which is hopefully consistent with where the gaps will go on in correlator.

F

It could just recall one moon point there so recently, then we also implemented the green functionality so before deleting a machine, so you could actually. This could actually be a part of the Machine set and I'm, not sure whether it is already there in the existing implementation incapacity. Okay, so we can actually leverage nice feature from the Cuban is itself so long back.

F

I think there was a conversation that, when the rolling update is happening from the Machine deployment, we also want to make sure that the application is turning on top or somehow we ensure that the applications have actually respecting the application SLS. So the PD B's are being respected right so to solve that problem, rather than implementing it by ourselves. What we did instead before deleting a machine, we initialized the parallel train functions and you could see the tube Bundys itself, while the parallel drain is opening respects the PDP's of different applications.

F

It's really possible that if you are dilated scaling down three machines in parallel and before scaling down those machines, if you are playing with brain function in parallel to all those three machines, then the applications running on those three machines are by default, going to respect the application PDP's- and I.

A

Think why can't you said: what did you initialize that cause the drain through respect the pdbs, so.

F

Before deleting any machine, so we by default by default in machine controller manager deleting them before deleting a machine. They execute the same drain function that we have in cube city L. So we basically copy the cube city on the screen. Function, put it inside before the delete logic, and what happens is that when I, what Minerva machine deployment with ten replicas and I want to make it fine? Basically, five machines will be parallely. Good.

F

Five machines will basically be going down well in parallel fashion right, so these five drinks are happening in parallel and because kubernetes grain function is implemented in a way that the application PDP's will be by default, respected and that's how we can ensure that the application is also respected. So back then the conversation was that it can be very critical, very complicated, complicated to implement it, but cortisol would much much easier. Yeah.

A

And it's also been a discussion about moving that train functionality into the server, so you can actually API server to do it for you, because right now, like gke, has implemented client-side drains, you guys have implemented client-side drain its copses and pulling it client-side like everybody who cares about this is implementing client-side drains on their end, whether they're shelling out to take you cuddle or rendering that code in or rewriting the code, which is a little bit silly as it'd be nice to just have that code written once they can reuse inside the server right.

A

Then cute cuddle drain can just say like great pay. Server do the drain and the machine you know controller because they hate server do the drain and we got that cut price so because we can poke people and see if that's on the up come towards map I.

G

Think the issue, especially with the server-side draining and there's everyone keeps asking about it then can nobody actually start start working commits so I, don't know it looks like the best approach is to have a step as the pre source on do not probably or something like this, and then you send like a request.

G

Yeah we should, I I, don't know which approach is actually better to implement it. To diplomatic, then go go to the AP machinery or which actually is. We should be responsible for this one. A paper machinery Custer, stick.

C

G

C

Say abs for workloads, hmm since that's the oath rescheduling absent workloads off of a note in respecting their essays.

A

Yeah I mean maybe it's maybe part of the reason it's I'm making progress is because the ownership isn't clear, because, right now, it's probably owned by sig CLI rights as part of cue cuddle, but when it gets pushed to the server, it's I think it's a little bit unclear whether its API machinery or workloads or scheduling, but it has to do with scheduling, also and sort of who the owner of that becomes. So we can.

A

We can poke at people and try to see where what the status, oh, that is, and if it's not anybody's near-term roadmap. Otherwise, we'll probably, like you said, end up doing the client side ring for now and just saying that we will replace that in the future when server side rate becomes available.

A

All right with that, it's now 11 o'clock and.

J

One last question: so currently no one is putting effort into it to create a first PR or the type definition should we should I, or should we take care of this, creating a first type definition and then creating a peons, and we can discuss on the or should look like sure, that'd be great. If you want to take that as an action yeah.

J

A

All right thanks, passion all awful day, I in the notes for you and it's it's not a little bit past 11:00 o'clock, and so we're gonna go ahead and call this meeting over. Thank you. Everyone we've had a I think a very productive and useful meeting today. A lot of big discussion and help see everyone again next week take care.