Kubernetes SIG Cluster Lifecycle, 16 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Cluster Lifecycle - Cluster API Provider Nested office hours 2021-02-16

Description

#sig-cluster-lifecycle
#capn
#capi

A

Good morning, everybody uh today is february 16th, and this is the uh cluster api provider nested office hours. I'm gonna trip up on trying to say that every single time but anyways um this is a recorded meeting, um will be posted to youtube. So don't say anything: you don't want post it up there. We have a pretty short agenda so far. um So please make sure you add your name to that. Google doc.

A

We have two items on there and we don't have any new folks. So no introductions or anything like that, and I don't have any uh psas to call out. So we can just jump into this vince if you want to take over and uh start talking about copy, dev environments or test environments.

B

Yeah, so I need screen sharing capabilities.

A

You should have it.

B

All right, um so we briefly talked like last time about like test end. What's so great about it, what's not so great about it, and I guess, like there were a few questions about like how do we.

B

You know like a go from like having nothing to actually having like a test environment um and also like how do we get to a place where, like we could spin up multiple test environment if we wanted to um control, runtime actually adds like a lot of these things, like it, they're they're, built in control of runtime.

B

um This is the cappy main branch and we're right now we're using controller runtime082, which is one of the latest releases um and I'm gonna walk through like just like. How do we set up these um controllers and like how do we test them? um And um there is also like some sort of like um super type that we have here like this test environment, which is built on top of controller runtime.

B

So, um if you're not familiar with like how cap is structured, but I think everybody here is like kind of like familiar with it.

B

It's like we have a management, cluster and workload cluster and a lot of times like we need to test both um and there's a couple of ways to go about this, like you could have, for example, you could pretend to that, like the same, execute control plane that you're connected to is both a workload and a management cluster, because that that is a support use case or you could create like a two different test environment so with the con with their own control, plane and lcd, then um what we're act like one as a management plus one is the workload cluster um for controllers.

B

um What we actually are doing here is like the first things that you you might see here um is that we have we define a test environment and we say: okay like I want to create a new test environment. But what does this do is actually like kind of like reusing um the end test from controller runtime to to say, hey, I want to spin up a new test environment. What m test does under the hud is actually like. Pretty nice like there is a fetch external binaries shell script.

B

um Q builder has published cube api server and xcd binaries for both darwin and linux. I believe so far.

B

I'm not sure if there are like uh m arms builds or anything like that yet, but um we can add support for it because, like we just have to cross build that there. um But what this script does is that, like it will download locally, if you don't have in a temporary folder the these two binaries for the version that you're specifying so, for example, here we're testing the minimum version of the management and workload plus for that that we want to support in, in this case, like it's 192 for the management cluster.

B

um So this will download those binaries locally and um make sure that like when you create a new test environment, it will spin those binaries up in a process group. Any questions so far.

B

um Not all the versions are published here so like if you need a different version like you might want to check out, like this um storage link. um Just this part before the archive name.

A

B

Should show you like which conversions are published? If you need other versions like a, we can work with the cube builder folks to actually publish them.

B

um So under the hood, the test environment is actually quite complicated like how, like it all ties, everything up together, um especially because it also supports web books, which it actually it's really nice, um but um it will spin up the aps server. It will spin up at cd that you can see here um and it will find those binaries first and then it will try to to spin it up.

B

It also configure like things like the rest config, so that, like you, can do like a super like a lot of requests like uh per second and like I have a high burst, etc.

B

um But the really nice thing is lucky. You could actually configure everything in a control plane. So like there's a whole thing here like about the api server, so you could set like secure port. If you want to, you could set the path where to find or like define the binaries or you could find the argument. If you have another std like you could potentially configure to use it, we don't do that.

B

We just pin up two two processes, and then um it has happened sometimes that, like we, have found the errors in in in the abs server itself, um and one nice thing about um testing, is that actually, like you can like say like attach control, plane, output and then everything will go to somewhere out in there um as needed.

B

um You could also use an existing cluster if you wanted to um so like this will just like defeat the purpose of like having um kind of like segregated, like environments. um But you know, if you have that use case, uh that's great as well. um You can also install all the crds so, like you could give the test environment say like I want all these crds to be installed. um You can give the crds in here, uh or you could say like I want them.

B

um This install options, which is like you, could give them paths, and so this would actually be set usually to the config directory that you have in here so like in the basis so like all of these crds, um so that, when test m starts like it will just go and find those crds and install them for you, and you don't have to do like anything else. um You can also clean them up after you use them, etc.

B

So there is a like a lot of good support about it and if you have web books, um this is also like why we kind of like have a structure that then has that environment struck is because we installed our own web boards in like web books, and we want to make sure that, like all the web books are registered and up and running for those crds um you'll see here, there's like a custom function like how to install web books- and you know we don't have to go through this, but like it's pretty much saying, um I'm gonna go find all the web pages and register them from the config directory.

B

This is probably something that we can add to control runtime itself in the future. um But what's so good about having my box in here, um you get conversion, you get validation, you get defaulting um all of in test m. So when you like, have this integration test up and running you're creating an object, it will actually go through the same code path that, like a qctl apply, would go through, which is really powerful. If you want to test like a user behavior as well.

B

um Once all of this is done like we, we usually return the struct, which the magic, client and config and then a suite in here like um we just create the test environment, and then we were just like all the reconciliation in here and then we start the manager like we were studying a main.go file um and then we wait for it to be elected.

B

We don't use legit election here. This is just because this should be called something else, but this waits for all the um all the reconciles to be up and um all the caches to have started. So this is pretty important to wait for that to happen before continuing.

B

um Also wait for web books to start because the services might take a little bit to actually like link that all up together and then it will run all the tests and then it will stop the the sweep.

B

um So what happens when we? We do run a test. So as an example, we could go into actually questions.

A

Yeah, I'm to throw out one one question, because this is fascinating and some of the stuff that I haven't actually checked out. Yet most of the tests that I've ever seen for controller runtime and keybuilder have been just like linked off of docs and they were pointing to like the databricks azure one, which has a very different implementation and similar, but very different in terms of like some of these, like gotchas that you found.

A

This is just a complete blanket statement. Is any of this ever going to end up more into those into qbuilder or controller runtimes docs a little bit because it doesn't feel like this is all like. This is so much more advanced than what we get out of like what I've typically seen and I'm stoked by it.

B

um Yeah I feel, like I mean we have been using it for two years plus at this point, so like we're pretty far ahead in terms of like what our requirements have been and how we want to use it in terms of dots. Like yes like. I hope that, like these things like actually are gonna like the goal is like actually to build them in here, but like bring up as much as possible and control the runtime, because then all the browsers can reuse it.

B

um I would love if actually, these things would actually be a little bit easier like so that you don't even have to register, like you know like you, could auto discover reconcilers in the same package, and things like that sort of like a really advanced scenario like we'll, have to think about yeah yeah.

A

Because it's getting so stupid yeah because this gets so screw when you do like the multi-group. True, like you end up with multiple uh sweet tests and they can't run into they can't run parallel because of the way that uh envy test is set up in in at least base q builder, and so I've run into a ton of gotchas around around that. And this is.

B

A

Uh-Huh we'll talk about it later I haven't, I haven't I've been meaning to finally go file, some issues to try and fix some of that stuff, uh where the like some of the ways that you can a scaffold, a project will screw you in the long run, for being able to do some of these things and a lot of it doesn't point to make it make your sweet test like main.go.

A

It's almost like test the individual controllers that are in here, especially when you go multi-group uh test the individual controllers in here, but then it really screws up like massive full test. Suite runs anyways.

B

A

That's an aside: this is fantastic I'll. Let you keep going.

B

Yeah we like we did the complete opposite we're like well, um we want to run all the controllers in a package and maybe even like other packages controllers, if possible, because the system has to be just together like um it's like really uncommon.

B

That, like I could test, for example like the cluster controller or like the machine controller without the cluster controller, unless, like I start making a test to behave like that cluster controller, which is another option, but truthfully like in in production environments like they will have to behave together um and kind of this like yeah, like the reconciliation, looks like I have to behave together, and you know cool thanks for that.

B

So in terms of like, for example, um so one of the change that we have made in cluster api versus um as a builder, you might have noticed, like a huge builder, has um so, for example. Let me take- uh I think this one still has it.

B

Has this before sweet and after sweet, usually um so gingka is not bad, uh but um one thing that we didn't like in copy is that, like you, you lose like the ability to run single tests. If you wanted to um so as an example like if I go into machine delete policy, if I click this run test in vs code, I cannot do that with with gingko.

B

But if I I can do that in here and if I click it like, it's not gonna break, but um it would run the test environment first, as you can see, and then we actually run only that test and that test only rather than like, try to spin up the whole so like yeah, like it just ran test machine to be deleted, but instead like, if I go into the other package, we haven't been able to do all of them yet so, for example, this is still on the old one.

B

I cannot run the suite or I cannot run uh these are like unit tests um but yeah. I cannot run this one only if I wanted to so like if I'm developing and I'm changing this one or like I don't know this line of code.

B

The only way that I can run this is like actually go and type that out like the whole package. I guess, or I could try to do, file tests or something, um but really like you can just say like. I want to just test this. It condition so we went kind of the opposite direction of this and we're trying to move all the other um suites as well to use test main, because this main you know, works well and it kind of yeah.

B

So I can just run one test or debug one test if I want to um so the one other thing that, like we have been doing here, so it's like if I get the machine deployment controller, for example, so like as you can see like, but it's like still some mix of things in here but like um which is fine. uh We one thing that we have been doing is the namespace um segregation of like these tests, um and this is sure all things that, like are gonna, be documented at some point.

B

So, for example, here we create a new name space then create the cluster. I think there was a question like nothing's like can I delete the namespace? Yes, we we were able to delete the namespace, for example.

B

So um after each test so like, if this is run, and then there is like a bunch of tests in here like after each test is run, um we create and delete the the namespace um so that, like the next one, is going to like a b separately in that namespace and the namespace is going to be created with. uh I guess like.

B

If this is using the same name, it should really use a generating name, but, like that's a topic for a different type, um whenever we can, we would use general name instead of like a name um and uh make sure that, like the, if you have a test, that's running um like that. If you want to do count equals two for two, for example, that like a goal of sports- um and you wanna run that test twice, you don't rely on naming to actually say like hey.

B

I wanna create this object and then delete it or modify, or something um in that way like. If you don't have proper deletion in in place like you, you don't get stuck with failing tests if they run like twice in a row with the same test environment.

B

um So that's like the reproducible aspect of it, um the other nice thing about test and is um something that we have found that you you folks, probably want to use this as well. It's like how can I test things that I don't know anything about. um I mentioned like we have the concept providers and like if these products have contracts um so like. How do we go about testing these contracts, um the thing about test and that, like it's, actually really useful that you can inject the crds on the fly?

B

So, for example, here you can see like we have an infrastructure cluster crd. This urd doesn't exist anything uh except for in tests um and there is pretty much like a whole um so like we create these crds in tests and we registered the crd as well when test them comes up so that you could do the same thing pretty much but as a whole hack and I'm trying to to find it that you can do to say.

B

I want to register these uh crds and the crd is like um we pretty much disable the open api spec so that, like it's just say, like plain object like you can go anything you can go in here and it will also preserve with the fields because, like newer version of the api server, actually won't, let you preserve the fields and unless you opt in into into that behavior, um so yeah we try to. We try to kind of like make sure that um that's in place any questions before I move forward.

C

um I've been saying charles, I have. I have some questions. um It's really cool. I I didn't realize control runtime provide this cool testing environment.

C

um I think this testing environment can actually be used for adding any controller that developed, uh developed based on control, runtime or any other like client go controller. Is that right.

B

um This would so like this probably work best with um controllers that are part of controller runtime, so like that you would create with controller runtime. I wouldn't see why, like uh it, wouldn't work with client, go plain informers and like just like more generic controllers, but I have not tested that. So I I'm not able to speak.

C

Okay, I see- and I and I noticed that we only have uh kubi api server and etc in a controller, a control plan. So if I wanted to test something related to deployment or stable set, is that possible.

B

um You need the controller manager, if I remember correctly, for those so probably so like you could create it, but it won't be. It won't actually like spin up like a deployment or it's like it doesn't have any nodes underneath.

C

Okay, so is that um is that in the roadmap like in the future, maybe we will add the controller controller manager into the test framework and we can do some more cool testing.

B

um I I have not seen like an ask for a thing until the runtime. It would be probably hard because this yeah, I'm not sure if because like there, might be no cri and cni like a local use like like it gets like exponentially complicated rather than just like, run the api server in ncd.

B

But um what you're saying is actually like. So these are like usually folks, like call these integration tests um right and um we usually do end-to-end tests with kind. So we have a.

A

Whole, like other.

B

Provider which is captive, which is in inside our cluster api.

A

But you could create.

B

Like um you could spin up like a a kind based cluster with a cluster api, so you could just have like a test cluster in there and then create a deployments, and you know like everything you want to create like on that: okay, okay, so.

D

B

The foundations, the english api to spin up like a a kind cluster and then you can test for um other things, but I see um but didn't control the runtime. I would see see if really yeah. It wouldn't be super durable.

C

So the so so this test framework is aimed at testing only the the workflow or things related to the api, server or etc. Like add, a new object, delete uh object is that right.

B

um I would say like it's: it's definitely meant to be testing like controllers behaviors, uh rather than like um the whole chain of systems together.

B

So, for example like if you see um cluster api has like so many controllers, and some of them have to work like really closely with each other. So, for example, here, like the machining cluster controller, the machine a machine cannot be created without a cluster, but the cluster has to be reconciled and some fields have to be in place for the machine. I see: okay, okay, okay, that makes sense yeah. So those kind of behavior like those are testable in here um other behaviors, which is like more like, for example, hey.

B

I want to check the status of a node in a cluster. um We would either mock it or create a fake node, which you know it won't be represented.

C

Yeah, I see yeah that makes sense yeah because sometimes when I write the controller, sometimes I wanted to reconcile some other workload like deployment, stable, set or even parts. So I'm thinking about yeah, we can add some fake or smoke objecting to the api server. Yeah yeah. That's that's all my question. Thank you.

A

We've actually done something very similar to that for a couple controllers that that I've been on testing uh where we've gone through and basically tried to. We were trying to do reconciliation based on specific states that, like a deployment was going through and if it got into a bad state, we wanted to handle it in a specific way, and so we just mocked exactly the deployment.

C

A

Getting passed into it which which worked but yeah it doesn't, it doesn't solve for the the full e to e test that you eventually want to run to.

C

Yeah, I see yeah, that's really cool yeah I'll, try it out yeah.

B

Feel free to reach out like, and I can post links to the cluster api once and like your folks, need.

C

um Is the class api one is uh in a in a master branch or.

B

B

um Back just here and.

A

Cool thanks for dropping that I'll put that in.

C

A

A

This is fantastic thanks for taking on that tour. I'm gonna ask a couple more questions, so the test main setup that you actually did uh and not using ginkgo for all that for gingo yeah, whatever it's called uh that project for this. um Do you see that permeating outside of outside a cluster api? I mean being that you, you are on uh you head up cue builder and control controller runtime as well. Is that something that we would be changing the community as well to do uh or exploring from that side.

D

We could because.

A

I agree all the same problems that you called out, I think, are like useful, useful cases for one of the for for reasons why we wouldn't be prescriptive in those tools.

B

So I have not contributed to book directly but yeah control runtime. We can draw the examples like how to do so, rewriting all the tests like I'm not a fan of, but we can probably say like hey, like maybe new tests, we can write with like the new, a new test mean or something um but yeah like those who are pain and like going forward that we actually don't allow liking caffeine. We don't allow new tests to be written with gingko.

B

We just say like hey, you need to kind of move to the the test mean unless it's you know, fixing the old test or something yeah.

A

Okay, cool and uh for the setup of that is that just is that just get called does test main just get called for every single test run, or are you calling it from somewhere else that that wasn't? Actually that I didn't see- or I might have missed, that.

B

Destiny is actually called from go directly when you do go test, um so it's the first function that actually gets called when uh a package after init uh the package. When you, when you run tests, it will run testament first.

A

B

It's basically.

A

The sweet, the sweet setup or the before suite call, but not for genko. That's what you're getting no it's it's built in and goes, which is just why it's super nice yeah, okay, cool that was fantastic. uh I I in the middle of this because it was so good. I called a couple more people from from our team in so you might have saw a couple more people jump in.

A

They might have missed a little bit of this, but we can give you we can give you some primers on on or some notes about what uh vince was talking about. uh Gabby and etia.

A

Thank you for doing this. I really appreciate it. Do you want to move on to wade? Do you want to talk about.

C

A

The synchronization proposal yeah.

D

A

Me share my screen.

A

Give me one second.

A

All right, you should have the.

A

A

D

um So can you see this screen?

D

Yep looks good uh singularly emote in in multi-tenancy thinker.

D

So basically uh I had a brief discussion with chris uh yesterday and uh and the day before, so we would like to have a sample code uh implement in current uh multi-tenancy sinker code so that everyone can use this uh as a base to synchronize some annotated crds from super cluster to the virtual cluster that tenant created.

D

So this document is uh try to give a high level of view what we want to do and try to get some feedback from the community that if this is a good approach or not, and then we will do some implementation after that, so the main objective, I think for this document- is try to uh install the crd from super cluster to the virtual cluster when the virtual cluster is started and also we will dynamically build the scheme and see our clients when the synchronization started, and we will also construct example, cr single code on the side of this current virtual thinker code.

D

We don't want to put all the crd sinker code in the current multi-tenancy thinker code base, because I think most cr code is a proprietary, so probably people don't want to share it on the community, so the the basic ideas in this diagram.

D

Basically, we want to reuse the entire multi-tenancy single code as a library or infrastructure, and we build our proprietary code on the sideline in a proprietary uh ripple and what we want to reuse is most of the component in the current virtual cluster multi-tenancy thinker code and uh one only one module will be introduced will be an optional module that, based on the configuration option, that this module is uh in the listener package that monitor the super cluster crd creation and this crd need to be have a annotation called a tenancy super public.

D

This can be a we can still discuss on this annotation, but once this crd has this annotation, this optional uh crd syncer will try to load the crd into the newly created tenant virtual cluster and for all the virtual cluster. We will install the crd uh without any uh any any uh other options, and this will be also automatically install.

D

This new crd scheme into the scheme, which is currently exist in the multi-tenancy code, so with this two options that all the crd has been installed uh and there then user can start to play with the synchronization code.

D

The synchronization code in a user repo is mainly a main function, which is replace current domain in this multi-tenancy code, because you need to have build locally and also we need to try to select the synchronization feature, including the cr, the synchronization, as well as the current core v1 or standard kubernetes resource synchronization, according to current code base, that we need to initial function that to make sure what we want.

D

The resource type need to be registered into the plugin so that this uh code can be start with the new uh resource, type controllers and start to reconcile all the objects. So this is a basically uh uh is this stretcher, but we think that all this need to be done at the compile time. We do not want to support the dynamically installation, like crd, when dynamically installed. We also don't want to support a customized resource thinker when this already running and user want to inject into a new thinker. While it is running, I think.

D

Currently we have quite some issue need to be fixed before we can do fully dynamic uh load of this new synchronization uh modules, and uh so this is a basically.

D

We were mostly using the controller runtime to do all this uh uh client sets and as well as the testing- and uh I think basically the idea is- uh uh is like that.

D

Do you have any questions or concerns based on this kind of a implementation.

A

Structure, we'll definitely want to get phase eyes on this as well um yeah, if you have any feedback as well, that'd be useful.

D

Yeah, so the basic idea, I think, is just uh put this two code base separate uh instead of fully integrated into it, and I'm going to implement this uh either in this directory uh example, or what pha is doing with experiment.

D

uh Exam experiment directory like to put this uh as a new main and uh order uh temple resource thinker, a custom resource thinker in this directory and make sure uh we have at least a template, or example, to teach user how we can implement customize the resource thinker. On top of that.

C

um So, are we going to create a new repo just for the customer code.

D

Yeah, so currently, I think I will put. I hope I can put into an example or another option is put experiment that I have a like the new code face. Doing we have a main, like is a scheduler code. We have a main. We have a package within this experiment or with example, code, so that we can build uh build this uh series thinker code, uh but to refer all this main component to the current multi multi-tenancy sinker code.

C

I see I see um yeah, I don't have any any any thoughts right now, but I think it should be better if we can that way to take a look of this. This proposal.

D

Okay, our ping pay to make sure he reviewed it and uh are going to implement uh example uh somewhere in that.

A

Cool thanks for showing us, uh okay, I've also cc'd faye on that on the issue. Okay, so I assume he'll get a ping from that.

D

Yeah, I also see that another folks working on this area, I think, with a face- a colleague and then also make sure he is aware that change and we can do the synchronization work together.

A

Perfect appreciate you doing that all right, we are at the end of our agenda. Does anybody have any anything they want to bring up any questions.

A

All righty: well, we can cut it about 20 minutes early and give everybody back some time thanks everybody again, uh and this will be posted to youtube after this we'll see y'all next week,.

C

Yep, thank you.

A

Yeah. Thank you. Thank you.