Kubernetes SIG Testing, 6 Nov 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Testing 2018-11-06

Description

https://bit.ly/k8s-sig-testing-notes

A

I would do the note-taking again except I'll, be demoing. First, will you.

B

Want to take notes, I can take notes during all right. So today is November 6th, and this is sig. Testing and I am Dana and we have a couple items on the agenda to talk, get a get a demo of from Andrew about yakety and then potentially talking about D github application of prowl, so Andrew. If you would like to take us away, thank you. So.

A

Yeah, let me share my screen here, start with Safari, I, suppose and then I'll have to switch. So what is yakety well I have a little BEC here and in with a quick demo and the agenda.

A

Let's get right into it, so they're really kind of three types of people that I've seen that work with kubernetes the operators, excessive men's people actually using carbonated kubernetes, the people, building and testing apps foreign with kubernetes and then finally, really I think a lot of us developers building and testing kubernetes and core kubernetes components, and so there are definitely solutions for those first, two types of people, but there's kind of a gap, a limited set of answers for that third type of people.

A

Certain the cube test is one of those answers, but that doesn't work for every platform. Currently, although we should all try to to work to extend it, and so yakety is actually the beginning of addressing that third group of people and extending cube tests. So how does it help? Well again? It's for those developers building and testing kubernetes and core kubernetes components.

A

It has a lot of features similar to cube tests, because, again it's going to be the basis of the cube test extension for vSphere, it's capable of deploying any version of kubernetes one tenant grater on generic Linux distros, and it can deploy single node, multi, node and even multi, control/play node, or what people would call multi master clusters.

A

This one was sort of important it is able to deploy nodes on with DHCP, but still have certificates that have both the node fqdn zone, ipv4 addresses and so no matter how humans is designed to present the nodes. You shouldn't have any chaos errors. Finally, it's actually it's single POSIX compliant shell script, making it fairly portable and customizable.

A

What is a yakety provision? Cluster look like well pardon this. It looks a lot better in the in the SVG formats. Just I can show that here there yeah I just use plant UML. It didn't look great when it generated the PNG, but you've got one or more control, plane nodes and one or more worker nodes control. There can be control plane, nodes that can also schedule workloads, but by default, yakety prefers control, plane, nodes to just be control plane. They have the control, plane, components they have at CD on each of them.

A

Engine x4 load that need to not use TLS for health checks and then each control, plane, node, run-scorer, dns unrelated to kubernetes and you'll, see why in a moment and then finally, the worker nodes use container d and then other things that should look familiar.

A

So what we're gonna do is a quick demo of the yakety OVA and it's designed to deploy yakety on vSphere or VMware cloud on AWS, which is just a nice fear on software-defined data center control plane and it's the quickest way to deploy yakety. But there are others, but the OVA actually does come with some additional features.

A

All nodes have the same host names and deployed clusters, so you can deploy five clusters and if all those clusters had three nodes to controller nodes and one worker node, each cluster would have nodes called Co one co, 2 and wo one, and that makes writing tooling very easy. It can bootstrap from a single OVA, that's great for rapid deployment. It has a service called cube update, and this is where it becomes really friendly to developers.

A

It gives you the ability to update core kubernetes component binaries on live deployed custer's without having to redeploy the cluster so and I'll give a demo of that. But essentially you can SCP. You know, updated, cube, cuddle, binary and updated cube.

A

Api super binary to any of the cluster nodes, and then that will be distributed to the rest of the nodes in the cluster services will be restarted, yada yada and finally, if you're deploying on VMware cloud on AWS, again known as DMC, the OVA will actually can create a publicly routable load balancer, since the cluster nodes will be private, so I'm gonna do a quick demo here.

A

I actually already have one deployed and that's because they can take a while to stand up I'm going to start a deployment process here, but not actually wait for it to finish just so that we can jump right into a working cluster. So the reason the Obey is important I should skip. That is because, with an OVA with VCR, specifically, you can deploy it from a remote location and that's not true of like a template or anything else.

A

It needs to be in an IVF or OVA format, and so that means you can deploy it to a vCenter or, or you know, be Center instance that doesn't have anything already on it conducive to standing up kubernetes, so I'm just going to name this codes cube the.

A

And she's my resource pool.

B

So um I don't think we're actually seeing the window that you have open. Oh crap, let me let me.

A

Do that I wish my desktop now. Thank you.

A

A

The only goats go back a bit then.

A

Sorry about that again what I was saying you can give it a remote location so that the instance that you're, to which you're deploying doesn't already have to have any information about you know any templates or any assets on that V Center, which is great because when I enhanced cube test, I don't want to have to expect people deploying to vSphere to have a template available there as they may not. And so that's why it was very important. This being OVA, we could have used a Nov 8th at had cloud in it in it.

A

But once you start going that route, it just becomes a little bit simpler to do your own customizations. Alright. So here's where you set the values, so you can choose any of the I. Don't know the proper word for it, but it's any of the version string patterns that match the release or dev buckets so release or see.

A

I should be familiar, but also importantly, a URL that can point to any staging location and as long as that location includes these tar balls you're good to go so people can employ staged, builds not staged by Hugh tests it just it expects these tar balls to be there and then I'm going to deploy three nodes to control, plane members and one control plane, member that can schedule workloads. So what that means is the there will be three nodes. Two of those nodes will be members of the control plane.

A

One of those two nodes can schedule workloads, which means you load, that is just a worker, and then we enter credentials, and these are for the cloud provider, but they're also for for this, VM is responsible for cloning itself.

A

And then, if you click this I'm not going to accept, makes things take longer it. The a kadhi will also deploy the kubernetes test, tarball to varla kubernetes /e de on all nodes that can schedule workloads, and we have an image for our conformance tests. That host mounts that, so that all yakety deployed clusters already have the test bits on them and just you can set logging levels.

A

um While that's deploying I'm going to show you what the one that's already up looks like so when it stands up its updates, its notes here with some information and we are going to enable remote app I, never said what yakety stands for it stands for yet another kubernetes, installer, thingy I, don't know I just thought that seemed appropriate.

A

A

Okay, so that that got the cluster information generating commands so I'm going to export this so now, I should be able to SSH into any of my nodes.

A

Oh boy, oh wait: I did not do my jump post one. Second, you can use a jump post if you don't have direct access. So if you explore drum pose, I thought the reefs have to reinitialize that.

A

Okay, so that no deployed you power it on and I'm going to put this over here so we'll start to see new VM to come up. Okay, now.

A

And so here's where the the host names become nice, so the I.

A

Always know what my host names are gonna be and that's why that core DNS is running on each member of the control plane it's so it provides DNS for the actual nodes in the cluster, and this is all just specific to yakety. um But I want to do if you've got old, git nodes.

A

And that's going through the load balancer, since those are privately routed and.

A

I had a problem with a bouncer, something I did there anyway, just ignore that maybe yeah we've been working on it and then the last thing to show you would be this.

A

Let me find oh I've made a change to the cubelet, that's what I did where did I put that here we go okay and then I want to show the cube update service.

A

My says no, oh.

A

I sit there sorry source.

A

Why it's taken so long to SCP it, but that's obtaining an updated cube cuddle to the to the server? And so, while it's doing that, you can see over here that what happens? Is this node clones itself or if times, that it needs to be clumped based on the number of nodes in the cluster, um so that will make it really easy to stand up clusters of any size. So, let's SSH into that node and and if we are long cube, updates.

A

Okay, so we can see here, it's processing the client upload cube cuddle. It skips the out load for itself, but then it uploads it to its other peers. So if I do cube cuddle now we see a Christmas here because that's a custom binary, and that should be the case on any of these.

A

Yep and so that's that's- the the cube update service. So that's that's. Basically, a quick demo I'm, sorry that there was something broken in there like I said: I'd been playing around the cubelet options because I then the reason being is because if you go to the site, you can see that there are directions for installing it using yakety with the OVA, but also there are instructions for just standing up a multi or single mode or multi node cluster with vagrant on Ubuntu, the OVA you just vote on OS.

A

My original testing was with synth OS 7 and then the vagrant work uses bone. So it's been tested on three different Linux distros and it I think it was because I had to change them, swap space settings for the vagrant box, but yeah if anyone's interested I believe I put a link in the doc and again this is gonna, be the basis of the enhancement to cube tests for its vSphere. But since other people do use, use fear outside of VMware I thought. Maybe this could be interesting or useful to people doing their own testing.

A

Now. This specifically the cube, but the cube update service has been really useful for some people. Colleagues, in my on my team, because they can just do a quick compile of a new cube, API server, for example, and just SCP it up and it gets rotated across all the different nodes can.

B

I copy it to any node or do I have to copy it to the co one node you can copy it to.

A

Any node just to slash, cube, update our sorry, slash, cube, update now all nodes know about all the other nodes, so yeah.

A

Anyway, this is just a project. I worked on in my spare time. It's not official for VMware I. Am it to be open sourced by VMware, we'll see how that goes, but it's already out there because we use it to do conformance testing Thanks. That's it. If anyone has any questions just line up, please thank you.

A

Real quick, then I think been once I think been told me. This he's never seen a forty three hundred line, long shell script itself. Anyone I don't know if anyone can beat that probably.

B

A

Oh no, it's somebody else had said that sorry I once.

C

Wrote an entire database replication management system in shell well,.

A

There you go and I used an only reason: I trust it as I use shell check net to make sure that it's it's compliant. It catches most errors, although I did find out today that doesn't support pipe fail, which is a boon to his default shell, which was weird anyway, thanks, guys, Unni.

B

uh That looks pretty cool any questions from people.

A

Alright, well, if anyone does have any, please don't hesitate.

B

So something I'd be interested in thinking about you know is if there any. You know, I feel like over time. I don't know if there's anything that you know would be useful, there's any patterns or whatnot. That would be useful to take into the general like chest. Infra. You know like, if you think you know if the copy's service is interesting or if the naming like here the the way that we you're referencing a version like. Maybe that is something that seems similar to how a cube test works. But you know maybe it's different enough.

B

It'd be nice. If there is maybe if we could start thinking about, you know ways to standardize on some sort of interfaces or, if yeah, but if there are, you know things that we could pull into tests infra that you'd be interested in sharing absolutely.

A

That'd be interesting, I'd love to share all of it. The I do think the cube update service could be useful for for Devitt tests and I mean there's some others like them running DNS and all the nodes, so that all of the host names are always the same I'm looking at adding ovn and OBS support so that, ultimately, even like, so all the control plane nodes would run the ovn control plane and so that all of the nodes would always even have the same ip's.

A

Essentially, and now there is a design flaw or design consideration, rather kubernetes cloud provider interface only ever provides the node name. So from VMware's perspective, the vSphere cloud provider gets the host fqt and from the guest. So I didn't demo this.

A

But if you looked at the host fqdn, it includes a hash in the middle to guarantee that the fqdn is unique, because we are running into issues where multiple clusters were spinning up for conformance testing and subsequent ones would always fail, and it was because the cloud provider could never resolve the host since, and so that's really just more attuned to the cloud provider. Interface, but you had certain patterns like that I think could make their way into other things.

A

Other projects which is really I, think one of the main impetus for sharing it's like. Maybe there's some useful ideas here.

B

Cool yeah all right. Well, thank you for that demo and we have a few minutes left to potentially start will. I, don't know how you are feeling about things on mic and technically this you know mostly ends at 1:30 or you know, 30 minutes pass through the hour and not sure what time zone you're in, but we could start talking about the you know wasted d github of hi-pro, and we can see your video now. So you no longer look like a black box. Okay,.

D

Yeah yeah great so yeah I'll just try to be brief. Just because, like you said, we have about five minutes. So thanks everybody for being here and listening. So basically, the idea is so I'm part of the Jenkins X community and I've been doing a lot of work in Jenkins X around our support for various get providers and we've been using prowl for a while.

D

Now we use it, you know internally to test Jenkins X and some just managed the project similar to you know the way Cooper the kubernetes projects use it, and we also are, if you're not familiar with what thinkin's X is just very briefly.

D

What it is, is it's a attempt to kind of modernize Jenkins and really it's an attempt to build an opinionated application, development and delivery platform for kubernetes, where the various components, including the CIA engine, if you like our pluggable, so you could use Jenkins, you could use prowl, you could use something else, and so basically, so we've been incorporating prowl in Jenkins X with that in mind and and what we kind of like is sort of in line with what I understand you folks have been talking about for a while now, which is just making it less tightly coupled to github and kind of just more generic and flexible in terms of you know, specifically, at least in the short term, in terms of providing support for these different get providers.

D

But then also, you know, as a more general note, I, don't know what the kind of what kind of conversations you've had about this more general idea of you know, kind of just making proud more flexible and generic in general, but I feel like this. The this get piece is just kind of one piece that could be part of a broader effort to do that, to kind of just make it. You know, be this standalone project which I think it.

D

You know, I think that consensus, at least from our side is that it's, you know, Prowse awesome, we love proud, and it certainly has the potential to be that you know if it can be made more generic and so really I just wanted to to bring that up here and to kind of just say like hey I'm here and- and you know, we would like to start contributing to this, to whatever efforts are ongoing and and kind of be part of that conversation, so really I kind of more than just then talking I really wanted to just sort of hear from from you folks.

D

You know where those efforts are what your thoughts about about kind of what I've said so far are.

E

So, actually, prowess not only working for github right now, I made proud to work with Jared, and we also make proud to work with Papa. So imagine you can just shoot. We currently only report the project result to a putzi, but potentially we can also trigger a project from pops up. You want to make proud for a new gift provider. It's as simple as write your own trigger and write your own reporter report. The result back, mm-hmm I, think the biggest thing is github hat. We have a lot of get automations example tied.

E

It's based on github, get up queries, nothing, that's very easy to mimic on other get providers also other we have autos get laid github label management, stuff mm-hm soon, I mean I. Think the root issue is the AI will be they're different across different get brothers yeah.

E

B

Sorry good, what will do you? First of all, are you on the Proud there's a proud channel now on the kubernetes slack and so in terms of for going forward? You know this might be a good place to you know start having these conversations beyond you know. The scope of this meeting and yeah I do think that uh began in general. I think that everybody's interested in you know reducing the cost of integrating with a new provider with new providers. Yes, are you go ahead? Well,.

D

Yeah I was just gonna say that I, if I understand correctly in the meeting, I watch, the recording for the meaning of July I, think it's July 17th and some of you folks talked about just having like sort of a generic get provider, interface or some kind. We have something like that in Jenkins X. It would have to be very different for prowl, just because Jenkins X were it's currently a single process, and so you can just make it a you know an interface and with different implementations of it.

D

There doesn't have to be I, mean there's no IPC happening right. So it's it's a lot simpler, so I mean I, I understand it. That said, that idea is a much bigger one when you talk about in the context of proud, but is that something that is still in the mix? That idea of just sort of creating this generic get interface layer that you can have multiple implementations of like for github or you know whatever? Whatever else hadn't like these first I'm.

B

Sorry, good cool.

F

I think we're certainly open the idea. I think the we're just a little eager to see an actual like API or interface that be consistent between all of these providers.

F

If we can actually get some sort of like plant without then that's certainly something we'd like to have, but it just seems a little tricky to pull off right now. Sure.

B

Yeah I think part of the challenge here will be, you know, potentially trying to distinguish like what things like, probably triggering a build and kicking off testing like. Maybe that is easier to abstract to everything versus you know something like tied or maybe a lot of our label. Plugins right, add or remove this label like I, don't know if you know that doesn't really have a great corollary and get-get and Garrett I don't know about other systems right, but yeah I do think that you know figuring out a good interface.

B

I know that there have been various a lot of the people who are working on the pipeline and build CRT are thinking about working on a trigger CRT and so at some point I. You know I think that would be a good thing for this group to participate in, or you know, figure out and I think that yeah so I think yeah I think everybody's very interested in all that I would like I'd love for there to be a world where it's very easy.

B

For you know some new provider there to hand prowl a thing which then can be you know essentially prowl magically starts working with that hosting service and so figuring. That out, I think is something that everybody is interested in. We just need people, and maybe this could be you who are willing to design it and implement it.

D

Okay, yeah well, in that case, I'll come up with some ideas. Maybe I'll come up with a preliminary document and just kind of presented. Some initial.

B

Yes sounds great and then I'm happy, you know maybe this weren't, some more discussion next week, I think we're a couple of minutes over and so maybe we'll box it there but yeah. This sounds pretty exciting and thanks everybody for coming and see you on online and potentially next week.

B

Okay, thanks great thanks. Well thanks, everybody else bite.