Kubernetes VMware User Group, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes VMware User Group Intro: Best Practices for Running on VMware - Steven Wong & Myles Gray

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Kubernetes VMware User Group Intro: Best Practices for Running on VMware - Steven Wong & Myles Gray, VMware

This will be a presentation by organizers and members of the Kubernetes VMware User Group. This group addresses running all forms of Kubernetes on VMware infrastructure. The group exists to serve users, along with those who develop, test and support running K8s on vSphere and desktop hypervisors. This group is also here to support those building/using cluster installation tooling for VMware hypervisors. Agenda: - Overview of vSphere cloud provider and related storage plugins – with coverage of recent features/changes. - Recommended path for migration from the deprecated in-tree storage plugin to CSI. - New features for running Kubernetes on "desktop" hypervisors. - How to get involved in the User Group to meet other users to share advice and experiences.

https://sched.co/ekHS

A

Hi this session is presented by the kubernetes vmware user group. This group is about running all forms of kubernetes on the vmware infrastructure platforms from a user's perspective, I'm steve wong, co-chair of the user group. I work on kubernetes and a few other open source projects as an employee of vmware.

A

Today, I'm joined by miles the other co-chair miles I'll. Let you introduce yourself and take over.

B

Okay, thanks for that steve. So uh when we get into it here, um we're gonna have a look through the agenda. So uh first thing that we're gonna do is give an overview of the vsphere cloud provider and the related storage plugins. That is the vsphere cloud provider out of tree and the csi driver that we launched about a year and a half ago.

B

Now so we're going to cover any recent changes and features in that, as well as the current architecture of the csi driver and how we're planning on doing a migration from the old entry vcp to the new csi driver that we have, then I'm going to hand it over to steve and steve is going to talk about the new kubernetes features that are built into our desktop hypervisors at vmware, and then he's going to close up with how you can get involved with the vmware user group meet other users of vmware technologies that are trying to run containers and kubernetes to share experiences and advice with each other.

B

So, um let's start off with the vsphere cloud provider and what it does so the vsphere cloud provider uh is is what makes kubernetes cloud native. It's uh the glue layer, if you want to think about it that way between kubernetes and the infrastructure, that means kubernetes doesn't have to be opinionated or doesn't have to have built-in logic about how to interact with the underlying platform. It just knows how to do that via these cloud provider plugins.

B

So there were two different models to begin with. There was the intrigue cloud providers and what that means is the core code for adding support for a new cloud was added to the kubernetes code base itself. It wasn't really a plug-in.

B

It was put a plug-in from the api perspective, but it was directly in the core code, so that meant, if you installed kubernetes, you had that um cloud provider plug-in just sort of installed by default, because it was just part of the core code that led to a lot of challenges, particularly around code bloat and the core kubernetes repo security on security vulnerabilities that weren't getting patched.

B

um If there was a bug, for example, in one of those cloud provider plug-ins, you would have to upgrade your entire kubernetes distribution in order to patch that you couldn't just patch the plug-in, because it's part of the core code. So there was a lot of drawbacks to actually embedding them in kubernetes. So community came together and decided we're going to move all this stuff out of tree into discrete plugins that you have to explicitly install.

B

So we built the vsphere cloud provider, which is the out of tree version of that, and it supports um stuff like how telling kubernetes how to interact with the vsphere infrastructure, underneath how to tag particular vms, how to figure out what az a given host is in vsphere, how that correlates to kubernetes, and it also provides a lot of the infrastructure that we use in the csi driver around the logic of how how the plug-in can talk to vcenter itself.

B

um There are a bunch of primitives that are supported by the kubernetes cloud provider, spec that we don't support out of the box with the vsphere cloud provider, so things like load, balancers roots and that kind of thing, because we have a separate cni that does that as well.

B

So if you want more details, go and have a look at the cloud provider, vsphere sig site there on the slide. So um I'm going to talk to you now about storage on vsphere as it pertains to kubernetes. So we're going to look at the vcp, which is the old entry way of doing things.

B

We talked about the cpi just now and by the way, cpi is what we're calling the cloud provider interface or the vsphere cloud provider, and the csi is the cloud storage container storage interface, and that is what we call our new out of tree version of the storage driver.

B

So the vcp, um to give you sort of an architectural look at things, was what I was just talking about the thing that was built into kubernetes natively. So you can see that here it's just a core part of kubernetes. You install kubernetes, you get the vcp and it enabled dynamic provisioning of volumes and kubernetes, which is pretty much a must-have whenever you're trying to deploy stateful apps on top of kate's. You need something that does dynamic provisioning of storage um and it did uh data service granularity at an spvm policy level.

B

So if you created a storage class within kubernetes that one-to-one maps to a storage policy in vcenter, so storage policy is a vsphere concept which is essentially identical actually to a storage class. So it's very convenient. We just mapped one to the other, but we talked about the drawbacks before. If there was a bug in the vcp or if there was a problem with it, you had to upgrade your entire kubernetes distribution, so say you're on 1.11.

B

You know because the vcp is quite old. Now, if you're on 1.11, there is a bug in the vcp say we fixed it in 1.12 and pushed that into the upstream repo. You would have to move your entire kubernetes distribution from 1.11 to 1.12 a whole major step. So, rather than just being able to update the vcp itself, you had to update everything, so there were some drawbacks there.

B

So with vsphere 6.7 u3, uh we introduced uh two things, so we introduced cns control, plane inside of vsphere, and we introduced a csi driver as well. So the csi driver took the logic that was in the vcp and made it out of tree.

B

That means that you can rev the csi driver without having to upgrade your kubernetes. So again, if there's a new feature or there's a bug or fixed or whatever, in a new version of the csi driver, you just update csi, you don't have to update your kate's version, and you can see that here I've sort of tried to highlight that by showing csi is just another pod. It's another workload that runs inside your cage cluster and it uses this new cns control plane that we built into vsphere.

B

So the old way of doing things with the vcp. It was kind of hacky. If, if you look at how it actually provision volumes, basically what it did was, if you requested a 100 gig volume, for example, it would create a vm with no resources attached to it, but it would also attach a vmdk of 100 gigs in size and the reason it did.

B

That was because there was no primitive uh to just deploy a disk, so it would deploy a vm with a disk attached, unmount the disk, throw away the v, throw away the vm and then remount the disk into wherever it was needed. Obviously, that's a lot of overhead that really didn't need to happen, so we fixed all of that stuff with cns and csi.

B

So whenever we actually made this a product in 6.7 u3, the new cns control plane uses a back-end storage class or a back-end storage, uh provisioning type in vcenter called first class disks. You can have a google for first class disks, there's a good blog by cormac cormachogan on that, and that's essentially like a global catalog for disks within vcenter.

B

So it allowed uh this individual creation of disks without having to have a vm that sort of managed or looked after it. um So we introduced that and that got a got rid of a lot of our uh technical debt that we have in the vcp. It's essentially a ground up rewrite of of the vcp from a storage perspective, and it did just the same stuff.

B

It allowed you to do dynamic provisioning of kubernetes volumes again very important whenever you're deploying stateful workloads on top of case, but it also added some really nice ui elements into vcenter as well for cloud native storage, so there's a whole new ui in vsphere for cloud native storage. So any of your kubernetes clusters in any of your applications that are deployed on top of esphere on top of kubernetes, because they're provisioned via cns.

B

We expose all that information straight into the ui. So you can see exactly you know an example.

B

You can see that there is a mongodb primary node mounting this particular volume and all that information is directly accessible to your vsphere administrators.

B

So they get all of that rich context from the application layer down at the infrastructure layer to more effectively help you troubleshoot things like, for example, we use the same pvc names as you have in kubernetes to identify the volumes in vsphere, just anything that we could do to lower the boundary between the two teams so that they they speak the same sort of language.

B

Then you know that was 6.7. U3 we've had two big releases since then we had 7.0 and 7.0 u1 and we added a whole dip, a bunch of stuff to cns and csi. For that uh the first one was. We offer uh file based persistent volumes now, if you're, using vsan as you're back in storage. So if you need rewrite many volume types for kubernetes, you can do that now. If you have vsan storage, so it will automatically provision nfs shares mount them into the appropriate containers.

B

For you, we added support for all types of vsphere storage for block. So if you only need read write once you can now use that with vsan vvols, vmfs and nfs storage, so any storage type supported by vsphere and additionally, we added a few other bits and pieces. So we have offline volume, resize support. uh There is support for snapshotting disks via valero, with our vadp plug-in, so we can do atomic snapshots at the infrastructure layer of volumes that are being backed up by valero and per persistent volume encryption as well.

B

So you can assign a storage policy that has encryption tied to it and it will encrypt every disk by default. For you, we also added support for exporting metrics from esxi and vsan into prometheus. So if you're using prometheus for monitoring, you can pull the metrics or scrape the metrics from the esxi hosts directly into prometheus. Do auto scaling or whatever it is. You want to do on it there or maybe just build fancy graffana dashboards.

B

So I think we all sort of know how this works, but we're going to go through it here. um This is the kind of the important bit whenever it comes to cns and csi as to how this makes volume provisioning work automatically.

B

So everything in orange here is vsphere. Everything in blue is kubernetes. So the first thing you or your administrator creates a storage policy within vcenter that defines things like raid level quality of service, all that kind of stuff, and then in kubernetes you create a storage class that points to that spvm policy. It's just a name, mapping really really simple and then you've got csi that actually ties the two things together. So you come along.

B

You deploy your cassandra app, it requests a volume, so it asks for that from a particular storage class storage class, looks at the provider and says this: is the vsphere csi hands it to our csi driver? It will then look up the vcenter connection information, so it connects to the correct vcenter. Cns instance. Cns looks at the request, says: okay, they've asked for a 100 gig volume, with a raid 1 policy applied to it.

B

Hands that off to the system creates two copies of it raid one and then passes that back up into the appropriate worker. Node gives it a file system and mounts it back into the container. So fully automated end to end once you have created your sppm policy and storage class, which you only need to do once now. The thing that most of you are probably wondering about is, if I have this old vcp driver, how do I migrate to the new csi driver?

B

That's very good question, so this is something that we've been actively working on in the back end. It's not fully baked yet, but we do have beta support for this as of case 119 and vsphere 7.0 u1. So if you're on an infrastructure level of vsphere, 70, 7.0, u1 or above you know, whenever you're watching this um and you have kubernetes 1.19, you can enable migration from the vcp to csi.

B

There is some documentation up on the github site. So if you want more info on how this actually works or how to do it, go have a look there, um but essentially what it does is whenever you enable this at the cubelet level on each of your kubernetes nodes, it creates a web hook for any calls that would go to the vcp and redirects them to the csi driver. So the csi driver will then take those calls and just translate them into cns calls so it'll do that for any newly created volumes.

B

In addition, you can add a flag or an annotation to your pvcs and that will say migrate. These volumes, you know- or this particular volume. In the background then, because, like I said, we added this new first class disk concept, fcd concept, we need to do a disk migration at the vsphere layer layer. uh Whenever you add that annotation it will then go and unschedule the pod.

B

That's attached to that move the vmdk from just being a standard vmdk to being a first class disk, and then it will mount it back into the pod whenever it spins it back up again.

B

So it does the back end uh data migration for you as well um like I say this will also require a new csi driver as well, because uh the current csi driver that that exists out there doesn't have this baked in fully supported, but if you want to try it in beta, it's in a patch release, so the projected cutoff date for the intri cloud provider is supposed to be kubernetes 1.21.

B

You know it's been said to be 1.16. It was said to be 1.18.

B

The goal posts keep moving, but at this point you know we have a built uh migration path that is in beta, so I would be fairly confident that 1.21 is a good prediction for the vcp driver to be removed from the core kubernetes code base. So I would imagine that that's sort of our cutoff date there, the out of tree driver the csi driver, is already recommended for all new installations.

B

um There is no reason that you should be using the vcp other than it's the only thing that your distribution supports, but you should be able to to install custom csi drivers into whatever distribution of kubernetes you're using.

B

So if you can use the csi driver from the get go, it'll just make your life a lot easier, older vsphere versions, the vcp supported from 6.5 onward, because of this new core code change that we had to do at the infrastructure layer that support will be removed and you'll have to be on 6.7, u3 or above to use the csi driver.

B

So that's sort of that covered. I am going to have a little look now at any of the recent changes. uh What they were. There wasn't a whole lot since the last time we did this at uh kubecon europe. um So there's only just two bug fixes listed here. um We have one where we fix a bug when discovering the vm's ip address by subnet or the network name, so that's been fixed and um fix the bug where the node may be prematurely uh deleted.

B

If vm tools is slow to respond uh to report its host name, so that's been fixed as well aside from that not a whole lot, because we are still developing some of the big major functionality, I'm going to push that up later in the year. So at this point, I'm going to hand it back to steve.

A

Thanks miles, I'm going to move on from the vsphere hypervisor to using desktop hypervisors to host kubernetes clusters. Why run kubernetes on your laptop?

A

Well, yes, for production, workloads hosting in a public cloud or an on-premises data center is recommended, but a local dev cluster is useful in some circumstances.

A

When you want to a learning environment- and you already have a computer, it may be cheaper than paying for cloud hosting when your internet connection is limited. It might be your only option when you want rapid turns, as in early stages of development workflows.

A

A local dev cluster can be useful, but you need to be careful about how this fits with your cicd, tooling.

A

These are the options for kubernetes in a desktop hypervisor, first kind. Second, mini cube and third running a lightweight distribution could be k3s, microcase upstream cube, atom or something else inside a vm, a desktop hypervisor, whether it's workstation fusion virtual box or something else, can support all of these variants and if you're running linux or a docker platform, maybe you can avoid using a general purpose. Hypervisor altogether kind stands for kubernetes in docker and, like the name implies, it runs the various kubernetes subsystems inside docker.

A

It is lighter weight than the other options at the expense of being less like the clusters that you would deploy in a cloud.

A

Kind's lighter weight tends to lead to a life cycle of very short-lived clusters, it's easier to tear them down and then rebuild them as you need them, and these might be more reliable for a local dev cluster use case. You aren't tempted to postpone patches as much as you are with some of the other variants, and maybe you can afford to test against multiple versions of kubernetes.

A

Today, I'm going to demo kind in a workstation hypervisor but see the kubecon europe recording, which is linked here for uh instructions on running minicube in a desktop hypervisor as an alternative, I'm intentionally to do doing this demo on windows for two reasons: first, it's a little less straightforward and if you have a windows laptop, I think you might be a little bit more in need of an assist from what you might learn from a demo.

A

The same thing will work uh on linux or mac. uh Although a few of the install steps will vary a bit they're close enough that I'm pretty confident you'd figure it out.

A

About a month ago, as I'm recording, this vmware released a new version of of the workstation infusion hypervisors for windows, linux and mac. The key new feature that was added for a kubernetes audience is built-in support for running containers and kubernetes clusters, with a new cli called v-cuttle. Now, you've always been able to run kubernetes in linux. Vms and minicube has been supported for years, but now you can directly deploy a kind cluster using the cli and I'm about to show you how this works on windows.

A

I've pre-staged some of the necessary steps, because you've seen installs before, but obviously you start by downloading the installer binary for the desktop hypervisor and installing it the cube cuddle cli is not bundled in with the hypervisor, so you need to install that independently, and this is just a standard.

A

Kubernetes component- that's well documented in the kubernetes docs, um if you're on windows and a kubernetes newbie I'd recommend the option of using chocolaty as an easy way to do it with a one line, cut and paste to an admin command line, and you can see the that command line right here. The other thing I've pre-staged is the installation of a tecton cli and the reason I've done.

A

This is because I'm going to host a tecton, ci cd platform on the kubernetes cluster that I'm about to deploy, so I'm going to cut to a demo screen. If you want to try this at home later in the published deck, I'm, including slides that detail all of these steps with the exact entry that you need to type in on your command line, so download the deck and you can cut and paste commands from it later.

A

So the new container and kind feature is based on technology, similar to what went into vser7 for hosting containers within a rapidly stageable. Vm kind is a binary produced by the kubernetes project for deploying kubernetes components into a docker runtime with workstation 16. What you've got is a version of kind that thinks it's talking to a standard container runtime interface like the one used by doctor, but is hooked up to the desktop hypervisor, so the desktop pipe providers start with a cli vcuttle that is intentionally similar to the docker cli you're already familiar with.

A

Let you let me show you this by typing in v: cuddle with no parameters, you get usage guidance and what you see is the familiar command options of build push, pull run, tag etc, which I'm sure you're familiar with from docker.

A

uh Let me prove it by uh doing a docker or a v cuddle pull of the classic hello world container.

A

I know what you're saying wow steve messed up the demo already, but trust me. I did this on purpose. The error tells you what to do to fix the issue. The container engine needs to be started.

A

This is intentional, so that you, when you don't need the container run time, it can be completely shut down something you might appreciate when you need the most out of your laptop's memory, cpu or or battery so now, I'll type in that v. Cuddle system start that the error message suggested.

A

And after a little while we're going to see that a container run time is available, uh we can type in vcuttle system.

A

A

And this shows what is available to the container runtime if it's needed, this is configurable and you can see the output explains what the current settings are and where the config file and the log lives. Now that that container runtime is available. Let's try that docker hello world exercise again. First, I'm going to pull the image.

A

And then next I'll run it.

A

What we want is full of kubernetes, not just docker, let's deploy a kind cluster, so the instruction for that is simply v. Cuddle kind.

A

And you'll see a new window popped up and this window is set up to use kind and it's going to make it a little bigger here and after maximizing this window, I'm going to do a kind, create cluster.

A

This is going to take a few minutes, probably more than one minute and less than five. The time depends on the speed of your system, as well as the speed of your network connection in the interest of time on this recorded demo, I'm going to cut the elapsed time down, but expect it to take between one and five minutes.

A

Now that that's finished, we should have a cluster, but let's uh prove it by trying a few standard kind, docker and kubernetes commands first we'll try the cuddle images, the equivalent of docker images.

A

You can see that we have that hello world image we pulled as well as a new one in the list associated with kind.

A

uh Next, we'll try cube cuddle version and we'll see the version of the kubernetes cluster, as well as the cube cuddle, client, you'll notice in the last decimal point, they're a little off but they're both version 1.18. So that's fine!

A

Next, let's do cube cuddle.

A

To look at what's lurking in the system namespace, and we see a lot of kubernetes components that are self-hosted in the cluster. We can also do a cube, cuddle cluster info to take a look at what we've got.

A

And the kind binary will also show us what it thinks it's deployed. As far as clusters.

A

We have one cluster called with the name of cli kind, uh we'll use cube, cuddle, get nodes, we can use, become ps to look at running, docker containers and, finally, we'll do uh use the cli for the hypervisor to see what has happened with regard to running vms. We've got one vm running so at this point we have a standard kubernetes cluster with standard functionality nice, but for that developer workflow I talked about it's a foundation, but still not a complete tool kit. What about ci cd?

A

Well techton is a popular open source framework for driving cicd by hosting steps and pipelines in a portable way that takes advantage of kubernetes.

A

The goal is that you can build test and deploy anywhere, and your ci uh pipeline can't tell the difference between when it's run on my laptop or when it's run in a public cloud, the result should be the same anywhere with no snowflake build steps going on. We can use our kind cluster to host the additional tools we need uh for a techton framework.

A

uh This isn't going to be a talk on techton, so I'm just going to get started, but let me at least show you how you can use a desktop hypervisor as part of your cicd process, we'll start by installing tecton into our kubernetes cluster, with cube cuddle, I'm going to download and apply the yaml from the tecton open source project.

A

So there we have cube cuddle, applying the yaml downloaded from the tecton project and you can see a bunch of stuff is going on. Let's take a look, cubecut will get pods all name spaces and I'm going to predict we're going to see the couple of tecton pods they're, not in the ready state yet but they've been deployed.

A

So we'll wait for those to start.

A

A

So now we've got uh techton running tecton runs cicd steps as something it calls tasks. It's a framework for running a ci cicd process on your choice of kubernetes, rather than run a full build test deploy cycle in the interest of time. I'm going to run a techdon hello world example that pretends that emitting hello world is a step that is part of your build process. This comes from a blog post from brian mclean that I'm going to link in the deck.

A

So let me deploy from the yaml in uh brian's github and um we've deployed a techcon step that emulates a a build step.

A

I'm going to use the tekton cli describe command to examine that build step, and you can see that it's started in status of running.

A

So I'm going to try again to look at the state of that technon step, and you can see now that it had a duration of two minutes and now has a status of succeeded and I'll see. What's in the logs.

A

And you can see that logs show that hello world was echoed, so this content is brought to you by a kubernetes user group and if you're using kubernetes on vsphere, I invite you to formally join the group. We have meetings each month where we present tutorials and best practices. The agenda is user driven. So members are encouraged to nominate presentation, subjects and discussion topics, including discussions of feature requests.

A

We have two user tech leads who aren't here today: bryson shepard and joe cersei, who helped us get this group started and we're looking to grow this group into a diverse set of worldwide users. The group is also running a slack channel, which is a great place to ask questions.

A

The kubernetes user channel might be a better place for generic kubernetes questions, but if you have a vsphere specific topic to discuss, the vmware users channel is a great place to find other users and also to reach code and documentation contributors.

A

So the next user group meeting will be december 3rd in the north american time zone. You can go to the kubernetes community calendar to get a conversion to your local time. Time zone and add it to your calendar, use the link in this slide.

A

You become a group member by joining the mailing list, using the link shown here and then finally, there's a link to the group's slack channel miles and I are going to hang around for q a here's, a link that will get you this deck and these are our github ids and you can also get in touch with us and the broader user community inside from inside the groups. Slack channel, thank you and I hope to see you in a future meeting at this point, I'm going to turn it back to the cncf conference.

A