Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Provider vSphere: All Things vSphere Working Group - David vonThenen, VMware

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Provider vSphere: All Things vSphere Working Group - David vonThenen, VMware

Cloud Providers and CSI drivers have reached the mainstream Kubernetes community. Members within the community are deploying persistent stateful applications on Kubernetes clusters both in the cloud and on-premises in production environments. A lot of the design and planning effort architecting these deployments has been through a significant amount of trial and error and painful iteration. The discussion in this session will address various deployment strategies on VMware infrastructure to make your clusters run smoothly, efficiently and more importantly... stress-free. Some topics that will be discussed: - Brief vSphere subproject updates and housekeeping - Cloud Provider (CPI) and CSI driver basics - Discuss multi-tenancy and role-based access introduced in CPI v1.1.0 - Deployment recommendations/strategies for large and multi-tenant environments These topics will be facilitated via a demonstration to help encourage and support discussion with attendees.

https://sched.co/ZevZ

A

Hello and welcome to provider's vsphere the vsphere working group, we're going to talk about everything, uh vsphere related and vmware. My name is david von. Then I am a cloud native engineer at vmware, um my twitter and github handle is at d von. Then my community participation includes kubernetes, obviously cloud provider, sig and, of course, the vsphere working group. Here.

A

My day-to-day activities include recently test infrastructure, related things, ci, cd pipelines and everything kind of related to the cloud, so that includes azure aws and the swords. So the agenda today we're going to cover some brief vsphere working group housekeeping items. Then we're going to do a very, very quick review of cpi and csi and we're going to talk about multi-tenancy and resource constraints as it applies to vsphere and the virtual infrastructure, and then we're going to do a demo of vsphere roll based access and kubernetes and kind of the topic for this presentation.

A

Here is like different deployment strategies. You might be able to use for large and multi-tenant environments as they relate to vsphere users and uh kubernetes clusters.

A

So the first thing uh vsphere working group housekeeping items so where to find us, the working group meets every first wednesday of every month at 9 00 a.m. Pacific, the slack link is right there and then we have meeting notes that are obviously publicly available.

A

We talk about everything that's going on with uh cpi csi, and um the meeting is obviously open and you're welcome to drop in and post and add your topics for whatever you want to the agenda, and we can discuss that.

A

There's also a vmware users group meet that meets the first thursday of every month, so basically the following day at 11 p.m: pacific um and then, where you also might find uh some vmware fellow people at in various areas of the kubernetes community, so we're actively involved in sig cloud provider, sig, storage and sig cluster life cycle and I've dropped the community links in the kubernetes repo. For so you can find the times when they meet so that we're going to talk about the cpi and csi projects.

A

Cpi is the cloud provider interface formerly known as the ccm or cloud controller manager, cloud provider interface. What it does is, it allows you to hook kubernetes into the underlying infrastructure and kind of make the underlying platform visible to kubernetes, and it provides various mechanisms to do high availability zones, regions, load, balancers and and whatnot. We just before this presentation. We cut a release of the cpi for vsphere and it's uh version 1.2.0.

A

It includes nsxt support um which is uh currently at an alpha and then also we have a yaml-based configuration. That's now available, so you can express your cloud config using yaml. um Also, we still continue to support the legacy, any interface or the any configuration but yeah well configures, to make configurations more can come like complies with and is more in tune to what kubernetes offers um the github link is there for cloud provider vsphere.

A

You can take a look at all the other bug, fixes and other features that were implemented just calling up the two big ones that are in uh cpi, so the container storage interface driver for vsphere, so container storage drivers. What they do is they provide uh storage lifecycle, management for the underlying infrastructure or cloud provider that they're on and case for vsphere you know, has all sorts of capabilities for like create uh crate volumes or vmdks mount on mount volumes for particular kubernetes workloads. The latest release at the time of this presentation is 2.0.0.

A

There is a release, that's scheduled, maybe a couple months after this. um This latest 2-0 release includes support for read, write many volumes on vsan and then also extend volume capabilities for block volumes if you're interested in all the other features that are present in the csi driver for vsphere, the github link is there, you can go ahead and look at the release. Notes that are available also would encourage you to look at the uh user group uh vm vsphere user group session.

A

That was uh happened just two days prior to this, I believe, was recorded. It's by steve, wong and miles gray definitely check that out, because they'll cover a lot more uh various other topics of relating to vsphere and vmware.

A

So the first topic we're going to talk about is multi-tenancy and resource constraints, so um just want to do a little quick, like level set for like what we're going to cover and not cover in this presentation, so we're not going to cover multi tendency within a kubernetes cluster. Although some of the things that you're going to hear in this presentation can apply to that, and so it's kind of gets you thinking about different ways.

A

You can apply what we're going to talk about to doing multi-tenancy, doing zones and regions, and that kind of thing when it comes to multi-tenancy and kubernetes clusters, the recommended thing is to have individual kubernetes cluster per tenant or per whatever. Your smallest organizational unit is, and even maybe kubernetes clusters per user so like when that user doesn't need his his or her cluster anymore. They can go ahead and just reclaim it and and or you know, reinstall the new one to get the latest version.

A

So what this presentation is going to cover is how to efficiently and effectively allocate vsphere compute resources. So the biggest problem, when you're dealing with uh physical infrastructure is that you have constrained resources. You don't have an unlimited amount of hosts out there.

A

You don't have an unlimited amount of servers that are out there in your data center, so the biggest problem is resource constraints and because you have resource constraints, you have to have shared infrastructure, so you have vsphere clusters and esx hosts that need to be shared among many many many many different users organization, business units and some organizations might have a lot of infrastructure. A lot of esx hosts a lot of vsphere clusters.

A

You know in the orders of hundreds, thousands, even if you did have thousands tens of thousands of esx hosts and clusters out there. um This you still have the problem of. If you were to dedicate you know, take the kubernetes recommendation and dedicate like a cluster or a group of esx hosts to a given user.

A

If you have hundreds of thousands of users, you know you could easily run out of resource, and so, if you can't afford to have dedicated hardware per user great but in all likelihood that's not really going to happen in like in reality, you're going to have to share your virtual infrastructure with other users. So how do we efficiently and effectively do that in vsphere and share all those resources, so the best and easiest way to do that is use vsphere, role-based access and apply that to kubernetes.

A

So vsphere has native built-in rbac for user access controls. I've dropped the link there for the documentation for vsphere 7., so kind of just to give you like a really quick overview like and like logically, what happens you have like? Normally, you would have an admin account. They access the vcenter server, they see the data center and in that data center you have two clusters: a and b now what you can do with rbac and and having uh like user accounts like for a given user instead of using the one root admin account.

A

What you can do is you, can create uh user accounts per user and then give them each individual access, so role based access based on what their what they have access to so like is in this example. Here we have a tenant or user one who accesses vcenter server, sees the data center and then sees his or her cluster called cluster a and then you have a 10 in user 2 access the vcenter server.

A

They can see their cluster b but, like we said before, if you have limited resources, limited esx hosts a finite number of clusters. You can't have a user have uh dedicated esx host or clusters. You know for that user, because you're talking then about a scale problem and you're gonna have you know it's gonna be costly, and so the best way to do that is to share the share.

A

The virtual infrastructure and esx hosting clusters that are in your vcenter amongst many users, and so one way you can achieve that- is doing our back doing our back plus resource pools and what a resource pool effectively is. Is you have a logical grouping of esx hosts in the form of a vsphere cluster and within that cluster?

A

What you can do is you, can carve out individual resource and and create a logical container or pool that users can create virtual machines within and so that resource pool will constrain, that user to x, number of cpu cycles x, number y number of memory, and then you can have multiple resource pools contained within a cluster, so basically carving out that particular those resources in that cluster to multiple resource pools, and so in the example here.

A

The little picture here you have an admin, his global view, since he has access to everything, can see the cluster can see the tenants, the tenants or users resource pools. You have tenant or tenant one user, one tenant two user: two he has: they have their own resource pools.

A

They have their virtual machines and they're just kubernetes nodes and that's what the admin sees now, if you have user accounts per user, what you can do is you, can uh have those user accounts and apply rbac to them and effectively give that user or tenant only access to their resource pool.

A

So when they log into vcenter, they see the data center and the cluster, which is in read-only, and then you have read-write access to tenant one's resource pool and they have their kubernetes nodes, which are backed by virtual machines and they're, able those that user is able to, like you know, add more cpu or memory to a given virtual machine, and you know they're out able to shift around and allocate and create new virtual machines within their resource pool and same with tenant too tenant two has their tenant, two or user.

A

Two has their credentials, they log in the vcenter server, and they only see the re, the the their resource pool. You know the all the the virtual machines assigned to that resource pool and they don't intermix or commingle and have access to each other's uh slice of the pie. In the virtual infrastructure, so what that looks like from the vsphere perspective, we've kind of talked about it, logically, so we're actually looking at the vsphere ui using vsphere 7..

A

um You can see from the admin point of view here. You can see right the virtual center at the top. Then you have the case dc, which is the data center. Then you have the cluster and we have four esx hosts there, and then we have two resource pools.

A

One called tenant one and one called tenant two and each tenant has their own kubernetes configuration, and so, when you log in to the vcenter server using their individual vcenter account- and you have our back applied to them, um but you basically do our back through the the vsphere ui and the the vcenter ui, and you do the our back configuration there.

A

So when you log in on the left hand side you have tenant one who logs in, can only see the tenant one resource pool and then their virtual machines there, which correspond to kubernetes nodes and then likewise, tenant or user. Two logs in with their creds, can only see tenant, two's resource pool and then their virtual machines. So that gives you an effective way of carving out resource, so compute resource cpu memory from a given set of clusters or a given set of esx hosts and carving them out to give them out to individual users.

A

So now the question then becomes so we have a good means of carving all that stuff out. So what happens when you have a given tenant or a given user, or they have a large number of esx hosts and they need to be able to carve out their own like individual users and carve out their resource pool and dish out x, amount of resource to one user or an organ business unit or an organization and carve out some more resource for a different user so effectively?

A

What I'm saying is what happens if we have tenant two here user two um they have a they need. They have a need for a large amount of kubernetes nodes and large amount of virtual machines, and so what you can do is have more resource carved out and given to the tenant, two's resource pool and they need to carve out and sub further subdivide that resource pool to individual users and business units.

A

So what you can do is you can take that resource pool and effectively have nested resource pools in uh vcenter and in this example, here's tenant one and we're just going to ignore it's just the same configuration that it had before and tenant. Two now says: hey.

A

I have two organizations within my uh within my group um and we're just calling tenant two like the engineering organization, and they have two groups within that engineering organization: one's like the ux team and then one's the back end, services, team and so ten and two might decide to create two resource pools, one resource pool for the back, end services, team and then another resource pool for the ux team, and you know further subdivide the global, the bigger tenant, two resource pool into two other resource pools.

A

And so what you do, then, is you create user accounts for those and apply rbac to those individual sub resource pools? So you create a ux, tenant or user account that can modify and have read, write access to the ux services resource pool, and then you create another user on the backend service, back-end, tenant or back-end user account, which has uh read-write access to uh the back-end services resource pool. And when you create the rbac settings for that particular user, you can effectively.

A

You have the color coded like what they would have access to when they have the need to manipulate resources and access to the and to the individual nodes and kubernetes.

A

So the reason why you do something like this is, if you have like one cohesive team, but you have that one is in charge of like ux, and we have another team in charge of back end, but those services and individual pods that they implement are are tied together and you want them to be accessible without having to like route them through through service mesh or like some other cni mechanism, and you want them to be able to have access um fairly easily to services and endpoints that are exposed by each uh pod that each team is creating, and this is like an easy way of like biting off that bullet.

A

So here's an example cpi config for tenant, two kind of the most important things to highlight here are you have a vcenter section, and one capability that was added into cpi recently was the ability to specify tenants- and you have multiple user accounts within a given vcenter server section here, the section here so and the way you do that is you create a two entries? The first entries we're calling it uh tenant.

A

Two engineering ux, which it represents, is just like a tenant id, and then you have the user account associated with that tenant or that user and in this case we're doing engineering ux. And then you provide a password same vcenter server.

A

You have a user name engineering, ux same uh and then password and then because we're talking about a single vcenter, server, you're talking same server, ip address and then also the same data center and then on the right hand, side. You have uh tenant two engineering be for back-end services, they have their user account engineering be at vsphere, local, their password same the same server, vcenter server and same data center, and by having these two tenants defined in the configuration these center server is able to use rbac and effectively for the cpi interface.

A

Here, it's able to query based and get inventory for each user account apply discovery through our back but cohesively to the whole. For the cpi. It's able to manage the entire engineering organization through a single configuration on the cp, so that covers the cpi and what that looks like in the csi configuration, because you want to be able to keep the the data and, when you're, creating uh vmdks or you're, creating persistent storage for individual pods that are being deployed in your kubernetes cluster for the ux team and the back end services team.

A

You might want to do a configuration where you have two different csi deployments, all right: one csi deployment and configuration where you deploy csi just to the engineering ux team and just using the credentials for that particular uh that particular ux uh engineering, ux user account. So here, if you look at the vcenter server, you see the user engineering ux, vsphere local, and then you do the same thing for the backend services team.

A

So you have a second second set of vsphere csi deployment, configuration yaml, and then this the configuration looks like this very similar, with the exception of that the user account is different and the reason why you do this- and this might be beneficial- is because when you create vmdks and you do storage related operations, each uh each all- the storage that's related to each account will be tied to that individual account.

A

So they'll you'll have the less likelihood that you have like pvcs that are called a certain name and have a collision between the engineering, ux team and the engineering ve team, and so they are able to create their own individual pvcs without worrying about collision and then, at the same time, if there is collision, you don't have to worry about um that particular vmd cave that exists on the ux team, get mounted and placed over on the uh the engineering be team, and it's basically because of these user accounts you're going to have that restricted access, so that it's only aware of the vmdks that are associated with that particular user account.

A

So what this looks like in terms of a diagram so that it's just really easy to digest. So if you have your vcenter server, you have your cluster, you have your tenant one has you know it's just the same configuration that we talked about. It's used kind of just as an example for contrast, but cluster two is where we're talking about. We have a larger engineering resource pool. We wanna subdivide it to two teams: one ux team, one back end team and we have two resource pools for each team.

A

They have their own kubernetes node in the cpi configuration right. We have both tenant user accounts listed in the cpi configuration, so they can have a holistic view of how it does workload placement within the kubernetes cluster, um that's represented by the engineering team, but each tenant has their own csi driver and it has the benefits like, as mentioned before.

A

It has the benefits of only providing access and amount capabilities for vmdks that are associated to that user, account so to that tenant, ux user account so that backend services, ten, that user account that's for each individual csi driver prevents collision accidentally mounting other vmdks from other users to other pods within their their group. I have a demo for you, which actually is going to be the exact configuration that we have here.

A

We have it's going to be a tenant, one which is the very simple same cpi, csi driver for that one individual tenant and then just like in the the previous picture, we're going to have a tenant two which has a two resource pools for a ux team and a back-end team and we're going to deploy a cpi driver that can manage engineering services team and then we have two individual csi accounts that basically create csi drivers for each individual team to prevent that you know prevent access to each other's resources for storage.

A

So my demo configuration here, I'm logged in the vcenter server under the admin account. I have a data center called kate's data center. I have a vsphere cluster called kate's cluster, with four esx hosts on them and just like in the powerpoint presentation um I in the last configuration I have two tenants. um The first tenant is: has a resource pool called tenant one. It has a kubernetes cluster already configured there, that resource pool has simply a user account called tenant.

A

One that's configured using rbac to only see the resources underneath this resource pool right here called tenant, one so tenant two.

A

um We said that in the last configuration that we have to imagine that, if tenant two was a very large kubernetes configuration where it may be required, uh two different business units or two different teams working with each other, that we might have um one engineering, back-end services and another team engineering ux to be underneath tenant, one to subdivide the resources uh under this tenant, two resource pool so tenant to the resource pool here configured using an account called tenant two uh under our back, so that it can see everything underneath ten and two.

A

um It has two resource subdivided resource pools. um One engineering backend services configured with an account called engineering backup our engineering backend services, um so that it can only through our back can only see these two worker nodes and same thing with the engineering. Ux can only see these two worker nodes under its resource pool, and so we're going to do is we're going to configure the cpi or the cloud provider interface for both of these tenants and we're also going to configure the csi driver for both of these tenants.

A

In the first configuration the first tenant it's going to be fairly simple, it's going to be a simple cs: cpi driver and csi driver that gets installed and for tenant. Two we're going to just like in the the last slide, we're going to install a cpi or cloud provider interface for tenant two and that's going to manage the entire kubernetes cluster, so all five nodes so that it can set up zones and then for the csi driver, we're going to configure and deploy two separate csi drivers, one for each subdivided team.

A

So one for engineering, be it's a back-end services, another one for engineering ux. So let's go ahead and do that now on the top window here I have logging into the kubernetes cluster for tenant. One.

A

And I have a simple script called tenant1.sh, which we can take a look at, and it's very simply does exactly what I said. It's just going to go into the cpi folder deploy the cpi for 10.1 and deploy the csi driver for 10 and 1, and let's go ahead and kick that off.

A

And then, if we do a get pods cube, ctl get pods all name spaces.

A

We can see that we have a cpi or cloud controller manager deployed for tenant, one that's going to correspond and do all the work required to surface everything vsphere up into the kubernetes cluster, and we have one csi driver controller, which does the operations for like create and delete volume and uh three csi node uh pods, which handle all the mount operations for this given cluster for tenant, one.

A

So now for ten and two we're going to go ahead and log in to the kubernetes cluster, that's represented by these five nodes, master, one master, node and four worker nodes, we're going to execute a script called tenant. Two. We can take a look at that really quickly um for tenant two, because it's a little bit more complex right.

A

We said that we're going to deploy a cpi driver that manages the five nodes, the kubernetes nodes and the reason why it's important is because that cpi is going to uh create zones, uh regions and zones so that we can divide the the placement for where we want to drop things between either. The backend services resource pool or the engineering ux resource pool, backed by the kubernetes nodes that are in those resource, pools and then, like we said, for the driver.

A

The csi driver we're going to deploy two separate csi drivers, one for the back end services, team and then another one for the ux team go ahead and get that.

A

A

And if we do a cube, cuddle get pods all name spaces.

A

We can see that we have one cloud: controller manager or cpi uh gets deployed to manage these four worker nodes and one masternode and then sets up the zones for all of those hosts or nodes there.

A

And then we have two csi controllers right: one controller for engineering, backend services, one controller for the engineering ux, and then we have one two, three four five nodes for uh the each corresponding to. If you're going to do volume out operations on the the four worker nodes in the master node, so there you have it. um Thank you very much. I hope the demo you found was interesting. um Like I always try to do.

A

I always try to make everything available so that if you want to try this out on your own you're able to do so, and um you can find all of this on github, so you can find out the link there, all the scripts, all the ammo and everything um I included, pictures there. So how you can duplicate the environment and go ahead and you know try to try this out on your own and thanks a lot. I hope you enjoyed the presentation and I guess we're going to be doing q a thanks a lot again.