Cloud Native Computing Foundation KubeCon + CloudNativeCon + Open Source Summit China 2021, 18 Dec 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Best Practice on Windows Workload Management In Kubernetes - Benjamin Wang & Wenli Wei, VMware

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Kubernetes 中 Windows 工作负载管理的最佳实践 | Best Practice on Windows Workload Management In Kubernetes - Benjamin Wang & Wenli Wei, VMware

随着 Windows 容器的成熟，大部分 Windows 应用程序和服务正在迁移到 Kubernetes。即使拥有 Linux 工作负载管理的成功经验，大规模管理 Windows 工作负载也是一项挑战。您知道 Windows 工作负载的暂存空间吗？您是否曾经因过度配置的暂存空间而导致节点崩溃？我们如何避免孤立磁盘？它怎么可能在滚动更新时陷入困境？如何优雅地关闭守护程序？组托管服务帐户 (GMSA) 作为在 Windows 上运行任务和应用程序的更安全的方式，您知道组托管服务帐户是如何集成到 Windows 群集中的吗？你还在纠结于 Kubernetes 上的组托管服务帐户与动态目录的集成吗？

As Windows containers become mature, a large portion of Windows applications and services are moving to Kubernetes. Even with the successful experience of Linux workloads management, it is challenging to manage Windows workloads at scale. Did you know the scratch space for windows workloads? Have you ever had a node crash caused by over-provisioned scratch space? How do we avoid the orphan disks? how could it be prone to get stuck at rolling update? How gracefully shutdown Daemonset? Group Managed Service Accounts (gMSA) as a more secure way to run tasks and applications on windows, do you know how gMSA integrates into the windows clusters? Are you still struggling on gMSA integration with Active Directory on Kubernetes?

A

Hi folks welcome to my presentation.

A

The topic we are going to talk about today is best practice on windows, workout management in kubernetes. We will share some appearance related to windows clusters in product environment.

A

There are two speakers, including benjamin one, is me and wendy wei. We are both coming from vmware. I mainly focus on kubernetes and csi related work and when he is focused on kubernetes bundles related technologies when he is busy with some customer issues recently. So I will deliver this presentation myself alone.

A

There are three items on agenda, firstly, is the background and is mainly about the issues from our customers, product environment. We summarize some experience and lessons based on the issues. That's exactly what we are going to share today. It includes some operation tips for class lifecycle management and how to integrate with the group management service account.

A

First, let's talk about the background. We received some issues from our customers in the approval environment, for example, nissi drive may run out of space when there are a lot of workloads.

A

Our pv couldn't be attached to new workouts during running updates. Sometimes we see some overdues after moving a cluster, it's also challenging to manage credentials on windows platform, and there are some other issues as well. We will share the experience on how we resolved all these issues or concerns.

A

The first tip is about the information each container on windows platform has its own scratch space and actually a water deformities is created, for each container is 20 gigabyte by default.

A

You know usually the root directory for container runtime locates on the c drive by default. So if there are a lot of polygram on a windows note, then the c drive may run off the space.

A

You know these seasonal fives are in the c drop as well. So dc drive is important for the windows os to operate correctly, so we should avoid running out of space on the c drive.

A

The best practice is to configure the container runtime's root directory on a larger trial such as e-trial. In this example, the c-slash-raw slash recap: slash. Data is a link to another. Larger drought, such as the target driver, may change, but the soft link will always be the same path.

A

Sometimes users may need to update the cluster.

A

It is important to make sure that there is zero damn time on the workout, so we should perform our only update, which means to update the workload one by one, but if they are not working out, it may take a long time to finish the learning update, for example, if there are, for example, if there are 50 working out in the clusters- and it needs about 3 minutes to update each workout, then you need about 150 minutes in total to finish, updating the cluster, which is two and a half hours, is a long time.

A

The solution is to let the user set the maximum number of work vms, which can be created or updated concurrently. We call it as marked in flight. In this example, we set the magnetic flight as file, so you need about 30 minutes to finish update. The cost obviously saves a lot of time in the meanwhile. It can also meet the zero downtime requirement.

A

The next operative is also for the early updates. When updating cluster, we need to trim the windows work node, one by one. We assume the max in flight is one for simplicity. Here.

A

The rough workflow is described in the diagram on the right side. The fourth step is to condom the working out. So as the market node as unscalable, any new part will not be scheduled on this workload anymore. Secondly, to trim the work node so are to erect all the existing parts which are running on these windows void. Node, please note that we need to ignore the given set, because the gimbal set cannot be trimmed usually after this step.

A

Two all the parts on the notes should have already been evicted, but somehow some parts may steer running for whatever reasons. In this situation, we need to firstly cure such parts, and it's exactly the third step too.

A

Next step is to watch disk until all block volumes are detached from this work. Node is important. You know. If a block one is not detached from this workload, then it cannot be attached to another workout.

A

So the part assuming the pv is not healthy. Please note that we need to ignore the information created for the container, because we should only care about the block volume for the pv.

A

The last step is to remove the node from the cluster.

A

Okay, the next tip is about deleting clusters. Sometimes users may want to delete the cluster. We need to remove the pivots beforehand, otherwise it may result in often disks if the reclaimed policy is retained, we need to remove the pbs as well previous. We received some cases related to this one. You know some users complain that they still pay for the disk which are not used anymore, so actually all user. They are open disk.

A

The next topic is about the gmsa. Tmsa means google manual service accounts. It could automatically manage a password and simplify service principle. It is widely used in windows, application from kubernetes 1.16 docker runtime supports cmsa for windows working out and is stable in kubernetes 1.18 in kubernetes gmsc credentials back are configured as customer resource.

A

These are background information for the hmic, and you are recommended to read the official guide on microsoft to get much more information on the gmse.

A

There are some preparation in order to use the gmse. Firstly, you need to set up a windows active directory server and create the kdi root key using the domain admin accounts. You also need to create a script group and get the hmac credential associated with the security groups.

A

Okay, this page is the highlight diagram, highlight workflows. The left diagram shows how the geometry work for the windows cluster.

A

Firstly, each windows workload needs to be added into the actual directory, and then it also needs to be added to the security group and our credit spec need to be created on each windows. Work note: the left diagram shows how to integrate the javascript into kubernetes, firstly, install the create credential, spec, crd and, secondly, install the webhook, including the mutating webhook, and the validating webhook.

A

The mutation method is used to expand reference to the jr message into the full credentials back in json format. Within the pulse back, the validation map is used to issue. All reference to the tmsa are authorized to be used by the pod service account.

A

Next, we just need to follow the standard outback pattern, create the cluster row and config the permission on the javascript credentials pack and then create a service account and assign the row to the service account using a class rail planning.

A

Lastly, we need to set the service account in the port and set the credentials back name in the security context to the gmsa credential graph, couldn't just fact reference in pulse back afterwards. All parts running on this windows work node could join the aed and the security group automatically.

A

This page is just the detailed description on the diagrams on previous page, so I'm not going to repeat it.

A

That's all for today's sharing, please let us know if you have any questions. Thank you.