Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Secure Multi User HPC Jobs in Kubernetes with Kyverno - Trey Dockendorf

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: Secure Multi User HPC Jobs in Kubernetes with Kyverno - Trey Dockendorf, Ohio Supercomputer Center

Sites running traditional High Performance Computing workloads are more frequently also deploying Kubernetes for infrastructure. By leveraging the Kubernetes infrastructure, HPC centers can supplement their HPC batch environment with Kubernetes for some very specific use cases, such as interactive HPC jobs. This approach to securely using Kubernetes to support many user’s interactive workloads has been presented and well received at HPC conferences such as Supercomputing 2021. The Ohio Supercomputer Center is currently using Open OnDemand and Kubernetes to securely support virtual classrooms that require running RStudio Server and Jupyter. The benefits of multi-user Kubernetes workloads can also benefit sites who are not traditional HPC but maybe wanting to allow staff a secure place to run containers themselves without necessarily involving the team who maintains Kubernetes.

A

Hi, I'm trade off north of the ohio state computer center, and this is secure, multi-user, hpc jobs and kubernetes with kyverno um a little bit to describe our hpc workload using kubernetes. It's mostly lightweight interactive workloads, so they have minimal cpu requirements compared tradition compared to traditional hpc right now, it's mostly jupiter in our studio.

A

This is serving up numerous classrooms being used by distance learning um quick overview. The technologies we're using so open on demand is how we make the hpc jobs or how we submit them to kubernetes. It's essentially a web portal for hpc access. A key part is, it runs. The web process runs as the user logged in to the hpc system supports multiple resource managers like slurm, torque and kubernetes.

A

Then there's uh kyberno, which is a kubernetes policy engine. You deploy policies using communities, native resources and these policies we'll go over a little bit. Some of the challenges was supporting hpc jobs and kubernetes.

A

We wanted to treat the jobs similar to traditional hpc jobs and allow on-demand apps to submit to both slurm and kubernetes, uh because communities pods can run as root um that made that presented challenges for shared file system access like gps or nfs home directories, uh and we had to ensure that the users processes running in the pod were all being run as that user's uid and gids.

A

This was a big part of this was mainly to treat them uh as them for the file system access um some design patterns, uh all user pods are in a namespace with user dash, prefix um and then their username.

A

These namespaces are bootstrapped by on demand when you log into ondemand, there's also roles and access controls um given to them in their namespace to allow them to do just enough to run on-demand jobs on the uh as both on a managed, pc jobs, um communities itself, authenticates with our key cloud idp and uh the oauth. The oidc tokens for on-demand are allowed to use the kubernetes thanks to an oauth2 audience.

A

uh We deployed a tool. We wrote called kldap config map, so this maps ldap data into config maps that can be used by kyverno, so policy solutions with kyberno. We have policies to ensure a uid and gid and supplemental groups of a pod match a user's ldap record. We have policies to restrict host host path, access amount locations to paths we want them to have access to such as nanotest home directories, gpfs and some files for slurm.

A

We disallow all forms of privilege escalation. We also enforce max resource requests and max runtime of pods. um Here's an example of how we're selecting resources in the policies. So you can see uh over here. We have the user prefix user dash prefix, which made it really easy to match all the user pods by namespace.

A

One thing we learned very quickly was you can only validate on, create and update when you're dealing with ldap data, because if the ldap data changes, um while the pod is running, it might become impossible to delete here's an example of getting ldap data into the context for the policy. So here we're getting the uid mapping, um here's a validate on the run as user. So we're using the uid to ensure that the um we're doing a lookup, so here's an example what the data looks like.

A

So it's user dash my username with my uid and we're ensuring that that uid is what's being used for running this user.

A

um We also validate supplemental groups, which is very important for file system access. So here's an example what the data looks like it's a json string of an array and we're using a denying condition.

A

Here's some additional policies uh this one, sorry, this one is to restrict the host path. They can access, so we allow them to access uh slam, configs, munch, socket for slurm and then user uh home directories and gpfs most of the other policies we use um come from upstream. Comparing the policies helm chart. uh These are mostly around security.

A

um Some of the ones I showed here are custom ones, and here are some of the resources so upstream policies for caverno um the osd policies that we deploy with helm and then a link to the 88 k8 ldap config map, which is uh how we get the ldap data available to kyberno.