Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Enabling Multi-user Machine Learning Workflows for Kubeflow Pipelines - Yannis Zarkadas & Yuan Gong

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Enabling Multi-user Machine Learning Workflows for Kubeflow Pipelines - Yannis Zarkadas, Arrikto & Yuan Gong, Google

Kubeflow is an open source machine learning platform built on Kubernetes. Every service in Kubeflow is implemented either as a Custom Resource Definition (CRD) (e.g., TensorFlow Job) or as a standalone service (e.g., Kubeflow Pipelines). As enterprises start to adopt Kubeflow, the need for access control, authentication, and authorization is emerging. Kubernetes CRDs come with their own auth story, but what about Services with their own API and database, like Kubeflow Pipelines? In this talk, we explore how we enabled multi-user workflows for Kubeflow Pipelines, in a Kubernetes-native way. We present how we combined open-source, cloud-native technologies to design and implement a flexible, Kubernetes-native solution for services with their own API and database. The talk will include a live demo.

https://sched.co/Zeok

A

Hi everyone and welcome today we will be talking about how we enable multi-user workflows for kubeflow pipelines. Let's get started first of all, let us introduce ourselves. My name is janis and I'm a software engineer at arikto working on kubeflow I am presenting together with yuan, going from google who works on kubeflow pipelines.

A

Together we worked on designing and delivering multi-user support for kubeflow pipelines.

A

In this talk, we will explore how multiple users can work together on kepler pipelines in a secure, isolated manner. This completes the kubeflow pipelines experience, which allows you to simplify user onboarding with the kubeflow pipeline's, intuitive ux, accelerate pipeline development by writing, pipelines as python code and collaborate in a secure and isolated manner.

A

So, let's start at the beginning, what is kubeflow kubeflow is a machine learning platform built on top of kubernetes dedicated to making deployments of machine learning, workflows, simple, portable and scalable.

A

Now then, what's keyflow pipelines, kiplo pipelines is a cornerstone of the kubeflow platform. It provides a powerful platform to run machine learning, pipelines on it comes with a powerful ui and a diomatic python sdk to define pipelines as code rich visualizations.

A

Linux, tracking and more so, as we said, q flow pipelines are a very important component of the kubeflow platform. However, when admins try to bring pipelines to their users, they quickly stumble upon the problem that kubeflow pipelines has no isolation, authentication or authorization.

A

All pipelines are running in one namespace, one search environment, which means that secret and data isolation is impossible. How are we going to isolate, authenticate and authorize to the kubeflow pipelines? Api kubeflow pipelines is not the kubernetes native api. It is not based on crds or an aggregated api server. Instead, keyflow pipelines has its own api server and database mysql and object store in the initial kubeflow pipeline's design.

A

There is no mention of isolation, authentication or authorization. So how can we extend the initial design to enable multiple users to work together in a secure, isolated manner, to design how to isolate, authenticate and authorize for the kubeflow pipelines api? We first turn our attention to the kubernetes api for inspiration for isolation. Kubernetes uses namespaces almost every resource lives in a namespace for authentication.

A

The kubernetes api server supports multiple methods like authenticating user's, poi dc. That is open id connect, authenticating machine accounts via service account tokens or getting the user identity from http headers set by a trusted proxy.

A

Finally, for authorization, kubernetes uses role-based access control over our back. For short, permissions are defined as roles and given to subjects via role bindings. Now, let's see how we can apply those principles for designing isolation, authentication and authorization for the kubeflow pipelines api.

A

First of all, let's talk about isolation. Ql pipeline's initial design has no isolation, primitive whatsoever, all pipeline runs are shared and everyone can access them. We introduce the namespace as an isolation, primitive same as kubernetes. In the queue flow pipelines, data model there are pipeline definitions and pipeline runs, which are instantiations of those definitions.

A

All pipeline runs reside inside of an experiment now to extend the cubesat pipelines data model with an isolation, primitive, we add a namespace attribute to experiments pipeline runs further. An experiment by looking up at five minutes in the story pipeline runs find their namespace by looking up their experiment.

A

In addition, the experiment api endpoints are extended. All api calls must now specify namespace same as kubernetes apis.

A

Then, let's talk about authentication as we saw the kubernetes api server supports various ways of authentication. We want to support that in the pipelines. Api server as well. We want to be able to authenticate users as well as service accounts, so we designed the following ways of authentication.

A

First of all, a user can authenticate with oidc at the history gateway provided by kubeflow and then authenticate to the pipelines. Api server via http headers. Here the steel gateway acts as a trusted authentication proxy for machine authentication. Htmpls is a good solution, but it can't easily be used with all kinds of workflows, for example around the completion workloads like argo workflows and kubernetes jobs, don't fully support running with an issue sidecar.

A

For that reason, we also designed a third way of authentication authentication with the service. Account token now be careful. We use service account tokens with a custom audience issued by the token request api.

A

These tokens are only usable at the pipeline's api server and not the kubernetes api server and as job video will tell you, handing a service account token. An unbound service account token to anything other than the kubernetes api server is a very, very bad idea.

A

Now we saw what we did about isolation authentication. Now, let's talk about authorization. Authorization is all about answering the question of. Can user do action on a resource in namespace, for example, cansara list pods in namespace kubeflow in the first implementation, keep flow pipelines are using the queue to access management, service or k-form for short to authorize requests.

A

Kfam is introducing a simple view. Edit admin access control list, abstraction for all actions on top of kubernetes are back, as you can see from the permission table below. There are only three levels of access available, so an admin can specify a more fine-grained policy, like a user, can view and create pipeline definitions, but not start pipeline rounds. This method is deprecated and will be superseded by our bug and subject tax review.

A

After talking about the current way of doing authorization, let's also see where we want to go. The endpoint for authorization is to map every kubeflow pipeline's api endpoint to enter back permission. For example, we get the api's, v1beta1 runs id api call to get runs, and so on. The information about permissions lives in kubernetes roles and role bindings. Just like with native kubernetes resources, then permissions are checked using a subject.

A

Access review poll now be careful and make sure to note that these resources are not crds, but we can use the subject: access review api to authorize them anyway, so we are essentially using kubernetes as an authorization system, an authorization database and enable admins to assign fine grain permissions. In the same way they do for kubernetes native apis with roles and role bindings.

A

This method of authorization will be implemented by arito.

A

So we designed authentication authorization and isolation for the kepler pipelines. Service isolation is achieved using namespaces same as kubernetes for authentication. We support multiple ways for both human and machine accounts for authorization. We want to move to using native kubernetes or back permissions and subject tax review.

A

Now, let's see how it all comes together, a user logs in at the http gateway and is authenticated. Then the easter gateway will proxy the request with the user's identity in http headers, when inside the workload with no human interaction, for example, a jupiter notebook, we can authenticate in two ways we can authenticate with a service account token, with a custom audience issued with a token request.

A

Api call. After issuing the token we send it via the authorization better header in http requests to the pipelines api server. Finally, we can authenticate via istiomtls, if that's available, for a workload.

A

Finally, the pipeline cpi server now that it knows who the user is and what the user wants to do will ask the simple question of: can user do action on the resource in namespace and issue a subject: access review code to authorize the request, depending on the answer of the authorization decision, the pipelines api server will then execute an action on its database.

A

So all in all, we explore kubernetes design for isolation, authentication and authorization and use it as a guide for the kubeflow pipelines api we isolate using namespaces authenticate using oidc service account tokens or s2mtls and authorized with kubernetes are back for the next part of the talk. I will hand it over to yuan, who will be talking about isolating visualizations networking implementation considerations and more.

B

So thanks janice, this is rian at google cloud working on kubeflow pipelines, janice just introduced three ways people may authenticate, but for the current phase we have only designed and implemented using authentication info as a http header. Now that the kfp api server takes a plain text: header for user identity, it sounds dangerous right. So how do we make sure clients cannot pretend to be others by faking a request?

B

This is where s2 mutual tls comes into the play. You probably know that so here's a really simplified explanation.

B

First, we have a client talking to a server through plain text http after turning on mutual tls, the request is proxied through an istio sidecar in the client pod. Then the client sidecar knows how to secure the connection to the server sidecar and the server sidecar talks to server container in http again.

B

In this way, the client container and server container are still talking in plain text http, but the connection between pods are secured and by this mutual tls. Amazing. Isn't that so in addition to that, the server-side car now knows identity of clients. Talking to it. Therefore, we can configure some authorization rules like only allow requests from this namespace or like these service accounts.

B

So, with these hto features, we are able to set up a set of security rules to first allow traffic from instill gateway to kfb ui allow traffic among all kfp components, allow traffic from kfp to dependent services and the most important one being denying traffic from non-kft sources to kf internal components.

B

So, finally, with all of this, when kfp api server receives a request, we can be sure it's coming from the trusted. Istio gateway.

B

So it took us a lot of efforts just to securely pass the user identity to our api server. That's, unfortunately, just the beginning of this story. The api server also needs to support multi-user separation.

B

Think about the generic problem of adding user separation to a service. There are typically two options to do this. The centralized option is to build a service that take, for example, here namespace as a primitive concept and enforces authorization rules on it. It takes more efforts to implement because we have to do intrusive logic, changes throughout the application.

B

Basically, we need to add a namespace argument, authorization and application logic to every api method and, don't forget, db, schema migration tool. On the other hand, the decentralized option is to start independent instances of the original service for each user. We, then we need a gateway in front of it and we need a controller that auto sets up the services in new username spaces.

B

This option is non-intrusive, we're just composing existing application instances in a way that they look like a single service with multi-user support, so for kfb api server. We decided to build a centralized one that takes namespace as previously described, so a single instance means lower operational and computational cost.

B

This is important, especially because usually a platform team can handle kfp systems operations like upgrade or backup, so data scientist teams can focus on their own work. So other reasons, including reserving the possibility of building cross namespace features like sharing, and this comes with a cost, but it's a cost only for us. We did a lot of code changes through all apis made. The namespace parameter, primitive concept and added authorizations in place.

B

Also remember: kfp provides end-to-end, reusable machine learning workflow. So it looks like this seeing visualization right inside kfpui. We also provide tensorboard, self-service instances and artifact preview right inside ui. However, users may decide to store their artifacts in external storages, like google cloud storage, so cafe. Ui server needs to fetch these artifacts for them in order to show them on the on the ui. So how do we separate kfe ui permissions?

B

For this case, we decided to take the decentralized option here, because these artifacts tensorboard and visualization services are stateless, lightweight clients. So there's not much operational cost anyway.

B

Also users are free to configure their own instances with correct permissions in this way.

B

Additionally, one contributor is adding volume support for data passing a decentralized factory servers can even support mounting persistence volumes because they are also in user name spaces.

B

So, comparing to kfp api server, we took a different approach for artifact servers because they have different characteristics and tradeoffs that, in the end, both different options make sense for different things. We are considering. So it's interesting that we got the experience of building multi-user support, with both options for different components in kfp.

B

There are still some missing pieces for this design. First, how do we integrate other two types of authentication methods with in-cluster traffic authorization?

B

They can allow other workloads in the cluster to authenticate to kft api server. Second pipeline definitions are still shared. That basically means code. Pipeline code is shared, so this is okay for some organizations, but not others.

B

Then menu object, store and machine learning. Metadata db are dependencies of kfp. They don't support separation themselves. So it's still an open question. How kfp can support separation for them.

B

So after discussing all about the design, let me share interesting things. I learned when implementing this feature. First on background, I was learning still from the beginning through this, so I hope these experiences can help someone who's. Listening to and disclaimer, I was working with st 1.1, which was a fairly old version. Kubeflow was using it's likely. Some of these have been improved without my notice.

B

Okay, so let's start lessons. I learned the hard way uh before using any istio features. Clients and servers need to have istio sidecars injected. That became my immediate blocker right away. Remember kubeflow is a platform of machine learning applications on cloud. There are a lot of applications other than kubeflow pipelines in the same control, plane, namespace pipelines was actually the first applications to require is to security in this namespace. So I had to turn that on turn it to sidecar injection on.

B

um So there are several things that can configure it. There's a cluster-scoped config map that changes, whether parts are injected by default, there's also a namespace label that is required before any auto injection can happen in the namespace. Of course, there's also a pod level annotation that works for just the parts.

B

So in order to turn on istio sidecar injection, what I wanted was that all pods in the shared name place, should default to not inject, while the kfp parts can use pod level annotations to turn on injection, but that's actually not feasible, because the default policy is a cluster scope configuration uh while other namespaces in kubeflow have already been depending on that behavior.

B

uh It would be a bigger problem if I wanted to change the cluster scope default value. So in the end I had to add the disable injection annotation to every other part in the shared namespace, with a really huge pull request and got some really hard time, troubleshooting all the pre-summit tests, but it was necessary- and it was a great step for the entire qfloor community to move towards integrating more issue security features.

B

So when gradually migrating to istu at first, I was tempted to first set up some authorization rules without turning on mutual tls, um because who knows the documentation looks complex for mutual trs, and so I I thought, probably I could skip it, but please be careful that wouldn't work for many authorization rules, because a rule that refers to even only name spaces requires mutual tls.

B

It still doesn't magically knows the namespace of a request.

B

Mutual tls is on these two side cards mutually authenticate each other, so they know each other's identity, for example, service account a in namespace b. So that's why it still can authorize requests by namespace to put it in another way. This is actually um listen carefully. This is actually a live takeaway. So before giving anyone money, don't forget, authentication, you need to ask them prove who they are.

B

First, this is as important as authorization, so these were just showcases of the hard problems that took me on still learner really a few days to figure out troubleshooting.

B

So my takeaways are: do not skip the basics. Really understanding is two concepts save more time later and it's good to learn. Troubleshooting techniques for istio system end to end, um there's a great documentation for that. But from my experience most authorization problems can be solved with just one section in the huge troubleshooting doc.

B

That's the ensuring proxies enforced policies correctly section. It tells you how to enable debugging logging um for a site card. Then you can observe what information is still parsed from a request and whether it was.

B

Denied so at the end, remember that this couldn't happen without these amazing people. We all worked on the design together, while ning chen and I at google cloud implemented current features.

A

B

A

Let's proceed with our demo in this demo we will see two kubeflow users working on the same kubeflow pipelines, installation or orpheus and yuri dies are two users of kubeflow who are working on the same kubeflow pipelines installation. But, as you will see, the resources are isolated. Let's see what happens.

A

First of all, orpheus will log into kubeflow.

A

When orthos logs in he will be greeted by the kubeflow central dashboard, where he will navigate to the notebooks page and start a new notebook server here, orpheus is using the codelab tutorial.

A

The titanic codelab tutorial orthos will also select the allow access to crypto pipelines and allow access to raw configurations which add specific service account tokens to the pod in order to access the pipelines, api server and the rok api server, when the notebook server is ready, orfes will connect to it and open the jupyter notebook he has prepared beforehand.

A

Now, in order to convert the jupiter notebook to a kubeflow pipeline, orphans will be using the kale tool with the kill tool you can just tag your notebook cells and with a click of a button, gail will parse your jupiter notebook and convert it into kubeflow pipeline.

A

As we can see here, kale converts the notebook, submits the pipeline to the pipelines, api server and will finally present orpheus with a link. So he can track the progress.

A

Orfus visit visits that link, and here he can track the progress of its of his run.

A

Now, let's go to the other user.

A

Yuridise also goes to the kubflow page and logs.

A

A

After that, yuri dies will list all pipeline runs and, as you can see, she cannot see the runs which or face hers have started. These runs are in namespace, cuteplay orpheus. While she is working in namespace you play yuridize when she tries to visit the link for one of our faces runs.

A

She gets an error because it's not allowed to do that.

A

And when going to the experiments page, you can see that she cannot see or faces a titanic experiment as well to find it to prove that we can list the pods for the namespace of our fails. Called cube. Flower fails and see that all the around pods are there, while in kubeflow, uridize no pods exist, and this is how we isolate kubeflow pipelines for multiple users.