Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Building a Global Supercomputer with Virtual Kubelet - Dmitry Mishin & Adrien Trouillaud

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Building a Global Supercomputer with Virtual Kubelet - Dmitry Mishin, University of California San Diego & Adrien Trouillaud, Admiralty

Nautilus is a global Kubernetes cluster, product of the Pacific Research Platform (PRP) project at The University of California San Diego (UCSD) and many collaborating campuses. It aggregates compute resources from around the world. While that's impressive, there are issues that a single cluster cannot solve: in this case, decentralized governance. PRP participants delegate the control of some of their compute resources to UCSD. That wouldn't scale to a global network of independent institutions. So PRP, along with peer projects nationwide, decided to adopt Admiralty's open-source multi-cluster scheduler, which allows for decentralized control plane topologies. Dmitry will demo the federation of Nautilus with the clusters of other large scientific networks; Adrien will explain how Admiralty implemented Virtual Kubelet and the Kubernetes scheduler framework to make this possible.

https://sched.co/ekBh

A

Hello, everyone: this is the talk on federation in kubernetes, I'm dmitry mission. I work for university of california, san diego.

A

This talk will have two parts in part one. I will describe our cluster kubernetes cluster, how we use admiralty, how our project came to the current state, it's in and in the second part uh adrian, the creator of admiralty will describe how admiralty is working. Also, we will have the demo on how admiralty works.

A

So our project is called prp, pacific research platform and it grew from a project measuring network performance.

A

So the initial problem was that university of california, san diego, had multiple campuses connected with, what's called science dmz, a scientific network uh 10 to 100 gigabits per second, so pretty fast, and sometimes this some segments of the network doesn't perform the way they're supposed to and prp was measuring uh network performance by sending test data between those locations uh that was done manually and at some point uh there were just more nodes that prp could handle manually and there was an idea to deploy kubernetes on top, so that kubernetes would orchestrate those measurement software.

A

uh Once we deployed kubernetes, we realized that now we have a cluster of nodes. uh We just occasionally ascend data, but mostly city idle, and we started providing those hardware resources to scientists uh to just do their computations and we saw exponential growth of our cluster because it was really easy to onboard our our system, uh bring in the container, run it and get free access to the resources for scientific use.

A

B

This cluster nautilus.

A

At some point, more projects started donating hardware to us. uh We partnered with internet too open science grid donated some notes. uh More networks were coming to us, and this is how now it looks in us.

A

So we have a scenic network in california, providing us the internet access, and there is a bunch of regional networks that also donate hardware to our cluster. So basically, we have national kubernetes cluster aggregating, a bunch of computing resources.

A

At some point we started bringing in gpu nodes, storage, nodes and so on. So now our cluster is pretty big and at some point it started growing internationally, so an international scale. uh We have notes in europe in asia and pacific region region. We have known in singapore, korea, guam, australia,.

A

And if we're talking about nodes themselves, those are called fiona nodes, flash io network appliances. So those nodes are optimized for fast networks, so they can actually leverage those speeds and some nodes can fit up to 8. Gaming, gpus and some nodes have a bunch of hard disk which we use for storage for ourselves.

A

A

So when we talk about federation, our current state is that we have our prp notice, cluster national for us and some international nodes. um Now it has 7000 cpu cores more than 500 gpus 2.5 petabytes of of self storage.

A

Additional to that we deployed a couple of smaller clusters. We have a development k3s arm cluster. We have development windows, kubernetes, cluster agonist, gpu cluster, that's for gaming and 3d visualization, and we spawn to aws cloud.

A

So all those clusters are separate. It's really hard to integrate all that hardware in one and we need a federation to work with them. Also, we partnered with several other smaller clusters.

B

A

There is a federation already established between those locations, so basically all clusters can finally talk to each other, but are controlled by different organizations and different people.

A

That's the huge benefit of admiralty, because all our other projects are seeing federation as one cluster controlling others. It's not one level federation between them and what we are looking forward to is when expense supercomputer is deployed.

A

It's also going to have the kubernetes cluster as a part of it and admiralty will be used to federate expense with notos and other clusters, and that will bring a theoretical expansion to 100 000 cores. Of course, we will not get all of them, but we're that's a pretty big increase in what we federate with.

A

So how can we use resources between those clusters? uh As I said, uh most of these clusters belong to different organizations who want to set their own rules. So we cannot federation cannot dictate uh policies for running stuff in different clusters and scientists who use those clusters usually have access to only one or several namespaces. They again don't control the whole cluster and so federation should uh allow working on namespaces level when people don't have to bug the admin to federate. But they can establish federations between the piece of the cluster that they control.

A

Additional to those small clusters which are on-prem, as I said, we federate with clouds uh amazon, microsoft, azure, google cloud if we need some resources from those clusters. If we have some big workloads or we want specialized hardware, we can always federate with those by simply running some temporary cluster in those and then establishing the federation.

A

So how do we use federation now.

A

Gitlab ci is what we started with it's super convenient to like when we need to build a container on some specialized hardware like arm or windows uh instead of manually, creating a gitlab runner somewhere else or trying to join this node in our big cluster. We just federate and that will be in the demo. So now we feather it with windows and arm and our gitlab by just tagging. The job can build the container in some other cluster and this container will be stored in gitlab registry network monitoring.

A

Right now we have a bunch of monitoring ports in our large cluster, but the goal is that any other cluster that wants to have automated monitoring can just join our federation and get all the monitoring pods.

A

uh Jobs bursting if some cluster has some unused resources and another one is overloaded. It's always useful to just burst to another. Cluster and clusters can share their computing resources without actually.

B

A

Nodes, so that's super convenient medical data uses a big one. uh Some data can is highly protected. It cannot leave some cluster, but this data can be used for uh some computation and then results of this computation can be shared when it's anonymized, so federation again, can let you spawn your computing job in another cluster.

A

Get some product anonymize it get it back. So this is super convenient special devices, internet of things. Some devices are really small and tiny and they can't just join a big cluster and just monitoring pots will kill it. So again, this is super useful to federate, and some iot devices are again controlled by some other people and federation helps establish those connections without getting or requiring access to the specialized hardware.

A

uh Future of federation, we are now working on the project that will allow us to uh create on-demand player to a fast pass around the world.

A

So there is a that's a project with surfnet, it's called nsa autogoal, and this project already can create a layer, two connections and tear them down on request, and what we are writing now is a special crd and operator which will allow pods that spawn in some remote location to create a path, for example, to storage in some other location, and this path will exist while the pod is running and then this path will will disappear and federation will help us to run this across several kubernetes clusters.

A

So if some cluster wants to talk to another cluster, it just creates a layer, 2 connection and that will be controlled by standard kubernetes api. So this is a really great project that we're working on is the demo on federation.

A

In this demo, I will show how federation works in our production cluster. I will not go through details of establishing the connection, it's all covered in documentation, and this is supposed to be the short demo.

A

So now we're looking at the nodes in our production cluster, and we see that the first node has the role of cluster and master. This is virtual node created by admiralty. So basically this represents the whole remote cluster and federated parts will run in this node, while the actual node is below it. In this demo, I will show how our gitlab can spawn parts in other clusters and get results from them.

A

So here is the jobs table of our gitlab. The these jobs are built in container and putting it in gitlab registry, but while uh most projects are just building uh containers in our.

B

Regular cluster.

A

This uh project is setting up this architecture arm 64 tag on the jobs which make them go to special runner in git lab.

A

Let's look at our gitlab namespace, so in this namespace we will see that it's labeled with multi-cluster scheduler enabled this will tell admiralty to look at this namespace and wait for parts that we mark as federated.

A

If we go in this namespace, we will see several runners running already on regular notes. If you look at deployments, there is already deployment that I created called gitlab render federated. We can look at it.

A

We will see that in template for parts it has the multi-cluster administrator elect annotation. So this again will tell admiralty that parts from this deployment should be federated, so admiralty will decide where to send them to.

A

Let's scale this deployment to one.

A

And we will see that this new pod is scheduled to run on this virtual node. So if you look at all uh ports, all of them are in our cluster, but this one part is running somewhere else.

A

We can look at our arm cluster.

A

Again, this is gitlab namespace, but this is now remote cluster federated with our large one, and this is the node. This part is running on, so we see that gitlab runner actually started on remote node and in our large cluster. We only have the proxy for this pod.

A

Let's switch back to our main quest.

A

So this runner already registered automatically in gitlab and if we rerun one of the jobs.

A

The runner will run it remotely.

A

So if we go again to remote cluster, we'll see that new.

A

A

B

A

Finished this is pretty quick and the result of this job went into the local container registry in our gitlab. So this completed successfully, this allows gitlab to run runners in federated clusters without manually setting them up and control them from one single location.

B

Hi, my name is adrian: I'm, the ceo of admiralty admiralty, is the company behind the open source project of the same name. Admiralty is what makes the decentralized federation presented by dimitri possible. It's a multi-class control plane that uses common kubernetes extension patterns. This talk will focus on virtual cubelet and the scheduler framework.

B

What is virtual cubelet? This is a screenshot from the virtual cube website. So like.

A

The kubernetes.

B

Cubelet that runs on each node, a virtual cubelet instance presents itself as a virtual node, but instead of running containers on a local vm, it runs them on a remote system and there are 11 known providers to the state, including admiralty in the case of admiralty virtual cubelets represent remote clusters to help us understand the concepts that I'll explain later.

B

Let's consider this example use case where a user submits jobs in their own cluster cluster, a but containers run in other clusters, b and c, and so in cluster a there are virtual nodes that represent the other clusters. The bots, created by the kubernetes job controller, are mutated by admiralty submission controller into what we call proxy pods.

B

We call them like that because they represent other pods, not because they are proxy in the networking sense. Admiralty's proxy pod scheduler schedules those spots to the virtual nodes and creates candidate pods in the other clusters.

B

Some of those candidates become delegate plots that are bound to real nodes. Those real nodes could actually be other virtual nodes. You can imagine several levels of inception.

B

Admiralty also includes a bunch of controllers to update bot statuses, make config maps secrets and other dependencies follow. Pods notice that cluster a needs to talk to clusters b and c and we'll see different ways to do that in a minute.

B

Let's go back to virtual cubelet and how admiralty implements it. Virtual cubelet has four main responsibilities. The first responsibility is to register a node object. Admiralty creates virtual nodes based on user created targets and cluster targets.

B

Those are custom, resource definitions that basically uh give a name to a virtual node and refer to a secret that will be used to talk to the corresponding target cluster. The second responsibility is heartbeat.

B

Kubernetes needs to know that a node is healthy at regular intervals. Otherwise, it'll evict the pods if the node is not responding. So the virtual cubelet in admiralty um checks the health of the target clusters and updates the condition of the node.

B

The third and maybe the most important responsibility is to handle pods so to run the actual containers somewhere um and in admiralty. This is most of the logic with multi-cluster scheduling, status, feedback, cross, cluster, garbage collection and so on know. Of course, there are some uh uh other features like uh handling logs requests and exact requests and in admiralty's case it's very simple. We just forward those requests to the target clusters. Kubernetes api.

B

A

B

Rows require two configs to call the target clusters.

B

So, let's talk about that in admiralty clusters are connected in the control plane in one to one relationships, and we say that the source and the target cluster are connected when controllers in the source. Cluster can call the kubernetes api server of the target cluster.

B

For that we need three ingredients. Routing authentication and authorization routing may require a vpn for a tunnel if the clusters uh on public or if they're in different vpcs authorization, the last one is, uh is quite straightforward. uh He uses object, resources in the target cluster and that's very important to dimitri and his colleagues and partners.

B

They want cluster admins to be in control of who can do what in their clusters and authentication can be done using different methods.

B

The the nice thing about having one-to-one relationships or connect cluster connections is that you can uh build any kind of topology cluster topology with those as long.

A

B

It's a directed graph, it's valid, and so the first one that comes to mind is probably a management cluster. Talking too many workload clusters or you can do cloud.

A

B

Where a cluster is its own target, in the case of the research platform presented by dmitry, we have a decentralized federation where you know any there's. No, uh no.

B

Leader, I said I would talk about cross-cluster authentication. This could be a talk, um a full talk um just by itself, so I'll go very quickly.

B

uh The the the simplest way to uh to achieve it is to uh take the service account token from target cluster and export it and save it in the source cluster.

B

You could do the same thing with the certificates api. Instead of a token, you would have a certificate. The problem with those two methods is that you're using the target clusters as an identity provider and not the source cluster, so you need to distribute and rotate the secrets yourself with other methods.

B

You can use an identity provider available in the source cluster to get your tokens or certificates and present those to the target cluster and that target cluster has to be able to recognize them, and so, if you're in the cloud you can use um cloud like a kubernetes service account in the in a source, cluster can impersonate uh like an aws. uh I am role or google cloud service account azure search principle. However, you name workload, identities or machine identities and and then use that to connect to the target cluster.

B

If you're in control of the master nodes- and you can change the api server flags- that's great because then you can use web hook. Token authentication authenticating proxy.

B

If uh you need a kind of a a solution that uh a one size fits all is uh you can use an impersonating proxy that uses kubernetes impersonation to uh authenticate, and uh you can check out the kubo ibc proxy um project as a prime example and admiralty cloud uses the same print, the same concept.

B

Admiralty has two schedulers: the proxy scheduler and the candidate scheduler proxy scheduler handles proxy paws on the source cluster side and the candidate scheduler handles the candidate pods, created by the proxy scheduler and they're both built upon the scheduler framework.

B

The scheduled framework is is great: it's a set of go language interfaces that allows you to build your own scheduler, while retaining all the features of the standard, kubernetes, scheduler and adding yours at various extension points in the scheduling cycle and the binding cycle, and so uh the dots here mark where uh our like multi two schedulers, extend the the kubernetes scheduler.

B

So you can, for example, add some filters like how to filter nodes or you can wait before actually binding to a node, and this is useful in admiralty's multi-cluster scaling algorithm, because the two clusters talk to each other um using annotations on uh pod chaperones. Let's, uh let's see uh how this works.

B

So in this uh sequence, diagram I have the source cluster on one side and very started clusters on the other side that have the same components with different timelines.

B

When a source spot is created and annotated to use admiralty as a multi-cluster scheduler, it is mutated. We need to make a few changes to the pod like a scalar name, change the scheduling constraints so that the pod can tolerate the virtual node. We save the original scheduling constraints. For later.

B

We add a finalizer for cross cluster gabbage collection, different things check the documentation for details, while the proxy part is being scheduled and the virtual nodes that it tolerates are filtered pod. Chaperones are created in the target clusters that correspond to those ritual nodes. A part chaperone has two purposes: it is the vehicle of annotations, um both schedulers annotate, the chaperone, to communicate if they annotated one of the other pod.

B

Instead, that would invalidate the cache of the scheduler. So it's best to use uh another component. Another object for that, but also maybe mostly most important from a feature perspective.

B

If the source to target cluster connection is interrupted and the a pod has been running for a while a candidate part has been sorry, a delegate pod has been running for a while, but is evicted because a node that runs on a real node that it runs on is um being coordinated, for example, or stop responding.

B

We need a way to recreate the pod, and uh so so we need a local controller and that's the pod chaperone part. Chaperone is just a pot template that creates the pawn all right enough about the chaperone.

B

So each part chaperone creates a pot that looks exactly like the pod chaperone, and uh it includes the scheduling constraints that were saved in the in annotations on the proxy pod, so that we can ensure that the intention of the user to schedule on a node that had gpu, for example, is met at the real node level in the target clusters.

B

And so, for example, here only two of the target clusters out of three can find a node for the candidates, their respective candidates, so they annotate the uh the podca prom. The proxy scheduler sees that selects one of the two using some topology spread constraint, all other other proxy pod level, scaling constraints that you can add and when one is when the highest scoring node is selected. The point shot chaperone in the corresponding target cluster is annotated again this time to signal the candidate scheduler that it is allowed to bind the candidate pod.

B

That becomes a delegate pod.

B

The proxy scheduler sees that binds to finally binds the proxy pod and the other candidate pods are deleted via the pot, chaperones and garbage collection.

B

In summary, it is possible to build a global kubernetes cluster. You just need super fast networks and some custom built notes, but even then sritri colleagues and partners found reasons to use multiple clusters.

B

The federation around oculus, has over 10 000 cores currently and will soon be expanded by an order of magnitude with the addition of the expanse of supercomputer federation uses admiralty, which itself uses virtual cubelet and the scheduler framework, if you're interested, if you're interested in joining the research platform around nautilus contact them and if you want to build your own federation check out admiralty.

B