Cloud Native Computing Foundation GitOpsCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: A Multi-Cluster, Multi-Cloud Infrastructure with GitOps at CERN - Ricardo Rocha, CERN

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

A Multi-Cluster, Multi-Cloud Infrastructure with GitOps at CERN - Ricardo Rocha, CERN

CERN relies on a large distributed infrastructure to store and analyze the multiple petabytes of data coming out of its experiments every year. In the last few years two important trends have been changing the way we manage our infrastructure: kubernetes and cloud native technologies on the deployment side, and public clouds as a potentially cost effective way to get additional capacity to cover for workloads spikes and accessing resources that are scarce on premises, such as GPUs and other accelerators. This has made efficient management of multiple clusters even more relevant. This talk will describe how we've been evolving our deployments to rely on GitOps and tools like Flux and ArgoCD to centralize and standardize our highly distributed infrastructure. It will cover the three main layers of our deployments: the "underlay" infrastructure, and how with a bit of yaml and tools like crossplane we manage our on-premises clusters and those across multiple regions of the different public cloud providers; the "base services", including monitoring and central aggregation with Prometheus, OPA policies, cost calculation, etc; and finally our end user services. A quick demo will show how clusters can be added and removed with a couple lines in a matter of minutes.

A

All right, so thank you very much for the invitation and and uh yeah. It's really good to to to be at a conference again with all the challenges of uh rediscovering how to to the presentations live again. um So today I'll be talking about uh how we've been using githubs at cern um and especially the needs we have for multi-cluster multi-cloud deployments, some some quite large deployments we have to handle so where this all comes from and then hopefully I'll leave some time for q a so very quickly. I'm a computer engineer at cern.

A

I work on containerization kubernetes, also some machine learning and accelerators. I also do some work in the cncf um in the toc and uh I call it the res sncf research, user group, so cern has a quite large private cloud. The kubernetes deployments are also quite significant. We have more than 600 clusters, uh like thousands of nodes, uh tens of uh more than ten thousand cores today running on kubernetes.

A

So it's pretty significant, and this is our world on premises, and this is where I will start to talk today is to describe uh what are the needs. We have to to support uh this. This kind of infrastructure and what we try to do to to ensure like hiv, uh proper upgrades and how we use githubs to to help with this so starting on our on-premises world.

A

uh The reason we decided to go with multiple clusters and not have very large clusters, uh there's quite a quite a few reasons for that. The first one is uh to isolate workloads to have some sort of multi-tenancy by just splitting clusters.

A

The state today probably would allow us to do it differently, but uh there's there's still quite quite some advantages to doing it like this. Also, we had some issues with the control plane of the cluster, not necessarily working always very well so kind of splitting clusters reduces the plus radius as well when we do upgrades. This is another area that we struggled quite a bit so just doing.

A

Cluster replacement instead of actual upgrades of the clusters is, is another way to to handle this and I'll talk a bit more about that, and initially we also had this. This needs to split clusters to get access to heterogeneous resources. This is not necessarily true today, but it's still something in some cases. We need so when you start handling multi-cluster and deployments over multiple clusters. Being able to do it in an automated and fast way uh is key and gate. Ops is uh is the answer we we used.

A

So when we started looking at how we did this. uh If we look at the fast like before we move to containerized deployments, we were using traditional, vms or physical machines and using things like chef or puppet to deploy, mostly puppet at cern and in reality, if you were a service manager, very most of your work would be done already in git by just updating the here values for puppets uh the configuration in puppet, and then this would propagate to the node.

A

So the notion of like doing githubs was kind of uh there already in in this way of interfacing with the deployments.

A

This is also very nice to introduce people to containerization and to kubernetes a lot of them will will go through a quite steep learning curve to to start using kubernetes and if they can start just by editing, yaml and doing configuration changes in multiple deployments.

A

This way they can more confidently build knowledge of the kubernetes stack as well, and then, when, when we start doing this, uh then we also gain all the benefits of using kubernetes and containerization, which is to avoid configuration drift by having immutable deployments, uh the much better reconcilia reconciliation of the deployments that kubernetes allows by having this declaratively declarative state that then propagates to to the different applications and services.

A

So how do we do this internally? We have this notion of clusters as cattle very much like vms, as cattle existed before, so we try to have multiple clusters serving the same, the same application of service. So on the left side, we see like a load balancer on top that will split the the load from one service across different clusters with 50 50 uh distribution.

A

uh If we want to upgrade this cluster say we have two clusters running one kubernetes 121. We want to upgrade the clustered 122.. We actually usually don't do input in place upgrades apart from a few exceptions. We just add a new cluster. We configure in our github in our github deployments that the the service should direct like 20 to this new cluster gain some confidence with the confidence with the new deployment, and eventually this will allow us to slowly get rid of the old clusters and move the the workloads.

A

We can also do things like uh cluster auto scaling to size the clusters, according to the load they have so one one key thing we do here is that- and this is something that you will also configure in our git git configurations is that you will need to to have a way to expose a service type load balancer into a single central load, balancer instance, and this we do by having a slight change on the normal cloud provider, where we say this service should actually link to this load.

A

Balancer pool with this uid- and this allows us to to do this- spread of the of the load so from this state that where we had three clusters, one with 122 with 20 percent, then eventually we move to what we have here on the left, where we got rid of one of the old clusters, and then we split the the load, half half between the 121 and 122 clusters, and with time we deploy a new 122 and we get rid of the old ones, and this is kind of how we do the upgrades of clusters.

A

One thing that is pretty obvious here is that if you don't have a very good automation of your deployments, this is extremely hard to maintain. So this is where we put our investment.

A

uh The other part that would like to discuss uh regarding our on-premise deployments is even for for components that need to be kept up to date, even if you're doing like version clusters with the kubernetes versions. There are a lot of components. You will have to do security patches and do small upgrades with time, even within the same cluster.

A

So we keep this. We have this idea of release channels where we try to have um each of the clusters linked to a branch uh of our githubs configuration uh where we evolve them independently.

A

So, for example, say you have a 122 cluster, you will link it to a 22 branch that has all the configurations we want for that specific kubernetes versions for the base components in the cluster, and you can also choose if you want like a stable or kind of more development branch that or qa that will allow us to to like roll out uh gradually the changes first in qa, then in stable.

A

So the recommendation here would be that you have your version clusters, but also you have maybe a qa and a stable cluster so that you can validate in your deployments all the changes as we roll them out gradually. So this is. This is um like how we try to to structure our deployments in a way that gives us confidence in for upgrades of the different components.

A

So if you, if one one one key thing, is that uh when you deploy a cluster at cern, you actually deploy also what we call an umbrella home chart and this uh home chart has a bunch of uh dependencies that have basically all the the the base components we need in a cluster. So you can see here base which is like a again a kind of a multi-component chart.

A

But then eos is our storage system, uh the gpu deployments uh fluency to to forward the logs to to our central lock collection, some internal components here in storage and there's a bunch more and what you do when you deploy your cluster, you basically can you all you have to do is specify this uh values file.

A

That has like the base definitions in in our in the in in the chart itself with the the versions we recommend, but actually users can override these values if they want, so they might disable enable some of the features or, for example, for fluency. We have this notion of a producer that will tag the logs with the with the application that is forwarding them, so that then we can do correlations between the different services.

A

So all these pieces allow us to to have like quite simple way of managing all the dependencies and quite simple way of managing all the multiple deployments we have which like, as I mentioned, it's like more than 600 clusters today,.

A

So when we do that, like getting people into git ups, a lot of our users will probably just start with writing some some yaml files with their deployments and all the kubernetes resources they need, but actually, when you, when you want to onboard them uh with github, so it's quite essential to to do a very uh big push for training and dissemination.

A

So this is what we've been doing, the last maybe two or three years um people need to know like. If I want. If I have multiple clusters, how do I handle the load in my services, the distribution or, if you're running some sort of batch uh workload? How do you redirect batch uh jobs to the multiple clusters? How do you handle like the queues in the different clusters? Things like this, so we wrote a couple of uh simple uh getting started uh tutorials, uh I I think they are open, so you can have a look.

A

They are getting outdated because we, these tutorials, are still for flux and argo cd v1. We actually started by writing one with flux, and this was the main tool we were using. Some users uh started using argo cd as well, so the one of them actually forked our tutorial and wrote the equivalent in argo cd. These tutorials will tell you everything from like recommending the structure and how to handle environments uh using either multiple directories or branches to to managing secrets uh with something like subs.

A

So in both cases we actually rely on subs for secrets.

A

So this is something we push internally quite a bit, and this is the key to to being able to have this uh multi-cluster deployments that we we want to keep so now after we did this, we started jumping to to the public cloud, and this brings quite a lot of challenges.

A

I will mention a few, but it, but there is one good thing is that, because we already rely on the kubernetes api and all the kubernetes ecosystem for most of these deployments, most public cloud providers offer this managed kubernetes services, where a lot of it can be reused, and this is this was the key to to like ease our transition to the public cloud.

A

There are quite a few motivations for an organization like cern to to use the public cloud, even if we have a very large data center and access to quite a lot of resources internally.

A

But one thing, if you imagine like cern, will have a constant load in our in our servers, but we also have like a physics conferences that come, for example, during the summer where people will have needs to to run a lot more analysis than usual, and this comes with periodical load spikes. If we would provision our resources on premises to to this kind of load, we would have um quite a lot of waste during the rest of the year.

A

So there's a motivation to do on demand uh to use on demand resources for this to cover for this and then with the with a lot of machine learning happening everywhere, but also at cern. We have a needs to for a large amount of accelerators, and these are like gpus and dedicated accelerators like tpus and ipu's for gpus. We have some in-house, but we need a lot more. So we try to go to the public cloud.

A

For for for this extra resources for tpus, which are this tensor processing units from from google, for example, we we can't have them on premises, they still have a value, so we we use them on the public cloud.

A

Also, we use public cloud resources to get access to new uh type of resources and evaluated, evaluate them before doing big tenders on premises and also disaster. Recovery is one of the use cases we have, so it's quite a different use from what we had on premises, but in reality, because we already had this multi-cluster.

A

Deployments we kind of knew how to to manage this in a similar way. So just just a very brief uh thing uh regarding the the needs for for public cloud. Just to give you an idea of why we need this. A couple years ago we did a keynote at kubecon, where we did an analysis in in five minutes of where we analyzed 70 terabytes of data using a kubernetes stack, and this works very well.

A

When we proved that we can scale, we can scale our our deployments for this kind of um load we need, uh but actually and yeah. We built the higgs plots the that gave the nobel prize in in 2013 live on stage. This was a really nice way to say: okay, we can use the public cloud for real things, but then, regarding the deployment, this was kind of a kind of a stunt.

A

We, we did a one once once once in a time deployment which is not really useful for anything else, so we didn't even get halfway to what we need to to to use the public cloud in um in a generic way uh things that we will. We knew we would need is supporting multi-cloud multiple cloud providers. We cannot have one single provider support multiple regions. If you use the public cloud, you know that the resources available in different regions can vary, so you it.

A

You need to be flexible in this way, then the integration with our in-house tools, I mentioned like storage systems, but we have a lot more that we we need to kind of propagate to the public cloud for our applications to be able to run.

A

We need to have some sort of centralized monitoring and log collection tracing of the applications so deploy at multiple public clouds, but also collect it centrally and then accounting costing and yeah. You can keep adding things that you need when you start using the public cloud. But again we we had these components well, structured. We had an api that is well available in all the cloud providers, so there was. There was quite a lot of work already there.

A

I will just mention uh in the same way that I had this this. I had a slide about doing things at scale just to strengths and the the need for the public cloud for us, but on the other hand, we also learned that the resources are not infinite, even in the public cloud, if you use cpus at tens of thousands you're, probably okay, but if you start going larger than that, you might need to go multi-region if you're using gpus. This can happen even in thousand gpus or a couple of thousand.

A

You probably want to start splitting the regions and there's one thing that is like the cloud: can you can have a huge benefit if you use things like preemptable spot instances that are really cost effective and uh the availability of this is also quite limited, so you need to be kind of smart and exploring multi-region also to to reduce the cost here.

A

So the the main message I I wanted to put here is that these are just these are really reasons to to really stay flexible, on your choice of uh not only clouds but even regions within the same cloud, and in the same way that I explained that to have multi-cluster run premises. We need to have automation in githubs if you go multi-cloud multi-region. This is even more important.

A

So I'll finish with this. uh With this example, uh I was talking to cornelia about machine learning just before so we have some cool machine learning use cases and to to have an idea like if we we have use cases, one gpu, you might take like uh an hour to do a single epoch and then, just by using the public cloud and exploring resources, we we can speed up uh a.

B

A

Times that uh so at the same cost, so so there is really a push to to explore these resources.

A

So I will try to spend the rest of the presentation describing how we handle the public cloud. It's quite similar in terms of the stack we had on premises. What we add here is that we start managing the clusters themselves using githubs.

A

This is this is wasn't the case before people would deploy their clusters and then register by using flux or argo cd, uh where the application should be coming from and and add this to the cluster, but in the in the public cloud, because we have, we need this flexibility of multiple clouds, multiple regions we started looking at, can we actually do the same for what we call the underlay, which is the clusters themselves? So we would have basically three three parts that we would need.

A

We would need to support the clusters, multiple clouds, multiple regions and manage them in an automated way, have the base services I mentioned and then manage the applications themselves. So we end up with three groups of components. The first one is again the underlay, the clusters themselves, the infrastructure, the base services that we need on all the clusters for applications to be able to run, and then the services and applications themselves which are cluster specific and we will allocate them to different clusters as needed.

A

So we do this by having a single git repository, where we have all the configuration for the clusters for all the definitions of the base services and the association of the services and applications with each of the clusters.

A

So all the complexity of a very large deployment with- I don't know you might be having a deployment with uh aws uh azure and the google cloud in multiple regions and basically you you are looking at a single repository to manage all of this.

A

So this is a view with the rcd, where we have again this infrastructure, the underlay, sorry and then the infrastructure and then the services and and the applications running. And we use this notion of like app of apps that I think in v2 is called an application set. uh But it's the same concept where we aggregate things into groups.

A

So this is the view of uh what the deployment would look like at cern. So we have one cluster at cern that basically is responsible for managing all the public cloud deployments, and this will count uh the definition of the clusters and then the definition of the base services.

A

So again you might have a cluster in gcp, say: west europe, west 4 region, and then each each in this cluster will have multiple node groups and we will have node groups that auto scale, for example, for different types of gpus and then maybe a tpu and then we'll have a similar one where we have, for example, here we see the needs to have multiple, multiple regions, for example in west 4, you won't have nvidia t4s. We need to go to west one to get to t4.

A

So basically we need already two regions here, just because of the access to different types of resources, and then we need aws, because this is where we got arm graviton true, and then we need azure for gpus or ipu's, for example. So this can be quite complex to manage, but in reality, at cern. What we have is a one big yml file, one very big, yaml file that basically tells you for all the clusters. I need. uh I call it like this.

A

This is gcp cloud and then I tell it the the region, I want to use the cluster version and then the different note pools that I need to to have so in this case, like I have a notebook for nvidia, 100 gpus, and you would see this continuing and then the the system.

A

uh The the key thing here, if we're talking about clusters, is that the cluster has to become a kubernetes resource which is not like the kubernetes cluster being a kubernetes resource is kind of like the to to to start with, but but actually this is a really good idea. The same way we do the declaration and reconciliation of any kind of resource in in kubernetes. We can do for for for cloud resources as well, so for that we rely on a project called crossplane where they have this notion of a provider for the different clouds.

A

uh So in this case, you see like a template where we define a gke cluster with uh with the name and then for a bunch of node pools um like a template for a node pool that will define, uh like the max mean, node count which kind of accelerator it should use if any uh the type of image and disk that it should use- and this this is basically expanded. Based on this yaml file that I showed you here so again, one single bit repo allows us to to do all of this.

A

So if you look at our argo deployment in this case, it will be really large, but with this app of apps and navigation, you can you kind of get a nice uh feedback loop, all the way up in the status of the resources, and you can see the reconciliation happening in two cases here. So in this case, we we added an eks cluster that is being reconciled, and then you can see for, for. I had this west 4 example cluster with all the node pools, and you can see that they are already reconciled.

A

So the up the layer above this is the infrastructure layer. Again. This is extremely similar to what we do on premises, but it does have some some particularities because say, for example, at cern we have this registry which is based on harbor. Then we have prometheus now if we start having this very distributed infrastructure, we started looking at how to to handle this, so we actually deploy thanos and in each of the clusters on the cloud in each of the regions um that we we have a cluster deployment.

A

You will have a replicated replication of all these services, so the registry will also be deployed there and in addition to that, we also define like the replication rules of which images should be replicated uh in advance and like proxy caches. If we need any any kind of policies, for example, things that we want to express is uh who can run in this cluster in this cloud and we use uh based on labels, for example, that have to come with the applications, some storage systems that we need and the monitoring as well.

A

So this is uh something that is quite easy to to to deploy, and in most cases these are like home charts or, I don't know, customize uh based deployments in other cases, like the replication rules, we have, we actually have small tools that will do api calls to the to the systems, and this is a view of uh like what a dashboard would look so in. If you would have the on-premises gpu monitoring, you would see uh like a single cluster, a couple of clusters on premises here. You can actually see also like per cloud.

A

You can do an aggregation of the queries in reality for the service managers. This is all hidden. They don't really have to bother with all the details of the deployments. All they care is about these dashboards and one key thing that is also important. Is um our users? Don't really see all this complexity?

A

They care about the services they use daily. So what we do is we integrate? For example, we integrated git lab. I here I give an example of a gitlab ci. We have a bunch of runners to run, for example, jobs on gpus.

A

We had already the integration with kubernetes on premises. We actually integrated the runners also with our external cloud premises. So this would be one of the applications running across all these clusters. Gitlab runners and our users will just submit via gitlab their ci jobs, and they will run anywhere. That is appropriate, but they won't realize this.

A

So it's kind of a nice way to to expose all this uh to the end users- and we do this with like machine learning, uh machine learning uh frameworks as well, where the the jobs like a distributed, training job might run on premises on the public cloud, depending on whatever is happening without much trouble.

A

So I come to the end. uh uh So basically, we we are like githubs is the key for us to be able to manage hundreds of clusters on premises like thousands of deployments, tens of clusters already on multiple clouds and regions.

A

So we use githubs for base services, but we started using them for for the clusters themselves and you can add a cluster to our infrastructure on any public cloud and have it serving production workloads in less than 15 minutes, which is quite an achievement with all this stack that we rely on the flexibility also allow. This flexibility also allows us to have much more cost-effective deployments, and then the kubernetes api is really the common language and all the the ecosystem is the enabler for all of this.

A

So if you are maintaining uh any of these projects and or contributing to them like we.

B

A

To say thank you yeah, you you're helping us in a huge way and with that I come to the questions. If there are any hopefully there's time.

C

um We have several questions if any, if any of our attendees can access these slides later on.

A

Yeah sure I can upload to the to the gender yeah.

A

C

Other questions.

B

Given that you're, connecting from your on-premises system to multiple different cloud providers, how do you handle the networking between your your own data centers and the public cloud.

A

Right, so um it depends on the service for most of the use cases like I I mentioned gitlab ci runners. They will just pick a job and run the job in the public cloud. It really depends on what the job needs if the job only needs, for example, access to our storage systems. We.

B

A

Replication in place of whatever storage system is needed. One of the base services is this: this replication to to kind of extend the storage to the to the to that region. uh In some cases, if the service really needs uh tight interconnectivity, you need some sort of like vpn setup between the the the different data centers.

A

We we don't have a lot of use cases onboarded into our public cloud deployments that need that we we really use them for kind of batch like totally independent workloads that fit much better without this need for interconnectivity.

A

Cool. Thank you.