Red Hat OpenShift End User Talks at OpenShift Commons Gathering KubeCon EU, 4 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cost & Resource Management of Modern CI/CD infrastructure with OpenShift at SIX Digital Exchange

Description

OpenShift Commons Gathering 2021 at KubeconEU
May 4, 2021

Title:
Cost & Resource Management of Modern CI/CD infrastructure with OpenShift

Speakers:
Saulius Prasmantas (SIX Digital Exchange)
Radu Domnu (Red Hat)
Marcel Haerri (Red Hat)

Abstract:
Operating modern CI/CD infrastructure requires a balance between fast resource availability during peak development time and cost optimization during downtime. With scalability features such as the Machine API and cluster autoscaler, OpenShift 4 can provide on-demand elasticity for development teams by enabling self-provisioning build servers and deployment environments, while at the same time optimize cloud platform costs by automatically expiring deployed workloads and removing temporary resources. In this session experts from the SIX Digital Exchange (SDX) and Red Hat Services will share their expertise on how to optimize GitLab CI runners on Openshift. You will learn - how easily you can run elastic Gitlab CI runners on OpenShift - how cluster autoscaler can help to scale infrastructure up and down based on developer demand - how to cut infrastructure cost, by automatically down scaling expired development environments - how to address provisioning latency of VMs on cloud platforms

A

All right welcome to our talk on cost and resource management management, on ci cd infrastructure by using openshift. So a few words about ourselves, my name is salus prosmantas. I work as a sre at sdx.

B

My name is marcel harry. I work as a cloud architect in the services team at red hat. I mainly focus on customers within the alps region and there on the infrastructure and openshift forward topic.

C

Hello, my name is rodrigo. I work as a thought consultant in the services team, as marcel in the arts region, I'm focusing mostly on openshift and cloud native ecosystem.

A

Okay, thank you guys, so we all work at sdx. At the moment. This is a six group company providing financial market infrastructure services in switzerland.

A

There are over 200 employees and we build dlt-based solutions, um be part of blockchain network. Primarily we use r3 corda, but, as you can see, we also partner with hyperledger and enterprise ethereum alliance.

A

So we build a number of products where we provide trading infrastructure for digitalized assets, also centralized security, depository market surveillance market data, quite a bit of products which requires agile and the scalable infrastructure. So companies started 2018.

A

We provided proof of concepts and later on, we adjusted to the our vision where we striving to to build next generation: digital assets, infrastructure which bridges, traditional financial world and new digital, um including crypto asset world.

A

So this requires scalable cicd infrastructure and the first question: why does it have to be container driven? um First, we want to build it once and run it everywhere. It means developer, builds on on the laptop compiles. It um pushes the code.

A

um Cicd's system releases securely signs it, so it has to be scalable first of all, to accommodate week or weekends uh loads. There's a end of sprint: when there's a lot of things happening, it also has to be rapid and with uniform tools, meaning we want to use same tools for building the code or deploying the infrastructure and for fintech companies. Securities is is number one priority, so we have. We want to enhance it with the infrastructure and we have to have flexible configuration to segregate development from from release and or deployment workflows.

A

So improving security is extremely important for us. Also debugging um same as like developer debugs on the laptop. They need same cable capabilities to to have it. In cicd pipeline we chose gitlab, here's the the architecture of gitlab, which is on the left. You can see the gitlab server where developer, pushes the code and there's a api where gitlab runners connect.

A

So first of all it registers with the gitlab token, and then it constantly pulls uh if there are any pipeline and jobs available and based on the type of of workloads. It starts um a number of runners jobs. It scales through the nodes in in the pods, get lab runner types.

A

We can segregate github runners into multiple types based on on security requirements or based on type of sizes. So, for example, developer feature branches. They could be run on shared, auto scalable runners and release. They need more secure environments, which will lead to openshift features. Okay, secure builds. um This is one of the, as I said, important features to run these bills separately from from each other. So when we run a new job, it has to be separated. So there should be no previous artifacts or workloads.

A

Openshift provides a non-privileged user by default, so we segregate by user id also separate projects and namespaces.

A

Can can run separate jobs, it integrates quite nicely with the hashicorp vault, where we store all the secrets and it also provides dynamic secret capability. So when we log into gitlab or s3 every time news is generated for for releases and for extremely secure loads, we run it in special hosts with a trusted execution environment.

A

So why openshift this is comprehensive, offering which provides the enhanced security and enhanced scalability. It has some extra robustness and performance, it also compliant and very development friendly, and we also get great support from our friends at red hat, where radu and marcel could beef us in. How did we achieve scalability and reducing the cost of of ci city workloads marcel.

B

Yeah, thank you solis. So, let's talk about one of the features that are coming with, openshift is, and especially when we combine openshift with a public cloud, like usher. Maybe next slide, please.

B

um And when we are developing in such an agile environment, we actually um kind of like need to address, also very elastic demands, so people are kind of like busy, especially when it goes towards the release, they're busy, spinning up um new features or actually stock fixing things. And- and so we get a lot of things being deployed. And when we look at what sdx deploys um within a single deployment or for a member network, then it's actually multiple deployments.

B

Databases see the jobs, everything together. So it's quite some kind of resources and if you are able to just spin them up with a finger snip, then it kind of like you end up with tons of things being spawned. Then, since it's so easy, but still development happens mostly only during business hours, so maybe during eight to nine to ten, maybe a little bit more hours, but at least hopefully not at the weekend, which in the end means um more or less. Your cloud is going to be idle during around two-thirds of the time.

B

And when you look at how you consume cloud, you consume clouds based on on on the usage. So you pay for what you use, so if you don't want to use it, you just have resources idling around, and this can make a cloud environment quite expensive and especially if it's idling two for um of the time. So let's make that environment also more elastic um towards a cloud perspective. Next slide, please.

B

So when we look at how um british scheduler works, um we can we can use this behavior to actually then drive how things are getting scaled out and when a new part is getting scheduled to schedule. The looks um at the part that is being scheduled.

B

It checks for its valid validated that it checks whether the part is valid or not, and then it looks at all the available nodes and checks which one might fit, and then, if there are some found it prioritizes and the cubelet will start um the part on the note now next slide, please, when we look at an integration with with a cloud provider, we actually um get openshift tightly integrated with our provider api, which means that we are able to spawn machines just like parts.

B

So as as we spawn a machine, we we get it, it joins automatically automatically the cluster and it will serve workload.

B

Now, with with a pod, you actually also have replication controller, which just watches whether your parts are still alive and whether the desired amount of replicas is still running and kind of like parts of that and similar is the machine sets for machines and they are kind of like the template that define in which availability zone in which network, based on which image your machine should run next slide.

B

Please now, if, if we we look at the feature of the autoscaler, we have um something that watches how resources are getting scheduled and when there are no available nodes being found, a pod stays in a pending state with not enough resources available, and so the the cluster autoscaler then sees that and actually provisions a new node to the cluster.

B

That node will join the cluster and the part gets scheduled, and we can see that on that animation on the right hand, side where the part is pending, machine got scheduled, got deployed and the part is part of the cloud. Now there is the cluster auto scaler, which is the global configuration, and then we can actually schedule.

B

We can actually configure the cluster auto scale to configure per machine sets, and so you can have a different, auto scaling functionality for gitlab cicd runners, um and you can have another one for for your regular work. Notes next slide. Please now when we see the cluster autoscaler in action- and this is the view on one week- we actually can see that it automatically adds quite a bunch of nodes, then removes them again and so on and on the right hand, side. We have the flat line over the weekend where no development happens.

B

So this is kind of like we can see that we um kind of like scale out our used resources um kind of like one third of the cluster. It goes up and down next slide. Please.

B

Now this ballooning and up and down on on and releasing resources means we're able to cut quite some costs. In that case, it's actually some 10k 0 per month that we're able to cut things- and, as I mentioned, the cluster autoscaler also compacts cluster and compacting clusters in openshift always means that workload is getting evicted and evicting. Workload in opengive means that also getting killed and new replicas are spawned on another node.

B

Now on on a busy cluster when things are being developed, this actually can lead to failures in terms of let's assume you have an integration pipeline, running testing particular parts, and then they are suddenly evicted. So this might interrupt things and part of that is. You can actually annotate.

B

At the beginning of a ci run, you can annotate your parts that they are not safe to evict, and so the pastor scale will not try to evict them and the notes will stay, and then you can actually just remove that flag at the end and then the cluster, auto scanner is able to release the resources and now, since people usually don't clean up when they just spawn, um we also quite heavily invested into down scaling the away on the deployed, replicas and so on, so that the cluster automatically starts compacting now ballooning up and down, usually throwing things away is quite fast.

B

Provisioning new vms still takes some time all still also in the cloud environment and one of the things um where we've seen issues there are ci runs and radu is now going to tell you about how this looks um like nowadays and how we solved it there that we get faster provisioning times.

C

Okay, myself next slide, please so, as marcel mentioned, provisioning of vms takes in the cloud from from 5 after 20 minutes uh when running the cicd pipeline developer uh must wait uh already some time until the the image will be built uh and then to deploy to the cluster.

C

Imagine that the cluster now is full and it's no, it's no place to accommodate resources anymore, and um we, the the developer, will need to wait extra another five to twenty minutes for a new node provision in order for his workload to be deployed to the cluster in many use cases. This is not acceptable.

C

Next, like this.

C

So, in order to mitigate this issue, we came up with the idea of having empty nodes of cellular empty nodes that are always started, and basically they are ready to take over workload instantly. How did we do that? So we basically leveraged on the kubernetes concept of both priority.

C

uh For those who don't know, let me, let's make a short parenthesis, so each pod uh inc when it's getting deployed kubernetes. uh It gets a sign of scheduling priority if this is not specifically uh defined. uh The priority basically is inherited from the cluster default priority. Now.

C

The important part here is that if a cluster is full, so it's out of available resources and there are some pods with priority higher that needs to be needs to be deployed to the cluster, but in the same time there are no available resources anymore in the cluster, but there are pods with lower priority. Then the scheduler will evict this uh ports with lower priority in order to make room instantly for the possible fire drive. Let me close the parenthesis and let's explain how we design this dummy port.

C

So basically, this dummy pod, it's it's a code that runs any image which must be non-terminated non-terminating and has a lower priority than the cluster defined cluster default priority, and it also consumes all the resources in onenote.

C

So in order to have one-to-one mapping with a node. Another important bit here is that we don't specifically define uh custom priorities for the rest of the workload in the cluster, so all of them will inherit the cluster default. Priority acts like this. So let's see this in action.

C

Let's imagine we have an openshift cluster with three worker nodes. Two of them are uh real uh work or not, so they are um basically have pods with real application workload. We say we have application, one application to application field application four, and let's also assume that this for application consumes uh all the resources.

C

On node a and node b, we have an additional node, node c, which is this is so called so the node and cell the worker node, and which has scheduled this dummy code, which we were talking before now we have a gitlab application, a cicd pipeline that wants to deploy application 5.

C

next slide. Please, however, as the cluster is now full and doesn't have available resources anymore, um the scheduler will check if there are pods with lower priority in the cluster that can be evicted and, of course the scheduler will realize that actually, the dummy pod had a lower priority has a lower priority than um our application. Five and he start editing the spot next, like this yeah. So this pod is getting evicting. It's terminating next slide. Please and after this board is getting everything. Two actions will happen.

C

First, the application five will be instantly scheduled to node c, and then the cluster autoscaler will detect that the dummy point now doesn't have enough resources. The cluster is not able to offer the dynamic enough resources in order to get scheduled and uh the thoughts that auto scale will start spawning up a new node in order to schedule this time.

C

The important part here is that the waiting time for provisioning the node is now transferred from the real workload so from application 5 to the to the dummy port, which is acceptable next slide list.

C

Oh, how did we implement that? So? First, we must define uh a new priority class, this priority class. We will assign to the dummy board, as we said before, this priority class needs to have a value lower than the cluster default priority in order for this dummy pod to be evictable.

C

Additionally, this um priority class and the value of this priority should be higher than the cluster autoscaler threshold, because we want the cluster autoscaler to react when the dam input cannot be scheduled anymore in the cluster.

C

Considering the default value for the cluster or port priorities is zero and the default ratio for the cluster. Auto scanner is minus 10, which is a priority of -5 for our new priority class. Like this.

C

Next, we need to create a deployment for our uh dummy mode, so important bits here are the following. So first we need to define the replicas. The replicas is basically the number of of of these pods but, from the other hand, side, uh because one would mess one to one to one spot node. It is basically the number of the spotter we need for that specific node category.

C

Then the priority class name needs to be the priority we defined before the node. Selector needs to be the type of the node for which you want to create the spot nodes and the resources uh must consist of the node uh allocatable resources, so in order to fit the body to fit exactly in the capacity of one node, the image, as I said, can be anything but must be a non-terminating image next slide, please.

C

So, let's draw some conclusions. First, when we have a new workload that needs to be scheduled to the cluster, if the scheduler detects that there are no available resources, it can still make room for these resources by evicting these dummy points which have lower priority after the eviction the cluster. I would out the cluster sched autoscaler will detect the unscheduleable pods and will start the new node in order to schedule these stomach mods.

C

This will also help when we encounter a situation like like this. In the future, we will. We already have some spot nodes ready to be able to serve workload.

C

Now, when we consider that the cluster autoscaler is doing downscaling, so it's basically computing which nodes can be scaled down. We must also think that the dummy board will fill out the capacity. The full capacity of a node and the code that are getting redoused by the cluster autoscale will not have room to to be scheduled on on this spot node and will be scheduled on another on another worker note.

C

So the dummy pod will remain scheduled on the spot note until it will be eventually evicted by the scheduler. When we'll encounter a scenario like we presented before, and that's it about the spot notes, I will give the word to solus to do the wrong month.

A

Thank you radu, so just to finish few takeaways, so so autoscaling as you can see it works, but it requires proper configuration and some tuning to be done.

A

Gitlab turns out to be a very flexible framework which allow us to to configure building software for developers, and we found that container based hybrid infrastructure suits very well for these kind of work. So thanks a lot thanks for tuning in and see you in next.

C

C