Red Hat OpenShift Detroit 2022 OpenShift Commons Gathering, 25 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Building a Namespace as a Service on OpenShift at ING - Red Hat OpenShift Commons 2022 Detroit

Description

Building a Namespace as a Service on OpenShift at ING
Red Hat OpenShift Commons 2022 @ Kubecon/NA
Detroit, Michigan
October 25, 2022

Speakers:
Jan Willem Bijma (ING)
Arno Vonk (ING)
https://commons.openshift.org/gatherings/kubecon-22-oct-25/

A

A

Nobody, okay. Now, my next questions are irrelevant. Maybe we is learning opposite 4.10 on bare metal, on-prem.

A

And we did it with IPI installation all Encore OS in an Enterprise organization. Also, nobody, okay, I, will tell you about our story.

A

So what Surfers on top of ing container hosting platform- and it is completely separated we as platform owners, redevelop the platform, maintain a platform and make all the compliance and stuff there, because we are a bank so, and we also make sure that the internal audit department is decided off. The namespace is completely responsibility of the owner of the namespace. Normally the devops teams, our consumers and they are responsible for what is running in the name phrase.

A

So they are responsible for the runtime on the images that is running damn so can they our consumers, install or create a namespace with OC Epping. Now they can do that. They can't do that. So we have in ing our ig5 Cloud cell service portal and there they can request their namespace and what they get they get Resources with mini course and memory, but also they get a secure connection to the one pipeline where they can deploy this stuff.

A

They get also and registration in the CNB, and they get also an egress IP address to get connections to their back-end systems. Oh no, they get not one IP address egress, but I got two. They will always install for them two namespaces or two different data centers and we make it equal for them.

A

So how we can do that. We have a lot of own packages built around 36 to support this. One of this is the ihp API to orchestrate everything we have a project control to create the namespaces in the cluster. We have the cdos control to make the connections to the one pipeline, the CID CD pipeline.

A

We have the image comporter, so we can see uh within with an API what is running in the cluster which images are running and we have also a quota Auto scaler, so with resize automatically the the resources in the namespace services efficiently used. So there we go, make a picture. What we do we have two data sensors always and then we order a lot, a lot of bare metal surface from our internal provider.

A

We have a lot of apis for the CBB, from monitoring security monitoring logging available and on the other hand, we have the cicd ing1 pipeline.

A

On top of that, we have all our code and hit repos and we use the provisioning node to install completely automatically with the installer proficient installation, IPI method, complete everything on top of the bare Metals open shift, including the core OS on the verification.

A

After that, we have all our own packages, we install them also with an openshift github's method and everything will install on top of the open shift. So it's running there at that moment. From that moment, the consumer can request via the uh internal Cloud.

B

A

Of a naval space and then the namespace will create it with all these packages and after that five minutes, some minutes later he can directly install his application from the one pipeline directly in the namespace.

A

All is hands off, so we have a completely immutable platform and a computer immutable workload environments. So this is our environment and the environment will tell you more about how we built this uh for our consumers.

B

So we have a lot of dependencies that we need to install on a cluster. In the past we used ansible to install everything on all of the Clusters. Sadly, we saw that there were problems with configuration drift. We came across some issues on one cluster that didn't show up on the other, and when we moved from openshift 3.11 to openshift 4, we really wanted to have a solid installation where everything is equal and that we use correct versioning.

B

um So we came across open shift githubs and we started using it. Sadly, we had to implement it a little bit differently than what's supposed. How are you supposed to implement it? So what we do is we push and copy of the git repository into the cluster, and then we use Argo CD to pull in the configuration from the from the mirror inside the cluster. This solves some compliers, some issues. With the repository we have. We, we can't connect directly to it from our cluster.

B

uh How does it look so we have a CI CD pipeline that creates a output, repo image and that output, repo image is deployed inside the cluster and Argo CD application is created inside the cluster and that ensures that all of the packages that we need are then pulled in and configured and installed and in the end we get into a state where all the features we need are there.

B

We created a demo. So let's see if the demo works so first we have the first package, which is the we call it Azure deploy because we deployed with Azure uh at some point: it installs all the output, repo images and slowly all packages are being picked up and configured and installed, and synced and yeah.

A

It does everything I.

B

Think it always looks pretty cool. The total time span is about 22 minutes due to some timeouts that we came across and we saw seeing this movie. We saw some improvements that we can make in the future, but for now it it works pretty well and for us this is quite fast. So we're happy with the result, and the last bit always takes a bit.

A

B

Do come, I recorded a demo, so I know it'll.

A

B

Green at some point.

A

B

A

Saw it before I go further, it's over yeah.

B

That takes about one minute to demo, so the the beginning is always really fast, and now it's completely green, so yeah. um So one of the features that we install is the project controller. This is a controller that was made to ensure that all the namespaces that we deploy are deployed. In the same way, it uses a crd which defines the resources the namespace needs.

B

It hardens the namespace to ensure that only the people that are part of the development team that requested it can access it, and it also ensures that there are some default Network policies and yeah all those kind of things, and it's based on the on a framework that we created ourselves, which we call the Scarface framework. It's a it's a framework for operators and you can write it in Python and we build in that. It also does an automatic roll back.

B

So how does the Scarface framework looks like you create an app and you create a kubernetes object, a crd and um it it reads the custom resource and it ensures that whenever something changes or is created that the operations are taken in so later. In the second day, my love will show you one of the CRS for a project and then it will probably be more clear.

B

But what we Define in the CR is the we call it purpose code, which defines the Development Group the resources like CPU memory, all the limitations that needs to be set. If it's stateless, stateless or if it's a data service, so yeah, uh the Scoville structure has automatically created a event listener which has three uh possible. Three methods create, update and delete.

B

Each is called in order, and if one of these feel it will automatically roll back the previous one, they did so that we always ensure that the state is how it's supposed to be and if we don't have any incompleted configurations or namespaces on the cluster.

B

um But we have one problem. We saw one problem with the project controller because we used an outside orchestrator to configure uh the CRS in the cluster and that will then be picked up by the project controller and to ensure that uh we have the correct State across two clusters, because we always deploy in pair the ice. Hp API was created and that allows us to just call the ice. Hp API tell the ichp API, we need a namespace and that ensures that the same crd is the same.

B

Cr is created on both clusters and it keeps track of the creation and it also keeps track if you want to update the namespace.

B

um The ichpa API uses three stages for us because we have some dependencies inside organization. So, for example, we first need to have a network ID before we can create a cmdb entry and we need cmdb entry to connect it to the namespace, because otherwise we don't know who owns the namespace and yeah. It would be nice to know who to contact if they have an issue.

B

All these steps are sequential, but inside each stage, they're, all concurrent, which is really nice for us, because it saves some time with with the creation. Some cars can take a bit longer, but uh they are then already processed when the slowest is done um during the first stage.

B

We also ensure that we have a high probability that all steps that come after stage one are succeeding. So we check if the namespace already exists, we check. If the cmdb entry already exists, we tried to see if it might have already been. uh There might be something wrong with charging. So if an API is not available, but still, we cannot be 100 sure that everything will succeed. So in case the checks were successful and in the end the next race was unable to be created uh during stage three.

B

We rolled back everything everywhere so that we don't leave any orphaned or still um entries somewhere, so it either was successful and it is created or it failed, and we don't have any any left over somewhere. Another tool that one of my colleagues Robin sickman created was a quota outer scaler. What we saw in the past was that our.

B

Our consumers request more than they need and we had a lot of Hardware that was unused and still we were out of Hardware. We couldn't create new namespaces, so we created a tool to ensure that, um in a certain frame and automation can check what what the consumer needs and will scale down. If it doesn't need it, and it will increase if there's a burst of resources required, for example, during an update.

B

uh That's why we prevent that consumers are reserving uh resources that they don't need 99 of the time, this lowers the cost for them. It also lowers the cost for us, because we don't need that many many Hardware anymore as we used to have um so that's the second demo, for uh that I want to show you so I talked about an IPR. I talked about the project controller, so this is in the the crd that we sent to the project to the HP API.

B

It defines the quota, the CPU memory. It also tells us who requested it. The name of the namespace.

B

And now we created during the by calling the icehp API and with a request ID, we can see if it was actually successful.

B

So when we look it up, we see all the running all the stages and what the result is. We see the on the first few lines we see running stage check, which is the check to see if there's a probability that it will succeed, uh they all succeeded and, in the end the State Street was also successful. This is the ichp project, object that the project controller uses and there we see the actual configuration for a specific cluster and it has the same information as what we've sent to the ichp API.

B

And now we're gonna do a deployment and we have a limit of 4, CPU and one gigabytes of memory and we have a deployment that takes one-fourth of it. So one CPU and 250 MB of memory, as you can see here,.

B

And when we deploy it, it will of course be successful because we have enough limits available.

B

And it's running.

B

B

It will also succeed and it will also be successful, but we're now at the limit of our quota, which will be problematic if we would, for example, increase it to six which we're now going to do.

B

And then we will see in the events that it wasn't able to create, because we don't have enough resources requested enough quota available to us, and what now happens is that a quota, other scaler saw this event happening and it will do an API call to charging and an API call to our cluster to ensure that the extra resources are requested for that namespace and it's assigned to the namespace and then the Bots will be successfully running after a total of five seconds, and now we have six running Bots instead of four.

B

So that's the demo of the quota Auto scaler for us. It's often an issue with consumers really requesting a lot of unused resources and we made it uh yeah. We are now using it in our newest ichp release.

B

We had hoped to announce that we were open sourcing, the three tools that I just mentioned. uh Sadly, we are a bit delayed with open sourcing. It Our intention is really to open source the software that we show the the tools that we showed, but it will be a bit later.

B

Please follow github.com ing, dashbank Dash bank, because we, when we open source it will be available there and the names used here might slightly change, but yeah keep following it and uh it will be there at some point in time.

B

We take because we think just like some other presenters, that it's important to also give back to a community that has given us a lot um and now we're at the stated we're quite close to what's used upstream and we're not uh stuck behind on Oak shift 311 we're now also using option for so that's uh that's a good thing. I guess I want to thank you all for being here and listening and have a great day.

B