Red Hat OpenShift OpenShift Commons Gathering 2021 Kubecon EU, 5 May 2021

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: ODCN’s journey to connecting OpenShift clusters securely and transparently with Submariner

Description

OpenShift Commons Gathering at Kubecon/EU 2021
May 4th, 2021

Title: ODCN’s journey to connecting OpenShift clusters securely and transparently with Submariner

Speakers:
Martijn Straatman (ODC-Noord)
Stephen Kitt (Red Hat)

Abstract:
This talk will present a proof of concept built by ODC-Noord, from the perspective of the customer and of the engineers working at Red Hat.
We will discuss the proof of concept as well as the collaboration itself. Overheidsdatacenter Noord (ODCN) provides modern, sustainable and reliable IT infrastructure for the Dutch government, running Kubernetes containers since 2018 on Tectonic, then OpenShift, on an OpenStack-based cloud. ODCN hosts a growing number of workloads, on multiple clusters, and the next step is to improve reliability by connecting clusters in different datacenters. Submariner provides secure connectivity between Kubernetes clusters, with transparent and configurable access to pods and services in other clusters. ODCN and Red Hat worked together to establish whether Submariner could meet ODCN’s requirements. The proof of concept got ODCN up to speed quickly on this new technology, and helped Red Hat drive the project’s roadmap in a customer-centric manner.

https://commons.openshift.org/gatherings/OpenShift_Commons_Gathering_at_Kubecon_EU.html

A

Hello. Welcome to this talk about oblivion, dcn's journey to connecting openshift clusters securely and transparently with submariner. I'm stephen kitt, I'm a software engineer at red hat and with me is mark. Hang.

B

Yeah, I am I'm uh working as a cloud engineer for the dutch government yeah.

A

And so we'll start with an overview of what submariner is so that you can understand exactly what we're talking about submariner.

C

A

That is designed.

C

A

C

Kubernetes clusters at the networking level, it does so by.

A

Exposing new customer resources that are stored in the standard, kubernetes data stores and I'll go into that in more detail in a little while, it's.

C

An open source.

A

Vendor neutral project that was started by rancher and is now maintained by red hat, with help from contributors from a number of different companies, and one of our big goals is to make it easy to set up and use, and part of this is we make it available for.

C

Deployment using.

A

An operator or helm charts or our own tool, which is called subcastle, so some common use cases for submariner.

C

Are application availability, so this is this is where.

A

You have an application, that's available inside one cluster and you would like to use it in another kubernetes cluster, but you can easily deploy it in the other cluster and you don't necessarily want to make it available using uh publicly.

C

A

Endpoints some arena can help you with that. Another use case is disaster.

C

A

Where you want to.

C

Basically ensure that your data is replicated in a number of different geographical areas, for example, so that you can recover.

A

If you lose one of your clusters and.

C

Another big use case is data residency.

A

Guidelines, so you might have workloads that are available in a specific or that can be processed.

C

In a specific area.

A

Or you want to be able to use, for example, cheaper, cpu time and one availability zone, but your data has to remain.

C

A

C

Geographical area, for legal reasons,.

A

C

A

You with that too, so what submariner.

C

A

Is helped you with the networking layer and so to understand that we need to.

C

Go over briefly, what kubernetes does.

A

C

Kubernetes has a very.

A

C

A

View of the world by default, the cluster boundary is a hard boundary, um and so you can see here a fairly typical view of the network stack in kubernetes. You.

C

Have a number of pods inside your cluster.

A

Each pod is connected using quality networking, so each bot has an ip address and can communicate with other.

C

Pods in the same cluster.

A

On a private network, each pod also benefits from service discovery and load. Balancing workflows can be.

C

Made available as services and.

A

Pods can address those by name and have load balancing among multiple instances of the same service. And finally, you also have network policy which allows you to define rules which say which pods have access to which parts of the network and subreiner.

C

Extends this across.

A

Multiple clusters, so it doesn't touch the pods model itself. Pods are still private to each separate clutter, but it extends the.

C

Networking data plane across all clusters so that.

A

Every single pods can have access to.

C

Any other pods.

A

In any other connected cluster transparently, it.

C

Also extends service discovery and load balancing.

A

So services can be made available from one cluster to another and load balanced within each cluster. So.

C

Typically, for example, a cluster would.

A

C

A

Remote cluster, which makes a service available.

C

Unless the service is also available in the local.

A

Cluster, in which case it would provide the local processor and.

C

Also extends network.

A

Policy uh because.

C

A

Networking layer preserves.

C

Source ip addresses.

A

You can have rules which take into account the problems of traffic and it.

C

Does all this as.

A

You can see with the padlock in a secure fashion, using iv set tunnels by default and also supports wireguard. All.

C

The traffic is encrypted.

A

Between the clusters, what are the benefits then? So, as we've.

C

A

uh Direct e-twist port to the pods and port to service routing.

C

Across clusters.

A

C

Any kubernetes.

A

Cluster and it.

C

Can work with a number of different.

A

Cmis, so we test with obn, for example, which is the default noun open shift.

B

A

um There's some work, that's been done with calico.

C

And a number of other.

A

uh Cni's and we'll give links later to the documentation. You can look at it describes all these services can be deployed across clusters, so.

C

A

C

Basic level just part-to-service ip connectivity, but.

A

Also, as I've mentioned, service discovery and network policy, and it provides all this in an encrypted uh fashion, as I mentioned my.

C

A

Here's the diagram.

C

Of the high level architecture of submariner.

A

So the top section is what we call the broker, which is uh one.

C

Of the clusters that can be a dedicated class for just one of the drawing.

A

Classes which is used to store and share all the data, that's required across all the clusters and each cluster.

C

That you want to integrate as part of.

A

C

Of larger set of clusters using submariner.

A

Is connected to this broker and.

C

Uses that to obtain information about all the other.

A

Clusters and inside each cluster each node gets our.

C

Root agents: this is.

A

What uh takes care of traffic that has.

C

To go to another cluster.

A

And make sure it gets the right place, and at least one node in each cluster is.

C

Designated as a gateway.

A

And so that hosts the gateway engine which takes care of the tunnel to all.

C

A

C

And so traffic gets routed.

A

From each node to the gateway and then across to the target cluster and back again next slide.

A

Please never connectivity.

C

With submariner um so there's no impact.

A

On infra cluster traffic, this is very.

C

A

All the traffic that stays within the cluster.

C

Uses the standard network.

A

Setup, whichever that is inside your classes, so there's no impact from using subreiner on the local network performance um and only traffic that has to go to our remote cluster goes through one of the gateway nodes and as.

C

I mentioned source ip is preserved, which.

A

Means that wow traffic can.

C

A

The other way, of course, and you can also specify network policies that take.

C

The source ip into accounts.

A

And by default we encrypt cross cluster traffic and.

C

Another big feature in sabrina is that we.

A

Can handle clusters that have overlapping overlapping siders.

C

So if you set up multiple clusters without.

A

Necessarily envisioning that you were going to want to connect them at some point, and so they have conflicting ip addresses can add its own, addressing overlay uh to provide unique advantages across all the clusters, and this is called globalnet next slide. Please.

A

Service discovery, so this is the layer on top of auto service ip, which provides service discovery across plasters.

C

And this is actually an implementation.

A

Of a spec, that's not defined.

C

Inside submariner itself, this is part of a larger effort in the kubernetes multi-cluster service special interest group.

A

C

This is called the multi-cluster service api.

A

C

That defines a standard.

A

Or shared rather set of terminology and some shared features, the first uh term is cluster.

C

Set- and this is a group of.

A

Clusters uh that are all connected to each other and that share services, and this.

C

Introduces an important concept in the.

A

Multi-Cluster service phase, which is that all namespaces with the same.

C

Name are considered to be the same across.

A

The clusters, this is called namespace, sameness and then the multi-cluster.

C

Service api defines.

A

Two crds that are used to.

C

Share services, the first.

A

One is the service export crd, so this is an illustrator controlled crd, which is used to specify services that should be exposed across all clusters. Services aren't exposed by defaults and to export one. You.

C

Create a service export crd.

A

And this makes the service available across all the cluster sets in a new uh sub domain um under dot svc.clusterset.local. So.

C

It follows a similar pattern to services inside a single kubernetes cluster.

A

Across the cluster sets and then the other crd is a service import, and so this is the in-class representation of a multi-cluster service, but this is really just an implementation detail in practice that slide. Please.

A

So how all this is done? uh Sabrina service discovery uses a component that we call lighthouse.

A

So this builds upon the existing submariner architecture, with the broker and so on, and it adds a number of components, and so the.

C

General idea is that.

A

When uh apollo service requests, uh cluster set wide service, so it uses the svc.clusterset.local name that gets resolved developed by core dns or well. The main.

C

Dns provider inside the cluster.

A

C

It isn't aware.

A

Of all of this, instead.

C

The dns resolver is configured to forward.

A

These requests to lighthouse's own dns server and so that dns server server is aware of service imports. And so it knows about services that are available in other clusters and that have been exported and it will use the information from the remote clusters and from the broker to resolve those services to an ip.

C

A

C

Brainer can then.

A

Take care of, and so traffic can get to the other author, so that was a brief overview of what submariner does and some of, however, goes about it.

C

But the purpose of this talk.

A

C

Over our journey.

A

With obt nord and so.

C

Martin will present.

A

What odyssey nord is uh in just a couple of minutes, but uh I wanted before I conclude my section of this talk to give uh some of my uh impressions from working with martin and his team. So the.

C

A

The uh poc, with proof of concept with.

C

Those and submariner was to.

A

C

Our perspective.

A

Early feedback on features and usability. So when we're in this of engineering groups, we don't necessarily have much direct feedback from end users, uh and so this was an opportunity to get that and.

C

A

Of a big initial drive effect on ease and speed of installation and the.

C

Importance of being able to upgrade.

A

Because uh when we were working with uh martin and his team, uh they were very willing to test new things and uh iterate.

C

A

But that meant that we had to make it easy for them to do so, um and frictionless.

C

Because, obviously we didn't want to.

A

C

Them into a situation where they'd upgraded.

A

A bad release and they ended up stuck there or we'd end up spending a lot of time, helping them to fix things.

C

And one of the big.

A

um Gains from our perspective is that we were.

C

A

Get a good idea of customer well features that were actually useful for customers and so uh provided.

C

Us with a number of.

A

Feature requests um that.

C

We hadn't necessarily thought about.

A

uh And that were provably useful for end users, and so the.

C

First, one of these.

A

Which is insecure connections that was actually somewhat surprising for us. I mentioned that that cross-cluster traffic is secured by default.

C

And this was a big.

A

Feature uh in some arena from our perspective.

C

But it is not desirable in all cases.

A

um So, in some cases end users.

C

A

Private connections between kubernetes.

C

A

So there's no point in adding extra layers of security on top of them. It just reduces the network performance, so this was a big request from wiki north.

C

It's taken a while to.

A

Implement but it's actually landing in the next release of submariner another one with this is high bandwidth, so uh udc.

C

North has a high performance network.

A

uh Physical network and the desire was to be able to use as much as possible of this uh in some brainer, so this led also. Our investigations are on the high bandwidth and balance issues led us to implement a benchmarking tool uh because we knew how to run benchmarks and the obviously north engineers knew how to do that as well, but repeating that all the time was somewhat painful, and so we integrated all that into a benchmarking tool.

A

At their request, and in addition to that so network policies, they were an important feature for others: own customers.

C

A

A

Takeaway that I had from the.

C

Working on the proof concept was that knowledge is power.

A

We benefited greatly from having different people.

C

On the team, with extensive expertise in all the layers of.

A

This fact that's end up being used at oc, in particular the openstack networking layer and so.

C

A

Just openshift or generic network experts in the team we might have had trouble helping with you know some of some of the problems that they ran onto because they have a fairly complicated uh openstack setup.

C

That's enough for me and I'll hang over to my time.

A

Now, uh he'll give you lots more information about voice. You know.

B

Okay, well, uh thank you steven. um Let me introduce myself first, um I'm a mathematician, I'm a cloud engineer for the.

C

Dutch government.

B

um And uh osg north is uh north stands for north. We are located mainly in the north.

C

Of the netherlands is one of the foreign dutch government data center agencies.

B

C

B

Past we had many data centers inside the dutch government.

C

B

I think, eight years ago, to consolidate it to four data center agencies at.

C

B

Moment we have uh two.

C

Data centers running in the.

B

North, the two red dots you see in the picture, an atlanta.

C

B

Of the netherlands and the third one is available this year, um so a little bit uh perspective and the history from our um as.

C

A resident customers.

B

We are former coreos users, so in 2017 we started our container journey with kubernetes.

C

B

We saw inside the dutch government, many government agencies are moving towards.

C

B

And wanted to use container platforms like kubernetes, so we thought, hey, that's an opportunity to to create and make it just to create some kind of managed service for uh our government agencies and um over here. Until now, we are pretty successful in it. So um um since uh reddit took over uh coral as we moved to openshift, so um I think since january 2019 we're in production with openshift b11.

B

There are still some 311 clusters left, but the rest is all openshift 46 now and well yeah. Our main, though I think we have about three more than 3 000 cores running on multiple clusters, but we have main um we are ready for. We want to move to the next step and the next step is make these clusters more resilient for failures.

B

So our customers can make their uh workloads more robust for our that's yeah, there's people so.

C

um We were wondering and looking, how can we make.

B

These openshift.

C

B

More reliable, more high available, he did a lot of reading and proof concepts and um well we reached out to red and netherlands, and just please help us: how can we do this? They came up with the mariner project. We did see it when it was at in the hansel rancher and um we left it first.

C

But then red had to go over.

B

And we we gone.

C

Have an opportunity like.

B

C

To work closely together.

B

So um yeah so we're.

C

Almost ready for the next step so.

B

C

That exactly is.

B

um What what do we expect to if we just say let's say, install shift process on all these free data? Centers, that's what we want to achieve this year. um We want to be able to connect clusters, so we can sync data or we or our customers can send data. We want to have of positive part connectivity.

B

um As I will explain in my next sheet. We we are already able to. We have a highly high available uh network with bandwidth between the data centers. It was already present, so we were able to communicate uh between these clusters in these several uh different data centers, but we were not able to do portable connectivity, so we got a fairly static setup, which was not really helpful.

B

So that's why we want to have powerpoint connectivity. We wanted.

C

To be secure, we want to leverage the available bandwidth.

B

Not a complex configure, it had to be cloud flash network agnostic. So, like steven mentioned, we are running on openstack, but maybe we are using in the future, something else. So it also has to work on that. So it must be open source because we believe in open source. Everything is open source. What we do in uh in our in our company um and um the last two bullets: um low balance traffic across multiple clusters and do dynamic health checks.

B

So well we thought- maybe that's a little bit too much to to do in one year of let's say one and a half years. Let's focus first on the east-west scenario, and now that's what we can do exactly with sumero, so um we uh have been working with them closely together since uh june 2020. So I think it was version 1.04.4 and yeah. Like steven explained, we've been testing new features, new releases. We could test upgrades. We.

C

Have regular feedback sessions which were very helpful.

B

C

Learned a lot about our.

B

Performance, so just how much performance can we get out of the out of the network and out of the clusters, but it also led to several openstack optimizations, which were really really helpful. We didn't figure out ourselves yet so our our just our.

C

Entire cloud just.

B

Benefited from these other openstack optimizations also non-openshift container broadcaster. This was a really cool cool cool benefit and uh yeah. We just we made our cloud available to the submarine, so america team is good, just use it to uh to investigate issues or to make things better and even so important. It was very cool. It's very cool and professional team to work with you. So if you ever get this opportunity grab it because it's really, it can be really really helpful for your own use case.

B

Okay, something about our infrastructure. So, um yes, we're ready customers so uh we're using openstrike openstack at the moment. We're at train we're running more than 6000 vms, um installing openshift on top of openstack. So we all also using the nice openstack apis for single block storage. We also have an encrypted variant using.

C

Octavia for service type load.

B

Balancing manila for rewrite many storage and a roadmap issue, a robot thing, um we're probably going to look at the iron and bare metal integrations for next year. Chef has.

C

B

Storage layer, more than approximately 20 terabytes and we have s3, compatible object, storage, implemented.

C

On multiple regions.

B

The red dots and the data centers in the north at the moment are connected with dark fibers. We say delegated.

C

Fiber channels.

B

Which has a lot of bandwidth so and this this network has been implemented as a layer-free network, so this was a really good fit for zuma.

B

C

High level network architecture.

B

C

Moment so we have two dc's running.

B

Active active and the third one is in the making and probably operational by the end of this year, but what we want to do is to connect, let's say three clusters together, so we can make a very reliable infrastructure yeah. So I think one and a half years ago we had a power outage on dc1 and everything was gone. It's not good for uh for our business, so um um we want to make things more robust, so um these ceilings I've been uh uh drawing here, one is for the ceiling between cluster.

B

A and b is the connection steven mentioned.

C

We control it ourselves. We already have.

B

Encrypted that we've, our network network devices, so we didn't have the use case there to be to have an extra encryption layer on top of it. So that's the requirement we wanted to be able to turn it.

C

B

But for the other connection to a to c, and maybe two from b to c, we do not control the network path, so we can enable the extra ipsec and maybe just the in the future we are going to connect.

B

We can move as.

C

B

Public workloads to the public cloud, there is maybe also a use case for a ceramic, so I want to conclude my.

C

Talk with a small demo.

B

It's uh rgb of two open shift clusters on gc1 and dc2. I will show you in a minute and I've installed a copper hdb on both clusters and you know former cluster and I will try to create some kind of database.

B

Well, it's going to be created on the other end, so I will show you. First, we have a multiple clusters.

C

B

So you see here our first cluster is here here: gn2, that's dc1,.

B

C

Our second cluster.

B

You see it here is gm3 dc2 and inside these clusters we have installed a submariner, so I will show you the gateway which is installed on a spot running on a certain node. We do.

C

B

Delegated nodes for it yet, but we are planning to so. If I just use the.

B

B

Looking for a gateway, so you see how many instances do we have on the gateway? You have to look up the right name, space.

B

We do have you see two gateway instances running, and one of these is the leader. So I think is the this one. So this one to show you you can nice feature. You can look up in this gateway. What the connection is between the the other between the gate, the gateways on both clusters. So, oh sorry,.

B

You see here we connected. This is dc2, you see here we are connected to gdc 3 and you know all I have kinds of features where you can see the latency, the average loss max minimum where.

C

The status is it's connected.

B

Actually, this is the active one so position. This is not the the passive one, all kinds.

C

B

Backhand delivers one so all kinds.

C

Of information, which is useful.

B

To check the gateway is alive, what's also is nice is the um I will switch tabs now is the able to uh to look at.

C

B

So um submariner is exposing from each metrics to if you want to just look the you can plot these latency this connection latency seconds in a nice graph.

B

You could use the uh of course the previous workload monitoring, which is inside of openshift, for monitoring.

C

B

Alerting on it, so it's a really nice feature, they've added, um so yeah really cool. um So now back to uh the demo of just some proof. This is really working. So I've been installing two.

B

So this one is running on dc2 switch the um and this one is easy yeah. I will look it up.

B

This one is on dc3, so you see your pods are running the container, creating thing. um So if I hop over to the uh copper gb uh uh ui, you see it's connected. It has nine nodes, three parts on each side, so you can see these nodes are connected on gn2, dc1 and gm3.

B

C

Also see this on the other side,.

B

This shoots gm3 g2 gm3, so this proves these this all cockroachdb cluster is able to do uh uh to communicate with each other. You have all kinds of nice latency maps also useful to see how fast is the communication between these between these nodes.

C

We haven't been looking and.

B

Tweaking it in detail, but we do that later um so.

C

B

This is really cool, I think the show I think is working um just to prove the part on the potting is to create a database, see what kind.

C

Of databases are in this.

B

Instance, we have a demo database, I should have been deleting it. I will delete it now and recreated it. So I will go inside a uh progress. Tv.

B

Keyboard sleeping here it is oh I'm connecting to the cockroach.

B

B

B

So I've been connected now to the uh the database instance. Now, let's say I will create a database create database.

B

uh Demo boring demo too.

B

It's telling me it's been created, so if I go back to the uh cockroachdb ui, I refresh the instance.

B

I see I have a new database created and now needs to be. Also, on the other end, let's see yeah, you can see it's also being created on gn3. So this proves it's a really simple demo, but yeah it proves it is uh it's syncing. The data between the uh the both corporates will be instances through samara.

B

So oh yeah. This concludes my demo and I will go back to the presentation.

B

So if you like to read, look up on some error just visit these links, thank you for your.

C

C