Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Building HA Multi-Cloud Clusters Using WireGuard as a Network Overlay - Lucas Servén Marín, Red Hat

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Building HA Multi-Cloud Clusters Using WireGuard as a Network Overlay - Lucas Servén Marín, Red Hat

Kubernetes abstracts away infrastructure, enabling the community to leverage a single, powerful interface anywhere we like, from bare-metal to public clouds like AWS or GCP. However, not all infrastructure is alike: some clouds do not provide GPUs, while others do not offer data-centers in certain regions. As a result, users often have to choose between running their entire workload on a single cloud provider or region, and managing multiple clusters in different clouds. In this presentation, Lucas Servén explains how to solve this issue at the network level and create a single cluster that spans cloud providers and regions, facilitating high availability and avoiding lock-in, by using WireGuard as a network overlay.

https://sched.co/MPbF

A

Hello, oh it's on good morning, everyone I'm, Rita and I work on an open source service, mesh called linker, T and I'm here to introduce our next speaker. Lucas servin marine is an engineer at Red Hat and he works on one of my favorite open-source projects. Prometheus and there's many cool facts about him, including that he has a master's in robotics, an interest in deep learning and a twin brother, and he is here to talk to us about how we can build high availability, multi-cloud clusters using wire guard. Let's welcome Lucas.

B

Thank you very much, Risa all right, so my twin brother doesn't come until tomorrow to Barcelona, so I promise you I'm Lucas.

B

Yeah so today we're gonna be talking about building multi-cloud clusters with wire guard. These are clusters that can span different cloud providers like I can have some nodes in DCP, so I'm an AWS, or it's also clusters that can span across multiple regions. We're also going to be seeing how we can use standard wire guard tooling to add peers to the wire guard cluster that may be at the edge like IOT devices and finally, we're gonna see how we can actually create multi cluster services using wire guard, essentially for free.

B

Just using all these standard wire guard things Rijo introduced me so I'm really from Spain. You can't tell that from my avatar I'm working in Berlin with a prometheus operator team, but for the last six or seven months, I've been spending most of my time working and obsessing and spending all my free time on building these multi cloud clusters with wire guard so I'm happy to see a lot of people here are also interested and share.

B

My obsession so I wanted to a really quick poll who here is using wire guard at all: oh cool, a lot of people and who is using wire guard with kubernetes already oh wow, that's like more people than I expected I think we can probably just end the talk now and you can talk to each other or something. But please don't do that. So, for the uninitiated wire guard is a very modern and secure VPN. It's particularly performant and no small mean because actually runs in the kernel.

B

The configuration for wire guard is also really simple, which makes it very appealing to use, unlike some other VPN projects. The code base for wire guard is particularly small. It's like orders of magnitude smaller than Open VPN, for example, which makes it auditable and also it's possible to do formal verification of the protocol. So this is what a sample wire guard configuration might look like if I'm configuring, a wire guard interface on my local, like my laptop I'll, have to configure a private key and a public key pair.

B

A bunch like I would do an SSH once I want to peer with other wire guard nodes. I'll exchange, this public key out of bounds right. The other important thing to note here is that every wire guard peer when you appear with each other. You have to specify a white list of allowed IPs, and these are the allowed IPs that when I want to send encrypt some wire guard pack, it to my peer mmm XYZ it'll only accept it will only accept packets that are from an allowed IP that I configured for myself.

B

So, for example, I want to send a wire guard packet to ten 402 I'll. Look it up in my wire guard configuration I'll, say: okay, this is supposed to be using encryption. Key XYZ, I'll encrypt the pocket with this key and I'll, send it to the last recorded endpoint. This is the actual public IP address of this host.

B

When that host receives the the wire got packet, it'll decrypt it using my public key and I'll say: okay is the source IP of this packet, one of the allowed IPs for Lucas, if not I'll, throw it away, and if it is one of the lot of pieces, then we'll receive it anyways. You can see that configuring. This is actually pretty straightforward and really flat compared to some of the really complex configuration for Open VPN and like creating all these certificate and stuff. So that's very much more in line with doing things like SSH.

B

If you were interested more and more guard, please look at the white paper online. It's super easy to read. It has a lot of really cool detail, so without further ado, I want to make a really quick disclaimer, like I, said, I've been working on building multi-cloud wire guard culture for the last few months. I'm doing this in a project called kilo, you can find it on github.

B

A lot of the examples that we're gonna be talking about today are gonna use this, because it's what I'm familiar with, but also all the things we're gonna be talking about- are things that I research and developed while working on this project. So it's the stuff that I'm I'm most familiar with without further ado.

B

Let's talk about how does this actually work so if I launch a normal kubernetes cluster on AWS, normally all node to node communication and all part of our communication will take place over some private somewhere right, so that might be 1000 one if I later want to add some node to this cluster. For some reason from another cloud provider, I can no longer use private networking right, I can't route to my DCP private network from AWS. It just doesn't work, so the obvious solution here is why don't we take some public IP addresses?

B

This is obviously routable over the Internet, but totally insecure right. I don't want to be sending my node to node communication mypod stuff directly over the open Internet. So this is just out of the question. The solution that we've always done is, let's just create a VPN.

B

What I'm gonna try to convince you of today is that using wire guard, we can efficiently secure part of our communication and no to no communication so that we can extend our private subnets and route packet and what we can do is create a fully routable pod and network and fully routable subnet first service subnet that spans across cloud providers.

B

We can skip this and come back later. We have time all right. So why do we want to do this at all? Let's say that I'm deploying some kubernetes cluster to us West if I'm, lucky and I'm, using a really cool distribution or kubernetes installer I might get multiple availability zones, but if I'm not or if maybe the whole cloud provider is down for some reason or maybe I'm, the I'm relying on some cloud service like s3 and s3 is just down like it was last year.

B

In that case, all of my clients here that are trying to access my service are gonna, be out of luck. um One thing we can do to address this is we can say: okay, how about in the same region I deploy clusters both in AWS and GCP, in this case, they're both serving clients in the same region and providing low latency responses. But this way, if AWS is down, I can fall over to all my services running on GCP, and the benefit of doing this is that we can deploy.

B

Only one pod can like only one deployment and distribute pods automatically two nodes running in both clusters. Obviously another traditional solution to this is using Federation, but today we're talking about how we can do this solve these problems a different way using. Why regard up topology for this cluster? My look, something like this: maybe I have one gateway in AWS and one gateway in DCP that are providing the cross cloud communication. So all crosscut communication from A to F to DCP goes over one link.

B

If that link, for some reason goes down, it automatically falls over to another link. But the thing that's interesting is that I can keep all all communication within one cloud fast and unencrypted and not using wire guard and just use. Direct routing and I can only have to encrypt cross cloud communication. Another good use cases.

B

Let's imagine that, because of reasons we're working at some company and we have our own private cloud and we are in the unfortunate situation of providing a service it's too popular and now we have to scale elastically and add a lot more nodes. How do we do this? I? Don't want to have to buy a bunch more nodes and install them for traffic that we might not be having tomorrow. It might just take us too much time to do that.

B

A good solution that we can do is say: ok, I'll, spin up a hundred nodes in AWS and join them with my cluster in my private space and now I can elastically scale my cluster without having to create two kubernetes clusters or installing you nodes, probably in my private cluster or anything like this.

B

The same thing happens for what, if my private cluster my cluster in my private cloud, doesn't have some resources like GPUs? What I can do now is I can say: I'll have all of my regular cluster stuff running in our private cloud and I can launch a bunch of GP.

B

You know than AWS to complement the features that I don't have in my private cloud or if we're running in some private clutter and some public cloud like digitalocean or anything else like heads, McCloud or something in Germany that doesn't have DP use either or any other accelerator we can mix and match that features we want by creating multi-cloud clusters.

B

We can come back to this later as well.

B

Another good valid use case for creating clusters using my regard is who's ever run a cluster in some cloud provider that didn't actually provide isolated. Private networking like who's ever run in digitalocean before 2018. Exactly so, if we're running in some top provider with shared layer, two between tenants note to note communication might actually not be secure, and this means that other tenants can read our can read our Ethernet frames and stuff.

B

So one thing that we can do to secure this is we can say, let's mesh, all of our nodes in our cloud using wire guard, and now we can encrypt all node to node communication and make sure that no one else can sync our traffic. We can do this later as well. In this case, we'd be creating a full mesh routine. All of our nodes.

B

This is one really trivial solution, or one really trivial topology for wire guard, but we can actually create much more exotic topologies if we're running a multi cloud cluster that has some nodes in a private cloud that has to be fully meshed and some nodes in GCP, for example, that don't need to be fully mesh because they actually have secure private networking. Then we can say all right: let's keep that stuff unencrypted and let's mesh everything here fully so we'd end up what the topology looks.

B

Something like this by the way, all of these graphs are being generated using the kilo command line tool, and we can talk about that later of anyone hasn't any interest.

B

Okay, I've talked to you a little bit about the motivation for why we want to create multi cloud clusters and now we're getting into some of the consequences of this decision. It turns out that we can actually do really interesting things with standard wire guard tooling, like we can add peers from IOT that are IOT devices or nodes that are at the edge to our wire guard cluster, to provide a either services that we don't have, or maybe just to give them access to our private services. So here's an example use case.

B

Let's say that I'm, a corporation and I'm, providing some enterprise service that shouldn't be available on the public internet and I, want my clients to be able to VPN into my cluster. If our cluster is already wire guard enabled we can trivially say: let's make this client a wire guard peer of our cluster and they can access all of our cluster services securely. The thing that's even more interesting, though, is that the pod subnet of our cluster is fully routable from our client. So is the service of net.

B

That means that let's go back to this. Actually that means that from the command line, I can see, say curl 10201 from my laptop and access any pot or whatever that's running with pot IP 10201.

B

The same thing happens in Reverse. Actually, so, if I have a kubernetes cluster that wants to access another service, that's behind a corporate firewall, it may be running on like some bare metal machines, like some legacy machines that I haven't moved to the cloud yet and I don't want to have publicly available. I can say alright, let's make all of these bare-metal services appears on my wire guard cluster and now any pod, that's running in my cluster can say all right. I want to make an HTTP request against 10101, which is the bare metal service.

B

This works perfectly well for us, except it's not really kubernetes native right, like in kubernetes. We want to be able to do service discovery. Maybe you want to be able to address services using VMs, but luckily there's a really cool trick that we can do for that.

B

We can create a set of endpoints to mirror the service that we have running outside of our kubernetes cluster, and then we can create an actual service as well with the same name, and the end result is that from any pot inside of our cluster, we can use standard kubernetes DNS to resolve the IP of the things running out of the out of the cluster and access those services.

B

And the same thing is true for IOT devices like if I'm running some things at the edge and I want to be able to collect. Maybe some data securely from IOT devices I can also connect them to my cluster using VPN cool.

B

This is the third and final act of this talk, which is multi cloud services, so somebody might say to me Lucas all this multi class stuff sounds really cool, but I don't want to put all my eggs proverbially in the same control plane. This sounds like a really bad idea. I want to have some resiliency in case one of my CDs gets hosed and, and I no longer have this cluster available. What can I do again?

B

Traditionally we solve this using Federation, but one of the consequences of using wire guard is that we can actually create these kind of like federated services, pretty much for free. The same way that we said we wanted to peer one WaterGuard cluster with a node like an IOT device or maybe a laptop, we can trivially say I want to peer one cluster with another cluster at the end of the day, they're. Both just wire, Guard, endpoints wire guard, doesn't know anything about kubernetes, think the Lord, so it keeps it really simple.

B

This means that if I'm running one kubernetes cluster, an AWS and I want to access a service there from a kubernetes cluster running in GCP, I can peer the two clusters and then I can use the same trick where I mirror the endpoints and the services in the two clusters and and then I can once again from my pod in Edo us I can access the I can access the service in sorry, I'd work from my pod in TCP I can act as a service in a do yes, just using regular community discovery or DNS, plus or DNS, or anything like this.

B

So without further ado, let's get to the interesting part, which is the demo so for this demo, I'm gonna try to propose like some motivation, for what we're gonna do. Let's imagine that all of us here there's a bunch of us we're working for a company like nest or something, and one of the things we're trying to do is we want to provide some security cameras to our customers.

B

One of the parts of this service is that we want to let customers be able to stream this webcam feed the security camera feed from a web service like anywhere that they may be, if they're not at home or something. So in order to do this, we want to be able to securely connect to the IOT device and be able to pull the stream, but it occurs to us. Our product team is like hey. You know what would be a really cool idea.

B

It'd be really cool that we could do object, detection on this service as well, if we could do automatic object, detection and labeling of everything that's happening in the stream. I think our customers would really like that.

B

Unfortunately, because of reasons again, the cloud that we're deploying our web service and doesn't have GPUs to accelerate this object detection. So what do we do? We can create another gun.

B

Multi-Cloud cluster, with some nodes in AWS that are GPU powered nodes and connect them to our existing cloud provider, because we don't have an actual IOT device here, I'm going to be using my laptop as if it were the security-camera and Jessie the webcam feed directly so for anybody following along or that wants to try this again later, there's a github repo with all the instructions. I'll be copying things from here. Actually, so, if the demo gods are generous with us today, we will have fun. Let us see.

B

B

This is using typhoon to spin up a bunch of nodes, I've already created the infrastructure, just to save us some time, I've created some nodes in digital ocean and some AWS, and now we're going to connect them using wire guard. So, let's take a look I'm going to install the manifest that I have in this repo. Let's try this.

B

Cool, let's take a really quick look at what we have in this repo. Alright, we have this thing called. Maybe you guys can't see this I might be better all right. We have this thing called dark API. This is just providing an API for a yellow object, detection, neural network. We have a coop Connie au pod. This is the actual web service. That's doing the streaming to our client. We have an M JPEG thing, we'll take a look at that in a second, we have a wire guard kernel module installer again.

B

This is running on typhoon, so it's all container Linux and we don't have any package manager or package manager to install kernel modules, but we can actually use a daemon set to do that. For us we have an Nvidia device plug-in. We have kilo to do the wire guard networking and we have an Nvidia kernel module installer. Let's take a quick look at the mjpeg thing. This looks familiar. This is just the mirrored service and endpoints that we are talking about earlier.

B

So we're going to be mirroring the service, that's being provided by the IOT device so that we can access it like a native service from our other daemon set here so here towards the bottom towards the bottom. Oh, my god, whatever towards the bottom. We have this thing that says stream mjpeg default. This is just using the cluster DNS to resolve the IP of our endpoint, that's providing the webcam stream. Okay, let's go back to the interesting part.

B

Now. What we're gonna do is we're going to actually create a wire guard interface on our local machine.

B

mmm We won't get part of it. Let's do this again, sometime, okay, great, let's see IPA. So we have a wire guard interface here perfect. Now we're gonna generate a public and private key pair for this interface, a.

B

Come on, oh sorry, wrong thing.

B

This is again we're gonna generate this puppy.

B

Cool, so we have here private key public key perfect. Now what we need to do is we need to. We need to actually peer our local wire on interface with our cluster. That's running wire guard Kela provides a CR D that you can use to define a wire guard peer. You can just list your allowed IPs and your public key, and this will automatically generate the actual any configure for a wire guard on the cluster nodes. So let's do that.

B

Okay, so now our cluster knows how to contact our local node, but our local node doesn't yet know how to contact the cluster. So this is what we have to do here. We'll use the kilo command-line tool to generate the configuration and then apply it to our local node.

B

Alright, the last step here of setup is that we need to actually add routes to the pod subnet and the service subnet, so that I know that when I want to send a request to one or 10201 I have to route it through my kilo interface, my wire got interface. So let's do this really quickly.

B

All right, we created a bunch of IP routes. Let's see this cool, so you can see now that I have ten too low. This slash, 24 via wire guard same thing here. The wire guard ip's and also the private IP addresses okay. Finally, what we're gonna do now is we're gonna actually run the webcam service. That's providing the stream I'm gonna run it in another terminal. Here we can send it alright cool. So this is just using the this just running a dock container.

B

That's providing the stream I'm gonna, move it back out of the way, so we have more room, and the last step is that we're actually gonna open up the IP address of this of this pod that we have running in our cluster, alright cool. So this is doing object. Detection on me again notice that what this is doing is we have a service running in digital ocean. That's reaching out to my laptop pulling images from my webcam, sending it to AWS for GPU object, detection and then sending it back to me.

B

Alright, this is all powered by wire guard and made possible this way. Okay. This is happening on our.

B

All right, that's really the end of our demo here and it's pretty much the end of our talk as well. We have some time for questions. Otherwise, if you want to talk to me later in private about anything else, I'm really happy to answer any questions either and yeah thanks a lot for coming.

B

It looks like we have some people here.

B

Yes, gentlemen, a.

C

Really great demo thanks a lot. Can you tell us a little bit more about security concept, service mesh integration, storage, replication I mean what's the use of pots without storage? Well,.

D

C

Other things that actually make up the complete working application: okay,.

B

So the question is: how can we, how can we complete the picture right and do provide things like storage and stuff, also across multiple clusters or well.

C

Your scenario was having a failover scenario between cloud providers, so obviously some user data needs to be replicated there right Nintendo totally. You didn't talk about that. I assume! You have an idea how to solve that. In my opinion, that's the bigger problem than actually connecting pots to each other yeah.

B

C

Also load, balancing like global of balancing is a big challenge. Actually yeah.

B

Absolutely specifically for for providing data. So if we have like some database, we have a service, that's running on multiple clouds. We don't want I mean there's. Actually a couple concerns right number one: if we're concerned about latency, we probably don't want to be making requests for database across multiple clouds or across different reasons either. So we're definitely gonna have to replicate data.

B

If this is an issue for you, if you're running these staple applications and you're running across multiple cloud providers, it's probably going to be very important for you to either have like, mmm like some replication for your database or some just mash to some other you're on gonna have like some other database nodes that are ready for reads, maybe some like masters that are ready to be swapped out.

B

If the other one fails or something like this is very much an open problem, but I mean database replication is it's kind of an entire field of study that I? Don't think we really have to talk about here like I'm, happy to talk about it with you later, but I think it's a general concern with running stateful applications and highly available applications in general right, like even if I'm, not running a multi cloud setup and I'm running just an H, a database across multiple ability zones and one of my nodes dies.

B

I have the same problem right. It's not really particular to this.

E

E

Sometimes you need, you may need to run appear in an environment that is behind a firewall or requires accessing the internet through a proxy can't tell me about how compatible this would be with proxies and firewalls.

B

Okay, well, one thing you should notice is that, for this example, we're running this laptop right is connecting to our cluster is being matted right through our like IP. So what like this doesn't have a public IP address right, so we're being matted and still I'm, being able to connect to my I'm still be able to connect for my cluster to my laptop.

B

Obviously, wire guard is not like a layer, seven protocol right. This is like happening at UDP level, we're encapsulating all of our encrypted IP packets inside of UDP, so you're gonna have to have some UDP port available and open. This is one important thing, but one of the things that's interesting is that bar guard has built in roaming enabled so like the IP addresses for all these things can change and they'll update, automatically and kind of transparently, and this does work behind, not another kind of like standard gateway. Things.

F

Hi so I noticed you put some static routes in your laptop. So how does routing work in a cloud, and also can you do this with managed communities like DKE and stuff I'm? Sorry,.

B

Can you repeat the question I didn't quite get the first part you.

F

Edit static routes on your laptop during the demo, so we'll kilo add routes to the notes inside the cow clusters or yeah.

B

So the nodes I add a node that was running in AWS and another one that was running on digitalocean. All of the routes to route packets across the two cloud providers were automatically generated right, like those were our math generated and applied. It also automatically applied a route to route packets to my local machine as well yeah.

B

This doesn't work today on gke because they have their own networking provider, but I mean there's, definitely ways. There's a lot of projects out there actually they're doing there's a lot of projects out there that are doing wire guard for multi cloud stuff and there's a lot of other projects that might actually be able to support gke directly. But you can definitely do this on GCP I've done the same thing. On Google Cloud, like I, said there was like Google closers with a double yes clusters, for example. So.

F

Is this like us, like a CNI noise, yeah.

B

Absolutely yeah, it's a networking provider. One thing that's interesting is that this project has some compatibility with other CNI packages. So one thing that we saw here, let me show you really quickly.

B

You see that our Damon said here is called kilo typhoon flannel. So this is running kilo on typhoon on top of flannel. You can also run it without flannel and just do all the networking itself. This means that kilo will do like the cross cloud. Communication and flannel takes care of like the local. If you're running flannel. If you don't care, then you can just use kilo entirely, and the same thing is true for, like any wire guard cloud, networking provider yeah, we have someone up front here.

B

I think that's have their hand for a second.

G

Hi, thanks for your talk and I would like to ask about. The purpose of the talk was how to be in HHA blasters. But how can you build a say, links between two clusters, for example? It will support adding to the objects of peers, how it will fall back from one another or how it will be used yeah.

G

B

In general, or by design wire guard doesn't allow you to have two routes to the same allowed IP yeah, but that doesn't mean that we can't have two routes of the same cluster. So one of the examples we saw earlier is that we can create a full mesh between all the nodes in our clusters right. So that way, if one of the nodes, that's providing some pod cider goes down that pot cider is down, but all the other pod ciders are still available.

B

The other thing is that the today, the way that, like this works on Kela, for example, is that if any of the nodes there's like a keepalive mechanism, if any of the nodes you can stop responding, it just is taken out of the mesh and another leader is elected to be the gateway for example. So we can do things like this to automatically as long as the kubernetes api is available, mmm the kilo mesh well like we configure itself. Ok,.

G

And second question will be: if you have, for example, let's say four clusters, but it's the topology will be like a central cluster connected to all three, but the remaining three doesn't connect between each other. Can you do multi hop with guarded or it will need to be updated, the central cluster continually being synced with all the roads or how it will be that possible. Do.

B

You want that clusters to be separate clusters or yeah.

G

For simple three clusters and one of them connect to all three but the remaining three. The synchronous between each other. So, for example, that there one of the single clusters that connect to a central how it could be our whole table to the they're plaster that this.

B

Is not possible today, I even implemented this today is definitely possible to do something, we're you to find your own routes and stuff like this. This is an interesting use case. I can't think of right now, like an example that I would do this for but yeah I'm sure that you could figure this out like it's at the end of the day, you're just applying routes, and we need to have some configuration mechanism to say this IPO via this route, I.

B

Think we have time for maybe one last question.

D

Thank you for the talk can I ask: do you think with something like wire guard? Go and network name spaces might be possible to have pods that only have a wire garden phase in them and therefore don't need any routing configuration or special privileges on the underlying node. Alright,.

B

So maybe using like using wire guard as actually like, have RC and I plugin only provision wire. It interfaces for all of our pods I'm.

D

Thinking you create these namespaces, create the interface outside of the pod yeah and move it in, so that the pod can only ever speak wire guard and, if possible, to avoid having to load the kernel. Module, use the user space implementation, yeah.

B

This doesn't get us around having to load a kernel module we're, not loading the kernel module into our our pod that was running this web streaming service. That's just installing it on the host by the way, so we never have to install it at any pause like this is just provided by the host.

B

um You can definitely do this, like you'd, have to just write a new CNI plugin that takes care of this or instead of like an IP, an IP, something or like a VX, LAN see and I plug-in just do one that does it for wire garden interfaces. It's definitely possible, it's just some work, but this would provide really secure encryption between every single pod. Thank you. All right, I think we're out of time, I'm really happy to talk to anybody else. If you have any questions and thinks about first thing or listening.