Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2017 (Berlin), 11 Apr 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: High Availability Kubernetes on Bare Metal [A] - Muhammad Kamran Azeem & Henrik Høegh, Praqma

Description

High Availability Kubernetes on Bare Metal [A] - Muhammad Kamran Azeem & Henrik Høegh, Praqma

To run mission critical applications on your own datacenter you need high availability on every part of the system. In a kubernetes cluster, this includes your controllers, etcd, and even the load balancers. This talk will describe about setting up high availability mechanisms for Kubernetes Controllers and load balancing nodes. It will also cover creating highly available etcd, worker nodes, and other components necessary in a functioning cluster based on a real world project.

About Muhammad Kamran Azeem
Muhammad Kamran Azeem is a senior consultant for infrastructure and security at Praqma’s Oslo office, where has has been working for past two years. He brings with him about 20 years of experience covering programming, database administration, networks, information security and training. He is author of the book “Linux Pocket Reference for System Administrators”, and various CBTs on adapting and using Linux operating system. Before Praqma he was managing some of the largest supercomputers in the world at Saudi Aramco in Saudi Arabia. Kamran is currently working on a book project entitled “High Availability Kubernetes: a Sysadmins Guide”.

About Henrik Høegh
Senior consultant with more than 15 years experience within operations, DevOps, and Continuous Delivery on Kubernetes, Docker, Linux, Atlassian, Jenkins. As a DevOps consultant Henrik regularly works with build servers, Container technologies, version control, and agile task management systems but also spend a great deal of time working with general organizational, architectural, and process related challenges for customers.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hello, everyone, you feel, can you guys hear me at the end at the dust? Okay, very good, so welcome everyone. My name is Cameron Ozzie Mohammad, Kamran, Ozzy and I'm from pragma Oslo I come with a background of about 20 years of operations and for five years I have been managing some of the largest supercomputers in the world.

A

Saudi Aramco before this job right now with pragma for last two years, and we are doing very much fun stuff with the continuous integration and continuous delivery using docker using kubernetes and lots of fun, stuff and I have my colleague here, Henrik who's from Denmark Eric yeah.

B

My name is Sami food I'm from Denmark and one of the boring employees and Dermott at the sameness. Now he at my workplace and I'm white really boring and I'm a male and.

B

Really boring and yeah. My first computer was a Commodore 64, so I've been working mostly as ops and I've been at practice for two years as well. I think we started at about the same time. I was the first employee in ovals and come one was the first employee in Muslim and when we started we were eight people I think we have 45 now with 19 different nationalities. Yes,.

C

A ball of yeah.

B

A

Thank You Hannah, all right, so are you guys excited about this talk today? Okay, very good! So here it goes so the title says it all: you can go home now.

A

So who recognizes this.

A

It's a scene from some movie.

A

Okay, so it this is a screen grab from the movie importance released in tool 2010.

A

The idea behind this is just some funny coincidence, so in Greek mythology, so first I cover I talked about the word kubernetes, it's a Greek word and it's funny how we pronounce it as kubernetes and the Greeks laugh at us, because it's key.

B

A

B

A

So since the kubernetes is Greek word, so we I thought I should bring up something great, so this is according to Greek mythology. These are the gods, and these are five and count and I consider these as the five kubernetes components, three of them run on master or controller node, namely API server, scheduler and the controller yes, and the to run on worker node.

A

The queue let and cube proxy, so I thought a connection, so I just thought it would be just going to show this and in a way we expect our nodes and pores not to die just like these guys we're not supposed to die, but the Titans beat the crap out of them. So anyway, let's move forward. This is what a community stressor looks like, as probably most of you already know, so we have an API server at the heart and we have a scheduler.

A

We have a controller manager and these three basically make up your control plane and we have in this diagram. You have two nodes just to emphasize on multiple nodes. We have the same cube proxy running on two nodes and the cube that running on both nodes and some quads, which are being managed so in in kubernetes. Everything looks up to the API server, that is, the heart of communities itself and API server talks to XE, D and I've just drawn one instance or one depiction of a CD.

A

Here it can be a cluster and normally it is.

A

So we want it to set up a kubernetes cluster for a client and they want it in on bare metal. So we came up with this setup and in this we you can see, we have a control plane, we tend to call them controllers. Many people tend to call the masters, so I will use this term most of the time we have a control plane. We have some worker nodes, we have HDD cluster, we have some shared storage through NFS and this all makes up a kubernetes network or our cluster.

A

We also have load balancer here running traffic, and this is the corporate Rooter and we have other components of this corporate behind on this interface, whatever there are other servers and their users and whatsoever- and this is the public connection to the Internet.

A

Excuse me- and this is a DNS server somewhere on the Internet, and this is just some other computer on the Internet. So this is the idea is that we are running nginx somewhere here and we access. We are able to access it, not only through corporate internal Network, as shown here we can also access it. All. The way from this computer outside, which is come somewhere in the world anywhere, so that is what the setup looks like, which we wanted to set up and we encountered some problems.

A

So when we were configuring, we configured controller nodes, we configured at CD, configure controller nodes and then we went to configure worker nodes, and then we realize what API server do we point our queue proxy to? It only takes one argument as API server target, so this was a first big question.

A

We couldn't risk, saying specify only one of these, because if one of these dies, what next do we go manually? Configure the worker to start talking to the other one, because it doesn't take a string or a list of API servers in this, especially in the queue proxy. Then we have a load balancer and we didn't know which API server to talk to.

A

We had the same problem here, whether to use this one or to just use this one. That's second problem. The third problem came: how do we make this load balancer highly available, because all the it traffic which is coming in is coming into this load balancer? What if this dies? This becomes single point of failure. So we have these big questions here in this set up, can you guys see the I apologize for standing in your view? Okay, so.

A

In the cloud it is quite easy because cloud and take talk about any cloud provider, GCE Amazon, they provide you with load balancers and you can put load balancer in front or behind of anything like so, and that normally those load balances are stable enough or the cloud provider takes care of them.

A

So we could address one problem like so, and the second problem was solved by the using the cloud provided load balances, but this is cloud and our setup is on bare metal. So cloud is easy. We could just set up a load, balancer provide its IP just you want the queue proxy instances to also to our incoming traffic and everything is solved. But this is cloud not bare metal, so we still had those problems. So then we thought what to do.

A

How do we provide high availability to this set up, which we have.

A

What if we provide a similar proxy ourselves, but then what? If this ties to proxies and then work, if the other one dies and so on and so forth? So the problem was still there: the problem of high availability. We could now look up to the controller nodes, but we still have problem of high availability. This instance and this load balance are still single point of failures, so we saw that some people are using the u.s.. They set up a multiple proxies, multiple load balancers.

A

They set up a common DNS name with a very low TTL, etc. I would just say these two words you can read these, but, and there are many other problems associated to it, so no E&S no-go this is our solution. This is not our solution. Yes,.

C

A

Occur to me, sorry, since we were on bare metal and since I had some experience of setting up such clusters in past this was a natural thing to do for me, so we introduced carousing and pacemaker to this problem.

A

So chorusing is what used to be Linux heartbeat project, or was it IBS or LVS I, don't remember exactly now there was heartbeat and then there was Linux as we guess, which is probably now pacemaker and pacemaker talks to carousing and manages resources over its cluster.

A

Now you're going to hear to about you're going to hear about two different clusters: one is one cluster is going to be the kubernetes cluster we are talking about, and the second is this high availability cluster. Let's first see what purpose we can chorusing does. This is a very simple conceptual diagram here. It shows that we have some hardware, some services running on some machines and then pacemaker and chorusing running on top of that managing some resources and monitoring them with some control loop.

A

So to speak, when it detects hardware failure, it moves the resources to other nodes, depending on what you define in the policy so without further ado, let's configure controllers. So what we did was we said in this cube: Nettie's cluster, the big one we introduced a smaller cluster, we made controller nodes, a small 8a cluster using chorusing and pacemaker.

A

So these two nodes have their own individual infrastructure level, IP addresses, for example, dot, 21 22, and we introduced a virtual IP resource which pacemaker controls and that could be assigned to any one node at any given time.

A

So all the nodes, anything as trying to contact this IP address lands on either of these two, depending on where it is at any given time. So that falls not only what 2.2 problem, but also the high availability, the single point of failure problem. Now this is a small cluster. So if one node dies and if that node happens to be the active node can having the IP address, the virtual IP address the moment it dies. The IP address floats to the other node and the other node becomes active and nobody ever notices. What happened.

A

Dub operations continue fluently. Then we address this problem of load balancer in the same way here, so we introduced two load balances and we set up a small cluster for them. They have their own individual IP addresses. Yes, they get a virtual IP which I forgot to write here. They have dot, they have the IP address dot 40 these load balancers. You will see it in the probably the next slide.

A

So now the load balancers can look up to this and the load balancers have their own virtual IP, which the incoming traffic sees my apologies for not writing the IP address. It is 10 to 40 0.44 0, it's not written here. So this is how we cluster eyes or created sub-clusters inside kubernetes cluster, and it has no interference with kubernetes operations. You have to remember this and we went ahead and did the same for xcd. So actually has these knows.

A

Individual IP addresses dot, 11 or 12 or 13, but we gave it a virtual IP dot 10. So now, when master nodes or controller nodes need to talk to a CD instance in their code in the cube, API server configuration, we just specify one at CD address, and that is the virtual IP and it talks to the HDD cluster.

A

So before I go into explaining chorusing and pacemaker, let me go back one step, so we assign a virtual IP to either of these two nodes right. You understand that part. Okay,.

A

Controller node form an internal cluster. The kubernetes software forms an internal cluster itself. There is a leader which is elected by itself and then the other nodes become followers. So if you are thinking that what if the IP address is assigned to a follower, let's assume controller 2 is follower. Controller 1 is leader, but the virtual IP address right now resides on controller 2. It's not a big problem. It takes the request.

A

It just gives it to the leader and the leader does what it wants to do and then and business continues, and if, for example, the virtual IP was already in controller 1 and that was active, it just receives all the requests and handles them. So the the point is not a factor of the the thing that controller nodes have their own internal net. Internal cluster saves us from a lot of configuration on pacemaker level. We don't have to manage and bind our virtual IP address with the ladies services.

A

We don't have to find that find out. Where is the leader and then go and set up the virtual IP address on that node? We don't need to do that, so we we are saved in that aspect from lot of configuration so how it works. Yes,.

A

How much latency is there between the master nodes.

A

They are, on the same premise, same side same right, so so there is no latency.

A

Yes, let's just go ahead and ask questions this one.

A

A

The way we saw kubernetes working, we saw one node, acting as a leader and the others acting as followers, and we saw the same in sed.

D

A

So so that's why we came up with this design that even if a master node is a is a follower, then it can talk to still talk to the leader and then the leader can do whatever it wants to do.

A

So these controllers are not so small in the first place, theoretical theoretically, I have not looked into that. So.

D

A

Have to quite a lot of.

D

Or, if is ever will boost the first per second, so we could release a little bit faster and we find that we need to balance the traffic to several API.

A

Server: okay, we.

D

Mean more like the same than usual, but we set up two proxies in front there and they are be a following I'd be in demand me there. Okay,.

A

So you you are saying that you need a load balancer behind these two and we can make that load balancer highly available. Ok, we can do that. Yes, of course, thank you for for sharing this idea, but we did not do it. We can do it in our future products.

C

So you know the master IP from two different machine, and so the rapper chooses, which want to go to a based on IP forecast, and so you have not actually not only your fate of a machine but also a local entity.

A

Okay, we can. We can discuss this after the target, sir. This is very important. This is very important and we're really good talk, so we will discuss this. Okay, very nice, so how it works, Sakura think actually, when chorusing runs, it forms some sort of ring, as shown here, a communication ring.

A

It calls itself a totem ring, but let's call it a ring and it keeps listening to the heartbeats of all the nodes which are involved and also make sure that the quorum is there if it is configured and other stuff, and then we run pacemaker on top of which talks to carousing and extract certain statistics and takes actions based on what course Inc reports. So pacemaker manages the resource itself. The virtual IP here, which is moving between the nodes, that is, that is space makers, job, that's not chorusing.

A

Job core things job is to make sure that it's listening to all the cluster nodes, and here the cluster is this AJ cluster.

A

So when the node fails, chorusing detects that it tells pacemaker, pacemaker acts on that and moves the resource to the other node. So you see like this, it moves to the other node. That is the basic principle of co.

A

How chorusing and pacemaker work together and these resources can be anything here we are using IP address, they can be a service, they could be anything and I wanted to highlight that when you use a high availability, when you set up a high availability list cluster, it is very important to not have a single network, because that single network can become your single point of failure, because if the communication fails, the switch fails or something the nodes, both nodes will think that they are the master and they are the active nodes and they will try to get hold of that resource.

A

This is called split brain through its brain syndrome and it is prevented by using certain fencing mechanisms, but primarily to assist fencing. You need to have multiple communication channels between the AJ cluster nodes. So here is just an example. I have shown you, we have to load balances and we have three small separate switches connected to these, and all of these are different different networks. So this is 10 to 4000 / 24. This is 192 168, 1, 0, 1, . 0/10, for this is 191 68 1 0 2.0 / 24.

A

So these are three different networks. That means three separate crossing rings, which so, if one fails, coursing sees that the link has failed, but the communication is still there, so it doesn't go on and tell pacemaker that you need to move resources. No, it informs pacemaker that a link has gone down, but it doesn't. Let doesn't ask it to move the resource, and this is how you should well the purists of H a. They say that if there are less than 4 networks or communication links, it's not an edge, a cluster.

A

So sure the purists can think like that. But here 3 are enough for us. So in order to set this up yourself, what do you need? If you try to do it on Amazon? You will not be able to because Amazon does not support, broadcast and multicast and we need rod cast and multicast for chorusing 2. It sends its packets and this is to it and everything. So we can't do it on any cloud provider. Any software-defined networks, as I understand I, may be wrong.

A

I didn't see a unicast option in chorusing configuration sorry.

A

It works now, it is not. There I've searched everywhere in this man pages and everything. Yes, maybe you can. You can increase my knowledge, so what we you can do is to set this up. You can use V ends or, of course, go for the bare metal and use.

E

A

Their support, okay, I, don't know.

E

A

Good, so if, if the the provider you are using supports these multicast and broadcast, then please go ahead, use that and you need to install chorusing, pacemaker and PCs just three packages we are. We are red-haired fans, so YUM, no applet here for us- and this is a simplified configuration shown. This is more or less what you need nothing more complicated. So you, how do you setup a cluster? You just execute very simple commands.

A

Few few of them are missing here. I just wanted to keep things simple, but there is not much rocket science involved here, so you use these commands and plus few others, and you will have a working H. A cluster, for example, I show you the PCs commands status, and it says that it has two nodes and one resource configured it has. It has two controller nodes online and a resource called controller whip is running on controller one. That means that is the active node right now, which has this IP this.

A

This resource controller whip is dot 20.

A

Demo and for demo I request Henrik yeah.

E

B

Asked if I wanted to break stuff and don't say no to that, so the screen resolution method a bit up but I think you get the point here. Hopefully you can see everything so the two top windows are from my local machine. My host I just do a good control get spots and could control get CS.

A

And white to different colors of the screen, natural.

E

A

Of the screen, this one is showing for every half. A second is getting a list of ports right yeah, and this one. This one here is showing the cluster status, give the troll get CS every half a second. This one is the output of IP address command from controller node 1.

A

This is output of IP address command from controller node 2, and this is an active part, running nginx, I believe and watch loop, every half a second, and we are also pinging, the virtual IP address, which is 10.2 40.0 20, and these are our virtual machines running on k, vm yeah.

E

B

I think I have like 11 virtual machines running on my laptop, so it's a bit slow and the network is a bit shaky, but so what I want to show here is that, right now you can see that the controller one has two IP addresses the 21 and the 20, and the controller 2 has only the 22. It has its own IP, and the thing we'll watch is that when something stops working, it doesn't fail by default. So just so you'll need to look at the up time.

B

So if they stop moving, that's because it can't connect to it. So if we start by looking at these two and I kill.

B

A

B

You'll see that the virtual IP pops over to this one now now this one isn't updated in more because it can't reach it right, and this one connected continues to two to run the IPA Dadd command. So now everything is running on controller to everything in all the traffic and up here you see that everything is still working and it's fine now. The other example is that I have my my nginx running inside my cluster and I exposed it as an ingress. We have traffic traffic running on two machines.

B

We have lb 1, + lb 2 and they also have a shared IP, dot, 40 and right now, it's running on the width. The virtual IP is on lb. 1 can.

A

We look, can we kill this ICMP ping and can we ping dot forty here, yeah?

A

Okay, you use it. Everybody saw here that the ping did not fail when we were pinging the controllers, and now we are bringing the load balancer. What you lie P so.

B

This curl updates every half. Second, it tries to fill the production, dot example.com and if I kill my load balancer and we see that it stops updating and then continues again down here. So I mean you, you will lose a couple of connections while you do the switch, but I think you'll lose connections. No matter how you do it, I mean the problem with DNS and basic DNS.

B

Is that if two other three controllers are down, then two out of three connections world will will not reach the destination every time every single time until you fix it, whereas here you only noose a couple of connections, while you actually do the switch, that's the advantage here yeah. So now it's figured out that this is done and then again I can start it. These machines and this one should start acting again.

A

Yeah but the resource will not float back to that node, which we turn on, because they did not configure that policy. There is no point in moving a resource back to a previously fare node if it is working correctly on the other node. So there is no point in wasting switch over again.

B

So so this is a very simple test. I know it's not very scientific, but just a small test to show that the IP moves and we can still keep on and connecting and can.

A

We go to our slides again yeah. Thank you, Henrik. Thank you very much, so we just killed a kill de virtual machine. You saw it so, let's move forward- and this is the book address- it's work in progress, but it it. It is going to contain all the configurations for pacemaker and Cora sync and everything else how we set up the cluster and that's it so questions please. Yes,.

A

Thank you. Yes,.

A

Very very good question good question, so the thing is we had to deliver this cluster and we couldn't afford going into another research, loop of researching other technologies and bringing them under control and then putting them in production and since I already had experience with the HJ clusters, configuring so I said: let's go ahead with this. It's tested for, like more than I, think I've been working with these technologies for more than 10 years now. So so it was a natural for us to just use this, but we can definitely look in the other directions.

A

Yes, more questions.

A

Okay, that's it! Thank you very much for coming. Thank you.