Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deploying Rock Solid Applications with Kubernetes - Jelmer Snoeck, Manifold

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Deploying Rock Solid Applications with Kubernetes - Jelmer Snoeck, Manifold

Kubernetes has made it easy to deploy applications to the cloud. It’s even made it easy to deploy the same application across several instances. This, however, does not mean that your application will be highly available by default. To achieve high availability for your applications, there's a lot more involved. In this talk we'll look at: - creating secure Docker images and enforcing this at the cluster level with Pod Security Policies; - configuring Health Checks and possible caveats like Circular Dependencies; - limiting traffic between services with Network Policies; - tolerating node failure by implementing correct Anti-Affinity rules; - rescheduling pods onto new available nodes after node failure with Disruption Budgets; - setting up correct deployment strategies; - how to automate this with Custom Resource Definitions;

To learn more: https://sched.co/GrRI

A

Hello, everyone welcome to my talk, deploying rock-solid applications at kubernetes great, to see so many people more than I expected, but that's good, so I'm Almer I work at a company called manifolds. We do a marketplace for developer services. It's like an independent marketplace where you can get like provision like your SAS. Basically I work a lot on kubernetes in our stack. They do a lot of ups in that area, but also build a lot of application. Stuff with that and I do a lot of go in that area.

A

I've got a bunch of these stickers if anyone is interested can come grab them afterwards.

A

So this talk is about rock-solid applications, but what does that mean? Actually? So the first section we'll look at is how we're going to deploy secure applications. This is not going to be at the level of a container or application. This is at a level of kubernetes.

A

Liz Rice already touched a little bit this morning in the keynote, and then the second part will look at is highly available. Applications we'll go into that in a bit so we're all here, because we like kubernetes I, assume we work with it. We reuse it and kubernetes is let a lot of tools to like do a lot of things. It helps us out in other areas that we had to like do manually before, like it makes deploying applications easy. It makes killing even easier.

A

It's just changing a number, but it doesn't do these things by default. You actually had to like configure it and especially like if you want to do a little bit more advanced stuff, you have to like dive into what you actually have to do so in this talk, we'll look at how you can do those.

A

The purpose of talk is that you like have an idea when you go home to like look into these things, so disclaimer will show a lot of the ammo and I've simplified it most of it will probably not work like directly.

A

We copy paste it because that's like cutting it out for other slides, so you can actually read it, but it should give you a general idea of what you should look at a lot of the resources or, like all the resources I've you store, like all the topics of covered here, are linked at the end of the slides as well.

A

So if you look at the kubernetes landscape for an application, this is roughly yet in the century have a thought which is basically the lowest abstraction level in kubernetes. We start there and then we have a bunch of extra things that are linked to two parts and in this talk we'll go over a few. We will go over deployments, we'll go over our back network policies, post security policies and pub disruption, policy, pop user disruption budgets.

A

So the first section that I said we're going to look at a security again. This is at the kubernetes level. Know that, after that, at the application, level or container level and community has like three topics for this. The first one are back. Then we also have network policies and also have lot security policies will dive in with port security policies first, so by default. This is not enabled like you actually have to enable that within this admission controller, but then we can actually dive in, like she like install these.

A

So this is a privileged policy. So this is like gives you access to everything. As you can see, you can like escalate to privilege like to do it. You allow all the Linux capabilities. You have access to all types of volumes. Like config, Maps secrets, persistent volume claims you can access the host network and you can even run as root.

A

What we want to do is make it a little bit more restricted and that's what we get in here so now we say you can't go privileged. You actually like can't go root. You can only use secrets, so you can't use config Maps. You can't use persistent volume claims, so we can only use secrets. You can't touch the host network and you're not allowed to run this route, so we define that now.

A

But how are we going to actually like apply that to application and that's what our back, and so our back is the next phase in that, as our back stands for a role based access control and actually exists out of three ticks. So you have your service account switch thanks for your identity roughly and then you also have a role or cholesterol, and then you also have cholesterol, bindings or roll bindings, and so the roll is then defining what your access pattern actually is.

A

So we'll start by creating a service accounts. In this case it's a simple service accounts for cucum. Then we apply that and then we can create a cluster role.

A

Note that there's this is a cluster role here, as you might have remember from the previous slides, you can actually have a role as well, so cluster roles are scoped across your entire cluster, but just regular roles are scoped across just the namespace. So if you would have a role that is linked to your namespace websites, it's only valid in that namespace. So in your default namespace you wouldn't have access to that.

A

That's important to think about because pop security policies or cluster code by default, so I, don't have to define, don't have to define them per namespace, so we're gonna do with this cholesterol. Here is we're going to define that we're going to use these pop security policies and we're going to reference the one that we created earlier so the restricted?

A

Well, we can now also link that or service accounts, that we created earlier by referencing that cholesterol, so we say Keep, Calm security and then we're going to link that to subjects in this case or subject or subject as the service accounts. You also have some some different ones. We're not gonna go into that. So we link that to the cube from service account now, so we have this all set up.

A

Now we have our cluster or roles linked or service account, but now we have to actually have to bring that to our application as well, and that happens in the deployment or in the pot. More precisely so, we can actually define a service account name, so we're gonna reference, Keep Calm here as well, and now that means that that service account is linked to that deployment. So every time you do deploy that service account is taking care of like its identity for that deployment.

A

So we have roughly secure applications. Now we secure that, but, as Liz rice show dinner keynote this morning, we also had this API server, which pots can speak to. So how can we like secure that now? A simple way is to disable alt amounting that services account token.

A

If you remember from the keynote again, you can like load that token file, and then you actually have access to to the cluster API, which you don't know you want to do so you can disable that by default, but some applications might actually need to have access to to do a cluster. So, for example, if you use certain manager, for example, or if you use contour, they actually need to like speak to the API, to get some information out.

A

So you want to give some specific rules that they can actually do and how you can do. That is with our back again, so you can make another cluster all and I say. Oh ok, in this case we're gonna make a rule that says this accounts can actually use the secrets, so we can actually get the secret's out, so this one can only get the secret's out, so it can't read or account the lead or it can't create it's just getting them, and you can do that with any resource that you want.

A

So we now have secure pots, also security, interaction with those pots between between the pots and the API server. But what about the interaction between pots themselves? That's where Network policies come in.

A

So this is like a rough diagram that you can make you can say: oh, we have an ingress, maybe envoi, and then we have a few applications. Some are open. Some are very restricted, so you can actually like control that incoming and outgoing traffic. So this is an allow configuration which basically means obviously like a lot o. So you can allow all incoming and outgoing traffic, and this is a default. So if you enable Network policies, this is default, you're, not talking else, but then we can restrict that a little bit more.

A

So in this case we have court deployment and then we say for the network policy.

A

The pub selector is an important part in this, so we're going to find the port's that have the label links the app label linked, which are this application and we want to deploy and then we're gonna say the ingress, the incoming traffic, we're going to restrict that to the ingress controller namespace and we're gonna restrict that to the pot that has App Engine X link to it as well, so only ports that are engine X from the ingress controller namespace are allowed to speak to this. These applications.

A

There's one caveat to that and that, like it depends on the network plug-in that you have so some implement all of this, like he can control incoming and outgoing traffic, some only he do incoming traffic. Some of them do none of it at all. So, depending on that you'll have a little bit of differentiation, so we have a secure setup now we cover that part it's time to go into the high availability section of this before we dive in want to clarify us or what we mean high availability.

A

This talk is not about high availability for, like your notes, we're purely gonna talk about high availability for your applications and there's order of talks about like how you can actually set up a high available cluster and stuff and we're not gonna dive into that here.

A

We assume you already had that so the first thing that, like the first thing we wanna bet but high built, is like, if you have an application, if one of your application fails- and you want to make sure that your users to get to see something right, so you're gonna deploy multiply via these applications and then everyone fails the other ones take over, and you still have that interaction that also works for internal traffic.

A

So you might have a circuit breaker somewhere and then, if you go from app application, one application to have some communication wanna make sure that that responds properly and because we work in a distributed environment. We want to make sure that we, if something happens, like network fault, we can still, we can still get the response back.

A

So for this high availability, high availability parts, we're going to look at two sections deployments habitual size, replica sets and then also both both disruption budgets.

A

So with deployments there they're, basically a functionality on top of replica sets and the main thing that they add is replicas and update strategy. So with replicas, it's easy to scale. So you just changed that number and then it will scale to that and then the other strategy is actually quite interesting and that's how we can define how we roll out or applications. So we can say this is the way if we scale that it should happen or if we deploy a new version. This is how it should be rolled out.

A

This is a simple template and we're gonna start with that. So we just say: we're gonna deploy two replicas, we're not going to think about update strategy, yet we're going to just use a default that communities provides and we're just going to deploy to applications.

A

So what happens here is we deployed that we have two version: 1.0 1.0 and that's running nicely. We have service link to that that does load balancing and then traffic flow strips both of these. But what happens if we do a deploy, and so at one point, 2.0 version gets deployed and immediately the one point one point over and gets terminated, but the more important part here is that we lose high availability.

A

Basically, because well as you can see the one point, one always terminate it, but one point: 2.0 isn't booted, yet it takes a while to do and in that, like brief moment, we actually serve only one pot. So we have to think about that. Eventually, this one point two point output comes available and then we have high availability again and then the cycle repeats until all one point, one point OS or done, and everything is to do version everyone's.

A

But what? If that new clock doesn't come available, then you're in a bad situation where you only have one pop and because your deployment is on the new version, you can't really like trigger a new one point. One foot or you can draw, do a rollback off sleep, so a way to fix that or no way to fix that this is so. This is a default configuration actually so by deep by default, we're gonna have a rolling update strategy with a max search of 25% and a maximum available of 25% as well.

A

The important thing to think to remember here is that these numbers they get rounded up. So that means in case of two replicas: you actually get one maximum available. So that means that if you do deployments, whilst you're deploying kubernetes tolerates one unavailable parts in your cluster, the max search is saying that demand of parts that is going to create extra.

A

So if you have two is going to create one extra pop you could set up to one hundred and then it's going to deploy two hundred two extra parts and you have to like play with those values a little bit, but for this example we're just gonna have a look at what happens. They be set this to max and available to zero, which is a little bit more interesting and maybe a little bit more desires.

A

Actually just a quick note on that like if you have more replicas, you still want to think about these numbers because it, the behavior, is similar and you have to like fine tune that yourself. So again, we start with the two one point, one point or versions, and then we do deploy. So what we see here now is that this new version becomes available, but we maintain this high availability, which is good and then eventually the new version becomes available.

A

We go to all the parts and then kubernetes actually like terminates that last one or the old one, and then it repeats that cycle and goes to everything again.

A

But the interesting part here is: if that new deployment fails, like you don't have an issue you get alerted, because one of your pulse is not like spinning up, but you can like still like surf traffic and you don't have performance degradation and again, if you have like more replicas and you might not go to like one part, but if you have a hundred replicas, the question is: can you tolerate like, for example, 25 percent of that going down while you do in plot and which is probably not the case, so you want to definitely fine-tune those numbers.

A

The next section we want to look at is a so with affinity. You can say I'm going to deploy these applications on a specific set of nodes or regions. There's a few areas that you can, like apologies I, can look into basically we're gonna look at hosts here before we dive into the code. Let's just have a quick look at what happens if you don't defy anything, so we have two zones here with each with two nodes and all of the sudden no three goes out, which is fine, we're running kubernetes.

A

It's gonna, take care of this right, but note that we in node one. We only have one application running at the moment, so a kubernetes is going to do is going to see oh, this node one actually has a lot of resources available. So it's gonna reschedule those faulty applications on that node one and it's gonna be happy. The issue with that is there's actually two issues. So the issue that is that node one now has to app once so.

A

If that node goes out for some reason as well, we lose like a lot of capacity. But if you look closely application, one is only deployed in zone one now. That means that, if zone one goes out for some reason it happens like we lose application one. So what we want to do is we want to define some affinity rules. So again, this is the deployments, and then we gonna define affinity and we're gonna use anti infinity here.

A

So we're gonna say these tools like we don't want to adhere to that, like we don't wanna do it refers basically and we're using preferred during scheduling. You also have required during scheduling and the preferred during scheduling is basically saying to determinate this API like we prefer that you like do not deploy this on this machine if like, if these rules already apply there, but if it is, if it is the case and there's nothing else, you can still do it.

A

If you do require it, it's not going to deploy it'd be perfect, like we chose to do the preferred and then set up alerting on that to like make sure that we still have enough capacity, and then you can like look after it once we have the alerts. At least we still have the same amount of like replicas running, so we can make sure we can serve traffic. So what we're gonna do here is we're gonna.

A

Do that plenty of affinity term and we're gonna, do label selection again again we're gonna, select that same application and we're gonna say: oh, is this application already running on this host? So what we're going to do here is we're going to determine like yeah. Does this hostname, like the machines with this hostname? Does this already have this application on and then kubernetes is going to determine like it does so I shouldn't schedule it on that?

A

So if you look at that now, what's going to happen instead of putting everything on the old one, it's actually going to say: oh node, one already has this application one so I shouldn't schedule it there and definitely like Note 4- doesn't have this yet so I'll prefer to schedule it there. That's what that's a nice way to like make sure you, like spray, spread low across your notes, basically and your region, so the example here was cut out, so we actually have some rule set up to do that across regions and zones as well.

A

So we talked about like disruptions or like unavailability jutsu, like external events like know that pictures on a pitch, but what, if your application is having an issue like how does kubernetes know that, like when we talked about deployments? We say it like this application. You see like we want X amount of applications to be available, but how does kubernetes know that and that's where probes come into play so there's two types of probes: there's your readiness probe, which basically works for determining every application, is ready to serve traffic.

A

So if you have a boot like a longer boot time, you probably want to play with these initial delay seconds or period seconds, and that's basically like until these checks basically run until like your application is ready to serve traffic, and then you also had the liveness probe. So once your application is running, you want to make sure that, like it's still LT and you can still serve traffic to that, and that's where the the liveness probes come into play and usually those liveness probes have a shorter period ii and a shorter initial delay seconds.

A

You want to like regularly check that, like once your application is running once it's serving traffic. You want to make sure that it's still healthy and then community is going to do a rebus or whatever it needs to do to make sure it's running correctly.

A

There's one caveat with that and that's circular dependencies. So a lot of people set up these health checks and then say we're gonna check. I, have these dependencies and I'm going to check that that dependency is live and then, if that's, not alive, I'm going to fail my health check, but you might have end up in a situation where you have this, where you have three applications in this case a B and C that all depend on each other. So in this case a Leyden C C relies on being B relies on a the issue.

A

Here is that if she fails, that's not that big of deal yeah where C fails. The next step. That's going to happen is that a is gonna fail as well, which means that B is going to fail as well and to actually get out of that. You need to like either time it very well or not use a circular dependency. Basically, so we reinstalled de crédit, we added a degraded label to that.

A

So, instead of actually failing the hell check, we just say that it's degraded and that way like external monitoring services, can I, see, detect and just alert us as well on that.

A

So we talked about a lot of this captions so far, so we talked about like expected disruptions from deployment. So if we like take out some some applications, but also about unexpected failures or disruptions like an application failure or a node failure or zone failure, so kubernetes does a lot of scheduling and rescheduling, and you need to make sure that kubernetes like is aware of this and make sure that cuban it is like knows that you definitely want this application to always be running at this capacity.

A

That's where poor disruption budgets come into play.

A

So another example here is: we have four nodes and node 4 is just know that just spun up, so we're scaling or we just tolerate. We just had node failure. So Note 4 is a new note. That came out into our cluster and we also have application woman that is using a lot of resources, so application one is on node 1 and node 3, and then no just doesn't have that much stuff going on at the moment.

A

So what cubanía is going to do is going to say: oh I can take these like hi or fencing application. The applications that are using a lot of resources and I can just reschedule them on to an app to a node that isn't using a lot of resources.

A

If you notice that here station now is that, well again, our application is not available, it's being rescheduled. So that's where all destruction budgets come into play. So again we want to define this and it's important to like the finest correctly like to what you need, because these objects are immutable. So once you apply it you you have to stick well, you can delete it and apply it again, obviously or replace it, but you can't just update them at will. So it's like better to use a percentage here.

A

So that way, if you scale up it applies that percentage instead of like having a fixed number number and that wouldn't really scale and again the percentages are being rounded up, as with database strikes you so what's going to happen now again, we start with this beginning face and kubernetes just going to say: oh I'm, going to see redeploy this app one, that's using a lot of resources, I'm going to redeploy it on this note that doesn't have anything at all and then I'm gonna in the next face.

A

When application one has become available, I'm going to take application, one from node tree and I'm going to put that on to- and that's basically rescheduling so we kind of want to prevent all of this miss configuration, and how can we ensure all of this is set up correctly. So there's a lot to think about here. We have to think about what destruction, veggies I think that we set up these update strategies correctly, that we have replica setup.

A

You have to think about a lot of things and that's not always the nicest thing to do. Can we avoid that? What linting linting is like a lot of work. You have to lie, set that up correctly make sure it's like up today. It's quite fragile as well. You might miss something if your winter isn't the great like isn't configured correctly and but the biggest issue is that if you just don't, run the linter well, then you can still deploy all of this to your cluster right.

A

So what we want to do is want to use web hooks, so quickness has web hooks and, as you we saw in a keynote this morning, we can do validation. We can also do change this abomination. That means we can actually become bypass style, anything so it's actually in the cluster. So if you deploy that into your cluster, it will definitely get validated properly and again. Liz touched on this.

A

In her keynote she used open policy agent and in this example here we're gonna just use another tool called Barbossa, so quick overview of the web hooks again. So if you do requests already do keep CTL apply. You're gonna go to the API. It should be handler. That's going to do like some modification and authorization and then you're gonna hit the mutating admission controller and it's going to see for this object. So in our case it deployment is a web hook. That is watching this.

A

If this is going to go to that left hook and then it's going to do its mutation, that is needed, so you could even have a limitation mutating weapon that says I'm going to make sure that all my applications at least have two replicas running or you can do like automatically spin about disruption, budgets and stuff like that, then you also.

A

We also do like very light schema validation, so every object has Jason validation. Even if you use custom resources, you can find that that's quite lightweight, and then we also have the validating admission controller.

A

So the validating admission controller is going through a list show this morning and it's going to like take that object. So again, our deployment is going to take that object and it's going to validate that against a set of rules that we have defined and that way by having this reinforced at the cluster level that you have all of these policies set up and if you don't pass this validation, it's obviously going to error on your side.

A

If you do is going to be persisted to SVD so again, in our case, we're gonna have this simple example here: we're gonna apply that one replica, that's the important part and no update strategy, so we're gonna apply that and we see of this deployment is successfully created, but now we're gonna, say: I'm gonna have a high availability policy. So again, this is like specific to the Barbossa tool. You can do this with open as well. It's just the syntax is a little bit different. This is like with custom resource definitions.

A

So what we're gonna do here is we're gonna, deploy this high availability policy in a namespace and then we're gonna say this applies to all pots. So we use the selector here, which is to get in a label selector and reason for that is that you can actually start mixing. So you can have a policy that says all pots should have a label, for example team, and then you can start making availability policies based on teams.

A

So then you can have a selector based on the team level, which is quite named and then we're gonna say you need at least two replicas so yeah you want that to like at least have some high availability, and then we want to make sure that our strategy is rolling update, not recreate, because you don't like recreate and then the minimum surge that you're gonna do is 25%, which is fine and maximum surge is 50%.

A

That's to make sure if you have like a bunch of a bunch of replicas, because we have a minimum that we haven't specified maximum. So you could have like a and, if you would say max, urges under percents. Then, all of a sudden, you spin up 100 hundred extra parts which you can do, but you needed a cluster autoscaler. So we want to limit that as well, and then we want to say the max unavailable in in our case is going to be zero.

A

You can make this percentage base as well, so we're going to apply this policy great and then what happens? If we apply this deployment, we see that we have two errors. The first one is that the replicas is invalid. We want to at least do, but we have one so that's not valid, and then we also see that the maximum available is invalid as well, so we have spent five percents and again this is rounded up.

A

So if you have two replicas or even one, the 25% is going to be rounded up to one, so it's more than the at most zero. So it's going to fell on that as well, yeah, so webhooks. So this is using Barbossa. The main differences that proposed I would do the machine that focuses all that the over stuff works great as well.

A

So that's it if you're interested in all this stuff manifold is hiring so definitely come, reach out and yeah. That's it I'll be around for questions. If you are.

A