Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Choose Wisely: Understanding Kubernetes Selectors - Christopher Hanson, RX-M,llc.

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Choose Wisely: Understanding Kubernetes Selectors - Christopher Hanson, RX-M,llc.

Kubernetes uses labels and selectors to tie otherwise distinct API objects together. Application controllers like ReplicaSets use labels on pods to enforce replication factor; Services (via Endpoint controllers) also use labels on pods to identify endpoints for routing client traffic. However, not all controller behavior is the same! Some controllers add extra label metadata to avoid potential collisions and others do not! Through a series of live demos, this session explores the use of labels and selectors used by controllers so that you can clearly understand and use both to your advantage when deploying apps on Kubernetes. The talk will conclude with examples of advanced deployment patterns such as red/black, highlander, and canary that can be achieved natively in Kubernetes via labels/selectors.

A

Hello and welcome to choose wisely understanding, kubernetes selectors, my name is chris and I'll be taking you through this session. uh I work for a company called rxm. We are a cloud native training and consulting firm, so I do a lot of training. I do some consulting and I do a lot of speaking so you may have seen me last year at uh kubecon here in europe. Well, I suppose it was virtual because of co-ed and talked about uh roll outs and roll backs and behaviors with controllers.

A

Today, it's sort of related in that we're going to talk about controller behavior with selectors.

A

So without further ado, let's get started so before we talk about how controllers use selectors, we need to talk about what they're selecting right so in a kubernetes cluster most objects have some sort of label and they may even have some sort of annotation and this could be any object. A pod, a node service.

A

The nice thing about labels is that they're selectable right, and so uh we can actually use kubectl commands to create labels uh review labels. So, as you can see on the slide, we've got the show labels flag. We can add a label after the fact. So if an object already exists, we can use the label command to add additional labels. We can overwrite existing labels if we want a new value for a particular key and we can remove them as well for existing objects in terms of using patterns to select things.

A

There's three patterns here: there's the quality based, which means that objects returned by that query, have satisfied all the constraints uh they may have additional labels. So if you just selected one label- but there are several others, the the object would still get returned by the cube ctl command, as we see here, they're set based. So you may want to filter based on a key with several values, and you want to see the objects that have um one or more values and then the key name.

A

uh You can just use the key and and see all the values for a particular uh key value pair, and so you can see some patterns here. We've got the team key and we're using in scarif with bespin and we're separating the two values with a comma and that logically ends, and so we get uh all of the results back for the objects that have both team, equal, scarif and team equal specimen.

A

We can expand upon that by using a set base, selector and an equality-based selector. So a second example here is team in scarif best been with apical's rideshare and then, of course, that minimizes the results that get returned. We can even use labels with things like the logs command.

A

If I don't know the name of a pod, but I want to get its logs as long as I know, a key value pair that is on that pod, then I can just use logs with the minus l, and you can see here that we've got three different pods. That answer right and they've got 36.3 40.2 and 40.3.

A

Each has a different ip address, so we can tell that comes from three different apache web servers. We can even delete things with uh selectors, so we can.

A

In this case, we delete a deployment based on a team plus a release key value pair, so just like, we can use labels and selectors in cubectl commands controllers, we'll use selectors to find and manage other objects, so selectors are used by application controllers, like deployments replica sets and others to target sets of pods to manage and deployments and replica sets and jobs and daemon sets will support both the equality based selector, as well as the set base selector okay.

A

So I've got several node cluster here that that I'm going to be working with we'll just do some do something simple. uh First, let's start a watch, I gotta watch here so we're gonna watch uh deployments replica sets and pods we're going to show their labels so that we can actually see those.

A

I don't have anything deployed yet so it's just going to be empty for now and then we'll create a uh deployment and we'll just call it casey, deploy uh just very simple uh use, a particular image. The host info image that we're gonna use throughout the demos. Here, uh when we query it directly, it will give us its ip address and its host name, which will be valuable for demonstration purposes. We're just going to run three replicas in this case.

A

So we see the deployment the replica set and the pods get deployed and we can see the labels that are used right. The deployment has the app equals kc deploy label and it has that because we use the create command and we're using an imperative. So when you do create deployment and the name, then the key value pair becomes app equal to the name, so we called it kc deploy, so we've got app equals casey deploy, but we can also see that an additional label was generated.

A

The pod template hash with the replica set, and then that also shows up on the pods. So what that means is, if I run another pod right, so I'm going to run this kc deploy pod, just use the same image and then label it with the kc app or sorry app equals casey deploy.

A

What we would expect is that the replica set is going to ignore it, because it doesn't have that extra key value pair, that's being used by the selector that the replica set is using, even though we're using the exact same label, because we had that extra dynamic label. We avoid that kind of situation.

A

So what that means is that a deployment controller will use a dynamically generated key value pair and that will avoid any potential overlaps with standalone pods and the pod template hash. Key value pair really isn't intended for a user to create right. It's it's going to be managed by the automation and with every revision of the pod template that gets deployed. The hash is going to change.

A

The label's value is going to change too, and this allows us to again only control the pods that we intend to control and avoid any potential collisions. So, let's take a look at another type of controller. Let's look at the rs directly, so if I use an rs directly rather than using it with a deployment, the behavior is a little bit different. So, let's take a look at a demo using replica sets all right so this time around we're going to look at replica sets to see how they behave and we'll do another watch.

A

So this time we're going to watch just public sets and pods, but also taking a look at their labels again, nothing running cleaned up after the last demo and so first off we will run.

A

A pod that is going to use the label we'll use with our replica sets we're going to start with the pod this time and see that it's running first, rather than as we did before running the pod after the fact, and then I've got a replica set and I've done it in yaml form, because there's no way to do that. Imperatively, if you take a look at the slides, you'll see the yaml in the deck. So I'm just going to cube ctl apply that.

A

And just called rs.yaml, and what we're going to see is that in that replica sets uh manifest, that the desired count was three and so, when we deploy that it actually only created two pots that were deployed when the rs was deployed, and we can see that it has now taken control of our original pod.

A

So, unlike deployments using a replica set directly means that we do have potential. If the label selector is basic like this and matches one of the label selectors of an existing pot, it will maintain control of it.

A

Now, let's say we had a second rs, so I'm going to create a second one that called clash rs, it's identical, yaml file to the first one, with the same selector, just with a different name and we're going to see that it is going to create two replicas and, of course it does that without detecting the original pod. So it essentially ignores our original pod, because that has now come into the control of our original replica set. So what that means is they're not going to fight each other.

A

Now, if I coup ctl, delete and we'll do both of.

A

A

We should see that everything gets garbage collected so that all of our pods now go into terminating state, even though that pod was created first independently, because it's under the control of our uh rs controller, it is now terminated, along with everything else.

A

So what that means is that replica sets on their own, don't create that extra dynamic, key value pair to try and avoid collisions. So it is possible that a given replica set could inadvertently begin controlling a standalone pod that has identical labels that is being used by the replica sets selector.

A

However, once that happens, if we have a second or third or additional rs using the exact same label, query it gets ignored, so the only one replica set will pay attention to it. That way, we don't have any sort of fights happen between the two rss. You know, if one replica one had a replication factor of two and the other one had a replication factor of three right, you could imagine they could uh start to fight each other and lead to thrashing.

A

So that's a good question uh just in general and what about not standalone pods but controller-based pods that have overlapping labels right so again, the example being two rs's use the same selector one has a replication factor of two one has a replication factor of three: would they thrash right, adding and deleting pods until we went into crash loops in current versions of kubernetes? No, because the controlled objects actually have a reference to their owner right so uh around the one sixth time frame.

A

uh This was added to a number of the controllers and you can actually see it. If you use coupsy tail explain you can see that there's an owner reference and in the highlighted text here we point out that if the object is managed by a controller, then there's a subfield called controller that is set to true and of course, we use cube. Ctl, explain again to show that so really uh label overlaps really only occur with standalone pods, not pods, that come under a controller already like from the very beginning.

A

But then, of course, once the standalone pod is taken over by a controller, it actually receives the owner reference, and that's why we saw when we deployed the second rs. The first rs had updated the pod to give it that owner reference, so there's no way that the second rs could have taken control over it. Now, what about other controllers?

A

Things like daemon sets staple sets jobs and cron jobs. Each one of these controllers has implemented some kind of labeling scheme that allows them to avoid any sort of collisions with standalone, pods and, of course, pods that have been deployed by a given controller. Have that owner reference as well, so it's not just deployments and rs's, but each one of these controllers have had that owner reference added to them over time, but we can see like deployment based pods. The damon set pods are given a hash label.

A

A staple set pods are given both a hash label and a pod name label, so an extra label for staple set pods and then jobs and cron jobs. The pods controlled by these are given a unique label related to their controller's id. uh Just some thoughts about how we could use this, could those dynamic labels be used in a standalone pod? Let's say you wanted to bring a pod.

A

Of course the controller is trying to avoid inadvertent control, but maybe this is a desired behavior right uh yeah, you could, you could add a dynamic pod template hash, for example, to a pod that you wanted to add to a deployment after the fact, so you could use the label command on that pod with some caveats uh if the deployment was created first and all of its pods were deployed first, when you added the standalone pod, if you ran it subsequently and then labeled it, the replica set is basically gonna kill it, because it's last in first out, and so you actually have to create the pod first- and you saw me do that in the demo with the rss I created the pod first and then I created the rs with the three replicas right.

A

That's because if I created the rs first and then ran the pod, the rs would have seen the mismatch between the desired and current and just killed the one that was the newest.

A

You have to make sure that your pod, that you want to add to your deployment, is older and then, if you're, using something like a deployment, a rolling update will essentially replace that standalone pod with a pod based on the template, so you'll lose that pod and one of the ones from the template will come in its place with the change. A work around could be to basically disable that right, so you can do a rollout pause on the deployment, even if config changes happened.

A

Of course, the rolling up update would not trigger, but it's kind of a lose-lose right, because either you're going to lose your standalone pod. When the rolling update triggers or you're basically losing the rolling update feature so yeah, you can do it, but you know there's lots of caveats in place.

A

So moving on from controllers or application controllers, we want to look at another type of controller, which is this a services. Endpoints controller, so in this case the controller is used to target a set of pods and create a pool of ips for load balancing so that we can load balance, client and peer requests to a set of backends.

A

Unlike other controllers that we've looked at before the services or the the endpoints controller can only support a quality-based selector. So you can't do group a selection here in this demo.

A

I've broken up the terminals into three parts, because what I want to do is first, I want to establish a watch for all the objects we're going to use so we're going to use a deployment we're going to expose that deployment with a service and then we're going to use a standalone pod and we're going to again have label overlaps on purpose to show how the behavior of service selectors work. So, of course nothing just yet we will uh run our client pod. So in this case it's just a busy box.

A

So I can get a shell and run a loop now, in this case, I'm going to grab the loop here and paste it in, and so we're just going to query the service called kc and of course we don't have one yet so, let's create the deployment and the plan is just going to be called kc again using the host info image to give us. The host names give us three replicas, so just basic deployment and then we'll expose it.

A

So in this case again we're just exposing the kc deployment mapping, the port that it listens on to a standard port. To make things simple. Now that I have the service established, I can start my query, but what I want to do first is run a standalone pod that has a similar label.

A

So we can see this service is using app equals kc as the selector and so we'll run another pod using the same image, because I want to see the host name, of course, as part of the demo and then label it so that it falls underneath the uh the service and so we'll run this loop.

A

Oh, and the very first uh request goes to our standalone plot right, so um of course, we're uh randomizing our load balancing with a with a normal service.

A

So what that means is like, unlike other controllers, that we've seen the endpoints controller does not try to avoid any sort of selector overlap. So, of course, this could be problematic if unrelated, uh an unrelated pod or unrelated groups of pods inadvertently end up under a given service. But the nice thing is.

A

We can actually use this to our advantage, because application controllers support the implicit, config change trigger and rolling update features, but they don't really expose other strategies so, like blue, green highlander, dark or canary the staple set controller does have a partition where you could do a canary. But what, if you don't have a staple application? You have a stateless one and you want to use a deployment controller well. This is where we can use the service selector to our advantage for the blue green strategy.

A

We will start with a an existing deployment of pods, but we're not going to use the deployment controller. This is because we're going to need to make subsequent edits to the pod template if you're familiar with the rolling update. You know that that comes with the deployment. You know that when you make changes to a pod template in a deployment controller, the implicit config change trigger fires and you're going to have rolling update, but we don't want that strategy.

A

So we're going to use replica sets directly create a service that is going to use a selector that can be shared between two groups of pods, the the blue and the green, and then uh we will add the label that common label that we're gonna use with this the service selector to the pod. Subsequently, so you can see the replica set uh on the left here: has a single label demo equals blue rs and then we're just going to use the label command to add the pod uh labels directly.

A

So once we've got the blue pods in place, you know we've got them running for some time right. We want to do an update to the application. Then we create a second rs in this case. We're changing the image from the alpine version to the latest version.

A

We deploy the green, rs and pods, and if you want to do a dark strategy here, we just don't immediately add the labels to the green ponds and we kind of let them warm up in the background and then subsequently. We will then add the labels using the label command to the to the green pods and we'll have a big pool where the blue, pods and the green pods are all going to be rolled up under the service.

A

We can't immediately remove the label from the blue pods because remember that controllers and kubernetes have eventual consistency, so the endpoint controller is going to take a minute to not a literal minute, but uh it won't be immediate right to update itself um get the all the pod ips and cooperox is going to need to implement this. So in our case we're using the default configuration of coup proxy, which is using ip tables and, of course, cuproxy, has to go and do that work.

A

There needs to be a little bit of overlap during this period, you're, going to see that both blue and green pods are behind the load balancer.

A

Once we're satisfied that we have enough endpoints behind the service, then we can simply again use the label command to remove the labels from the blue, pods and the traffic will just be directed to the green ones. We can retain the blue pods for roll back or undo. If something is wrong with green or if you want to go the highlander strategy route, you can immediately remove the blue rs pods after removing those labels. So let's take a look at a demo of this particular strategy.

A

So again we're just going to watch deployments. Replica sets, pods and services see all the things that we have deployed nothing right now, we're also going to run a client, so we can actually see again we'll use the host info image.

A

So we can see those hostnames come back so I'll start the client right now you'll see it creating and then we'll again have another loop going just copy that the service is going to be called blue, green. So we'll query that, but we're going to wait so first, let's apply our blue rs.

A

And it's identical to the blue rs we saw in the slides, and so it just has the one label right now demo equals blue rs. We will then also apply our blue green service.

A

And the selector we're using is going to be method equal to blue green. So right now the service isn't going to have any uh pods behind it and we can confirm that uh with the endpoints.

A

And, of course, the blue green service has no endpoints, but if we label.

A

Try that again we're going to label pods that have the current label demo equals blue rs, which is our rs pods, so we're going to pass that command to the label command and then add the method blue, green. So we'll have some pods that will be behind that service and down below. You can see that we've got the now. We've got the blue green and we can repeat our get endpoints and of course, we've got some endpoints for the blue green service.

A

Now now I can start my client loop and, of course we get the blue rss. So we've got our blue version of the application deployed and we need to then add the green.

A

Let's go back to the apply and use the green rs like we saw on the slides right and so again, we've just got the demo equals green rs labels for these pods. So, of course we wouldn't expect them to be under the service right away. Now we can use that same label command that we used here uh just change the demo blue rs to green.

A

And this will add the method blue green to our green pods and we'll start to see them show up in the load balancing with the client on the right once I hit enter here, we go.

A

So now we've got some green mixed in with our blue, so the endpoints controller is updated. Q proxy has implemented the new targets and we've got the blue and the green mixed in, and then we can go ahead and remove the label uh actually we'll go. Do this one and syntax is just label but method minus that will remove the label that we don't want from the blue rss.

A

And there we go now: our load balancer is just sending traffic to only our green pods. Lastly, here we can do a canary experiment using similar technique. So in this case we can actually use deployments and you absolutely can use rss2. So your production, let's say your production set of pods- is using a deployment and we're not going to really do that. Many or really any edits to the production.

A

What we're going to do is add, baseline and canary, so it sort of depends on how you want to do it, so we can use a deployment with baseline and canary and deploy them with the labels that the endpoints controller is using and they'll just be added to the load bouncing pool or, of course, you could use rss, deploy the pods first and then use the technique we used in the blue green, adding the labels. Subsequently, it's really up to you how you want to do it, but either way you can achieve a canary experiment.

A

Okay, again we're just going to watch deployments rss, pods and services.

A

This case, I'm going to use the deployment method, so I'm not going to use rss, which means that my yam manifests for my deployments already have the correct label so we'll run this again, we'll run another client here, so we can see the load balancing across the production, the baseline and then the canary.

A

Let that go for a minute we'll set up the loop, but not start it like we did before. So. In this case the service is going to be called canary demo.

A

So let's actually take a look at some of these. So.

A

Look at our prod first right, so the common label that we're going to use is amp equals demo and each one of the deployments is going to have its own release version. So if we wanted to, let's say query for a particular deployment by its release type, we can do that and delete them. So I could use like a delete, uh minus l and then, like pick a release that I wanted to delete so uh same image that we've been using uh up to this point.

A

So we'll go ahead and apply that.

A

We've got some our production pods rolled out right, so let's also look at the service, so it is service canary here and so again, it's just going to be using the selector of app equals demo and that will be a common label across all of our deployments.

A

Just a basic service using a cluster ip, so we'll apply.

A

That ah it would help if I use the word apply right. So we've got our service, we've got our uh production deployment, we'll go ahead and start the the loop here for the client and, of course, we've got our production pods aptly named, so we know how the mix shakes out. So now we can look at our baseline and uh very similar. uh It's using the same release, which is the alpine release right. uh It should be the same release version or whatever experiment.

A

You're gonna do right should be identical to the production version and then we'll also look at the canary.

A

Right so in this case we're again doing the uh update of the image from the alpine version to the latest, and you can see that these pods are going to have the apicals demo label too, so they can fall underneath the service. Now, in this case, I'm going to do applies at the same time. So we have an honest experiment.

A

Because we don't want to have an experiment where we deploy one first, that would not be a proper experiment, so we'll apply baseline and we'll apply canary. At the same time now we've got a whole bunch of pods, some of which uh coming off the screen a little bit. So we'll move this up, and we can already see on the load balancing here on the right that uh the canary uh pod has been queried a couple times and there's our baseline right.

A

So we've got a mix of canary baseline and production and, of course, uh while the clients are communicating with the various uh releases right, we can collect all of our metrics uh uh measure and then determine whether or not the um the experiment is a pass or fail, based on whatever we're trying to prove right. Like does the change to the image uh introduce more latency to the application right? Are there more error rates or whatever it might be right? And then, at the end of this?

A

What we'll do then is simply uh say: oh okay! Well the let's say the canary experiment was a success, and so this is a long command. I'm just going to copy and paste it from some notes. I've got here and so we're going to delete the baseline, we're going to delete the canary and then we're going to set the image for the production deployment to the latest because the canary was successful. So we want to update the version.

A

What you'll see is the rs hash for the production pods in the load balancing will transition from the 5f hash to the new hash. So let's go ahead and trigger that change.

A

ah You know what I need to do is um do the right path for this delete. Well, we're doing the rolling update right so there's the new hash for production, and I just need to give the full paths for these guys. So.

A

And we'll get them out of there.

A

Right so, ideally, of course, we do that all at the same time really wouldn't hurt if it wasn't all at the same time like we did there, but of course we delete the baseline in the canary and we've got our new production deployment based on a successful test, and that brings us to the end. I just want to say thank you for your time. Come find us over at rxm.com.

A

We have quite a few training courses that are available as open, enrollments or also private trainings. With teams we even have certification, boot camps and self-study. So if you're looking to take the ccad, the cka or the cks, we've got, you covered once again. Thank you and enjoy the rest of the show.