Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Ready? A Deep Dive into Pod Readiness Gates for Service Health... - Minhan Xia & Ping Zou

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Ready? A Deep Dive into Pod Readiness Gates for Service Health Management - Minhan Xia, Google & Ping Zou, Intuit

This talk will be a deep dive on “pod readiness gates”, a new Kubernetes API for extending pod readiness with custom external controllers and introduce its usage in real world scenarios with service health management. Kubernetes clusters that integrate with external network infrastructure (such as a cloud provider) need a way to coordinate Kubernetes Pod “readiness” with setup latencies in the network fabric. Not tackling this problem causes lost packets and dropped connections OR requires inserting worst-case delays to pod startup to account for external setup. It allows Foremast, an open source engine to maintain the health of applications running on K8s, to remediate similar problems in a much smoother way.

Minhan and Ping will cover the coordination problem, how the new pod readiness gate API fixes the issue and experiences from both Google and Intuit using the API with real workloads. At Google, this API is used to ensure no traffic disruption occurs for user service across changes to the workload. At Intuit, this API is used to maintain application health during canary deployment using Foremast.

https://sched.co/MPaQ

A

A

Alright, good afternoon everybody, my name is min Han I'm, a software engineer at Google and my co-speaker is ping. He she's a software engineer. I enjoy it so we're here to talk about pot, readiness and part readiness gates and we're here to explore their meanings and their usages. So we have four parts. One is the quick recap on PHA status and also intro to part Ren is gate and a usage at gke and into it.

A

So quick recap, so kubernetes api expose reach set of containers status. So here's an example of container status here, like it has image IDs like container runtime, IDs underneath and explicitly has a readiness per container and also the state per container and communities. Also let users to define liveness and readiness via the form of liveness probe and readiness probe, so each container gets to define okay, a library's probe and a readiness broke and cubelet will execute.

A

This probe on the local node and for liveness is basically signals whether cubelet has to kill the pot and restart it. I'll kill the container or you start it, and the readiness basically reflects if the part is ready and also there's a rich set of pot level status. So there are parts scheduled which means that if the scheduler has already scheduled a pod or initialized means that if there's like all the any containers in the pot are like actually successfully started, and finally, its readiness right, whether all the containers are actually ready or not.

A

So here's a quick example of pot life cycle. So here is a pod with two containers. If no liveness probe is specified, then it's live by default. So you can see when the part is started. The both containers are considered alive and container to first become ready and then container one becomes ready later, then the pot becomes ready and they start serving traffic, and when the pot is deleted, a sick term like Kuebler, will get a signal, both containers, a sick term signal, and then they will gracefully shut down.

A

If the container image can handle it and then when the graceful termination pur is done, then it will Kuebler basically signal is to kill and kill the pod.

A

So what does part redness mean horrendous means that all containers are ready and then the pot is ready, then part is ready to serve traffic and what are the meanings it has. So, each consumer of part readiness is actually effectively extending its definition. So here we look at workload and specifically, we wanted to take a look at deployment and in deployment. You can specify the rollout strategy where you can say: hey.

A

What's the max surge and Max unavailable that basically configure how fast you wanted to like proceed, the rollout and what happened underneath is that the deployment controller will actually control like valuable in in one case of a rolling update. It's gonna be juggling between two replicas set and in this example it's in the middle of roll out and you can see generation 1 replica set has replicas 5 and generation to replicate has replicas 5. That means you have we have five old parts and five new parts.

A

So here's a typical workflow of the rolling update you can see part one's gets created and part. One gets scheduled and cubelet gets Kuebler watches the pod that is scheduled on its own and cubelet runs the pod in and start a part in to onto the container runtime and container under reported success.

A

Then cubelet says the part is ready and when the part becomes ready and cute deployment controller, observe the pot readiness via like replica set, and then it makes a decision to say hey it's it's time for me, based on the deployment inspect the rollout strategy, whether I should proceed to the next pod, and in this case it says. Okay, then, like the part, one is ready. Then I should proceed to the next one, which is part two and then part two go through the same cycle.

A

As you can see, part readiness is a core signal for deployment decision-making.

A

Well. Similarly, service is also a consumer of part. Readiness and and pair with sickness service is the endpoints object, and the endpoints object contains all the end points behind the service and for all the end, points for all the parts that are ready it should be contained into in the in this end points object. So this is a typical endpoint update flow. So when the service is created, the endpoint controller is already automatically creating the corresponding endpoint object.

A

So when the service is created, the end points already exist, the part gets started, Hawk gets scheduled, then cubelets sees it and cubelet starts the pod and reports the part ready when the container runtime everything is in in shape or the like. Readiness for reporting, ready and then endpoints controller are watching all the pods, and he sees this new part belongs to the service, has become ready. Then the endpoint controller updates the endpoint object and the end point once the endpoint object gets updated.

A

It's reflected this new endpoint is ready in the endpoints object itself and the endpoints object will be distributed to all the queue proxies and the queue proxies will further implement the actual load. Balancing mechanism underneath is either iptables or IV, vs or EVP F or whatever, like queue proxy, you use and then finally the endpoint is program. So you can see the color coding. The blue states means the outer state. The green state is the actual intended state in various dimensions.

A

So if we look closer, the endpoint status is actually not feed back into like anywhere. So it's just program, it's like sort of like a best effort and obviously the end point like whether the end point is program or not does not reflect back into the pot readiness and you can see, there's a gap. There is actually a programming latency from like the pot becomes ready and the endpoint is actually getting reflected in IP tables or IP vs, and also combined with the fact that it's a cluster, it has many nodes and then each no.

A

You have to program each no to actually reflect it. So so there's a programming latency between the pot becomes ready and it's reflected globally in the cluster. So so what is part readiness? It's a core signal to workloads. It's either kubernetes native workloads. Third party workloads operators like they all look at pot readiness and also it's also a key like signal to networking like abstraction, like services ingress because ingress reference services so inherently rely on pods and also network policy.

A

Although never policy does not rely on heart readiness, but it needs to be programmed so effectively. Part readiness is the round oval point of like the whole communities, API ecosystem.

A

So actually does these like do these API is work together right intuitively they should. They should work together if, if they don't like, why, why do people actually a table like kubernetes? Why do we have such a big cube card right? So the answer is yes, but not without caveats.

A

So if you, if we look closer workloads and pods, have a closed feedback loop, that means what close they managed the pods and also observed the pod. They they rely on the pot readiness for their internals like either for rolling updates or reacting to changes like, for instance, the note goes down, and then you wanted to maintain the certain healthy, the certain ready, replicas available, replicas stand.

A

You were gonna schedule, more parts which runs elsewhere right and on the other side, the the relationship between parts and the networking abstractions are actually one one directional right when so a single directional relationship. So so, if let's say, if the end point is program, there's nothing to feedback it back into the pod or whether the network policy is actually enforced.

A

There's no, nothing! You can feed back into it's sort of a best-effort, so why this hasn't been a big problem so far, why does one directional like a relationship between the parts and the networking aspect and then there's and then the workload effectively ignores the the networking abstractions? So this is mostly thanks to the robust implementation of Q proxies and the underlying API machinery. So from what we have observed is that in a like, like normal cluster, not under load, it's pretty reliable and fast to update a single end point.

A

Usually it's less than five seconds and the new end point will get reflected, go blow globally in the cluster and not much people, not much user acumen. A user has like complain about it and say hey. We have like the end point update, is too slow or something like that, and it's very unlikely to hit the corner case. Something like my deployment is done and I have rolled out all my new pause, but my service is still pointing to my ol pods right. This is pretty we're like like it's like very small cases like this.

A

So here comes the paw trainees gate, so cubelet dictates part readiness, so far, readiness probe only executes on the node and there's no external feedback allowed to influence the pot readiness. So what if Cuba is not capable of determine whether a pot is ready or not right?

A

And secondly, what if we want the workloads to be network? Aware, meaning that let's say when I proceed, the road out I want to make sure that everything is ready when what I mean everything's ready is that endpoint is ready. Network policy is ready. Everything like everything related, is ready right before I proceed, not only what cubelet tells me is ready, so and finally, there are more. There are more sophisticated service health management solutions which incorporate more application level, metrics like like latencies, and things like that. How can they better integrate with kubernetes internals?

A

So can we do something like part, ready, plus, plus and.

A

When we look at like when we are trying to solve this problem- and we actually face a lot of constraints, so the constraints are mostly summarized to backward compatibility, backward compatibility and backward compatibility, so kubernetes api is have been well adopted and it's many people are running in production and currents. Although current system is not perfect, but it works right, it works most of the time. So how can we introduce a new part ready plus plus mechanism without breaking the backward compatibility?

A

So some of the concerns are like how to make it work with existing part specifications how to make it work with existing cubelet control flows and I. Don't want to inject overly complex Kuebler control flows so that, because Kuebler runs there where it should be reliable and yeah.

B

A

Should just work and also how can the mechanism work with existing workflows right? There are many different workflows. There are replicas that they were like kubernetes, natives native ones, deployments they've all said, and they are user-defined controls.

A

How can we like make this change without changing all these controllers and finally like how to make it like transparent to the user, so that the user is not aware of this new change and the part rating scale was proposed and implemented, and it's now g8 in since one the company is 1.14 and here's an example of the pot renegade like configuration?

A

It's it has two parts: one is in the pots back, one can specify a list of part readiness, Kate and the pot renegades are each part, renegade reference, a pod condition and in the past status the pod conditions are in like indicating whether the readiness gate is actually ready or not ready. In this particular case, although I do not like. Basically, let's assume all the containers are ready and you can see that readiness gate one is false.

A

Hired the status is false and readiness, KB is true, so that means readiness, K a, is not ready, and that means the whole part is not ready. So we effectively change the pot life cycle into this graph right. You can see when the pot starts up. Everything is nothing ready, nothing's ready. So then, even though a container one and container to both up both have become ready, pot is still not ready because it's waiting for readiness gate right.

A

So only when everything is ready, the pot is considered ready and then it starts serving traffic and yeah, and then the rest of the the lifecycle is the same.

A

So what hot ready skate, container pot readiness means all containers are running, all containers are ready and all part readiness gates conditions are true.

A

So in order to capture the existing part, readiness which means like all containers, are running and ready. We introduce a new part condition, which is which is called containers ready right. So when the containers are ready, then it preserves the old definition of par readiness, which is all containers already. So whoever like wants to consume like this can ya can switch to consume then the containers ready condition.

A

Well so far we we talked about the API. How do we use it like with a seamless user experience? So there are two parts: one is how do you inject the readiness gate, as you can see, it's a part of the pots back. Well, there are already pretty mature mechanism to inject stuff into like any object in kubernetes that is web hooks.

A

So if you have some kind of infrastructure or platform specific reading scale, you want to inject write your own web hook and then who how to update the readiness condition right, because so far only cubelet gets to update the heart conditions apart, like part statuses, so as part of these depart renegade, the cubelet has been changed to use patch to update part status. It will ignore, like the fields that it does not belong to, so it does not own like cubelet owns certain fields and it leaves the other fields to other components.

A

So that means you can use patch to update a single condition in a pod, and this will actually cause no conflicts and also there will be no race condition, since the fields are like separated, separately owned. One part of the pasture is owned by cubelet and then something can own. The part, readiness gate condition.

A

You can even do it in a custom controller and then that automatically reflects the condition and, as part of the power ratings gate change, cube carro has been like a new field has been added, it's called radius gate, so we decided not to change the existing ready column, because so many tools- and so many people are relying on it and cube. Caudill get pods, it's sort of like the mostly executed cube, color command. So we don't want it to change it.

A

If, let's say many, people are adopting readiness power in this case, so we can revisit this decision. So you can see, there's a separate column and, like the ready column, ready column means that how many containers are ready for the pod and the radius K column means that how many reflects how many various gates are stat is true.

A

Okay, so let me talk about the kubernetes engine. Use cases for part readiness gate, so we use it extensively for the container native low balancing feature we have so on. The left is how a typical cloud load balancer integrated with kubernetes, so the load balancer does not no pods low bouncer, like basically targets the cluster nodes and then typically the low balancer points to the service, no ports and the no port type service, because they expose a consistent port on all the cluster nodes.

A

Any traffic flows to that no port will be like randomly so like forwarded to one of the backend pods. So on the left. The typical case is that there's a secondary load balancing happening on the nodes and further delivered the the traffic to the back-end pods. So this setup actually has many downside. It's actually worth a complete talk to describe the problems, so we're not going to do it here.

A

So probably next time on, the right is what we call container native low balancing, so the Google cloud load balancer can actually target arbitrary endpoints, so it has a new construct called NetWare endpoint groups. So you just need to put the endpoints into this construct and low bouncer can directly send traffic to the pods. So it has many. It has many advantages, so obviously, there's no second hop, so pods become first-class endpoints and like and also the cloud load.

A

Balancer features just works like, for instance, traffic, shifting, cookie affinity and things like that like, for instance, the traffic shifting right, if you are doing the secondary low balancing, if you shift traffic on mount vm's, it doesn't translate to shifting traffic among the backend parts, because any vm were just randomly for traffic to one of the back-end parts.

A

So for the with this container native low bouncing, you can actually actually shift traffic among the pods and also it can evenly distribute the load of accurately because consider if, let's say if the low bouncer is targeting the VMS, so the low bouncer is basically doing traffic accounting per VM. You think the lowness of things that oh I'm perfectly balanced and I'm gonna send traffic to my back-end in a balanced way.

A

Well, actually, because of the again because of the secondary low balancing happened on the node, it messes up that accounting and then it becomes irrelevant and with this direct like targeting yeah, so it's better balanced. However, it's not without his challenges, so the classic setup just need it's like for an endpoint update in the classic setup. We just need to update the IP tables and update IP tables it pretty fast right from what we what I just mentioned.

A

It's like less than five seconds, but when you're talking about like programming, an external low bouncer, it's a different story. Yet there are many flavors or low grounds that they are regional. There are like four for Google Cloud. It provides a global low bouncer. That means a single pot, a single endpoint change.

A

It needs to program a globally distributed infrastructure like on all the Google edges, and it's definitely higher than like five seconds so for the extreme case, since the pods can start out very fast, depending on your container image, some image can start at like less than a second and it starts and it become ready. Then the deployments say I can just proceed and then, like within seconds, your backends can finish rollout right within seconds, but but the low bouncer is not actually completed and completed of programming for new endpoints.

A

So in that case your own endpoints are all gone and your new endpoints are not programmed. That would, in the extreme case that it will cause an outage so how we solve it. So here we're gonna, walk through a rolling update case. So on the left. A blue replica set is the v1 replica version of the application and the green one is the v2. So at first everything is stable. The low browser is targeting the old version.

A

Then let's say the rolling update starts right. The new part gets started and from the pot it becomes part of it becomes ready. The containers becomes ready to serve, but but the the low bounce is still not programmed to target a new pod and the old part gets removed in this case in this transition period, there's a slight reduce of the capacity because it used to be three pods serving traffic.

A

Now it's only two and however kubernetes deployment will not proceed because part is not ready because there's an extra readiness gate to reflect whether the infrastructure is ready. So the the red new escape will wait till the Google cloud load, balancer to say: hey, I'm, ready, I'm ready for this new endpoint. What is this ready? The readiness gate will condition will reflect that and kubernetes workload and the cloud network infrastructure will work together. Seamlessly you can just use. The native kubernetes were close to manage this.

A

The cloud infrastructure with all that I wanted to pass the mic to my co-speaker ping. She's going to talk about use cases add into it.

C

First, thanks me want to invite me to present how, for most rabbits, pod readiness gate.

C

And let me first briefly introduce what is the foremast, and why is the foremast for most is intuitive, sponsor open source crowd native platform for ability to monitor the application health in steady state of operations running on kubernetes it lavish matrix locks tracings as a barricade signal.

C

It also used a I based anomaly detection and can perform auto limitation based on once. The nominee is detected.

C

So now, let's talk about poverty, miss gateways formats, the use case. So today, I'd like to like I, introduce two of use case. We can live it. The first one is continuously health check. So, for must not only can monitor the application, health but also pot health, based on the dodecyl signal.

C

Once for mustard detector, a particular part is unhealth our AB nominee, so we leveraged pod readiness gate feature to set par the readiness conditioning to force. So in this way the pod can take that particular part can take out of traffic. Then people can debug and/or troubleshooting it. So the second thing at the second use case is of order force alert.

C

So in the past we found was right after pod, ready and pod start to serve the traffic at the very beginning of first one to two thrashing appears have highlighted, sea or sometimes even learn to like a.

C

Error timeout, that is because of application, means some of applications component. Some of the component may still not fully initialized. All some depends on some external dependency, so we introduced the pod readiness condition features, so we first we checked.

C

Readiness resource signal is if it's ready and then set for the readiness condition to chew and then start to serve the traffic.

C

So today, I'd like to have a demo of the second use case. This is a brief demo fro, so I mark the step, 1 2, 3, 4 5, so first step is right.

C

When we started the month, the one application is deployed and the second step is we start to deploy v2 to repress v1 and now v2 is the v2 is deployed and a pod pod is ready and then we add additional.

C

Readiness resource check for v2 and then set up B to pot the readiness condition to choose between three and a floor. Pod start to serve the traffic 4 to 5. Is we found application? Have a high error spark so the light of orange sorry? The line is indicated. We have the machine learning model. This is the upper bound which is generated based on historical data.

C

So in number 5 is 4, must detect anomaly and issued autoroute look limitation, which is a low back in this case. Number 6 is v1 is a replaced the place to be b2, so the one application is there. Like a deployed, you know in this case, let me have a real time demo, for it.

C

Yeah step 1, just like I mentioned under.

B

C

Flow, so application is the demo version is a v1.

C

C

Now I just this is just the price is greater. We were hot name and a skill step. One I just give you more information v2, it's not there step two. We start to deploy v1 by replace deployment of e 1 by V 2. So, ok, ok, so you can see the department is replaced.

C

Just indicate application is now verging, is application is demo? It is already.

C

Now let me get spray, the part.

C

The veto I search, the father, readiness.

C

So the second graph is the readiness resource check. So now the readiness condition is set to 2. You can see one of world.

C

We won got a terminator here.

C

Now applications start to serve the traffic.

C

Yeah, you can see yes and also we we have the aromatic skin.

C

So here, because error appears hi so for must indicate application years in health.

C

So our limitation is Auto robot. So now, let's display.

C

What is the department version is robot to be one.

C

Here's some more information here is the v1 pod and then here just spray me. It's like there's a no, no video part anymore, yeah.

C

Top of graph is like v2 is replaced by v1 was an anomaly.

C

Yeah: here's the foremast e for mysterious open-source project welcome to join the forum. Ask.

A

Cool so that concludes our presentation, so yeah.

B

Question I think we're technical at a time, but they're, probably here for a couple minutes afterwards. The questions appear again. Please thank you for coming and remember to rate the the session on your app. Thank you guys very much.