Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Don’t Catch Feelings, Catch Issues With Kuberhealthy - Joshulyne Park & Shilla Saebi, Comcast

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Don’t Catch Feelings, Catch Issues With Kuberhealthy - Joshulyne Park & Shilla Saebi, Comcast

Kuberhealthy is a synthetic monitoring operator for both apps and Kubernetes clusters. Learn how to increase application and cluster observability by replicating real workflow and carefully checking for the expected behavior to occur. With Kuberhealthy, our team has been able to reliably monitor all critical Kubernetes cluster functionality in order to catch issues before our developers do. With Kuberhealthy, you can write your own tests of any kind in your own container and Kuberhealthy will manage everything else, including the creation of Prometheus metrics. As we’ve transitioned more and more cloud workloads to elastic, self-healing Kubernetes clusters, the job of keeping the clusters running smoothly has become more challenging and important. That’s why we’re so excited to share Kuberhealthy, a new open-source tool we built at Comcast to keep our Kubernetes clusters running at their best.

https://sched.co/UaXs

A

Welcome to the next session before we start I want to make a few announcement, don't forget to rate the presentation on sketch dot-com afterwards, since this is the last session before lunch. Don't forget that lunch will also be served on this upper levels here, behind the showcase and on the terrace not only downstairs, and the third thing is: please check the room changes due to the power outage and with that I think we're ready to for the presentation by Sheila, savvy and Jocelyn.

A

Park Sheila is working at Comcast and Jocelyn at NBC Universal, and we will hear a presentation, don't catch feelings, catches, fears with cuber healthy, welcome, Sheila and Jocelyn Thank.

B

You Federico welcome everyone. Thank you for coming to our talk. Don't catch feelings, catch issues with Kuber healthy, and since we recently, we recently developed the new version of cooper, healthy 2.0. We decided that this title might be more relevant, so cooper, healthy and operator for synthetic monitoring on kubernetes. My name is Sheila SEBI, and this is Jocelyn.

C

B

We both work at Comcast I work as an open source program manager, so.

C

I'm a software engineer actually not at NBC but at Comcast ecology, solutions. Alright,.

B

So I'll quickly go over the agenda. This talk is you're, mostly going to be hearing from Josh I'm gonna. Give you a little bit of a little bit of information on the open source program office at Comcast, and what we're even doing here at this conference also Josh will be telling you a story and we will talk about uber, healthy 2.0, which was recently released and then we'll do a live demo and then how to contribute to coober healthy. So, let's talk about our open source journey.

B

I won't go through every single one of these points, but 2006 is a date that we know that we were doing open source here at Comcast. I went around with my colleagues and talked to figure out who who's been doing open source. What's the latest date that we can get back to and 2006 was the date that we that we found was the most common, and we know that at that point we were consumers of open source 2012.

B

We launched our first open source projects on Comcast, I'm, sorry, github.com, /, Comcast and then back fast forward to 2013 the open source advisory council was formed. This is a group that consists of folks from our legal team, as well as some engineers across different organizations at Comcast and the open source program office.

B

This group decides ultimately what becomes open source and what does it and I will say that the acceptance rate is 99% in the last two years so and that's how our team is related to the Cooper, healthy team and other engineers across Comcast. They come to us and say: hey. We want to open-source this cool thing. We say most of the time yes and Cooper. Healthy was one of those and they came in around 2018 is when Cooper healthy was open source.

B

So, over back over to 2019 you'll see that we have several open source projects now we're showing up to these events. We're sharing our story, we're also open sourcing things. We have more strategy around our open source. We also did two events. Last year we did one in Denver and one in Philadelphia open source events that were open to the public, so open source conferences.

B

If you will and a little bit more about the open source program office, which is a team that I work on I started at Comcast back in 2010 I was an operations engineer on the exchange team. Back in 2015 I got my first I got my hands into open source, so I'm, actually pretty fresh to the open source community as well.

B

I started, contributing back, fell in love with the open source community and then I started working on the open source program office, which opens its doors in 2017 you'll start seeing that a lot of companies they have open source program offices. We're part of a group called the to do group, which is top openly do openly with other AHS foes that are there. It's pretty cool our teams, it's not over 9000 technologists. We have over 200 repos on github we're a six-person team.

B

Most of us are program managers, and we also have a software engineer on our team that helps us with our tooling automation, etc. And if you want to find out more on what other projects we're doing with open source, you can always visit comcastic github do and now I'm gonna pass the mic over back to Josh, and she will talk to you about coober, healthy back to.

C

You Josh hi everyone, Thank You Sheila, for that great introduction in Comcast earning an open-source, so like I mentioned before my name is Josh and park. People call me Josh and I'm a software engineer on the kubernetes team at Comcast technology solutions. Comcast technology solutions is a b2b organization that provides tools, software and platforms for other video streaming companies we're an office located in Seattle and there's a few of us here as well.

C

So now I'm going to get to the Cooper healthy side of the talk and I'm, hoping that during my portion of the talk that I can accomplish. Two things first, is to inform you guys more about what Cooper healthy is as a project and a lot of the development and work we've been doing over the past few months and second, hopefully get some of you guys who are communities, operators or developers interested in implementing Cooper healthy in your own Cooper, dense clusters, or possibly contributing to our open source project so story time.

C

I'm gonna show you guys a little story about Cooper, healthy and our story begins with two characters. We have our cube operator Jake and we have our application developer. Kim, two very common characters in our industry today, and some of you here may relate with one or the other more so Jake.

C

Our cube operator is a huge fan of kubernetes I mean it's the latest and greatest it's the shiniest new toy out there and he's super excited to get a bunch of the application developers in his organization to adopt kubernetes and start using it to ship and basically ship their products and services onto kubernetes.

C

Our apt of, however Kim is a little skeptical about kubernetes I mean she's completely new to the technology, not familiar with it at all and honestly she's just wondering like if I deploy my services and products onto it, how can I make sure it's not gonna crash burn and die so Jake, our cube operator in order to reassure our Optive kim, decides to add standard monitors, looking particularly at node health and applying standard metrics just to ensure cluster functionality.

C

Kim is somewhat reassured and says all right. Let's go for it. Kubernetes is the latest and greatest, and she decides to ship her services and products onto kubernetes.

C

Some time passes and more and more actives and Jake's organizations start realizing the benefits of kubernetes very quickly and more and more people start adopting it. I mean, despite the steep learning curve, that a lot of app does are facing they're quickly. Realizing the benefits cubed Ines is providing I mean. None of us would be here if it wasn't the case, so everything's good. So far, however, one day Ken's come kim comes back to us and says my pod is not working.

C

It's stuck in container, creating what's going on so jake decides to take a look at this, and first wonder is: is it a miss configuration error? Did he put a wrong spec? Is there something wrong with it and because usually that's the case right when an app dev comes to keep operators saying my pod isn't working, it's usually some sort of miss configuration error. However Jake realizes that's not the case, he looks at his monitors and they all look good next.

C

He looks up and know that the pod was provisioned on and the node looks fine as well. So Jake looks into this container creating error and he observes the following scenario: when, when a pod requests for a container to get created, cubelet observes this change and communicates with docker docker in turn communicates with the CNI to get an IP allocated for this container. However, something was wrong with the CNI, where was being slower buggy, and it failed to allocate the IP for that container, thus resulting in this pods being stuck in container cage.

C

Sorry fella you.

C

Get a good recording got it. Okay, maybe I should have just put my hair back anyways back to the story, so that is why this pod was stuck in container, creating Jenkins able to resolve this issue.

C

However, the next day another app dev Khanna this pod stuck in pending, then a few days later, another abduction with a pod stuck in terminating Jake, realizes that a lot of these issues that abdabs are facing aren't necessarily miss configuration errors, but something wrong or something or issues related to nodes, something's wrong with the notes and some of the problems that he's of identifying are CNI communication failures, which is the issue that happened with Kim other issues such as docker Cuba restarts for caches disk provisioning failures, hi Iowa times and container one time.

C

There's not just this, but Jake is also observing the following scenario: a bunch of kubernetes system services appearing healthy, yet their underlying pods are constantly restarting or crashing way. Too often, Jake understands that kubernetes is a highly distributed large platform and it's also very self-healing.

C

This presents a scenario where a bunch of the smaller components can go bad, yet go bad undetected, because on the top level, everything appears good Jake wonders if he can actually catch some of these issues that app dads are running into with their pods and services. So Jay comes up with an idea.

C

He thinks why not run synthetic tests against core Cooper nice cluster functionalities to ensure that kubernetes is behaving as expected, and so, instead of having app tubs run into these issues and problems, why not cache these issues before abdabs run into them? So if you guys know this meme, it's a very famous one with Odell Beckham, jr., famous impossible catch, and basically a cube operators want to be like him catch things before app does run into them. So Jay comes up with a bunch of synthetic tests.

C

He believes would test core cluster functionalities such as the daemon set Chuck a daemon set Chuck essentially ensures that all pods can be provisioned on all nodes in the cluster. Not only does this test whether or not pods can be provisioned on all the nodes in the cluster, but it also indirectly checks for node health, making sure that nodes can actually get pods provisioned onto them.

C

He also comes up with a deployment check which is actually really important, because this replicates a real dev users experience using kubernetes, because it ensures that a deployment and service can be created, provisioned and served traffic in a kubernetes cluster. A DNS status check to make sure that there are no failures with DNS, resolving endpoints within and outside of the cluster.

C

So long with these Chuck's, he comes up with a bunch of tracks that he believes would test core cluster kubernetes functionalities and with these Chuck's here, he wonders how he can get notified when some of these things fail.

C

So in order to get notified of this, he decides to create a JSON Status page and the student Status page has a list of all the checks that he's running as well as whether or not the check passed or failed, and if the truck failed a list of errors associated with that failure.

C

Not only does he include this JSON Status page, but he also includes an M 50 B, as well as a Prometheus integration, so that this check failed or pass information essentially turn into metrics themselves that can be monitored and with all of this Cooper healthy is born.

C

So that's how the Cooper healthy project came about and, like Sheila said, this project was open source back in 2018, so Cooper healthy came out, and now Jake gets to be that cool cube operator that catches things before his app devs run into them and AB does are now all happy.

C

Some time passes and other cube operators learn about the Cooper healthy project. I'm Wonder. Can they add more synthetic tests? I mean there are other synthetic tests. I give me Ron I got two Cooper nice cluster. However, with the way Cooper healthy was built, it was hard for other than code maintainer x' to actually get involved and add checks themselves.

C

Not just that, but app does learned about Cooper healthy and wondered since Cooper healthy runs a bunch of synthetic tests and checks in kubernetes. Can they also potentially act as synthetic monitors for their apps and services in kubernetes?

C

With these suggestions, Cooper healthy 2.0 was born and the whole goal of Cooper healthy 2.0 was to make checks extensible so make Cooper healthy, extensible, meaning that we want to give cube operators and app tubs the ability to create, remove and add checks. However, they wanted to so in order to achieve --is. What we first had to do was remove checks embedded into Cooper healthy and essentially create the ability to add external checks.

C

The way we achieve this was by using the keys in custom resources and creating the Cooper healthy check, custom resource, so, instead of having checks embedded into the codebase, we now abstract them out by using these custom resources, because now Cooper healthy check resource contains all information about your check. Not only does it contain all of your check information, but it also contains your check program itself, because your track program, all it is- is a container so with this abstraction, you're able to create and add more checks.

C

However, you want, and now Cooper healthy acts as an operator provisioning managing these checks for you. So let's take a deep dive into what this Cooper healthy check configuration looks like so you see our standard configuration and for this in particular, we're looking at the deployment chat, Cooper healthy resource config file.

C

It has your standard resource, which is a Cooper, healthy chuck. It contains your metadata information, including your check name and, as I said before, we're looking at the deployment chuck. It also specifies which namespace you want to deploy your check into. Then it contains your normal specs, which is the run interval that determines how often you want this check to be run. You also have a time out specification as to how long Destruction run before it times out.

C

You have the ability to add extra adaptations and extra labels and, lastly, we have our pod spec, which essentially looks like any other pots that you'd find in a kubernetes resource. It has all your information on the container, that's being run as well as your image of your check program, and this pot spec, essentially, is what's being provisioned by Cooper healthy and there you have your cage check configuration. So when this check KH check configuration is applied to your cluster Cooper healthy, recognizes this resource automatically and creates and provisions a checker pod.

C

This check pod runs your pots back and runs that program for your check, and so when this check runs, it has to also communicate back to Cooper healthy, whether or not to check, passed or failed, and the way this happens is when this checker pod is provisioned by Cooper. Healthy, Cooper, healthy, automatically injects this cage reporting URL and for us it looks like this. So all you have to do in your program itself is read in this cage.

C

Sporting URL environment variable make a poster cross to it in your check program with a following JSON body of okay, bull of whether or not you're, past track passed or failed, and if your truck failed a list of associated errors with that failure for those who like to write their programs and go, we actually created a check client program for you or packet. A check. A check, client package for you that you can import into your program and all you have to do- is run your check and make one of two API calls.

C

If your check failed, you report failure and if your check passed, you report success. So with the report failure I forgot to mention. Is you have your list of errors that you can also put with that PSA? We are looking for more client libraries in different languages, just like how we add for the go package, and if anyone would like to contribute that that'd be great.

C

So now I'm going to go to the demo section of this talk and what I'm going to illustrate is the deployment check like I mentioned before the deployment check, ensures that deployments and services can be created, provisioned and served traffic within kubernetes cluster. So with that, we should observe the following scenario: when this ka Chuck is applied, Cooper healthy will automatically recognize this cage check, resource and provision that checker pod to run.

C

The Chuck like I, said before this deployment check, creates a deployment and in r-spec in our configuration, if you have it, if you didn't notice, we set the replica set to four, so we should see for deployment pods come up and we should also see a service get created. So once the service is created and ready, we should we're gonna serve a traffic and it's gonna respond back to us with a 200, ok response. Once we get that 200.

C

Okay response: the checker Pasha automatically begin clean up, basically, cleaning up the service and the deployment plots are all created once that cleanup is confirmed, the check officially passes the successful check status gets sent to Cooper healthy and the status then gets reported back to our JSON status page, so I'm gonna run the demo now and hopefully it works all right.

C

C

Okay, there you go, I cannot see that screen, but I.

C

Am I applying? Did I right cute control nice? You guys will have to guide me on this. So cute control off. Where am I on the demo? Folder? No I'm, not.

C

Yes, so I have two terminals: open, did I write cute control on the demo, okay got it so I'm not doing that delete. Sorry, you guys are my eyes. I'm, actually gonna open, canines, their canines, open nice, so you'll seek to Cooper he'll, be pods running and I'm gonna try to switch over to the other terminal.

C

Am I on the other terminal nope keep going, got it great, alright, now I'm going to apply. Thank you. Oh demos.

C

Does I have the deployment demo full yeah mo.

C

C

Okay, let's see what's I'm, going to just hear everything.

C

C

Okay, all right is it still on demo. Okay, thank you. Cube control apply UPS.

C

Yes, all right! That is our deployment configuration that I showed you earlier now. I'm gonna finally apply that. So you see a bunch of resources created to run this check. So with this track, it takes a little bit of time to come up, but we'll be see. Is the deployment the checker deployment pod come up first, once that comes up, we should see for deployment pods where that replica set, which we specified also come up. So we'll wait for that checker Potiphar's, and hopefully it takes a little bit of time.

C

The appointment pod came up and soon we should see for deployment pods also come up as well, so we see it coming up and once it's running it should be ready to stop traffic and once it stops of traffic, you should do with clean up and be true also in terminating the status. So it's coming up. It's running it's ready in a sec, then it goes into terminating cleaning up mode and once it's finished terminating our deployment pod should be completed. Our deployment check it should be completed and there we have it.

C

So that is a demo of deployment Chuck back to my flights, so yeah flowers are damn l.

C

Thank you, and- and thank you so much for being my eyes as I couldn't see the screen. But, as you can see here on this page, not only do we have the deployment check configuration, but we also have a bunch of other tracks that our team has come up with. So as long as you have Cooper healthy 2.0 installed in your kubernetes clusters, you can apply these trucks and get them also running in your clusters.

C

To learn more about our project, make sure you check out our github page comcast la cooper. Healthy compound. Slash also contains a lot of other open source projects from Comcast, and also we have a sock Paige, a sock channel within the Cooper Nerys organization. So please check that out as well.

C

Lastly, so these are some of the main contributors or core contributors and maintainer x', all of this Cooper healthy open-source product, as you can see, they're all cube operators and they're all here today, so make sure to meet with them.

C

Ask some questions and, lastly, we also have stickers Cooper healthy stickers, if you guys are interested so with this talk, I'm really hoping that I was able to accomplish the two things that I set out earlier, which is one to inform you guys more about what Cooper healthy as a project and more the work that we've been doing in the past few months and second, hopefully incentivize you guys to implement Cooper healthy in your own Cooper news clusters or contribute to the project. Thank you.

C

A

We're done for the session. We have stickers here and catch Sheila and Josh for question and answers outside the hall. Thank you.