Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2017 (Austin), 15 Dec 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Monitoring Kubernetes: Follow the Data [B] - Ilan Rabinovitch, Datadog

Description

Monitoring Kubernetes: Follow the Data [B] - Ilan Rabinovitch, Datadog

At Datadog we help thousands of organizations monitor their infrastructure and applications. In this session, we’ll dive deeper into the several hundred trillion data points we’ve gathered to extract information about the real-world use of Kubernetes and see trends in container and orchestrator usage.

As we look at Kubernetes and container usage data, we’ll also discuss the top applications being used in orchestrated environments and, using the data, provide insight into which metrics you should watch and how to troubleshoot based on those metrics. In this session, we will also look at a framework for your metrics and how to use it to find solutions to the issues that come up.

We will cover the three types of monitoring data; what to collect; what should trigger an alert (avoiding an alert storm and pager fatigue); and how to follow the resources to find the root causes of problems.

Although the real-world Kubernetes and container use data is derived from Datadog users, the focus of this session is not tool specific, so attendees will leave with strategies and frameworks they can implement in their container-based environments today regardless of the platforms and tools they use.

About Ilan Rabinovitch
Ilan Rabinovitch leads the community and product teams at Datadog. He spends his days diving into container monitoring metrics, collaborating with Datadog's open-source community and evangelizing observability best practices.

Previously, Ilan spent a number of years leading infrastructure and reliability engineering teams at organizations such as Ooyala and Edmunds.com. He’s active in the open source and DevOps communities, where he is a co-organizer of events such as SCALE, Texas Linux Fest, and multiple DevOpsDays events.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

So the top it's the this topic of adoption of Technology is something I'm quite interested in so I've been working with our data science teams, the last couple years on various technical reports about how various technologies are being adopted and being used by our customers at data dog. Today we're going to dive into the most recent study that was published this morning.

A

It focuses that focuses on orchestration, kubernetes and ECS to be specific, but but orchestration in general, but before I get to the study, I think it's I think it's important to often it offers context about Who I am and what I do and why any of the data I give you today should matter and why you should trust anything I, say or not.

A

So we'll take a quick detour just to talk about who data dog is, and you know who I am and and where the heck I got all this information I'm about to dump on you. So how many folks are familiar with data dog, probably stop by our booth down in the sponsor showcase? Hopefully anybody using it today in your environments for monitoring, kubernetes containers. Anything else cool awesome, so that's a lot of hands.

A

It's always encouraging to see that it makes me feel good because it means that my works actually getting used somewhere, which is awesome so for those of you that didn't raise your hand, we're you know, we're monitoring, we're SAS, based monitoring service, we're collecting metrics, logs and traces from thousands of organizations around the world. Our goal is to help you make sense of all your application infrastructure and business data as you're trying to make decisions about your environments, whether it's are they up or are they down do I need to scale it out.

A

You know was: was this event I'm running a success? Maybe you may be folks who are doing how many folks were on call for Black Friday this past week, cool so there's folks in the room that were that were that, were you know, wiping sweat from their brow, as people checked out on online shopping carts right, that's the kind of stuff that we help monitor.

A

Now my role I work on the product and open-source teams at data dog and most of my time split between two things. The first is ensuring that we're good open-source citizens, so that might be showing up the things here like koukaon or we joined the CNC F on Monday, making sure that, when we're participating in things like open tracing that we're, you know we're doing so we're doing so in a transparent and genuine manner, making sure that pull requests from our community get adopted. Things get get get merged on time.

A

Things like that, but the other half of my time, I spend working with our customers in our community and learning and identifying the technologies that they're using and trying to figure out how best to help them gain visibility into it. So lately that's been working mostly with containers and orchestrators some cloud providers, but it feels like every day, I'm spending, my time, thinking about docker or kubernetes, or something something similar, and so when I say we try to figure out all the things that our customers are. Monitoring.

A

I really mean everything: it's everything from caches to containers, to queues to cloud providers and everything else in between. We really want to help. You collect everything, because we think if you have that all in one place it's much more effective as a tool than if you're looking at five different systems across different across across different monitoring systems or different technologies.

A

But you know so, of course we're here at you're here at cloud native cons. So I'll talk about the things that you know, at least from the CN CF ecosystem, that we're were that we either use or or adopt or monitor. We have a logging product that we launched we launched about a month ago.

A

Then it integrates with fluent D quite a bit, so we love loud fluent D. We also can monitor fluent D from our from our from our monitoring solution. Kubernetes we've been been playing with kubernetes and dr and container d since, since pre 104, all of them and then again on Monday, we launched our open tracing solution for go and Python and other languages are coming very soon.

A

So we're adding more things all the time. It's about 250 things out of the box that we monitor today. That's you know, which is just quite comprehensive, but you're not here for a day to dog pitch you're here for kubernetes adoption data, so we'll keep going so, but you know, as we get as we're gonna get ready to dive into the study.

A

I want to do a couple, quick, trivia questions, see what folks see see what your, what your thoughts are and what's being adopted, and then we'll see if you're right as we dive through the study.

A

So just again, another show of hands are gonna, be raising your hands a lot today, it's gonna be like being in being in Beck, being Ben grade school apologize in advance, so by show hands. Let's I'm kind of curious, which of these technologies you all think, is the what is most likely to be used in a kubernetes cluster that we're monitoring. So the first is Postgres I think that one at a single hand in the house, my sequel, okay, you could but then I, then it's hard Apache, all the hands go up for Apache.

A

Okay, we'll see we'll see we'll see if you're right, let's, let's we'll, keep going so again, as I mentioned, we're always looking to better understand what our what technologies our customers are using and the trends that the industry is taking and we want to be able to help, monitor them, be able to tie in to them make sure our customers can be successful with them. So how do I decide on what to focus next turns out?

A

There's a couple signals: I can pretty reliably count on as I'm trying to figure out the maturity or adoption of a given technology. The first and most important one is always hacker news right, I, just top a search in there, look at the numbers and, if they're high, clearly that's what I should be writing my applications in and you're likely. Writing yours in right. Just follow the latest trend. It's called hacker news driven development.

A

It's very successful in Silicon, Valley companies I've been through a few of them that take this approach. I, don't know about you. All anybody have co-workers that do this. Nobody wants to raise their hand for that. One I promise: the camera is not pointing at you, it's pointing at me, they'll never know, but if we look at the numbers, I mean it turns out. Containers are pretty awesome, like the numbers are high up there they're higher than most things.

A

I search for you know whether it's, whether it's docker, ECS or kubernetes, turns out according to this I should be packaging all my stuff in docker and running it on kubernetes. Who knew then again it turns out that rust in uber are cooler, and so apparently, I need to pivot and stop worrying about containers, start worrying about infrastructure and just build like a really rusty. Ride-Sharing service and I'm gonna be a millionaire so but anyways we'll keep going so again we're at cloud native con.

A

Another thing I could do is I could pull the audience at events like this, whether it be at meetups or something like that like we did at the start, but I guess that's semi-scientific to figure out if things matter, because I see that you all paid to be here and it's interesting to you and I can tell that it's important, but there's a bit of selection bias. I can't tell if it's I mean you may be early adopters.

A

You all may say that you're using kubernetes or using docker, and then you go back home and you're. You know you're you're still running some still running things on Prem and VMs or whatever it might be. So you know polls are not not the perfect way. Another thing I can do is look at the age of these technologies and kind of dive in, and so you know it's it's important to remember that almost everything we've talked about today is quite young and it's moving pretty quickly right.

A

If we look at this I mean Linux was released in in 94 right. Let's, let's, let's look at that as an example of a technology and how quickly it was adopted. I know this is a bit of an eye chart, but Linux was released in 94 and it took about what seven years before, but one day was in 94. It was released earlier than that, but took about seven years before we got some serious business adoption. This is the first time.

A

2001 was the first time that that IBM jumped jumped in and said we're gonna put a billion dollars into Linux. Anybody. Remember the commercials with a little kid. What will teach him will raise him? He'll be awesome. His name is Linux or something like that right. So that's that took about seven years. I took a while for people to really get on the open source and on the Linux bandwagon.

A

But here we're at coop con and I go to expo floor and you know obviously I think data dogs awesome, but you wander around and you see huge organizations that you see IBM, you see Microsoft, you see Cisco, you see Intel they're, all jumping in with both feet on this containers thing in this kubernetes thing, so it seems like we're moving a little bit more quickly.

A

I, don't know if it's you know our love for the cloud or the need for dynamic infrastructure now or what but like it's weird that, after only three to four years, like everybody is talking about this and and really as we'll see in the study, a lot of people are actually using it as well.

A

For us, our first signal, the real one is you know the real thing is when customers ask for it, so in 2014 we got our first pull request. Doctor hadn't even hit 100 yet, and some customer was cared about this enough to actually write a doctor integration for data dog and start collecting metrics. It was like well if they want to do it, I guess if they wanted enough to write it right, the integration forth and clearly it's important to somebody and so we'd open on it. We started partnering with them.

A

We took the data dog agent shoved it in a container. At this point, we've collaborated with our community. We've got. You know over I think over 5,000 contributors at this point on our various open-source projects and what what built a containerized agent, you can drop it on kubernetes or ECS, or plain old, doc or anywhere else that you might want and it it collects metrics and monitors your applications and does all the right things. So for us, that's important much like everybody else, we're also adopting containers we've been using them in CI forever.

A

We've been using ECS for about as long we're he's starting to use kubernetes in other places. Now, so so lots of clock and slots of containers everywhere at day to dog. To but again all this is really anecdotal. You might ask where's the data we are. Data. Dog data is in our name if I didn't show up with data.

A

You'd, probably lock me out of the room, so we kind of have to go back and fact-checked all this I can't just kind of do this poll go back to my boss and say: let's drop all that everything else from the roadmap. It's all kubernetes, moving toward so I start to look at other data right. I can look at look at baby, pulls from docker hub every year at docker. Con doctor talks about how many images have been pulled from their service in 2014.

A

It was about a million it took about over in in over a year it kind of it creased significantly from a million to a billion. 2016 was six billion 26:17 they just announced when I was in Copenhagen was 24 billion, pull requests and that sorry pull image pulls and that's just from public docker images. We're not talking about all the private registries that you all might be using, or you know no Red Hat has a new container or image service that they offer.

A

You know all the other places that it might be so clearly containers are taking off for us, though we realized we have real data and so remember we're monitoring infrastructure all over the world for thousands of organizations and all comes back to our hosted service, and so about a few years back. We started.

A

We started looking at that data in a fairly rolled up and anonymous fashion, right we're not digging into any individual persons data, but we look at it in aggregate, and we realized that we can tell from these workloads from these trillions of data points per day. Once we aggregate all that data, we can see technology trends in the tech world around us in you know pretty accurately and pretty quickly.

A

We can see as things shift, so we started as putting up these reports that we put the first one out in in 2015, and we looked at the first year of dr. since 100 and Dovan to docker adoption, and we saw some pretty interesting stats. We refresh that annually now so every year at docker call we put this out, but we're a coupe con. So we'll keep going so, but what did we see in the first?

A

In the last report we saw things like containers being up 40% in our customer base over over the previous over the previous year. So and that's you know, had been 5x up the year before that. So it's you know, that's that's clearly some good data. These are folks that are paying us to monitor docker. Clearly, that's some production use case. You don't tend to pay somebody to monitor things. You don't care about.

A

We then looked at it again. We saw things like 15% of the hosts we monitored at the time we're running we're running some amount of doctor on them. Nowadays, it's actually closer to 35%. If you look at organizations over 500 hosts you're, pretty much guaranteed to be running some amount of docker, but if you're over 500 hosts about 35% of you are running running some amount of docker. So that's you know, that's growing again, that's that's real data there, let's dive into orchestrators, though right.

A

That's what we're here for we're here at coop, con and cloud native con, so we raised hands before, for you know, for dr. orchestrators in general. Anybody just like still running just docker, like your doctor running everywhere, you got ansible or something else. Kicking off doctor runs for you a couple you, so we were curious about the same right. We were solved this docker adoption. We wanted to see. How is how our folks, using it and again this is from about about six eight months ago we dug into it.

A

We found that about 40% of organizations at the time we're running some form of organization if they ran docker. This is as of as of April and when we'd open and looked at even further. One of the things that we found is that if you were over 25 hosts running docker, you've got your probability right. You moved up to about 50 percent of the users running something once you get a hundred at 65 and beyond that, you're, basically guaranteed to be running. Some sort of Orchestrator turns out when you're running at scale.

A

You don't like SSH in a for loop or whatever crazy contraption you rube, goldberg machine you put together, is for like launching containers and rescheduling them. You actually want something to do this for you, so another interesting stat about 5% of you all are apparently running two or more Orchestrator technologies.

A

Who's got this Frankenstein environment out there raise your hand couple you what's what's with a migration story,.

B

A

Cool yeah, you guys have a really cool case study with with Google up on the Google site about how are you using GK? So if folks are looking for how how this guy here is doing all kinds, awesome stuff with kubernetes it's up on the Google site, thank you, so yeah I think usually I hear about people that it's migration stories other times it's it's folks, like you know where there's were acquisitions or or mergers or just large organizations where there's different teams and it's I can't possibly use what you're using right.

A

So it's a little bit of everything, something this. This dysfunction does come with us, even as we move into the new generation of technologies. But again, let's talk about the new data. Where I've been telling you all about this old data. That's you know it's! It's! It's I, don't know how old April's not that long ago, but but it's it's old compared to the stuff I released three hours ago. So so, what's new today released any report again, it was mostly focused on kubernetes and UCS. We see lots of other orchestrators as well.

A

We see nomad. We see meso, so we see swarm, we see other things, but really I. Think kubernetes and ECS were the ones we've had the most conversations about.

A

So we wanted to dive into those specifically, and so the first thing is: there's some pretty good news for kubernetes in terms of adoption in the first nine months of the year so again, I had you know this date is up through October through October September had to go back and put together, but to again through, as of October 2017, about 41% of all docker environments that we monitor have are using kubernetes to orchestrate them right.

A

That's quite high, again, a minute ago, we just talked about only 40% of orchestrators, that of our customers that were using docker, even use even use orchestrators. That was that that was from last year right now, we're saying over 41% of them are actually using kubernetes. It turns out that these days it's about 2/3 of organizations are using some some amount of some amount, some number of orchestrators. You know one or more, that's huge growth.

A

The other thing to keep in mind is that the number of docker users that we haven't we saw that we saw that growth curve right. The docker was on that's not slowed down, that's at the same pace. So if you talk about percentage of like 30 percent here versus 40 percent, here that's more than 11 percent gain. That's a huge gain! That's a lot of people jumping into kubernetes right! So I, don't think you all are alone. As you looked around the room and you're like who's, got your hands up.

A

You all have your hands up because everybody's running kubernetes, but the real question is: how does this impact? You know your let's? How does this impact? What we do right, Kelsey we talked to kill Kelsey, was on yesterday onstage giving a keynote, and he reminded us all. Nobody cares about your kubernetes cluster. They care about the business services that you offer they care about the products you deliver to your customers. I mean you care about your kubernetes service, but at the end of the day you know keep it up.

A

Move on go back to building product right, keep it boring. He said so. I think you know, the question is are is is kubernetes and his doctor helping us do that?

A

Well, we found we found some interesting stats around with this, what it did to host churn and can change container turn. Usually we see hosts that we monitor live about 23 days if they don't involve any amount of containers, but once we introduce containers it drops to 17 days, we once we had orchestrators like ECS or kubernetes. They vary a little bit by a couple days, but it's about ten days on average at a ECS or kubernetes hosts will be around four and I.

A

Think that's telling it shows that kubernetes and UCS make it pretty easy to drain workloads off your machines. Kill those machines bring up new machines. Have those workloads show up somewhere else? You don't really feel bad about about. You know about terminating those hosts and bringing back up- and you know this is this- is this is above and beyond what folks were already doing with VMs, where they've already we're, probably using auto scaling groups, probably using some other dynamic method for provisioning their systems. So I think that's!

A

That's that that's one sign that it's helping us move faster right, but not surprisingly, it also helps our containers be more dynamic. Remember that that SSH for loop around docker run it's pretty slow. It makes it hard to make changes quickly, right, folks, that are doing Bluegreen deployments. That kind of stuff are not doing that.

A

So you know once you once you start diving to something like kubernetes. What we're seeing it's, it's it's! It's just an assumption right that you're gonna be able to move faster. We scheduled and faster deploy faster, so one interesting, stat I thought we found here was that container joining on kubernetes live only about one and a half days on average ECS.

A

On the other hand, which is that I think the next most popular container Orchestrator, we see lasts about 12 days on average for a container, so I, don't know what that I, don't know what you all think that means about you, kubernetes versus ECS, workloads, I, don't have to I, don't have to point at Jah.

A

You know kubernetes jobs or the cron schedule, or other things that are bringing pods up for very short periods of time which ECS doesn't have, or if it's that people are just using ECS as like an artifact deployer, not necessarily like as a real container Orchestrator. It's interesting I'm curious to see what all the Fargate stuff that they announced at the keynote earlier this week with with Adrienne Coe Croft it'll change. Things too, especially since I won't be able to see the hosts anymore. It's just containers right, but it'll be it'll, be interesting.

A

So you know the next thing that we've all talked again. We talked about speed when we lock talked about containers. We talked about orchestrators, but the other thing we talked about is is density and efficiency right. How many folks are doing this for bin packing reasons, like you really want really efficient, 100% utilized all your infrastructure? You don't want to leave one cent on the table right. One. A couple people raised their hands right we want to.

A

We want to stop moving things like this, where we've got a lot of empty space around we're kind of shoving things between boxes, trying to make use of those workloads and be a lot more efficient, a lot more standardized right. So when we did our doctor report, we found that folks were running about seven containers per per host. So I'm curious is folks on average. Are you running more or less more couple more couple less.

A

So what do you think kubernetes does to this? Do you think it's going to give us more containers or less containers per host, so you're right, it's it's more! So when we delve into kubernetes, we were seeing, we see closer to nine application containers per host and what do I mean by application? Containers. Well, I mean your workloads. So when I looked at this I thought I, don't really care. I mean you're not really deploying Kubb dns for the sake of deploying coop DNS. That's like a utility.

A

That's a system container for you, you're, not deploying all of the kubernetes system. You know control, plane stuff as your own containers. That's part of the orchestration system, so we de head and grep doll that out exclude that from a report, and so that's only nine, if I put all that back in its like 14 to 15 containers per host, but most of that again is kubernetes itself.

A

So it feels a little weird to give kubernetes credit for just existing right like if you could have a completely idle kubernetes cluster and you'll likely have like five containers running on that box. It's like I can't really count that right, so we're excluding all that just because it turns out Google does use a lot of containers itself, which is cool UCS.

A

On the other hand, tends only have three containers on average, if I throw the fourth container in there it's gonna be ECS agent, I, just I gripped that I pulled that one out as well, because again, for the same reason it you know nobody's nobody's intentionally paying to run the yes agent, just like nobody's intentionally paying to run the you know their Cooper needs in for the the coumadin's control plane. So I, don't know really what that says.

A

I mean that's 3x, more density, again, I think it's it's sort of telling about the types of workloads. People do on ECS versus Cooper needs, but it need to dive into it more. So if you have your own thoughts, I'd love to love your hypothesis, so I can go. I can go dive into it and prove it with data.

A

Cool, so let's go back to that trivia question: let's play a little game of family feud. What would this serve it? You know we had before folks were saying what we said: Apache was the most popular most popular technology. That's running in of the three that I mentioned running in kubernetes. Is there?

A

Is there any other things you think are almost guaranteed to be running there, something that you're sure will be on this list when I, when I flip the cards, nginx, okay, anything else, message queues, any specific message: queues grab it: okay, redness, okay, cool, so turns out. It's things like nginx, so I, don't if I had socks or something I tossed it at whoever said nginx, but I, don't so come by the booth later, maybe I think we probably have some shirts for you.

A

Rabbitmq was on there elasticsearch MongoDB postgrads, my sequel, right they're, all in there who said Apache before it was all of you right patch is not even on the list. It doesn't even follow the top 15 things that I find in container environments. I, don't know why that is I spent. So many years, automating Apache to playing Apache running Apache I think I could on a whiteboard here right. It's like tactically, correct, Apache, config file for you and it's tilt and nobody's using it in container environments and I.

A

Don't I I, don't know why I mean maybe it's you know, maybe just a little heavier weight. I, don't know what I do find really interesting is the number of databases that are on there right, like how many you all are doing. Stateful workloads and your kubernetes clusters couple hands, but it's not. You know, generally folks, I run into folks the mass majority like if I asked you how many of you guys are running stateless services on your career days clusters.

A

All the heads are gonna go up right, so it's just interesting the number of when we look at this number of folks that are doing it. So to give you a sense of the ranking in kubernetes at least you know, nginx, of course, by far I think pretty much. Everybody is using it as a reverse proxy here, but you know the others are well within within each other's range. In terms of in terms of the data stores, there's quite a few of them, what do you think it looks like across other orchestrators?

A

Do you think it's about the same across things that are running containers or things that are running UCS?

A

It's the same technologies again. The difference, though, is that what I see in ECS is that most people are running their own custom-built images, they're, not building things off the shelf, they're things that they've built from their building they're tulsi their own applications. I. Think that's because when you run in something like Google or are you running something like like ECS with with with with with Amazon, they already give you all the services?

A

Why would you run a register at a server if they, if they like, they already give you a reliable Redis server that you just you pay less for and don't have to worry about operating like you? Just you take advantage of that whole ecosystem, so I think that's the real reason why on ECS, everything's smaller the stacks on docker, it's again it's about the same as just again the scales are slightly different, but it's about the same.

A

You know I started to look at also at container registries this morning. I wish it was actually interesting. It turns out that if you're in Google you're using the Google container registry, if you're in, if you're an Amazon using the Amazon container registry- and then all of you are using some amount of docker hub, because it's the thing that everybody uses, but if we talk about the best practices around some of that, how many how many folks are doing that are pulling latest from like docker hub all the time?

A

No none of you, nobody! Nobody I see some people going sometimes yeah.

A

Hey, that's what our Doc's say right, so you better be doing that yeah haven't you guys are mirroring how many you all are mirroring things back into your own registries or into private registries. I, see three hands four hands. Okay! So not a lot of folks. Do you think most people are? Do you think most people are doing about the same as you or better or worse somewhere in the middle.

A

Yeah, so so it turns out it's it's. It's a pretty healthy mix, I mean it's a scary mix, but it's a pretty healthy mix right about about 75.

A

74 percent of you all are doing doing a mix of good things and bad things about about 10 percent of you all they're doing only the good things which is like mirroring your containers or grabbing specific versions and not letting things not not letting me break you when I release a new thing sort of like I mean how many did we not learn our lessons like who didn't lock their bundle file their their.

A

You know their gem files or their requirements files and like in Python or Ruby back in the day and like have everything blow up on them on the next deployment. Did we not learn anything in the last ten years, anyways so about that? 16 percent of you are doing like the right again doing the right thing and only playing the only sorry, the wrong thing and only deploying latest which I guess that's. You know those are those those others, those Cowboys. We always hear about right. So yeah.

A

That's you know, these are that's that's the facts. I got for you today, I think I've got a couple minutes left here to answer any questions and dive in anything. I maybe didn't cover. If there's some metrics I didn't report on or something I was supposed to talk about that I said in the abstract and didn't cover here today you can you can quiz me on it.

A

Grow me on it, throw some tomatoes, whatever it might be, feel free to tweet that data dog or myself I saw you all taking pictures of the graphs there they're all online, so just I'll tweet them out in a second they'll, be linked here as well. It's up on the day to dog website. You can check it out. We also have it down in the booth if you want to check it out on the big screen.

A

I've got a bunch of resources here in the slides about how to you know how to monitor kubernetes, how to monitor doctor how to how to work in the cloud whole bunch. Other things there. The original doctor report I'll put these up on the slides for the for the coupon site in just a few minutes. Oh yeah. If there's any questions happy to answer them, we've got about I. Think about eight minutes left so I finished just in time. For that, no all right, it's been fun school's out.