Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2017 (Austin), 15 Dec 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Red Hat: Making Containers Boring (again) - Clayton Coleman

Description

Keynote: Red Hat: Making Containers Boring (again) - Clayton Coleman, Architect, Kubernetes and OpenShift, Red Hat

By ensuring everything about containers is standardized and boring, we can now focus on the overall Kubernetes experience when it comes to actually running containers. Freeing Kubernetes to just focus on orchestrating containers from now on and setting the stage for exponential growth. We'll take a brief look at how Kubernetes is prepared to explode in usage because the foundation has been solidified. From container standards to customer-resource definitions to pluggable hardware, Kubernetes is ready for broad usage patterns.

About Clayton Coleman
Clayton is architect and engineer on cloud orchestration and
containers at Red Hat, in charge of both technical direction for
Kubernetes and OpenShift (Red Hat's platform as a service built on top of Kubernetes) as well as the broader container and container content efforts at Red Hat. Clayton is one of the top contributors to both Kubernetes and OpenShift and has been involved in many projects in the container, platform-as-a-service, and ci/cd ecosystem over the last four years. He enjoys sleeping, but rarely has time to do it anymore.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

So I want to talk about why kubernetes should be boring and how Red Hat is helping make that happen. So I don't mean this kind of boring, even though this is roughly connected. So this is about infrastructure. Kubernetes is infrastructure, infrastructure helps us build things, but the goal of infrastructure actually isn't infrastructure itself. It's what we build with it.

A

It's how our users and our applications work better with it, and so, if I'm, a user who is using a service or using an application exciting to me means ok, you got a new feature as a developer. Exciting means building my application, but if I'm running an application for a user, exciting usually means something a little bit different. It means everything is on fire again.

A

So, in my opinion, if we want happy users, kubernetes needs to be boring, so everybody above the stack, can rely on us. Red Hat helps build boring software. You know, there's not a open sources and always an exciting thing. People have to chop wood and carry water, but the end result is that outcome for everybody that's better than if we all worked alone. So, rather than get all high-level, I'm gonna get really concrete.

A

Red Hat runs hosted kubernetes because you should never ship what you don't run yourself, and so one of those variants is open shift online starter, which is free kubernetes for everyone. It's a multi-tenant hosted kubernetes.

A

These are some of the largest and densest kubernetes clusters in the world, if not the largest, dense, the largest kubernetes clusters and, of course, when you have very large, dense free kubernetes clusters, they're very exciting, so one of the things I want to go over kind of what we've done to help make kubernetes a little bit less exciting in this respect. So events are a very important part of kubernetes. They communicate when something in the system isn't working. It's actually a bread crumb for a user to follow to know why their application isn't working.

A

But when you have lots of applications, odds are most of them are well they're, not most of them, depending on how good you are as a programmer. Some of them are broken all of the time, and so in kubernetes we were sending too many events. We found this because we love prometheus and we use Prometheus from the open-source community to make kubernetes better.

A

This graph is showing you all of the incoming API requests to kubernetes over a period of time that giant green bar is events coming from the applications in the cluster that just aren't working, and we also found and fixed a number of bugs in kubernetes.

A

They make metrics better at communicating back to end-users, so the fix was actually pretty obvious, which was you don't need to send the same event 32,000 times you just keep it on the client-side, build them all up and send a update every now and then hey your applications still not working, and this also set the stage for longer-term changes that we're making to kubernetes to make your Bernays scale better to make events better with other people in the community.

A

As these clusters got bigger and bigger, though we started to hit, we started hit the limits in the system, so the example for us is secrets, because we have a very dense multi tenant cluster. Everybody has their own secrets, those started to add up. One morning we woke up and realized that every time one of the server's started up, it was fetching 800 megabytes of secret data. 800 megabytes of anything is pretty bad and we were doing it pretty frequently. So this trickled through the system, there was a number of bugs found and fixed.

A

We set a long term arc for what we were going to do to make kubernetes better. So the next person who hits this is going to benefit from features that we've put into kubernetes that make it work at higher and higher scales. But we also got to pay down an investment. That's been building for years, so kubernetes has been built on LCD since the very beginning, and in the very early days we worked with the NCD community and team to set a direction for EDD CD.

A

That would meet where kubernetes would be three years from now and for us running in these very large, dense clusters. We realized that the time was right, but it's a very complex process, because you need to do a migration. Migration is critically important and we should never ever as developers give our end-users something that we haven't tested ourselves and surprisingly enough giant.

A

Complex, dense clusters are a really great way to test migration, and so we helped triage fix and identify issues and move those out into the community so that everybody can benefit from it and the end result for us was huge. We drive memory use on our large clusters down to a third of what it was before: CPU dropped significantly and API latency dropped as well, so unfortunately as well, when you have lots and lots of applications, you tend to hit almost every edge case that could ever exist in any bit of software.

A

All of these applications fighting against each other cleaning it claiming for resources, people coming and going walking away those edge cases we found and fixed. This is one particular example of we enabled cron jobs to early and an enterprising user turned it on what ended up, creating about 40, creating and deleting about 40 pods, a second and so through Prometheus and through the other operational tools that we developed our careers. We're able to spot this go and make those fixes possible, so I could go on right.

A

We've done a lot of work in kubernetes to make it better, but I actually would say it's really much more important that everyone here recognizes that we as a community need to make kubernetes boring to make it boring, predictable and reliable in open source, because at the end of the day, the only thing that matters is our users. So please come and be boring with us. Thank you.