Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 12 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Surviving From Endless Issues Coming From 7K+ Kubernetes Clusters - Wanhae Lee & Seok-yong Hong

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Surviving From Endless Issues Coming From 7K+ Kubernetes Clusters - Wanhae Lee & Seok-yong Hong, Kakao Corp

Speakers: Seok-yong Hong, Wanhae Lee
Kakao is the 'mobile life platform' company dedicated to renewing daily lives and the leading player in the mobile messenger market in South Korea. As a member of the private Kubernetes as a Service team at Kakao Corp, we have seen an impressive expansion of the service which was 2K clusters with 20K nodes last year to be a 7K+ clusters with 100K+ nodes. With an unprecedented growing number of the clusters in our service, we have faced several problems never met before. One of them is an ever-growing number of on-call issues that are barely manageable with a DevOps team consisting of a small group of developers. In this session, we are going to reveal the secret of how the small team could successfully survive from endless issues generated from 7K+ Kubernetes clusters. We will also illustrate what tools we have made and why we opensource some of them.

A

A

Thank you. Thank you for attending our talk on Surviving from Andrew's issue coming from 7 000 plus kubernetes clusters.

A

First of all, I'm very excited to be here because it's my first time abroad in three years since covid-19 pandemic, my Korean name is but you can call me or just Dennis I'm A Cloud native cell reader at cacao, developing a private cloud.

B

And I'm Juan here, I'm also from Seoul South of South, Korea and I'm, also working as a cloud engineer at cacao, Corporation and first time in North, America and cupcan also for cook corn. As a speaker, hello,.

A

Despite the enormous size of kubernetes clusters, I believe many of us here do not know what company this is. Cacao is the Mobile Life platform companies that serve messenger portal, Bank, Mobility, Commerce web terms and more. We cannot say everything here so if you're curious visit kakaokov.com for more information, one thing in common is that all of them are internet service that need a server.

A

As you can guess, most of the service in cacao are running in kubernetes. Now we started our transport from Apache methods to kubernetes in 2080 and 99 was complete this year when we decide to use kubernetes. There are many concerns about how to provide kubernetes to in-house Developers.

A

The biggest consolation in adapting kubernetes was whether to provide a single Raji cluster as a talent for namespace, or provides separate clusters for each organization or service. We consider various factors such as close to management, resource efficiency, security, freedom of development extra.

A

In conclusion, we decide to provide lots of small clusters because isolation and security guarantees are essential requirements for compliance, so we developed our own private kubernetes service to provide the kubernetes cluster to in-house development developers. You know automated way is called dkos it only take. It only takes three steps to get on new kubernetes clusters, login set cluster names and select region and done, and this newer.

A

We will create kubernetes clusters, not only in cacao's private clouds, but only about also in public clouds such as AWS delivering a kubernetes cluster in automate way, while ensuring High isolation and freedom through self-service has about good news and bad news. Good news is the number of kubernetes clusters grow grows successfully and more than seven thousand plus clusters are operated.

A

Bad news is so deep. Our Encore issue are.

A

We have moved the clouds successfully by delivering automated kubernetes service. This approach has advantage of it easily creating a kubernetes clusters and meeting individual requirements, but also has disadvantage of being too easy to make one and causing various Edge case and the growth of operate operational cost is barely manageable.

C

B

Let's talk about bad things further. First, too many clusters are not being used, developers made their cluster for Production, Service, developing testing or whatever reasons the problem is that they do not delete deleted, even though their purpose is done. It is okay if we have a countless server in a data center with Infinite Space, but sadly we do not.

B

And someone might think that, then why just find them out and delete all of them first, we do not expect our service growth this far, so we do not have used for metadata metadata for this task. Second, determining whether this cluster is in use, programmatically is not on each task.

B

For example, the cluster used using less than 10 percent of CPU could be a cluster not being used, but also could be a production CDN service which just using less CPU and the cluster having no deployment could be unused cluster, but also could be a cluster used by a bunch of custom resources.

B

The point is that we have exceptions for every Criterion, so we have to consider multi multiple factors.

B

The second problem is that we've got anchor issues that are very manageable. This is an example of our uncle request. It basically said good morning. Some of our parts are research. Ports are restarted at 5. 00 am we've got notifications for that, but we don't know why? So can you help us?

B

uh That's the request we received and the reason can be anything starting from a application block to non-linux counter issues, and we have to find out the reason from scratch every time what we did with this half year of 1000 plus Encore data, was conducting a research to be more specific. We could not take research based on ground grounded Theory, including open X-Ray and selective coding techniques.

B

So one thing I want to share from this research is: that is the fact that not all developers do know where about kubernetes, for example. We all know that using the container image tag latest could be caused unexpected version of container being run, but this happened frequently and they asked us why some of their parts are running unexpectedly or using the host pad host path to store some critical data, something like mySQL database and creating new VM to scale up and ask us why their data is gone.

B

All these issues could be fixed just by saying hey, you have potential problem here. You'd better fix it now, but the problem is where we have 7K plus clusters and it's never been a good idea to do it manually. So what we need is someone or something other than us.

B

The rust problem is that we forget what we know after a few years. This graph show how we handle our issues when the issue can be handled by user them themselves. We just show manual or Documents To Handle by themselves.

B

If it is a administrative request such as increasing quota or along public IP, we just allow or disallow and the rest of things are processed automatically.

B

If the thing is known issue we'll label it an instant, if not, we mark it as a problem.

B

When you find the reason for the problem, we write some documents and share it with our list of Encore members, but sharing knowledge or documenting is always a challenging task, so at best after a few years, we forget it and try to find reasons for one's known issues again, even though we have operational tours like Shepherd and monitoring tours. This is not enough for these situations.

B

What we needed was a detection as a code to examine multiple, to examine multiple factors for deleting clusters to let users know what could be a problem with our human intelligence and to find known issues automatically without waste of time.

B

So we made a detect show for detecting kubernetes known issues, which is an extensive problem, detecting command line interface through for Reliable, reliable kubernetes, cluster operations and Rapid problem, detecting made by gold, and thanks for taking photo of this, but I can show the link. Yes, we open source it and you can find it on github.com.

B

And this is how we use the Third.

B

B

Before starting the demo, this is our in-house service, so this demo will show a lot of Korean, but don't worry, open source version of detect does not have single letter of Korean and Earth so yeah.

B

So this is our list of clusters that I can have access and we will use cluster name Scottish code, which is my English name and requesting a new analysis, diagnosis running down. You can see the results of diagnosis.

B

uh As you can see, there is fatal error and warm things and there's a part port with no psyllium endpoints and service, with no Opera report, which is detailed, server yeah and we can see the part restarted frequently, which is 12 000 times oh God, and we have outdated, TRS certifications which is outdated 53 days ago yeah this was detected and that's all it finds the known issues which frequently happened and show say user to act, something or it's for the automation. We can do something automatically that can handle by automatically sorry no worries.

B

B

The tech has a pretty simple internal structure from variety sources like kubernetes, Prometheus, SSH or whatever can access components core collectoral in detect, try to get something like Port manifest deployment, manifest parameters metrics and infrastators and save it in memory in memory, key value, storage in detect and search of components called detector try to find useful insights for us, something like TRS certification will be expired soon, or some reports restarted more than Millions times.

B

Lastly, final report is generated in a form of Json or Yammer for further automation, or can be seen as an interactive form of HTML.

B

While both collector and detector components are extensible, so you can add or remove whatever rural suitable for your environment.

B

Now I show you a demo for creating new rules for detect. Let's say you are in a situation where deploying more than 10 000 ports will kill. You kill your control plane and want to detect to such situation. We are going to show how to add a new detectoral code. No more than thousand ten thousand parts.

B

B

Code: here's the reverse, the Clone, the repository of the tech and, as you can see, there's the directory called cases and detector. What I just said and I make some detector rule called too many parts starting here.

C

Type too many parts.

B

And some Thai painting for PS code.

B

Implements this interface- and you can see the detector detector is consists of two methods which is called get meta and do for doing it and in a game Metal. You can Define the informative things in here. Something like.

C

B

Pause description.

B

C

Than ten thousand part, where.

C

C

C

It happened, it's.

B

Library will be worn.

C

And it if happened,.

C

You have more than.

B

We need something like we need some data from collector core pod list.

B

B

Which is type of pot raised from kubernetes and yeah? That's the all the all the things you need in get meta and to detect. If there is actually the more than ten thousand part you can.

D

B

Tap typing I want to get some pot rest from Context cat.

C

Collector give.

B

B

And you will get published.

C

B

Or not handle this now.

B

And just returned report, which is which is has passed if length of Paris items is less than ten thousand.

C

And doesn't return rare yeah.

B

That's the order all the things that you need to define a new detector and.

C

Always show it right now.

C

Detect this song called.

B

Currently, my laptop it have connected with kubernetes, which has two walking roads and are run this time.

C

Sorry, our own.

B

Export our reports as HTML.

B

Yeah- and this is what he got as a result- reports from detail and as we just defined it, there's a room saying too many parts and current state is everything is normal, but there's no informative anymore. Oh sorry, I forgot something.

C

B

To some default information.

C

Description shop or pause data when polymers top item.

B

Writing report.

C

Yeah there's so.

B

Record too many parts detect nothing and there's a total of 56 Parts in this cluster and if there's more than 10 000 Parts their level will be warning and ask user to delete some unused part immediately. Hopefully, please, whatever.

C

B

Yeah data has pretty simple structure, as you saw just saying: some mirror and some rules to Define. What is the problem you can do such things like connect via connecting each node via SSH and check some Corner parameters or checking the container images from unknown registry or check wound care parts such a thing and, yes, that's all hope.

B

Sorry I miss my scripts, so I just saying something and yes hope this helps kubernetes users to use kubernetes more effectively and thanks for coming and listening.

B

B

Any question here: yeah.

E

Hi thanks thanks for uh really good talking to you guys. um I was wondering uh I took a look at the repo and there are uh maybe half a dozen fifty six or seven ten detection rules or something in there. Are you planning on publishing more detection rules because I'm sure you've got a lot from your operational experience. Yeah.

B

In in-house we have completed more than 50 rules in their spot. Oh repeating the question he said: if the there's currently a six detectoral rule in our public GitHub repository- and he asked me, did you have any plan to expand it? Adding more rules for that, and my answer is yes: in in-house we have more than 50 rules for detectors, but yeah. We don't have time to evaluate whether is this fine to open source it or we have to constraint it.

B

That is the distance problem and, yes, we have planned to add more rules in this year.

D

Hey uh just good question so uh for your scroll scripts uh since they're accessing uh some of those uh kubernet resources. Are they deploy externally outside your cluster and, if so like? How do you go about deploying it.

B

Oh sorry, my English skill is very poor. So can you speak more explicitly? Oh.

D

I, just I was just wondering if your gold scripts are deployed in like outside your cluster or inside your cluster, and if it is they'll play outside like where are you deploying it I guess.

F

So the question is that how you execute this script? It needs to be deployed, or it is this command line.

C

D

Well, I was wondering if it's deploy and if it has.

F

So is: have you deployed this externally or is it just a command line all.

B

We need is Coupe config file to access the Commodities cluster.

B

Oh that's not. You know, for distance.

B

Oh, you can do it in internally and externally.

B

Where we just show the demo, which is out of cluster- and we can- we already deployed this as a container, so you can deploy in your pod and there's example in our GitHub repository, so yeah.

G

uh One of the challenges we have with many clusters is handling upgrades like kubernetes upgrades. How do you manage that with so many clusters.

B

Where we also made API for that and.

B

We already made some live operation solution for this and we share it in Korean, but we are not yet sure we are not yet opened in English. So sorry,.

B

Kiddos, no, we cannot handle it, get off dance. We even cannot use a cluster API, since our cluster has more than 7K plus so.

H

A question here: yeah: how do you run those scripts uh is a necessary that runs the the scripts or it can be the developer. That knows to go to that site and run the scripts. Do you have a website to run those scripts.

B

Okay, somebody has to replace this sorry who.

F

Execute this scripts, is it for developer or Ops, or is it a schedule.

B

Separate those rules so developers deploy their manifest. We just checking if there's the problem, but before the problem we don't know nothing, and that could be your answer.

C

B

I, don't want to wake up at 5am. This is purposeful before asking me just try this. This is its purpose.

I

Hey, uh thank you for the great talk. I have a few questions about the 13 000 customers. So um all those classes created by your developers for developing and testing how many developers do you have like uh using those seven thousand clusters and how many classes do you use actually in Production, Services.

B

One thing I can only say in here is that there's more cluster than our developers.

B

You know, I can say no sorry.

I

B

F

Many I think he said that there are more clusters than the Developers. He cannot disclose the numbers. That's what he's saying.

F

So I have a question so to execute this, uh what kind of permission required for a developer or the operation guy, because you are seriously doing some inside information from the Pod as well as a system, so it require admin access. How the developer can use this, whether they have sufficient access to the cluster or not. Oh.

B

So yes, you're asking me how the police to access the closed.

F

No, no is that what access require for executing this, oh or.

A

Oh, we we have three about three thousand developers in company and oh excess one cluster excess developer about 10 10 to 30.

F

A

F

Sorry, let me repeat my question: is that to execute this is developer, require admin, access or developer will work on its own namespace.

A

Oh, yes, we separate all cluster, so cluster user is a gain old, namespace access.

G

Are you using bare metal or virtual machines for on-premise.

B

We have our virtual machine Solutions based on openstack, okay,.

G

Any bare metal, or just all virtual machines,.

B

Combined, okay.

J

I was like working the other week during that outage uh two weeks ago, with your data center issue, or you guys were down for a couple days or no.

F

Do you have outages, and if do you have outages on your data centers or on clusters.

B

We pretty much not want to say that yeah.

B

Yeah we have fire on the our data events database. Maybe last week, yeah last week.

A

B

We were very busy last week and I'm pretty much sure. I'm still sleep I want some sleep yeah thank.

F

You everyone for coming thanks.