Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2019 (Shanghai), 5 Jul 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Auto Operation of Hundreds of K8s Clusters in Yahoo!

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Auto Operation of Hundreds of K8s Clusters in Yahoo! Japan - Cheng Chen & Hiroki Katsuta, Yahoo Japan Corporation

Multi-tenancy in Kubernetes is actively being developed. However, single-tenancy is still the most practical way to ensure security and workload isolation between organizations. Yahoo! Japan is the biggest internet company in Japan, which develops and maintains hundreds of applications. Currently, we are running the most Kubernetes clusters in Japan. Different service teams, such as the payment system and e-commerce system, are using our clusters in order to serve 130 million Japanese users. In this session, we will present the motivation and procedures of building Kubernetes-as-a-Service in an on-premise environment, includes: ・Learning from our experience in production, such as upgrading clusters with zero downtime. ・Extend Kubernetes using features such as CRDs and Webhook ・Experience of managing hundreds of clusters in an on-premise environment with a handful of members

https://sched.co/NrnM

A

Okay, konnichiwa Nihao and hello, everyone I'm here, okay and here's Jan. Thank you for coming. This is soon we are going to talk about how to manage 200 courses in Europe and cooperation in the system, our presentation in English and he transferred its to Chinese.

B

Water from can walk through donation can see Fabio donated $10 convince' from the Yahoo ribbons. It's unable to open issue by the teacher that legal culture only harden we're a young human Fabio know your wall. I get. I define, come well yeah open the hangar. No one can deny that you have.

A

This is agenda. First, we will explain about Yahoo, Japan cooperation. Then we will introduce kubernetes obvious, which is a mechanism to manage many clusters and contents. We will explain how to manage clusters. The system's main features and planting new functions last beer is summarized is a.

B

Textual know who eat a cow socio generate a digester Yahoo Rupa once the kraaho well generated a thi woman Suzie, yes, father kubernetes service that you got Randy iguana teaching the teacher woman as she don't John Woo Haqqani teacher kakashi do not Hoshi own hotel at the Tiwi. The whole woman generations here.

A

My neighbor here Kosta I'm, aside to deliberate the engineer in Yahoo Japan corporations, governance across the management team and I've studied about colleges for two years and I got CK on CKD salmon.

B

You all, okay, that is our teruyama honking, whatever doctrine ham, wakfu, have a yahoo driven most of the kubernetes second edition. Nobuta and Agassi. Are you the true, a hydrangea kubernetes sanguinetti Rihanna rashida allah Tatiana are showing wahoo controller sec, hey Jose, hey I did yeah.

A

And it's my Patronus chan chan has my teacher about kubernetes and Chinese. He was a panco person. He was Yahoo Japan questions engineer but now only hope, Japan. He is reading Japan's alternative text, Oh.

B

Something wrong food or power: no, no, what well, she sure sure hot eh I'll wash it had a kubernetes, go to Manila sure Johanna. What.

C

B

Is Rikuo you been yahudah ego-driven that you are young room, y'all wanna! Thank you! Parent accords air. How is our show phone with? You are thorough, set up water, Yahoo yeah that I didn't take her? Oh you know. Actually they go chief about you go.

A

Okay about the Japan compilation which, upon cooperation as developing so many services, for example, such search engine accion shopping email map, epics PP PP is Japanese. Payments is premium service, similar to a leap a and which a pay so Yahoo. Japan question is the most be used in Internet company in Japan.

B

Okayama rebirth Yahoo, ribbon red ribbon, he wanna Sandoval or earlier Yahoo, so so yeah. My my yahoo, yahoo, yoshi yahoo densetsu help a parent engineer, foo happy, not sure, but not true. For us. Yeah taro says how on the toolbar hi ovation, tofu Thai parent I destroyed the entire body. Yahoo Laguna Honda Honda want handy.

A

There are so many service development teams. It is very inefficient to manage and operate infrastructures in each subsys. Your Japan question has crossed our management team. Like us, Yahoo.

B

Revenue hundred Akhenaten we're pleased it hundred awkwardly who you rated from 0 to 2 feet on your shoulder in societal Rubin. To finish our warm and dry under generate hand way.

A

Yes, we are across the management team and our team manager. It's almost every comment is clusters in Yahoo Japan Corporation, and not only managing. We also support how to use kubernetes, so we usually hold lessons support development and connect clusters to in-house systems.

B

Raekwon 801 the pnina city on quani Yacouba, not in control of kubernetes teacher Putin Jaquan. Do you have a college teacher woman who never could undo it? He won't coronated the shoe young, the pound rule. So any vomiting country, bank or chorionic kubernetes dog, Suzi Quatro make half a ton. Do it she won't be out. The truth hurts here: woohoo john kubernetes. Did you can come soon, a good, a condo? She turns into ETA.

B

D

Is our stems mission.

A

Of course, to provide stable infrasound as the top priority and we need to reduce management cost because we have any other tasks. So automatic management as needed and infrastructure needs hi a velvety and they required to continue to support the latest version to remove binary kiss.

B

Hundred alone, Donna human dignity torment on duty to our Tijuana hi onto the conveyor, went home. So he won't. You show unit, humble sis, don't wanna! Do piano, jump ephedrine, tonalá, her ta g, jr ratio, cocoa! You all khruschev and honorable mentions yahoo co to on agenda load on Honduran TV on. She don't want.

A

You noted those missions and to support Yahoo Japan cooperation services. We systemic rosters management, so kubernetes obvious is developed. It has already been running in which open question for about two years, Yahoo.

B

Ribbon woven dimension or machine part of tradition, quality community service at Agora, generating to Martha thanks a lot tae-young, even the another Yahoo River they go gee Tony. We got National.

A

Theatre, the main required features for the system: first: easy to manage kubernetes clusters.

A

It is manage, grab manage these clusters as its most important and, of course, some of the machines go down so to reduce management, cost automatic manager automatic machine recover as needed.

A

We design to one one service in one cluster, because yahoo, japan corporation, has so many services and they require various infrastructures. This design is so secure, come.

B

On sss service almond, rosetta, Katya, ter, photography, quality, teacher, gotcha de nada, the Johanna Johanna teacher God didn't come with you to come talk, see where our church, our universe, humble include the idea to be a phenomenal sousou-dono Yahoo de baño. They don't know that fool or Jamaica fools who put on daddy's home, oh sure sure. So you know I put on that hip and we turn to need a combination. Teacher.

A

And we talked about the movement of community service, but first as let's look back on the features of just kubernetes, you know: kubernetes manages pots, terrific, set and deployment governmentís new for these resources and operate how many and which version required to depart.

B

Tonometry cerkupo necessary, slow conversion, ha Oceania Haram cancer, Kunis's toshio-kun assistance region to tomorrow, pops Idaho deployment er just teaching one. If Hoda occupant, one document is Teresa with Hangul digits, well, Cooper national pension with her grandfather, the bamboo or Julia.

A

Official incorporations community, the service, has same government of given ideas, but management target like machine has a lot of custom list of definitions but, for example, to manage machines. Machine said, and machine deployment defined and custom controlled, controlled by custom controllers, Yahoo.

B

Driven the coroner says: serratus Sangala good cause, you know, I'm sure, no woman, no quantity on PO the pendulum, our machine. Surely you were trained, the people want what they need that concrete. She controversial much inside home, arson deployment.

A

In Japan cooperation, kubernetes clusters are running or water machines created by OpenStack so to monitor. The Soviets compares the virtual machine states and custom resources, specs such as most in said, and most in deployment and custom controls, create request for OpenStack API Tahu.

B

Liberty tornado warning, OpenStack or transient idea being wrong. Cooper nice addition is generosity them to sha, tsui kubernetes, the servicer congress. He who put on the phone ovince take a API without dig and drown. Her commercial inside commercial deployment at redound own city needs new entries and a highly.

A

Then come and risk as a service create one cluster. This is a cluster configuration only one solve it is deployed. He deployed to here and custom associates about machines, are divided into load, balancer, master, ingress, ad city and a nose. We are calling as worker and the automatically linked and created by a custom controllers. Kubernetes.

B

A service to Tron Django teacher Turkey got additional achievement rodeo, so they know who jump orphanage home for that young Chi master in choirs. It is 80 Oh Walker was it Tiana, go furniture service conditions only Indira.

A

There are two world branches for master and ingress operate account access to master water burn, sir. It's for internal access and service users, access to Internet and increase rules of answer, so it's 4x sauna. Access to.

B

The audio for that year, master who inquires to show you later. You burn from go master of footage in Hong chief, our API server, Chicana teacher, so it hurts to a neighborhood. Are you go for that? Johan, no ha food, that another shrooms owner shows your tongue whoa, you guys the further Johan chief Alan kubernetes attrition of war, so you know Harvey would provide up for that. Johan T.

A

We explained the main features first, one of the main objective as reducing of management cost, even if money, even if machines go down now, if we automatically detect and recover it, and it's easy to update its imported, apply new person quickly like foreigner with this. This is achieved with no downtime.

A

Besides, it is convenient to coupon it's as a service. Our users to, for example, users, can scale clusters with only one command.

B

Tommy Lasorda Satoshi on our show Gianna. You got ego to me auditorium so gentle, yet humble hundred on sensational come this year. We can treat open session fatigue of end here. Ariana Tom, Bianchi song ji-hyo, you go. You went here, yo yo, you go cha-cha-cha sofa, Johanna kakazu. Indeed, our egos in the bombuh Vanessa service, Nikita dafu, many authenticity on Dragon, Con, dong, xiang, ji to Tirana, cobranet service, annihilation, road, Hugh de la fórmula de la mia, beautiful new version decisions, engine Jota, ah ho dress, really humming NGO koala and.

A

Provided kubernetes is no customized, so users can use pure kubernetes and no customizing realize quick provision for users. India Japan cooperation, as if it's a new version, including fix of high risk vulnerability, may be applied to all clustered within 24 hours. Long of shell problem.

A

After this, we talk about above two features with figures moment.

B

Here on the kubernetes abandoned ping-mei of a shoka, so young who to adore the traditional kubernetes teacher locally, requested human core, warm entire otaku ribbon tito naval history, simple ohio, co-ed own shampoo, do namitha under our social name. If I you know, create, even which consonant with a solar conduit women, geography that there are who maintains you intruder? What hosting Leticia.

A

Expression of automatic recovery equipment, estas obvious, regulated check the cluster status by polling, request to open stock API and compare them and custom resources.

B

For kubernetes service in Oakland war overseas, like the API food origin, II Artesia, not wrong, hi pinky. What's the transition be zero.

A

No, this worker goes down thank communities, less obvious, detect difference between customer suspects and Muslim status by the polling check.

B

You go: Canadian nurses, Allah Cooper, mrs. Saracen atomic control; zero, her to go TG that one had at high.

A

After the detection, companies is a service request to create a new worker to the open stock API. Second,.

B

Read out hi Johanna coronation service, something war, open, Sega, API, twenty English India.

A

And OpenStack create a new worker and list of learning machines becomes similar to the customary sources Oh.

B

Psychotron English in the general Horner Shruti Shah contra la traigo, TCG tantrum, has to enter the enter and Ronald chunk fool well.

A

And the last equipment is a service, deletes, don't machine, and this recovery is done as.

B

Well now, caresses services, always shunt, will make or she's out the idea, no bundle.

A

Next, we will introduce how to update government digital service. Has these custom resources that cross the boss room and machine deployment and those are linked? Custom controls make pulling a check of the resources and compare it to Lowell associates.

B

Jerusalem church, how is our Google Xiang ji? Kubernetes services, don't volunteer in the Sun question. She some concrete jungle, Bhutan the hotels here, the entranc ha Peter 300 had a put home and I take the tuning on a woman's yeah so um be sure they wake up. Related videos he's on the panda.

A

In these few years, those resources are set. As one point sorry and machines is running at sole.

B

A

Now we have showed these clusters updating 2001 point forty first, this crisis up later updates custom resource about cross diversion to 1.40 operator only do this after all of the operating process after this population is run by automatically so.

B

Now I'm John, yeah sure she's confronted a leader is the Family Court huh Shoshanna totally new engine companies, Department is a leader is the true holds your daughter, Mara holy. You mentioned other channels, long-haul conversation with the domain controller from chicawa.

A

Okay and commented service is custom, controller will detect cross subversion plating, then updating a plate. Machine deployment to 1.40 after this machine's will be updated by rolling plate because the Machine deployments Peck and his machine status different.

B

Components has serviced the country, t-tommy country, dog, bun, bun, Xiao, Zhang, Jie Jie jump machine, the department that you consider it our East bomba yo, you can trade RTG permaculture, the embalmer put home Jake with teaching the dog, don't consider isnumber.

A

First, new version was acquitted and all all the machine is still running.

B

Vanessa service lock, Oceania Shimbun, bandage EDM, patrons yeah parents or the to do to that idea that unison don't go.

A

After the new machine is grated, it's wrong. One auto machine is drained by this. One ink pot in the machine finished no money and they are redeployed to another machine.

B

Image idea: true Hana petronia Hall shan't happen my oven T to the G idea. Not television, big orange hot wok Tillett joins hold on. Don't worry soon come to God GT sit on the powder, no dissipation, gonna polish I. Don't have it right now in the didn't jump in bed wrong since you don't.

A

After the read the point, the machine will be quest. Please shut down and repeat these tasks. I push.

B

One from Jolla to go to the theater: she don't have a son who they are. No gentle I'll take a concrete one sheet: the undulation, thoughtful and.

A

New machine is grated. She.

B

Not dissipate round yeah the.

A

One old machine is drained to.

B

The GC, no patrician, gig or Trejo to total and.

A

Machine is dated Roger.

B

Didn't be thankful.

A

Grated, oh yeah.

B

A

Dated now the crossbar plating is done. Thankful.

B

To hollered teaching the family from dinner to the Omaha.

A

By this role by this snoring updated update with no downtime its revised in the case of load, balancer or etcd, they are almost the same token.

B

Tina physicians, a jump away enough ooh reason: $10 to go down this generation, emotional, a turn-on, a lot implants or IEDs 18 are the function notions in.

A

Yes, and by using such kubernetes obvious, we so reduced the cost of cross the management. Now our clusters are uses by 100 services. There are 400 clusters and there are four thousand machines commotion.

B

Openssl service in a woman, Jenny Tara tardy on don't even wanna trauma syndrome, quani that chronicity channel TV Parkinson a fool Jordanna a poke was the baggage issue such a hot idea.

A

Cross is so many, but our team is very small. There are only about ten members. Of course our job is not only management. We provide many useful add-ons like Prometheus a lot Michelle dog, urgent and so on, and we also take our cross burner wiki's and updates them also.

B

To teach in our cilantro, you got sure and a shorthand in some way. Ahora para on the Congo put in general kubernetes, the teacher woman Shania with Hugo premier sources are not many agenda. Ohio logo shows the agenda. You unit come to achieve only Yahoo or Chihuahua June hands and her teacher. The angel dog paint a chauffeured hammer.

A

Okay, when we we talk about planting new functions, a lot more useful, we're planning, support, state rule, realized, I, observe a return and high flexibility. No spec and director burrito is setting around load balancer, also in the right figure. It's a method to return response to years, a directory without too low the bouncer. It's reduced a lot of load or answer and improve the last one speed. Oh.

B

No, no no I'm, novella de Tijuana warranty heart refers. The default state for I will take a fun one change. Your true color change the idea she do not a quarantine show of how foreign teams, who prefer sure you go for that. You don't see the Caixa to mood you bein to consume, usually yeah you could whom were to go for that you home she found food. Johanna turns FL, which ago the show cannot you for being routine war, that you honky Troiano click on that renunciation. For that you consider bigger food item.

B

Walk-In said he got how to do and.

A

About more secure, we are planning to realize automatic check of docker images, kubernetes audit, vlogs and container logs. We will use, for example, photo.

B

For me, aa woman, co-wash Regents, so don't want II know it's in something: huh hi audio kubernetes that hold our intruder: hi old record own children or shorty. What Mia Java won't cook higher and higher, and the fellow cut out wonderful.

A

Okay, this is summary we are managing over 400 clusters. They are easy to update, macabre automatically and be kept secure. The realized by Yahoo Japan corporations kubernetes as a service to.

B

Hong da Jie woman granular chose the back cover us room to work. One electro was the bakkoto kubernetes disha are secreted shangela Shante, AHA changer for gigantic equivalences our territory, Yahoo Rubin, the crisis service, the war.

A

Thank you very much.

A

Is there any questions.

E

Welcome that tilma numenera yeah what we all question Emily on the future. What city should saw the open stacked? What was an insurer open, cycling and dig a magnum sure, Magnum, the PTL woman, sweden, yet so allergic to share only upgrader who take auto-feeding Lilliana future coin. Iam come at you delegates. They don't show for or share yo yo yo yo holla city abilityto, you're running for mg, holding over psych the API to Gen turn unity, idea, written down down I'm taken to Shu, Shu eager. You go physical, that or you go early cool down.

E

They were during needed a go to go to the Shuji idiom koala for bahala na jaana. Yet so there you are male college for your file in the kool-aid. Quad doesn't need in the city pension either dig host national kinda nice, a night ecosystems on.

E

Cardi I'm working for OpenStack Magnum project I'm, the PDF of the project. Recently we introduced the auto healing feature as well, but, as you presentation, I think you just missed some details when you detect the failure of the the cluster. For example, you just mentioned the community inside the service just.

A

E

Api to detect the failure, but for me it's most like just the hardware failure. It's the word machine down. But how do you detect the failure? Freedom when.

A

B

Go to her Patti tells us. She's will be what how you got TG tamil archita coupon expansion also be anticipation that you know house check everyday is Johnny. Don't do sure. Are you sure, woman Suzanne? That's what we talked. My cell research shows a potential to be impatient that you go. Oh yeah.

B

Oh yeah, which is not any good, so don't know they go how to check this.

E

B

Tracy, do you you.

A

B

For Tito Tommy, a mutiny, okay,.

D

About how we check the governess- and it's not close.

C

Don't know everything.

B

B

C

Will be out watching as our email question she don't companies sold on Johar push. What role do you know a John Doe you, eighty? She, oh no ma ginger thing she told Todd, I gotta tell you, though, I'm killing machine harsher Lankan yeah. Neither yeah and I go nation. You know it is not easy enemy, yo, yo, yo, CR teach you thank you I mean you know, I teach it to each other. I wanted some. What you teach the other.

B

Emo emo say lu8, it's a no. This was all do Saturday with an ax in this car on Omaha Shawano, which you know Jim 9er. No, no, no Tim, Allen, not oniy, Arnolfini she's, a Shannon I know she's foliage they're, not honest about his hobbies or take a.

B

Now, what we do anyway, Dava our social channels done come on. You know when he was at Tony's yeah to her. Until you are happy with how they go. You may see unless you don't seem to do the back over the smooch allies. Were she imparted hope you miss. You confirm you lost son at home, yeah, you sure, George or to go to consider. Giada woman teaching is humble in gradual, take crisis, the carrot and a note Anjana higher is horizontal returned onto a coupon expansion unit and we're all touch with Shana is Homer sure.

B

To semicolon come here, so the do. How can you allow this is home to a Jewish town who would say it's Honda CR it on the hood at yours, I didn't question who target on? Do it now much a good handing out happy waiting on a CT with our yahoo pension that they go haven't a year? Would on Jen. You know she wanted magenta the neutron eager diversity at my palm, who got hundreds walk dr. wahrman, tornado, fanariotov, Cugat hundred Emanuel you'd Susie, the deposit omega-3 fat.

B

We go to Ankara near the Kohanga REO analyzed equivalent to the treasure. You know: hotel I, called polite to chakra furnace room from for a woman to woman teacher named with the community here will appear on around her trio system, Cooper system college. She react. We saw their home and party. She quit the jog-a-thon control trunk or even-tempered. Digger she'll see how they go. Pal iPad awarded you enable the mushroom kubernetes, a new heart for us, gender, dysphoria, I. Suppose yeah you should have water go.

B

Don't you don't report a community to show Hungary about you say: oh the yeah. We're gonna cooperate. Suspension seem to each year what your tissue you'd show, how that I mean you got school function, Oh, Oh, granny,.

F

I have a question for the customer upgrading when you drink one node and get to the add to the new version of node. Some workload may not work functionally because you have a version gap and if you have project are incompatible with the new version. So I think there was an important point in the upgrading is that you must do some pre check and post check for workload when you no upgrading.

F

So could you just describe what the post chart and pre-check you do when you upgrading so to make sure that the workload work functionally on the new version of node.

A

We automatically operate all machines, that's not too bad, no needed machines down.

A

If there are no need to update, we didn't mind out all machines update.

A

Do you, man, I.

F

Don't get that clearly, could you repeat another way so.

A

What's that your question so.

F

I want to say is that the version upgrading is operation with a risk so because the version is not very compatible with some rock load, because API object may various and some old, if you object, may not work functionally on the new version of node I. Don't know if you have income starter case, so I think the workload evicted and we start on the new node is a progress that need to be checked and ensured that the new node and the workload on it is work correctly. So to have some progress and operation for that.

B

What cannot you know once I hold it on? You know an inertia. The insurance are 2.0 by making the whole operation zeda quadrant on the return date change. A lot of your kubernetes take off hop emergency hot hot pan band returned away with general.

F

Check you it you walk loaded, she made sure I choose when you put them go and you you know, you couldn't push the near, handle the Tommy, Mottola or.

B

Cilantro to understand what the ego sure to show San Diego copilot at the company that generates Mitchell I J go to Canyon on the general doctrine, each other. It whole resources to go to do. Farhana, warmish, real Maharashtra it back up smooth under humans, the baggage issue in Georgia woman. Can you go food hungry? Aren't you gonna stand out sue decision now, Jade someone doubts about you in some pain, be actually legal deployment position. You should take a different view on yeah yeah yeah. Now yeah, oh, it took hundreds of sugar yeah.

B

The did you interpret you gonna pre-production to your head up. If we do, the more sure he's done on our tradition, you're free to go you'll download data center I'll take our data center from being a production teacher, okay, Oh Brock, with a monument.

F

A

Time it's over.

B

A

B