Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 24 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Embracing Upstream Kubernetes in Web Scale Organization - Lei Zhang & Jun Chen

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Embracing Upstream Kubernetes in Web Scale Organization - Lei Zhang, Alibaba & Jun Chen, Ant Financial

How does a web-scale organization meet the demands of performance & scalability without forking Kubernetes while still offering a fully integrated and secure offering? Actually, for any large scale organization or public cloud, we believe the problems we share are the similar: Should I maintain a swarm of small clusters or a single "monster"? What would be the challenge when I scale the cluster to 1k nodes or more? How to fix issues or add features without breaking Kubernetes code? How often the cluster would be upgraded? And how to make engineers think in a “Kubernetes Native” way? In this talk, we will share some experience & lessons we learnt from applying upstream Kubernetes in Alibaba ecosystem. We will list every challenge we faced, and the "upstream native" practice we fixed it. We will share our unique way of upgrading our Kubernetes with keeping pace of upstream as well.

https://sched.co/MPXu

A

Okay, so I think it's a time to start to speak. My name is Lee Jong I come from Alibaba and my co-speaker did not make here, because these are problems that I will give you a solo speak today, and the topic that I would like to share is embracing upstream Canadians in websql organization, which basically are some user cases and lessons we learn from maintaining kubernetes cluster in more than like hangs down ones, nose cluster in Alibaba.

A

So this is a simple out liable to desk topic and I will give you some little bit of talk about over background and how we achieved it, a hard multi-tenancy architecture and how we fix the container headache and, of course, what it comes in a headache and we'll talk about the workload management and why we prefer the predictability. And of course we will talk about scalability in this tent on the nose cluster and the last, but most important. We will talk little bit about how he danced with Reid the upstream okay.

A

So little bit background is that our team in Alibaba actually serves both Alibaba cloud and Alibaba Group. The difference is Alibaba. Group is the e-commerce system in China, which is actually one of the largest e-commerce platform in the world, but Alibaba cloud is actually the biggest Club ladder in Asia and we also provided the manager kuben a service like AWS or Microsoft, either allowing Alibaba Group. This is a little bit different because we basically maintain stable internal kubernetes cluster stand each cluster.

A

We will have more than 5,000 nodes and most of them, under the most one of them, had like tens down the nodes, at least, and we use them to serve this world's largest e-commerce platform, and that is actually the user case. I will talk about in today's talk and all of this technique. Technology technologies, tank under Alibaba Group, is actually based on open source technologies, including kubernetes operator, content, D and one C and card containers.

A

So this is a high-level architecture and, as I mentioned, that we are using a very large cluster to serve all the business unit business unit from Alibaba Group, and that also means this architecture should be a hard multi-tenancy model. But before talking about harmony, tenancy and information that what components we use in Alibaba Group, so most of these components are open source and upstream components.

A

For example, the Eco be API server, it's totally an boundary percent upstream component, but we do not use the default scheduler because we have a lot of customized scheduling, policies and algorithm. We need to implement and also we focus a lot only scalability issue. So we write our own version of the scheduler.

A

We also use Kubus could be controller manager and we use several of the controllers, but we also implement several of our own controllers because we have our own Road policies and I will talk about this more in the later and slides, and the the whole cluster is actually bare metal cluster. But this bare metal cluster, all the physical machines, are manager by Alibaba cloud. Bi-Metal instances is something like on-premise instances from a lobbyist or either and because we have the actually itself, it is a fixed size, cluster resource pool.

A

So we also have the connector which is implemented by were to cooperate. So whenever there are some traffic burst or shefte peak, and then we can actually be we can, we can actually be able to schedule their part to a another resource pool which is managed by Alibaba cloud by virtual machines, and this is actually implemented by a virtual kulit. If you happen to go to my talk during our couplet even know about them.

A

So this is basically the high level architecture and I will go to more details, and the first of all I want to mention is the hard multi-tenancy model, which is a very unique in our your cluster, because we are using a virtual cluster best hard multi-tenancy model, and the key idea of this model is that we hope already for every tenant have a tenant cluster. So it's something like a could be on could be architecture, but the difference is every tenant.

A

Cluster is actually composed by a by a dedicated kubernetes control plan, which is totally upstream components, but we stay where were shown notes. So there's no real notes for lieutenant cluster and the diversion notes are basically if you object, we implemented in the HDD, and this object are virtualized from the real notes by Union small engine. Twitch name is version node engine being engine and it is basically a proxy on top of crew blade, so it will handle the inbound and outbound requests between crew blade and the tenant master.

A

So it's specific basically like her proxy and if we also have virtual couplet in this place in case, we want to schedule pod to other resource poor. That is ideally the same. We will provide a virtual cluster to the f2 every tenant, and by doing this we actually achieved two co-locating, the workloads from every tenant in the same cluster, and we share resources and that's why we do not need to run could be scheduler in tenant cluster because we have a unified scheduler.

A

The unified scheduler will handle every tenant were closed because it is tenant aware and do the fight until the five grand scheduling pest, only conditions of the hawk cluster to achieve high resource utilization and all of this architecture and components will be open sourced during the Kubik on North America of this year, and if you are interested in that, you can just check the upstream talk be proposed in the upstream community. So this is the multi-tenancy. A second thing I want to mention is actually how we migrate applications from Alibaba to container.

A

This is very interesting because Alibaba has been using container for very long time, but I will say they are using container there's an anti container pattern at one of the example is so-called a rich container. So I was very curious. What is rich content even when I first joined the company, because I'm thinking? Okay, why this container the rich then I deep dive into the project? I found it very interesting, because you can basically fight everything in studies container, for example, you can fight. There is the application that you can also further.

A

There are a lot of start and stop scripts inside and you you will then fight something like, as I said she did demon by the log ancient demon. You will fight morning to demon your father there, the cash catch system inside your file, there's the virtual IP client, you guys Clyde your father there, a lot of proxies service service image, engines, everything's, basically in the same container, and it will also follow that is the first process of this container- is not to the application. It's a system D, so uh I just got it.

A

So the rich container means you can fight everything in a container just like the pocket of a dollar a month. So that's why they didn't reach, because it's not because it had a lot of money and I also realized that the reason the people want to do. That is just because they want to make this container act as a VM. So they can microwave applications more easier to this container. But this actually caused a lot of trouble because the developer team becomes used to use this special container.

A

They are using container like the virtual machine, for example, they want to start the container first and then log into the container and then startup application that don't think that container has the same life cycle as their application. That's the first problem, and the second problem is that they just want to write files freely inside a container that don't they have no concept of volume, so they just the right files right that user data in the container at the risk of losing the container some day.

A

So this is really a terrible situation that we want to fix and the way we fix that is using, of course, using concept at a pod. I think you guys might know that pod is basically a group of containers which has very tightly coupled relationship, so in Alibaba a typical part in over clusters like this, it will have an application container, of course, and then has a group of other static hearts.

A

One of the static cars is the live, upgrading static, car, which is used for LA use for live upgrading data or application, which is supported by many java applications. So the live. The live, upgrade container live a quest. Sidecar will copy data from its image to the volume which is shared by the application container. So application container can then load the data from this shared volume and the trigger reload. So we don't need to redeploy application.

A

The other study cars, including the service mission service, mesh-like car, including some assistance, are like cash in as client, also including some monitoring or logging agents that which is classified as the operating side cars so Coelho here is because you know in a politic container, actually share volumes that we can exchange files, do the share the volume and they also share the same namespace. So all the exotic cars container can communicate with each other freely.

A

But another thing I want to mention that as long as you are Union pod, you need to pay attention to five grand container lifecycle management in this application, just at least a very typical application or party or a cluster. So besides the image or the parameters of this application, we prefer that you need developers to defy the post start hook and a priest on hook for your pod.

A

For example, you need to write your start scripts inside of these hooks or even throughout your clean up scripts in the pre stop hook, and besides that, we also require developers to defy clearly what is your live nice probe? That is what is your condition to make sure that your application is healthy, so this is actually how we fix the container headache by using part from kubernetes to replace the rich container stuff.

A

Another thing I may be automation in that. As long as you have a lot, you have so many kinds of static card. You have huge number of static cards. So how can you manage restart cards so we actually implemented a static operator. The static operator is actually, first of all, it has a CRD to collect all the information of the whole static cards in your cluster and a second it provided you with operators that we can defy policies of static hearts, but grant we quantify policies of what cadeau static.

A

Are you want to inject into what kind of applications or whenever static heart is whenever static? Our image is upgraded. You want to make sure you can wrote all the static cards in your cluster, so all of these operations will be handled by this unified operator to handle your all of your study cards containers in your large-scale cluster, so this is static operator and then how about manager work? Clothes in your super scale cluster. Before talking about workload management, we need to make it clear that in kubernetes there are two concept.

A

First of all, the application application in kubernetes is actually a bunch of yamas. There's no such thing urban areas, which name is applications or no, that kind of things. So it just kind of it's just a group of Yama files which can be generated about helm or very customized. But the lesson we learned from here that managers these kind of Yama files in a super scale- cluster- is really the nightmare.

A

I will explain how we do that later and a second concept in the world close the workloads are actually a group of controllers for you to defy how to roll out applications or help to manage your applications. So all the concept like demon, state, job, cron, job staples that are actually or workloads. So that's why we say work. Clothes are actually predefined models of how you do row out how you do instance, recovery, how you do batch, deploy, how you do cannery or blue blue green deployment.

A

But lesson we learned here is okay, so kubernetes a lot of building work loves the the in most cases they are verified and work very well and they are very easy to use. But in some other cases they may not work, and you expected so a talking about the application management. We are talking about the llamo and management. The llamo files management is really difficult in your super skill cluster, but they're.

A

Actually, some tools you can use to solve this problem and the most important part I want to mention in this talk is we actually use get ups with customized to manager all the AMA files in our cluster? So we basically do not revise our motels or 82mm effectively. We use customize to do that and we build a huge kit up spaced pipeline to generate here models from your from oversea our CD system and then apply them to the kubernetes cluster. So this is application management and the second thing is as a mission.

A

The workloads may not meet your requirement in some cases. That's why we created open source project which naming the crews we'll be announced next month and the cruise is actually what we call the kubernetes workloads advanced, it's actually a fleet of customized, the rDNA controllers, which will operate applications and a wave scale.

A

It is totally fully plugged ball because it's basically a group of plugins and it will be a 100% open-source the next month and one of these dumpling two mission in the cruise project is a in play state which is actually our own version of stifle state. The reason we have implicit is because you know largest bill closer, especially when you are running an online business platform. Then the predicate ability is really critical in your system.

A

We don't want to see that whenever we do route we just kill all the part and credit them in most cases. We hope that we can upgrade the applications in place that we serve. We don't need to reschedule apart or regret apart. That's why we really prefer pretty credibility and on the other hands. Actually, we can't use stay full site to do that in most cases, but the problem stay for state is staples that only provide you, this very limited role to policies. That's why we implemented is the in-place state.

A

I just gave you a very, very simple table to compare between the deployment stay for stand in place, a don't talk, it a little to talk to target a lot today, but we can focus on three parts here: the first of all in gameplay state we implement a policy which name is in place that so that means you can row table applications without recreating a part. Another node everything will happen. On the same note, everything will be recovered on the same note, and another strategy is retry on our nodes.

A

So today, if you are using some building controllers, for example, if the node is broken, so application just cannot stop. For summary, just cannot start for some reason. Then this controller will just retry again again on the same node. It will not. It will never try on on another node, so we just add a new policy to in flew in place state so when, if they are timeout for starting over application, this controller automatically recreated the pod on a new node instead of drawing the same node.

A

Another thing is that we added the pre and post update hug. The role of strategy of implicate is that you can just add your own logic in during the road. You know all the policies for your applications. For example, you can say I want to do something after 50 percentage of the applications has been robbed.

A

You can add a script like that, so this is hook so by implementing this kind of customized controller, we actually make it possible, for so many business units on running on top of our cluster can can use can run their applications as it will. So. That's why that's? How we handle the various requirements from over a player users?

A

Ok! So let's talk about scalability, but before talking about scalability of animation, that you need to know that what is your requirement, because whenever you want to deploy, deploy a cluster with kubernetes cluster, more than 5,000 xenos you to think about what is your goal? So that's why at least our requirement here in current kubernetes, our team, the nose limit I mean the classic scale limit is like 5,000 nodes, but the over cluster. They require at least 10 K nodes in one cluster that hit over call. So we require a lot of nodes.

A

That's the first thing! Another thing is we require like 300 down the part in this in this cluster. If you make a calculation well valid, we actually has no such too much requirement for the port number, because every part in over cluster their size is big, so we generally run 20 or 30 parts on over node on every node. So we don't need to handle some cases which have lot of parts in our cluster. So you need to make sure you understand the equipment before you do a scalability improvement.

A

That's why we said a non golf or a cluster is to care about the total number of the containers or parts per node, which is a little bit different. It may be from your case or from your company. Ok, so this our grummond, please remember that we care about large cluster, but not large number.

A

The part and another thing I did not mention here that we have huge number of C, RDS and operators running over cluster, because we need to store a lot of be nice unit on top of our cluster and the basically everybody's and unit will have like 100. Is the artis divide to integrate with our past with our class with our kubernetes cluster, because they may have their own platform and service? They may have their own East Joe. They may have their own service platform, so I read a really.

A

Every platform will use the rd in a controller to integrate with over kubernetes. So, basically, we have huge number of ste, RDS and CRS running in our cluster, which also give us a lot of burden for the scalability part. I will explain this a little bit later, but before you figure out, ok, so after we figure out your requirement- and you want to do some research on your cluster, the first priority is how you can discover the scalability issue in this tank ace in this tank annals cluster.

A

So that's why we actually developed a group, a performance benchmark toolkit, and this there's no magic here, because we are basically using the open source project which named the cubing mark, and it is total open source and the own upstream. We just made a little bit modification to interact to integrate it.

A

It weighs over a cloud infrastructure and it will also implement a HTTP HTTP interface on top of qu, be an auto on top could be mark because we want to make it more easier for us to trigger the scalability performance test, and the most important concept of could be mark is that it will help you to create Holub nodes, which is basically a group of a pause, and in order to do this performance test, we are actually using over production cluster online. To do that, we do not.

A

We do not create a new heist cluster, because we want to make make sure areas. Everything is consistent with the production environment. So before we do the performance taste, we will tend and agendas for for this probe for this performance test, environment and the devil run, the application will run the performance test application by just extending HTTP request like this. We just standard should be request to over cookie mark and it will run up around the test and the typical test cases in our web scale.

A

Cluster is like we care about the startup time during scaling pod, and also we care about the time of creating and deleting part in this cluster, and we care about the depart listing return time and we care about the fare accounts of course.

A

So whenever you have this kind of performance test system, then you can just try to run your test cases again and again to discovery to discover the scalability issues and according to our tests and, of course, based on awkward' acquirements, we found that there are several issues in the current option. Kubernetes cluster first of all, is of course, SED. We actually invested a lot in post as in both HDD, and it's data, bank, bank and storage, which is about TB and most of issues happens in LCD, is actually related to concurrency, locking and datastore.

A

That is a party beam and the stake on this thing we found in this critical path, which is between HDD and ApS server, is that the APS server is actually very sensitive to the data scale, especially when there is large amount of data. The ApS slower will actually suffer from high pressure.

A

It will just reject your request, and this happens a lot when you have a huge number of CRTs or controllers, keep watching people watching or released in your API server or just to keep listing, for example, every node or a pod in your super super skill cluster.

A

Another performance we also be also preserving our cluster is come from the customized controllers, which, as I mentioned almost every business unit or every customer, will actually install the install their own version, customized controlling over in over kubernetes cluster by implementing their own operator, or something like that. So this cutter pressure also drag allow to the performance of our API server with edge steady past. This is actually the critical critical part of the whole cluster, but as I mention before that, we have our own version of scheduler here. So there's no such performance.

A

The issue in our scheduler here, but if you're using, could be scheduled on upstream I, think you may want to pay attention to the QPS of the path scheduling of that project. Maybe I think it's a little bit. Maybe it will be a little bit problem for you. I, don't know the exact number right now. So, as long as we know there is performance issue, so problem is how to fix that, and we can take a look at step by step so first of all, a city park as a mission.

A

The SD part is mostly related to concurrency, loking and data storage problem. So we actually did a lot of work on option to solve the concurrent, for example, concurrence the concurrent to read and to remove a lot of unnecessary logging in the codebase, and you can actually check all of these PR any issues on option. We actually cooperative is a community to do that, but we actually have no any in-house solution. Every contribution has be contributed.

A

Every patch has been contributed back to the option like this like that, and also we noticed the both DB, which is used as the DHT bank and has a lot of performance issue, because this project is the action not very actively actively maintained.

A

It actually partially met maintained by a by Korres right now and, for example, I just listed here: IV fix the bug in the ACD of DB part by replacing instead structure from from at least to a Hashi map which improves the performance like 24 times, comparing to the baseline and after we have cherry-pick all of these fixes on upstream. We also did a performance test again and we found the result very good, so it has resulted like this. We have 100 client to connect with this HDD.

A

We have three nodes: physical nodes, to run the SV cluster. At the end we tested ways: it wins: 1 million, running key value, pairs and retrieved, the makeu of five thousands and the completion. Time of this test is around 200 seconds and the latency most of the latency is less than 100 milliseconds. So this is a very winnable result. If you have already picture picked all of the patches from upstream the latest release of EDD, actually content I think most of the patches, but there are still some words were not included.

A

So if you are interested in that part, you I think I recommend you checking all of these issues and PRS. So this is a CD part and the second part is Kubiak a story. We also cooperate with the community to do that and they are stable, outstanding fixes. I want to share with you guys today. The first of all is listing indexing, and this actually, this page had not been contributed to option, but we will do that very soon by adding index by any index.

A

For least in part, you can actually save a lot of performance improvement in this critical path of your API server between your con, your controllers or operators, and also there is a very interesting issue on option, which name is the Walsh bookmark. It is actually what we are doing with Google recently to implement this feature. The bookmark will enable you to add a bookmark to your walsh api. So whenever you want to restart all your controllers, your controller turn don't need to restore all the watch connections, just as we just doing to that.

A

It just restarted the connections from a from a certain bookmark, because it know that there's nothing changed and before that bookmark. So this is a very important improvement if you want to especially you have a lot of CRT the controllers running sorry or cluster, and also we recommend you to cherry-pick stable of the upstream patches here, and one of the most important is the incremental heartbeat, which is actually something like for now the nodes. So now the notes don't need to report.

A

Harpy hood harpy to API, slower pre-order, claim it just updated, harpy to API server at owned amount, for example, whenever the nose is updated or there's too long certain time. There is no update that this is actually a one of the pro requests which has been merged on AB Jim, and we highly recommend you to cherry-pick this patch because, according to our performance test, the result very good.

A

So after we have chirpy kids or all of these patches into our cluster, we also do performance test again and an old test is like this: we have kinky nodes, we have 100,000 existing parts and we test the the the operation of skill to stun the parts in this cluster and the QBs is more than 130 parts per second and most of the operations actually completed in meetings three seconds. Another problem we actually observed during this performance test is, which is still ongoing, is that the metric data from aps server can actually crash premises.

A

When you have a super large-scale cluster and this part of it, I still think out finger out of the root, Coast and I think maybe two weeks later, we will commit. We always tend some passion or issues to the option to talk about this part. Okay, so this system could be IPs the right part, the LCD and could be a API server it for now see a critical parts be observed in our clusters.

A

Of that, that's why we actually stand a lot of passion exposed to the upstream and also to our own version of a code. But before talking about modifying the code, we want to mention that to concept that we are, we we of course we modify the code of a criminalist, but we we don't think we are working kubernetes, because there are some differences. So when we talking about fork normally we are saying that we are.

A

We will look down to some certain version of kubernetes or we modify the api of kubernetes or we just build something on top of kubernetes to hide the API server from the our customers or we just bypass some of the kubernetes kolodziejczak. For example, we bypass the controller pattern or kubernetes, or we just bypass some interface of kubernetes, or we just replace the couplet of we just like replaced Kubla to some other engine to your cluster.

A

So we think if you do, that, you're actually forking kubernetes, but what we are doing or we think the best practice, the more like. First of all, in our cluster, we will keep upgrading our class diversion by by keeping PI clip to release lag from the option. We have already done the speed. We have already done this rebase for a long time and it works very well so currently, our cluster is one pod, twelve and the next I think next month will upgrade over cluster to one port 14.

A

So this is how we keep tracking with a version of the option. Second, we will never change the API of kubernetes and and and essentially, we have no patch, only a server-side, and we stand every scalability patch to option and also when we think it is very important to over respect kubernetes philosophy, that is the declarative, API and the controller pattern, and if we also leverage the kubernetes extender, expeller extension ability ponds like hissing ICSI, to implement every feature we want to have in our cluster.

A

We will respect every interface here and, of course, I think. The most important thing is we will never replace crew blade with something else. We will respect crew blade as well as the CI. So that's how we keep our cluster, keep upgrading healthily and rapidly busy with the kubernetes option, with that, we also build a very interesting model in our company, which name is a key which is a given kubernetes. The equipment is, is actually a tiger team visiting the Hulk Alibaba company, which which were responsible to cooperate with the criminal sub team.

A

In order to do that, we actually build an internal up team, which name is equivalent, is inside Alibaba, and this is more like a mirror of the kubernetes upstream instead of the company, but it will carry the some patches or critical fix on on the unsecured pass, and the upstream team will cooperate this upstream with the crudeness up team based on the equivalent is code base here and, on the other hand- and we also maintain on downstream code, which have to version lag with a kubernetes code here.

A

So we, basically, basically men- tend to code based in your company and the the downstream actually are using the internet upstream instead of the kubernetes option. The reason we do that is because, as we mentioned, we need to serve post Alibaba cloud and Alibaba Group. Well, Alibaba Group is well Alibaba. Group is more like a union house cluster, but Alibaba cloud is what we have. We need to facing Z and the user. So we need to make sure that everything used by Alibaba cloud. It is fully option.

A

So that is why we use another option code base for our cloud provider and by using these models, all of all all of our partners and over Philly the companies but actually consume the Alibaba kubernetes infrastructure, which is a downstream of the equalists code base. So this is how the co-operative is upstream and for now this model works very well. That is also how we contribute patches to the option we do not just we do not just pour code from the downstream code based we actually.

A

First of all, we contribute to the engine option and at the stand, Pro requests from the internet upstream to the real community option. Okay.

A

So this is summary of today's topic, so, first of all, they use kubernetes post for post and any user, which is Alibaba Group and also as a public color provider and also we are using the virtual cluster based model tenancy model to serve all the customers. Cuttings customers inside Alibaba Group, and we use the container design pattern to migrate.

A

The applications from Alibaba from the old Alibaba infrastructure to the new coolest best infrastructure, and also implemented our own version of a customized workload management system to handle various requirements from the top layer of users, which may be different from stateful set and deployment. And another reason is that we really prefer predictability in this case, and because is this webscoe credit class are huge. So we think that performance testing system will always be your first priority.

A

If you want to build this cattle cluster in your organization and we actually modify some part of the code of kubernetes. But we will follow the philosophy that we will never and it's also how we start both internal and a public code public cloud provider, because we have two code bases and every cooperative each other, and the last thing is that I think you guys may also try to build a small upstream team. It's really fun and rewarding. So this is n-no. My two days share and thank you very much.

A

If you have any questions, please just ask.

A

Yeah, actually, we use up us in our cluster, but it's only for part of the user cases and the other user case. It will be mostly you Dina is part yeah yeah, please.

A

So this actually a step by step program and the we started this program at the middle of 2018. So for now is like one year. We just migrated hit a third, a percentage of whole Alibaba infrastructure to kubernetes, and the goal is by 2020. 90 percentage of Alibaba will move to kubernetes.

A

It's already be tested in 2018, there are 13 percentage of traffic is handled by kubernetes.

A

Okay, so thank you very much and you can talk with me. Offline.