Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2016 (Seattle), 16 Nov 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Good, the Bad, and the Ugly of Managing Production Systems with Kubernetes

Description

The Good, the Bad, and the Ugly of Managing Production Systems with Kubernetes in Chinese Enterprises - Xin Zhang, Caicloud

"Kubernetes has rapidly evolved from theoretical trials to empirical deployments in an increasing number of US enterprises. However, the Chinese enterprises unveil different traits when it comes to requirements, platforms, and the tech-savviness of the operators, rendering the upstream guidelines and references a far cry from enabling successful Kubernetes production usage in varying circumstances.

In this talk, we leverage our unique experience with using Kubernetes to manage production systems in large-scale Chinese enterprises, with a stab at stereotyping different categories of common usage scenarios not covered by the official guidelines. Peering through the mist, we aim to glean insights into the usage patterns in different industries (carrier, finance, e-commerce, and traditional, etc) to use Kubernetes more effectively.

We start with the standard Kubernetes features people are most thrilled about, then unearth the glitches and pitfalls when running Kubernetes in the wild, including dealing with Internet inaccessibility, unique security requirements, dancing with traditional, stateful applications, etc. Finally, we share our open source efforts and tools to tackle those issues in pursuit of wilder kubernetes adoption."

About Xin Zhang
Xin is currently CEO and co-founder of Caicloud (https://caicloud.io), a startup that fosters Kubernetes community in China (https://kubeacademy.caicloud.io) and provides Kubernetes-based products and services for Chinese enterprises. His team has helped tens of well-known Chinese enterprises to run Kubernetes in production, and has held multiple Kubernetes meetups with over a thousand attendees in total.

Before founding Caicloud, Xin was a Googler for almost 4 years working on Cluster Management and Google Cloud Platform, with close relationship to Borg and the early Kubernetes (in fact, he has engaged in bringing Kube-ui to life with Kismatic folks).

Xin obtained his Ph.D in Computer Science from Carnegie Mellon University, during which he has published 30 research articles in top-notch venues and delivered over 10 academic speeches at international conferences around the globe to over thousands of audience."
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

It's so good. Already, oh I thought I mean.

A

I'm going to prick actually help put.

A

In count as the solutions with the stories were based on or like on cameras with strong load,.

A

Generically so producers, so we are it um Joe, China and from the peak Oh his baby. It's also there the olive that quarter is based, so our funding members are not thing contributing to the internal Google cluster management system and the open source kubernetes projects. Also as a proud member of CN CF we're all coding.

A

Of like meter training session deification as public like articles or even books. So if any of you, whoever wants to like speak at one of our events, so do contact us. So it's very welcome us as a company providing risk product to help transform their IT infrastructure.

A

So the stories and the morals were a gift from our own customers from a wide array from the e-commerce Finance carry.

A

Like in the industry, stateful app and up legacy,.

A

So yeah just a first quick check, so not no Corbin at ease.

A

Bananas has become a kind of a common sense right so in Q Khan or in the US. So the question is how it okay studies that need to speak too loud.

A

All right, so, you may wonder like how kubernetes is doing in China. You know these photos were taken from our events, so they can pretty much give the answer we have held about like to meetups every quarter in China and there are like hundreds of attendees in each of these events. You know we have got about 3,000 followers on our kubernetes study group, so that is the status of the kubernetes in China.

A

So basically, there is the thriving community with growing user base and engaged partners, so before I dive into the details, I wanted to give a quick summary of what we found upfront from our case studies. So, first the positives.

A

We found one of the biggest reasons that people buy into kubernetes in China is due to the strong community and the big brand names behind it, and also people loved its native, service-oriented design and elegant API and architecture, especially it is very extensive and also it provides rich functionality compared to swarm or mesos. So it goes really beyond just scheduling and a resource management. It also provides a lot of like config configuration management, Network policy, secrets, etc.

A

So they're not just about resource management or scheduling, but in reality, so we also found there are some factors that affected kubernetes user experience. So from the list you can see there are.

A

Some of them are due to like the different environments between the US and the Chinese enterprises like the different cloud providers and also people's operational habits, because in China, so most of the operators are more familiar with user interface right, so they're, not so much a big fan of commits lines when people do get kubernetes up and running, so there are also some pitfalls. We found that need to also be addressed. You order to run kubernetes continuously and successfully in production systems, so this one is a short list, for example, for the first one.

A

So even though we know kubernetes has the notion of resource quota and the limits, but in practice how to set the exact values for them represent a big challenge, because if we set the quota too low, then applications may crash because they usually require a higher CPU spikes when they start up. But if we set the quota too high that may result in a very low utilization right. So these are the challenges we have to resolve and again.

A

So these are the common challenges we've seen in our own customers, but they could be seen in other places as well. Alright. So in the second half of the talk I'll cross over some of these challenges and present our solutions and the good news is so, some of the solutions are open sourced. So contributions are welcome. So the first thing we have to do is to get kubernetes deployed even without internet inaccessibility right. So when looking at the title, people may think of ok.

A

So this is probably due to like the firewalls right, but in fact surprisingly, it's a quite frequent requirement in some traditional industries or some like state-owned companies like in energy or in banking or finance, where all the communications have to happen within their own private corporate network.

A

So to compliment the upstream deployment instructions, we devised a ansible based deployment mechanism, so the gist is basically to pre-download kubernetes packages, Python packages and a bunch of young sources files, and we also need to localize our docker registry by populating it with a bunch of pre saved, docker images.

A

So speaking of deployment from our own experience, deploying kubernetes in our or in on-premise or bare Mendel environments accounts for about 70% of all the cases and a the other 30% of customers deploy kubernetes on public cloud, but in China pretty much all the 30% is Alibaba cloud right.

A

So in fact, Alibaba cloud is already among the top three largest cloud providers, or the public clouds in the world, and and also it has the fastest growth rate, so which means we have to integrate kubernetes with a bunch of like components like the computing service, the load, balancer and also the object, storage service right, and we also put together a pull request with our work in the PR link here.

A

So once we have a running korver Nettie's. The next thing we have to do is to migrate existing applications into the new platform, and one of the things that developers actually are concerned about is how to do logging right.

A

So if we recall this is the the standard way of doing logging so where we recommend applications to write their locks onto standard out or standard error right instead of writing to file and then docker or some other container rent runtime will like intercept the messages and a write that into a local JSON file under a darker directory and are now cubelet will comes into play and set up a soft link to map that file into a directory that flinty can recognize right and if Lindy we're now script, the files attached some additional metadata and a sent the file into the elasticsearch store that comes with Corbin at ease right.

A

So this is the standard model and uh it should work in most cases. But, however, we found there are some limitations where this model doesn't work as well as expected. So the first thing is in some traditional industries. People or legacy applications still write to files right. So if we wanted to require to use that model, it means people have to change their code or applications. You order to use kubernetes that may become a big hurdle to some enterprises, and also surprisingly, so that's not the only reason.

A

So sometimes people have a kind of legit scenario where they wanted to use different files to actually segregate logs even from the same application right. So if we are using the standard out approach, so basically, all the messages from the same application writes to the same JSON file right generated by docker and also we found some enterprises already have an existing VLK or start storage platform for the logs. So they wanted to reuse their existing platforms instead of using the ëokay stack built into kubernetes.

A

So here's our enhancement so now in order to allow people to still write files for the larks. So we first set up a empty directory or the empty dirt storage option for kubernetes and amount that into the application pot. So from now the application we will read logs into that empty directory, and so the next step is to modify cubelet to still set up a soft link.

A

But this time is not mapping from the Dockers JSON file, but it is mapping from the empty directory generated by kubernetes and again map that into the directory that Flint it recognizes right and now Soph Lindy. You can do a similar thing, but it'll also attach a metadata about the file name from which the logs are collected.

A

And, finally, when sending the logs, we also twig off Lindy, so that we can point fool indeed to send the logs to anywhere beyond just the building in elasticsearch platform. So this is our enhancement, so another possible option would be to use the set car pattern where we have a set car container for each application pod and that set car will collect the logs and a pipe that logs to any existing like a login platform.

A

But after experiments we found that approach actually requires more resource, because you have to deploy a sidecar for each of the application container in your system.

A

So after logging, like kind of a twin problem, is monitoring right so again, let's first review the current standard approach for monitoring kubernetes, so essentially we're running a CO divisor agent on each node or each machine, and on top of them there's a centralized hipster that collects and aggregates all the metrics from Co divisor persist. Those data in in flux, DB and finally graph Ana will visualize and present all those metrics in very nice format. So again, so this works pretty well in most cases, but in a large-scale production environment.

A

So it represents it incurred some challenges, for example the hipster or the influx DB themselves. Our single points of failure, which also incur scaling issues and also if people are familiar with the internal cluster management in Google, which we call which we call Borg. So there is also a Borgman, a system that monitors the Borg system itself.

A

So if we look at kubernetes, even though this architecture monitors the applications running on top of kubernetes, but nothing monitors kubernetes itself right so because kubernetes itself is also a complex software system with like cube, dns cubelet and all the networking setups. So there is no such Koopman to monitor the system components and finally, there is no alerting so.

A

Operators in our case have complained about so they said we cannot stare at our like refine a dashboard 24/7 in order to detect any anomaly right, so we need to have a automatic and intelligent or alerting module to send any like learning emails or short messages.

A

So this is our enhancement, and this is the overall picture, so you can immediately noticed we have replaced two components, so the first one is for the storage. We used open, TS DB instead of in flux, TB, and also we used Prometheus instead of hipster for scraping the metrics, and these are the rationales why we made the change- oh and also I forgot to mention so there is also a alert manager and that aggregates the metrics and also performs like alerting based on the user's rules.

A

So now these are the rationales of why we did these two replacements, so the first one. The reason we're using prometheus is because it supports really powerful querying language. So it's very nice. You can very easily craft your own query to see how the system is behaving and also we are using open, TS DB because it is based on HBase. So it has much better scalability in nature and it can be horizontally scaled, but also there are like challenges coming along.

A

If you want to use these two approaches, for example, to use open, TS DB, so you have to deal with how to run it on top of Corbin at ease and also you have to deal with some very demo. Details like solving the reverse, IP lookup problem, alright, so after logging and monitoring. So what is the next tricky business so to us? It is achieving external load. Balancer right so recall that in our cases about 70 70 percent of all the deployments happen on premise or on bearman dough. So how?

A

In these cases, how can you expose external services without a cloud provider right? So if you're, if you're, using like AWS or PCP or ager, so you can use their like public load balancers to balance traffic from the outside into your internal services right, but it's a luxury for on-premise or bare metal deployments. So, let's see what are the options currently in upstream, so the first one and probably the easiest one- is to use a note port right. So just a first, a quick recap.

A

So essentially, if you want to expose a service to the external world, so you can use a open, a node part which opens a high port on every machine right. So in this case it's it's 31, 0, 80 right. So it's open on every machine in the cluster and under under the hood. So essentially, kubernetes will set up a bunch of IP table rules to actually redirect the traffic's. So if any traffic comes to that port, so these rules will come into play and randomly redirect the traffic's to the multiple back-end instances behind that service.

A

So this is the North node port approach, so we can think about. So what are like the limitations for this approach right? So any guest, so the first one is all the outbound traffic or the or the external traffic has to target at a single machine and the destination address is like IP Colin port right. So it still represents a single point of failure and also incurs scaling issues and also because the load balancing is done via IP tables. So it only supports level for load balancing.

A

So you cannot do like session affinity or like a load balancing based on like like client regions, etc, right, so just a quick summary in ops trip. So currently we can do external load balancers through like cloud load, balancer providers, node, port or ingress. So ingress is a new concept right so in in kubernetes 1.4.

A

But again they all have some limitations. Like I already mentioned so public cloud providers are not always available. Note party is kind of hacky and uh and not highly scalable and an ingress so which is the new cool concept. It's nice, but its current implementation in upstream only supports using GCP to perform the load balancers right. So what if we wanted to do a on-premise or a bare-metal deployment? So here is our solution right, so we call it hi available ingress, so in order to gather so, let's first see how the default ingress works.

A

So essentially, if you want to use a ingress, so it'll deploy a ingress pod in a one of the machines in your cluster, and this ingress pod will do basically two things, so the first one is to set up a cloud load balancer which could be on TCP and also it'll it'll automatically generate forwarding rules based on the user's, the user's preference and after that, so all the incoming traffic will hit the GCP load balancer first and then, according to the rules, it will be redirected to any back-end instance.

A

So this is how the upstream ingress works, but we can see so one of the the other limitations for that is. There is only one ingress part running in the system right so again, not highly available, not really scaling. So that's why we need to have a height of highly available ingress. So from this example picture we can see that so first, we can deploy more than one ingress pod in multiple machines.

A

Right so probably the fun is too small, so it's not really readable, but we can just imagine so this one is the ingress controller right, so we have two of them all right, and so now it has a new problem. So now we have two ingress controllers. Before for all the incoming traffic, it can just directly hit into the only one ingress controller, but now we have two of them. So how can we do the load balancing for the ingress controllers themselves?

A

So our approach is to basically use keepalive D so that keep alive D will maintain a stable virtual IP, and this virtual IP will be assigned on multiple machines which run the ingress controller and for all the incoming traffic, so keep alive. D will guarantee that there is only one active instance: one active ingress controller at any time and all the incoming traffic will be routed to that active ingress controller and within this ingress pod.

A

So there's also a H a proxy or engine njx running in it, so that we can configure level-7, forwarding rules to support like session affinity and more complicated load. Balancing rules right and at this H, a proxy or ngx will be in charge of further directing the traffic to a back-end pod for that service. So this is a basic model for for the high available ingress right. So yeah things are looking better. So then the then the single single ingress controller case, but there are like still remaining problem with this approach.

A

So if we want to make a guess so we can see that all right so before I get to that. So, let's first see how this hi available approach will handle failures. So again, back to the picture we have to ingress controllers or to ingress pots running on the two machines on the right side. So assuming now like the the ingress controller on the right most failed and then so migration will happen.

A

So, first, the application containers running on the failure machine will be migrated to a healthy machine on the leftmost site and also the ingress controller pod. Will also be migrated as the the arrow denotes to the healthy machine on the the leftmost side. All right so now keep alive D. You will come into play and it correctly also maintain the mapping from the stable virtual IP to the new machine so that all the further traffic will come into the new healthy machine instead of the failed machine right.

A

So this is how the high high ability, the high availability, helps in this case all right. So now, let's think about the problems with this approach. So the first one is, we may encounter what we call a reload storm because remember all the forwarding rules are now being taken care of by the H a proxy or the ng X. So if any of the positive with the sub behind the service failed- and there is a migration- so we have to update so by we I mean the ingress controller pod.

A

So it's automatic, so it's better than manual, but we still have to update like the config files for ng, X or H a proxy right. So this is for one service, but imagine if we have like hundreds or thousands of services and each with like tens of pots. So the problem, the probability for this update is very high and we have to reload the config for the H, a proxy or ng x, pretty often right.

A

So this is what we call the reload storm and also there is the scaling issue as well, because, even though we have multiple instances of the the ingress pots, but they still have to carrot of the forewarning and a load balancing for all the hundreds or thousands of services right. So it can still represent a scaling issue and to solve this problem, we actually introduced the scheduling, like it's scheduling, of jobs, that a concept into here. Alright. So here is the gist. So we essentially perform a sharding.

A

So suppose we have multiple ingress controllers, instead of letting all of them to handle all the other services. We perform a sharding so that some of the ingress controllers are only taking card of a subset of the services and they will only perform load balancing, as well as like forwarding for those subset of services right so now, so it basically resembles a scheduling, scheduling scenario where we have like a bunch of services and we have another bunch of ingress controllers, so we have to assign which ingress controllers take care of which subset of services right.

A

So it's a scheduling problem. So to solve that problem, we introduce a new concept, called the ingress claim right. So a ingress claim, if you recall the PVC, the persistent vol claim is pretty similar, so you define a class of forwarding like service upfront so again, so the fun is too small to read so, but basically these two black boxes define two ingress claims, so each with different, like performance guarantees like what is a delay.

A

What is what is a timeout and what is a like the bandwidth for this ingress claim, and now, let's see how things work so, first, we add an EO controller into the master component and we call it the ingress claim controller. So what it does is whenever there is a request for a new service to be exposed externally, so it submits a request to the master saying that hey I want it to be exposed externally.

A

So please assign me a ingress controller to take care of my Alma forwarding and load balancing, and also it requests a certain ingress claim class saying that I wanted to have a very low like a latency to do my load. Balancing and this ingress claim controller will look at. Are there any suitable ingress controllers already in the system and, if not so, it'll create the certain number of ingress controllers for that service right.

A

So in this example, there are two instances and the number two actually comes from the ingress claim, so it's also property for the ingress claim class right, because that means how reliable you want it. You want your like load balancer to be alright, so that's pretty much about our high available ingress mechanism and the last tricky business business I wanted to mention is how to achieve high availability masters.

A

So you may already noticed it's kind of like an option already in the upstream, but the problem again is the upstream solution requires to use like TCP or AWS load balancers. In order to achieve this high availability right so, but in bare mental or in on-premise cases, we basically have to devise our own mechanism to achieve this. So the reason is intuitively.

A

If we want to have high availability master, we have to deploy multiple instances right so as depicted here. So we have multiple master instances running in here. But again the problem is now when a traffic comes so which of these master instances, should we redirect the traffic again, so we need a load balancer, and so it's similar story. So we use a keepalive, D and H here proxy in front of all these master instances to achieve the high available load balancers themselves. Alright, alright.

A

So in the remainder of the talk, so hopefully I'll talk about like something more interesting. So for the previous like, like the issues we mentioned, so there are kind of the enhancements to the core core Benares like components, but in order to run kubernetes smoothly in a production system, so kubernetes along is not enough right. So we have to have a way to do like continuous, continuous, like integration and a deployment, and we also need to have like a registry, a docker registry or like a email registry, to securely store the images.

A

So now, let's see how we solve those problems with some open source efforts, so the first one I wanted to introduce is a system called cyclone. So this is our open sourced, CI CD system and it's more kubernetes native, so people today may have been using like a jenkins with like either like plugins or some custom scripts in order to like integrate jenkins into a container platform. But cyclone is kubernetes native, so it is more lightweight and it is much faster.

A

So it basically shapes your code directly from from your source repository to any kubernetes cluster that you specified. It also has some very nice properties that jenkins or some other CICE tools do not have, for example. So we know the modern world is is all with micro services right. But another challenge with micro services is now it has too many components so when you're doing a push or you're doing a release, so there are like multiple components that you may have to update at the same time and they may have like interdependencies right.

A

So if you change the API of a back-end, you probably want to update that first and then update your front-end. So your order to better support that. So cyclone is a dependency aware, so it supports a API and in the near future, a UI that that's a user to use a drag-and-drop approach to first set up a dependency graph like shown here in the picture, and so then, when you are doing a release, so it will follow the dependency order and it will start from the root. So that is dependency awareness and also.

A

We know that about 60% of the doctor images have some like suspicious portion it in them, so security for the doctor. Images is also very important, so we integrate in cyclone with a clear to basically scan each layer of the of the docker images against a database of known like vulnerability signatures in order to detect any any like suspicious code in the in the base image or in any layer of the image.

A

So this is a like a sample output from cyclone all right, so this one is how it is designed, but I'm not going into the details. So you can visit our like a github page for more details, so I also mentioned.

A

We also need to have a private registry to store the docker images and the options out there are probably like either docker hub or like the standard, docker registry distribution project right, so those are open source as well, but we are using a open source project called harbor, so harbor was initiated by vmware and we are one of the facial partners on that project. So this link here is our fork, but from there you can look at the app stream, the VMware version.

A

So the reason we are using like a fork here is because so in our version, so we have a more concise user experience or UI. So we're on a single page. You can see various image details about like a different images like where are they deployed when are they like, updated and who did, which updates, etc and also another novel feature we introduced here is labeling. So if we think of the canonical dr.

A

image URL, so it's basically just three layers right, so you have the registry location, slash the project name or the name space right, slash, basically the repo name. Of course there is the label in the end, but so in our cases, so for some of the large enterprise users, they wanted to have a more flexible, like labeling mechanism, to really enable them to classify and imagine managed images more flexibly.

A

For example, they want to classify images per their customer, so they may in a like ISV industry right or they may store and I classify images by types or by regions right. So, if you think of just the connect canonical talked about the image names. So how can you like express this information? So, of course you can put everything into the tag right in the very end or put that into the image names. But these are like a key, so we basically borrowed the concept of kubernetes right.

A

So, if you think of how kubernetes manage its resources, so it is using like two labels right. So we're doing the same thing, so you can tag any doctor image with any number of labels, and then you can use the api's to query those labels in order to perform like management and classification right and beside these features.

A

Of course, it is highly available compared to the like the docker distribution or the docker registry, and also it has a nice feature of automatically syncing images between multiple instances of this Harbor as of Harbor registry and what is even better is we also enable on a one-click deployment of hover on kubernetes so that we can make hover even more highly available.

A

So this is an example like user interface for harbor, alright. So the last point I wanted to mention, although I'm running out of time, is a big data support, because it's pre is a pretty often Airy requirement to run some big data applications on kubernetes as well, although it is not very well officially supported in the upstream right, so people maybe are still running more like stateless application on kubernetes, but we wanted to beat data.

A

So besides, like Hadoop and spark so here with maybe one or two minutes, I wanted to highlight our work for running tensorflow on kubernetes, so for those of you who are not familiar with tensorflow. So it's a deep learning tool, also open sourced by Google, which uses neutral networks to perform some more efficient learning on non structured data right like pictures or like audio files or etc. Right.

A

So essentially, the architecture for tensorflow is a is at the server and a worker model so where the server essentially stores all the parameters for the training model and the worker is the one that really performs the prediction and a classification and the the actual job. So the order to run tensorflow on kubernetes the gist is to use the job concept in kubernetes so that you first start the tensorflow server and whenever it needs to start a worker, so it basically creates a kubernetes job to do that, and also to achieve more reliable.

A

Like effect so we can set, we start on failure policy for that job. To finish so, to summarize so our takeaway for how to increase the kubernetes user base in China or maybe in a global scale, is basically threefold. So the first one is either deployment is not only about having like a one-click deployment solution, but it's also more about being able to deployment kubernetes in a more like diverse environments, either with or without internet access, either with or without AWS or OpenStack. So this is number one.

A

So number two is in order to use kubernetes in a large, especially production systems. We have to do more like performance tuning and optimizations and finally, that's the case for any active projects, so we always require like better documentation and a training as well. Sorry, and probably for like none english-speaking countries, so any internet internationalization is also preferred.

A

So that's pretty much conclude my talk and finally I also wanted to thank two of our team members, who are also kubernetes contributors they're also here, like durian, and mean ciao, so they're also helping a lot with all these work and are preparing this slides alright. So it seems we only have one minute to go so I would like to take any questions.

A

Yes, they they do because, like historically like Hadoop and a spark, these are more like a mature and more like a well understand tools through the customers. Yes, so they're using Hadoop and a spark actually more fruit, complete than tensorflow yeah inside kubernetes customers, clusters yeah, so data storage. So these are different options, so 30% of the customers who are using like Alibaba cloud, so they basically use a like the object, storage service, it's a public cloud, but for our like on-premise storage, we basically use SEF in order to do that.

A

All right, so if you have further questions so I'll be around to chat. Thank you very much.