Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2019 (Shanghai), 5 Jul 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Adapt to Unified and Pluggable Cluster Management Platform at LinkedIn - Tengfei Mu & Abin Shahab

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Adapt to Unified and Pluggable Cluster Management Platform at LinkedIn - Tengfei Mu & Abin Shahab

RAIN is a cluster resource management system developed at LinkedIn. It manages resources for tens of thousands of hosts per cluster in multiple datacenters including Azure to support scheduling both long running and batch jobs. It is integrated with existing LinkedIn cluster management ecosystem. The goal for our next generation cluster management system is to support heterogeneous compute workloads quickly to improve developer productivity and server utilizations. We have evaluated and decided to adopt K8s' declarative API and extensible architecture. The adoption process has quite a few challenges for integrating with existing ecosystem at LinkedIn scale. We first give an overview of LinkedIn cluster management ecosystem. Then we talk about our evaluation process and adoption challenges. We will then share lessons we learned during production and integration process.

https://sched.co/Nrm9

A

I guess: let's get started so hi everyone, I'm, Tom, fameux, engineer manager for the image container team at linking my colleague, Irene Shahab, stop engineer at Lindy. Today we are going to talk about living, is journey of adopting Konami's for unified classroom, a common platform.

A

As you can see from this picture, our journey is a Star Wars story. Assuming many of you have watched, Star, Wars, Star Wars.

A

um Here's the agenda in absolute view, we will start with introducing existing nicotine class or management platform how it encountered. The new challenges does revaluated kubernetes as our new hope, then how we quickly started and approved value by support a different kind of machine, machine learning, related workloads on cover aunty's and then eventually we're extending kubernetes and unifying our platform. So.

A

Linkedin is the word or its largest professional network with morning half opinion professionals in more than 200 countries and territories worldwide. Our mission is to connect the world's professionals to make them more productive, productive and successful.

A

S'posed of us working the team called LPS LinkedIn platform and service. Our mission is to provide a compute fat forum for linking engineers to make them more productive and make efficient use of our hardware resources.

A

So we have four pillars: enable innovation by providing the right building blocks and abstractions increased developer and operations, productivity and increase hardware utilization.

A

This is our scale. We manage thousands of services. Every day we have hundreds of thousands of builds. We have tens thousands of deployments daily. We manage hundreds of thousands of hosts men in multiple datacenters with millions of containers, so this is our existing cost management platform is simplified, layered view on a very bottom. We have in ops which manages Hardware asset and their lifecycle. Then we have ready its resource allocation, meaning thing and live is for linking deployment control plane.

A

So on top of it, we have the pass layer. Race is for auto scaling and auto ram addition, orca is for short-lived job. Orchestration new on t is for storage service. Stop, stop storage, self-service provisioning on the topmost we have mistral. The idea is, we are moving into this intent-based service blueprint.

A

In this architectural diagram you can see we have multiple datacenters. Each defender has multiple clusters. We call it a fabric. The fabric has its own control play. We have one main artifact two clusters and another for disaster recovery purpose on in another defender. All the fabrics have their own artifactory proxies to scale out the route traffic.

A

Let's go through a typical manual deployment. Workflow developer first comes to way to allocate or change resources for their services. They normally create a resource, slice or profile. Then they add instances or remove instances from it after resource allocation, they can trigger the performance through our deployment, control, plane, lead, which will check our back against a check. Their deployment policies and trigger config compilation and public published process then generate and stand. The deployment plan to physical host.

A

After the deployment plan gets through the host, the host launcher will download the artifact image and competes then start the application in the locker container, which is our abstraction layer overrun seen during the container start process, we decrypt come secrets in a config, opens necessary firewall ports and download the TLS certificate for the service so that service to service communication is HTTPS days.

A

Other than the long-running service job we also have batch jobs orchestrated through Oregon orca, was started as our internal Jenkins replacement. Basically, it creates a tree of short-run jobs with priorities and then utilize spare capacity in our common pool. When there is no available resources in the common pool, the table will remain in queue.

A

So all car presented this challenges in term of a capability to support growing in batch workloads and then another challenge is how do we model the emerging, for example, AI workloads in our existing platform? Last but not least, having this portable stack across on Prime and a juror is critical to us. Nowadays,.

B

We felt kubernetes and other modern schedulers gave us a new hope to deal with these challenges. Oops backwards.

B

We set up the experiment. We set up a 1602 Gorillaz cluster and ran as many orca post comment, jobs and there as possible host commit jobs, are a job that run when a LinkedIn engineer, submits code or Mardis code or creates a cool request, and we found out that kubernetes performed admirably compared to our own stack in measures such as. How long does it take to take a job from queued to running to Varennes performed really well.

B

We we performed several other more tests and holistically. We realize that kubernetes and other modern schedulers can provide a generational boost to the productivity, operability and utilization of our application fleet beyond what our current stack made of rain rain lid Locker cap.

B

Among all the systems that we tested kubernetes, was the one with the most capabilities it's built in this kind of layered way, where there's a robust ecosystem around it, and a lot of components have been built around that which has extended its extensibility, which I'm sure you guys are already aware about. Overall and because of its extensibility, we felt that it was the scheduler that we could integrate the best with LinkedIn's architecture.

B

So we now had two paths to go forward to whether we could start and researching on our pluggable scheduler architecture, idea where we could decide how to integrate kubernetes with the rest of LinkedIn and complete that work and versus whether we could also pick our first use case. There were a lot of teams looking to move into a new school earnest and we wanted to. We had a need to support them, so we decided to do both, but first I'm gonna talk about our first use case.

B

The first use case was due supporting Jupiter notebooks on kubernetes and when we proposed this to LinkedIn security team, their question was kubernetes is not integrated with any security system that we have in LinkedIn. So is it going to bring down LinkedIn security, or can it can it be integrated? So we decided on our integration strategy with the kind of security first in mind, we deployed kubernetes on rain, to both the control plane and the application plane.

B

The cubelets are all on rain, so when they get deployed they can get a certificate from our own certificate server and they they also can send metrics to our own metric service. So this allows us to kind of discover and may manage kubernetes cluster. Just the way we are.

B

We've been managing and discovering rain when we deployed Jupiter hub on kubernetes, so jabarut hub is mainly to pod and orchestration pods, which is called the Jupiter hub and then the actual worker or minion part, which is the kind of the single person notebook which is like an AI engineer, would use to do their work. So both of these types of pods could get certificates from our certificate server and could talk to the rest of LinkedIn using LinkedIn sewn certificate authority.

B

The users workflow is like this user logs in to our ml platform UI, and then they get a two-factor authentication token from our certificate server, and then that token is used by Jupiter hub to figure out which user they have to spawn the notebook pod for and then that notebook pod gets this token as a secret even mounted on it, so that they can. It can use it for subsequent actions.

B

Users notebooks get loaded from a get back storage service that we have in LinkedIn, and so this allows the user to kind of do their work and save their work in that get back storage repository and when they come back to the notebook platform, they can start over from where this, where the laptop.

B

The other thing is because that token is available to the notebook as a secret that can be used to talk to our HDFS cluster securely. So when the user launches a query to our HDFS cluster, we use a spark context, manager called Apache levy and and the Apache levy will use that token to validate identify the user and talk on behalf of them to our security. Faster.

B

Once we and productionize jupiter hub, we thought that kubernetes has journey at LinkedIn will be very smooth, and this is usually what day one kubernetes looks like, and then that brings to our first war story, one Piney being our notebook pod started crashing, and we thought that the flannel Dean overlay network that we were using, that's the one to blame. That was a root cause of it. That was the root cause, and when we looked up advice about this on the internet, the raining advice was with a new car cluster and start over.

B

Luckily, we didn't do that.

B

We kept digging into what could be the issue. Our next root cause hypothesis was that it was the cordilla nest pod that we were running to support all DNS of all pods and when we decoupled corniness from the rest from the Jupiter hub, a notebook pods, the issue still didn't go away. So that hypothesis was not correct. A few days into the debugging of this issue, we were started having internal arguments within the team that whether we should actually listen to the advice on the Internet and kill our cluster and start over.

B

Then we got our first clue. It was that all portable networking was not working and this particular issue is not related to Jupiter hub or notebooks. But it's it's a it's a deeper issue, so we had to take a deep look at particle networking and really try to understand how that works.

B

So this is how you clear, don't books, sorry Google in this pot, the pod Network. We should work all pods have their own IP address and say when the Jupiter hub pod wants to send an IP packet through the notebook. It just addresses that IP packet to that notebooks IP address and it should go there.

B

However, in reality, this packet has to go out of the host, where Jupiter notebook is travel to all the routers and switches that are in the direct data center and then go into the power into the host, where the division of the Potters so for to do that. The IP address. That's on that ID packet has to make sense to the router than switches and internal IP address using kubernetes in the pod do not make sense to the routers and switches.

B

So the way it works is when a packet is sent from github. That packet address to the notebook pod makes it to this plan. Ld daemon on the host and what lousy daemon does is. It will wrap this packet with the IP address of the destination host of the host, where the notebook thought is and now that now this IP packet can make it out of the host where there you better, have the hostess and go to the host where the notebook is, and so this packet, which is double wrapped.

B

It now makes it into the host where the notebook is, and it goes to the flannel, the app whose job is to unwrap this, and once it gets unwrapped. It now has the address of the notebook pod, so it makes it to the notebook.

B

Unfortunately, there's more to this madness.

B

In both cases, when the packet is going out of the host, where the Jupiter hub is or going into the host, where their notebook is we're asking their hosts to act as routers we're asking them take this packet IP packet, which is coming from an internal network and send it sent to the external network and, in the case of the ingress they're telling take a packet from the external network and send it to the internal network and Linux operating systems, would not do that unless they're told explicitly told by studying a kernel plan, and that is the ipv4 forwarding flag once that's we set this ipv4 forwarding flag, it starts working.

B

So what happened? The root cause of this issue was: we have an under relying automation platform which sets up our hosts and that automation platform had gone in and unset this flag, ipv4 forwarding flag in a bunch of our hosts and that caused this problem. So once we fix that issue in that underlying automation, platform, no notebooks, very happy and users- very happy it's after that. After this, we thought. Ok, we're done with war stories. Kubernetes is going to do a successful adlington.

B

This brings to our second war story, one fine evening, Noble Park stopped working.

B

We thought that the plan LD daemon, was to blame and when we looked up advise on the internet, the raining advice was nuclear cluster that over this time, we quickly identified that hotspot networking was not Beijing.

B

We have to take a closer look and found out that our Jupiter hub pod was complaining that it it cannot reach the API server and that's why it started crashing and dominating a crash into the notebooks. So we take took a look at how the Jupiter hub pod is constructed. Then our case, then our Jupiter hub pod has three containers and in its container that gets certificates from our certificate server.

B

And then this init container makes that certificate available for the subsequent containers, which is we have an engine, X container, to set up routes and then a hub container to orchestrate pods and.

B

In this particular case, it was the nginx container complaining that it doesn't have this kubernetes service host environment variable set. Therefore it doesn't know where the API server is and because it doesn't know where the API server it doesn't work, it just crashes.

B

So, in our clusters we set the kubernetes service host environment variable on every pod, using a pod preset a pod preset. If you guys know it's something, you can use to say that okay, they do this before the pod is setup and in our case we basically say: okay, setup, the Cougar and the service host environment, variable on the pods environment.

B

Before you start the pod and it's always said to the fqdn of the API server and that ensures that user can securely talk to the API server, because the fqdn is the only thing that's in the API server certificate.

B

So before kubernetes version 1.14 in its containers did not honor hot presets, so hot Wizards could not be set on init container. So what we did was we hard-coded the kubernetes service host value in the init container, so once we upgraded to 1.14, the in its container issue was fixed and 1.14. We didn't know that so pot pressure controller tried to apply the kubernetes service host value on the image container and it found the conflict.

B

They found that there was a value with the same key, but a different value for the kubernetes service host and once it failed there, it failed to apply that environment variable value in any subsequent container.

B

So once we fix this issue, things started working again and the loot pod was happy. So after this we extended our use case to a batch batch. In this case, users wanted to be able to law, distributer tensorflow training right from their notebook. So we enable that we're using cube flows, pairing library, users could annotate their notebooks notebook code and then once they ran that it would actually talk to the TF operator on the cluster.

B

And then the TF operator would launch a tentacle cluster where the workers distributed across the cluster and these workers would be able to talk to the Nvidia device plugins on those hosts to get GPUs allocated to them. So this enabled them to do a very large distributed job and also take advantage of GPUs. While talking to our secure HDFS cluster.

B

Our next use case was another online use case where serving serving models directly from our kubernetes cluster.

B

So let's say there is a production service that wants to identify the image of a cat, so it will send that image of a cat to this model service, and then that request, would it what it expects from the model service is the word cat so that request gets forwarded to the model serving TF serving pod in our model serving cluster, and then the TF serving pod will route it to a model, that's deployed by our model deployment system, and this can also take advantage of GPUs on the box.

B

So it can answer a lot of queries in parallel. It can take. It can cache queries and things like that.

B

With these three use cases, we had started supporting a model offering to Jupiter notebooks model training through tensorflow, distributed training and model serving through TF serving, and this where significant parts of an ml pipeline. So naturally we now had a issue with figuring out: do we buy an entire pipeline from the open source, or do we build it in-house?

B

So this is because leg dinner already has a significant ml infrastructure that built on the Hadoop yarn stack, and we had to kind of compare the strengths and weaknesses of the huddle here and stack versus the kubernetes stack. Now, how do we turn stack is very well integrated into LinkedIn, and also it's has capabilities, such as hierarchical queues, preemption that we use a lot kubernetes, on the other hand, is very strong with mature container support and it also it's a PI is implemented by all the new ml training Poona frameworks.

B

So we have to bring this both of these world together, because there's strengths on both sides and we, our next-gen ml pipeline, had to adopt the best.

B

So to bridge the gap we started initiatives such as allowing users to be able to securely access HDFS directly from their kubernetes cluster, so Kerberos is what's used in LinkedIn all across our HDFS clusters. We are building a product that we may open source is to be able to support coverage directly from the kubernetes cluster.

B

So the way it works is when user logs in and submits a job to kubernetes they are able to submit a Kerberos ticket along with it and that ticket is handled by a delegation, token controller and a delegation token service, that we will build in the kubernetes cluster and this delegation. Token controller and service then talks to the HDFS cluster to get a hadoop delegation token, and then it mounts that into the into the worker pods.

B

This allows that pods to directly toxeus GFS cluster securely and once that worker is done, that delegation to token could be revoked. If that workers lifetime spanned longer than the delegation tokens expiry, then that gets automatically renewed by the delegation. Token service.

A

All right so after supporting both ally and batch AI workloads on our core bernardi's platform, which will show the value and the confidence for the team, we are working on extending kubernetes and unifying them platform.

A

So here are more in questions that we're working on past layer in the question was raised and Orca we're also working on unified topology in question. The unified apology provides this single shared view of Lincoln's application fleet and it's a topology or basically, what runs where so we have many system.

A

An internal system depends on such a view because, for example, dot like certificate management, server, distribute the firewall system and PN's discovery, and we also work on our back into question with they'd watch using automation, wipe oops, pod certificate, inspiration using unit containers or the physical host. We also have implemented the CRI for our Locker containers. The idea is, crew.

A

Bladder would be able to speak with Locker CRI out of box, so the unique container in a locker CI would decrease the secret in the config and the open firewall or, if necessary, as you can see with all these integrations kubernetes adoption and linking is really picking picking off.

A

So, let's wrap up kubernetes the whole ecosystem is really powerful and going in very fast. However, they do operation and integrations, especially with a huge legacy infrastructure can be challenging.

A

Supporting our ultimate goal is to create a unified compute platform on top of this coburn at ease.

A

What started with supporting emerging AI workloads to prove the value and shew confidence was a great kick start boss cloud native is morning cloud: Oly, kubernetes and its ecosystem embodies cloud centric best practices and provide this cognitive approach to modernize our laxing infrastructure. It has a huge value for large scale enterprise companies.

A

All right. That's for this talk. Thank you for coming.