Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2018 (Shanghai), 21 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Smart Workload: Automated Routing, Scaling of K8s and Serverless Functions - Enlin Xu, Turbonomic

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Smart Workload: Automated Routing, Scaling of K8s and Serverless Functions - Enlin Xu, Turbonomic

Developers have a dream that applications can run anywhere without modification. This begins with Kubernetes and extends to serverless, which promises developers to deploy and run functions without having to manage the infrastructure. To achieve this dream workload must be self-managed anywhere regardless of platform. How would you balance resources between serverless functions and other workloads co-existing in your k8s cluster? How do you scale functions without hitting resource constraints? Should you scale the cluster, or could you route workload to another less-loaded cluster? To find answers we have been running experiments to automate smart workload routing and scaling using Istio/Kong/Gloo over various platforms such as OpenWhisk, OpenFaaS and AWS Lambda. We will share what we have learned in assuring performance while allowing applications to run anywhere, on any cloud.

To learn more click here: https://sched.co/FuKW

A

So hello, so this session is going to be English but feel free to ask questions at any time, both English and Chinese. So before I start I really want to say thank you for staying until the last minute, because when I received the schedule, I was literally having to scroll down to the very bottom to see my session and I was really worried that I'm going to face an empty room. So thank you so much for being here today, I'm going to talk about smart workloads, automated writing, scaling of kubernetes and service functions.

A

So a lot of buzzwords, as you can tell from the title- and there are a lot of more to cover like edge computing, smart workloads and myself. My name is Ellen Shu I'm engineering, director at tokomak, I'm, pretty sure not many people heard about ergonomic. So we are a company actually based in Boston, founded in 2009. Myself is based in New York, so we feel very lucky to be here because compared to a lot of presenter in this conference we are a much smaller company.

A

We have around 500 employees and the whole company is founded on the idea that the software should manage IT resources, not people. So we are. We have been awarded the 2018 cloud words most best 100 company cloud companies in the world and we are growing real really fast. So that means we're hiring. So if you're interested just contact me after this session. So if you're interesting this presentation, so also, you may notice that I posted a Twitter from Kelsey Hightower, so I'm pretty sure that Kelsey is much more famous than our company in kubernetes community.

A

So the reason is why I post his Twitter here is because when he says here, here's my application run it for me when and where I want it securely. That's the end game. That's exactly aligned with what our software is trying to deliver. The value we are trying to deliver is that we help you to automate the IT management for people using software instead of people.

A

So that's the endgame, so people developers have the dream that they want to run application anywhere and how to operate these applications, whether it's a application, containerized application service application, whether it's a cloud infrastructure arm, prime infrastructure, how do I manage it? This is exactly what we are trying to solve, so we have over 2,000 world-class companies that trust to economic across different industries. So without saying much about my company, here's what I'm going to cover in this session, so first I'm going to introduce what is the challenge to deploy and operate?

A

Applications in the cloud then I'm going to talk about what is the concept of smart workloads and what is the concept that of edge computing and how the smart workloads engage competing, look like and a large portion of this session is going to be a demo, so we create a testbed to actually combine a lot of technology together.

A

That's a really cool testbed, as you can tell later that we want to show you and we're going to break down these technologies and show you one by one and how this mark low a smart, more close in edge computing will look like and how smart they can be, and last but not least, I'm going to summarize with a few key takeaways. So as I mentioned during the session feel free to interrupt me.

A

If you have any question so challenge, I want to run some code in cloud, and so things are always more complex than you thought. So, even if you want to run some workload in cloud, it's may be easy in the z-row, but definitely not easy in day.

A

One day, two and am/pm, first of all, in day 0, you may want to consider about the operational complexity, whether you want to package it in a virtual machine which you have to configure the Machine, the os/2 storage, the network, or you want to configure the container in which you still need to configure the application itself. The sizing is scaling how to automate the scaling. Things like this and serval is still.

A

You have to configure the memory configuration where to run whether you want to run it on a cloud service platform, or we want to run some kubernetes based service platform like a native open risk and that in day two and they ongoing business, you want to see the capacity management. Even the cloud has infinite capacity, but that's not your budget capacity. You don't have infinity budget and scalability. You always want to infinitely scale your applications, but you only want to pay for what you really need.

A

So you want a highly scalable cluster with highly scaleable workload, they're always automatically scaling and now so that you don't pay extra for what you need. So you may easily reach to an conclusion that civilization seems like the easiest way to deploy applications, or is it if we take a look at the service use cases, for example, an image upload to s3 triggers alarmed out to computer some now to post on Twitter I'm, not going to read all this aloud, so you can easily find a few servers use cases.

A

So by looking at these use cases, you can easily tell that the complexity, the first complexity comes in the fact that service application itself, the service function itself, doesn't have to be the only component in the application. So usually the application itself is a hybrid application that is composed of service function. Some database services, some other services, either running container of VMs. So what you are facing, the first challenge is the complex service chains that you may have multiple service functions, that connect to your RDS or connect to your dynamodb.

A

And if you don't have a you, don't have a well operated service configuration. You may face troubles it cause of of this cascading, in fact, and then what about the challenging data to an operation data operations? So what? If the functions aren't properly sized so we're under under sized? You will face some high response time. Well, over sized, you will face some unnecessary high cost accumulates. So if you absolutely poor everything on the cloud, they are charging you based on the memory sizing.

A

So if you oversize your function all the time, then you are facing extra cost and then it's also possible that you may face some cross region data transfer cost, which is also going to be severe.

A

Even if you think that service should be very, very cheap, so the solution- maybe you should be placing this lambda functioning every region, but still you should be able to have the capability to reroute the traffic to a different region in case it will cause some high cross region data cost and then, if you actually choose not to deploy your service application in the cloud instead, you want to deploy a service application anywhere based on kubernetes platform, for example, k native, open, fast, open risk that you may face the same scenario that you face for containerized applications or virtualized applications that you may face the performance degradation due to the underlying infrastructure resource congestion.

A

So in this case the solution can be scaled cluster or reroute the traffic to a different cluster, which is the traffic engineering. So in this session, in fact, we build a test path, which is to showcase this reroute of the traffic between different platform, one deploy on the edge, which is a key native service platform and the other deployed on lambda, and why you have the resource congestion happens on the key native platform. We automatically route your traffic to the lambda function that serves the same function so.

A

This is exactly the demo. I was just talking about, so smart workloads with edge computing, so edge computing has become harder and harder topic. So in fact, this morning that could be edge has announced that they are going to be open source, so that means that edge computing is actually not very far away from us. So what is H computing being?

A

That means that you're moving intensive workloads from the cloud out to the edge of the network, so as computing as you can tell it will be highly valuable if you're actually running a lot of IOT or cloud. So we have heard a lot of hardware vendors that start to thinking about deploying their data center instead of in different regions like 50 different regions in the world, they are considering to deploy the data centers in a cell tower.

A

So imagine the intensity of how many cell towers you are going to see and all of them will be equipped with data center. So when you actually are driving a car when you're stuck in a traffic, you try to reach some server side application from from your mobile device. Instead of reaching to the cloud you may actually served by a beta server server that is deployed at the edge computing side, so edge computing is really really cool and what is smart, workloads, smart workloads here we define as these easy to place and size their short-lived.

A

They are very small. There's no need to continuous place. Maybe don't have to move this workload, you don't have to migrate these workloads. So it's one time placement and the size is usually small and it should be traffic. Engineering chemical means that it's stainless. We don't consider in this demo. We are not considering, if they're actually trying to get a lot of data from the data that you have to consider how much data is caused and how far you are away from the data. So this is not.

A

We are considering as smart workloads, so state state's workload is also of important criteria here. So multiple instances of the same function can be placed across different clusters: different regions, different edges, to satisfy the client request.

A

So, as I mentioned that we are going to start the demo, most of the most of the part of this session is about going through the demo, so I hope that you can enjoy. You can enjoy this time of break down.

A

So, first of all to to create this simulation of edge computing versus cloud, we first choose to deploy the cloud service platform and the choice we choose is the K native platform, which is a how many I wonder how many of you have heard about K native great, thank you so key native as a platform that is actually announced by Google this year.

A

So it's some pretty cool things which we believe that it will be a new standard for run service workload on kubernetes, so we actually in the beginning which so anything you see here at a native alumna. They can be swapped by any other service platform. So this is really just for proof of concept, so we initially we use open risk.

A

Then we think a native so cool, so we decided, let's play with key native, so we deploy a service platform key native at the edge and then aw database lambda in the cloud, and within these two platforms we actually deploy the same applications, one ink-stained native, and why inside of this lambda? On top of that, we have deployed some API gateway called con.

A

So again, this call can be replaced by any other API gateway like glue, for example, another cool start up in Boston area that they develop some API gateways and what we call so the purpose of having this API gateway is that you can easily configure some static routing between creative and lambda, so that when we actually push the policy through is do then go through this call.

A

It knows where to route these traffic to so there, so some of the people may ask, so you can imagine that I just mentioned is do so is to is another routing layer on top of that. So since it's tube it's becoming very high, so why don't? We add is to into the picture as well. So it still here serves as a policy enforcement tool, so basically, in this demo we're using the label based policy routing to actually push the routing data to ISTE, oh, so that we can routing the client based on label.

A

If it's a label foo it goes to a key native. It's a label bar. It goes to the lambda and the reason why, when you come here is really because Kong has a very good integration, with a lot of servers platform that to save some time to building this test bed, we actually choose to integrate with call. Instead of writing some extra code to use sto only but Kahn is actually not unnecessary parking. This in this whole demo picture, and then we have the client start to visit sto.

A

So what we want to create the scenario here in this demo is that the client keeps calling it still and then they get routed some of the client routed to creative if the label is full and when the key native itself is built on kubernetes cluster, that is at the edge ok. So that means that you can actually monitor the resource resource consumption at the kubernetes level and what we have. What we have here is that the resource gets congested at the edge and the response time of the same application.

A

That is satisfied from k native and a double islam that they differ a lot. So the lambda gets a much better response time application performance, while ki native because of underlying infrastructure congestion, they are getting a very poor performance. So what we do here is we actually use two anomic, which is a our company that is absolutely serve as a decision-making engine to help you automatically choose the routing bass.

A

Sound is so in this case we identify that there's a resource congestion in key native, and there are same function at the different edge or different cloud that you can actually satisfy the same client. So we will route the traffic from K native to a double anda, so a little bit breakdown with more details, so the service platform, as I mentioned, that you can actually choose any of the flavor in this diagram. So we choose one from the cloud we choose.

A

One from the kubernetes base service platform to showcase, so we believe that if you actually deploy the edge computing, it must be depends on the kubernetes or it must be. The kuben ad must be the infrastructure abstraction, so Kay native. We, we love Kay native a lot, so we picked a native as the showcase.

A

But again you can pick any of the flavor in this picture and then API gateway is really a goo to actually combine the glam-doll is the key native so that we don't have to write actual code to make the sto directly routing to lambda or K native. You just need to have some static mapping configure in call that would serve the same purpose and I will show you with more details in the demo, then the routing layer, which is the east, your part, so so the service match.

A

So just a brief introduction about service match so service match. There are a lot of definition about what is service matched one. For me, one of my favorite definition is that when micro service overlaps with the network functionality, that's where the Civil Service Service manage start to play. His role so, for example, telemetry authentication security. All these kind of things where micro service wants to overlap with the network now service match, takes the control of how everything should work.

A

So there are a lot of functionality that service manager still can provide telemetry policy enforcement traffic engineering. So here we basically take advantage of the traffic routing functionality of sto to finish this demo. So here you can see this Yama file, which we actually create a virtual service object which and create a rule in this rectangular box. So the rule is very simple. So if the app has the label fool, then it goes to the natal okay native.

A

If the app has the label bar, it goes to the holo lambda and last but not least, is the decision-making engine. So we want to make sure that the decision can be made in different aspects. So what so in this case, in this term, the decision is made by if the application performance has been decorated or not, and the degradation can cause by the underlying infrastructure, congestion, so the decision-making you need to make in a day-to-day basis, including scaling capacity placement.

A

So in this Timon we're focusing on the placement so I know this session is not supposed to be a banda pitching, but I really want to just give you some introduction about how this turbine amic works in the back, which I think is a pretty interesting idea that I want to share with everyone, so the secret ingredients. Why so many customers is using coop turbine Amex, because we create an abstraction which is the most important layer of our our system?

A

So so, regardless, what kind of infrastructure people are using, whether it's on pram or cloud, regardless? What type of application people are using application, VM or containers? We actually create an abstraction layer, so abstractions is most important thing in the data center. So, in fact, one of my favorite one of my favorite Kubik on presentation is delivered by the CTO of box, a Sam goat. He actually mentioned.

A

One of the biggest benefits of kubernetes itself is that it creates the abstraction that abstract, abstract away the complexity of underlying infrastructure, so that you don't have to configure persist, volume load balancing differently because you choose different infrastructure so having kubernetes, absolutely apps are abstracts away the complexity and simplifies the operation, because you only need to know how to operate on kubernetes. The same concept applies here, so Tobin Ave actually create the abstraction which abstract away different entities like apps, container VM physical machine fabric.

A

They are all service entity in your model and then every single resource, CPU memory, flow, I, ops, latency, large page GPU, any kind of resources you can think about or any kind of constraint like label, selector affinity and the affinity tense toleration. They are basically a commodities and service entity will buy and sell commodities.

A

So the interesting part is that, on top of this abstraction we apply economic theory. So since we talk about commodity, so commodity is liked. Our real life commodity right every commodity will be associated with a pricing function, so the pricing function of this commodity is the higher the utilization of the commodity, the higher the price of the commodity. So, as I mentioned that every service entity in this market is basically nothing but the trader, so we don't care if it's a container, it's a pod.

A

It's a load balancer, it's basically a trader that buys and sells commodity. So the fancy part of this algorithm, the elegant part, is that we make every trader makes their own decision about their sizing, scaling and placement decision by themselves. So it's a highly distributable highly distributed algorithm that, for example, if I'm a virtual machine, I buy CPU memory from a physical server. That will be my expense and if I find another physical server has a much cheaper bundle of physical CPU memory. I will move myself to another physical machine.

A

That course that will generate a move action from one physical machine to another physical machine. Naturally same time, if I'm running kubernetes service I have some application that is highly utilized because I'm hitting a lot of transaction. The revenue that sort of transaction and the response time that is generating revenue for me and if my underlying infrastructure is not so expensive, that's my expense.

A

My I will be high enough for me to expand my business and I will clone myself that generates a provision action on the other side, if I'm not making enough money, while the underlying infrastructure is congested, I'm easily go bankrupt and cause a suspend action, scaling action. So everything in this model is so elegant that everyone, every trader itself, is making their own decision of scaling placement and sizing to achieve the best return on investment. So everything is economic principle driven so far, so good.

A

So after this analysis, ng, the output of this decision-making engine turbine amic is basically provide continuous, optimization in real-time that drives continuous house emplacement size improvisation, provision across different layers of your IT infrastructure, so container parts VMs every layer, every entity is making their own decision based based on the economic theory based on the economic principle to make their to achieve the best hour. So they will try to place in a cheaper place. They will try to size up or down based on their ally. They was tried to provisional scale based on hour as well.

A

So on top of that, we also provide the capacity management which give you the capability to create a what, if scenario, sandbox, so that you try to create actual workload, try to replace your Hardware try to migrate to cloud or the scenario you may face that you want to change your data center in the future. You can use this sandbox in reality. In the back of this application, all we do does is basically adding supply or adding demand and simulate the whole market behavior and let the market play the magic itself.

A

So so everything is very, very consistent, so now demo time.

A

So so I will I will just so. Let me go back to this slide, so I'm going to show you every every layer, one by one in this demo. First of all, is the civilization layer. So so we use lambda and the K native, and we have the connective building my kubernetes cluster and I hope the network actually works. Well. So in both platforms we have this cool app defined in boast lambda, as well as the creative platform which is running on one of the kubernetes cluster.

A

So we consider this creative as one of the data center that running in at the edge, so both applications serves the same purpose. So this cool app is so cool that all it does is basically echo a message to show where you actually get the silver respond from. So if you actually get the response from km, ativ you'll get a message like hello from cognitive. If you get the server response from the lambda, then you're going to receive a message like hello from lambda.

A

So this is the service service layers that you have both applications running a post platform. Then the api gateway layer.

A

So if we look at the api gateway layer, so you can see, we have deployed a call part as a replica controller and what contests, if you can so, as I mentioned, that the only reason why we try to use coin is because it actually provides a static routing that helped us help us to easily map the core to the corresponding function, to deploy in two places. So if I show you how everything configured in con, then from the route side, you can see that we have two different passes.

A

One is cout app in the cloud which is corresponding to this service. The other is qu app at the edge which is corresponding to another service. And if you do the same query, sorry.

A

If you do the same query on the service side, you can see that the first service that we talked about here, if you map the ID there will be the lambda service. The second is basically to the K native service. So oh it does. The Khan is easily so that you don't need any code change for us in this demo. All we want to show you as a proof of concept of how everything works. So colony is not the most important component here.

A

Oh it does is just a static mapping about how how this routing based on URL works. So it's your part. That is the interesting part. So we do create a virtual service object in the kubernetes cluster, as I mentioned in the slide. So so all it does is to specify the track. So it's to specify the routing rules.

A

So, as you can see so one second, so so, basically the client side from the client side or it does- is trying to make an API call on the endpoint kulap. So they they don't have any idea about whether this cool app will be satisfied by the server running on lambda or by the server running on creative. So all it does is to trigger a request to coop. So what happens behind the scene from the kh2 perspective?

A

Is that sto checks, whether this client has a label fool, then in route to the it overrides the URL instead of cool app, it actually override to cool app at the edge and for for the label with khabar instead of qu app, it overrides qu app in the cloud. So having said that, after this rule, whenever you receive a request based on the label, it will automatically override the URI. After that there was no ambiguity. There can't knows that we're exactly to route this traffic. So what does that mean?

A

That means that what the decision-making engine turbo Nam ik is going to do is that it's going to manipulate the label based on the fool bar to actually manipulate the traffic, so it can always override the client label too, so that it will map to exactly you eye to map the traffic.

A

So how does everything looks like so? If I, if I, do a a curl from one of the part in this kubernetes cluster with label foo, you can see, you will get a hollow firm edge and if you ask the curl from application that with label bar you're going to receive holo from the cloud, that's a setup. So now the decision-making engine. So that's the epidemic part.

A

So I'm not going to show you a lot about turbo now, because it's kind of if we want to do a deep dive, it will take a long time so so just a little bit overview of how this turbine amick look like so the entry point of to economics. Basically, user comes here as the target configuration page to tell us what kind of data center you have.

A

As you can see, this is pretty much the prerequisite to let any decision making engine happens because first step is to understand your topology, your resource consumption, everything from your data center. So we do have all these kind of technology supported out of box. You can add these Saints from hypervisor to cloud to cloud native everything that you can find like kubernetes k, native Prometheus out of box. You can just add them.

A

We use the API, so we support the Vista API and if you don't actually see the flavor you're using, which is quite we're in our target configuration you can actually use our open source go SDK to support it yourself, it's not so difficult. So having said that, we add all the things that we covered like including AWS lambda, kubernetes, K native into tab, anomic, so immediately you're going to see the foods that visibility that combine everything together.

A

So you may see partial view from either of the technology I described, but by having all the data coming to the same platform, I eat ergonomic that we are able to identify, which other entities that actually common to the platforms that we will merge them together. So with that, you can actually see the full staff visibility like from your services running in kubernetes, your K native application, running kubernetes, all the way down to the VMS running on AWS things like this and with the decision-making ng.

A

You can see all kinds of actions that recommended by tool anomic, for example, you can you you want to move a volume in AWS. You want to move a container part in your communities which is actually another functionality. We provide because placement can happen every layer right. So the whenever you find a cheaper placement, you will move your container part the note in the same cluster so, but so all these scaling moving can happen with our analysis, energy, so I'm now going to cover a lot of details about all these decision-making engine.

A

So let's focus on the creative part.

A

So we basically saw the king, so the K native service function in here in all mod data model, it's called virtual application, so you can see it is running in one of the part in kubernetes that is in one of the kubernetes cluster on top of an AWS cluster. So this is how the everything set up so as I mentioned that, since you add all these targets into anomic already, so we are able to give you of stiched full stack visibility in a single pane of glass.

A

So what happened here is that you can see this Kafue, which is the application that running on creative, is actually suffering from the performance, which is because the virtual machine that is in this cluster is heavily congested. If you look at, if you look at the virtual machine that currently hosts this community, if you can see that the virtual memory is heavily congested, so this will cause the performance degradation for any kind of services, because you cannot guarantee that this is the cluster.

A

Only for you right, you will be a multi tenant cluster that you may have some noisy neighbors that will eat or the fight for the con, just fight for the resources which may cause some performance degradation. So this is exactly what is happening in this case, so we see that the hollow foo is actually have a lot of problems with the performance.

A

So what tobin AMEX suggestion, which in the back again, is an economic principle that we find that another provider that is running in lambda can satisfy the same request which can provide a better performance. So we suggest that you should move this fool Kafue from T native to lambda. So on top of that, we actually build this graph on a dashboard to show how the response time looks like before and after the move. So before the move.

A

Let's just take a look, so this car foo actually has a much higher response time than the car bar, which these data are all coming from premises. So now let me just run this one more time before I execute this move, so you can see now this car foo is actually echoing back hollow farm edge. Now, let's execute this move.

A

See this happens really quick.

A

So now it is routing to the hollow from the cloud, so everything is happening by overriding the label using a steel. So we absolutely make the decision only and leverage the orchestrator system like sto to actually performing the corresponding actions that actually can help us throughout the traffic and what happened afterwards. So you can see the performance start to getting much better, because that we try to avoid the resource congestion on your creative platform, which can happen any edge that we identify that you have a similar provider.

A

A similar application that runs in a different platform then can give you a much better performance. So this is exactly what we are trying to help. Okay.

A

So this is a demo, so, just to summarize so I hope you enjoyed this demo because we actually spend a lot of time to explore all these technologies and combine them in one demo and I hope. You think this is interesting because I don't think this is an introduction session.

A

What we want to really achieve is that we want to prove of a concept that edge computing can be really really flexible, flexible and can be really really helpful when you actually try to reach and satisfy your client IOT workflow from the edge, but on the other side of this flexibility, it's the complexity, there's a lot of decision you need to make so the key takeaways deploy and cloud is complicated.

A

Even the service function, because workflow needs to be smart, they need to be resizable, scalable and routable and take advantage of edge computing, there's a much higher bandwidth and low latency. Because it's near you right, you don't have to reach the cloud regions which may be far away from you. You can reach a cell tower, which is much closer to you, and real-time decision-making is critical because decision making today what I show you today is really only about congestion on the underlying infrastructure, but in reality the scenario can be much more complex.

A

The performance is what I show you today later, you may actually consider what about the cost efficiency, as I mentioned, that cross region data transfer is very costly. So at the same time, if you actually can route the traffic to avoid these cross region costs, that may be another criteria to rather traffic differently and compliance. You may face some data sovereignty problems that you want to move the workload routing the traffic to so to make compliance work. So, on top of the demo, we are showing you today.

A

Next we are going to add more dimension in the decision-making, as I mentioned, that there is compliance, cost efficiency. Also, we want to introduce the micro move, which means that we are not going to route the entire traffic from one application provider to another application provider. We want to do a more fine, fine, grained routing, which is like partial traffic, 90% or 80% to another platform, to give you a better performance and cost efficiency and which is also leveraging stos the orchestrator.

A

So thank you. So that's my presentation.