Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 24 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Delivering Serverless Experience on Kubernetes: Beyond Web Applications - Alex Glikson

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Delivering Serverless Experience on Kubernetes: Beyond Web Applications - Alex Glikson, Carnegie Mellon University

Knative is a great platform to host web applications on Kubernetes, offering seamless request-based scaling as well as scale-to-zero. As part of research at the Carnegie Mellon University to better understand the advantages and limitations of serverless platforms, we assessed Knative for two workloads that may differ from typical Web applications -- Machine Learning inference (ML) and Interactive Computing (IC). We focused on properties which are important for workloads with significant amount of state - such as ML model, or IC session - while still maintaining the ability to scale down to zero (and back to one) efficiently. In this talk we will present the initial results of an effort to design a Kubernetes-based serverless platform optimized for such workloads, focusing on two properties: 1) ability to efficiently persist state, 2) ability to deliver low latency under load spikes.

https://sched.co/MPem

A

Hello: everyone, my name, is Alex and I'm, going to talk about serverless computer platforms on kubernetes and specifically on some experiments that we did or building such platforms for different kinds of workloads, and this work has been done. Collaboration with some students and the Carnegie Mellon University, where I'm currently at and it started when it was back in IBM Research in Israel.

A

And so I'm going to start with, let's say yet another attempt to define what serverless is, or at least in the context of this presentation, what do I mean by serverless and then going more specifically to serve this compute, because there are servers nowadays or services applied to different areas, not just to compute I'm going to try to distinguish between like the role of function as a service in this space, which would hopefully also help us understand what other kinds of service platforms could make sense besides functions as a service and then I'm going to talk about a couple of use cases that we try to implement one dealing with interactive computing in hope, and hopefully we'll have time for in demo or at least screenshots, and the second use case is deep learning and so I guess the the advantage of being the last is that we probably don't have anyone after after this.

A

So if folks are interested, we can go into in more detail because, like presentation contains more details, that I will be actually able to cover and it is available or online. And so, let's start with their server lists, so there are lots of definitions and what server this is or isn't. It is extremely popular and there are kind of attempts to give it different definition. So, for me, in a nutshell, server this is so it started with not managing servers.

A

But to me it's very kind of an example, so it doesn't have to be about not managing servers. It's more often not doing some tasks which distract the developers from developing their actual core business capabilities, which is typically writing code and those tasks can be outsourced to someone else which could be a cloud provider could be ITIL IT department or if a person has multiple personalities, may be the same person doing some other things.

A

So it's essentially tightly related to separation of concerns and being able to outsource as much as possible to someone else and the crucial part here is that to find this, someone else who you can trust and they can actually do this well and it and hopefully for them. This is their core business to manage the infrastructure or to provide services that you are using in your application.

A

So this is a bit broader, I think then, for example or lambda, but hopefully it will help us understand what other kind of solutions make sense from this space. So essentially a service is about being distraction-free and separation of concerns and.

A

So all of those are so this is actually not. You know right, so the separation of concerns and focusing on your business logic is something that is around for many many years. So there are many iterations of the same approach. So so why now service is so important, or specifically, let's say Amazon um Dan White is so so popular, so I think it's more related to the developer experience than to actual programming model. So so that's right.

A

There are some innovations in the programming model space as well, but to me this is essentially boils down to developer experience and efficiency and the maturity of those services that you could outsource to. So that's kind of a side note, but essentially serverless is about writing code and doing someone else doing the other. The things that you are less proficient in so a couple of examples which I'm sure that you're familiar with so I'll start with non compute examples.

A

The object store, is considered or is considered the server this offering because you essentially you're just focusing on managing your day time and you couldn't care less.

A

What exactly the Amazon s3 operators are doing at the back end on the so there are lots of destruction that you as a developer, that you don't need to worry about if we take micro services and is another example, so here the core is to develop the actual services interfaces, how they interact and so on, and there are lots of other things you that you prefer the platform to give you that you can just use and don't worry about, and this of course also includes things like DevOps.

A

How do you deploy things, how you manage versions and so on? And the third example here is about two event-driven or asynchronous workloads with things like Amazon lambda.

A

So once again here the core is to build the actual event processing logic and to define what kind of triggers should trigger what kind of events, and maybe some workflow and everything else. Ideally, you should get from a platform that someone developed for you that is scalable, efficient and so on.

A

So so this is service in general, an hour if we zoom into the compute space. What would be the the core and the distraction elements in the if we are focusing on compute?

A

So essentially it's I think the most fundamental part is that servers, compute platform hosts some code that the developer writes and if brain bring your own codes approach then often, which I think makes sense. It is optimized for a particular kind of workloads or application patterns.

A

So historically it it is often related to event-driven applications and their elasticity is scaling to zero and so on. But I think that's not exactly I mean that's, certainly nice, but that's not something which is very fundamental to the definition of services and at least in my opinion- and there are many platforms when, for example, scaling to zero. Many applications where scaling to zero, for example, is not that important.

A

So if, if you're, if you're running a web application, that has millions of requests pretty much all the time- and you have thousands of thousands of servers, then probably scaling to zero is not that important. But elasticity in general is important.

A

So I think the the idea here is to understand what kind of applications you want to run on this platform and to design the platform in such a way that it will be optimized for the applications that you are targeting and for the usage part in Detroit, targeting and in terms of destruction that we would prefer to to avoid.

A

So there are two areas that are important: one is related to management of that code, that the developer, writes deployment, scaling, monitoring, updates and so on, and another area is integration so that so the more we outsource capabilities to someone else, the more integration aspects we have with other pieces of code. That may be hosted on other platforms, which could be event sources, or some integration, around data communication, and so on.

A

So and now I'll try to to see how this definition feeds that the function is a service and few other examples that I'm going to dig deeper next so for function of the service or so-called general purpose. Functional service, like lambda or various kinds of functions, was public cloud and open source.

A

So we have arbitrary functions that we want to host somewhere. We want to just develop the function, push it somewhere and then the rest will happen automatically.

A

Then I came up with this relatively long definition of which what seems to be the deep time, optimization point for for this sort of workloads, so those functions are short-lived, but probably not too short, because otherwise you will be like the scaling granularity, for example, in lambda, is 100 milliseconds. So if your function takes one millisecond and it's probably not too good your feet and also charged because per 100 milliseconds from our own triggers triggered by events, requests so scaling to zero is related to or elasticity and specifically scaling to.

A

Zero is related to the ability to accommodate high variability of loads, including some items. So if you don't have item s and you probably don't care about scale to zero and so on, and also once again based on the existing implementations, may most of them are not very low latency, although there are some efforts to improve it, but essentially I. Think majority of this functional service workers today are somewhat are not very sensitive to latency.

A

So there is back to the question about to the point about developer experience, so it was not entirely clear to me whether this particular kind of applications is suddenly so important that people are so excited about function as a service, but still I, think it moves the industry in the right direction of building this sort of platforms.

A

So, of course we want everything to be fully managed and what is different about functional services that we have functions and functions with invocations as first-class citizens, which was not the case in most of the previous platforms, so, for example, for monitoring on tracing. This is a quite different model and integration here.

A

The the most important integration points are around event sources, and so so that's a function of the service, and now there are other platforms which also deal with functions but in a more specialized context, suggest as an example, there are various solutions to run functions at the edge of the network like problem functions or lambda edge, so it is called functions, but it is targeting very different kinds of applications and well, although those platforms are not open, I would guess that they have very different design decisions under the covers.

A

So still you want to run arbitrary code, so maybe you don't need as much of a variety in terms of programming languages, but here the the requirement is very much focused on high throughput and low latency. So, for example, I'm pretty sure that they are not provisioning, docker containers for for each packet.

A

So we still want them to be fully managed, of course, and in terms of integration, their main integration goal is to integrate well with the actual platform that those functions are running on. So, as you can see here, well, it's called function is a service I mean it can be considered, there's a special case of functional service, but with very different set of requirements.

A

And yes, so there are some efforts in the open source that try to address. Also this kind of use, workloads but I think maybe over time there will be more such efforts.

A

And now the next example is also from a different kind of workload. So if we, if you look at there, the ETL space so Amazon, for example, has in a service called or AWS bloom, which essentially provides a server less hosting for for ETL for big data processing functions. If you wish, it could be as far something that the SPARC job that are they host on their cluster or it could be just plain Python jobs that you can use, for example, for pre-processing.

A

So here again, so you can think of from the developer perspective. They write some code, they want to host it somewhere and they want to be able to trigger it similar in their simple way and to integrate it with theirs with the ecosystem. But once again here the the application pattern is very different and also the the platform underneath is probably somewhat different. So even for the for the plain Python jobs well.

A

Well, one could think maybe you could run those Python jobs on lambda right, but well, that's not exactly the case or because, for example, those Python jobs can have I, think 10 gigabytes of disk space available, which is quite different than 500 20 megabytes on lambda and also in terms of runtime. Those jobs are not limited in terms of fun time and I'm, pretty sure that the entire architecture on below those or that offering is very different but well I, don't have inside information.

A

So well still, we want to all of that be fully managed, of course, and also the integration here is more towards the data catalog and there the access to data.

A

So the last example before, where you moved to our experiments, is a native essentially which I'm pretty sure that, if you're interested in server list that you're familiar with it, maybe I already heard some sessions this week.

A

So cane ad with these are essentially yet another example of service compute platform, which is not exactly functional service, because here the code is essentially a web application, but primarily HTTP based, although now there are some extensions to handle other protocols and the application button here is also very different. So we're conceptually we're talking about long-running applications and to scale out that you can have different number of containers and what is important here or why or so?

A

How can we not deal with the scalability, for example, is that we are assuming that there is some linear relationship between the research demand and the request. So there is request based scaling and with such modular it is typically, it is typically a good fit for traditional web applications, which are high, throughput, low latency or so.

A

Essentially there, the K native, takes out many of the destructions related to scaling to build and to ingesting data or events into the service and leaves the developer to focus on and also creative takes care of, upgrades, blue/green deployments and so on, and the developer focuses on the actual logic.

A

So the question is what other kinds of platforms or what other kinds of application patterns could require slightly or very different platforms. So hopefully we can run all of that on top of kubernetes. So hopefully, kubernetes is generic enough to host all kinds of workloads, or maybe some of them will require some some low-level changes and we've experimented with two kinds of applications in this regard.

A

One is the interactive computing, so this is sort to main use cases here that are inspired this. Those experiments, one is just developer working with some VM in the cloud, and the second experience is the second use case is a data scientist, so data scientist using, for example, Jupiter notebook, so most of the time that Jupiter notebook is not doing anything because it's an interactive workloads, but yet at the moment there are no good solutions to essentially to reuse or recycle those underlying resources in a kind of service manner yeah.

A

So this is an example of Jupiter notebook.

A

Is there here who, who is familiar with Jupiter notebooks, raise your hand? Okay, so it's in their interactive video web application with some with a web server the backend, so you can essentially work with different programming languages and at the backend, the Jupiter server runs something some sort of sandbox called the kernel that actually runs the the text here that the user enters in the UI.

A

So the way it works is that the Jupiter server the web server here sends the actual sell here as a payload to the colonel Colonel runs and some sort of interpreter and returns back the result.

A

So what we've we tried to do and to see how we can decouple the architecture here so that the actual compute happens on a shared cluster, using on-demand resources and not alone, groaning colonel, so colonel today is a long-running thing and the question is: how can we make it more serverless or more elastic in a sense or more or capable to scale to zero in this case, because here, unlike like web server here, we know exactly when the user needs those resources or doesn't need those resources right, because we control the the entire experience here and unless the user clicks the run button, we can know for sure that those resources are not needed.

A

So there are two architectures for in general, so one that leverages a spark cluster at the backend, and this is not what we're targeting here, because maybe not everyone wants to write their programs using spark. There are lots of general-purpose libraries, so maybe your scale, our scale out, is less applicable here and maybe a model that could make sense here is actually scale up.

A

So maybe it could make sense to allocate different amount of resources for a particular Python process, as opposed to scaling to the cluster and if one is just working with, for example, a linux shell, it's somewhat similar and I hope you can see the similarity here so try to use the same definition here for this kind of platforms.

A

So the code here are those cells or the actual command that the user enters.

A

So the application pattern here. So it's a iterative invocation of stateful non parallel computation intensive ad hoc tasks triggered by explicit user instructions. So there are a few things here that we can use to derive some design requirements or architectural requirements.

A

One is related to iterative invocation of stateful tasks, so we probably want to persist the state across invocation sufficiently as opposed to, for example, everything every time retrieving, data from s3 and then writing in back to history. The second aspect here is related to non parallel and computational intensive, as I mentioned. Scale-Up could make sense here.

A

This third is ad hoc tasks, so that we probably want to consider say sending the code as a payload as opposed to put it as part like deliver it as part of some sort of C ICD pipeline to image registry and so on, and the last one is so because this is a ad hoc and there there is a user interaction here. We just probably it probably is important here to scale to zero when, when I, don't so, and so.

A

So we'll build a proof-of-concept of this sort of such a platform that we call the run box is actually uploaded it to a publicly reported a so it's a proof of concept. So it's essentially this box, so looking at the architecture. So if this is the developer machine, so there is the this run box proxy, which in this case in our implementation, is just a Python script.

A

There is a user that interacts with this proxy, either directly or indirectly for examples through Jupiter kernel, and there is a file system which is which keeps the state that is probably important to synchronize with the cloud and we're essentially managing which we call the run box on a kubernetes cluster, which is currently implemented as just a simple or deployment with of size, 0 or 1. So we have a pod.

A

We have a container within the pod, but actually the more important part is the the data volume which is persistent as opposed to the pod, which is ephemeral. So another thing which is important here is that we want to be able to resize this pod essentially do a vertical scaling of this pod. But as opposed to a vertical scaling like VPA kind of vertical scaling, which just keeps statistics of your resource usage and decides what is the right size based on some statistical or history?

A

So here the the policy should be different, because it is somehow related to the thing that the user is doing at the moment and not the history of the last hour. So at the moment we're just doing it explicitly. So the user can specify how much resources they need for the particular operation. And so we can just sorry.

A

So we can just just scale the deployment or there is someone when work on in place, pod vertical scaling, which actually could be very useful here, because it would do the vertical scaling more efficiently and we actually managed to integrate the two, and indeed it works faster than the regular scaling. Naturally, and there is also a controller which well, we haven't implemented yet which, but conceptually the controller could decide when to scale those individual pods and their run boxes based on some policy, for example, based on available capacity and so on.

A

So let's see a couple of screenshots I'm, not sure we have time for an actual live demo. So so, with once we create the run box, so we can specify an image that will be running in the actual worker container.

A

So what happens under the covers is just essentially used cube, CTL, create to create a deployment and and the pod within the deployment. And of course there is also replica set in between.

A

There is the actual worker container and the site card that we are using for our purposes and we can just run commands there as if it was like SSH, for example, and the difference here is that that pod is ephemeral, so it can disappear and come back live based on the policy on based on some specific triggers, and this is transparent to the user.

A

So the next thing that we implemented was the file system synchronization. So we can say: okay before I'm running this LS command. I would like to synchronize the folder here to the remote volume, so at the moment we're we're using the sidecar container with unis on tool to do it, but it could be nice if kubernetes could support something like that. Natively like remote access to your data to volumes, and the next thing is what I mentioned before.

A

We can actually use the cube CTL to scale that deployment to zero and yet till the next command will work. Well, it will take a little bit longer, but it worked. It will work because it will scale it back to one and then it will run it and it will connect to the same persistent volume as before.

A

And we can also specify how much we can essentially resize the container on the fly per command in this case. So you can see here we're just displaying the see group memory limit.

A

Like with regular invocation and then we're allocating factor of two of resources- and we can see here that twice more memory and once again in between the container could the actual pod could be recycled or not depending on the policy.

A

So another thing that we did is integrating it with Jupiter. So we implemented a couple of kernels that used the same Python script at the back end, so the kernels essentially wrapped the code that arrives from the from jupiter server from the cell and send it to the to the run box and by the way, at the moment, we just use cube CTL exec to run stuff in the in the container.

A

So, for example, you could have a very simple or Python codes and here that to assign some value to variable a and then we can go and we can.

A

We can recycle those resources so that we don't have any pods running and then we can come back to the Jupiter notebook and the same variable is still available. So, in order to get it, we had to tweak a little bit the actual Python interpreter session so that it saves the state and then we restore the state, but the proof-of-concept layer level. It seems to be working fine and, of course, the underlying infrastructure is entirely ephemeral from the compute perspective.

A

And here there is an example related to resource allocation, so there is some little bit more complicated computation here of some combinations of letters. So here are partly there are not enough resources and to do it and then, if we resize it to four times more resources, then we can see that it actually works. So, of course, in real life. It might be nice if this exact factor could be somehow computed there automatically, but at the moment, even just letting the developer to specify it per per cell.

A

In this case or sounds like current useful and so.

A

There are some more details on the architecture here which I think I'll skip and.

A

So, maybe just very briefly and on the deployment use case, so this is very different. So there are. There is lots of research on how to do deep learning. It is useful in lots of different applications.

A

Model training model inference, and all of that is very resource-intensive and lots of new algorithms new frameworks.

A

And so the question is you know this is an example application and we want to focus on the classification side so which is kind of well behaving web application in nature, but and we were trying to use K native to implement it in a kind of serverless manner and which we encountered some issues so specifically at the two issues that we encountered were around the ability to use GPUs, which is not something which is trivial or just not enabled, and another issue was around the actual persistence of the deep learning model that is needed for classification, which is essentially the cold start.

A

So the cold start of the containers in this case is about 20 seconds or even more so, with all the efforts to improve the connective side of the house still for certain applications, the scaling, the actual provisioning is the dominant part. So so what we did I'm skipping this just going to the architecture.

A

So if this is our like the GPU nodes and the CPU nodes of the deployment just doing and blue, it has nothing to do with going to the deployment and we want to run some sort of hybrid service. On top of that that leverages pods, which GPUs and pods without GPUs.

A

So obviously, we cannot just have a 1k native service on top of those heterogeneous resources between, because scaling will not work. The scaling should be done differently.

A

So what we did in a in a prototype that we built was to use different native services for each side and to have a load balancing on top of that which is aware of the actual workload, and it would send some requests to this side and some requests to this side based on some policy. For example, typically GPU based classification requests will be faster or it will be more efficient if we have a large batch of requests.

A

But if we assume that amount of GPU nodes is limited and we have other applications that also want to use those resources, there is a contention here and GPUs. So it could be nice if those resources could be somehow associated with priorities in kubernetes, so that kubernetes will somehow help sharing those resources among different workloads.

A

But this doesn't seem to be the case at the moment, so we had to implement our own scheduler, which did kind of arbitration between services and provide provided some sort of fair sharing between those limited number of GPU resources between different services and the second part that I mentioned regarding cold start.

A

So we addressed it and we also try to address it and the way we did. It was inspired by function of the service solutions that sometimes many J I maintain standby containers to run functions.

A

So we essentially implemented a similar mechanism for creative or maintaining a pool of running pods and based on some policy and some coordination with the scaling of the individual services instead of provisioning, a new pod, we just resigned running pod to a particularly native service.

A

So partly it was not terribly difficult, so we had to do some tweaking of labels, and we also, we had to tweak a little bit the history of routing and their stats mechanism to make it work, but in general, so it seems that this could be a reasonable approach, although maybe there are other ways so here essentially, we had to implement our own logic around sharing of those standby pods between services, and it could be nice if, for example, or kubernetes, priorities could be used to let the regular scheduler to decide, for example, when those pods needs to be added or removed.

A

So if, for example, we could, as part of the individual services, creative could just run more pods than we asked if there is enough if there is free capacity and kill those pods when there is less capacity. So, although this this approach worked it, it seemed that there might be some need kubernetes native solutions that could work better.

A

So there are some design details, so this is pretty much it so, just to conclude, our service is about being destruction, free specifically for compute its round. Remember, you're, doing your own cold, plus distraction-free.

A

There are different requirements for different workloads. Probably there is no one-size-fits-all solution and there are lots of opportunities to build this sort of platforms on kubernetes Boston. The examples that are explored here for deep learning or interactive computing, but I'm sure that there are others and that's it. Thank you.