Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Leveraging Service Meshes for Accelerating Serverless Workflows - Paarijaat Aditya & Manuel Stein

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Leveraging Service Meshes for Accelerating Serverless Workflows - Paarijaat Aditya & Manuel Stein, Nokia

Serverless platforms increasingly provide support for function composition (e.g., Knative Eventing, Fission, KNIX). To reduce function interaction overhead within a workflow, platforms may choose to co-locate multiple functions inside a single container along with a custom local messaging mechanism. In this design, the functions of a complex workflow may have to be partitioned across multiple containers, requiring invocations to a downstream function to be load-balanced between its replicas across containers. Service meshes could provide such load balancing; however, they lack locality awareness for the workflow. This talk explores how service meshes can be extended to be co-location aware and to provide a single data plane for both intra- and inter-container messaging. Lessons learned implementing such a co-location aware data plane, its advantages and pitfalls are also presented.

https://sched.co/ekEe

A

Hello, my name is manuel and I'm here with my colleague packageart in this presentation. We want to take a look at serverless workflows and how using service meshes can reduce their completion times.

A

Let me start with the obligatory introduction to serverless. The idea of a serverless platform is that developers can solely focus on writing code. When the code is ready for release, it would be stored for the platform to use it, and we also need to specify what triggers our code. This may be a scheduled execution, a reaction to arbitrary cloud events, a request from our mobile apps from our shops or factory site, just anything that could trigger the code.

A

The serverless platform handles those triggers and manages deployments autonomously function as a service means that the platform creates container instances equipped with a runtime and an event loop that calls the function for every trigger it processes, so the developers would never have to think about servers again, but naturally developers don't just write one function. There may be plenty of functions involved in an application, large and small to produce the desired results. Of course, over time it gets more complex. So we would organize the work into reusable function code.

A

When we put the functions in the order of execution, there might be some branching conditional, invocations mapping or error handling. A single trigger causes a couple of actions to be invoked, all of which we'd want the platform to manage and run for us, and this is what we'd like to call a serverless workflow like any modeling approach. The serverless workflow captures the process in a common notation, so developers and domain experts can agree on a formal definition, a serverless, workflow composes, multiple invocations.

A

It specifies the order of executions and a control flow, and it puts all actions of a workflow invocation in a shared context, depending on the workflow language. Invocations may be called actions, steps or tasks, and the flow may be represented as a task graph. A flowchart or state chart data maintained in the context of a workflow are called artifacts or just workflow data.

A

The cncf has given serverless workflows a sandbox project to create a community-driven workflow language that meets our requirements as application developers, and I think the foundation is the perfect environment to create a vendor-neutral specification.

A

If you'd like to know more about the work stop by the project, spruce or visit serverlessworkflow.io, the workflow language is independent from the runtime implementation of the platform, and we asked ourselves how can our platform achieve fast workflow completion from a workflow perspective? The platform does two things that are on the critical paths. We need to pass control from one action to the next and we need to pass along the workflow context.

A

This talk is organized as follows. First, I'm going to talk about our decentralized design approach and four ways to deploy on kubernetes in search for the fastest and most flexible communication, and second, my colleague parachute is going to discuss how we can group functions to avoid most of the communication and still be able to balance load across resources. We allocate on the cluster, followed by design to dynamically control. The rebalancing.

A

To run workflows, we could do a central workflow engine that dispatches invocations to function workers we want our workflows to complete fast, but that way we'll always have a centralized component. That is on the communication path of all our workflow executions, and it could easily become a bottleneck that we would need to scale separately, so instead we're using a decentralized setup. This way we can hand over control directly between instances and we can pipeline workflow invocations workflow control logic is evaluated at each instance. Let's explore four options to deploy this on kubernetes.

A

Our first pattern are just plain microservices. We could use kubernetes deployments and expose them as services to pass control from one step to another. We can simply use service requests. The event loops here have to be implemented as asynchronous apis to hand over control and not interlock the services to benchmark this. We've set up a series of five steps and we've traced the time to handover control between every step of the flow.

A

The implementation sends a cloud event with one kilobyte of random data. The chart shows a box plot for each of the steps with the time. Measured between making the service request and receiving it at the next step, the median time for transmission across all steps is only 620 microseconds. Our next option is canadian serving using sdos increase.

A

Every canadian service deploys with a queue proxy connective, serving uses a gateway to accept connections for a running service. The request goes through the gateway and q proxy before it hits hour runtime the gateway can redirect requests, even when there are no deployments, which means canadian can scale the deployment to zero instances and spin up cold ones when there is new demand.

A

This also allows for traffic splitting, for example, to migrate load between revisions of a service, but the in direction comes with a cost. In latency we've measured again our series of five steps and the time it takes to hand over one cloud event with one kilobyte of data. The implementation is the same. Only now we're using k native services that have already running instances and no code starts in this case.

A

Half of the deliveries take about 2, milliseconds or less native serving introduces some latency, but we've also moved to a solution that can scale to zero and that can split traffic for revision management, basically using a service mesh to do so. Let's take a look at this third option. Canadian eventing has been around for two years and provides composable primitives to enable late, binding event, sources and event consumers, one of these primitives to define a message passing topology is the sequence: it's a simple pipeline that automates the creation of channels and subscriptions.

A

When we trigger a sequence, each stage makes synchronous calls to the destinations and the result is used for the next invocation. This makes it pretty easy for developers to use. It also provides multiple technology bindings. For example, there's an in-memory implementation, a nuts binding and a kafka binding available for channels.

A

We've used the in-memory channel implementation that handles the communication in our scenario with a single part, but message passing almost always uses a store and forward pattern, and we would expect this to add some latency. So, let's take a look at the benchmarks: we've measured a sequence of five stages with plain kubernetes services, so we're using simple deployments for our runtimes again we're measuring only the time it takes from the event, leaving the service until its arrival at the next service and again we're using cloud events with one kilobyte of data.

A

Interestingly, we found that with the in-memory channel the time between the second class and the last stage in a sequence always has a much higher latency, but since we're looking at the median of all deliveries here, that doesn't make much of a difference. Fifty percent of all the event deliveries take 1.45, milliseconds or less or 1.4 milliseconds.

A

If we left out the last step, our last option combines both the canadian eventing sequence and co-native services, because it gives us the features of both the decoupling and late binding of the eventing sequence and the ability to scale to zero and migrate between revisions of a connective service. Let's look at the numbers again we're measuring a sequence of five stages. Only here we're deploying canadian services and again we're measuring the time it takes for one kilobyte cloud event to be passed from one service to the next.

A

Fifty percent of all the event, deliveries take four point: five, nine milliseconds or less or 4.7 milliseconds. If we left out the last step now, let's compare the four options and let's also take a look: what happens when we transfer larger events, the combination of eventing and serving at a median of four and a half milliseconds for a kilobyte of data, but for two megabytes 50 of the deliveries take almost 90 milliseconds or less the eventing sequence alone. Invoking basic services requires 60 milliseconds to pass on 2 megabytes of event.

A

Data canadian serving without eventing channels delivers the event from one instance to the next, with a median of 54 milliseconds and, of course, basic services achieve the best figure here, with a median overhead of 40 milliseconds to pass on 2 megabyte to the next service.

A

When we look at even larger data sizes, we'll find that the linear trends continue, we can see that the absolute overhead of sequences increases with larger data sizes, which would be due to the store and forward communication pattern when brokering messages.

A

To summarize this part, we can state that every additional indirection adds latency, whether it's a proxy or message broker.

A

Proxying http transactions is faster than using a store and forward pattern, which is especially noticeable with large data when we want to use communication features like complex routing or message queues, it's always better to separate control from data, and we've also got a feeling now how much latency each of the solution causes.

A

Microlig parachute will now present how we can avoid transferring data completely by grouping functions while still being able to balance the load across allocated resources.

B

Thank you manuel. So till now we have looked at communication mechanisms that can be used to communicate between functions of a workflow.

B

We looked at how the overheads of these mechanisms increase as we try to transfer more data between the functions, and we also measured that calling the function directly with the data versus a store and forward approach is actually better from a latency perspective.

B

Eventually, we are trying to look for ways to reduce the overall completion time of a workflow and another factor that affects the completion. Time of a workflow is how you package the functions of a workflow together, essentially how you group the functions together in all the previous approaches which manual presented. There was an implicit assumption that all the functions were running in different containers or parts, and that are not explicitly sharing resources.

B

Now this choice can affect the workflow completion time, particularly uh when there is significant data that needs to be transferred between f1 and f2, and in this case either f1 directly calls a remote instances f2 with the data, or it calls it by reference by first writing the data locally and then sending a reference and then f2 pulls it or it could write to a global data store and then f2 pulls from there, and we have measured the workflow completion time of this approach in an experiment as we increase the size of data that needs to be transferred between the function and the legend here.

B

The file ref means that we are passing data by reference and then pulling it, and data call means that we're calling the functions directly with the actual payload of the data.

B

Now, what is interesting here in in this approach uh in this graph, rather, is that what happens when you compare this approach with an alternate one where both f1 and f2 are residing within the same container or a part that is explicitly sharing resources.

B

Now, here, f1, either directly sends data to f2 or via a reference, but to a local instance of f2, and here you can see, you can clearly see that there is a difference between both these approaches and keeping the data. Local is performing better now. This is not very surprising because there's a lot of research that indicates that we should try to put compute and data together to speed up.

B

So what I want to indicate here at a high level is that there are certainly benefits to be gained if we were to simply relax the assumption that all functions need to reside in different containers, but instead package, multiple functions of a workflow together inside a single container and potentially we can package all the functions of a workflow inside a single container to speed up the overall workflow execution time, and this strategy can work well as long as the container has enough resources to handle all the incoming workflow requests.

B

But now you're also dealing with a larger unit of deployment, which is a workflow with multi. Multiple functions that need to be scheduled together and balancing load in this strategy can become tricky.

B

Consider a situation where you have multiple replicas of the workflow available, and now there is a request going through the top container, which causes one of the downstream functions f2 to start doing some heavy work. The top container may continue to request. More sorry continue to accept more requests, as the initial part of the workflow may not be aware of congestions happening downstream in the downstream functions, particularly if the workflow structure is more complex than that is shown here.

B

But now that we have admitted this request in this container, our choice of staying local would continue to push the request down locally, potentially degrading the completion time of all the requests going through this container.

B

Now, even though we have admitted a request, we can potentially redirect or rebalance this request internally from the middle of a workflow to a remote instance of f2. If that is not loaded, and that way, we can still benefit from additional resources if they are available elsewhere or we can even create resources on the fly.

B

What I'm trying to indicate here is that when we group functions together and then treat that as a deployment unit, one may have to dynamically decide between keeping the requests. Local versus sending some requests down remotely to other instances of downstream functions in other containers, and this situation can change dynamically depending on the load, and this makes the load balancing more tricky and what happens if you now also have to transfer data between functions while rebalancing a request to a remote container here.

B

This request will now have to take the hit of pulling the data first in f2 and then processing it and which may also affect the completion time of this rebalanced request.

B

So we did an experiment to try that out and what we had was that there were already two requests that are concrete, concurrently being processed by the top container. And now a third request arrives at the same container.

B

Now we make a choice between pushing this request down locally within the same container or calling a remote instance of f2, so that we rebalance this request while pulling this data.

B

And then we measure the completion time of this third request, and what we find interestingly here is that the situation is completely reversed, even even under load and depending on your applications.

B

As the size of the data is increasing to be transferred between the functions. It might still turn out to be better to send this third request remotely and take the hit of pulling the data than to push this request down in the local instance of the container.

B

So overall, what I want to say is that co-locating, multiple functions of a workflow inside a single end container can actually accelerate workflow completion time, and we should certainly take advantage of that.

B

But treating workflow as a unit of deployment, particularly under load, will also require internally rebalancing workflow executions from somewhere in the middle of a workflow, which you may not know beforehand where that needs to happen, or this situation can also change dynamically.

B

Now. The communication mechanism that you as a serverless platform, developer choose for communicating between functions should be flexible enough to support these use cases. The mechanism should ideally provide dynamically reconfigurable routing at runtime without having to modify the core logic of your application.

B

It should also provide the possibility of fine grained observability to monitor the situation and should also support configurable load balancing strategies, without your application logic having to do all that now. Fortunately, we don't have to do all of this from scratch. Service meshes, such as onward proxy, already provide most of these functionalities for microservices, and we could adapt this to our use case.

B

Essentially, a service mesh takes on the communication on behalf of a service and provides a large set of tools to do that effectively such as retrying load, balancing active health checks between services and all this can be configured at runtime via a centralized control plane.

B

So when say, when service 1 is trying to call service, 2 service 1 simply hands over the request to the local on y proxy, that is running as a sidecar. The proxy, on behalf of service 1 load balances. This request to one of the replicas of service 2, while providing facilities such as retrying and active health checks.

B

This load balancing can also be configured dynamically via the control plane to say something like that now, 70 percent of the requests to service 2 should go to this specific replica instead of a purely round robin manner or we can even have more complex load balancing policies which all y already provides.

B

So we feel that service meshes such as envoy is a nice choice of communication mechanism that can be adapted to our use case, where there are multiple functions of a workflow co-located inside a single container or a port. It would provide a dynamically configurable and a unified communication mechanism for both intra-container and intercontainer communication.

B

So now, when f1 wants to call f2, it would simply hand over the request to the local on-voip proxy, which would which could be configured to pass down all requests to the local instance of f2. To begin with, and if now there is back pressure exerted by the local instance of f2 say via 503 status responses to multiple health checks, then the control plane could intervene to reconfigure the proxies to now only send say.

B

20 of requests locally and 80 percent of requests should go to remote instances of f2, and this situation can change again depending on the load conditions and may require further reconfiguration.

B

So this way we can see how we can leverage the service mesh as an enabler for accelerating serverless workflows by providing a unified communication mechanism which lets you benefit from locality, while at the same time, being dynamically reconfigurable to support rebalancing of requests when needed, and the benchmark experiments that I presented that measured the workflow completion time without load and under load with rebalancing were actually conducted in this kind of a setup with the on y proxy, redirecting invocations from f1 to f2, either locally or remotely to other replicas of f2 running in other containers.

B

So, overall, in this presentation, we explored different communication mechanisms for function to function, communication in serverless workflows and we measured their overheads. We also measured how co-locating functions of a workflow can accelerate work for completion time. We looked at the load, balancing challenges, this approach creates and the need for dynamic, workflow execution. Rebalancing.

B

We also presented how a service mesh can act as a unified enabler for this dynamical rebalancing situation, while also being leveraged for locality, aware communication, and we have already instantiated some of these ideas and concepts in our open source, serverless platform that we are developing at nokia bell: labs called knx micro functions, which provides workflow support as a first class citizen.

B

We already accelerate workflows by co-locating all the functions of a workflow inside a single container and then load balancing externally between replicas of these complete workflow units.

B

We also provide a k native and an istio based implementation, where our workflows are packaged as k native services and currently within the container hosting the workflow functions.

B

We provide a custom, local message, bus for communicating between functions, but we are moving towards utilizing the on voip proxy that comes packaged with istio to be used for intra-container communication as well, and then further utilizing the service mesh in conjunction with a control plane to provide dynamic, workflow rebalancing with locality-aware communication, so do check us out at knx, dot io and our github repository, which is koenix dash micro function, slash knx, where we are actively developing these ideas and also check out our slack channel, and you can also find the code for all our benchmark.

B

Experiments shown in this talk also at our github repository koenix dash micro functions, slash workflow mesh. So with that we've finished this presentation and we thank you for your attention and we are happy to take questions now.