Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Spark on Kubernetes: The Elastic Story - Bowen Li & Huichao Zhao, Apple

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Spark on Kubernetes: The Elastic Story - Bowen Li & Huichao Zhao, Apple

Apache Spark is a unified analytics engine for large-scale data processing. People are moving Spark and batch workload to Kubernetes due to its uprising popularity. There are many challenges to running Spark efficiently on Kubernetes, for example, supporting autoscaling-based workloads. In this talk, we discuss building a large scale Spark Service on top of Kubernetes. We will also walk through autoscaling on a multi-tenant platform with advanced features such as physical isolation, min/max capacity setting, bin-packing, scale-in and scale out controls, and more. These improvements show significant CPU and memory utilization savings for Spark on Kubernetes.

A

Hi everyone thank you for joining us today. It's our pleasure to present you the elastic story of running spark on kubernetes natively at massive skill for apple. My name is bowen lee. I lead the batch processing and interactive analytics areas of the apple am ammo data platform.

A

My team builds and operates called native services like batch processing powered by spark on kubernetes, interactive data science service with interactive spark and jupiter, and interactive analytics service powered by presto and trino.

A

So we serve hundreds of data engineers and scientists every day to improve our ei and mml products like siri and apple search with best-in-class data and analytics and processing infrastructure, which, how is an engineering from engineer from my team, who has been focusing on how to run spark elastically and cost efficiently on kubernetes?

A

So here's the agenda today.

A

We will first talk about the benefits of cloud and our design principles to elaborate those cloud native characteristics and then the architecture of our cloud native spark on kubernetes platform and why we need to like auto-scale our spark service based on cost saving and elasticity need now, which I will dive deep into design of the reactive, auto scaling and the productionization of it and our learnings and future work sounds good, let's get in there, so um you know why we are moving to cloud it's.

A

This may not be a new topic, but I want to iterate our unique perspectives, um so cloud and kubernetes can help solve lots of the problems of legacy infrastructure have, for example, it is igel, resources are consumed on demand and the user can pay as you go.

A

Second is elastic and scalable we can acquire resources, we need and return them when we are done so that saves us lots of money and the compute and the storage are almost infinite skill.

A

Then kubernetes enables us to build services in a container native way, with strong resource isolation, so users workload only impact each other.

A

This supports our like multi-tenancy and isolation guarantees with cloud and kubernetes. We can leverage cognitive, cutting-edge security techniques to build a privacy first, the data infra and lastly, you know kubernetes and the providers of the cloud took away lots of those heavy liftings from us which enabled our developers to focus on building and improve business critical batch processing service. To achieve you know higher roi.

A

So it's a no-brainer a couple years ago for us to decide to all-in cloud and kubernetes.

A

With the benefits of cloud and kubernetes in mind, we set a few critical design principles for ourselves when designing the system. First, we want to fully embrace public cloud and cognitive way of thinking and building infrastructure.

A

That is quite a message. A mindset, change, for example, uh you know in the cloud native world when we want to upgrade our infrastructure, we don't have to do any in-place upgrade which expose a huge risk to our infrastructure and our users right in the new world, we can just spin up a completely new environment and gradually roll traffic over from our old environment to the new environment and instantaneously switch back. If there's an issue. So that's kind of flexibility is a huge win for our devops.

A

Second, everything should be containerized for elasticity, agility and reproducibility. We aim to scale and replicate our infra very fast to cater to business needs and full containerization enables us to do so.

A

Third, compute and storage have to be fully decoupled, so they can scale independently according to business needs. For example, the shuffle data science can vary significantly from spark job to job, and then we have to build our own spark service, be able to handle that in a flexible way, rather than one-size-fit-all solution, the security and the privacy and user experience I'm talking about them together, since they are related.

A

um You know uh in our new infrastructure, security and privacy are first-class citizens in the design stack and we leverage fine-tuned.

A

uh You know techniques like rows, policies to govern our data services and, at the same time, we still want to make it super easy for users to run smart jobs by following those governance. So, instead of having users to run spark submit directly, we expose a rest api that has exactly the same parameters as spark submit.

A

So we can enforce security at the rest, api layer while still giving users a very familiar development experience.

A

Last, we have apple internal distribution that we decided to use.

A

Next, I want to present our cloud native elastic spark architecture, so you know we can start from the data plan where, in the back end, we have multiple spar, kubernetes cluster. We use the spark kubernetes operator to submit smart jobs and manage these job life cycles.

A

There are a replica set of spark operators, so they can load balance and achieve high availability. Each tenant on this platform have their own resource. Cues powered by apache. Unicorn unicorn plays a few key roles here, like it's, for example, uh one is, um it is a multi-tenancy support and resource quotas. Each queue for each tenant is fully isolated from one another.

A

Second, unicorn queue runs all the resource, scheduling for spark workload. You know from basic ones like gas scheduling, requirements to more advanced scheduling, policies like fifo priority or preemption.

A

Lastly, unicorn handles elasticity of the queues by independently scaling resources for each tenant.

A

We have many of the expired clusters uh in the back end, the multi-cluster and multi-cue strategy pro provide us with many folds of elasticity and linear scalability. Without a single bottleneck.

A

So in the control plane we built our own spark service gateway which exposed the rest api I mentioned before it is itself, is a container native and can be deployed and skilled very easily as a microservice on kubernetes when submitting uh jobs through our rest, api users can specify additional parameters like qname and the skate will route the job to the underlying queue.

A

On the client side, we provide rest api, a simple, easy to use cli for users to run jobs from terminal and a corresponding airflow operator, so users can run scheduled jobs.

A

We have also the data science service, where our data engineers quickly iterate and build their spark etl pipeline and data scientists build and train their models interactively. We aim to share a unified backend for the two spark services. So, as you can see, our interactive spark workload that comes from jupiter notebooks went through its own. Interactive spark gateway and workloads are running on the same infra structure. On the back end. This way we achieved the goal of reusing most of our infra without reinventing the wheel.

A

Lastly, we closely collaborate with our security and privacy team and observability team to develop and uh integrate on those two fronts in a fully integrated way, so our spark service has been running in production for a year so far.

A

It currently supports many business critical workload for apple aml. The development skill is massive. We are running hundreds of thousands of vcpus and hundreds of terabytes memories with supports. You know hundreds of thousands of smart jobs per week. The job skill is also very large.

A

Our users biggest jobs can consume up to you know thousands of executors and cpus. At the same time as it runs for hours, we have been very active contributors to the unique apache unicorn project and have grown commuters and pmcs organically from the team. We are also planning to open source, some of the components in the stack.

A

There being super successful, we initially have been operating all the resources statically for users, um for example, our unicorn queues are of static amount of resources, and we see a massive opportunity to make the stack more elastic and save cost.

A

For example, workload patterns can vary from time to time in a week or even during a day right and- um and they also vary quite a bit from use case- to use case, for example, from running only scheduled jobs to mostly ad hoc and interactive jobs, or mixed of both or occasionally super large-scale backfill jobs.

A

When using a fixed amount of resources, it has to account for the max usage and will cost waste. So we have been investing heavily into auto scaling, spark on kubernetes and have achieved great results so far by cutting down cost for our users by as much as you know, 70 to 80 percent on q basis next I'll hand it over to huichol to talk about how we achieved that and our learnings and roadmap on that direction.

B

Hi folks, this is richard from aimil data platform in apple. Now, let me walk you through the architecture and the design of which is reactive, auto scaling feature in our cloud native spark cluster we delivered recently.

B

First of all, let me talk about the auto scaling cluster node groups layout as a multi-tenant of the skating cluster. We provide physical isolation among system components, spark driver and the sparca executors, and each of them are located in their own node groups.

B

Here the system component, including such as node problem, detector, ingress controller spark, kubernetes operator, unicorn and so on.

B

Also by mapping different tenant queue to their dedicated executor, node groups. We can oscillate different tenants from each other to minimize the potential impact and also help us to generate the cost usage reports per tenant very easily.

B

We provide the mean capacity setting per q, so there is amount of guaranteed machine that keeps running over there to support the long running and the smaller cadence in workload.

B

The maximum capacity setting can provide a guide reel for each queue and workloads will be weighted in a queue if they are exceeded the maximum threshold. Until there are related resources, our scheduler find.

B

This is the workflow, however, clutter size being changed based on the spark workloads per node group when users submit their job to our gateway. The skills service will create the crd on the corresponding cluster. Firstly, it will create the driver pause on driver node group to make sure the job can always be scheduled and then x. Guild reports will be created by spark operator in the pre-assigned node group, scheduled by unicorn.

B

We can also see once kubernetes clutter, auto scaler find the pending part in whichever node group. It will talk to cloud provider to scale out the suitable numbers of nodes in the specified node group here, which is mapping to our unicode resources queue vice verse wants you to find that there are idle nodes. It will terminate the node to save the cost.

B

Beside this, we also provide some customized scanning control to our auto scaling clusters for skill in control.

B

Our backing will only apply the skill in on executor, node groups and the scaling process will be triggered only when no running executor port on the node.

B

We have enabled beam packing provided by unicode to minimize the number of instances to use the default allocation policy of the scheduler will try to evenly distribute, distribute the support to all the nodes.

B

The bin packing policy can sort the list of nodes by the amount of available resources, so our scheduler can efficiently allocate the parts to the underutilized nodes, firstly and zinc to the idle nodes. So cluster autoscaler can trigger the scaling in a very efficient way.

B

The the right hand are ec2 machine utilization dashboard. The top one is the matrix of static queue without beam packing. We can see most of cpu and memory utilization is only around the template page. Only a few of machines can approach to 45 days.

B

The bottom dashboard is shows a matrix. After being able to be packing on auto scaling cluster, we can see there is a pretty good usage rate on both the cpu and memory compared to the massive wasting before.

B

Regarding to the skill out control, we provide a skill out only feature to the bucket driver node group, which all our users always get their driver paws launched, so they also can check their logs over there. Always we also speed up the skill out latency by tuning some spark configurations.

B

Now, let me talk about our production status. With this new feature till now, we have embodied more than 19 internal teams to our auto scaling clusters for more than three months so far, and the average cost saving range is around from 20 percentage to 70 percentage.

B

During migration, we have found that all skilling events works as expected, and the machine will not be removed. As long as there are running or active specular parts, the scale out latency is consistent, which is keep lower than five minutes here. The maximum scale out range we are talking about is from two to two hundred machines.

B

Moreover, auto scaling feature can work with various uh type of resources usage patterns such as ad hoc etl and mixed patterns. Meantime we also found that compared to the massive over provisioning approach before runtime of workloads with auto scaling enables may increase. However, this is expected, which is due to the very good usage rate of cpu and memory compared to the maximum wasting before, given this user need to take this into consideration and optimize their jobs. If there is a strict data delivery time required.

B

I know we have covered a lot in this short time. Here are some key takeaways dealing with develop and deliver this new feature on our platform, physical isolation and the ming max capacity is very important per customer requirements. We can leverage node group mean and max settings and unicode resources quarter. Setting together to achieve this, it will help us to support budget based control going forward.

B

Auto provides guarantees that no impact to existing smart jobs when skilling happens is the most important feature for production jobs. We need to apply some customized skill in control based on different node group types to provide this guarantee in time. We also need enable pin packing to improve its efficiency.

B

The skill out latency is important to large scale jobs by using the dedicated driver, node group and the tuned spark configurations. We can keep the scale out latency as low as possible.

B

Forward there are still lots of improvement. Areas need to be explored, such as how to support mixed insulin type per cluster and how to fully support dynamic allocation.

B

Support instance is much cheaper than on demand instance, which we are using right now.

B

It will be another big win if we can support it with the help of remote shuffle services or similar disaggregated, compute and storage architecture, then we can trigger the skill in more aggressive and even separate the computation and the storage independently, with different off scaling control in future. How to provide a predictive, auto scaling feature to the platform is another interesting topic.

B

That's all today's sharing thanks for your time.