Cloud Native Computing Foundation ArgoCon 2022, 21 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Running Cloud-native Spark Jobs with Argo Workflows - Caelan Urquhart & Darko Janjić

Description

Lightning Talk: Running Cloud-native Spark Jobs with Argo Workflows - Caelan Urquhart & Darko Janjić, Pipekit

Companies with large computational workloads often use Apache Spark combined with numerous Python packages such as PySpark, NumPy, MLlib, XGBoost, and more. Unfortunately, as teams add the number of jobs running on a single Spark cluster managing dependencies becomes a nightmare. Kubernetes makes it easy to use numerous packages for large data jobs in distributed environments, and Argo Workflows is the best way to run pipelines on Kubernetes. This talk demonstrates how to orchestrate common Spark jobs with Argo Workflows, from the architecture to resource and workflow definitions. We'll show how to provision Spark and Argo Workflows on Kubernetes to process large data jobs. We'll also show how Argo Workflows and Kubernetes provide distinct scaling and stability advantages for Spark users by running some example jobs. We hope that listeners of this talk will learn the pros and cons of orchestrating their Spark job on Kubernetes with Argo Workflows, instead of traditional local or cloud environments.

A

So today, we'll be running some spark jobs on kubernetes with Argo workflows.

A

What we'll cover here we'll be learning about the scaling and stability advantages of running spark jobs on kubernetes versus traditionally running them on yarn, we'll learn how to use Argo workflows, to deploy and automate spark jobs on kubernetes and not just run one job but run multiple spark jobs successively or in parallel, and once we're running our job on kubernetes.

A

We will also learn what else is possible to do so, a little bit about us, I'm, Kalyn, co-founder and CEO of pipekit, who you probably just heard about, but at pipekit we use Argo workflows to scale data pipelines for Enterprises.

A

We help them with a SAS control plane that sets up Argo workflows and gets them running on production using kubernetes to build faster and have more reliable data pipelines without having to know everything about Argo I'm, a contributor on the Argo, workflows, project and I was supposed to give this talk today with my talented teammate Darko, but unfortunately he couldn't make it from Serbia. So just shout out to Darko for the contributions on this talk.

A

Let's get into background of the tools that we're using so first Apache spark. So it's a really popular Big Data compute framework, that's open source, so, in addition to being super performant for batch data jobs, it's actually a really versatile tool. So that's why it's maintained its popularity over time, even with upstarts like dask and Ray, coming onto the scene that are really even better at doing batch jobs.

A

But spark is possible because it's versatile, you can adapt, use the framework to also run streaming jobs alongside your batch jobs has a handy machine learning library creatively named ml lib, so you can transition right from ingestion and data processing into machine learning and do that with a bunch of common languages. So we work with a lot of companies that use spark alongside all these other Frameworks and do it on kubernetes. So we want to talk about a bit of the advantages today, because more and more spark jobs are moving away from yarn.

A

There there's a bunch of issues that we run into when we Deploy on yarn first is requiring Global installs, so the risk of sharing libraries between your spark jobs on one cluster is is really hairy to deal with, and often you want to use a different spark version for different apps that you're building and not have to go back in time and refactor Spark jobs as versions upgrade, but this isn't really possible with yarn.

A

So that's a big advantage of adopting kubernetes, of course, Docker using Docker natively is another big advantage of moving kubernetes on yarn. You have to do a lot of leg, work to be able to containerize your applications and take advantage of the faster Dev experience and the improved dependency management that Docker offers again. There's there's a resource benefit as well. So, instead of running a jvm on each node, you can take advantage of kubernetes PODS and then Auto scaling is the big final advantage that we really want to take advantage of so like.

A

How can you use different CPU or GPU thresholds on the same cluster, and how can you adapt to spiky data jobs depending on the workload you need, but we often ask just why use kubernetes for data scientists, data science and to quickly touch on this for data teams who are newer to kubernetes? Often when you use containers as you're building blocks in the pipeline, you get reproducibility of the pipeline, so you can share parts or the full Pipeline with different people on your team and you get improved reliability given better dependency management.

A

Kubernetes also gives you a declarative approach to pipelines, so this is much more advantageous than so that you can essentially Define what outputs you want and let kubernetes handle the auto scaling, and so that's where vertical Auto scaling comes into play.

A

Kubernetes is a great complement to the horizontal scaling that a framework like spark offers you, but now introduces the ability to also vertically scale, depending on the compute that you want, and finally, we'll talk a little bit later about how the cloud native ecosystem can come into play to bring your data pipeline fully to production. We've talked a lot about Argo workflows today, so we'll be using Argo workflows and the big benefit here is it's generalizable.

A

So beyond just running your batch jobs, you can now use Argo workflows alongside spark to then trigger ml training, jobs, trigger a model, serve model, surveying or deployment, and ultimately we're using this as like the workflow engine, the orchestrator to automate many many smart jobs, and instead of having to run these manually, taking a look at what the solution looks like so on the cluster here we'll be deploying Argo workflows and the user would submit a workflow to Argo in the Argo namespace Argo then talks to the kubernetes API to schedule that workflow, which is going to be a series of spark jobs.

A

Kubernetes starts the workflow and then the workflow starts to step through and submit spark jobs for the kubernetes scheduler to then spin up the spark driver, pods and then spin out the executor pods for the driver to then assign tasks to, and the big Advantage here is number one workflows. Argo workflows is managing the state of these jobs. So that's how we can then begin to string jobs together into a longer Pipeline and the second big Advantage is kubernetes is handling this.

A

The spinning up and down of all the executor pods so as pods are no longer needed. Kubernetes will spin those down and save you resources and then, once the driver's done it spins that down as well, so you're not holding any sort of resources idle, so yeah. All in all we're really seeing the these the the problems we saw initially solved by moving container native, so we're now running different spark versions and different jobs on the same cluster. We can manage dependencies more easily and we can stop downloading those dependencies at runtime.

A

The other big thing that we get is now lower resource requirements for running our jobs and auto scaling vertically, but wait there's of course more that we get so the big next. The the first thing that we want to call out here is it's duplicable, so this architecture can be used. If you have a team that wants to move Beyond spark um start adopting or adopting task array, you don't have to completely set up a new cluster.

A

You can spin up a similar architecture on the same cluster run some jobs with spark some with task and Ray and all and use our workflows to orchestrate. All of that, you can extend it so now you can bring in other data processing or machine learning tooling, into the same cluster, which we'll talk about in a minute, and you get a bunch of other benefits by being Cloud native.

A

You know you're agnostic and you can start to pull in other tools like Prometheus, for your logging and metrics, so quickly, I'm going to jump into a demo and like how we get this done. Let's see.

A

So what we're going to be running here is a couple of basic spark jobs, so we have a bike share data set that we're pulling off a kaggle, we're just doing a simple map reduce so we're going to count our bikes by bike type and total up their um their the amount of hours. We've rode.

A

And then what we're going to do is create an augural workflow. That is a dag that runs these two spark jobs in parallel, so we're going to run the bike type count job and then we're going to run the ride length job in parallel. So what we do is we create an Argo workflow and we start to define the dag steps so, first down here we're going to define the bike type step, we're going to define the um this. The success and failure conditions which again is Handy for when we want to.

A

um We want to string together tasks depending on how a step completes or fails. We then deter. We then designate that this is going to be using the spark operator.

A

We vents, designate the language so we're using Scala for this job. Sorry, the videos blowing up on me here we're going to set the mode to Cluster, which we have to do for running spark on kubernetes.

A

We also deter we um put in our Docker image here and we set our main class, which is the Scala file that we're that we defined our job in and then we set the jar location and the spark version. So this is where you can change. You know within the same workflow, you can have one step that runs version 3.1, one step, that's running version two and then down at the bottom. Here we start to Define our resource requirements.

A

So, first for the driver um which we've set up here to run locally and then next for the executors and of course we can scale these up if we want, if we're going, to move to our AWS account and move off a local, but for now we're just going to run two instances here, which would mean that we're going to spin up two kubernetes pods to execute each spark job.

A

Let me just skip so now what we've done, let's see, go we're going to take our workflow, put it into Argo and run it.

A

And now we can see our dag that Matt's back to our workflow file. So we have our bike type count and our bike length count, and this is going to run and now what we can do is we can go into Kube, CTL and monitor it.

A

So we see that we have our driver, pods, two driver pods and two executor pods for each driver once they complete I'm, going to just grab the logs here, and we can see that we get our output of um our count for byte type and then we'll also grab the logs for our ride length by bike type.

A

And since this is a demo, we're just using the print statements there, but that's an example of how we can just run two spark jobs in parallel on kubernetes with Argo. Now we're going to shift into how we can actually parameterize that workflow file so that more people on our team can run workflows on their own by just simply passing in arguments.

A

So, for instance, this is an example of a workflow that we're actually going to be using the workflow template ref to call in a spark workflow template and instead of writing all the yaml that we did before. We just simply pass in template parameters that we've defined in the workflow template. So, as a data scientist, we can just Define what our Scala job is going to be the file the image and where we want to run it and what resources we need to run. It.

A

And so what this looks like on on the workflow template side is a bunch of input parameters um that map back to our original workflow, but instead we set some defaults and we allow now any user to adapt this template depending on the spark job they want to run so just jumping back into um next steps. So now that we're running spark on kubernetes, we can next team enable Chrome workflows. So if you hop into the GitHub repo that we have online, we have an example of that that you can check out as well.

A

We can start to Define different node pools, so if you want to scale up gpus for certain jobs or not with a few lines of code, you define those nodes and select them and with Argo workflows, have the logic to then run certain spark jobs on on more compute or or less compute, depending on your needs start to use cluster Auto scaling and then extend the pipeline with more capabilities.

A

An example of that is this architecture, where we're deploying Argo workflows, but we're also then deploying a rest API that can feed events into through Kafka and into Spark's structured streaming app. We can ingest that alongside running some batch jobs that we're pulling from a database and we can use Argo and Argo events to trigger these spark jobs simultaneously, and then we can now that we know the state of these jobs. We can trigger ml model training, hyper hyper parameter tuning and even model deployment as the pipeline progresses.

A

So you get an all-encompassing way to automate your ml Pipeline on kubernetes, with Argo workflows as far as resources and next steps feel free to check out our repo, where we have all the code in the demo hit us up on Twitter if you're curious about chatting about this any further and make sure to check out a couple extra resources. If you end up diving in thank you, foreign.