Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Scalable Monitoring Using Prometheus with Apache Spark - Diane Feddema & Zak Hassan, Red Hat Inc

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Scalable Monitoring Using Prometheus with Apache Spark - Diane Feddema & Zak Hassan, Red Hat Inc (Intermediate Skill Level)

As spark applications move to a containerized environment, there are many questions about how to best configure server systems in the container world. In this talk we will demonstrate a set of tools to better monitor performance and identify optimal configuration settings. We will demonstrate how Prometheus, a project that is now part of the Cloud Native Computing Foundation (CNCF), can be applied to monitor and archive system performance data in a containerized spark environment. In our examples, we will gather spark metric output through Prometheus and present the data with Grafana dashboards. We will use our examples to demonstrate how performance can be enhanced through different tuned configuration settings. Our demo will show how to configure settings across the cluster as well as within each node.

About Diane
Diane Feddema is a principal software engineer at Red Hat Inc Canada, Emerging Technologies Group. Diane is currently focused on developing and applying Data Science and Machine Learning techniques for performance analysis, automating these analyses and displaying data in novel ways. Previously Diane was a performance engineer at the National Center for Atmospheric Research, NCAR, working on optimizations and tuning in parallel global climate models. She has a MS and BS in Computer Science and has presented at Apache Big Data, Red Hat Summit and Openshift Commons Conferences.

About Zak
Zak Hassan is a Software Engineer on the Emerging Technology Group working on Data Science and Machine Learning. Zak previously worked as a Software Consultant in the financial services and insurance industry, building end-to-end software solutions for clients.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Thank you for joining us, so my name is Zack Hassan and I'm. Here with Diane Fatima, we work at Red Hat in the Emergency Technology Group on data analytics I've been at Red Hat for the past four years. Our project is rad analytics, which is a platform where developers build data science applications on our kubernetes, offering which is OpenShift along with building the platform, I also develop. Intelligent applications do things like image, recognition and other machine learning, AI type of workloads and.

B

I'm Diane Fatima I've been doing big data work for about the last five years. At Red, Hat I started out working with Hadoop, initially for a couple years and I've moved into using spark and I primarily analyze performance data, I, run experiments, collect that data and then use the machine learning algorithms in spark to analyze them.

A

So today, we're gonna spend the next half an hour to talk about the story of how we set up Prometheus the monitors, our spark cluster and run and our running spark jobs live in kubernetes pods will dive into high-level concepts. We'll look at code will actually show you a live demo with Griffin prometheus. All running in kubernetes are open shipped offering so we're also gonna. Do some slight code changes and we'll show you performance improvements of code and then we'll run a job with the the not being tuned, and then we'll run a job.

A

That's tuned and we'll show you some performance improvements and quicker runtime.

A

So our goal for building the the platform was actually when we built the platform, our goal was to try to improve the runtime of applications to allow data scientists to do more rapid experimentation to provide. You know if you prove, if you write a program and you get quick feedback rapid feedback, then you do more experiments and also when you're doing integration testing.

A

You can do more ambitious testing when you know exactly how much memory is CPU and whatever resources that are being consumed in kubernetes, and one of the compelling reasons for even using spark for developers is actually because of its parallel processing. It's in memory pipelining and data intent, optimization.

A

So our story. Basically, this whole experiment, I started looking at this experiment. Last year, like around mid June I looked at since we were running spark in containers, I looked at what, if we put an agent in there and then expose the metrics and then at first we were using Jolokia and then later we decided to use Prometheus endpoints later, as the experiment continued, we we created a CRD custom resource definition with a spark operator.

A

That's able to deploy spark cluster, discover the the the workers in the panda master and then dynamically configure the targets to be consumed by Prometheus.

A

We set up alert manager and you know, and and you know what that wasn't enough- actually, we we try to even do further optimization where I got in contact with Diane and we set up Diane help actually set up the CRO final integration with Prometheus and we're off to experiment on doing actual spark jobs. So this is not going to be an introduction to Prometheus.

A

This is going to be more of actually using Prometheus or if this is the, if that's a talk, this is the talk for you, so I'm gonna briefly go over what prometheus is?

A

Who does not know? Prometheus, okay, so Prometheus is a modern solution and it is a times. Is it's a time series database, so it collects the metrics running in your application and you could either instrument your application or you can use an exporter that would export out like if you have JMX metrics being exposed, then you can use JMX exporter or there's different exporters for different tools in your infrastructure. Luckily, if you're using kubernetes that comes out of the box, already pre instrumented.

A

So, what's SPARC anyway, so SPARC is actually a data processing engine. It's parallel processing. It lets. You spread your data across many machines. You can do in memory processing and it has optimizations to form, queries, machine learning, stream, processing, workloads and more. You can choose to run on yarn, Mei, sauce or standalone. So for our setup, we're using stand-alone there's. Actually some interesting work, that's happening at Apache foundation. Our group and other companies actually are working together on using kubernetes as the cluster manager talk to me offline.

A

If you want to know a little bit more about that, so this is actually the building blocks of SPARC itself. So you have the core spark core. Then you have different api's and libraries that you would use for whatever particular problem. You were trying to solve right, so you can use sequel. If you want to query the data spread across your cluster, you can use a machine learning. You could use graph, you can use streaming.

A

If you have data that's being streamed into a Kafka topic, then that's that's one option as well and you can actually access your data from different data sources, so you can access from HDFS Amazon s3 SEF s3. If you want and you can write jobs in Java, Python and Scala I- think some people also use our. How many people here use our okay.

A

So rdd's is one concept to understand, because that's the foundational thing with it with inspark, so rdd's are resilient, distributed datasets and basically your data is split into partitions and it's spread across a cluster, and then you it's it's like. Basically, you have a collection of rows that sit on a physical, physical machines, and you have this this concept of executors that execute your work and these executors are actually in the spark worker.

A

So you might think okay, why do I even need partitions?

A

Interesting thing?

A

Is you need partitions, because when you execute work, if you have only one partition and you have a thousand executors, your parallelism is only one if you have a thousand partitions, but only one executors same thing: parallelism is only one, so you want to have more partitions and more executors and we're gonna go into a high level of what a spark cluster is made of so spark a plication, as I mentioned before, a spark application is basically you get data from a source right, let's say HDFS or let's say s3, and then you have a spark job.

A

You write a job, so you have to decide if your job is gonna be streaming or if it's gonna be a job. That's that's! Just a batch job either option you could you could use the libraries that that that come with spark and then you want to store your data somewhere. If you want to train a model, you might want to store that model somewhere and one thing to that I mentioned earlier. Remember that point when I said Y spark is popular. It's because of its parallelism right.

A

It's lazy by default. Its core data structure is immutable. Let me actually explain that. So what it does is. It performs a transfer, but only until the very end when an action is actually called on it. So internally spark builds up a dag and it makes a plan for the transformation and then in a polite, applies that transformation to the source data and generates a result. So under the hood it actually uses rdd's.

A

Remember that term that I mentioned resilient distributed data set, but what you can do this library on top of that called data frames that makes it super like a little bit more simple to use we'll talk about that more.

A

So this is our setup within kubernetes. We have positive familiar with. We have containers that are running inside those pots, and these containers have a spark master, a worker and you can have more than one worker and then the application. That's running it's running as a driver application. So I want you to remember that term. The driver, the driver, is basically running actually doing the job right, so we use source to image to when you have good source code. You point to the gate source code.

A

We we pull down the image, it builds an image and then it deploys a job. You're gonna see all that stuff in the live demo. So some interesting metrics that we got back from the driver is things like block manager.

A

Things like the tag, scheduler things like JVM and some of the internals of what's happening within the JVM inside the container that's running and we found some interesting details.

A

So this is just a high-level view of the the cluster. So Prometheus is gonna, go and scrape these endpoints, so we have Java agents running in the spark master and worker and also on the actual driver application. That's running it's all running in kubernetes, so we get information about the executor. How much memory is being used?

A

We can get more information about. You know how much memory is actually being used by the whole kubernetes cluster and other different details as well and then but the thing is, you said how often you want it to scrape so I'll, explain that part in a later slide.

A

So this is basically just the when we're building docker image. This is the Java agent who is familiar with Java agents who has used Java agent before okay, so 10% of the room. Okay, so Java agent, you pass a Java agent at runtime when you're running the the worker or the master, and you can pass it a jar and that jar is gonna. Put for this particular example that jar is gonna, go and look for JMX metrics and expose it over rest in a format that permittees understands.

A

So some prerequisites you have to have JMX enabled.

A

So this is a view of we can see the network CPU memory of kubernetes. We can see application metrics and we actually interesting thing is we actually ran OpenShift ansible to set this up, which is on github, and then we have to do a little bit more experimentation for the spark side and we'll talk about that more.

A

This is the configurations for an example of configuring, something to scrape an endpoint in prometheus. This is an example of a simple, very, very simple alert rule. You can set different rules and then you would use prompt QL. So if you see the up equal equals zero. That means, if any of these nodes that are being monitored if they go down, I want to get notified. An email, slack notification or whatnot from qo prompt Q is what we use when we want to do.

A

Visualizations and Dianne has some visualizations that she's going to cover so just remember, gauges are like when you're driving a car gauges. Tell you how fast you're going right. So that's a gauge Prometheus. You have gauges and then there's something called counters. Think of that as an odometer, how many miles you've traveled so far, so without further ado, I'm going to pass it on to my teammate Diane. Furthermore,.

B

Thanks, Zack, okay, so for the second part of the presentation, we're going to talk about how Prometheus and Ravana can be used when you're doing performance analysis, which is what I do in my job. So there are a number of things that you do get with spark out of the box that tell you what's happening with your spark job as it's running.

B

There are many important things that you want to know that the spark web UI does not tell you and that's what we're gonna get from Prometheus and Griffon a-- really important things that help you not get out of memory errors when you run your job, it help you tune your job, it just sort of turns the lights on, and you suddenly know all the their resources that are being used for your job. So we want to know things like how much memory is.

B

Are my workers and my master using how much memory is available on the entire system if someone else might be running at the same time as you is, there is the network being saturated by my job and what might I do to eliminate that bottleneck? What's the CPU usage on the container level in the pod level and on the node level you can with Prometheus, you can look at a Pernod level of four all these metrics.

B

You can also look at a per application level for all of these metrics, so we as Zach said added these endpoints in spark to scrape and we're also scraping for career Nettie's, to get all this information to help us do our job and there are just hundreds of metrics that are being saved in this time series manner and we can't really go into all of them in this format in this presentation, so we are going to concentrate on memory usage and the duration of the job, so we're going to just show some example code.

B

We're going do some tuning and we're going to improve the memory usage and the timing for the job.

B

So efficient memory use is important for SPARC, but it's also important for any data intensive system so impossible. We want to keep us met as much information as we can near the processors, and we don't want to spill to disk. We want to avoid spilling to disk with SPARC jobs whenever possible, and we want to figure out what we want to cache. These are Dedes that Zak talked about earlier, they're, often costly to compute, and so we want to cache them instead of having to recompute them.

B

So if you have multiple stages in our job, you want to cache the rdd's between the stages, so we're going to create Prometheus and graph on the dashboards. What we already did and we're going to demonstrate them to you and show you how that helps us to know our spark job and it's actually really fun to create these dashboards. I mean sometimes I. Forget that I'm actually working when I'm doing this. It's I mean it's just a lot of fun to create these graph on a dashboards with an input from Prometheus.

B

So SPARC is an in-memory compute engine. It very aggressively uses memory, and even though these nodes that we're gonna do our demo on have 256 gigabytes of memory each so a cluster can easily have a terabyte of memory, there's still contention for memory, and it's still worth tuning your memory, because these large analytics jobs very aggressively use memory and in a spark job, you can easily have a hundred gigabyte heap spaces. So it's worth it to tune that and we'll talk about how to optimize that for overall performance improvement.

B

So now I'm going to talk a bit about Sparx memory management model. Just so you'll understand why our tuning helps once we employ the tuning we're going to go with here. There are two memory usage types and SPARC execution and, and they share the heap space- the execution memory on the left here oops here we go, no animation didn't start yet execution memory on the left is used for short live results. It buffers intermediate results that are your computation is performing so, for instance, in our example code.

B

We're going to do a group by in order to do a group by you have to create a hash map, and you have to build that hash map somewhere, it's going to get built here and the execution side of the heap and after we have performed this group by it's good. The the result is going to be an RDD. This resilient resilient distributed data set as I, was talking about and it is going to be stored over in the member. The storage side of the heap.

B

So here's a little example of your code running and the sharing of the open space here in the middle is a space on the heap that's available, so the storage area say your caching, you put dot cache on a bunch at the end of a bunch of your your a Scala or Python lines of spark code, and you want to you want to cache a little bit more of your data. So do you suppose this? What do you think? Do you think that you're gonna be able to put this blog here?

B

Yes, you know: there's open space, we're gonna get to do that. Once we have contention, we have to decide which block is going to be evicted and what happens is the execution side always takes precedence over the storage side, because the processes that are relying on this intermediate results that are in the execution side would fail if you were to evict their blocks so that just forced some storage memory to spill to disk, which is ok, sometimes people over cache anyway, and it's no harm you can just reload what was in the cache earlier again.

B

The execution side is going to take precedence, it's going to cause this to spill to disk, but there is a limit that you can set and we can tune it, and this is something else we can view with Prometheus and and play with this value you user can set this spark memory storage fraction, and that is the unev ICTA below part of the storage memory.

B

So again, execution takes precedence over storage up to a point where you have any victim blocks, so one more thing I want you to understand: is some api's we're going to use. So when you look at this example, code it'll make more sense.

B

Spark sequel is essentially a library that sits on top of spark core and it gives us three additional api's. It gives us spark sequel syntax. So if you're accustomed to using relational databases, you can make sequel calls to and interact with spark that way. It also gives you data frames, which we're going to use in our example, and data frames are similar to a relational database table with named columns.

B

It also gives you a data sets which are similar to data frames, but the columns are not named.

B

Additionally, so we're going to use the data frames. Additionally, you get okay, you get two back-end optimizations as well, and this is where spark sequel sits in the stack and in your user code you can intermix the use of spark, API and spark sequel API.

B

So we are going to do a performance, optimization, we're going to show a non-cash version and a cache version in our live demo. You need to figure out how much space your RTD takes. So this is something they spark. You I will tell you you put dot cash in the end of the line that creates your data frame. Then you run your job. You bring up the spark web UI, you go to the storage tab and you see the size in memory that your RDD is going to take.

B

So then you're going to set that that storage on a victim storage region to be at least as big as all the rdd's that you want to cache. So this is our code. This is the before we create these data frames. We do a cross join, which is a Cartesian product, and then we do a group by operation and because we reuse an RDD and that group by we want to cache it. So in our improved version you just put dot cache at the end of the line. That's going to cache.

B

Those are Dedes and we're gonna see what that looks like if we ran after and our timings and in our memory usage. So this is the query plan on the left for the non-cash version. You see that that's creating those rdd's but they're, not in memory on the right. It shows that those are in memory tables because they've been cached, so this results you're gonna, see in the demo as well, but just as a preview, we looked at all this running and spark.

B

We look at how much memory is being used with viewing it through Prometheus and Griffin ax, and the result is that, after our optimization, we went from about 5.3 gigabytes per worker to about around 2 gigabytes per worker after we cashed that RDD.

B

So this is my relative index of performance that I used in my performance. Metrics, but here it's the same value as a percentage change, so I'll just explain it as a percentage change.

B

We got a 76 percent improvement in memory usage, so that's memory reduction and we had a 10% decrease in runtime by adding that optimization. Ok, so demo time.

A

So this is Griffin. Oh and Diane's. Gonna show us running a spark job and okay.

B

See quite you'll.

A

See you'll see how performant it is so.

B

So first we see here that there is no spark job running and I'm gonna go to my B deployment. Configs am I gonna go to my nan cached version.

B

I'm gonna deploy this non cash job.

B

Then I'm going to look at the spark master. We should see this job showing up here.

B

Take a second there. It is there's our non cash spark job.

B

We can see that it's running. We can see her under the storage tab that, in fact there is nothing because we haven't cashed anything now we can go over to Griffin ax and we see that it's going to start. The job is going to start showing up here soon. Here it is so we see our CPU usage going up to start with. We just have like the see that the met the network is quiet. The whole system is pretty quiet, but as this job most in here we're gonna see our memory usage going up.

B

Let's just go and make sure that that job was still running.

B

Yeah, it's running. Okay, that's good! It's all good, all right! So.

A

Yes, so it's not looks like it's starting so.

B

You see that the cluster CPU usage is going up. You see the memory usage is going up. Pod, CP usage, okay. So now what we're interested in here? Is you see that, let's see.

A

B

They read the numbers, you think, okay, so um so you can see that we've already we've got five point, two gigabytes for these four top entries here. Those are our spark workers. This is the non-cash version. Okay, and there is only one pod per container, so the the pod metrics are the same and down here at the bottom. We have our JVM pools.

B

We have the Eden space, the survivor space and the old gen space, which is the Eden spaces where our new Java objects are being created after the first garbage collection, things get moved over to the survivor space and then our cache items and things that make it through another garbage collection are going to end up in the old gen space. So we can use these Griffin Oh dashboards at the bottom here to optimize how we're going to use the JVM for this part for these SPARC workers in the spark master.

B

Now we see we're up at 40% of CPU usage for the cluster and that's because that Cartesian product is happening right now. It's very CPU intensive operation and you can see our two nodes here that are sitting in Boston doing this work as we watch it, and so here we have the memory high-water mark for the workers at five gigabytes, five and a half gigabytes each. So now we'll go back and see that spark job is finished. I happen to know that it's finishes in about three minutes. There are two point: nine minutes: it finished.

B

Okay, so now we're gonna go back and deploy the cached version.

A

So if you notice there Dyan pressed deploy, so this is a deployment config and when Diane was was building her application. She created a she's sourced to image to build an image and then once Diane had her image, we used the deployment effect to roll out the the spark job and then, when she went iron, wanted to rerun that same spark job. She just had to just press deploy and I really ran the same job yeah.

B

One of the challenges and making something that you can do in a five minute demo is: you have to cut an actual spark job into something that will finish in the amount of time that we have up here. So I've just submitted the cache version.

B

Now we're going to look at storage, see we have the the two rdd's there that we've cached that's running live right now and now we can look over at go fauna, and you can see that that last job that drop-off right there is the last job that finished we're gonna see the next job coming through. So.

A

There's a little bit of a lag because we set the intervals a little bit with a little bit more time in between.

A

So if you wanted it to see, you know more more, you know metrics a little bit quicker than you would configure those things.

B

Instead of scraping every 10 seconds like we, you know so.

A

You could set to scrape every 10 seconds or you can set lower, but so.

B

Now the second job is running: we've already hit the high-water mark for the containers that are holding the workers. It's showing right here saying for the pods about two gigabytes. Each again we can go down and look at our JVM pools. We also I noticed that the network was very quiet. Network is not an issue with this example at all, and it's there's no bottleneck there. But the important thing here is that these values were not known to us previously and it's a guessing game.

B

If you don't have these metrics to look at so this is incredibly helpful when you're tuning a spark, job and I'm sure applications in general. Just the fact that you can look at how its load balanced across the nodes, then you can dive down deeper and see at the pod level in container level exactly what's happening. So that's it.

A

So we'll take any questions if anybody has any, we also have office hours for about an hour. We're gonna be at the Red Hat booth. So if you want to stop by ask any questions of anything that we showed up here, because everything that we did was all open source and.

B

Reproducible and reproduce.

A

Do it yourself, thank you.