National Energy Research Scientific Computing Center (NERSC) Data Seminars Series, 31 Jan 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020-01-31 - Avani Wildani - Data skeletons: IO workload characterization for the modern age

Description

NERSC Data Seminars: https://github.com/NERSC/data-seminars

Abstract: In a not-too-distant future, our devices will not just be networked, but share huge volumes of data required for instantaneous and vital decisions. Technologies such as self-driving cars, smart cities and homes, and augmented reality all depend on handling massive quantities of data quickly and reliably. Understanding the characteristics of workloads such as these will be critical to reducing storage costs and making the cloud of tomorrow accessible and equitable for all. All storage systems exist to serve some group of users and applications. Tuning a storage system is a delicate balance between reliability, availability, security, performance spread over the users, or workloads, that the system serves. For cloud storage, elasticity is a crucial factor, but misconfiguration in reactive storage tuning has been cited to be a leading cause of production failures. Transferring and transforming provisioning insights to match dynamic workloads will break ground for system improvements spanning from the power footprint to cache management to selecting an appropriate reliability configuration. This talk covers the current state of workload-aware design along with our current work to improve storage provisioning and trace characterization.

A

I'll sort of hang out around here.

B

She's an assistant professor in computer science and neuroscience at Emory University. She got her PhD from UC Santa Cruz.

B

She does research in storage and retrieval access across a wide range of models and applications. It means and she's going to talk to us.

A

Today about data skeleton thanks Taylor, so hi I'm, a Moneyball Donny, and thank you and I'm here to tell you a little bit about figuring out the core of what's going on in your IR workloads and so I'll explain the data skeletons name in a little bit here, but I'm sure. All of you kind of run into this problem of we're, designing storage and at least I know from the stories of science perspective we're focusing on what can we do with this reading class point and sometimes I.

B

A

Michael bit more.

B

A

Okay, awesome, so it's often easy to lose track of what we're designing, storage for and that we're designing, for is the end user and their goals are changing they're changing rapidly right so, whereas before we had kind of you, traditional archives or a scientific were closed. Now we're designing for areas with very, very high ingress, like you know, mix or areas where you need to process data really quickly like self-driving cars, where they're generating on the order.

A

One point five terabytes an hour and if you do not very quickly process that, like we're talking within about ten seconds, to do a CNN on that data, you crash literally, and so that's my situation where you know, storage providers are now kind of on the hook to make sure that we're designing systems that keep modern applications in mind and, unfortunately, right like while we understand that there's a lot of data out there, we haven't really figured out what the new problems are, and so this is just the you know.

A

Everything has to slide in these talks about how we're making lots and lots of data. So this is my slide about how we're making lots and lots of data, but this is stolen from and the key thing in the corner. There is by 2025 they're projecting with an IDC projection for 63 exabytes of generated data a day- and this is data- that's actually referred to- and this doesn't count things like duplicated emails.

C

A

Like checkpoints or other data, that's designed to be thrown away, this is actual active data, so clearly right we're making a lot of data and used to do that. Data is changing by this informal talk. So if any point you feel like stopping chatting, you know snarking, please feel free to stop in ask, but all that data that we're generating more and more of it isn't well structured right.

A

So, as storage people we're pretty good at handling structured data, so if you give me a database, I know exactly how I want to stir that database right. I know what it's access pattern is gonna, be I know how to design a cache for it. I know what kind of storage to give it and I can give you a reasonably good idea of how quickly things are gonna run right. But if you notice here that little blue in the bottom is your structured data and the rest of it is all unstructured right.

A

So, most of the stay that we don't have these nice parameters for for how to deal with it and then, of course, the typical labels that we have for data like archival means very very little. So this is your traditional archive. This is Iron Mountain, which is full of tapes.

A

There write once read: reburying, maybe yeah I think there was a recent a where the 50% of these weren't actually readable. But this is your typical archival. Look right, like you write data, and maybe it's for compliance reasons. Maybe it's for some other. You know reason giving you come on high and you don't expect to ever see it again. Modern archival data is not like this right, so this is snap in my email.

C

A

Most of the data I will never look at again. There is lots of it, but I have not intentionally archived it right. It's kind of archived by accident and when I want a piece of that data, I want a piece of that data right now and it had better be there, and this is true of photo albums right, and so that is the blu-ray storage that Facebook has to store their photos. So again, most of the Facebook photo storage is archival, but it's archival by accident right.

A

So these are archives where if, for example, a celebrity dies, people are gonna want to see every photo of them ever created, razza celebrity lives to a ripe old age, nobody's ever gonna care, and so it's really difficult to predict what parts of this data are going to be non archival ahead of time.

A

And you know what people call our ival again Ashley's vastly changes. So this is, we did a meta-analysis to see. You know people label data archival what properties that these different data sets had in common, and so in the same year there were several traces and.

A

C

Very different file.

A

Size and then across here is right. You had very different file sizes, so there were no traits in that paper that they found that were consistent across types for different traits.

A

All right and then, of course, anytime, you design, storage for workload, they're gonna change workloads are highly dynamic. Again. This is old news to probably everybody in this room, and so.

D

C

Couple slides may get in trouble for.

A

So HPC data is also changing and your typical scientific workloads right of doing a lot of data, ingress, computation, checkpointing story, data and then kind of loop, its occasionally changing, although I have learned today that MPI will never die and, of course that is, that is true. There's this catch is still there, but we also have other more modern, HPC workloads, right, which are say.

A

Maybe you hook, Bitcoin mining situation going or you are have a cluster OTP use and you're trying to do a large, deep learning workload, and so now, all sudden you don't have this assumption is sequential and periodic I.

C

A

See questioning eyebrows? Are you good, yeah, yeah? Yes, okay, again, stop me. Stop me if you would like to heckle this is about where I expect the heckling. This is actually one thing I really want to hear from you is you know what do modern HPC workloads look like and what assumptions can we keep right? So in the research community? We have this idea that sequential and periodic I/o is on its way out, and so we need to kind of redesign how we think about HPC workloads, provisioning.

A

And then, of course, when you think about where to put it right, so the underlying storage media is also changing. People been saying disk is dead for years. Listening tape is dead for years and, of course, these things are still around, but they are on their way out. So now, you're seeing these like a Templar and their prize scale, libraries and then, of course, now we have things like 3d crosspoint, and then you know kind of more exotic lis, so the corner there we have, you know DNA, based storage, for your long-term ur cuz.

A

We have shingled disks which give you density, but you trade-off the ability to easily do rites, and so again this is depends very much using these new storage media and very much an understanding. Your workload very well, because now you have many many more knobs to turn in how you can place things to maximize density, minimize costs, minimize power, maximize performance, whatever you might care about.

C

A

Yeah, so if, for example, you are a Bitcoin miner right, you're not going to get a lot of sequential.

C

A

I have a nice listing the data, but my intuition is that a lot of deep learning workloads are not going to be very sequential and that's something where I'd love to get a better understanding from people in this room about you know, is that intuition actually true, but yeah I'm expecting that if you can't actually reasonably batch your work, if you're trying to wait for a lot of these intermediate results, then it's gonna be hard to get that kind of sequential.

A

D

Definitely cook them.

A

This is where I was I was saying that the hardware is changing out from under us, and so, even if her work was we're saying the same which they aren't, we need to kind of revisit how we consider the interface between storage, workloads and storage media all right. So we want to know what actually still matters and what we do with it, and we know very little so we're trying to change this. But if we did know things right, we could do great things. So this is just a small example.

A

What we could do if we understood our work blows bell enough to do caching. So this was a project where we looked at a workload and we found that it had working sets that were somewhere along the sides or like between four and a thousand blocks right and just by prefetching. These working sets into a fingerprint database. We were able to drastically improve the deduplication performance on this computer, and this is something that we've actually implemented now in enterprise.

A

D

A

D

A

Was a bunch of students doing experiments.

D

A

No, it was actually so. There was a university and we took all of the Student Activity on a server, and that was the workbook. It.

C

Was it wasn't it.

A

Wasn't anywhere close that was design for this project. It was a storage workload that was around.

D

A

Playing with email playing videos, you know playing quake on the side. It was all in there.

A

And then again, so all we can do here because we calculated working sets. We can index I'm.

C

Not gonna go into this.

A

One too much because this is mostly here as an illustrative example that if you can understand your workloads, great things will happen and we actually have some current work now in this doing a multi-level cache, and so we found already that for a particular workload, um it's not necessarily gonna be hierarchical, so I, don't know we had a slide to show you. One of my students gave.

C

Me the wrong image, but anyways.

A

So we found that we could have situations where you actually want more DRAM, for example, and less SSD and more hard disk again, so you kind of have this pyramid, or this hourglass instead of a pyramid, and so there's some exciting things happening there, but just for this LRU cache we can get significantly better efficiency with me. Do this fingerprint grouping.

D

A

D

A

Entirely that level there's no real ratios can be done at higher levels. So we have another project that uses exact same technique at the file level, but.

A

All right, so our grand plan, our grand plan to quantify this, is something that we're calling Skeletor and so we're looking again for kind of bones of your data right we'd like to understand you know, apart from these old labels that we have of like archival or enterprise, you know what's really going on between data sets, and maybe things are more similar across these different areas and they are within these areas. So.

C

I'm going to briefly.

A

Talk about some, what we're doing in education and then focus more on what we're doing with the actual extraction.

A

So we have one project where we're actually trying to understand what features are happening without understanding anything at all about the data. So this is also at the block.

A

Where we're trying to just understand what higher dimensional feature is exist between different data sets,.

D

A

These are just um they're, actually the same as here right, there's space and time. So you have some sort of spatial dimension in this case, I'll be a.

E

A

Then tiny mention and then similarly here right so in this case we're taking the entire. In this case we had teachers that were extracted so you're, looking here at the hidden layers of siena res here, you're looking at the entire trace.

D

A

This is this: these are entire trees. That so I mean each image is about a week. These.

B

A

And so just even looking at these right.

C

A

Can tell that there's some correlations.

C

A

Things that have acted similarly on our systems and so we're clustering. These and doing some other work to try to figure out what again, what visual features are important, and so there is a privacy dimension to this. That is future work, Chris right now, where.

C

We can actually look at.

A

These applications running in some cases, so a database.

C

For example, like a nice diagonal.

A

Line right up- and you know you can actually find Journal pretty easily.

A

And so there are couple patterns that you could just pull out like you know, February and.

A

Yes, yeah: this is all done on the surface.

E

People doing mixture of all these things at the same time,.

D

A

So that's actually something I'll be talking about in a few slides, that's something that we ran into as a problem also. So this was a single client and then we actually had multiple data sets from the same sort of place, and so we combined them together and then resep rating them out. The cigarette was going on at a you know, plant level. It was very hard so again we're trying to find good metrics right. So we took a look at what metrics were standard, and this is again a meta-analysis where we pulled just okay.

A

What do people look at and they did not at all match the things that we found. So we hit that problem figuring out tenancy and we ended up first, you need to figure out how many tenants were, even in our system right, so the technique that we need and we are like. Okay, we have to where we have 300, we don't know so. Let's say we can figure that out automatically and again people. Please stop me if there's anything where I'm blowing by something anything very quiet.

B

A

This was literally, we looked at every single paper that did a work live analysis that we could get our hands on, and we just you know, made a histogram of the times that these features were used.

A

They were used within the classification that the paper was focused on.

C

A

Is the suspend a stick, so this was across several different I/o traces from different file systems.

B

A

D

Year, the trace was collected.

A

E

A

Know there were I, think 30-something so.

D

It wasn't, everybody was.

A

B

The next line is showing so what.

B

A

Right so for the result that we wanted, we were curious to see again how many cannons burn our system, and so we did a time series analysis without any prior bias about what features might be important, and this is what we ended up coming out with. So the address complexity mattered quite a bit more than anything else that we looked at, and that is very strongly tied to the spatial locality of the data set.

B

A

So this is our time series analysis features, and so it is a function of the spatial locality and I can get.

B

Into the whether.

B

A

The proximity: it's it's: a ratio of proximity between locations.

A

B

What are you so.

A

We're trying to and I'll go into that and a couple slides here, but the list feature analysis was to predict the number of tenants that were within our rig low trace.

A

A

Of course, this ended up being very reliant on which trace we were using. So this has many of the same features and as we change the trace that we were looking at the feature importance very loudly, so even for a very simple problem like this. Finding a core set of metrics is a complex, open problem right and so yeah.

E

A

The bottom yeah yeah, so this.

A

On that rainbow pot, its relevance per trace.

E

A

And the light is good.

A

My apologies for the lack of axes, but it got really really messy very quickly, so just wanted to quickly show that they were different. You.

C

B

A

You need and there's no consistent set of features that we found so far. So this doesn't, you know often right especially early on in this project. We got a lot of feedback. Oh, we know what features to use right.

A

The same features we've always been using, but we're trying to show that this is actually an interesting open problem, because the features are highly dependent on the workload and I should point out right, like workloads, don't exist, independent of their infrastructure, so two people can be running the exact same thing and if one person has twice a cache or the other they're gonna look very different in this post, cached I/o workload, and so, if you don't.

C

Have that sort of metadata.

A

Right, you can't necessarily say that these situations are gonna, be the same metrics the same layout, the same handling in any way, and so you know one thing I'm really interested in is: how do you tease out the functions from the infrastructure right? How do you figure out what somebody is doing as an outside observer? If you don't necessarily have that CPU information, if you don't necessarily know how often they block on other tenants, if you don't know how much cash they were going through, yeah.

A

C

A

Yeah, so that's coming together: no! No! No! It's fine! I'm I jumped a little because I wanted to show that the metric extraction problem was but I. Maybe should've started telling you more about this project before and other other questions.

A

Alright, so yeah, so it's problematic, you don't have any ground truth. I completely agree right, and so, as a result, we want to use a few metrics as possible and of course we, if you want to actually separate workloads, we can't define our classes combination of metrics. So our goal is to minimize metric redundancy and maximize metric coverage and so that that's gonna write Beckenham I jump ahead. A little bit.

D

A

Minimum block IO all.

D

Right, that's like just numbers written all the time.

D

A

I mean at minimum rate some sort of locality and some sort of time parameter, and then you get any questions with sampling and that's actually very hard right, because if you want to do sampling for a lot of these sort of like burstiness or cyclic questions or figuring out what the phase shifts are in your workload right like that, there's no good standard, I know of on how to do that. So in an ideal world you collect everything. But of course that gets to be its own storage problem very quickly.

A

And so then there's you know some sort of compromise and how much she would collect. But yeah I mean.

B

More is better right.

A

Like so, if you could tell us, you know UID whatever or any sort of, maybe.

E

E

A

And I mean it's not an easy problem right like knowing what you want, knowing you know how to line things so I didn't talk about it, but one thing that we tried to do at point with collect data at different levels of the stat and try to align it, and that was really hard. We were surprised at how many times you know we what we thought had been collected.

A

It actually not actually been collected, so there are dropped taxes or dropped logs, but we can see at the block level things happen right and then we see the application level and then back down and things are offset, and so it's also not an easy problem. Data is dirty yeah.

C

Ground ground treats for I/o is like an order of magnitude, ordered them for them. I believe.

A

That I believe the right answer.

C

A

Yeah right, like it's all about where you're collecting it and what frame I think you're looking at it.

D

A

D

About this, because.

A

I mean I have I, have I can propagate to you, but honestly, because there is so little I/o data out there I think most of the community have kind of learned to live with IO data, just like lock, IO data, there's even a group working with like single snapshots. So right you get like one snapshot with lots of metadata, but it's like one time point per week and you're surprising amount with even that so I think it depends of a question you're trying to answer. We.

E

Can talk about.

A

E

A

A

A

All right, lemme, let jump ahead. Talking about these interleaved workloads, a small tendency situation right, so what we can do.

C

A

We have a ground truth is do boost deconvolution and say this is using ica.

A

And we do pretty well right, it's not fantastic. I could be better, but even with simple ICA on this mixed workload, we can get reasonably good, mean squared error and the problem with that. This is five workloads over ten minutes. The problem is that we don't always down many workloads. We have in fact we never know how many workloads we have.

A

So, even if you tell me that side people are using the system, you haven't tell me that five people are you doing system differently, and the goal here isn't to figure out how many people are there. The goal is to figure out how many things are going on how many different use cases are happening right. So if you have 100 different clients, but you only have two real workloads going on I, don't want to design a hundred different subsystems.

E

A

Absolutely right, and one thing that we're still trying to figure out is that, once we tease these people apart right, what is the right way to reallocate you know our run? Our resources should be, should be separate them, or should we allow them to be in the same space.

E

We have back-end looks the thing we've never done is figure out the correlation between them. The other thing this department does know everything. This tension, universes governmental people, editing's source code, around four thousand, whatever interactivity people moving to a river at the center.

D

So we have the dream to instrument.

E

Everything everywhere we're getting.

A

E

A

Like it would be fantastic to actually see this correlation, like you say, cuz, it's not an obvious question answer right like what do you do once you have these people separate it out like what is the.

E

E

Logical extreme is to give everybody their own file system right. Then they're all happy some version of happy, but follow so clearly. That's not the right answer either. But what's the right says one thing that we've been talking about a future generations of how do we go from and then we guarantee a quality of service to somebody who needs it maybe charge them a lot more than someone doesn't right and the first step of that is how we would even be able to communicate. To the first system.

E

There is person is more important, maybe we're just given their own. Whatever I mean.

A

You can imagine like if you have two real sets of clients: you'd make them give them to file systems if you can sign some sort of clustering or grouping across your tenant space. Yes,.

E

Sir, we could pretty easily you know to say.

E

Terabytes, we'll all over there. Actually, that doesn't work, because that's name right.

E

A

D

A

Tricky and they, of course, there were closed. You change very quickly right, like we've noticed that say, shows tend to happen between every 10 and 30 minutes. The faces don't ship that much so we have like a model workload space now. So, instead of having these labels, we have essentially different axes within our feature space, and so you take whatever basis you're closest to, and you say, okay, this workload is kind of solved under the spaces right.

A

Are you questioning here just focusing too much on.

B

Particular worries yeah your description to Jenny here, yeah.

A

So I mean that that's a way to think about it, um we're not strictly obviously creat. You know creating a matrix birth, each workload, but that would be ideal if we could. So you can imagine is a similar thing, like you take this n dimensional space and you have a feature array that represents each workload right and so the model of our clothes really are the basis of that space. Now, like they really are the you know your your directions,.

D

Really swapping to our aspect- and you know it kind of right now- there's no there's no systematic good way of doing it. All right you go ahead. Look at the traces, maybe make the plots notice that this one party person is doing so they stuff that we can get into and tooling and even approaches. Do that then will have an impact operationally for a lot of the pathos, and you know somebody opening a few million go to less than ten thousand files per second at one point on a very small job.

D

One of our saturated must have a lot of cases like that. Reading.

A

Backwards, 300 a lot of things.

D

A

Any way like even predict, based on their initial behavior and what they're gonna be doing.

D

A

Architecture of the experiment that created that rainbow graph earlier, we were trying to figure out again how many tenants that we have so we're again we're taking our air is we're splitting it up in the futures using this time series analysis and then.

A

We're gonna be looked at and anon.

C

A

The Emory IO dataset from earlier this was submitted to line change that around. We saw that, with the time series analysis, we actually do fairly well for detecting the number tenants we had and we have ground truth here. So we artificially designed different use cases for ours, and then we actually talked with people who made the data sets in the other cases. So we know exactly what was going on in each of these systems and how many different functional users we had, and so relatively high confidence in the detection. In this case,.

B

A

So solid at baseline and slashed us upper is better so the access percent accuracy, and here one approximate and to approximate it means that so say that you had five clients. When approximate gives you credit for for six, your Christ might give you credit for three or seven.

A

So now we can figure out how many clients we have now we actually have a shot of doing this in early workload, separation and we're looking for my data size now to try that out on we're doing Rawlins benefits we have, but obviously our data sets aren't representative.

B

A

You know I would love to see how the scales I think it should, but we do not have any data sets that don't have that many different functional clients in them, and even if we take every day, that's that we had and combining together, I, don't think we'd hit over ten.

B

A

Here, right we're classifying to find out how many actors functionally distinct tenants. There are.

C

A

No, that's exactly I'm talking about right.

A

So that's actually a worry of ours right because we don't want to Train on her testing set and we're doing a feature analysis before we do this.

C

C

A

What you're saying yeah that could work I think that that that would certainly work for some some of these types, but I think some of be very.

C

A

Yeah no I mean when I say, tenant, I'm talking about a functionally distinct usage of the system. Right so, like you know, classically just the person who comes in and like reads, data diagonally right like that would be a tenant, regardless. How many different people are doing that or how many that are you? And if you do that, sometimes right like so one user can be five different tenants in this case.

A

Well, you know it's exactly: it's defined by features.

A

All right and so again now we figure out how to measure it right. So we figured out how many a time, by the way.

A

All right, günter this a little bit quickly, because I would like to talk about well, actually, no, this again isn't it I took without already good job, so for figure out how to measure again right. We forgot where to instrument the questions that we have and then kind of what to do with the data at different levels. Everything I talked about has been either the block or the file level right.

A

Obviously, at the object level, there are going to be some things that we can keep and some things that we need to change and then aligning what we've learned in lower levels with these upper levels is an open question right, so we're using it until DSS to propagate information off the stack right now, and ideally, we can do better so kind of from here right we're trying to figure out like how our to work was meaningfully different. That's a core question.

A

Trying to answer is what does it mean for a workload to be similar in their workload or not, and then from there we'd like to separate a functional types and then finally be able to tell you what hardware to buy based on the workload you have, which of course is gonna, make lots and lots of good things happen. We're looking at improving power with a we have projects open in caching, right now in performance and and so with that questions, if you've saved any for the end.