National Energy Research Scientific Computing Center (NERSC) Data Seminars Series, 14 Dec 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2021-12-14 - Ahmad Maroof Karimi - Characterizing ML I/O Workloads on Leadership Scale HPC Systems

Description

NERSC Data Seminars Series: https://github.com/NERSC/data-seminars

Title:
Characterizing Machine Learning I/O Workloads on Leadership Scale HPC Systems

Speaker:
Ahmad Maroof Karimi
A.I. Methods at Scale Group
National Center for Computational Sciences (NCCS) Division
Oak Ridge National Laboratory

Abstract:
High performance computing (HPC) is no longer solely limited to traditional workloads such as simulation and modeling. With the increase in the popularity of machine learn- ing (ML) and deep learning (DL) technologies, we are observing that an increasing number of HPC users
are incorporating ML methods into their workflow and scientific discovery processes, across a wide spectrum of science domains such as biology, earth science, and physics. This gives rise to a diverse set of I/O patterns than the traditional checkpoint/restart-based HPC I/O behavior. The details of the I/O characteristics of such ML I/O workloads have not been studied extensively for large-scale leadership HPC systems. This paper aims to fill that gap by providing an in-depth analysis to gain an understanding of the I/O behavior of ML I/O workloads using darshan - an I/O characterization tool designed for lightweight tracing and profiling. We study the darshan logs of more than 23,000 HPC ML I/O jobs over a time period of one year running on Summit - the second-fastest supercomputer in the world. This paper provides a systematic I/O characterization of ML I/O jobs running on a leadership scale supercomputer to understand how the I/O behavior differs across science domains and the scale of workloads, and analyze the usage of parallel file system and burst buffer by ML I/O workloads.

Bio:
Ahmad Maroof Karimi works as an HPC Operational Data Scientist in Analytics and A.I. Methods at Scale (AAIMS) Group in National Center for Computational Sciences (NCCS) Division, Oak Ridge National Laboratory. His current research focuses on the characterization of HPC I/O patterns and finding evolving HPC workload trends. He is also working on analyzing HPC facility data to characterize the HPC power consumption and building machine learning based job-aware power prediction models. Before joining ORNL, Ahmad completed his Ph.D. in Computer Science at CWRU, Cleveland, Ohio, in October 2020. His Ph.D. dissertation titled “Data science and machine learning to predict degradation and power of photovoltaic systems: convolutional and spatiotemporal graph neural networks” focused on classifying degradation mechanism and performance prediction of a photovoltaic power plant. He received his M.S. degree from the University of Toledo, Ohio, and B.S. degree from Aligarh Muslim University, India. Ahmad has also worked in the I.T. industry as a software programmer and database designer.

Hosts of Seminar:
Hai Ah Nam, Advanced Technologies Group
Wahid Bhimji, Acting Group Leader, Data & Analytics Group
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory

A

But I welcome everybody to the nourish data seminar. We are really excited today to have our colleague from oak ridge here with us ahmad karimi, to give us a talk on uh characterizing machine learning, io workloads on leadership, scale, hpc systems. We um we saw uh the you know that this talk being presented, or you know that it was going to be presented at mascots 2021, and so we were like. Oh, we have to get this here at lbo because I think the work is really interesting.

A

So ahmad comes to us from the analytics and ai methods at scale group at nccs, at the olcf at oak ridge, and he is an hpc operational data scientist.

A

Specializing in that characterizing hpcio patterns, so with that I'm going gonna, let you go ahead and present your work thanks. So much.

B

Thank you. Thank you for the introduction and hello everyone. I will turn off my video for the talk so just to have enough bandwidth for our like slides.

B

So, as already mentioned today, the topic of my uh presentation is characterizing machine learning. I o workloads on leadership, scale, hpc system and arnab and fey were my like co-authors on this work and it was presented at mascot last month.

B

And in this talk I will start with first introduction and background and then a brief description about darshan, which is an hpc io characterization tool. I think many of us would know that, but it will be good to like uh describe it here briefly, so that everybody knows that, and then I will talk about a method that we used for extracting machine learning, workloads out of all the workloads that that darshan captured and then the analysis that we did on those extracted workload and then, finally, conclusion so a little bit of introduction.

B

So as we know that hpc workloads are always evolving, especially when we have more and more of ai and machine learning jobs which which has like a different io, which expect different io patterns than the traditional workloads like simulation and visualization, and maybe computation and those kind of workloads, because we know it has a distinct pattern like hpc. First, we load the data and then we process it and then train it.

B

So this has its own distinct pattern and then the work that we did was uh I mean the workload that we used were collected from summit, which is also called olcf4 machine, and it has two storage systems. One is parallel file system, which is ibm gps, ibm uh gps and it has a different name, is spectrum or something? And then the other is like the burst buffer, which is like node local uh storage.

B

So every node has its own burst buffer and then the analysis that we performed, where, like the data, that we had, we kind of classified based on science domain, from where the jobs were being executed or the size of the job. Based on the number of nodes and the temporal trends that we observed over a period of like uh for one year.

B

This in this image and like we can see how the hpc machines have evolved at oak ridge, national lab and over a decade and a half, and then we see that we have summit here and especially as these machines have evolved, so as the workloads on these machines right and especially like with the with more and more of machine learning workloads running they, they they have their own. I o pack, and so that was the motivation like we want to see how what what are the characteristics of these kinds of workloads.

B

So for this task we used uh darshan, which is an like well-known hpcio, characterization tool. I think developed that argon national lab for ibm, gene, ibm, blue gene series, computers and but but it's now very popular and is being used at different hpc machines across the nation and even beyond, so for it. The darshan has like different instrumentation modules. So for this work we have used uh three different modules like posix, mpi and standard io, so for darshan like how it does so for all the ranks I mean when a job is submitted.

B

It's it's running on multiple ranks right, so it will record the uh uh like io traces for all the ranks uh independently and when the job finishes, they are collectively written to one single log file. So that's how it does so. For example, mpi io would record the traces whenever mpi file read or mpi file light functions are called.

B

Similarly, posix would record for read and write function, calls and a standard io for for fprintf, fscanf and others and yeah, and now here I'm like showing a sample, io tray or a part of a sample address, because these trays can be very big right. So the the different features that we, our attributes that we see here, are module, which is like again the instrumentation that we use, for example, pos-x, mpi or standard io.

B

Then it has a rank on which processor it's running then record id, which is kind of the hash value of the file name, and then counters shows the different metrics which are like collected at darsh by darshan. Then the values of these metrics then filename.

B

So I have kind of um uh hidden the file name, full path of the file name, because just I thought it would be better and then so. This file name is like really important for our work, because this is what we use to find whether a job is a machine learning job or not a machine learning job, and then there are some other attributes as well.

B

So so we will come back to this again when we talk about how we, how we extract machine learning workloads so for this work we use the data set from uh uh I mean for workloads which ran on summit from january 2020 to december 2020, like one year of data and in total we recorded like three hundred thousand. Are we this darshan regarded three hundred thousand jobs, which is like not all the jobs because of the limitations of darshan?

B

It cannot record all the jobs, so this was roughly around 40 percent and out of these jobs, we found that around 25 000 of the jobs are machine, learning jobs there could be more, but from the method that we used, we could find out 25 000 jobs and the way we did it was that we used proof.

B

So primarily we we kind of uh we were looking for not we were looking for like we found a method by which we can at least extract those machine learning job which have used like python or our libraries, and then how do we do? That is first.

B

We uh like list down the list keywords and this these keywords are based on primarily based on the programming languages like if, if a machine learning job is being run by python on python, then what are the different libraries that it could use or if it's being run by art, then what are the different libraries that it could use? So so from that, we we kind of came up with these keywords and then also we kind of manually like also observed the data set for one month uh over one month for for like logs.

B

Obviously it's a huge log, so we cannot go all, but then we wanted to see. Are we covering uh like most of these keywords? So, yes, we we were covering, but, as I mentioned again right, there could be more keywords as well, but at least we we are starting with something where we can uh explicitly.

B

Okay know that these are machine learning jobs compared to like the other job, which could be uh traditional workloads yeah. So so these these keywords, the ones that are listed here. What we do we we then like, go back to this data and then map these uh with the file name. If for one job, we see any of these uh like file name, which includes the file path as well matching with these keywords here we would say that that workload is a machine learning workload and that's how so we would go with all.

B

We would scan all the 300 000 jobs, and, out of that, we found that 25 000 jobs are the ones which use uh which has files um matching with I any one of these patents, at least any one of these patents.

B

So that's that's like I, I would say the crux of the work that we did like kind of come up with a method to to uh to identify what are the jobs which are machine, learning job, because unless until we don't know that we will not be able to.

B

uh You know like like study about the uh uh the ml workloads uh or or the other other way could be to you know like run a job, and we know that this is a machine learning job so go and analyze that so that's one way to do, but we want to do. We wanted to do cross-sectional study like running for all the jobs that are there on uh that ran on summit.

B

So first, so then now our analysis begins now, from here onwards, we will have all of the analysis. Part and some of the results are are here. So, for example, we want the first thing that we did was we wanted to see the distribution of ml jobs, and we found that out of all the domain signs that are there and I think they're around 30 or something uh only nine of them.

B

We're like using uh machine learning, ml jobs from obviously from the method that we used and then how the unique users and unique applications uh distribute. So physics has, like physics, has like maximum number of users and application.

B

For example, computer science also has like maximum users and application, and then the other one is oh just this hide so yeah yeah and then the biology is the third one. So these three are the and especially computer science and biology, has a major contribution of the machine learning workloads and when we see, we also know that, like bioinformatics is one of the area and then computer science is the other area which would do that.

A

Hey I'm mod. Can I pause you for just a second to answer. A question um from cadilla? um Are the file names user defined so uh from from your previous slide.

B

Oh, these file names.

A

Right right, so those.

B

A

Defined files but you're also tracking our keywords.

B

Yeah, so if I, if I understood the uh like question correctly, it's like the question is like: how do we see these file names or who is defining these file names right yeah? So these file names are so I don't know like who is defining that it could be from the libraries itself. It could be from the user data or it could be from the source application as well. So there could be three different sources.

B

So, for example, I'm running a like a machine learning job right, so I would use let's say: train dot pi, so that would come here, but then train, train, dot, pi would import tensorflow right. So that would come here as well. So if you see here, these are the full file names. So we see repetitions here because for every count, because we have the counter values for our, for we have different counter values for one file. That's why we see reputations here, but we would see, train dot, pi and it's different counter values.

B

Then we would see tensorflow dot pi, let's say then their counter value. And let's say: if there is a data called data dot uh parkey, then we will see that as well. Here, that's my understanding, yeah.

A

Okay and and you have darshan running system wide right, it's all always running behind the scenes. Yeah.

B

It's always running and.

A

Can users disable it like they can disable like alt d, like a module, unload kind of thing, yeah.

B

Yeah yeah, it's it's a module so which is like loaded by default. If you don't want it, we can unload that thanks yeah and then there are different kinds of uh like darshan modules. I mean different with different functionality. For example, darshan dxt is the one which usu, which is usually not done loaded, but if we want like a real time or or temporal data information, then we can load that and it will have much more detailed data set, but usually that's not turned on.

B

Did that clear, the or answered the question.

A

Yes, thank you. Thank you.

B

Yeah so yeah coming back to our presentation, uh so yeah so three different, so these were the three different domains which had like maximum contribution to ml jobs and then so that now the other thing is we wanted to see what are the job size. I mean the number of nodes that these jobs are using. So for this we made the three different classification like flagship jobs, medium jobs and small jobs, and how do we classify them?

B

It's based on the uh the the scheduling policy of of uh of the summit, so some it has a scheduling policy for a different number of nodes. For example, if a job is running for like uh on nodes from 1 to 45, it will have wall time of maximum of two hours, so we classified those as ml, sorry as small jobs and then the other jobs that we classified as medium jobs are running on the nodes 46 to 921 and again there is like a there.

B

There are two categories in here for smaller for a smaller number of nodes. It will have a maximum wall time of six hour and for the larger category, a larger number of nodes, it will have 12 hours and then flagship jobs are like more than 922 nodes.

B

So these are the different different categories here and we want to see the distribution of each of these nine domain signs uh our contribution of ml jobs across these nine domains and nine science domain. And then we see that so, let's start with uh this small scale are: are the flagship jobs right. So we see that for flagship jobs, we see computer science and physics, jobs as a major contribution and also biology, and then we see that for medium scale, jobs we see biology, has the maximum number of contribution for small scale jobs.

B

We see biology physics and computer science, so this kind of makes uh one more thing that for for, like flagship jobs, use users are usually running. You know, like small scale, jobs first, which is kind of understood as well, but this is kind of giving us a qualitative perspective as well that, since these three domains are the major contributors in medium and flagship scale jobs, they are also the maximum contributors here.

B

That means these jobs very first, uh we can like kind of derive this conclusion that okay, these jobs are first ran on a small scale and then now they are running when they want to run into the production skill. They then I mean, then they run on the large scale, whether it's medium or flagship.

B

Then now we also wanted to see the the file system that these jobs are using right. That's one of the like kind of the important thing we want to see whether these jobs are really using uh the bus buffer or not and and if they're, using in what percentage.

B

So that was one of the things so that this is uh surprising to us in the beginning that out of all these nine science domains, only four of these are using burst buffer and the others are. So there could be different reasons right.

B

The the one of the thing could be the the jobs or or the the workflow that are designed for them may not be easier for them to move it to buzz buffer, but then also, maybe they are not aware of the burst buffer or uh it may not be comfortable for the other science domains to use uh the buzz. But so that was like one thing: that, on the limited number of users are using buzz buffer and even even in those science domains like, for example, computer science and biology.

B

The amount of birth buffer jobs are less compared to the jobs which are running on gps. Also, I and I think it will be also good to define like how do we. How do I define what are buzz buffer jobs so so again like when we go to the file system, so I mean to the data. There is also like. uh uh There is one more field here there, uh which is like mount path or something where we see whether it's like. What's the what's the file system, whether it's bus buffer or whether it's gpfs.

B

So if there is a file in in all the files for a job, if there is even one file which has used buzz buffer, we will call it as a buzz buffer job, so so bus buffer jobs can have files from gpfs as well, but it will have at least one file from the burst buffer system file system, whereas uh gps jobs are exclusively gps, jobs like they do not use burst buffer at all, and then there are other file system, but I primarily focused on these two file systems.

B

C

The the distribution for um all jobs or for the ml jobs for.

B

B

All the analysis that I will be talking from here on are for 25 000 ml jobs that we extracted from 300 000 ml jobs.

C

Yeah, that was the the origin of my question. It looks like on this scale. You have a lot more than no it's a long scale, so maybe that is 25 000., okay,.

B

On the previous slide,.

C

No right here this it looks like you have two entries here that are around ten thousand, um so it wasn't sh it looked like for a moment. It looked like a lot more.

B

C

Thousand but it's a log scale, so the.

B

The ones that are.

C

A little bit smaller are actually quite a bit smaller.

B

C

Yeah, okay, make sense.

A

So steve has a question about collecting data for jobs reading from node local storage, but I think it's also. Maybe could you explain what burst buffer means on summit, because it means something different for us locally.

B

So burst buffer.

A

Is note local right.

B

Yeah yeah yeah, it's it's a little bit different from nurse, not a little bit. I mean it's a different architecture yeah, so I I I in, I think it's a luster file system, but here we have like a like a local storage within the node so and which has like, which is nvme ssd base. So I don't know how it's on nusk. So so that's the difference here we don't have like a separate. uh Like I mean the storage server for for buzz buffer. Every node has its own uh local storage.

A

Is it a terabyte or what was the size of the version of node? uh I.

B

Yeah it's in terabytes, yeah, okay, yeah, uh and so they are like individual like nodes. But I think when, when we run, I'm I'm not sure about this, but I think when we run uh like a let's say, a job on five nodes right, so I think they will make a network and we make we can distribute the data across like different um those five nodes. Even even though we call them node local, but I'm not 100 sure about that part. Yeah.

B

So yeah so again like we now we want to see the distribution of jobs um on two file system, but also across the different uh job sizes, flagship jobs, medium jobs and small jobs. So so hey- and so here we see something different from the usual trend and which is like the computer science jobs are are running on.

B

There are like first of all, there are very few jobs which are on the flagship compared to the other jobs and then but but the trend here is that we see more of the computer science jobs on burst buffer than on uh gpfs and the other science domains. They do not at all use the burst buffer when, when, when they're running the jobs in flagship, however, we see, but because again these are like the fewer numbers. So that could be one reason.

B

But when we see on the medium-sized jobs, then we see that they are all for science domain use buzz buffer, just like the small jobs. So this is one one of the thing that why like they would not use buzz buffer on the flagship jobs. We don't know the reason, but that's one of the finding here and and then we want to like characterize these ml jobs. Also based on you know like how much what's the whether they are really intensive, writing can save or read, write intensity.

B

Usually we think that they are read intensive, but they, but that's not the case, always that's what we found and first, first of all, we define like what are read, intensive, write, intensive and read: write jobs. So we we use this equation here uh like read: bytes minus right, bytes over read, bytes, plus right bytes, and if the value of this uh like number of read by this equation, lies in the range of minus one to minus point five.

B

That means there are overwhelming number of right right, bytes and then we will call it the write, intensive jobs, then.

B

Similarly for others as like, if the if the value of this equation lies from point five to one, then we will call it read intensive, because there are overwhelmingly number of read bytes and if the result is near around zero, which is like point five to five. We call it as both read and write so when we like distribute.

B

So this is a density plot which we should, which shows that the denser the plot, the the region is the more number of data points are there, and it shows that uh that for gpfs and for burst buffer file system there's a different trend here for for burst buffer. Usually we see that it's it's like uh um divided into like multi model like, for example, whether it's like either the jobs are read intensive. That means the number of bytes are very small.

B

Here are the number of bytes our bytes written, are very high and then very small read bytes. So this is something different from what we see here on gps. So one thing the for gps, the number of bytes, which are sorry number of jobs which are in read, write. Our range uh are far more greater, where we don't see anything here rather than few few sample here.

B

So that's one thing different and when we see it quantitatively, this is how it looks that that read, write jobs are very small here, but we do have uh significant numbers here and then the other finding is that gpfs. Like has computer science and chemistry, has I it's still higher number of read intensive jobs in gpfs, so there are opportunities. If they could move to bus buffer, they will have like a better performance.

B

Now the other thing that we want to see is that usually- uh or we know that gpfs or the file systems are good, are not good are suited for small calls right for them. We have designed burst buffer and if, if the job has that kind of a I o access patent, then they should uh definitely use buzz buffer. So that's what that's what we wanted to see like? How are the? What are the range of these job?

B

uh These I o calls and then again we see that uh that overwhelming number of calls are very small, like less than one megabyte compared to the other other other call sizes, and if these call size are are done on purse buffer, it would be it would give better.

B

I o performance, so that's uh another thing again from the data set like we don't get like less than one megabyte like there are far finer uh size of uh bin, but we combine all of those bins into one category of less than one megabyte, and these are pretty consistent across all the science domain. But in few cases we do see like some bigger numbers, even in larger bins, but it's still the percentage. If we see it's still like very high percentage in less than one one megabytes.

C

And the the reason there's fractional values, because they're averaged across all the runs within that domain. Yes,.

B

B

uh I I think they are not average value. These are like aggregate values.

C

This was trying to understand how a call could be fractional, so I was thinking it must be some kind of average or normalization.

B

Oh, it's also, these are not fraction right if you, if you expand these number so.

C

One greater than one gig gets 2.48.

B

Oh yeah yeah, so yeah these are average. Yes, you're right, yeah, okay, so I think we aggregated for one job like all the calls and then if there are let's say, 100 jobs in that call, then we would take the mean of that yeah.

B

And, and not now, in this slide, I'm showing that distribution of jobs having persistent files and when we say persisting files here, we meant that how many of the files are that are being written to the burst buffer are then being persisted to the persistent file system, for example, parallel file system here, r or the other way around. If, if we are seeing uh files on the bus buffer, then what fraction of files are being copied?

B

If, if we are seeing a files on on gpfs, then what fraction of those files are being copied to um bus buffer and then being read from there? So that's the like the idea here, like looking for the common files on both file system during a job so so, and we have divided in based on the science domain and then based on the size of the job here. So uh I think for for for biology for almost all science domain. We don't see much of the activity here other than the computer science which has like.

B

So let me first also explain these bars here, so I think it will be uh make it more more like easy to understand. So we see a blue bar here, which is right, right, re, read intensive files, and then we see reading things if common files again, I call it common whether it's like whether for read or for write right. If we see a common file on both file system, then we will call it a common file.

B

So there are so, for example, there are no jobs in biology which are uh which are common, but there are files which are like read intensive files so and for then for write intensive, let's see for right, intensive, there are files, uh there are jobs, and then there are equal number of jobs which have like common files as well.

B

So if, if a job is using a if a job has like, if, if a biology has only one right, intensive job, then that job is sharing its data on both file system, it could be whether the file has been written and then moved to the gps or the whether the file has been first moved to the burst buffer and then read from uh buzz buffer. So that's how we will see these plots here and it's. The idea is to see like how the different domain sciences are using the file system.

B

So so we don't see uh so much of like a diversity here, but only in computer science domain and and when we see this across the uh like the size of the jobs, it's like flagship medium and it's small. We see much more diverse city here and one of the reason could be it could be because, like now, computer science jobs are dominating in flagship right and they will also have some percentage of the jobs in these two categories. So I think this pattern is being reflected here.

B

So that's that's the reason I think one of the for or an explanation for these kinds of plot for this behavior of the plot and then now moving further and not so now we saw the number of jobs right having the common files, but we also wanted to see the like the number of bytes that are being copied. So if, if we have like a uh like us, a file which is being written again and again right, but if it's being copied only once to the parallel file system, so is that is there?

B

Are we able to track those kinds of activities? So that was the idea behind looking at uh the number of bytes here what we are looking at so, for example, for this read plot, we are looking at total number of bytes in a in a job and then total number of by bus, a bytes red and then the total number of bytes persisted. So total number of bytes read and burst, but and number of bytes right from the burst buffer are almost same. But then none of those are persisted.

B

So I so here what we can see from the pre com, if we compare it with the previous figure that it kind of support that that if we are only looking at the file, it kind of gives us a general idea of the number of bytes as well. There is no such difference that between the number of bytes and number of files, so again we see we don't see much of a like a different pack in here. They all behave different. Similarly, like we will see a similar.

B

um These two box have high value, but then the persistence box have less value in these three domains other than computer science, and the pattern is similar here as well. These are all zeros, because there are no jobs for for write, r or.

A

B

B

And now uh so now we saw the different behaviors of the of jobs and files on both file systems. Right and now we want to see their average behavior on different file system in terms of bandwidth.

B

So some of the about things are like kind of known, but it's kind of giving us a quantitative value than what we already assume. For example, we see that burst buffer has a higher performance than than gpfs in terms of like megabytes per second for read and write. Oh sorry like this has a bigger value right. So overall we saw that read by sorry.

B

Jobs has five times better performance on buzz buffer compared to gps, and right has four times better performance compared to gps, and then we also kind of divided uh distributed the data across the science domains and saw that are these behavior consisting so in so they are kind of similar uh across the uh like.

B

I mean that if we compare their mean values like it's almost similar, there are some variations, but we cannot uh like just conclude anything from there and so is for for right, right jobs here, and we see some zeros here, because there are no jobs for that category, so that that was like the performance and the cross-sectional view of all the machine learning jobs that we could find found out from the uh darshan traces.

B

And now here we are trying to show the how that are there like uh machine learning, jobs, evolving and by evolving I mean? Are we seeing more and more of machine learning jobs on summit over a period of one year that we that we were analyzing, and these are the trends that we find so, for example, if we see uh the cdfs of machine learning jobs that are running on gpfs, then we see that there are uh okay and before that uh these red lines shows the 50 of the jobs right.

B

50 of the jobs are on the left and 50 of the jobs are on the right and the x axis is that is the month. So we start with january february march and go on until december here. So if the, if the red line is in between, uh for example, something like that, that means the 50 of the jobs ran in the first half of the year and the next 50 of the job ran in the other half next half of the year.

B

So so, if we, if we keep that those points in the mind, then we will see that most of the red points uh lie towards the latter part of the year. And that means like in september october november, that that shows that more and more of the machine, learning loads or jobs are running uh are in the later part of the year, or we see increasingly large number of machine learning jobs, uh jobs as we as the system matures and but there are some some, I mean anomalies to it as well.

B

We will talk about that, for example. So, but before that, for example, we see computer science jobs, they are like that. We see the trend is rising and- and so is the case for other other other science domains.

B

But we see some of these sharp rises as well, and even though we see us, we are seeing this this kind of a trend here, but this also attributes, but we cannot take it as a face value, and by that I mean uh that there are like very few users who who who ran the job during this month right and then nobody used across the other period, but it's still uh like. So we cannot say anything conclusively right now from this data, but we still see that.

B

Okay, like these things, happen at the later part of the year. That means more and more jobs are we can foresee more ml jobs being run, and then I talk about.

B

I will talk about this anomalous behavior here, and this is for computer science and for buzz buffer, like where 50 of the jobs ran in january itself and the the the so when we saw this plot, then the discussion that we had was that it was, uh and I'm not saying it conclusively, because it's just based on the discussion that we had is that there were like lots of test run for this case. This scenario.

B

That's where we see uh lots of jobs which were executed in january itself, so which is like different from the usual pattern that we see so now coming on to the conclusion of the work. So we we can like say in general, there is a continuous rise in machine learning workloads.

B

We see large number of small files, uh jobs uh or we see large number of jobs using small files which can, uh if we move them to the burst buffer. If they were running on the bus buffer, they will get better performance.

B

Many users do not use buzz buffer, which there could be different reasons to it. But if it's really, but if we associate it with the ease of use, then there is a need for a architecture design like we should come up with a user friendly solution. So that's one of the way to make it much more user friendly for not for just for the computer science people from the computer science background, but for other users as well and um there's a need for to develop a generic model. I think that's what we are working on.

B

So, as I mentioned in the beginning right, we extracted 25 000 workload by certain map method, and that method is not exhaustive, like we could. We might have missed different workloads there, because if they didn't fall into those keywords, then like working on a model which can bring, uh which can like detect even unseen, like even the even the jobs which are not from those keywords and then the other thing.

B

That was that we think we we should have done or we want to do- is that compare these patterns with the other workloads, I mean which we call as a traditional workload right and then we can actually come see if there are packing difference and then, and if there is then what are the different pattern or how we can quantify those differences. So these are the things for we are working on that that's for the presentation, dude. Thank you very much, and I want to acknowledge the uh olcf and doe for this, for the support.

A

Thanks ahmad, that was great. um We have uh well, so um uh we definitely have time for some questions, so you can meet yourself to ask questions or raise your hand, but we we have one question in the chat still remaining from annette and it was about uh um how much of the behavior you see, um you know is a response to messaging and user training, and I think this is the burst buffer utilization.

A

um You know so, for example, is it possible that people have been cued to use the burst buffer specifically when their job involves a lot of reeds or a lot of rights?.

B

So I I didn't understand the question. So what's the question.

A

Well uh so the question is, you know the utilization in the burst buffer? How much do you think that is based on you know, user training and kind of what the facility is saying to users about how best to use the burst buffer versus a particular user preference like they've come up to come to it on their own.

B

Yeah, so uh we I think this work doesn't cover that part, because we were like looking at the cross-sectional study right, not looking at the individual job or for user, how they're performing so so right now we can just say that for a particular science domain, like what was the pattern by by one of the slide where I presented the number of smaller bytes read or where far much greater than uh for the other larger calls, but saying that how much of that was for training or for based on the user preference.

B

I don't think like that discovered in this work.

A

Yeah, do you get a sense of it? You know, do you know um maybe what kind of training the lcf does regarding first buffer usage.

B

No, I cannot say that means. I do not know here.

A