National Energy Research Scientific Computing Center (NERSC) Data Seminars Series, 7 Feb 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020-02-07 - Mariam Kiran - Time-series Analysis of ESnet Network Traffic

Description

NERSC Data Seminars: https://github.com/NERSC/data-seminars

Title: Time-series Analysis of ESnet Network Traffic: Statistical and Deep Learning models

Abstract: Predicting network traffic can provide information on large data movement and details on how users interact with the network. As the area of ML matures, we are investigating building predictive models that can provide suitable predictions into the future on how traffic will behave on the network links. The goals of this prediction is to investigate anomaly detection and understanding congestion patterns on the network to help manage them more efficiently. In this talk, we will present our results on time series analysis and also show we plan to deploy these models to perform real-time ML predictions.

A

Okay, hello. Everybody thanks for joining us today very pleased to have one of our local quickly lab friends with us: Mariam Kiran she's, a research scientist for the joint position and yes, net and Ciardi and charity shorts and the scientific data management.

B

A

C

A

Software machine learning techniques to advance system architectures, particularly high speed, no.

A

Reward from oscar in 2017, so whenever you're ready, they're. Okay, thank.

B

You so yeah! Thank you again. So it's been some time since I've done a talk to internally, so so this is work which so which we're doing under the oscar ground. But it's two years in so we still have three more years to go so, but we've had some recent results and I wanted to present what have so far so so the okay. So.

B

So in this work, we're going to be talking about how we're experimenting with time series prediction particularly to do with network traffic.

B

So the outline of the talk is I'm gonna start off by talking about how we see the future of networks and how it will leverage AI techniques to actually optimize how we currently use our high-speed networks. In particular, I'm talking about wide area networks for the techniques we develop for wide area networks can be transferred to IOT and even smaller kinds of networks.

B

So we look at network traffic prediction as a classical problem, and we've been experimenting with various statistical techniques and deep learning techniques and various models to see how we can improve the predictions and actually anticipate what kind of traffic we will be getting on es net before we any optimal decisions. So the last part is how so once we've got the algorithm it's working and we actually need to deploy them into production and somehow link them in with the tools so that we can actually leverage the decisions coming in from the AI.

B

So we've been building a platform or a complete infrastructure which is hosted on GCP. Our Google cloud platform and I'll talk about some of the ways in which we've done it, where we can actually deploy multiple models in the wild and test them with real-time traffic, as it comes in on the network.

B

So just a background, so I lead a team where I'm trying to leverage AI to aid in the network research. So we are looking at various aspects of this, such as predicting how our network really resources, are being used which goes from Reuters, which is how the bandwidths are going to look like in the future and also how we can turn our network devices into cell phone devices. So we're leveraging a lot of deep reinforcement, learning to actually work with the devices.

B

So the controllers look at what is happening on the network and then they change their behavior and how they see fit. So a lot of these approaches have deep learning in it, but the the key difference is a lot of the data we work with is system logs time series logs error, reporting logs no images at all so far. So that's the unique challenge we find in our team learning approaches that we need to work with very boring sets of very large data sets.

B

So this is a picture of how esn't does capacitive projections, so this is showing a graph since 1990 and predicting what the traffic is going to look like next year 2020, and we see that over the years, we've always had like an exponential growth of traffic, and we have a 62% growth every year of traffic, which is on TV. So obviously it has a lot of operations, challenges such as managing the scale and different kinds of heterogeneity.

B

We see this new era where a lot of AI workloads are going to be running on our networks and obviously super facility, as well, so with an anticipated.

A

B

Petabytes and maybe March 2019, how can we address these challenges from an infrastructure point of view and make sure that the network doesn't go down a so a lot of its? Not six efforts which I am currently not part of, but um on the site of are concentrated on addressing the scalability challenge, so they're, looking at it from a very core network perspective network engineering perspective and how they can update the current network environment to actually spoke with these.

B

My work sits on the side where I'm just trying to see where we can bring in AI and leverage that, with an es, not six and possibly push it, push the border a bit further.

B

So if we look at so with that anticipated growth, if we look at the current bandwidth usage on yes net, this is a graph which shows you, the bandwidth usage of the whole twenty nineteen from January to December and I plotted a histogram of what was the capacity being used of the links during the whole year, and what it shows is that most of the year, only forty percent of the links are hi are the ones which are used.

B

There are only some links which go above forty percent, and these are particularly transatlantic links, and they always they sometimes also go above one such as they have very high traffic, which goes on to the backward, the back, the back up links now, a reason why this is is because the network is designed to have to go up to forty percent, so we do have a cap, and the reason is because we do see a lot of bursty traffic on our networks, so we have to count for a sudden burst.

B

So there is a sudden big data transfer coming from CERN all the way to LBNL, and can can there be enough space on the network to actually cope with that extra burst. So with all of that, we also provide very high quality of service, so we are known for being a lossless Network compared to AT&T and other providers, and we do up specialize in delivering big data science transfers as well. So we monitor separately.

B

We have a number of monitoring tools on the network where we can monitor loss throughput and then you can also monitor what kind of traffic data the amount of data is moving from different links across.

B

So the question we're posing is that, given this is the current infrastructure, can we run the networks hotter? So right now we are optimized to go up to 40%. Can we do something where we can go up to a hundred percent with the current infrastructure we have? So some of the inspirations we draw are from operational wide area networks which include Google and Microsoft, so they have these seminal papers and 2013 where they discussed their ways in which they have optimized their networks and gone up to a hundred percent a lot of those.

B

So there are very common themes in those papers which talk about central decision-making and with an optimization function, so they monitor what is happening currently on the network and they have a lot of interesting, optimization to actually move traffic from one link to the other, to demonstrate that we can actually go up our network utilization.

B

However, what happens is that those solutions are specifically designed for their networks and they are not transferable to our networks. That's one issue. The other issue is that these networks have worked with applications which they have designed themselves.

B

We work with networks where our applications are written by scientists, so we usually don't see what an application is doing on the network until it actually interfaces with the network so and also do e traffic tends to be very varied from very large, bursts to very small file transfers with R, which are many in number, so the traffic nature is very different, particularly because we deal with science datasets.

B

So that brings the motivation for this work, so current approaches which we follow in ES Net have or in wide area networks have to do with manual configuration, so they rely on standard routing parameters which come from experiences of network engineers and usually the most robust algorithms are the shortest path calculating algorithms.

B

Now the challenge which these algorithms have is that they can pretend lead to a number of connection points. So we have situations where there are certain links which are always hot or very highly utilized, but at the same time there is another part of the network which are not being used at all. So these are underutilized links, but have many number, but because of the way the routing algorithm works, those aren't those are usually not selected.

B

So we want to see if we can use prediction to actually do intelligent routing decisions so try to utilize these underutilized links see if we can actually move more traffic on the current network. We have so allow basically more signs as possible, so we don't have to wait for a particular slot where the network will be good. We should be able to utilize what we have and also use traffic prediction patterns to actually work out.

B

If this path is not being used right now, can we just use it even though it's not what a shortest possible route result. So we drew inspiration from Google Maps here, so this was an idea which so our director had this idea. So credit goes to him. So so we were thinking about how Google Maps actually does this, where you can put in a source in a destination, and it gives you what the next 24 hours is.

B

Gonna look like and based on the graph you have you can choose which slot you want to do your travel on, and you can also get real-time updates. So there is an accident or there's congestion on certain roads, and you can choose different paths based on this. So what we want to do is we want to bring that inspiration and those ideas into our our system here.

B

So we're tapping into all real-time data, which is not has right now, such as persona logs, which are monitoring the lost happening on the network traffic statistics coming from the SNMP Rooter interfaces.

B

We also have access to netfloor logs and also we can use all of this to actually give you a transfer completion time so based on the prediction results, it would tell you that if you do the transfer within this lot, your transfer will be finish the next five hours, so that would allow us to actually build up a number of workflows in that process. So that's the golden golden prize at the end. So the way we handle this google map idea is to leverage the time series predictions.

B

So we take in current a network traffic statistics coming from every router, and then we try to predict how the next 24 hours is going to look like in what this is focused on is just 24 hours right now, but we can expand it further, but right now we're trying to see what are the just-in-time pots we can take so basically we're trying to create solve the traffic engineering problem, which and other patterns where we can reduce congestion and how we are currently utilizing it another way of looking at it is that if you see what the next 24 hours is going to look like engineers can make intelligent decisions that okay I can turn I can I can not turn, but they cannot actually leverage which, which particular links they want to experiment with in the next 24 hours, because it's gonna be less busy, for example.

B

So they can do software upgrades hardware, upgrades anything and then plus. If we have predictions for the next 24 hours, we can also link it in with raising alarms. So that introduces a lot of Network automation capability which can alert the engineers if a certain traffic is happening on the network which we didn't anticipate, etc.

B

So the problem we found is that networks are actually very noisy, so this graph, basically just shows you the transatlantic links from esn't, and this is the total traffic in the year 2018. So we see that all of this is very noisy. It's very difficult to predict what the season seasonality in this data is. There is also a lot of missing data, because some of those switches were not working or the rooters were turned off and there was some issue over there. So actually working with this data is a little bit of a problem.

B

So we investigated like various methods, we started with statistical methods first and then we moved on to deeper, as we got a bit more mature in our research. So we tested ARIMA, holt winters answer email, crotches, and then we moved on to a deep yearning.

B

So I'm now going to go to the the results we got from the prediction algorithms themselves.

B

So a lot of literature analysis was done before we started, so he did find that there is no application or time series on real network data, and the reason for this is because network data is difficult to get and being an es net.

B

We have the luxury of having access to these datasets, so in eres, a good plus point that we can actually test our algorithms with real time data with real network data, so statistical methods are good, but they are difficult to tune and scale, and there is a lot of free stuff which you have to do to make sure that those statistical methods would work. So we found a lot of literature which sometimes supported a statistical method, especially if you have seasonal patterns in your data sets.

B

But then more recent work is supporting a lot of deep learning approaches because they can learn much more complex, datasets and also reduce the processing time.

B

So this is the kind of traffic we use. So all of the data is time series traffic statistics. So every Rooter interface has this kind of a monitoring tool which can actually record the number of gigabytes moving in and out from from the Rooter that is recorded in gigabytes, and it is recorded at 30-second intervals. So we collected this data from es net. We also found open source datasets from UC San Diego, who does a lot of internet research, and we also found the Internet backbone data from Japan and us.

B

So we used all of these data statistics to actually test out our machine learning models.

B

So we first started by looking at seasonality.

B

What we found is here that with ARIMA you need to so what are the way I Remo works is that it converts your time series into a stationary series and which basically means that it assumes that there is your mean and your variances is stable and it's much more of a linear regression approach. If your data is not stationary, then this algorithm fails in Holt winters.

B

It assumes that there is a trend and seasonality in your data and also it smoothes it out the way our network traffic is up and down up and down up and down. So that smoothing thing actually does not work very well with our predictions. The best results we found were with sarima in these approaches, and this was because it had additional four variables, which includes seasonality into our data sets.

B

So we have a few results and using Serena for over here we also experimented, and we are still experimenting with Fourier transforms, which basically takes your time series data and converts it into a frequency domain data, which is a series of sine curves and cosine curves, and what we did. This was basically, if you once you convert it into a frequency curve, you can filter out the low amplitudes and the high frequency components.

B

So this also smooths out your data, which makes it easy to then predict, but again, in all of these approaches, what we found is you're, smoothing out your network traffic, so they do require periodicity if these techniques have to work. So we can also do things where you can find what the trend in is in your data set or what the seasonality patterns are. You can also work out how much noise you have from the data using these techniques and actually filter that out.

B

So all of this reduce allows us to smooth that out, and you can actually improve your predictions if you were working with that kind of data set, and if that was your goal, the pro another problem we had was a lot of. Our data looks like this, where we have missing data sets in the in the box over here.

B

So statistical predictions do not work well because of the nature of our data, so we are very, very random. We don't actually have a lot of seasonality from what our analysis showed, and these missing datasets means that, even if we smooth the data sets out we're actually getting a lot of prediction errors when we use these techniques, so this is so Serena was the one which was able to give us pretty good results in that sense, so we did a lot of pre tuning of the variables.

B

So the way we did it is that every link- yes, that has a hundred and forty nights in total, which connects all of more than 70 degree sites and universities. So we looked at traffic hurling and we built a model purling. So that means we had a hundred and forty models, so we did a lot of pre tuning of all the models using this max function here. What the advantage of this approach gives you confidence intervals which grow further as you go down in the future, so they these are good.

B

If you want to have those confidence intervals in your predictions, however, again because we have sudden traffic bursts, these are still difficult to anticipate.

B

Sarima was also able to give us this result. So this is a sudden burst which we had in me 2018, so we took that data set and we plotted a correlation matrix of all the links active at that time. So what it showed is that whenever there was a burst the burst was on this link, which was a Sacramento Denver link.

B

There was a corresponding burst on the Denver Kansas link, so we can actually build up relationships using this kind of algorithm, but whenever there is a burst on one thing, this is the effect which or the traffic flow over it's going to happen. The other links at that time.

B

Yeah, so this was so we did a multivariate analysis for this. So that's how this worked out. This correlation.

C

B

Had a sliding window, so we moved to moving up that was 24 hours.

B

Yep, so from these statistics, after experimenting with these statistical approaches, we actually decided to go down the deep learning road. So the assumption was that deep learning allows us to learn more features because we don't have any seasonality and each link was behaving differently.

B

So what we did was we combined features such as we, we wanted to combine features such as conference times and weekends with our traffic data to see if that would improve and allow us to add more features in that. So the way we did it is we treated our time as discrete sequences, so all of our time was intervals, and then we had discrete values for every hour and then we built an lsdm model for those and with those we also added features such as the time of the day.

B

What the day it was if it was a weekday or weekend so right now we just added these features and we try to see how the lsdm models were improvement.

B

So we did start to start with recurrent neural networks. There is a problem: it's recurrent neural networks that it cannot learn because we have. There is a vanishing gradient problem with RN ends, so it wasn't able to capture long term what was happening so we switched to LS DMS, and that does give you a better prediction. So by default we should always do LS teams if you're trying to do time series predictions, because you can track long, long time periods in this.

B

So we so when we prepare the data, we bend it into hourly intervals, and this was the moving window technique where we used batches. So the batches were used as a moving window to actually train our LST M. As we moved forward looking at literature, a number of lsdm models were discussed. So again, there is a plethora of different models which everybody kind of tends to come up with.

B

So there is some papers we found where just one lsdm two extremes would be better or stacked lsdm, and then there was a lot of support for sequence, to sequence, learning, which is actually encoding, putting in an autoencoder in the middle of various elastin layers, because it allows you to learn. So what we wanted to do was to test which so again this is not an architect, so this turned into like which architecture is best for this kind of an experiment. So we wanted to test which architecture would be the best.

B

So just a little bit on sequence, the sequence approaches, so this has been favored as one of the best approaches, because they've used it to predict what video frames are going to look like in the future based on the current techniques are, and also it's been very successful in text translation. So we wanted to see whether it works well with time series prediction as well. So the way it works is that you have an input set and then you have a forecast so again not doing an encoder part and a decoder part.

B

And then you move data between those two and in the end, you learn a thought factor which is the learnt representations, and then you use this thought vector to actually forecast what the next feature is going to look like.

B

So this allows you to do look back and extract useful features because you're actually compressing the data and then expanding its unique draw out what is relevant and then yeah. You use that to do the forecasting, given the input data set so hyper parameter. Tuning became a big issue for us because we were working with the hundred and forty link traces with 140 independent models. The way we did it is that we had a predefined set of parameters, so the number of neurons were say between these three sets of thing.

B

We had a batch size between these two values. We experimented with different layers, hidden layers and how many new defined in those, and then we already were experimenting with different lsdm architect. So just number of layers we were using. The lesson we learned from this is that this is essentially a multi objective, optimization problem, because there are too many variants in an LSTA model, so we do have to go down a Bayesian approach or genetic algorithm to actually learn what the optimum hyperparameters are per model right now.

B

This is a scaling issue for us, because we have too many models. We have to train, so we are now going into I'm working with Judy to see if we are idea around this problem, where we can do this.

C

B

Data is different, so the model has to be suited to the day-night. It is working.

C

B

Just so that's so I'm gonna come to that.

A

B

Right now we're just working with an essence.

B

Yeah, so a precursor to your answer is that we once we trained every kind of variant, we found that different variants were giving better results for different links, and the reason for this is because of the different nature, all the links themselves. It depends on the site, characteristics, the kind of traffic which is moving on on those links, so we cannot come up with one model which is best for all things. Yes,.

C

B

One whole year, so no.

A

B

We converted one here into one-hour intervals, so 8640.

B

They were no, they were distinct patches.

C

B

B

So the model comparison performance comparison we have here, so we did this across multiple links. This is a very busy table, but I want to focus on just two columns over here, so we compared our results with the traditional Arena and home winters approaches on the different variants. We were experimenting with, so in particular, the IFR London in Birds link has very strong seasonality compared to.

A

The other links so.

B

In those cases, we actually got a better result from the statistical approach in Washington CERN. In word link. We had a lot of problem with the data.

B

In the data, so in those approaches again.

C

B

So we yeah, so we we use the standard. We don't do like from a human perspective, but if the prediction is a little later, we didn't consider it yeah. So maybe that's a good suggestion. We can work on next. Thank you. So, in most of the approaches we do find that Alice teams are better at learning certain patterns and each linked us behave differently, a much more graphical representation just to show how the thing works. Again. There is a lot of randomness to deep learning.

B

Every time you run it, you'll get a different results, so we had to average multiple runs to get so here we did adjust three times and there's blood election results. Lsd aims tend to flatline quite a lot. If you try to bootstrap as you go further into the future, they have first two links. First, two concert when you go up and down and then just flatline.

B

If so, the way we solve that is basically just add, more training data, so the more training data and the students were able to learn those patterns up and down up and down up and down better which prevented it from flatlining future.

B

There were a lot of parameters just tweaks, so we had a student who was working with us last year and what she did was she basically just compared the different activation functions to take out sigmoid, replace it with soft AK soft max Hanoch realm, and then she tried to see what the means where I married with all of those activation functions as well as, and we found that the best one was for it with Hanoch activation functions. So again,.

A

B

So many variants that we can play around with to actually make your model fit the data itself. So that's just what I wanted to show here. So some of the learnings we had from this time series prediction work is that statistical methods are good to go with. They are very quick to process very quick to write, but they do depend on seasonality in your data and there is a lot of pre-work you have to do so. There is a lot of parameter search. You have to do for all the global variables.

B

You said in a statistical model, and these can influence the predictions you make in LS diems. They are able to pick up seasonality patterns given more data sets, but you do need more ears. So now we are experimenting with two years or three years of data as well, but there's a lot of trial in there. So again it's a scaling problem because working we just don't have one thing we're working with.

B

So what we're doing next to answer your question is to actually build a bigger lsdm model, so we're trying to go down this route of a graph based on STM, where we have multiple inputs and multiple outputs and try to just actions that way. Then we just have one model to train we're also thinking of adding features such as conference deadlines into this, such as at the time around SC or other big HPC conferences to see. If that could influence the model predictions but again, hyper parameter.

B

Optimization is an issue and there any more work happening at the lab in this area just to general, so yeah.

B

So now that we've done the the various analysis and we've got like a hundred and forty different models, we wanted to actually deploy them in the wild and test them out with real network traffic. So this is a typical machine learning lifecycle where we've got the data, we've trained the models and we've evaluated them, and it's best. So now we want to deploy those models into production. The problem we have is that AI is expensive.

B

Network devices don't have very powerful computer tasks for them, we're not an HPC facility, so the machine learning algorithms also need to be tested so yeah. So this is a common problem we found so traditionally machine learning research is trained and tested on historical data sets, and that's how you evaluate how good your machine learning is doing our networks changed constantly every day, so we want to see how the machine learning algorithms are doing in real time, so we're trying to test our models in real time model.

B

Inference is also a challenge because our devices don't have a long time to wait. So the inference time has to be reduced as much as possible, so adding more layers will improve, increase the inference time of a model as well. So those are things which we have to consider if we want to actually deploy these machine learning techniques in so our goal is to productionize the machine learning we're doing, monitor how well the models are are performing and also how well they're performing real time data coming in data which they've never seen before.

B

We need a standard way to deploy.

A

B

Such as a plug-and-play technique, where we can have API is attached to different and you can switch different models at different times and also want a scalable support for many tools to access these models.

B

So the way we did this is we built this on Google Cloud. So we needed a system where we could have much more of a real-time streaming kind of a technique where we hang and access to api's. We deployed the machine learning models here and monitor their performance. We have a centralized model hub. We can perform real-time predictions, so it takes in what the real-time data is and then feeds into our model and does an entrance and sees what the next predictions are going to look like, and you can also compare multiple models in together.

B

So one model is giving you one performance, but you can have another chart showing, so you can actually switch in and out between different models. One thing we did we kind of cheated in a way to bring our costs down, because we pre trained other models on francium and nurse so and then we basically just saved the m5 files and then loaded them in. So that's how we kind of saved the costs of actually treat.

C

B

I'm gonna go into details now so the way the system architecture works. We have a lot of es net monitoring tools which are hooked into a streaming engine which was designed in firebase. So this is much more like an online based system and multiple devices can be feeding into the streaming thing. So that's how we're getting the data? You also have pre saved data, which is in the offline phase, where you have a data link and you can train different models using that, but you can also update the models.

B

You've trained using a certain schedule with the real time data which is coming in so once your pre trained models have also been updated. You can save them into come people which is cloud buckets which we used here once the cloud the model has done trained. What we did was we use cloud functions to load those models up as an API, so when the data came in, it calls those clouds.

B

So when a data came in, the cloud function ran and it called a model as an API to produce the predictions. So those predictions were offered to our service, which was not predict and other automation services, but you can also have a performance monitor or a trust dashboard, which is running in the background to see how well your predictions are performing with the real-time data which is coming in, so it's so so I have a few slides where I go into more this, so the system features we built in our flexible time scales.

B

You can set up different times at which you want the models to be updated. It's scalable, so the inference time had to be so. If we, if we had one cloud function, which was doing the inference for all 140, it was taking a half an hour. So the way we scaled it out was have a hundred and forty cloud functions for everything reduce the time magnitudes. So we can. This gives us the option of scaling the functions in and out with the number of devices we are working with.

B

We pre train those models, so we saved a lot of money on that and I think others into API. So we could set up alarms that if one particular machine learning model is not doing swap it out with another, one was performing well at that time, and this way we can provide automation services. So now, if we want to compute real time predictions to just get the least congested path in real time, it can actually go through this process and bring up the results.

B

So this is an screenshot of our service, which was demoed at SC in research exhibitions.

B

So what it has is right now, so you can define a source and a destination here, so we put an LV now in Denver, so this is kind of like a drop-down thing once you do that it gives you the next 24 hour predictions over here and then based on the different hours. You click. It gives you the alternative thoughts as well, so it gives you the least congested path in that hour. If you want to schedule your trance during that time, it also works out how long your transfer will take.

B

If you were to choose this particular slot. So it gives you like an end time for for your thing so, and we found that at certain times there might be other cuts, which would be the least congested ones, so we can actually do better pot routing decisions. This way it is right now so, with this we're actually making decisions based on what we predict, the congestion is rather than the standard network, routing algorithms.

B

So this is a very different way in in which how we can decide routing decisions in the network, so this screen also links into a trust dashboard, so we want to so if we show this to the network engineers and if they want to use this, they want to see if their machine learning ones are actually performing, actually make operational decisions, so we had so the the screen actually links into a trust dashboard where you can compare different, deep learning models and their over time.

B

So again, this is all real time, so you can grab so this will show what the real time date is and how your models are performing right. Now you can test friend models together. You can put them together and you can also see that certain models who actually performed very well that's on certain days, are deviating as they go forward. So this is again because network traffic is quite random and we have different behaviors. So at that time we can see that maybe another model actually starts.

A

B

Predicting forward so with real-time data, this is what I predict forward, and then you enter the me with what I predicted and what was reality and that's how I guess the question.

A

A

And you need those who look and see how does.

A

A

B

So there so where so, it's kind of like a backward-looking things, so this model was performing very bad, a very nicely with yesterday's data, so there is a backwards. There is a dependency on with.

B

Yes, so it's actually looking because no, you have 24 hours in advance right.

A

Yeah and you take the model every hour right, no.

B

A

C

B

Now this graph, the trust dashboard- is next hour. We want to push it to 24 hours, so that's current work right. So the the good thing is that the flexibility of this kind of an infrastructure is huge. We can hook him to any es net tool or device.

A

B

Can scale to as many devices or any changes side of things, so this can exist as a completely independent system which just hooks into what engineering decisions are being made.

B

You can also try out new models as research matures we see that we might have a very good time series prediction model today and tomorrow someone will come and say that my model is better, so this allows us to basically easily switch different models in and out and work with, real-world challenges, so we're connecting to various es Ihnen devices, but this kind of infrastructure also that to tap into IOT networks and IOT sensors.

B

So we can do the same work which we es meant devices with IOT networks, and that is one of the areas in which we are also expanding. Now the cost is a big problem. When you work with a commercial class, we have a lot of expertise. I am a software engineer myself. Our team has a lot of software engineers as well.

B

We played around with a lot of cloud services to actually bring the costs down, so we experimented with different Redis databases and other databases, and we tried out different things to actually bring the cost down. So this whole infrastructure now runs for 10 dollars per week and I think I have to manage, monitor this, because I'm actually charging this to my project, so I cannot afford to pay like multiple dollars per per month.

B

Now the the advantage of doing this is that this makes a very good case for the facilities, because I can then take it up to the facilities and say this won't even cost you ten dollars if we were to provide this service to users to use the hardest part of all of this. So again, the inference time is problem is problematic.

B

We, so it's not only just the machine, learning, training and the accuracy of the machine learning models, but we also have to produce the inference time because you can imagine users or engineers using this service, and they don't have a lot of time to wait for money, converting all the code to cloud functions. This is good and there is an additional criteria that cloud functions have to finish within profits. So all of your code has to be optimized to finish within nine minutes.

B

If you are using cloud functions, if we go down the route of using virtual machines, these are very expensive on GCP and then we yeah we played around with different cloud services and various database models. So yes, that also has a net beam project, so they have a lot of expertise and.

A

B

We can optimize how GCP can be used and peter was part of this team to do this as well.

B

Now what we can do is so this infrastructure is now deployed, so we can train per es net link or even site. We can train hundreds. We can strap them with real-time data coming in. We can automatically update these models whenever we want and offer many services and yeah also expanding to IOT senses.

B

So so this is my last slide, so we have been running simulations to see whether using congestion information can improve the traffic, and we have found that there should be a 30% improvement if and in lesser congestion points. If we were to do transfers based on what our time series prediction algorithms are giving us. So this works very well with simulations. So now what we're gonna do is we're gonna start doing this in action network, so we're interfacing with Oscars, which is another es net tool which allows you to set up virtual links.

B

So what we're gonna do is that Oscars will take in information which the time series prediction, algorithms are giving to actually set up those pots rather than using the standard routing algorithms which Oscar uses so we're working with a lot of ESN engineers to actually get some time on to network where we can safely do these experiments working with commercial clouds has its advantages, so we.

A

Had the situation.

B

Where there was a power outage during SC and a lot of demos fails because one of the devices were turned off, because our demo was hosted on GC P. We didn't have that issue, and that is something which doesn't get noted enough that to have kind of a capability, because again, services have to be kept online and running, and how do we work if we try to so? We do want to explore if there are an in-house solutions for this. But these are things we also have to think about so ongoing work.

B

We are continuing to work in deep learning models to improve predictions. We also have a very strong stream of work happening in the reinforcement, learning optimization, so we're also using those techniques to actually allow network controllers to learn, as they interact with the network, we're working on improving how the trust dashboard works, so that it adds confidence in our machine learning predictions and also interfacing, with other es no data sources or right now we can take in inputs from SNMP data sets, but you can also super lace.

B

It with other data sources, which are normally not combines and then make your predictions with multiple data sources, so this would not be possible with this amazing team, so we I don't work a little, but we also have a lot of acknowledgments from es net engineers who are constantly supporting the work we do and that is it. The app which I showed is actually hosted on this website. So if anybody wants to go and play with it, please do.