ONNX June 2022 Community Meetup, 13 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Improving the online shopping experience with ONNX

Description

Building and deploying AI solutions to the cloud at scale is complex. With massive datasets and performance considerations - finding a harmonious balance is crucial. This session will outline key learnings from deploying a Serverless application running inference on a sci-kit learn model using ONNX Runtime, and will share how to utilize the capabilities of ONNX runtime to improve the online shopping experience for shoppers and global brands.

Matthew Leyburn is a software engineer at Bazaarvoice in Belfast, Northern Ireland. After graduating from Queen’s University with a BSc in Computer Science, he joined Bazaarvoice where he has focused on improving the online shopping experience through the use of AI. Matthew is involved in delivering e-commerce machine learning solutions and optimising cloud performance at scale. Matthew is passionate about harnessing the capabilities of innovative technologies to solve real-world problems.

A

Hi everyone watching live at the microsoft silicon valley, campus and watching virtually as well. The single biggest change in the shopping experience in recent years is the digitization of commerce, brands and retailers have gone from fighting over limited shelf space and physical stores to fighting over the infinite shelf space across the internet.

A

Standing out amongst the crowd is now harder than ever. One advantage. E-Commerce companies possess is the scale of data they have and collect. Creating impactful insights using ai can be the differentiator between winning and losing. Today we're going to explore how to tackle big data problems with lightweight solutions with the help of onyx.

A

Before we get into that a little bit about myself, I've been fortunate enough to work with a range of innovative tech for several years now, I'm currently a software engineer at bazaar voice and over the past year, or so, I've been focused on harnessing ai to improve the online shopping experience by delivering ecommerce machine learning solutions to the cloud.

A

So let me try to illustrate the scale of data in e-commerce and why the application of ai is so compelling by using bizarre voice as an example, bizarre voice manage and process data for over a billion shoppers a month.

A

They have the world's largest network of brands and retailers, and they provide over 8 billion total reviews for these brands and retailers. The variety and data and scale of data here would cause any data scientists mouth to water. It's obvious, there's massive opportunity here to innovate in this sector using ai.

A

It's the reviews I want to focus on today. Specifically, the bizarre voice network allows brands and retailers to share product reviews across cat logs, meaning one review on one site can be shared to every place that product is sold online.

A

More views equals more consumer trust which ultimately equals more sales. The process of sharing reviews is called review syndication, but this is a topic for another day.

A

We want to explore the concept of sharing reviews today and that concept is powered by product matching product matching is the process of determining if two products are the same, it's the core function of the machine learning model we built product matching is performed by comparing unique product identifiers, but whatever is left over and can't be unmatched is done manually or in other words, by humans.

A

In fact, over a million product matches a month are performed manually. You can imagine how tedious this can get and expensive as well, but how did humans product match if we take a step back well, if we take a look at these two products on screen now, how can we tell they're both the same?

A

Well, we look at the picture. We read the text price, written reviews, the description and gather enough data to come to an accurate conclusion and subconsciously we're carrying out techniques that machines use to solve this problem, numerical processing, text, processing and image recognition, and these are the basis of how we built our model and, of course, we need data.

A

We gathered a data set of over three million historical product matching human decisions, to train our model to 99 precision rate.

A

So that's the first part of our solution, the model, a scikit-learn model built with python, exported to onyx format. Now that's all well and good, but how do we implement this into our services to generate real business value?

A

Well, let's now look at approaches to deploying machine learning solutions to the cloud. For me, the ideal solution is an implementation that is the most lightweight cost efficient solution that also maintains performance.

A

With that in mind. These points here in this slide are based on results. I've personally seen in the past, the y-axis is complexity on the x-axis. It's cost from high to low. First of all, we started looking at virtual servers like azure virtual machine or aws ec2s.

A

This can give you the most flexibility, as you have an actual os to work from, but comes with some unnecessary configuration costs, especially if you're using it specifically for machine learning I'll touch a bit on costs later, but why over complicate things when we don't have to the next consideration?

A

Is a cloud ml platform like azure machine learning or aws sagemaker? We now have the benefit of dedicated ml tools at a lower cost in some cases in a virtual machine, but we must remember at this stage. It may be the case that the machine learning engineer hands over the model to an engineering team to implement, and personally I want a development pipeline. The software engineers who don't specialize in machine learning can pick up and use seamlessly.

A

That's when we arrived at our third option. Serverless functions now like azure cloud functions or aws lambdas. These remove the overhead of managing servers, configuring software and patching operating systems and they're also very cheap. So this is the option we explored.

A

And this is the solution we came up with, so we developed a psychic learn model exported to onyx forma and built it with python, deployed it to a serverless function, and we used a node environment to run inference on a onyx. Runtime onwards gave us the flexibility to be able to build a model in python and deploy it to whatever environment we wanted.

A

We chose node because that was based on similar approaches, we've chosen in the past, so it really worked well for us, and by running this we can send predictions out to our network, and just by using this lightweight solution, we are able to process hundreds of millions of pending product matches within the timeout limits and deployment size limits that are part and parcel of server's functions.

A

What's also great about this solution is the modularity of it. We can plug this in anywhere to any of our services quickly and easily, but how do we scale this solution to hundreds of millions of predictions?

A

Well, if we look at this table, this is based on results. I've seen in the past, with machine learning inference on serverless functions using onyx runtime.

A

These metrics help us find the optimum memory size to avoid timeouts and ensure the best performance while keeping costs low.

A

I like to use the metrics memory used and execution time. If the memory used nears the total allocated memory or the execution time nears the timeout limit, we know we need to scale higher. This is an instant indication that we need to scale. If we look down the layout at allocated memory, we can see that at four gigabytes, performance, plateaus and therefore two gigabytes- is the ideal memory size for us as it stays within the memory usage and easily within the timeout limits and is also the lowest cost there at 2 800.

A

Now, if we compare this same model with the same compute loads across service functions, applied ml platforms and virtual servers, we can get a great grasp of the cost savings here. You can see that service functions is significantly cheaper in our case platform and significantly again cheaper than a virtual server.

A

So there's great cost saving potentials here by using this lightweight approach.

A

One consideration, I'd like to bring up is that python packages add up very quickly.

A

We can see with this chart that the orange and a blue line, donate deployment, size limits of service functions and that popular machine learning frameworks like tensorflow and pi source already exceed these limits, and that's even before we add any logic to our serverless function.

A

Onyx helps us reduce the the size of our models during deployment, but sometimes our deployment sizes are too big. There also are workarounds around this, like fashion or model from a different cloud storage source into our serverless function, but again, onyx helps us reduce the size of these deployments.

A

So the two main considerations to these approaches is the deployment size limits and working within the timeout limits.

A

But I hope this demonstrates the power of onyx and onyx runtime when deployed to a service function.

A

Using this lightweight approach, we are able to match hundreds of millions of products across thousands of client catalogs at high speeds, resulting in over 15 million extra reviews for brands and retailers, proving that this approach is fit to process data on a grand scale.

A

So we've explored just one way of utilizing the capabilities of omix and anextron time. If you want to hear more or want to share your use case with me, please reach out I'd, love to hear it and as someone actively working in this space, I'm both intrigued and compelled by where these technologies are taking us. Thank you.