Cloud Native Computing Foundation Cloud Native Security Day EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Weaviate Vector Search Engine - Enhancing Cybersecurity - Bob van Luijt , SeMI Tech

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: Weaviate Vector Search Engine - Enhancing Cybersecurity - Bob van Luijt , SeMI Technologies

This talk is an introduction to the vector search engine Weaviate. You will learn how storing data using vectors enables semantic search and automatic data classification. Topics like the underlying vector storage mechanism and how the pre-trained language vectorization model enables this are touched. In addition, this presentation consists of live demos to show the power of Weaviate and how you can get started with your own datasets. No prior technical knowledge is required; all concepts are illustrated with real use case examples and live demos. Although Weaviate could be applied in a broad set of use cases, Weaviate has high potential in cybersecurity. Automatic classification of attacks, clustering of incoming threats, and better search through data in the security realm could improve the cyber security of systems and assist security analysts in analysing and preventing attacks.

A

Hello, everybody thank you for watching this video, I'm going to talk about the viviate, vector, search engine and then specifically from the perspective of threat analysis in cyber security, on mostly we're gonna focusing on I'm gonna tell a little bit about what weave it is and then we're gonna focus on the demo, um and I'm also gonna show give you some pointers. How you can start working with it yourself.

A

So we've hit is a vector search engine and it's a it's a full across support database. But the most important thing to bear in mind is that the the difference between traditional search engines and vector search engines like we've yet is that we've yet focuses on the vector representation of the data that is stored within it and those factory representations are given to it, based on machine learning models and the simplest way to show you. The difference between a traditional search engine and a vector search engine is the following.

A

So let's look at this data object, so we have here data object representing a in this case, cybersecurity threat group. So we have here fin6, and then you see like a group that has focused on an attacking point of sales devices.

A

Well, if he would store this data object in a traditional search engine and we want to retrieve it and now we say, for example, we are looking for something related to finance hardware attacks. Then it will not find anything because we know that fin6 is related to these. This query, but well, the um the exact keywords are not matched in this data object.

A

However, when you do this in a vector search engine like we've, yet it will actually return fin6, because it know that there's a relation between finance hardware and attack based on a group that focuses on attacking point of sales devices, and that is the most important added value of these vector search engines like we've yet and, as you will see in the demo, a lot of new use cases come from this.

A

So when it comes to the cyber security domain, our main focus or our main question that we want to answer is like. Can we go from like a rule-based approach to a prediction-based approach?

A

So a little bit about the core features. So with that two core features, the first one is search or, as we like to say, search and discovery in your data, so the demo that I'm going to show you is completely focusing on search and how to find things using we've yet.

A

But on the other hand, we also have classification, and the easiest way to think of classification is context- is that we've yet can automatically make relations in your data sets. So where search there's some form of human or machine input to get to insights with classification, you can ask we've yet to do that automatically.

A

Then we have some other well unique features from eva's, so, first the modules as I just described, we have a bunch of them. If you have ideas for modules, we're more than happy to hear, um you know how you think we can improve vva's module ecosystem by the way. You can also create your own modules.

A

Then we support any media type, so we've had not only the vector representations that we've yet can store, but also the data objects can be of any media type. So, within one single vp8, you can store textual text objects and text effect vectors to represent these data objects but also, for example, images or video we've hit itself has a graph like a data model.

A

So what you will see when I show the demo to you, we use graphql graphql is used to not only do the machine learning matching, but also to make traditional graph relations and, most importantly, to mix and match them. And then, of course, we focus on we being scalable and fast, and we're constantly updating and improving we've. Yet so, with every release, we get a bit more, a bit more faster and a bit more scalable.

A

So when it comes to the um the vertical of cyber security, um uh we mostly focus on intelligent threat analysis, um meaning that we we've had focuses on structured data, so that can be descriptions, text, documents or those kind of things that are related to a threat analysis.

A

So for this demo, we're using the metro tech framework for the simple reason that it contains a lot of graph relations, but also a lot of unstructured data.

A

We use a vva transformers module, which is fine to fine-tune with cyber security data. uh In this case we use sentence birth, and that brings me to the demo.

A

So when we are in the we've hit console, we get an interface where we can use graphql to query true vp8 and the setup how ev8 works is actually very simple so um with it has three core functions, meaning that's aggregate that is explore and that is get and aggregate is used to well get an as an aggregate function.

A

So, for example, how many objects are stored in wv8 explorer is used to search for the complete vector space, but get is used to make a mix of vector searches and graph searches and that's what we're going to use for the demo inside vv8. You have a graph like data model, so that means that you can create any class and any property. So in this case our class and property structure is based on the meteor attack framework.

A

So let's take a look at the first one. So, let's take a look at thread groups, so the thread groups have a name- and this very simple query says like get me thread groups and show them show me their names so running this query. You see a bunch of thread groups they're, not organized in any way, they're just randomly shown. Based on this query, we can also go add properties, so we can say, for example, say like a show the description and there you see, for example, copy kittens.

A

You see the the description of the the thread group. So let's go one step back now. What we now can do is that we can enable the machine learning model and in this we've had instance, we have two modules, enabled the factorization module and the q and a modules. So, let's start by looking at the from the perspective of the q, a module. So what we can do is that we can say well. We want to ask a question.

A

Then, of course, we have the question, and um the question that we want to ask is: who targeted the government in the middle east make an array? That's incorrect should like do it like this.

A

We can se the properties that we want to search through, and that is something we want to do, based on the descriptions and then we have so-called underscore properties additional or we can say well. This is where we want to see the answer to the question and we want to see the result so very simple: question, a query where we say based on these thread groups, search for the descriptions and show us who targeted the government in the middle east. So let's run this query so here we'll find the answer oral rig.

A

What we now can do with uh with y8 is that we can say like well, for example, also include the name of this thread, the description of this thread group, and then you see that the name is actually also oil rig.

A

But if you're also gonna look at um uh the description, uh then we see that actually inside the description, there's more information about oil rig. So let's go one step deeper, so let's say like uh what kind of attacks does all rig use and we find that in the document white ulric, the answer is found, which is supply chain attacks.

A

um We've now been looking at the graph from the level of the thread group, so let's take a completely different approach and use the supply chain effects in there. So, let's start from the perspective of the attack technique, and so, if you now also say I have this simple query say where we say: get attack techniques and show me the names and if we run this query, we just get a bunch of attack techniques coming from the metro attack framework.

A

Now, when we hit, we can do question answering, but we can also do a near searches, so we can also search for neighboring concepts. So let's say, for example, here we have near text and we're going to look for concepts and then for the concepts.

A

Let's look at the supply chain attacks and let's say for this query limit that to the first 10 results.

A

So if I now run these queries, then with supply chain attacks, then the most neighboring concepts are, of course well: the supply chain, compromise, the supply chain compromise or the compromising hardware supply chain, etc.

A

So a port was opened at the hosts firewall so same query, but we're now going to say well inside the the attack they somehow opened. uh They opened a port at the hostess firewall.

A

We want to be at least 80 certain, and we want immediately shown what the known mitigations are for these solutions and we want to see names for these types of attacks. So thank you very much for listening. My name is bob fallout feel free to reach out to me on bob at seminar technology. If you want to learn more about how bv8 can help in your domain or with your the challenges that you have on semi.technology, you can also find our open core software, so you can play around with it. You can try it out.

A

So I'm looking very much forward to hearing from you and hopefully we can help you taking the next step in intelligent threat. Analysis. Thank you. So much for listening bye-bye.