Nebula Graph Community Meeting, 20 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Fraud detection in NebulaGraph, NebulaGraph DGL integration

Description

- Project heartbeat
- Fraud Detection with NebulaGraph by Wey Gu
- nebula-dgl the project for NebulaGraph DGL integration, by Wey Gu

ref:
- https://www.siwei.io/en/fraud-detection-with-nebulagraph/
- https://github.com/wey-gu/nebula-dgl/

A

Welcome again to our never growth community meeting, so today, I will give apart from the project heart piece. I will personally get topics around fraud. Detection with graph and I will share some podcasts on another graph and the deep, deep crop library integration works for myself, too um yeah, um let's introduce ourselves first.

A

um This is way um from the we sub team I'm working on the never graph community and yeah. How about you, John, hey.

B

Away nice to see you again, um this is on um I'm also working on the uh Community.

B

um So uh even though there's only the top of us, so hopefully some of our other members would find time to join us uh later on so on. Let's keep rolling okay.

A

Thank you all right.

A

Okay, so we will have this meeting every four weeks, and so anyone would like to share anything I made. This I want to share with you something I want to discuss with you. I would like to possibly we can invite someone from Community. Just let us know so anything about nethergraph or your craft. Things are welcome to the community.

A

um Yeah I will start with project Heartbeats. uh One of the small announcements is that we will have another uh minor release uh in the upcoming days, so there will be some hot faces um yeah and for the core uh site. uh We have a bunch of newly merged um uh PRS, and uh these are only uh the major ones. uh One of our community contributor uh fixed uh fixed this uh issues um and we have a bunch of optimizations, for example.

A

uh This is uh these two are related to the uh the K Group queries. uh It's supposed to be mentioned that we initially we changed our third party, which is our.

A

We include all the third party uh projects in those republishes our dependent when I want to comply a network graph, we used to change it to 5.0, but later we wrote back to the 3.1 record correctly yeah. This is related to k-hub optimization, and this is a new interface uh in in the storage layer.

A

So it's addressing some specific goal queries, but we want to squeeze more uh performance, is slightly slightly uh same as another uh existing interface, but this is more optimal in certain cases and we have like uh this is for uh count uh star optimization, and this is another, uh yet another authorization rules uh to put filters down in the inner drawing case.

A

um Oh this is this. One relies on this change. Of course, uh yeah. If you are interested just to check out our GitHub repository I'm, not good I'm, not going to dive into all the changes on the other side. uh This is a some surrounding uh projects of the community.

A

The first one is that we are open source this project, so it's a front-end product handling, uh one of our commercial uh proprietary projects, so we open source the core part, it's related to rendering your tags or uh flowcharts in the browser and another one is uh still do have another minor release with the battle handling on the connections um and our Community maintained uh project called antibactics. So you can tell it's related to my batteries.

A

It's actually accurately maintained and evolved in last couple weeks we have a bunch of new PR's merged. Another Thing Worth to mentioned is nebula python. So in this project one of our new contributors have raised and merge two PR. So it's related to um more easily handling the return results. Value types, so we first play introduce the casting uh in this SDK. So it's quite sweet change.

A

Another thing is, uh I am worked, I was I, had been working on a side project called neblo djl, and this this is. This is a another topic today. I will share later so you can do something around mammograph plus the bgl. So this is a a paper, that's in archive.org. So if you are interested, you can check out it's. uh The author are guides from our team, so it's related to design of Naval graph.

A

uh The final one is last week: we uh we add another release format deliverable for the naval graph, HTTP Gateway. So previously we used to have these Docker images being released for this project, but we, we forgot to add um for about last two releases because we decoupled this project in our Studio Projects.

A

So now in studio, we don't rely on this project, so we forgot to add the release of this project and the uh we make it. So previously you have to use the binary packages for HTTP Gateway, but now it's come back. You can still get get Dr proof from the docker Hub. So if you are using it to check out it's easier um yeah- and this part is a so in the future- I will like every uh two or three months.

A

I will give this topic share, um maybe not necessarily by me, but we will have some more specific uh drum map uh being reviewed. So this is uh some uh planet uh features that we would like to address in next cycle.

A

I will give it quite uh briefly. So if you have any questions, just comment on YouTube or um you know- drop a discussion uh thread on GitHub or Slack. uh The first one is. Finally we uh we will support the filtering of the vertex properties for uh find tests and get sub graph um yeah it's requested by many already, and we will have some more uh finecast performance optimizations.

A

Third, one is: uh we will provide the session two for our officially supported clients and, as I know, there is an ongoing PR. I'll never go today, so by still work in progress. So please expect most of the clients comes with the session for officially in like two months.

A

um We will redesign our tagless or uh bare vertex behaviors um in next readings. So ideally um will remove this concept because it brings some burden to users and not really. There are certain scenarios that we actually needed and around the Studio Plus the navigraft Explorers still do it. You can consider Studio as the Explorer light.

A

We will have the visualization for the graph modeling, so you can drag and drop and create your schema and finally- and we will have some other optimizations like we will support a comment in in a console inside of the studio and the Explorer, and we will have some optimization in batch executions.

A

The final one is about never graph dashboard and we will have some more features. One of them is. We will support the magic search in the dashboard um yeah. Then we come to.

A

um We come to the first topic. uh This is about fraud detection with graph so I. Recently I did some research to um compose a blog post around how we should how we could do the fraud detection thing with the graph Tech, mostly together time with the graph database and I studied some of the setup, art matters and pro approaches and provide you some sort of summary and the playground that you can actually get a hands dirty.

A

So um there are more details in this URL, so I I translated the origin, the Chinese or blog into English today, um so I will start with the graph modeling. So what is fraud detection? So it's basically, uh so. Fraud exists in in different Industries, for example in the fintech.

A

um If a transaction comes to the system, you want to know if this, uh the owner of this, uh the account of this uh transaction behind is, uh is not a fraud, and in case you can have a high confidence to say yes, probably to prevent it from proceeding. uh Will you know, save a lot of money so in this kind of cases graph can help- and this already is proven to be quite performance.

A

So this is a sample graph. Modeling, but it's not related to the transaction but related to the loan you can, as you can tell so. Basically, we will set up this uh Knowledge Graph on the certain um entities that may be related.

A

For example, you want to record everything around the applicant and some information about application, um and if, if your system is mostly online, you will have some uh devices or meteor um related with those application uh actions, and do you have more information either is comes from the application itself or some some other sources you can grab like um the the corporation, the workplaces or some education in information or the phone number Etc. So if you put everything in this graph, you can do something interesting and I will share you later.

A

So, for example, you can you can simply um do uh queries. For example, this is a query that um how do you check from certain uh say when a transaction comes or when loan application comes, and you can query on certain people or certain applicants with a query like this, so it will help you find um put all the the other existing uh information, we've shared device or shared phone numbers.

A

So in this single query, you you you could tell uh one thing is the the orange node here is actually a phone number you can see with phone number, so the word is quite small anyway. So this is a typical uh scenario that, after your query, you could tell there is one um applicant or there is one phone number associated with associate associated with the a lot of different other uh people.

A

So in this case you will, uh you probably will see this uh phone number is uh risky and from that you can uh ideally further and analyze analysis uh other related people, and in case this phone is uh connected with your new uh applicants, so you can then Mark it as high risky or even reject it. So this is one of the typical uh use case that you can simply use a graph query to help you um evaluate the the risk level, so uh there are actually a bunch of different queries.

A

You can do that and some of them are based on your experience somehow or your risk control, expertise or certain patterns that you uh you summarize right. Another part, is you or you already have a bunch of labels like certain devices, certain people or certain cooperation or masks as a high risk flag?

A

So in that case you can simply query something like this like if it's somehow related to uh connected to another applicant or are they marked as fraud or even a phone number Etc? And that's another uh typical queries? You can identify frauds found silver. One of the um one of the issue is sometimes you don't have enough labels right or you don't have access to for your co-star, your projects and your compliance um requirements. uh Don't allow you to.

A

You know, connect to other data that comes with some labor so like, for example, you only have like 20 label data, but you have millions of uh vertexes. So how do you leverage, so this seems quite uh useful and the performance, but to in the real world, you may not have the um lock to have this label. So what can you do? You have a bunch of approaches to help you uh extend from a limited number of labels to have more ideally have more.

A

Labels, like you, can do some graph algorithms, um one of them I, consider quite useful, uh is called the label spread, so basically, uh actually I have a full um full solution or with the code implementation in in my blog post, I'm, not I'm, not going to dive into in this session. But if you are interested, you can check out. So this is similar to our you know a well-known the common label propagation, but is slightly different. uh The use the purpose of this uh label propagation here or we call it a label spread.

A

So it is firstly uh named in this term in the initial papers, so the purpose of this label spread thing was just to address the scenario I mentioned the before, like you have a limited number of nodes marked as from, and you want to fully leverage your existing uh information in your graph, maybe later in a similar module, as I mentioned before uh and in theory you will know this information can help. You lead some conclusions, like the other.

A

uh Some of some of the other nodes can be treated as a risky as well based on your limited ones at the starting point. So this algorithm is helping to help you to doing so. So, ideally, you can first doing this algorithms like every week and help you extend your um accumulated limited labeled nodes, and on top of that, you can apply some query based, rule-based pattern-based, broad detection. So this is another interesting method um with graph and after you, you computed those labels.

A

The best ways use the written to write them back to the ground. Up is because you can query them on demand in real time right, okay, so next I uh introduced some other uh methods.

A

um Our machine learning related.

A

um So you can roughly divide them into two, so one is the Legacy machine learning, Masters, plus the graph features. So this is uh this is a horse already quite mature matters uh brought a couple of years before that. Previously you don't consider you don't use anything around graph. You just have everything in your.

A

And you train, uh you do some feature engineering to pick like 50 of the data of the properties of your tables are related to the fraud and you train a machine learning module, um maybe just uh uh to uh separate those um to predict uh who are in high risk a fraud.

A

um So the the only difference here is a plus a plug on top of those uh features you just compute like, for example, graph, related uh structure.

A

um Things like you, compute every node page map so put the page rank value as another feature, and it was proven and that this certain method is better performance. It's more performant than the Legacy non-graph feature machine learning, algorithms, because, for example, background it reflects the importance of certain nodes and that importance the information is related for is related to the you know the the case that we want to detect fraud.

A

uh You can leverage not only uh the the Patriot, maybe some some other uh Community detection. Algorithms could also help. But in that case you don't want to put the you know the idea or label of the notes at your feature. Maybe you want to use some date wise uh features, but that will help you, because the the community graph information is also related to if it's fraud right. So this is the first approach, so the second one is uh the graph neural networks are related approaches.

A

So this this method, comparing to the first one, uh can be. uh You know helpful, because when it actually uh the NGM, we actually embedding the uh the the the the whole graph into um they didn't see Matrix. So, although it's embedding but it it remains a bunch of graph related information like the locality how this node are um directly or two Hoops connected to other nodes.

A

So in in this, in this representation and plus the the methods in the TNN, it magically consider the information uh hiding in the whole graph, comparing to the the Legacy graph feature machine learning methods.

A

So another thing is uh with the GN approach: um you don't have to do a bunch of fish engineering comparing to the the other one, but you do. You still need some sort of feature engineering, but it's a lot more easier uh based on the observation uh and the research.

A

Those two uh approaches are comes with some more details and even playground in action uh content in my blog post, so you can check out and actually for the GN uh uh case. I I provide you a end-to-end solution. So this is the whole thing of this approach. So basically, in the left side, I uh I make uh this is the architecture or flowchart um how we um we use a node classification uh approach to help us learn the learn function from the existing graph.

A

We have graphed that some of the nodes are marked as as cross and after we learned this, this uh GN module.

A

uh This module is a function that you can give a subgraph and it will help you predict uh all the nodes in the sub graph if those nodes are categorized as fraud or not um I in this, in this example, I'm using a graph algorithms, that you can train your module with the old graph and you can feed and consume this module with the whole new data, and uh so the right side is how we we are going to now.

A

Reach This module, um you know, and uh so imagine we have this module trained here and in in the in the online fraud detection system based on this module. uh When we have this uh transaction or uh a review from Yelp uh sending to the system, the first thing uh in the system, we will going to insert everything related to the graph to our uh never graph, uh Knowledge Graph. And the second thing is: we want to get a subgraph related sitting close to this new newly added nodes.

A

So we have a smaller subgraph like how 3000 nodes and a bunch of connections among them. Sorry, and then we will send this small subgraph to our inference. Api we're underlying. We are calling our trained module, JN module, and this module will help. You predict whether you're newly adding node are considered high high risk um or not.

A

So this is a I made a I even made a front hand for this project, so you can actually see what is happening so the the Apple side of is the dashboard or uh user interface of this system online system. You can see this is a table. So, every time you have some a bunch of uh queries that uh this one uh would like to give the uh underlying this product uh projects I'm leveraging a public, a real world uh data set called Yelp data set so Yelp so because it doesn't really have to yell.

A

So every transaction here is not actually a transaction. It's a review of a given a restaurant or hotel. So this newly added review Send us to here. First, we will query from navigraph to get its related subgraph, and then we will send this. This uh subgraph information to the inference, API and inference API will feed this um subgraph to the module and then we will send back the information.

A

And finally, we will update your friend hand with the socket IO, so you can watch out the upper side is the console I want to emulate email simulate, the the query- and you will see this this section- it will have the the log. Okay I will show you the demo yeah. So the query is sent- and this is, if you are familiar with the the machine learning thing you will notice.

A

This is the uh how you want to processing the uh the data in in the uh Pi torch or DGL, and then you will feedback the prediction and you can see now it's too fast. uh Now there is an actual line here regarding this newly added record, and this marked, as is fraud Force. So this is the you know, end-to-end demo and you you can have every every line of code in GitHub and everything in details are um describing in my blog post.

A

If you are interested and for anything you'd like to discuss further just ping me or uh you know, comment on the blog or or discuss it in in our community. So this is the full uh view of this uh article and I. Remember I um briefly, uh introduced or described here so be sure to check out if you are interested and yeah. This is the um uh the first topic and the second one is related djl work.

A

um Actually, in the last demo, I uh heavily used this djl thing. So djl is the the most popular deep learning library uh up. There is open source and you can uh so. This topic is about uh how we uh we can leverage egl to working with your network graph. So um I didn't prepare um uh slides for this, but this is the uh the key point of this topic. uh This is a project called neblo DGL and it's literally means the the adapter or connector between a naval graph and the djl.

A

So with that you can you can train your um your DN module on your graph, persistent in level graph and that's quite straightforward and easy. So, ideally, if you have or your data already in your navigraph something like this and you can have uh put- you can load your graph in double graph in just like three line of code. It's quite easy and you can run demo and playground uh here or you can refer to my uh uh my fraud detection uh projects too. So they are all based on level graph and djl So.

A

Speaking of um how you like to um you know your tgl module will do things on top of your one of your ground safely in another graph, but how you would like to map them. So this is the most important thing. So this is how I designed the uh the never loader API so I, designed it in into a yaml based configuration file.

A

So basically you want to um tell nebulograph on tgl that which tag or which Edge type of data you want to extract or load from never graph, and you know that in gnm every graph are number based, but you don't have to worry about your vertex ID. Even this string type, never ground video will help you to handle that dirty work and map it it's transparent. But when it comes to property, you know we need to make our graph features in number formats.

A

Either it's a float or ink, but you cannot take a string, but in real world a lot of uh a lot of properties are by Nature's green. So you need to do the transformation. So previously you you can do with without uh nebular uh without uh Naval djl. You can do this by your ETL or pipeline you. You have some big data infrastructure.

A

uh You uh extract the information from Naval graph and you make some transforming uh tasks and like mapping your string or other type properties into your number typed features, but with never graph ejl. You can just describe them directly uh in in my mapping API.

A

So I introduced uh three types of mapping of filtering one of the amazing functions, so you can put, for example, here: I want to inject all the vertex uh intact layer and I want to extract the feature based on age, so HR number already, but I want to make it normalized. So I can write this Lambda function to just to make every age to um to have this normalized to be smaller one than one.

A

So this is one type of uh transformation. Another type is just to give the pair of value. So, for example, here I want to extract the follow ads and I just keep the value itself so yeah. The the degree of the the degree is a number. Eight properties will be your uh one of the features in your uh follow type ads. So another type that in fact you can do every kind of um transformation with the function but I introduced.

A

Another type of uh may have been called the immigration, so, for example, this in in this case, I, want to extract load, a team, the vertex with the team tag and one of the features I want to use is called host. So that means I list all the West codes and the East Coast team emulations, and so those one will be mapped at zero and those ones be mapped X1. So uh that's fair right, that's straightforward! So these are uh now uh I designed this three types of the Transformations.

A

So this is the most of uh of the progress for now. So in the future, I will add more uh things. For example, we don't have the the capability to make your data save written back to navigraph. So I will add that feature in the future and there are some uh many more things that still need to be done. So this is a very uh an issue. uh Proof of concept States.

A

So if you are interested in using never ground with ddl, just be sure to let us know and or you you can help us to improve this project, and let me know if you want to make some contributions on this. So this is the um the everything I would like to share uh today and I. Think uh I come to the end of my session shared, so uh we don't have too much yeah and uh the yeah. So uh any questions Joe about the.

B

Topics yeah, but thank you very much for putting it up together. um So so.

A

B

Very good sharing, um even though it's only two of us uh for now, um due to time zone differences, uh the recording will be shared very soon right, yeah, all right.

A

Okay, thank you. So um let's see you in uh next four weeks, so uh ideally we will have uh some guys from other open source community in in next uh meet up from the uh Apache API six so uh be sure to join us. If you are interested in that project,.

B

Yeah um so yeah, please find us on meetup.com, GitHub or slack Channel, um so drop any drop any questions or ask uh on either those channels. So we will be very happy to get back to you.