Nebula Graph Community Meeting, 18 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Nebula Graph community meeting [2022-05-18] How I generated Graph Data with community structure

Description

Nebula Graph Project update:
- Core
- subgraph last step pruned when outputise vertices only #4226
- MATCH perf optimization #4122
- Seek by ID no filter #4249
- fix listener can't receive snapshot anymore #4245
- Nebula-Go(All by Tiago Peczenyj)
- remove remote services, the code is not necessary to use this project as a nebula driver #200
- Create integration tests, now we could run pure UT w/o heavy container-cluster up #201
- PR REVIEW WIP: Code of conduct; IPv6 support; Multiple GO version tests support; Connection string support and session pool 🚀
- Nebula-node
- ping function bug fixed by jjsimps https://github.com/nebula-contrib/nebula-node/pull/12

Ad-hoc topic:

Graph Data Generation: Experience on create dummy datasets with graph structure

A

A

A

A

A

Yep, just a second.

A

A

A

Let's get started: okay,.

A

Hello- everyone, um hello graph community, so another community meeting, so we don't have new joining so um okay, I will get started um so today I will update you some um new uh news of the nabograph projects and then I will give a quick uh sharing uh on a small topic that uh what I had summarized and experienced on the graph database, uh the graph data set generation.

A

So we don't have new ones to be introduced and as euro, so anyone want to join the community meeting just be sure to check out subscribe to register in our zoom meeting. So you can, you can check out this link. This is the ether pad that you, everyone can put your proposal, your questions there and be sure to join our slack channel.

A

We have a in every graph slack site. There is a community meeting channel and you can get everything notified there.

A

So we have we have this meeting by bi-weekly and we will regularly uh have the projects updates and at least one of the ad hoc topics, and if anyone want to have open floors for this discussion on any problems, just come and join, and if you want to involve someone you or someone in specific domains, just let us know before the meeting and yep so today um the project of updates uh they're uh not too much in this cycle, because we just have a one release of 3.1.0 uh weeks before and our next release will be around july.

A

um So uh from the nablograph uh core database like uh we, uh we have, we only have like less than 10 uh prs merged, so I will go through the major ones.

A

uh So this one is optimization in in the sub graph that previously, when we query sub graph, even the last node, we will have this all its neighbors, which is actually next step to this node, but that that's actually not necessary. If you don't want to yield those, and so we we have this optimization uh in the plans now. So please expect performance uh improvements and another one is the related.

A

uh So this is the issue uh number, so you have a bunch of different match performance optimizations, uh so one of them just merged days before is, is related to when, when you are doing match with the starting node with vid.

A

In that case, you don't need to use the index right.

A

In that case, uh all the other filters in the where clause is actually not needed, because you already pointed to a starting point specifically, so this optimization was applied with this mindset and- and there are actually uh some ongoing pr's on this topic, so they are quite interesting. So if you're interested just check out this issue number uh the last one that worthy to be mentioned, is uh we fixed one of the rare case related to rough snapshots.

A

Of the listener, so it's a quite interesting pr. So if you can check out the numbers, um the the other big news news is that I want to share, is uh never go so last last last time of meeting tiago joined our meeting, who is a so very great man so and in last two weeks he contributed on a a bunch of different pr's on never go so two lambda two of them were already merged, uh and this one is the uh they.

A

Yes, the first one is uh in 200 200 pr number uh he removed some. uh You know some some code, that's not actually needed and and that part of code is blocking us from leveraging newer version of go and that's uh thousands of lines of removal risk which is quite satisfying.

A

Another big thing is it's already immersed, and this all of those pr's are contributed by tiago that previously we put every test uh in one go. There are no uh guarantee. uh There is no distinguish between the uh integration test and the unit test so actually most of the developer. The contributor only would like to run the unit test to justify their modification. Don't don't break anything, uh but you know by default.

A

Translate is quite heavy to uh both strip uh a whole clustering container with a uh docker compose and now uh tiago bring a broadcaster changes that you can purely run the unit test without the heavy integration tests.

A

So there are a bunch of more uh one of them that I would like to quickly uh review and emerge is thiago helped assemble the very first version of the code of conduct for the nabla graph gold repository that I'm thinking of bringing to all the projects as well.

A

Here he was also working on the ipv6 sport on the netflix gold client and he brought multiple go versions in tests. So this is awesome and really uh great work for the nebula go another one is um this is a huge change and it is. It is a huge pain point for all of our clients. For now we will really lack of proper wrappers and the elegant design in how the server address could be configured.

A

So the term of the leveraging proper designed uh connection string will be perfect. So I'm looking forward to for this this pr to be merged, and I would like to um promote this, this design to the uh all the other uh clients as well. It's quite cool. Thank you tiago! So much so. uh The next one is on nebula node. There's a node.js client, it's not in the v soft uh org, it's under enabler contribute.org.

A

So this pr just was just pushed last night by um uh djsi mps, so he or she helped solving uh a ping about in the ping uh function, which is quite cool. I think maybe it's the first contribution that he or she contributed to nabla graph community. So thank you so much so um that's the uh project heartbeat part, and then we will go to the um um at hot topic. So I will give this topic as europe.

A

I will invite more more guys uh to share topics in the future, um so this is about. uh Actually I have the experience to uh generate some data set. Some dummy data sets on my own to you know, test something or uh create something based on that, so they are all uh graph structured uh data sets. So I'm thinking, maybe I can share some of the methodology and experiences that I um like some problems. I have I faced and and mitigated so here here is a topic.

A

I hope it helps you, so I want to um start with the problems that I encountered. Some of them was a little bit challenging and not that straightforward to be solved. The very first one is a very travel one, but that we want to um generate graph a data set.

A

You know everything is random, it is dummy, but still uh you want it somehow make sense. So we we're generating generating data um mapping to our property uh graph module. So the properties like the names company names, the the role and someone worked at some other uh cooperations in in specific roles, their age, their addresses their names, their gender. So those all things will be too lame or too boring. If you just give random words as the specific vote that will be working, but it's not elegant.

A

So uh that's a problem and luckily there are always a faker uh library in almost every languages, so um I was using uh one of them. The next one is uh the relation uh for address relationship generation.

A

So uh it's easy to be understand that you can generate nodes or uh vertex uh with the like faker, but how do you generate randomly uh relationships between different nodes, so one of the the way that I would do doing that was making the the vertex id in a number or case, and it's making it continuous.

A

So in that way you will know uh the range of a valid ver uh nodes or vertexes. So then the a randomly generated relationship will be referred to randomly generate two two numbers in specific ranges. So that will be easy and that's the way I do it. I think everyone who encounter this issue will come with this uh solution is quite straightforward, but it works well.

A

So a next problem or challenges is.

A

Sometimes you want data pressure of data to be true, you know connected to the real world, and that- and I encountered that issue as well. uh I was uh I wanted to uh generate dummy data set up reflecting to a knowledge graph. That's defining um you know the the the people in the society living in different addresses that they were reflect uh the virals, the chrono virus, the kobe 19.

A

How is spread in this in the in the you know, society uh knowledge graph, that every node is a a person, um so I have some experiences that I want to share later.

A

So the most challenge now for now that I managed to figure out how to fix is uh some of the purp for some purposes that we want to leverage some generated random data set. You want to have some sort of graph wise structure. For example, you want your generated data to have a community structure and even some community structure that you want to configurable to be like you want. um You know, 1 billion of nodes, that you want to have 300 communities that they are clustered in some way.

A

So that's not able to be done randomly in a naive way, so I managed to find one of the papers. That's working well on this purpose, but it's becoming uh tricky because most of the algorithms was doing the work to generate that assuming every nodes and every uh relationship I.e, one single type. But how about a hydro genius graph that you have different type of nodes and different type of relationships?

A

How could that be be that I will share my approach that works. The final uh problem or challenge is: um how do you label data like you, want to do um fraud, detection, the in your graph data is randomly generated. It's not real-world data, but some of the nodes should be flagged as high risk, some other or not. You want to do some graph, query or graph learning to find the relationship between the you know the the ones with the flag and the one without. But how do you make this flag making sense?

A

So it's quite tricky to me. So then I want to quickly share the solutions. So, as I mentioned, I use the pi, uh the python faker uh module to generate the properties.

A

As I mentioned already, and also how do I generate a relationship, you just make the vid, the vid, sorry, maybe id and the containers number, um so you can do the range randomly generation of the range to have the uh the edges, so you can. Actually. This is the very first data set that I try to generate and it works well.

A

uh That's also also the data side uh running on the nabla graph playground really reflecting the relationship between the people and the crop cooperation and the some between them. How, when people hear a whole share of other corporations in different jumps of relationships- and I will introduce that later so- uh how do we connect fake data to uh real world data? And that could be done? You just grasp those from the public domain.

A

You can make some pre-process processing uh works and that that solved the work in this repository I create a data set generator to um create randomly people and with randomly a flag that they leave somewhere. Those uh specific address like the building name or the room number they're, totally fake. They don't exist in the real world, but what this place belongs to a street and the street belongs to a district. District belongs to a city. City belongs to a province.

A

Province belong to the country, so those are all real data and I make it uh I make everything connected and it works like china electron.

A

So next one is: uh how do you contain the you know, maintain a data in in a define or expected control controlled graph structure. So in this case I was, I want to um create a data set for fraud detection with a specific community structure, so I couldn't find a straightforward way. So I tried to search some papers and I found this so this is a paper related to fast random graph module generation with the community structure and the the author even shared their workable.

A

You know runnable module code in julia in github, so their module named abcd refers to artificial benchmark for community detection. So, and- and that's where I leveraged this uh package to generate.

A

Community data, but the next challenge will be those modules, only generate one type of points or nodes and one type of relationship. But how do we apply to our heterogeneous graph? I will share more later.

A

I I make some transformation to make it consistent, make the structure not change and apply to a hydro genius graph, and so the final challenge that I encounter is the labels in data like how do you make your uh part of the nose with the flag? In truth, the other involves how to decide you just randomly do that in most of cases that works, I just randomly uh created them and make the you know the proper possibility, um the chances, how it's setting true uh configurable, and that can be easily done.

A

You can check out my codes regarding those parts. So that's what I do now, but in the future, uh if I have the requirements I have the time and to make it even more uh useful and meaningful.

A

I was thinking that maybe I could connect reuse, some some of the real world data, for example.

A

If you want to create some some knowledge uh graph based on randomly generated data, most of like the structured, the properties, but you want to keep the some sub graph of the community or connections uh more like the uh the real world, you can leverage some other open data like the live journal or twitter data. That's actually in in real world stands for the social network, but maybe in your data set is it's for another purpose, but it doesn't matter.

A

Natural extracted structure can still benefits, and even uh and this um this theory can apply to the flag domain like how do those people how how many of them are considered as fraud, some other or not, you can leverage most of your data are randomly generated, but the flag, the flag, when the label one reuse, the real world data that I think that's a doable, feasible approach, but I didn't try it.

A

I will share my further experience when I got banned ways to do with approach so another, and you can do it either together or a totally different alternative. Is uh you can do it with your domain? Knowledge study to be consistent with your generated uh uh community structure or other graph structure, for example in in the fraud detection.

A

You you you generate with like uh a bunch of like 100 communities and you you know which node belongs to which community. So when you are flagging them right, even you are doing it either randomly or connecting to the real world.

A

You can still leverage uh you know like if this community one of this community are flagged as uh um as a high risk, and you can make the other nodes in the same community in a higher probability and higher chance to be labeled also as a higher risk, so that will make the whole data even more um meaningful.

A

Hopefully, so I didn't try that, but that's in my mind, maybe I will do it in the future.

A

So before I finish this topic, I would like to give you a more detailed introduction of the three projects that I created to generate random graph data and that and all those uh experiences comes from those ones so where the first one is the very basic one that I basically just applied. The approach of generate meaningful properties and randomly created the the edges, so the code is, can be found here.

A

So I just generate the nodes uh like in in a specific range, also generate a corporation just range, so their names address and those rows like the property in the edges. They are all generated from faker, so they're meaningful, but totally random and those uh relationships you just randomly choose two number and make them sitting in the range of the different count like you have 100 person, so you you have.

A

You cannot make the starting point of this lag larger than 100 right, so you can do that in a random lay fashion and it works well for me.

A

So um the next one is uh how I connect the the real world data uh to the generated data. So this is the module that I I randomly generate. A person and address and the streets really belong to the district they belong to. Town belong to, uh city city belongs to the province. Those data are real-world data and I I grasped it from the public domain and put it in this project, and yet the belong to lag here.

A

That belong to here is also from real world, but the people live with person and pers person with it they address. Those are randomly generally lags, so it's with the same method as the last one so be sure to check out this code. If you are interested and the last one uh is how I applied the sorry, the community structure, data to the hydrogenesis graph.

A

So this is the module in the left part. We have the people, they will apply as the law applicant, so they will apply for a long application and we have some other information like the people in this. In this application uh table, they will fill the information like they work for some uh corporation uh when they are doing this uh online, they are they they used to use specific device.

A

So you have a unique device id of the like phone or computer and people they will have their phone number and the corporation help will have phone number and people will have their emergency contact with another person and those persons are having another phone number, so they could be. You know connected in some way, but how do we generate search graph to do a fraud detection test and the data should have expected community structure like you want to have ten thousands of a person, and you want to have like 200 communities?

A

How do you do that? So, as I mentioned, I I checked for the abcd. You know the paper and make it working. So what um the paper will help us generate a community. A lot of points about vertexes, with one kind of connections between those one single typed vertex and those connections will generate like 200 communities. For you, it's not directly configurable. You just configure some other angles of the parameter because it will end up with the like two handles.

A

You can tune the the number to have how many the scale of the community structure you would like to, but that's details. You can refer to the documentation of this project, but you know you have this this kind of relationship, but it's for single typed data. How do you make it to community structure with the different kind of multi-hook relationships?

A

And that's? This is what I do so I data generation in the code part will be reflect to four steps. So the very first step is you just randomly generate the person as we do in other projects. So in the second part we are leveraging the abcd the the paper to help us generate person and percent their relationships so and that contributes to the structure with the specific community, the clustered uh structure.

A

So how do we make those structure into hypogenius uh hydrogen genius graph?

A

I just defined a specific way of the multiple connections and make it and break down all the direct, simple connections into those different kind of you know. Multiple connections, like you have a hundred of this, the blue connections from person to person, and then you randomly choose like you can define the ratio like uh you want 100. uh You want 10 of the connection to be in the pattern of person device person. So it's a shared device connection.

A

You, then you will have 10 randomly connect, 10 options actually in this way. Similarly, maybe you have 20 of the connections in this one to have. They are sharing phone numbers, so you can do this for all the different patterns you define and the key point here is.

A

You have now generated a hypogenius graph with the structure exactly same as this one. If you in you, you count those connections as the connections and they are weighted in the same way. So if you are running a algorithm to compute how many uh to detect the community communities, it will result in exactly exactly same structure with the sec, the one we generate in the second step.

A

So this is how I did it and it worked well. So I think that's worth just the most worthy to be shared part in this step and then finally,.

A

You have the persons between the relationship between the persons and that keep your community structure and then finally, I just randomly generate the applicant apply for loan application relationship in the final step and and all the connections here, never break the community structure generated in step two and you can check for the visual check by code, because I actually, uh uh I have wrote a quite beautifully uh commandland uh interface here, so I want to share it to you. So we're almost finished.

A

Okay, the network is not perfect, so be sure to check oh yeah. This is the how it's running in your command line. uh It is quite satisfying to me to have this fancy. You know the command. I love it.

A

Okay and that's that's the final part of the the uh one so the future work last, but least uh what I could do in the future. I would do more uh studies on how I should make the label data making more sense. So yeah. As I mentioned, I want to leverage the personal uh real world data. Maybe it's on different domain bus. Do we can make it more meaningful and I want to combine the domain knowledge and and the graph structure and make it uh consistent?

A

uh If I'm mad, I put some effort to make it more self-consistent. um I may get some more meaningful uh results. I guess so. Another part is the performance. So if you want to generate like 10 of billions of nodes and my current implementation will be the bottleneck because it's running in python and it's a single it's not it's a single thread. So maybe in the first step I may make it more concurrently or parallelized computing way of working.

A

Maybe I don't use python or use the multi-processing to make the async things or some some of the steps can be do in parallel. So another thing I will looking forward, I will watching some new papers. Maybe some new algorithm will be faster, yeah, that's uh the other part. I would like to improve yeah and that's all part of this uh topic.

A

Okay, and uh so see you in two weeks be sure to uh check out our slack channel to know how uh and in our uh in our github wiki. You will find the zoom link to so you can register the meeting here uh and we are having our uh managed service, our uh nabla graph as a service cloud in asia. So you can search graph database in azure portal and you will see here. This is the nav graph cloud and be sure to check that out and ask let's call this a day. Thank you. Bye.

A