Nebula Graph Community Meeting, 20 Apr 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Nebula Graph community meeting [2022-04-20] (Nebula Graph on Spark)

Description

Project heartbeat

- #4098 `RETURN extract(a.player.name, "\\w+")` by @wuxiaobai24
- Extract the substring that matches the regular expression from the string
- #4095 Refactor FIND PATH:
- reduced RAM usage of allPath & multiPairShortestPath
- reduced data copying by integrating conjunct path into the path operator
- #4131 Disk fault recovery improved with multiple data path awareness
- #4129 The single server now supports auto adding hosts
- #4146 Optimizer refactor:
- now only release symbols of top layer nodes in all matched nodes
- #4162 PATH optimizer refactored:
- redundant path removed, time complexity reduced

Spark on Nebula Graph, details also in https://siwei.io/en/spark-on-nebula-graph/

A

A

A

New members to join for now, okay, it depends.

A

So I guess I will do it all along again: okay,.

A

A

Let's make sure I'm.

A

Recording, okay, so hello, everyone um welcome to the community meeting again, um so the big difference uh in in this meeting is we listen that one of our contributors in community goran he suggested we shift to another time slot, uh probably more europe friendly. So we switch to this time slot so in case um anyone.

A

You would like to propose new time slot that you that otherwise is blocking you uh from joining the meeting. Just let us know in the stack channel, so we can change. uh You may probably to make more people with a higher profi ability to join um so agenda. As usual, um we will introduce the new member which uh we don't have to do in this. In this time uh we I will update you the project news and I will give you an uh ad hoc topic.

A

This time is again again for me and uh if someone is joining, we can have a sync discussion and the qa section.

A

So again uh we will put uh the information in this ether pad and it's also in our meeting um wiki in our github, and we will have this meeting bi-weekly.

A

So anyone can bring their proposal stories or topics here previously before animating and just let them let us know, and so, since last two weeks we have a couple of uh major pr's merged and I will give a brief introduction on some of them.

A

So these this time, most of them all of them, are for nabla graph, the core core part. So this one comes from the uh who sell by 24, which is our first time contributor, welcome to the to join the community and he helped us to introduce a new function named extract.

A

So this extract will extract literally extract the substring that match the uh regular expression from a given string of a variable and yeah. You can do it and afterwards it should be at least returned.

A

um So this is the first one and we have a bunch of optimizing and refactor related uh prs merged in last two weeks, so this one is related to fine path, and this refactor um helped reduce the memory usage in the all paths and multiple pair shortest paths, and also it reduced the data coping uh by integrating the conject conjection uh path into the pass-up operator.

A

So, if you're interested just to check out this pr, so it's not the only one that helping our sign pass performance so um and that this is the another pr that's making nebulograph it's more um thought: disc, fog, tolerance when it comes to data paths, multiple database awareness, so initially we assumed every instance have only one. um You know one block device mapped, but it turned out in some bare metal case. Some user will plug like 10 different disks in one big server.

A

But in that case previously we didn't handle that so well, and now it's optimized, it's it's better now. Another thing is that we we know that things uh 3.0.

A

We have changed the services discovery um mechanism and we actually, we added introduced actual step to you- have to explicitly add hosts to enable a storage instance after it's registered automated lay uh register to a meta to meta the uh the user should give ad hosts and address of the storage to explicitly make it activated.

A

Otherwise cannot it cannot be used? The data cannot be balanced to it and now.

A

uh This it will be silly if you are uh running a single server where we we put the storage and the gravity in one service, and uh we know it's really only one one server. So this the storage instance in that case should be added automatically, and we we bring this possibility with with the optional, with the configuration in this pr, and this one is another uh refactor case and it's related to all the optima optimizer.

A

So now, with this pr, uh you know the optimizer is actually uh changed, the specific plan from one to another to make it more optimal and previously on all the uh all the same symbols.

A

A

Okay, now only we only release the symbols of top layer nodes in all match nodes. When it comes to nodes, it means the operator the planned operator to be optimized in the optimizing row.

A

So if you are interested in in the details, please check out this pr, so it's actually quite underlying part of the of the uh graphic part. The file, the last one is another operator, but it's only related to the path. Fine path operator- and this reflects refactor- helps remove the redundant path. So it will reduce the time complexity.

A

It will improve uh a performance in uh in a good way. So uh the the author of this pr even put a performance before and after in in the description so be sure to check out it if you're interested. So please expect fine pass performance improvements in in upcoming release. When it comes to upcoming release, we will come to another major uh sorry, another medium release, which is 3.1, and ideally it will be released in in one or two weeks yeah.

A

So I think we we finished the um the project heartbeat part, then I would. I would like to uh give give the a hot topic today and this will it will be presented by me again and hopefully we will have more uh contributors to share their ideas and their sparkles on network graph and today's spot. It is about spark and the trigger that I would like to share this is I was.

A

uh I would like to doing some pre-study on bring the uh the pipe spark support, and but during the study it turned out, it can be done without code change. So it's just some tweaks when we are leveraging the pi spark with the nebula spark connector, and I will share that in this topic later. So everything is is already covered in my blog. So please check how this this.

A

uh This url is the art that this article covers everything that I will introduce in this topic, and this is the article and you can check the tlc here um and now I will present it in another slice.

A

Okay, so uh speaking of uh speaking of uh never graph on spark, I would like to share that what spark related open source projects are. We have on another graph community.

A

Actually, we now have three of them and they are the spark connector, the nebula exchange and the nebula algorithm. So.

A

What are they, and actually, uh I think most of you are already familiar with about so I will go through it quickly, so nabla graph spark connector literally it's just a client that you can.

A

You know, call the data, the data you can read and or write data uh towards the navel graph cluster within the uh the spark. So it's just a spark library and the other two actually are spark applications, so you can call them from spark with the summit or you can call them in this in the in your scala code, calling them as a library and underlying they are leveraging the the abilities of the nebulous bar connector actually.

A

But what are they? The what's? Never exchange and nabla exchange. It literally means a application. You can exchange data towards nabla graph, that that's saying that you can read data from mysql from postgresql from click click house from hive from hbase, etc. There are more than 10 of different data sources already supported out of box now, and you can contribute or request more of them. So, with leveraging nebula exchange, you can consuming data from those different data sources and write them into network graph, and that's only one of the paths please be noted.

A

There is another writing path to nav nabla exchange that you can also optionally generate the underlying roxdb sst files, and that will enable you to leveraging the spark neck spark cluster to help you sorting. The the data in uh you know cheaper way to be ingested into nabla graph and like when you are have take a daily massive data like uh hundreds of billions of edges or vertexes every day, and you want to offline batch import them.

A

So sst with the nebula exchange is your um is the go-to tools that you are seeking for because it will be much faster than you directly write into never graph cluster, but in the normal size, waxing normal for nabla graph is already a giant, a huge data volume.

A

So only when, uh when you're writing directive, network graph cannot fulfill your requirements, then you go for the sst writer. I will give a more detailed introduction later. So the the final one is the naval algorithm.

A

So literally it means you can you can leverage this tour to run the graph algorithm on top of the the level graph, so you can so in your spark connect your in your spark cluster with the this nablogra nabograph algorithm project, or you know, package you can run like like page rank, lpa, etc and algorithm, and this algorithm will seeking will scanning for the graph date time when it needed from the network graph and load it into the the memory and make the iteration on this algorithm execution inside of your spark cluster so by it, comes with a couple of different algorithm outer box and you can actually modify the algorithm either on how it's computed or how the you know how the data was outputted output and you can also introduce your new algorithm when the existing one cannot fulfill yours.

A

So you can upstream and uh open sources to our community and that's basically a short introduction of all the three different projects. I hope you make more sense on what they can do and what they are uh regarding of the spark connector.

A

So this is the gita github repo and this documentation that here I just gave the latest released version of 3.2.0.

A

So this is the core version and the course corresponding algorithm documentation is just here and you can actually, you can download this binary jar package in the center um repository of marvin in this url and for sure you can still uh comply for yourself and and clone this code in the copies right, and uh this- and here is the code- examples uh how you you could call uh spark connector in a scala code.

A

So it's actually a path in in the code base, and I would like to give you a brief demo that you can check more details in my blog post. So this is a simple example on how you would like to read vertex from another graph in spark.

A

So all you need to do is to put the information that the spark read dot read dot nebula would like to you would like to know like the first config is telling uh your where your, how the spark can reach out to your nebula cluster so like here, you are providing the meta d instances here and the second configuration here is actually telling the the read uh the reader of the spark reader uh which space and which label here label refers to number graph, vertex tag and the ver the properties of this label.

A

You want to scan, and if you want to sample with a limit you can provide- or if you omit it, it will scan the whole tag or the whole label of this graph. In afterwards.

A

If you call this function, it will read all the player with the property of name and age of this naval graph space net basketball player. So this is the example of the read vertex and there there. There is another example in the github examples on right, vertex or right edges, so it will be similar, but the the one big difference is that level graph. Spa spark, it is only used for the analytical or computation scenario. So when we are reading data, we don't do it go through the gravity.

A

Instead, we directly talk to the storage and that's why we we don't use, we don't compose any queries and we only scan for the whole graph table or our label.

A

But when it comes to right, there is no such uh interface for spark to write directly directly to the storage c. Instead, it will, it will uh generate uh in in certain batch sets of the ngql in insert queries, and in this time it will call to the graph d. So so you will need to provide weight graph address similar, something like that in the configuration. And then you called the right interfaces. I'm not going to go through that, but it should be similar, and I hope you have a you know.

A

You have a picture on how the spark should be look uh should uh looks like and how it should be used. So it's actually a library. So you call it from your code just like this, and uh then we can hands on it. I'm not I'm not going to uh demo. It on the fly here, but to you. Ideally, you can run smoothly without any actual uh environment preparation, just following steps in my um following specs in my own uh post, so I I I cherry p.

A

I picked some of the lines, the major lines, so uh I assume that you have a linux server with the internet connection so and it's better that you already have the document.compost print stored and even you didn't, you can do with this one liner, this curl one liner um powered by enable off it's a script. I um I created to help you try never graph in one command and afterwards you, you should load uh the graph data uh with the nabla graph console and this is covered by the introduced by nabla.

A

So you can do this in one line to add get into a container with a naval graph console installed and in with this line you will access to the network graph with the console and you you should firstly to do this. Every time you create a new 3.0 uh cluster, you have to do add host so just copy and paste this line. In other words, you you give this line and it will help you load a test data size named basketball, player and ins.

A

The a graph space which is the namespace um describing a whole graph with this exactly same name will be created. It will it will. You know it will need around one minute, just to be patient and afterwards, after this, this command is executed and released. You you can query the data, just follow the documentation, um and then you can you exit of this console and run this command to setup help. You set up a spark playground with the help of the big data europe. They they are doing really great things and they open source it.

A

So you can do this one liner to have a whole uh usable spark with the hadoop. It's very. uh It's really a sweet project. In other words, you you can access this spark with this line. You will access the bash shell into the spark container and then you can download and install the mvn, which is the marvin uh you you utility with this lines.

A

So from this type I I will show you how to build um the uh nebulous bar connector, so we clone this code outside of of the container and then we get into the container. Remember we mapped in in this step. We actually mapped current pass pwd to the root home inside the container of the spark.

A

So and then we can uh cd to this repository. We clone it outside of container, because we don't have the gate installed in this spark container. We have to clone it outside, and then we just call this mvvn in store. It will help us, uh please be sure, to skip this test, so it will be faster it will. It will be help us to build this spark connector and then, with inside this repo, there is actually a spark connector reader example code. It comes with all the needed dirty things.

A

You don't have to create your pom xml file, you just reuse what the repository already provide you up inside of the example path, and then you can change the code of this this this file, because the example provided in this repo is not based on the data site. I just created by default, so I'm I make it uh smaller and easier for you to understand and try it. So you just uh change this and and.

A

And actually pop up.

A

In in this article, um I put more information, so this is the the code of of this reader part, so you can see inside it. The main things is, this function called the um you know, read vertex, so it will read the vertex and edges for you and, and with this uh sorry, oh and with this uh file, be being changed as the.

A

So, with this file being uh modified with vi, you can then build this example code. So you sit into the example folder and run the mvn again and after a couple of seconds you will have uh example: hyphen 3.0 happens snapshot.jar.

A

So this is the example code you already built with the read read code inside it will underline calling the nebulous bar connector and you just submit, submit it to the this spark environment running inside this container, and you you can see the class name is the uh base of the example. Connector nebula spark reader example and that's exactly the class name of this file.

A

So afterwards you will see uh it won't. Take you too long, because it's quite small graph and you will see the graph data will be uh read scanned and printed in the output. So you actually run your own code, calling the enabler, spark connector and congratulations.

A

um Another thing I want to share here that normally spark speaks scala, but spark comes with the python support and r support. So I spent some time to make it make the demo end-to-end demo working on the pi spike, which is the python spike. So underlying python spike is just making uh internally it's making a pi 4 j, making a a a a java layer, a compatible talking communicating with the python layer, uh so in so in that enables us to, uh you know, actually call the nablograph spark connect.

A

You know, spark connector is a scala and it will be, uh you know, build a compliant compiled in into a jar which is the jvm binary by byte code and with the health of the pi spark, what everything already provided by pi spark. You can do that in your python so and you can write it. Everything is, or the more detailed version is in my in my blog and you can just enter this spark, and you just uh I in in this case I didn't I didn't build, I just.

A

Instead, I download the spark connector jar package and I call I I get into the the interact development environment in which is provided already provided by the spark in this container, with the parameter of the driver, glass pass and the jars. Where we put this jar file here and inside here. We just you know we just use the spark.

A

Read function and we put everything I needed in the format of our connector class and we put other things equipment equivalent to the example we in the nebular config and the gla and the reader config and all the related things are into turbo of the strings in the option. So it's a chain of option call, so I put everything here and you load it and boom. You can have this data frame show.

A

So this proves we can actually run navigate spark connector in pi spark. So this is the first time uh as I know. First one first time someone make this work. I I spent some a couple of hours to do this.

A

Okay and that's our part from the connect smart connector, which is the you know the base of all the other spark applications.

A

Then we come to the nebular exchange and remember: nabla exchange is a is a spark open source spark application or library to enable data being exchanged without her data sources towards nabla graph. So this is the github repository documentation. You can also download a jar package, and this is the configuration examples uh when it comes to configuration.

A

It's because this package can be called directly with spark summit, but how can we tell uh let this uh exchange know where we should call the data source which data source and how is this data was extracted or mapped to nabla graph, the schema, so those all of the information are being written in in this single uh application.com file. So it's hoc on format, it's similar to json, quite easier, both for the machine and and the per and the human being.

A

So we will come to the uh configuration examples later, but before that I won't give us a a brief introduction of the exchange again. So exchange comes with three part, which is the reader, the preciser and the writer. So we already know that the reader and writer is underlying leveraging the upstream the option of exchange here. One of them is the spark connector. We just reviewed it for you, and the processor here is the most important one.

A

So the reader will read different sources like myc core another nebulograph, a neo4j, hbase or postgres or pauser kafka, and when read the data, the processor will make it into different.

A

uh Serializable data depends on how you want to write, so there are two ways of writing data. One is the server-based search writer that the server here refers to network graph server or navigator. So that means you will write directly to this cluster.

A

So that means the preciser will make all this information of the edges of the vertexes into the ngql insert query or update query. There are options to support, update or offset, and so this is the the normal path. Then other path is when you want to sync or sink the sync or write data into sst files. So in that case, sst processor comes into work.

A

The ssd will it will read the information from the level graph on the fly and then sorted your data and make it into the und under nebulograph and the line ssd files and put it in your um hdfs, centralized storage, endpoint. You know it's, it's been para processed in parallel and will be right into a one centralized place and afterwards you just trans.

A

Afterwards, you can um download those ssd file from the nablograph storage key. There is a command named download and that will require you have the https client in your storage d and then it will download a needed ssd file there, and you will give a command named ingest. It will uh site load your ssd file, generated by the exchange and write it to nabla graph, so that, basically, that that's the whole process of the uh exchange how it works.

A

So everything you want you, let you want to let exchange know will be reflected to the configuration files. So this is a hands-on example on how we will uh exchange data from one of the source. Here is the same place, simplest, which is the csv file, and it's very smaller. Only with two lines and three columns.

A

One column is the vertex id and the second column is, is the name property called the name and the third column is the age and we want to exchange data or information from this source into our network graph. So we will make a configuration here, something like this. There will be more lines, but the major ones one is from the nebular part. You you tell the exchange where to write which cluster the user, the password graph address and the metadata address, and another thing is how you want to map your data set data source.

A

Here our resource is csv and, of course we will sync or write data with the client. The client means never graph plan graph decline, and this is the file the path of the file you can use the file schema or you can choose other storage like hdfs schema, and you should let the exchange know how the different columns will be reflected to the nebular schema. So here in this example, we have the what c1 here mean.

A

So the column uh index started from zero, so c one will refers to the full bar or another name, uh column. C. One refers to name and c two here: twenty four and 21 attend this 321 refer to age and the c0 here player 800 will be reflected to the vertex, so this means the vertex id, so it will be similar when it comes to edges so just different way. You just put in corresponding needed information here.

A

So uh after we we have. This configuration, we can do it our own, so everything I put in the post is runnable. You can do it, you can reproduce in your in your own linux, environment with the internet and that's all you need. So we you enter the the spark uh container and you cd to the the root path and you download the exchange jar package, but instead you can. You can also build this on your own and then you just create a configuration file.

A

Something like this, but the full version of this configuration file is already in my blog post just to refer to it and then we just you, can see. We have the configuration file. We have this the data source you know in in the configuration we put this in the files, and that means it will be in the local folder when you're calling the spark, but it normally. It should be something around on top of the hdfs, but I'm just you know just demo it.

A

So we have. You have all three of the files here, so we call it with spark summate, and this is the classroom name local, and this is the class name of the exchange, and this is the jar package, and this is the configuration that you will tell this jarvis. You want to find everything on the contest and the information from this configuration file and boom. It will help you exchange data from csv source to nabla graph and that's all from the navigraph uh from from the exchange part.

A

Then we come to the final one which, which is the algorithm. So in this case I'm not going to walk you through how to do it, uh how how it's running on the fly um and as euro. uh This is the code based documentation and and jar package, and actually you can call the number algorithm either from uh sub uh spark summit or from the library.

A

So you in different case you you, you, um you follow the configuration and the the the and the code example.

A

um So when, when we are calling the algorithm with submit it's easier, you don't have to write code and build code, but uh you can only leverage what algorithm provided you out of box. That is only um you, know the provided algorithm and a predefined um output format.

A

So you cannot change actual things to be output or different format of the output of the algorithm, but actually, um and that that's how it is, and but all the configuration configurable things you can put are in one configuration file similar to when we are doing it for exchange, and uh I I actually put uh another post regarding how we want to set up another graph in the algorithm in in container and how we want to import a live journal data set into nebula and how we want to run the page rank on top of that.

A

So this is a just if you're interested in how you want to render your first time, you can follow my blog posts of this this url and but when it comes to library, if you want to calling the nabla graph algorithm from your scala code, and that brings you more flexibility to you. Can customize how this algorithm are running? You can change the steps.

A

Maybe some of the details cannot fulfill your requirements. You want to modify it and yours are even in other case, you want to modify the output of the specific algorithm. You can do that you have more control and- and the example is already provided in in the level graph algorithm uh code base uh that I'm not going to show it here, and I think that's more. uh That's all uh for my part for um for this topic.

A

Okay, so we're back to here.

A

Okay and then uh sorry, I I used too much time this time. So then we just finished the ad hot topic.

A

Finally, please be sure that please be known that we will have to meeting two weeks later and be sure to subscribe and register this meeting in google calendar and do so everything is in our wiki page and uh of the github of the github repo of nebula and also one another thing I want to remind you: is um we just launched naval graph cloud on the microsoft azure in the marketplace, so you can search graph database in the portal of the ager.

A

You will find this never graph cloud, so it's still in open beta is quite cheap, so welcome to try it out and that's all from today. So thank you see you next two weeks, bye.

A