Nebula Graph Community Meeting, 9 Mar 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Nebula Graph community meeting [2022-03-09] (ARM supported, GetProps Pushdown, How to build/debug)

Description

💡 Ad-hoc: the workflow of contributing to nebula graph core: Build/UT/DEBUG Flow get started.
00:00 - Intro of the community meeting
02:40 - 🚩 Project Heartbeat:
02:47 - Interesting ISSUE:
- https://github.com/vesoft-inc/nebula/issues/3941 thread pool perf
- https://github.com/vesoft-inc/nebula/issues/3940 multi-get rocksDB
Surrounding Merged:
- https://github.com/vesoft-inc/nebula-third-party/pull/37 arm build supported
- https://github.com/vesoft-inc/nebula-operator/pull/117 Operator: Helm Auth supported
Core Merged:
- https://github.com/vesoft-inc/nebula/pull/3904 use one global lock in meta
- https://github.com/vesoft-inc/nebula/pull/3896 permission for kill queries
- https://github.com/vesoft-inc/nebula/pull/3913 auto_remove_invalid_space default true
- https://github.com/vesoft-inc/nebula/pull/3839 https://github.com/vesoft-inc/nebula/issues/2533 GetProps supports Limit push down
09:14 - How to contribute to Nebula Graph Core

👉 Join the meeting from the link: https://us02web.zoom.us/meeting/register/tZ0rcuypqDMvGdLuIm4VprTlx96wrEf062SH

A

Hello, everyone so it's time for another community meeting, so wait for maybe 13 seconds. As I know, lisa will join us she's, one of our community engagement girl, okay, I will start first.

A

A

This is the agenda today.

A

um Maybe we can introduce the new coming girls later, which is lisa sport, um so we will go, go with the project heartbeat first, um before we start as we as euro. I will give a brief introduction of the community meeting. We will have it bi-weekly and every time before a meeting, everyone can propose a topic to be shared or to be discussed to be requested and feel free to reach out to to us from the slack channel or about the documentation which is which is a etherpad online.

A

Anyone can add it and after a meeting we will share the whole record in the video and text and it will be archived in youtube and our github wiki.

A

So during the meeting we will have the project heartbeats the ad hoc topics. Anyone can bring us, you can put your rfc your idea, your I made these stories here and we will have open uh discussion.

A

Okay, welcome lisa, so I will have you being introduced later after. I give this brief introduction and we will have open floors uh for any questions or discussions, and you can propose it before before.

A

um I will meet you first lisa before every meeting and we will try our best to arrange anyone that you would like to discuss with, and today I will give an ad hoc topic as well as zero, so uh the first one first topic is the heartbeats of the projects. As you wrote, and the first one are the issues. So we have two interesting uh issues and they are not the is still under discussion and the implementation is not yet started and they are all related to the optimization of the bluegrass core.

A

So one needs to uh use the multi multi-gat interface of the roxdb. Instead of the you know, the single gas and the other one is to refactor the thread pool of the storage site to ideally improve the uh performance. They have very good discussions there so be sure to check out there if you are interested and regarding um hi lisa.

A

I I saw your you turn your um maybe I can put it here yeah so lisa is in our new uh office of the wiisoft. I will introduce you later. um Then. Then we come to the the the latest merged.

A

Prs the first one is uh related to our meta uh daemon, so this vr is already merged, introduced a global log, and previously we are using segment log in different plus places and for for the reason that the the right traffic for mata isn't that heavy as it was before or as it was, uh we thought it was, and our designers decided to use a global login step and let's see how it goes uh in the coming months.

A

um This this vr is facing an issue that previously we don't have a precise uh permission, control on killing queries, and this one improves that- and this is the new flag uh default value.

A

It's related to uh like when you removed a space and underline the data will not be deleted physically, but instead it will be marked deleted. So this behavior is related to whether during a restart it will be deleted during the booting up or it was done uh waiting to a next compaction.

A

So by default now is it is in true now before it was uh false. So this is the the last one is in. The core part is related to the um get property supports on the limb. Push down so and now it is implemented, uh is done and and it it's a sub task of the whole uh push down works. uh Please check out this issue and you will have the whole picture of the progress.

A

So, on the other side, one big thing is we eventually we have the arm build capability ready again previously it was added, but it was removed during our third party uh changing and now it's added back back. I know everyone uh using an m1 macbook pro is eager for this feature. It's back and actually, from last week, our netly built docker image is supporting both x86 and the arm architecture, so you can enjoy it from your arm based macbook or resin raspberry pi.

A

Another thing another merge is related to our kubernetes operator, and now we fixed the configuration capability to support helmholtz and the last one is related. Our nebular community contribution org- and this is a new repo donated by uh our community member, that they they were doing. A a fantastic job based on the euler euler is the open source job project on the deep learning platform, so they replacing their in-house graph database storage layer of this project and use the enable graph so check out this repo.

A

If you are interested- and we have the related uh posts- blog post posted- maybe in in chinese for now only but in new future, there will be in english version, so check out that so um we have done our project heartbeats part. So maybe we can introduce new members, lisa hi lisa.

B

A

Nice to meet you.

B

Here I am very nice very happy to see you in this channel uh and I'm the new member in the number graph community.

B

Now I'm the operating specialist and responsible for our community health yeah and maybe talking with our users uh and in this meeting uh I expect to meet more and more members from our community. So we can have a very happy discussion about our products under our community like that things. Okay, that's all.

A

Fantastic, uh thank you lisa, so lisa is, is a quite uh kind-hearted girl and uh being everywhere in our community, like slack or um uh forum or github or twitter. You can find him here also in linkedin, so our instagram, so thank you lisa and yup. So we'll move forward um yep. So I will start the ad hoc topic today, so I will give some sharing on how to contribute navigate core and how to build it in in a smooth and easy way and how you can debug it and later.

A

This is also the outcome of this topic will be also shared in our uh documentation for developers so um and yeah. I will start so. I will start with the architecture of network of core, so I think you may be familiar with, or with it already.

A

There are three types of binaries or services of network graph, so they are graph surveys, meta service and storage service. So the graph service uh is our ngql traffic interface and is the only interface exposed to end user or ndql clients.

A

So if we are talking with nabla graph only with the query, so only the graph service should be connected to and underlying there are other two services which is the meta service. Then the storage service. So for meta service uh the metadata are including the authentication, authorization and the sharding information of the other stories layer, the service discovery of the gravity and the storage d, and how the graph spaces are isolated and distributed in.

A

Among the you know, the other storage instances, the schema information and it's also maintaining the long run, jobs, the lifecycle of the jobs and, like other information like the data, ttl et cetera, et cetera. So they are all maintained in the mentality and if you are implementing them in the multiple replica mode like three and it's actually in in active standbys and by mode, so only one of them, which is the leader in the rough consensus, is accessible.

A

um And yes, so they are uh synced in in in raft protocol and underlying like, for example, when you you are making a logging, the gravity will talk into me for us, so any user won't need to talk to matadi. In in case you are only using ngql, query and story. Z is the the one behind that to serving the storage layer and um it's a distributed storage, so the data will be sharded in a stat static, hash way. So it's not using you know, uh then it's not using a ring.

A

You know, um and it's also uh uh can consensus the with the raft protocol and so per per h shard. If you choose to have multiple replicas so and the line is also the uh the raft and underlying in in the local layer, it is actually leveraging the roxdb as a local storage engine and it's exposing some like gap, neighbors gap properties, interface to the uh graph d gravity, will call the neighbors of properties.

A

It's already a a graph, a graph term of the interface and the underlying is the kv storage in in ruxdb and then, as I mentioned, revealed something on the communication.

A

So they are communicated in different ways and by default all the service will expose a thrift uh ser as a serving as a stripped server, so they are by default lacing uh on different parts here.

A

um So, for example, if we are using using nebula console which is command line, ngql query client, we are it's underlying a a thrift client talking to the graph service implemented in gold and so gravity uh will actually call the meta d for different purposes. As I mentioned before,.

A

For example, if you are doing a login here, it will make the uh swift call with a meta client to the my uh so here in the in this collect. The the meta client is a thrift client to the swift server in the mac d and also for the service discovery. As I mentioned before, all the services, all the graph and storage services will be, will push report there uh not notified mahadee services that I'm a newcomer, so you can register me as a new service.

A

So it's it's also done uh through this uh thrift service, thrift interface and, uh for example, when we have a query like a match: n written n limit 100. uh We we have an ngo query to graph d, so graph d will uh first talk to the meta to get the related uh information like related spaces and how the data was distributed. And where can we get the data for storage? So what are those stories these instances? So it will talk to the metadata first, then it will talk to the storage d just like here.

A

It will also talk the stories so when, when calling, when the graphic is calling the stories, it is leveraging the story declined so and the line is a uh a street thrift client talking to the thrift server served by the story, service and also storage d will will do the service discovery thing similar to gravity talking to the metadata and as storage they will have the raft traffic between or among all, the storage the instances. So this is the brief view of the communications underlying.

A

uh So, for example, there are actual http uh interface per each type of service. You can do some admin or statistic purposes on that interface as well, but that's not the main communication traffic so for details you can check out more documentation the configuration and our code base and the architecture more architectural information can be reached out from this documentation chapter here.

A

Yeah for folder, actually uh we have a couple of uh uh blog uh documentation, uh blog posts in our uh uh blog here, so they are tagged uh in a very quite uh easy way. Like you have you have tag of uh architecture, you have type of uh source code and type of feature introduction. So this is the soft code explained uh theory series of the blog posts, so you have more uh information related to how it was designed and how the dots were connected.

A

Yeah so feel free to check those uh posts. If you are interested and then we will move on so next topic is on how we can develop. We can make some changes or debug the network graph, the core database.

A

So this is the uh the I consider the easiest way to make everything uh prepared, ready and clean way yeah. So I will uh so. We will start with the clone of the repository, and here we are assuming. We are working on the uh 3.0.0 uh tag and then here is uh we. We we need to create a developing container. So the good, the good side of this approach is, you will have every third party ever developing.

A

You tell ready, after boss, in this docker repos image, so in this docker image and you and the second one. On the other hand, is uh you don't?

A

You won't make your uh operating system dirty during preparing of everything here and another thing: is you can make isolated environments in parallel if you are doing on different versions, development or if you are working on other clubs plus projects, so it won't make your uh environment dirty so and before I I uh initially this uh dark dark container, I actually create a docker network, so this this will be handy for us to make this.

A

So it's a disability system, you can debug on one of the binary and you can still make it talking to existing uh cluster and that's running on docker and and our navillograph compose uh project is handy for this, and by default is, is creating uh isolated, uh docker network. So, with with this step, you can talk to all the other meta and the storage the um in in the easiest way, so so be sure to do this.

A

I will share this in our development documentation later and then you just every time you want to access this long run container. You can stop it anytime, you just stop it so the name here I use the nablu and underscore with your username, so you can rename it as you as you prefer, and then you can access it in this way.

A

So how we, after this, we can build it without dealing with any dependencies or utilities of the you know, the c plus plus and the dependencies and every time, the first time you you will need to build the source code without any changes. So afterwards you can. The build will be much more fast and much faster, because you will only rebuild the delta part that you changed.

A

So this is the uh this is the the long uh configurations of the cmake. So, for example, I I might make the uh testing off because we want to build it for the first time we don't change anything. We don't need to verify the unit testing our um for the existing uh code base. uh You can.

A

We can enable the debug, because we want to use the gdb later to see where how the code was going through inside code base, and we you can make uh in different uh numbers according to your cpu number and then you making store, you will have all the built binaries and the built uh configurations and other files in in in one place here. So you will have all this here. I will. We will leverage those files later when we are moving on debug part. So then we will start the debugging of network graph.

A

So in this example, I'm I'm showing you how we just debug the graph d part so, but it will be similar if you want to debug the meta or uh storage part so uh before so I'm assuming that we will set up a cluster with the docker compose, rather than start all the needed binaries yourself.

A

So here I check out the compose file from the github and I am be sure to use the same tag of your build code right if you are using v3, so you be sure to use v3 as well and make it off and after it's up so in v3, be sure to connect it with the console like this and then be sure to add all the storage service.

A

This is to activate all the storage instances. So this is the behavior change in v3 you can check before and after on the show hosts before it will be offline. Afterwards it will be shown online and then you can use uh in in this same uh console session. You can use the uh play baseball player.

A

It will help you download the basket player data set for you, so you can you can you can tweak tweak, uh ngq, query and debug it further and afterwards you can next exact, uh existed uh exit, 8., sorry and after we have a cluster, and now we can add our debug, our new standalone uh graph, the instance joining to this cluster and then we'll connect to this.

A

uh This debugged version of uh gravity and we will um make breakpoint on this uh session and this process and we can do or check the vari uh the variables and how it goes in in the code base. So first we um we accessed the uh the development container we created in the previous steps, and we can start this. um This binary with gdb, which is then for the noon debugger, which is handy for the c cnc pulse, plus debugging, and before we start running the uh the process.

A

We need to um set some flags because it will fork different process. So we we should use the fork fork mode child, so we we, we can be ensured to uh jump to the corresponding child, um and then we stack the uh the arguments. So this is actually related to configuration of this instance of the metagraph uh gravity.

A

So we actually, we are configuring, our local ip, with our staff container name so in in in docker. Your container name will also be treated as a domain name. So just like we are talking to the method d0 with this name. So this is the equivalent address of of our graph d in a debug mode. So assuming we, my user here, is named tom, so be sure to replace this part to your username accordingly.

A

So then we will have everything set set up on the uh gdb and that then we can set a breakpoint. So I use b- and this is the function name of where you want it's to stop and you will check from there. So here I just gave an example of this uh function. Name and after this you can hit run. It will set up the process for you and after a while, you can, after you be sure to uh render.

A

Are you end run the the service will be up, so you can now run your console again to connect to here to connect to your debug graph d instance. So here is the address I'm assuming we are. We named it tom. So then we will connect to the graph, the debug graph, and you can see if it's booting up properly the service itself will be registered to the meta.

A

So after you are using show ground host graph, you will see our debug instance here in the same version, so this should be in same version or you will have some unexpected behaviors and then you will make a query here so here I'm making a a query with the ids vertex id at the starting point, so it will be stopped.

A

Remember I put a breakpoint here. It's related to this vid extract visitor. It will be held in there and you can continue your debugging from here. So you will see something here like this in in your gdb command line session and you can leveraging all the powerful tools in the gdb. So here I'm just demonstrating the the most trivial uh ones. uh For example, if you hit f enter, you will see where you are so this is exactly this is the exact uh function.

A

Remember I use b space of this function name, so you will be stopped here in the this file in line of this line, and then you can list the surrounding lines with l. You can see. uh So this is the our uh function here, and this is the related lines and then you can use the print space with the the the variable of current scope. Here, I we can see this variable here, so you can print it.

A

So this is very handy right and you can use the b t which stands for the back trees, and here we can see all the batteries from this line. There are more of them. I I all made them here and you can choose with next, which is the end command to continue to next line. Also, if you are reaching like to a for loop or a while, if or calling another function line, so you can use s enter to enter that deep layer of that call.

A

Instead, if you are un, it will execute that line to the next line afterwards.

A

So this is a step step in command and you you can- or if you have everything you needed you can use continue to move on. It will not stop until another breakpoint, so this is basically uh all the trivial uh the most frequent used uh commands of the gdb. So, with all above information I share, so ideally you will help.

A

You will have a handy environment and the workflow on how you want to um debug the graph database and if, after you're debugging with this flow, you will know where you want to change a bit and you will, after you change everything you can rebuild the code just uh just like this, but in this time you can either enable the testing to just select the unit test to help.

A

um You know regression your change or you turn it off with this debug. You can continue this process continue of the gdp process to debug your changed graph version of grafti continually. So after that you will have a handy environment and workflow on how to contribute and debug nabla graph.

A

So that's all from this topic, so we don't have new commerce, so we will move on um yes and before we call this end, uh be sure that we are having this meeting bi-weekly and be sure to register our meeting in zoom and- and we have a google calendar, you will have it reminded in your gmail and put your uh topics on this easter pad.

A

So all these information are in our wiki page and uh two things to be uh highlighted before we end it. The first one is the nabla graph cloud on ager is open, beta, so be sure to check out on azure cloud.

A

You can search graph database, so you will see navigraph top in the marketplace here and the next thing is we just have we just celebrated our um 2022 international women's day, and this is a great tweet uh posted by lisa, so be sure to check out so this I really enjoy this, uh this art artwork so by our designers in community.

A

So it's at a lot list, which is the very initial coders in the world, so she's a very cougar happy women's day, and I think we can call this a day, lisa any question or any information to be added.

B

uh Actually, no okay! Thank you. This meeting is amazing. Yeah.

A

Thank you. Thank you so much for joining us, so I will call this again so see you two weeks later and be sure to reach out to you to us in in in flag github discussions or forum. Sorry.