Nebula Graph Community Meeting, 9 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Nebula Graph community meeting [2022-02-09] (New Feat.:UUID & Standalone mode,Algorithm on k8s)

Description

00:00- Intro
03:40- # Project Heartbeat

03:49- `2.6.2` Core released 🚩(https://github.com/vesoft-inc/nebula/releases/tag/v2.6.2)
- fixed memeory issue/CGROUPv2 issue/full-text index issue

04:28- Master Core
- Snowflake ID supported in `UUID()` (https://github.com/vesoft-inc/nebula/pull/3500)
- Single Server(standalone) Support (https://github.com/vesoft-inc/nebula/pull/3310)
- fixed a backup issue if after v2 to v3 upgrade (https://github.com/vesoft-inc/nebula/pull/3855)

08:35- ⎈ K8s Operator (https://github.com/vesoft-inc/nebula-operator/pull/109)
- `unsatisfiableAction`, configuration for **POD Spread Constraint**

10:05- `3.1.0` project created: 📅 https://github.com/orgs/vesoft-inc/projects/28/views/6

10:30- nebula-agent v0.1.0 released(https://github.com/vesoft-inc/nebula-agent/releases/tag/v0.1.0)

12:20- # Ad-hoc topic: Nebula Spark-Connector/Nebula-Algorithm in K8s

33:21- The community meeting calendar

- Arch. of Nebula Graph
- How Nebula Services are landed in K8s
- Service Discovery in Nebula
- Nebula-Algorithm config explained
- How-to & Hands-on 🏂🏻 (https://github.com/wey-gu/nebula-operator-kind)
👉🏻 (https://github.com/wey-gu/nebula-operator-kind#whats-next)

A

Days agenda as euro we will have introduction of any newcomers.

A

Then we will have project updates in about five minutes and again, if any um at hot topic, we can, anyone can bring the topics like any certificate certificate, um any uh specific issues to be taught- or you want to bring some works- that you have done related to nablograph. You want to share to community anything or some interesting topics to be discussed and there will be a sink discussion of the open floor time in the end.

A

So today I prepared a ad hot topic related to nabla graph spark, connector and level graph algorithm um working with the naval graph cluster running on top of kubernetes. So this is a hot topics discussed in the community in last couple of weeks that I consider worthy to be shared.

A

So all the uh agenda and topics uh you can anyone can put in this ether pad it's a online pad that useful for the community collaboration and more information. Please uh refer to our wiki and slack channel. You can reach out to us in the either from the pad or in the slack, and I think let me check newcomers.

A

Oops, maybe okay, I didn't share my screen.

A

Okay, I'm sharing um okay. There is no new commerce, unfortunately, this time, so I will go solo again.

A

A

This is me way as a euro.

A

um Again, I want to give a brief introduction of our way of working of the community meeting and you can see that what will happen bi-weekly and you can subscribe our registrat on the zoom or subscribe our google calendar. You will have a reminder before the meeting and during during the the two meeting points you can propose your topics or you want to involve some part of the community in next meeting or coming meetings.

A

You can request that from our uh different channels and we will try our best to arrange and during the meeting week we will go through the the routine topics and there are ad hoc topics, your proposal, your idea of story, um and afterwards we will archive everything, including the video and the text, version of the information, and you can check it in at a time. So that's the way of working and then we will start our first part the project heartbeats of the last two weeks, um the very first one is.

A

We have a minor release, which is the you know, the the dot to the final doctor, which is the minor version, and there will be no break change, but only hot fixes, and this um actually only three issues was fixed fixed in this release of the core part and the the main part is it's a memory issue. You can refer to the contents of the reset release page, and this is the trigger of this release and we also put other two hot fixes in this cycle.

A

Regarding the master core, um there are mainly only three three parts uh merged uh for two reasons. So it's not a lot of contents, because uh one is uh the last two weeks are a public holiday, a long holiday in china and most of our community members are chinese.

A

So there wasn't too much work in last two weeks and another reason is: we are in a time that we already have a release, basically uh released candidates of the the mer, uh the major release that we already introduced the v3.0 and we are in the face that doing the stability test, so the major part was already merged. So only two new features.

A

One is the snowflake id, which is um the universal uh unique id generation uh function, that user users uh are already eager to have this and they requested in a couple of different issues.

A

We managed to make it happen, then merge in the master and the basically, it will be included in the v3.0 and the uuid uh uh clause was bring back. So um if you are old user in apple graph, you know that in version one we already have a similar clause, but you have to put a seed inside a string seed as an input argument of this function.

A

uh This is because in we want, we only support the the number of the int format of the vertex eddy. So when your identifier of your vertex, our string, you can now reach this uuid function to make it into a number to hash it.

A

But when it comes to v2, we have a string format of vid supported and we remove this function and in this time, when we want to bring the uni universal, unique id function, generation function. We use reuse this keyword, so you can refer to more information from the pull requests and there will be documentation um in in coming weeks.

A

Another major function is we now support the single server or the standalone. So what's different between this single server and you deploy three different services in single server, so the difference is now. There will be only one service and especially for the uh graph d function and the storage d function. They will be in one um thread process, sorry, and there will be no rpc call between the uh gravity and the star hd, so it will cost less. So performance will be much better in the single server or standalone uh scenario.

A

So, as I recall, the the manta, the the cause to metadata still rpc, but you know it's not a major part of the the performance so have fun with that. You can check out the um the purecast and there will be documentations on how you can configure it, and you will have only single service to leverage your single server mode.

A

The next one is a bug, fix that and there will be corresponding uh prs on the pr site, so it it. So it's observed that if your cluster is upgraded from v2 to v3 and your br uh will have some issues and this fix so on the other side uh on the kubernetes operators, welcome hera.

A

Hi herrera welcome, so you are the very first member join joining our meeting and I will continue on on this topic and and between uh before next next topic. I will uh I invite you to give a brief introduction. Okay,.

B

uh Do you mean me, okay, yeah,.

A

Yeah, yeah, okay, okay, okay, thank you. uh So I will continue the kubernetes operator part. We have a new pr merge and, as related to the pause spread constraints, so that's related to the spread policy, so you can refer more information from the pr that this configuration uh flag was introduced so and by default.

A

Is I don't remember, is I don't remember the default value, but uh if I record it correctly, is that if your um spread situation is not satisfied, it will do nothing, it will not create new resources, as as I recall you can, if you have no questions you can discuss with us. So this is a new function on the kubernetes operator of the nabla graph.

A

Another thing is that we have created a new project which uh cam kanban boarding in github uh regarding the next three cycle, which which will be um approximately uh three months later 3.1, so you can check out this projects and there will be. um You will see the the major uh works tasks that is ongoing or done from that view. Another thing is uh um nebula agent, as we introduced in 3.0 preview is a new process introduced uh mainly for now.

A

It's mainly helpful on the backup and restore so maybe in future there will be more functions included. So the very first three release was was just done yesterday, so check out this uh tag page. So basically, that's the main part of the project heartbeats.

A

So before we move to next, we will welcome our new member of the community meeting hera. So hara, can you introduce yourself maybe turn on your camera? If it's comfortable to you, it's not necessary.

B

A

Up to you, it's up to you, okay,.

B

Okay, then, I will not today not looking so good, so uh I'm hera and the one of the mega graph community member and just want to know what's going on for the product and the community, so I can continue. Please.

A

Okay, okay, okay, thank you. uh So hera is one of our my uh mace, my uh workmen colleague in the community operation teams, so she's, a very uh kind-hearted and very um funny great girl, feel free to reach her uh for the questions in the future. Thank you, hera. So we will continue yeah um yeah. The next part is the ad hot topic that I prepared it's regarding how we can run a naval algorithm on top of that graph deployed on kubernetes, so why this is worthy to talk about.

A

You will know it later, so I will start with a couple of different uh perspectives, which is I consider most of them are the um prerequisite background information for this topic.

A

um So first I will give some a brief introduction of the architecture of nabla graphs. It's crucial to this topic, so this is the page of the documentation on the architecture overview.

A

We all know that there are three services for nebulograph and they are the graph d, which is the graph service or graph daemon. The method which is the meta service and the storage d, so gravity is actually in charge of the graph query handling.

A

That is when you are doing the ndqr query, the match: query and your session, uh your client will will be talking directly to the grafty welcome jamie, and I will invite you to give a brief self introduction later, when I finish this part. Okay,.

A

Maybe your audio device is with some issues, maybe you're talking. I see you are not muted, but there is no wave of your voice.

A

So I think you already hear me so I will continue and the after I finish this topic, I will kill you okay, so I think it's okay, okay, um so anytime you are making, uh for example, from your nebula console or other clients. You are making queries or write data into network graph. You actually are talking to the graph d directly and only talking to graph d directly and like, for example, you are making a logging session.

A

The metadata will be involved because all the metadata or the authentication data and the the cluster distribution data are, you know the metadata of the graph, so they are all persistent handled by the metadata service and for your, like your user and password information.

A

But when you are looking logging with the gravity session that you are not direct, you are not directly talking to the manhattan service and instead gravity will be a proxy to help you to check with the metadata service so and when you are actually querying the graph data, the gravity will talk directly to the storage statement or story service.

A

So all the queries are only directly through the gravity services to the end user or the client users. So that is the uh the basically uh basic information regarding the service, but there are certain cases that the end user can somehow leverage the clients to directly talk to mandatories and the storage service. But that's not the the most general use cases that I will review later on that cases.

A

So- and this is the basic information regarding architecture part and then I will not going through all details, but only the related ones on this topic. So the next one is how nebulous service I landed in a kubernetes environment. So I will demo you uh from this. So this is a kubernetes environment that I deployed, and this is the service view. So we can see the part wheel.

A

Let me see oops.

A

uh Sorry so out, this is the part wheel. You can see that there will be storage d instances, the meta d instances and graph the instances, and they are all the um you know, the part of the uh combination, services, uh resources uh and underlying they are just containers right. But how can we talk to them? So this is related to uh services.

A

So when it comes to service, we notice that, for both gravity metadata and stories, they are services, but only the graffiti is actually serviced with the cluster ip and for ha and for media and storage. There are headless that is without any cluster app here, that is by default.

A

When we spawn a nebulograph cluster through the nebula operator, there will be it's not possible to access the naval graph cluster directly to the gravity and uh sorry to the metadata storage.

A

The end and that's by design for reason that I explained in last part that for normal use cases we only talk to the gravity right, but uh so that is the main information that I want to review in in this part, let's back to to the main page, so uh I will explain uh you already know that it's by design and and that brick has um and that's for good reason, because we only normally we only use queries to talk with the cluster, but I will review more information on the later part on the nebula algorithm, which obviously require more than that and that's why we need to talk about this topic.

A

um Then we will go to a next part. The service discovery in nepal graph and that's a major part, help us to understand why we should or should not do this during this uh scenario, so how um nabla graph services discover each other. So uh the thing is the the all.

A

The service states are actually maintained by the manta service, and you know the service information is also meta for the you know, graph, nabla, graph cluster so and, and literally um they should be uh put here right, but how are they fund or discover each other?

A

So the thing is: uh when we put boot up a cluster of navel graph, we should first put the metadata service, always because that's the you know the meta of the graph right and then a manhattan, but if you have more than one metadata service this they should be aware of each other and how could they know each other and that's uh of the first part of this discovery, so we should configure them in each of the method instances. So that is you.

A

You should configure your metadata configurations and let it know all the other metadata addresses. So it will check through that uh ipn port to communicate each other. So metadata service is ready. Then we will start the storage. The storage need to uh know uh where the map data, where the meta service is so they need to know the same information through the storage. This configuration so stories. You will know all the meta these instances and it will communicate with them and also the method will in in this communication from story z to the meta d.

A

The medi will know the uh the stories these address, so that's a service discovery typically right, but but how? um But how uh metadata will know exactly uh the storage. This uh ipn part it comes to the configuration again and the information uh infoflow will be go from the storage. The configuration so stories uh d will have a configuration of itself on uh which ipo it will be listened and which part it will be listed.

A

Listening and then this information will be passed to meta d and how grafty will discover the storage d star the gravity, then after star uh the stories they will be up, the gravity will also be up. Finally, and the the flow will be, the gravity will uh config to the metadata, so it will know where the melody is, but there will be no storage deconfiguration to the grafting gravity will get the storage the information from the meta. We already know what madhai is measuring uh doing right.

A

So the main information here is that normally the client said or or the consumer of the storage, the will not know the storage con address or ip directly. Instead, it will get information from the metadata and our takeaway is where the meta, the. um How will maddie know this storage these addresses it will come from the stories. The configuration and stories d will. Let the manhattan know this information. So that is the key point of this part of how this service was uh discovered in nabla graph.

A

So the next part is nebula algorithm, so negr nabl algorithm. We we already know that it's application to enable nebula uh graph to have the the graph algorithm on top of the the graph data enable graph. So you can check more on the related either on related github, repo or from documentation here, but I will not go into details, but I will give you some brief introduction of the configuration of the algorithm. So every algorithm is underlying a spark application, so you have to run another algorithm on the spark cluster or spark runtime.

A

So, each time you want trigger algorithm computing, you will let the algorithm program to talk to the nebula graph cluster and to know which algorithm you want to run so how we should let algorithm to know that is uh configuration of course. So here uh I will show you the part uh in enviro. In my environment, I set up a a pod, that's running a spark inside and and then we'll go inside this. So I will uh show you the configuration example of the nebula algorithm and that's related to our survey to our uh problem.

A

So by default also, we will start the configuration part. So this field is, uh you know the spark related configuration. uh We will give this spark app a name so in this example of a configuration it's the lpa algorithm, so we give a name and I'm not going to dive into all the configs, but only related, so on data part, it's related to how this algorithm application will get infer reader info from the source and the sync will write in for two another target.

A

So the source here is nebula and there are other options of csv and json. Of course, so nebula refers to the nebula. Algorithm will talk to the nebula cluster and the sync will sync to a file.

A

Also, you can choose to sync it to the nebula cluster, but in this case I just sync the computed information in in a file- uh and this is the nebula part. If you choose the source as nebular, you will configure the related read inside the network configuration, but the main part here is related to the addresses.

A

You will know that the only information you taught regarding the connective connectivity uh of the nabla nablograph cluster is the meta address address. We already know that with the meta address you will know all the other service addresses.

A

So that's reasonable right and here we you will notice that we, we put a domain name with the port and the domain name is so long and a little bit weird. But but I will show you why we can see uh this is the console connected to this nebula cluster.

A

So when you see show hosts meta, you will know that by default, uh a cluster spawned by the nebula operator, the address configured by the uh naval operator will be something like this and that's why we put it uh in in a configuration like this, and this is a space, and this is the the the label which is a tag of the the graph we want to compute, and this is the edge the properties of the edge which is start here. So this is the main information that I want to review for. Why for how?

A

We want to run nebula algorithm on uh kubernetes. So with all this information combined, I think you already know the key points of the.

A

How we want to do this, and actually so, the main takeaway is you need to um specify the the meta um address in the application and and and by by specifying that the underlying requirements is, your spark will need to have the ability to talk to the manhattan address, which is, I showed you, which is the headless service, and then actually, your uh your nebula algorithm or your spark environment will need to talk to the um the storage service as well. uh You can see from here that.

A

The storage uh addresses config in your storage service and then discovered by meta. The information can be created by uh source host storage and they are also heavy services and, as as I already shared uh in the first part, that uh second part, uh all the headless service cannot be accessed outside of the kubernetes cluster. So you have to put your spark inside either you put your spark inside your of your nebula cluster or you make some hacks needed in operations configuration without reaching the kubernetes ingress.

A

But in this case uh in in a simple case, I will show you that I'll put the spark in the same cluster so so details. uh How to I will I already put on this repository. Also, this is a repository um help user to um you know play with the nebula operator in a kind, so kind refers to kubernetes in docker. So it's a a very small footprint on your single server of even desktop play with the kubernetes, and I put everything ready to use on this environment.

A

You can in one liner, you will have a kubernetes in docker and a nebula cluster inside that, and there will be a chapter called. What's next uh I will click this and you will see what's next here in this read me of this repo.

A

You will see the steps on how you want to run a nebula algorithm and actually the key point I already shared with you that you will you have. You will have to run your spark with the same network of the nebula cluster and you want to configure the corresponding addresses in the headless services and that's and and and if after you follow this step-by-step tutorial, you will have everything ready to use so, okay and and and this is the the main part of this topic, I think so. uh I put the major steps here.

A

So this line is referring to you spawn the cluster and that's one: is you want to create your um your spark uh part with this line, and then you will have this line to access the spark console, and then you will continue to uh config the algorithm rhythm uh configuration and then run the algorithm application.

A

So I think that's all part of this um okay. One thing to mention is that when I was preparing for this I I found a bug in in the algorithm and the nablo java clients. I make a pr that was merged yesterday on the master, but to not yet merge in the 2.6 branch.

A

So I make a patched uh package so when you are referring to my repo, that's underlying is referring to my personal build, so maybe in next week we will have a formal uh release on the nebula uh spark connector and algorithm, so I will update accordingly so yeah. I think that's the that's. The this add a hot topic.

A

So uh jamie are you there. Can you.

C

Yes, yes, I'm here.

A

Okay, so welcome jamie uh jamie is, is our uh nebulograph uh community manager, and uh maybe you can tell me, tell us more about you.

C

Oh okay, I'm not quite prepared for that uh so and thank you uh sway for such an amazing community meeting and I um I'd say we're going to keep doing this and providing more information about the nebulograph community and keep the community updated about. What's going on in the community, yeah and hope, more and more community members can join this community meeting and share their ideas about what they want, uh yeah, how uh how they are, how they want to um join the community, yeah and- and I think, that's all yeah. Thanks again, we.

A

Thank you jamie awesome. So then we will go to uh next part. Oh final yeah, uh so we basically finished all the agenda uh topics and uh so uh be subscribe. Our calendar or register our uh zoom meetings to not missing one and will next meeting will happen two weeks later and reach out us from uh from our uh sorry from the ether pad and propose your topics and put it there or talk us from our slack channel. And uh finally, before we finish it, uh we have two uh news uh to be to announce.

A

The one is uh regarding our the vsoft team that uh our nablograph cloud, which is the nablograph as a service providing will be open beta soon, maybe later in this week. Hopefully so stay tuned, and you can play with that managed service on a public cloud infrastructure.

A

So it will be easier for many teams, as I believe another news is, I will be giving a talk regarding nablograph and some know hows on the dok, which is the data on kubernetes community later in this week, so you can reach out to subscribe. This live streaming on youtube from here and I think that's that's all from today and we can call this end.

A

So I see only jamie uh joining on this meeting. So we we won't, have a uh sync discussion, uh so we can say goodbye and see you in next weeks.

C

See you we bye.

A