Nebula Graph Community Meeting, 26 Jan 2022

Previous Meeting

⏯

youtube image

►

From YouTube: Nebula Graph community meeting [2022-01-26] (3.0 preview, dgraph and PostgreSQL import)

Description

# Topics
00:00- Intro
01:48- Community meeting kick-off and way of working - 5 mins - wey
07:02- Project heartbeats - 5 mins - wey

10:45-## PR:
- LDBC test cases introduced: https://github.com/vesoft-inc/nebula/pull/3537
- K6 supports Nebula Graph v1: https://github.com/vesoft-inc/k6-plugin/tree/nebula_v1
- `SHOW HOSTS` now reports version: https://github.com/vesoft-inc/nebula/pull/3702
- Configurable `max_expression_depth`: https://github.com/vesoft-inc/nebula/pull/3722
- Limit number of sessions per client: https://github.com/vesoft-inc/nebula/pull/3729
- PostgreSQL support in exchange: https://github.com/vesoft-inc/nebula-exchange/pull/62

14:26- Dgraph Data Import to Nebula Graph
- option 0 https://github.com/vesoft-inc/nebula-exchange/issues/57
- nebula-spark-connector & dgraph-spark-connector
- option 1 https://github.com/jievince/rdf-converter

17:03- Why is nebula in python slow?
- https://discuss.nebula-graph.com.cn/t/topic/7285 (Chinese, but with python code)
- TL;DR:
- CPU-bound in case of huge returning data(unserilzation), multi-process should be considerred
- IO-bound in case of long query(like `count(*)`), multi-thread helps only in this case

19:25- GraphD crash on CGROUPv2 OS/Container host OS
- https://github.com/vesoft-inc/nebula/issues/3278
- 2.6.2 is on the way

22:06- Open floors for a sync discussion/Q&A - 0 mins

Check more on https://etherpad.opendev.org/p/nebula-community-meeting

A

So waiting for anyone to join.

A

Maybe I started too late.

A

A

Okay, so maybe I will start because we didn't promote this uh very first community meeting on time, so there isn't too much guys participating us. I will start because we will archive the meeting anyway, so everyone can see this recording.

A

Okay, so I will start a solo meeting, so let's start hello, everyone for those, maybe not join now, but in the future. I'm super excited to. I'm super excited to share with you that we will launch our community meeting for naval ground community community and the agenda today, four of them.

A

So uh apart from the first one, which is the introduction of the way of working of the community meeting, the rest are actually the routine uh process of a meeting that you will see how it looks like actually later so um first is the way of working of the community meeting.

A

So as we can see that the meeting now was designed as uh bi-weekly, so this is like this is the week four uh today and next two weeks uh we will have another meeting so before um before a next meeting, everyone can prepare and propose their topics and put it in our agenda in our in the ether pad that uh the etherpad. You may already know that this is the wiki our homepage of the meeting, and you can click here to have this ether pad.

A

So, for example, before this very first meeting, I have put my topics here with my uh uh approximately uh time and and my name and um in in a typical case, I will suggest you to uh also ping us in a slack uh we will have. We have a channel named the nebula community meeting channel in slack, so you can ping me and or anyone else or you can start questioning on how you would like to make this discussion topics there and for other uh async communications.

A

You can still do that in slack, github forum, etc.

A

So, when it comes to um actual meeting, we will first introduce the new um members. I will do that later, so, maybe only myself, we just said the story, but not that sad, we're only very beginning.

A

We will have some heartbeats updates of the project. For example, some interesting pr's issues, some interesting threads in the forum or offline discussion that we consider to be meaningful to be shared and yeah. This is the ad hoc topics part that everyone can prepare and propose it here. You can give your proposal introduced.

A

Give your idea discuss, share a story either it's related to naval graph or not, if you like, or you bring something you made it. I made this and I want to share it with you. It's more than welcome to be demonstrated here and the last part will be the open floors. If anyone want to have a sink discussion, some questions, or even you, want to involve certain parts or contribute towards of the community. You can propose it and let us know if you would like someone to be involved.

A

We can try that and after the meeting, everything will be archived and published on internet and you can find them in archiving youtube and enter etherpad. So anyone else in the future can learn from that. Hopefully, so, basically, uh this is the way of working of the community meeting.

A

Okay, I see, let me see if anyone joins.

A

A

And so we start with the routine, so this is our very first only me, so this is weight. I'm the developer advocate in vesoft, which is the team behind the nabla graph, and I love open source. I, like coding- and I love graph text I like to collaborate and help you out there in the community, just ping me from the twitter slack or forum.

A

So that's, that's. All the members join this meeting today. I think okay, um so we move to the next section.

A

The project heartbeats, so the first part I want to share is about a next release that actually um like yesterday or today before yesterday, we already have a session on the preview of this release, which is 3.0, which is the third major release of nabla graph, and I have given uh like 30 minutes uh talk in this in this uh youtube. It's a live streaming talk, and this is the slides and you can.

A

I will quickly go through the contents of of the new release, but you can you can check more details in the video we. The main features introduced are some more support um capability on the open, cipher.

A

um The very first one is: we support the multi-pattern mesh, so it says it will uh significantly improve the expression ability of the match pattern and also we support a parameter that you can let the service at helped you to validate your query. You can send the parameters and the template together to a server side, and this is the second one. Is the birth eye view of data. That is, you can look up or match data without you necessarily needing the the index.

A

The only requirements here is you put a limit of the clause, so that will sample personal data of the of your query without needed of without needing the index, which is quite convenient another one is we support the bare vertex, which is the vertex without any text, so you can create a bare vertex without any text or you can detach all the text from a vertex and it will. It won't be deleted anymore. So this is a change uh comparing to the version two, and we finally will release the backup and restore in this cycle.

A

So and it's actually redesigned we decoupled ssh and we keep the user the api and changed, but it's more friendly for the cloud native case for enhancements. We re visited and refine our metrics and we introduce a lot more metrics like 15 more and we have the support of the kiwi separation, which is the outcome of the nebula hackathon in last year.

A

We refined and refactored the index execution plan and the raft we introduced the tr support for the elastic search client and we implemented the log rotation from the server side. So you don't have to use your external utility to do that anymore and that's the first part on the preview of the next release, which will probably happen in february.

A

um So this part that I will do it in every meeting in the future are some highlights of some interesting pr's. So I will go through very quickly for you. If you are interested in uh them, you can discuss with us or check the pr's.

A

The first one is with the help of the open, cipher improvements, the new features um we bring a lot of ldbc cases in our unit tests, so this is the pr that you can refer a lot of multi-pattern expressions in in the code in the test code and k6. If you don't know that k6 is a tooling to do the performance stress test and the nabla graph community has a k6 plug-in to stress to put pressures traffic on double graph and recently there is a pr to make it supporting the v we want.

A

So this tool was introduced in the v2 uh time period. Now it supported the legacy v1 as well, and the this is the recently merged uh pr that show hosts now can uh reports details uh version previously. You have to do show host meta show whose graph, because uh in in history, we used to separate different parts in different repos, so the version of the github, the gate, hash information vr- is a max expression depth. So this is a new um configuration.

A

Previously we hard coded it in a value like 500, 1 and and 12.. Now it's configurable, the expression depth actually means. When you are uh parsing, your expression, every uh part will be considered one less as one layer. So depth means like one plus two refers to uh dapps in two. So that's a more uh smaller um um it's a smaller piece in adapts, rather than the jumps or pipelines.

A

You can uh look uh check up to details in spr and next one is the this vr that one of our community contributors introduced a limited number of sessions per client. That is um with this work. You can optionally limit session and numbers per ip address of the client side.

A

Another one is also uh not merged a pr. It's it's appear in the nebula exchange, so for your information, nebula exchange is a spark calculation application that enables multiple data sources uh to be imported to a network graph, um like my c core, like hive, neo4j, etc.

A

So this pr will introduce the capability to import data from the postgres database and it's still being reviewed. So that's all from the pr in last two weeks.

A

Yep and next part, I will share some interesting threads or discussions among the community, so there will be three of them.

A

The first one is someone actually is not the first one that someone was questioning that how we can import data from d graph, so actually after box nebula exchange doesn't support the source yet. But you can you actually can do that in several different ways. One of them proposed by our maintainer of nebula exchange is that, but you for details, you can check on this issue.

A

uh You can use nebulous bar connector and uh the ground spot connector. So we running on spark. You can leveraged two connectors and uh to parsing the date, the source of truth from day graph and uh persistent uh persisting or writing it to nabla graph that won't be quite complicated, but similar way you can use any other programming languages that we already have a client, that's comfortable to you, um like nebula python, together with the python clients of the d graph.

A

It's a similar case, but maybe spark is helping on leveraging multiple uh computation resources so, but you can never leverage any any clients in theory. Another option is that gm, who is our community contributor maintainer of nabla graph core introduced this this project, which is the rdf converter?

A

This is the repo here with that you can. Firstly, you can export your data from draft as the rtf format, and then you can use this converter to make it into uh csv files, which is compatible to nebula importer. That importer is a go binary file to enable csv file to be imported to network graph, so that's also an option so another topics uh uh in our forum, but it's in in chinese, but you can um get some information there, because the sample codes are in python.

A

uh Someone is questioning that nebula python seems quite slow and our enabler one of our maintainers in nabla graph share this um experiments that to demonstrate how or why is slow and how we should solve them in certain cases, and so in short, um there could be two reasons. Mainly one is uh one case. Is that your traffic is your cases, scenario is cpu bound, that is, the python code is basic, doing the unserialization work.

A

So in that case, in that case, normally you have a bunch of data being returned from the server side and the python was baby. You know, python was wasn't quite that's good at doing this cpu bound tasks.

A

So in this case, uh your way out, if you uh stick to python, your way out will be leveraging the multi-process to you know, then you will not be limited to the gl um with this case right. Another case that your query is actually uh quite slow. You don't. uh The python code is not basically um handling the data and serialization case. Like you, query a lot of data and give a pipeline account starts. So that will I, in that case.

A

It's actually, I o bound, because your client is just waiting for the server to return. A long run query. So in that case the multi-thread will help you, but this is a typical uh I o bond case, which is quite typical um experiment and quite useful for the newcomers. I think so. The next topic that I want to bring is um the support um of the secret way one.

A

um So this is typo crash on c group. We too actually.

A

A

And the background here is that uh we we now we introduced a mechanism to uh check the utilization of the memory and that we assumed every host os comes with a secret v1, um but actually in in some modern uh environments. It comes with the secret v2 and that will crash the graph d.

A

um So this is the related issue. Actually, this is not impacting most of the production environments as we observed, but unfortunately, uh last month um the docker desktop have a 3.4 4.3.0 that replaced their uh you know. The docker desktop is a tool provided by docker and they actually put a vm on your windows or mac os, so that vm was replaced with a newer version of the operating system introduced the secret v2 since 4.3.0.

A

So if you are running nablograph on your laptop's docker desktop your graph, your gravity will crash. So in that case you can go back to the last version, which is the 4.2.0 that darker desktop. That will help you to solve that issue, and but if you are running uh nebular in container environments and your, uh for example, your kubernetes node is quite new.

A

uh Please just wait a couple of days that we will have a new uh release of 2.6 uh in near future and that will bring this bug fix cherry picked so that won't take you long and that's all parts of the project updates and, I believe, maybe yeah. I think we are reaching where enough top topics. Now let me see if anyone still join okay.

A

So before we um call end of this meeting, please be noted that we are bi weekly meeting and our next time will be here in february, and this is our ways to join the meeting. Our zoom google calendar uh be sure to check the ether pads on. If any topics are interesting to you and be sure to join our slack channel, we have a community meeting channel that you can talk things around the community meeting here and I was there anytime.

A

So maybe we can call this and okay. Thank you see you in two weeks, bye.

A