Nebula Graph / Community Meeting

Project Heartbeats - Wey GU
Introducing and Demo of NebulaGraph 3.3.0 - Wey GU
Demo: How I model data from multiple sources in tabular format into NebulaGraph with dbt and NebulaGraph importer. - Wey GU

In this community meeting, Dr. Xuntao CHENG shared us his work on refactoring NebulaGraph BindingTable.

Jingchun Xia, Head of QA & DE talked about how we test NebulaGraph this time.

- Project heartbeat
- Fraud Detection with NebulaGraph by Wey Gu
- nebula-dgl the project for NebulaGraph DGL integration, by Wey Gu

ref:
- https://www.siwei.io/en/fraud-detection-with-nebulagraph/
- https://github.com/wey-gu/nebula-dgl/

This time the ad-hoc topic is about NebulaGraph Cache design for now and future by Wen Hao, our contributor in Storage.

Ad-hoc topic: A reference solution for Data Lineage

ref: https://siwei.io/en/data-lineage-oss-ref-solution/

Project heartbeats

- Core
- Shortest path optimization #4071
- Opt: retrieving input/Variable property from index instead of property name #4174
- More config added with meaningful default values #4310
- MATCH
- Move input rows of Join based on lifetime analysis #4186
- rule: removed useless project #4157
- Stop scan edge opt when build path #4053
- Push vFilter down #4260
- Push eFilter down #4270
- `tuned` Profile #4255

- Nebula-Agent
- Support incremental backup https://github.com/vesoft-inc/nebula-agent/pull/16
- Nebula-NodeJS
- Sew release v3.0.1 with v3 supported by @wujjp & @wey-gu
- CI Publish/UT added by @wujjp
- Nebula-Exchange
- incorrect partition id in sst https://github.com/vesoft-inc/nebula-exchange/pull/82

Participants
- Lindsay Liu
- Zhong Tian
- wey gu
- Eric Laquer
- Marco the handsome dog

Ad-hoc topic: Nebula-UP (v2) All-in-one introduced

Now Nebula-UP can enable a lot of nebula graph tools to be installed in one line for fresh users to play and test:

👉🏻 https://github.com/wey-gu/nebula-up

🚩 Project Heartbeats

- Remove useless AppendVertices https://github.com/vesoft-inc/nebula/pull/4277
- `MATCH (v)-[:like]-() return v` no need to append dst vertex when only returning src
- Explain/Profile Dot format fixed https://github.com/vesoft-inc/nebula/pull/4280
- `query_concurrently` bugfixes https://github.com/vesoft-inc/nebula/pull/4288

🎙️ Hao Wen will talk about Nebula Graph in Sigmod 👉🏻2022.sigmod.org/program.html

Nebula Graph Project update:
- Core
- subgraph last step pruned when outputise vertices only #4226
- MATCH perf optimization #4122
- Seek by ID no filter #4249
- fix listener can't receive snapshot anymore #4245
- Nebula-Go(All by Tiago Peczenyj)
- remove remote services, the code is not necessary to use this project as a nebula driver #200
- Create integration tests, now we could run pure UT w/o heavy container-cluster up #201
- PR REVIEW WIP: Code of conduct; IPv6 support; Multiple GO version tests support; Connection string support and session pool 🚀
- Nebula-node
- ping function bug fixed by jjsimps https://github.com/nebula-contrib/nebula-node/pull/12

Ad-hoc topic:

Graph Data Generation: Experience on create dummy datasets with graph structure

Details refer to release note: https://docs.nebula-graph.io/3.1.0/20.appendix/release-note/

Project heartbeat

- #4098 `RETURN extract(a.player.name, "\\w+")` by @wuxiaobai24
- Extract the substring that matches the regular expression from the string
- #4095 Refactor FIND PATH:
- reduced RAM usage of allPath & multiPairShortestPath
- reduced data copying by integrating conjunct path into the path operator
- #4131 Disk fault recovery improved with multiple data path awareness
- #4129 The single server now supports auto adding hosts
- #4146 Optimizer refactor:
- now only release symbols of top layer nodes in all matched nodes
- #4162 PATH optimizer refactored:
- redundant path removed, time complexity reduced

Spark on Nebula Graph, details also in https://siwei.io/en/spark-on-nebula-graph/

🚩 Project heartbeats
- 3.0.2 released with ARM support
- #3997 support `MATCH p=(v:player)-[:serve]-() WHERE (v)-[:follow]-() RETURN p` now
- #3929 support `MATCH (v)--(v)` now
- #3677 More comments in codebase
- #4085, #4067 more information on `SHOW JOB `
- #4088, #4017 Balance Data
- #4100 Notice in Storaged service status for Not added hosts
- Nebula K8s Operator Supports 3.0 Now

Ad-hoc Topic:

Nebula Graph on SBC/Raspberry Pi
- https://www.siwei.io/en/nebula-graph-on-pi/

🚩 Project Heartbeat
- #3993 Something error when the version of tag/edge schema is greater than 256

- #3946, #4060 UDF progress

- #4063 Query optimization summary based on cache experiment

- #3771 persist learner info

- #4030 refactor: use STL facilities instead of 3rd party libraries(Jun)

- Feature:

- #3989 clear space
- Others

- #3905 console in binary packages
- Nebula .net, Nebula jdbc with 3.0.0 supported

Ad-hoc topic

How I use Knowledge Graph(with Nebula Graph) to help solve Chinese Wordle

💡 Ad-hoc: the workflow of contributing to nebula graph core: Build/UT/DEBUG Flow get started.
00:00 - Intro of the community meeting
02:40 - 🚩 Project Heartbeat:
02:47 - Interesting ISSUE:
- https://github.com/vesoft-inc/nebula/issues/3941 thread pool perf
- https://github.com/vesoft-inc/nebula/issues/3940 multi-get rocksDB
Surrounding Merged:
- https://github.com/vesoft-inc/nebula-third-party/pull/37 arm build supported
- https://github.com/vesoft-inc/nebula-operator/pull/117 Operator: Helm Auth supported
Core Merged:
- https://github.com/vesoft-inc/nebula/pull/3904 use one global lock in meta
- https://github.com/vesoft-inc/nebula/pull/3896 permission for kill queries
- https://github.com/vesoft-inc/nebula/pull/3913 auto_remove_invalid_space default true
- https://github.com/vesoft-inc/nebula/pull/3839 https://github.com/vesoft-inc/nebula/issues/2533 GetProps supports Limit push down
09:14 - How to contribute to Nebula Graph Core

👉 Join the meeting from the link: https://us02web.zoom.us/meeting/register/tZ0rcuypqDMvGdLuIm4VprTlx96wrEf062SH

Project Heartbeat

00:00- Intro
03:20- 3.0 Released 🚩! https://nebula-graph.io/posts/nebula-graph-v3.0.0-release-note/

03:42- Master Core
- Prop pruner (https://github.com/vesoft-inc/nebula/pull/3750)
- `--disable_page_cache` (https://github.com/vesoft-inc/nebula/pull/3890/)
- GetProps LIMIT push down (https://github.com/vesoft-inc/nebula/pull/3839)

05:43- Spark-Connector and Algo(2.6.2)/Exchange(2.6.3)
- fixed address invalid when it's domain name

06:40- Nebula-Contrib
- php, .net, nodejs client moved here

07:38- Topic: Nebula Graph Index Explained
- 📚 https://www.siwei.io/en/nebula-index-explained/

00:00- Intro
03:40- # Project Heartbeat

03:49- `2.6.2` Core released 🚩(https://github.com/vesoft-inc/nebula/releases/tag/v2.6.2)
- fixed memeory issue/CGROUPv2 issue/full-text index issue


04:28- Master Core
- Snowflake ID supported in `UUID()` (https://github.com/vesoft-inc/nebula/pull/3500)
- Single Server(standalone) Support (https://github.com/vesoft-inc/nebula/pull/3310)
- fixed a backup issue if after v2 to v3 upgrade (https://github.com/vesoft-inc/nebula/pull/3855)


08:35- ⎈ K8s Operator (https://github.com/vesoft-inc/nebula-operator/pull/109)
- `unsatisfiableAction`, configuration for **POD Spread Constraint**


10:05- `3.1.0` project created: 📅 https://github.com/orgs/vesoft-inc/projects/28/views/6


10:30- nebula-agent v0.1.0 released(https://github.com/vesoft-inc/nebula-agent/releases/tag/v0.1.0)

12:20- # Ad-hoc topic: Nebula Spark-Connector/Nebula-Algorithm in K8s

33:21- The community meeting calendar

- Arch. of Nebula Graph
- How Nebula Services are landed in K8s
- Service Discovery in Nebula
- Nebula-Algorithm config explained
- How-to & Hands-on 🏂🏻 (https://github.com/wey-gu/nebula-operator-kind)
👉🏻 (https://github.com/wey-gu/nebula-operator-kind#whats-next)

# Topics
00:00- Intro
01:48- Community meeting kick-off and way of working - 5 mins - wey
07:02- Project heartbeats - 5 mins - wey

10:45-## PR:
- LDBC test cases introduced: https://github.com/vesoft-inc/nebula/pull/3537
- K6 supports Nebula Graph v1: https://github.com/vesoft-inc/k6-plugin/tree/nebula_v1
- `SHOW HOSTS` now reports version: https://github.com/vesoft-inc/nebula/pull/3702
- Configurable `max_expression_depth`: https://github.com/vesoft-inc/nebula/pull/3722
- Limit number of sessions per client: https://github.com/vesoft-inc/nebula/pull/3729
- PostgreSQL support in exchange: https://github.com/vesoft-inc/nebula-exchange/pull/62

14:26- Dgraph Data Import to Nebula Graph
- option 0 https://github.com/vesoft-inc/nebula-exchange/issues/57
- nebula-spark-connector & dgraph-spark-connector
- option 1 https://github.com/jievince/rdf-converter

17:03- Why is nebula in python slow?
- https://discuss.nebula-graph.com.cn/t/topic/7285 (Chinese, but with python code)
- TL;DR:
- CPU-bound in case of huge returning data(unserilzation), multi-process should be considerred
- IO-bound in case of long query(like `count(*)`), multi-thread helps only in this case

19:25- GraphD crash on CGROUPv2 OS/Container host OS
- https://github.com/vesoft-inc/nebula/issues/3278
- 2.6.2 is on the way

22:06- Open floors for a sync discussion/Q&A - 0 mins

Check more on https://etherpad.opendev.org/p/nebula-community-meeting