Nebula Graph NebulaGraph Ecosystem, 24 Aug 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: NebulaGraph for Beginners - Exchange SST (CN Version & EN Subtitle)

Description

This video explained the new feature of NebulaGraph Exchange SST for data importing. In the video, Siwei will show you how to operate NebulaGraph Exchange SST in detail so that you can follow him step by step to try it. Welcome to discuss on our Slack: nebulagraph.slack.com
Tips: if you want to see English subtitle, please switch the subtitles to English in the settings.

A

Hi ~, hello, everyone, I'm thinking this is as hera, hera, hi, ~, hi, ~, hello, everybody, I think, is very happy to meet.

A

It is this issue we introduce new features, nebula graph, 2.5.0 of corresponding nebula exchange.

A

A period that import data into.

B

The data of sst,, please think of us first, to tell you about the good that we hera first to recall our earlier, had revealed a content.

B

That is what is nebula exchange.

B

Nebula exchange is a spark. It is the application of our community to support from a variety of external sources of data to nebula graph itself is a tool or a bulk flow of imported.

B

Nebula exchange, that is how it works inside the nebula. It has three parts which are reader.

B

And processor processor, as well as a write writer writer.

B

That today we have to discuss this topic. Exchange is inside the sst. It is actually the.

B

First part of this green arrow, which path it is.

B

This writer's our normal nebula graph. We also flow inside the other processor and exchange writer writing process.

B

We can notice it even though a lot of reading performance, nebula graph- writing is very good, but the process actually involves a lot.

B

From graphd to storaged process, or some of the steps that we have not.

B

Calculated by bypassing these steps might be better performance, then the answer is yes.

B

Path as shown in the pink arrow exchange sst way, it actually is to make use of external computing power.

B

Directly generate sst file so that you can liberate some resources on our nebula graph storage cluster, which has.

B

Reached such a purpose and capabilities of the ultra-large-scale data injection, so the above is a brief introduction of the sst nebula exchange functionality.

A

That sounds good. It can be a great sst under the scale, can also be quickly (import data).

A

That would like to ask so what kind of data is more suitable to import it.

B

Right hera key scene is so fast sst in this way.

B

It is characterized by an external force by the count of clusters in exchange for written throughput.

B

So it is actually very suitable for the kind of super super large quantities of data import injected scene.

B

That us corporations, like micro-letter team, actually have a lot of line up to use this feature. You can go to our public forum. These.

B

Numbers to see if they share for more details of some of these use cases of.

A

Good xiexie si import this set of operations for the exchange sst. You can demo it for you right.

B

Now that the good taste to show you what the minimum environment inside such a test exchange, sst demo.

B

First, we have a linux server inside. We will build this environment that we actually can refer to the.

B

Entire environment, to build a shared articles, I have on my personal website: url is this: siwei.io.

B

That, here the edge we need to do, what does.

B

The first is to build a nebula graph that here we are with the aid of a docker way before I wrote a little program called nebula up.

B

You can use this line of code to put on a linux inside a nebula graph to pull up.

B

Extra that which we have to do is pull up the environment. After doing a little bit of change to its network becomes externally accessible so that we can use to access other docker, to which.

B

We will nebula graph in the other. Docker go inside, hadoop and spark run behind will involve so we can see that this is the environment.

B

Then we can see that I actually changed the docker compose file.

B

And then adds this network is mainly followed. What I for convenience, I have also deleted the two meta default. It is three meta.

B

Meta I deleted. The two major changes is to do.

B

The second part of that environment is to have a ready spark of our operating environment to run exchange this environment, and it.

B

Can be network access to our network of hadoop? Hdfs must also have access to the nebula graph cluster.

B

By means of this, we already have this big data: europe, an environment. I can pull it up in my environment that we can go to test.

B

So we already have a spark environment for a while I'll show you how to call it.

B

That other part of it is to take a hadoop environment. I also use the docker.

B

Clone it this after compose documents, do we do a little bit of modification to modify part of it? That.

B

I have actually increased the list to the same network that docker containers with nebula graph resides in this article, so that.

B

They can be interconnected between this part is then we have a good run, hadoop environment.

B

That the environment after it is ready, we will start to do the rest of the steps.

B

We can have a look at the environment that we can perform exchange application of such a generation, sst sst. Then, let's.

B

Go after the sst to do this download and injected nebula graph in the work.

B

That we first look at if we are to implement this generation. Sst is what we need to write exchange corresponding to such a configuration file, pieces of.

B

Paper that we have already described what we want to show up in the file inside nebula related information graph. Secondly, we have to describe what it actually is to.

B

Turn into one data source that there sst file for simplicity, we use the simplest, is to csv files. This document we've actually made good to give us a look at kazakhstan.

B

We can enter into this spark an environment where we can look at.

B

This is that we made a good.

B

Simple, only one type of edge csv file.

B

It first column is the id of the middle point of this column. It is a such a property called name name of the last one is the point of his age called a property age.

B

That we then look down is worth noting. We want to fill this meta address that there. Let me give you an example.

B

We saw before we made changes only a meta that it is in fact three.

B

Of the 9559 foreign listen mapped to an external address that corresponds to.

B

It is this 49377 here the edge. We need to do to fill this in this config file inside 49377, that I will give you a look at.

B

The example I wrote such a configuration file, so you see here, is below the nebula graph meta and then write that this password then graph space.

B

Here we specify that the sst.

B

Files after exchange generation that position remote is actually the final output of that this remote path is actually hdfs.

B

Then its address is such that, in fact, we've just had the hadoop up and running environment.

B

Will give you a show you the list of the remaining part of this document? There is information that describes you this side of the exchange. These points describe the relationship of such a corresponding.

B

Bit like inside importer, similar config file as it is the player we have just seen.

B

In the interior of the container path running exchange environment that I've already seen this document to you that this.

B

Is the first description of its column and row inside the second row.

B

Nebula and corresponding, so this is the name and age part of that article. So a generation is an example of the exchange to generate sst file itself, the configuration file.

B

That, then, we will be able to carry out sst file's also described how we csv this data. In this configuration and (data) sources, files are.

B

Copied to the interior of the container that the implementation of our spark of that I've done that.

B

We can now in the process of implementation of it. We can turn this on us up. Nebula, nebula, docker, compose pull up with the environment in this program of exchange.

B

Then we log the following: this is our sp. We have put the ark.

B

Exchange environment configuration files and csv files are put to the exchange, also put itself into a jar and put that inside.

B

And the way we use it.

B

Later submitted to the executive.

B

Execute it, we can see what we have sst-generated files,. We can, then we can look at.

B

The good news where we enter into our hdfs inside that namenode, our hadoop inside of the inside.

B

This is what we run: the exchange to generate sst file.

B

So corresponds to our overall, what steps we have completed the sst generated now by accessing exchange.

B

Nebula cluster and then read the source data.

B

Has been calculated to generate the sst file and put it existed, hadoop's hdfs, which.

B

It then we went to the next step to put this file injection the nebula graph, which is to this point that this step does. In fact, we can also refer to this document.

B

Is actually the first two steps called the second step called ingest, download, download, nebula graph server is, in the end.

B

It will pass the hdfs client needs. The sst file is downloaded to a specific directory where the file system inside storaged inside.

B

Its path that it actually is such that data- storage, nebula and space id is drawing space.

B

And the bottom is just sst download all the files clip all of the following documents must be placed in this place. Is the download part.

B

Because, in the actual situation, we need to ensure that the service side of our nebula graph environment, of hadoop, hdfs on the relevant client installed.

B

And that this fact,, so it is too much trouble. I gave you one step. It is equivalent.

B

To manually operate the manual, I go to copy all the files to the appropriate path there that I already know.

B

That there is such a worth, noting that we look at the id is space. What there is not a name string of space but its numeric id.

B

Describe space that we have here with this name, sst, that this situation is in fact 49 this work I have done in my environment, where.

B

We can do now. Look at this is to use this command. Is that we have, it will help us, through the nebula up to automatically install console container environment installed.

B

So I've executed after into the container, and then we can connect directly to us this space.

B

And we use sst is our the space just now,. Then we can see if our id is also 49. after.

B

The download is ready,, we ingest can do direct operations of.

B

This step is the last service will end all storaged from this path to go inside to its underlying rocksdb. Based on this document, do an import.

B

That we can pay attention to it at the same time. Look at the top part of the log is this part of the log.

B

We execute this direct ingest.

B

We finished this instruction has been executed and then we can look at the top of the log.

B

So we import more than it is a process sst.

B

We can refer to my article that can help you in your own little environment that try to do a more detailed content of it in the official documentation, hera.

B

Is also a good reference.

A

For that, our good xiexie si content of the video on here, which we later imported using the sst process.

A

If you have any feedback or questions, are welcome to leave a message exchange with us. Thank you. The next issue to see the process of bye bye import, if you have any feedback or questions, are welcome message to communicate with us. Thank you. The next issue to see bye, bye, the process of.

A

~ import, if you have any feedback or welcome to issue a message to communicate with us. Thank you. ~ ~, bye, bye, see the next issue. The process of import, if you have any feedback or questions, are welcome to share with us. Thank you message see next issue. Bye, bye, ~, ~,.