►
From YouTube: Nebula Graph community meeting [2022-06-15] Data lineage with Amundsen and Nebula Graph
Description
Ad-hoc topic: A reference solution for Data Lineage
ref: https://siwei.io/en/data-lineage-oss-ref-solution/
Project heartbeats
- Core
- Shortest path optimization #4071
- Opt: retrieving input/Variable property from index instead of property name #4174
- More config added with meaningful default values #4310
- MATCH
- Move input rows of Join based on lifetime analysis #4186
- rule: removed useless project #4157
- Stop scan edge opt when build path #4053
- Push vFilter down #4260
- Push eFilter down #4270
- `tuned` Profile #4255
- Nebula-Agent
- Support incremental backup https://github.com/vesoft-inc/nebula-agent/pull/16
- Nebula-NodeJS
- Sew release v3.0.1 with v3 supported by @wujjp & @wey-gu
- CI Publish/UT added by @wujjp
- Nebula-Exchange
- incorrect partition id in sst https://github.com/vesoft-inc/nebula-exchange/pull/82
Participants
- Lindsay Liu
- Zhong Tian
- wey gu
- Eric Laquer
- Marco the handsome dog
B
B
B
B
B
C
Terrific
yeah,
my
name,
is
eric
lacaire.
I
don't
care
how
you
pronounce
my
name,
I'm
a
data
engineer,
but
a
fairly
long
career
40-year
career
I've
spent
the
last
seven
years
specializing
in
market
logic,
which
is
a
multi-model
database.
One
of
the
models
is
graph,
so
I've
done
graphic
scale.
Implementing
kind
of
big
systems
like
healthcare.gov
is
based
on
this
technology,
so
it's
just
some
lexus
nexus,
some
pretty
big
stuff,
and
I'm
looking
to
be
able
to
bring
that.
You
know
this
technology
lower
price
point
to
smaller
clients.
B
D
A
E
Thanks
for
for
hosting
this,
this
is
john
tiana
coming
from
china,
beijing,
I'm
also
a
member
of
the
I'm,
a
new
member
of
the
the
nebulograph
family,
so
I'm
so
new
to
a
grappling
habit
as
well.
My
past
experience
has
been
on
the
cloud
space,
so
hopefully
I
will
learn
a
lot
from
you
guys.
Thank
you
guys.
B
Thank
you,
okay.
So
let's
continue
so
I'm
super
excited.
We
have
so
many
people
this
time,
so
we
will
have
this.
This
meeting
bi-weekly
and
every
everyone
can
join.
And
if
you
want
to
bring
some
topics
to
us,
it
will
be
fantastic.
So
just
let
us
know
before
the
meeting
anytime
on
the
slack
or
you
can
put
things
on
the
ether
pad,
so
everything
can
be
found
in
the
slack
or
the
wiki
of
the
community,
repo
and
yeah.
B
So
there
are
basically
no
new
functional
improvements
in
this
time
and
there
are
a
bunch
of
bug
fix
that
I
didn't
highlight
here,
but
some
other
improvements
I
can
quickly
go
through.
If
you
are
interested
you
just
dive
into
the
the
ripple,
and
so
the
first
page
is
around
the
nebula
core.
So
the
first
one
is
around
the
fine
pass.
So
when
you
are
finding
shortest
paths,
this
optimization
is
quite
interesting,
so
you
are
actually
doing
the
bfs
from
the
two
point
starting
and
the
destination
point
in
in
parallel.
B
So
but
when
you
have
even
a
number
of
the
shortest
paths,
so
it
will
be
different
that
I'm
not
going
to
dab
deep,
but
we
precisely
removed
the
actual
get
property
perform
execution
to
make
it
more
a
little
more
a
little
bit
more
performant
when
you
are
doing
a
shortest
path
finding.
So
the
second
one
is
the
optimization
regarding
when
we
have
the
input,
property
or
variable
property,
so
there
are
difference
of
them.
B
The
input
is
actually
a
pipeline
and
the
variable
is
like
your
clear,
a
variable
equals
to
something,
and
then
you
give
a
comma
and
another
sentence,
but
either
way
there
there
will
be
a
improvement
now
that
we
we
instead
of
query
the
data
from
a
property
name.
Now
we
make
everything
in
a
turbo
and
the
query
get
data
during
the
execution
from
the
index
of
the
turbo
so
because
it's
actually
immutable.
So
this
is
another
performance
improvement.
B
So
there
are,
some
of
them
are
a
push
down.
The
vertex
filter,
push
down
the
edge
filter
push
down,
and
this
is
the
bug
fix
that
I
put
it
because
I
contributed
this
so
yeah,
and
this
is
the
optimization
rule.
That's
removed
a
new
slice,
actual
project
when
the
rule
is
applied,
so
it's
kind
of
helping
the
performance,
and
this
one
is
more
complicated
that
I
couldn't
understand
the
details.
B
So
if
you
are
interested
you
can
check
out,
and
the
final
one
is
is
actually
just
a
documentation
that
one
of
our
old
contributors
provided
this,
because
we
noticed
a
lot
of
users
were
not
tuned
their
operating
system
in
in
the
best
practice.
So
he
contributed
this
guide
with
a
a
tune
d
configuration
so
tund
is
actually
a
linux
utility
binary
package
that
you
can
easily
make
some
tunings
on
top
of
operating
system.
So
his
guide
is
quite
easy.
B
So,
with
the
configuration
up
on
the
tune,
the
after
box,
you
can
run
one
or
two
commands
to
have
your
operating
system
in
a
better
starting
point
so
and
that's
more
compatible
with
with
the
navel
graph
and
that's
all
of
the
core
part
and
regarding
the
other
surrounding
two
chains,
one
of
them
is
the
nebula
agent,
so
nabla
agent
words
a
a
a
stand-alone
legend.
That's
helping
some
of
the
initially
we
bring
this
to
nebula
because
we
want
to
make
our
backup
and
restore
working
better.
B
B
B
So
finally,
previously
in
because
this
is
not
a
officially
sponsored
repo
by
resolved,
so
we
didn't
make
it
quite
polished,
so
I
I
sometimes
contribute
on
it
in
my
free
time,
and
now
the
publish
can
be
done
automatically
as
well.
Previously
I
do
it
manually,
which
is
ugly
work.
The
final
one
is
it's
a.
This
is
a
working
progress.
B
Pr
that's,
which
is,
as
I
consider
worthy
to
be
shared,
that
this
is
related
to
when
you
want
to
generate
sst
files,
so
that
will
help
you
to
generate
the
underlying
file
format
of
the
nebula.
So
you
can
do
t
plus
one
huge
amount
of
data,
for
example
user
like
ovchat,
so
they
want
to
make
about
a
lot
of
data
every
day.
So
in
that
time
you
can
choose
to
use
sst
and
site
load
the
information
in
your
nabla
graph.
B
B
We
didn't
use
the
vanilla
one
and
when
we
are
when
we
are
creating
the
exchange,
we
we
don't
have
this
context,
so
we
use
the
manila
one
so
in
a
corner
rare
case,
so
it's
there
like
for
a
year.
Finally,
some
of
our
users
hate
this
rare
condition
and
we
figure
out
and
one
of
our
contribution
contributors
fix
it
in
spr
to
use
the
exactly
same
behavior
when
it's
doing
the
hashing
in
the
core
part.
So
it's
quite
interesting
and
finally
I
would
like
to
share
my
ad
hot
topic
today.
B
So
it's
related
to
some
of
the
work
I
have
done
around
the
data
lineage
and
the
metadata
management.
So
I
will,
I
will
introduce
the
amazon:
is
another
open
source
project
to
our
community
and
some
background.
So
what?
What
is
a
mental
data
governance
system
and
what
is
data
lineage?
So
from
my
understanding,
I
think
most
of
you
already
have
a
sense
on
that.
B
You
you
don't
know
when
you
can
change
it
or
you,
you
can
delete
it
or
you.
You
can
do
some
work
on
top
of
that
right
and
also
you
need
to
make
everything
in
a
single
source
of
shoes
too.
Everyone
can
query
and
check
it
instead
of,
like
everyone,
have
to
fan
out
a
a
large
email
loop
or
check
how
ping
everyone
in
in
the
stack
group
to
see
hey
is
every
anyone
know
something
about
this
table
and
this
oops.
B
B
So
that's
that
will
waste
a
lot
of
communication
hi
eric
welcome
back,
so
that
will
waste
a
lot
of
efforts
in
communication.
So
what
way
is
data
lineage?
So
this
is
an
even
more
fancy
word
right.
So
it's
more
about
the
traceability
of
your
your
data,
so
the
data
and
information
flows
in
all
life
cycle
from
how
it
was
generated
from
a
raw
data
and
it
can
be
consumed
by
other
application
service
or
regenerated
by
another
table.
B
Another
view,
and
there
are
there
will
be
a
batch
or
streaming
pipelines
to
you
know
talk
to
the
data,
so
everything
should
be
controlled
precisely
to
have.
You
know
a
guaranteed
the
performance
right,
and
why
do
we
need
that?
I
think
the
what
part
already
explained
the
the
reason,
because
data
grows,
and
even
we
have
only
two
people
in
a
small
team,
so
you're
from
day
zero.
You
you,
everyone
know
who
is
in
charge
of
which
table
and
you
have
no
all
the
history
in
your
mind.
B
So,
instead
of
putting
everything
in
the
outdated
non,
you
know
lots
of
sync
wiki
pages
or
you
occasionally
add
some
comments
in
your
database
when
some
of
you
change
schema.
So
it's
it's
not
a
trust
data
working
environment,
so
we
need
to.
Ideally
we
need
to
bring
a
single
source
of
shoes
off
all
the
metadata
and
including
the
data
lineage.
B
So,
and
that's
why
I
I'm
trying
to
make
this
reference
projects-
and
this
is
my
one
of
my
opinion,
because
I
I
was
withdrawing
witnessing
a
lot
of
community
users
that
so
when
they
are
asking
about
a
query-
and
I
noticed
their
query
is
about
data
lineage.
So
I
realized
a
lot
of
people
were
doing
recreating
their
own
wheels
to
make
their
own
toolings
around
the
data
lineage.
B
So
I
want
to
help
so
maybe
I
can
leverage
some
of
example
schema
or
leverage
some
of
the
open
source
projects
so
and
and
make
some
steps
study
and
it
turned
out.
I
found
that
instead
of
creating
a
a
single
standalone
service
dedicated
for
data
linux,
we
should
instead
leverage
a
whole
metadata
management
system,
because
the
linux
is
actually
generated
from
the
metadata.
So
when
you're
extracting
it
you're
actually
talking
to
all
the
other
methods
of
the
your
data
lab
and
but
why
not
manage
them
together?
B
B
The
open
source
project
called
the
amazon,
so
it's
it's
initiated
by
a
lyft
data
team
and
it's
donated
to
linux
foundation
and
a
bunch
of
big
and
small
teams
are
using
them
and
they
are
contributing
to
the
the
project
and
it
has
a
very
great
data,
module
and
api
and
a
fancy
friend
a
front-end
interface,
of
course.
So
this
is
the
official
website
of
manson.
So
basically
you,
when
it
comes
to
the
front-end
interface,
you
can
have
this
single
source
of
portal.
B
You
can
search
starting
from
the
search
everything
is
indexed
like
and
when
you
are
searching
some
of
the
tables,
some
of
the
column
or
dashboard
or
pipeline,
the
all
the
metadata
are
listed
here
and
you
can
manipulate
on
top
of
them.
So
everyone
is
seeing
the
same
thing
and
there
are
some
statistics
of
all
your
data
owned
by
whom,
who
is
the
frequent
users?
You
can
tag
things
with
specific,
meaningful
tags,
something
like
that,
so
everything
is
connected
and
beautifully
controlled
in
the
system.
B
This
is
like
the
day,
one
module,
it's
quite
not
naive,
and
you
don't
have
too
many
things
that
they
already
support
in
in
now,
but
in
our
time,
but
but
I
think
it's
more
easier
for
us
to
new,
fresh
users
to
understand
so,
basically
they
they.
I
hope
you
can
see
it
clearly,
so
they
they
module.
The
components
like
database
and
database
has
a
cluster
cluster
instance
can
have
a
couple
of
different
schema
and
the
table.
Your
table
instance
belongs
to
certain
schema.
B
You
can
tag
the
table
and
the
table
can
be
generated
of
from
a
air
air
table
dac
and
you
you
will
have
every
single
column
managed
separately
in
the
system
and
you
have
a
user
user,
can
read
something
and
can
own
the
table
and
or
other
components.
You
can
have
the
description
separately.
There
are
a
lot
more
of
them
nowadays
in
medicine,
but
just
this
is
just
a
reference
for
you
to
make
sense
of
how
it
looks
like
leveraging
the
graph
capability
so
yeah.
B
B
One
is
in
charge
of
extracting
or
ingestion
all
the
those
metadata,
including
the
data
in
lineage
things
from
different
sources,
so
they
can
be
different.
The
databases,
data
warehouse
or
some
other
utility.
I
can
show
you
more,
but
this
they
don't
don't
draw
too
much
types
of
sources,
but
there
are
more
of
them
and
they
have
a
bunch
of
data
builder
things
to
support
the
mainstream
data
sources
after
both
and
and
you
can
even
custom
them
quite
easily-
they
abstract
them
beautifully
and
underlying
they.
They
persist.
B
Data
in
single
software
choose
in
a
graph
database
and
they
leverage
elastic
search
for
the
full
text
search
and
the
the
other
part
is
the
the
service
part.
So
you
can
either
query
is
melody.
Api
underline
is
calling
the
graph
database
and,
of
course
you
can
search
or
you
can
do
it
from
the
front
hand.
I
will
demo
you
their
front
hand
gui
service.
So,
yes,
here
is
the
demo,
so
this
is
actually
a
mansion
running
on
top
of
elasticsearch
and
tableau
graph.
B
So
you
can
search
here,
for
example,
users
test,
and
you
have
about
a
bunch
of
different
resources.
You
can
bookmark
it
here,
so
it
have
a
lot
of
different,
useful
information
here.
This
is
the
owner,
the
quality
related
information.
You
can
ingest
and
integrate
with
the
quality
data
quality
service.
Here-
and
this
is
the
metadata,
so
you
have
everything
here
and
you
can
read
or
write
on
top
of
that
you
can
tag
your
your
metadata
and
this
is
the
data
lineage.
B
So
there
are,
there
is
one
upstream
of
that
table,
so
this
is
jumping
to
its
upstream
and
you
can
you
can
see
all
the
data
lineage
in
the
graph
view.
So
it's
quite
interesting,
but
there
are
more
than
more
of
that.
I
just
review
some
of
his
cool
features
and
but
it's
on
top
of
nabla
graph,
so
what
we
can
do
else
with
the
nabla
graph
right.
B
So
I
think
this
is
maybe
one
of
the
reasons
why
a
mansion
choose
to
put
data
only
on
data,
a
graph
database,
because
most
of
the
graphics
database
by
nature
will
offer
you
explore
a
data
visualization
tour.
So
the
meaning
here
is
everything
you
want
to
explore,
or
you
want
to
get
inside
from
in
in
your
metadata
service.
It's
a
it's
actually
a
web
service
right.
B
You
have
to
define
your
new
api
or
new
gui,
but
when
you
are
running
data
already
on
graph
database
with
the
something
like
nebulograph
studio,
you
can
have
another
console
visually
to
get
your
metadata
or
lineage
inside
just
to
click
your
your
mouse,
so
you
can
optionally
imagine
you
are
leveraging
never
graph
animation.
You
can
optionally
expose
your
studio
for
to
the
people
who
knows
what
they
are
doing
so
they
can
have
those
those
metadata
inside
on
their
own
in
this
beautiful
way.
B
It
is
quite
and
plus
on.
On
top
of
this,
you
can
also
expose
your
graph
query
interface
to
them.
So
it's
like
your
dsl
for
your
metadata
lineage
system.
You
don't
have
to
write
your
code
to
bring
your
new
apis.
You
just
expose
your
graph
database
interface.
Maybe
you
limited
to
a
read
permission,
so
that
will
be
quite
beautiful.
I'm
not
going
to
walk
through
everything
I
recorded
already,
but
basically
you
can
query
the
metadata
in
in
in
your
visualization,
providing
from
your
graph
database
like
this.
B
So
I
think
it's
cool
and
what's
underneath
so
actually
underneath
they're,
just
a
bunch
of
graph
queries
right.
So
this
is
the
code
from
the
amazon,
and
this
is
the
nebula
proxy
that
I
contributed
to
enable
the
capability
of
nebula
as
a
back
end.
So
when
they
are
calling
execute
table
query
so
underline,
is
this
giant
match
query
and
if
you
change
the
code
into
actual
query,
you
can
actually
make
it
towards
the
level
graph.
So
I'm
proving
I'm
not
cheating.
So
you
can
see
this.
B
This
is
the
beautiful
part
of
the
graph
database.
So
in
the
single
query,
you
are
fetching
a
bunch
of
information
in
one
go
and
in
no
time
it's
quite
fast.
You
actually
query
a
table
and
with
the
id
of
this
this
vid
and
optionally,
you
are
already
matching
another
10
different
patterns
in
this
one
go
and
finally,
you
want
to
process
your
return
values
in
in
in
a
format
that
your
api
would
like
to
see.
B
Finally,
so
it's
it's
beautifully
done
with
this
pattern
and
yeah
so
enable
to
help
to
introduce
a
mention
to
our
community
better.
I
actually
created
this
reference
project
and
this
is
the
article
describing
that
so
in
this
project,
an
article
I
I
want
to
dive
into
details,
but
I
want
to
give
you
a
birth
view
of
what
I
have
provided
you.
You
can
do
it
very
easier,
so
I
enable
to
have
everything
connected
in
a
detailed
way.
B
I
actually
set
up
a
whole
data
stack,
so
I'm
I'm
a
newbie
to
data
stack.
So
please
don't
don't
love
eric.
So
maybe
it's
quite
naive.
I
actually
bring
a
dummy
data
base.
I
I
choose
to
use
postgres
as
your
your
data
warehouse,
your
database
in
your
in
our
lab,
and
I
bring
a
data
data,
ops
utility
with
the
etl,
and
I
also
I
I
put
a
bi
tour
into
this
stack
and
also
a
job
orchestration,
which
is
the
air
air
flow
and,
finally,
the
amazing.
B
So
in
this
in
this
project,
I
I
put
everything
runnable,
so
you
can
just
copy
and
paste
and
run
everything
on
your
own
docker
environment.
So,
firstly,
you
will.
You
can
run
this
this
chapter
to
have
your
postgres
in
the
container
and
you
will
have
some
dummy
data
inside
and
then
you
will
run.
I
use
the
mail
tunnel,
which
is
a
beautiful,
open
source
project,
leveraging
the
singer
for
its
extraction
and
the
loading
and
the
use
dbt
for
the
transform.
So
I
I
make
this
connected
to
help.
B
You
generate
some
data
source
from
csv
and
to
to
your
database
and
and
make
some
transform
of
your
existing
table
in
the
first
place
in
the
first
step
too,
so
you
will
have
relatively
toy
data
set.
So
then
I
I
bring
the
apache
superset
to
this
stack
and
you
and
I
even
create
a
bunch
of
very
naive
chart
to
become
a
a
a
dashboard.
B
So
it
looks
very
ugly,
but
it's
a
dashboard
anyway
and
finally,
I
bring
the
amazon
together
with
elastic
and
the
nebula
you
just
one
go.
Everything
is
up
in
the
dock
compose
and
I
give
some
examples
with
the
data
builder
to
load
some
dummy
data
in
the
in
our
system.
So
you
can
click
and
check
the
everything
inside
and
finally,
I
I'm
I
create
a
runnable
examples
on
how
you
actually
make
every
source
of
the
stack
as
your
metadata
source
and
I
create
the
example
of
the
running
jobs.
B
So
you
can
make
everything
from
the
postgres
the
dbt
pipeline
and
your
even
dashboard,
so
they
every
component
inside
the
metadata
component,
can
be
extracted
in
my
reference
projects.
So
then
I
bring
the
superset
at
the
pre
preview
of
the
data
of
the
medicine.
And
finally,
here
is
the
final
example
to
help
you
extract
the
lineage
information
from
the
dbt
manufacts.
So
it's
a
genius
combination.
B
So
if
you
have
the
luck
to
leveraging
dbt
for
etl,
so
you
don't
have
to
write
your
own
extracting
rules
to
you
can
just
put
together
everything
you
can
extract
the
lineage
to
your
system.
Of
course
you
can
write
your
own
one,
so
it's
quite
frankly,
it's
quite
easy
because
they
have
a
beautiful
abstraction
on
doing
that,
and
so
there
are
a
lot
of
details
in
this
article.
It's
on
my
website
and
the
nebular
official
blogs.
B
And
finally,
let's
do
a
recap
of
this
topic,
so
we
with
the
month
I
mentioned
on
double
graph,
so
we
don't
have
to
struggling
or
put
a
lot
of
effort
recreating
the
wheels.
So
you
just
need
to
connect
with
it.
There
are
a
bunch
of
extractors
you
can.
Even
you
can
bring
most
of
your
data
components
in
your
lab,
your
data
center
into
a
mansion,
and
you
can
extract
the
lineage
with
dbt
and
everything
is,
is
running
on
on
nabla
graph.
B
B
B
In
other
words,
you
can
have
the
beautiful
interface
with
a
men's
and
front
hand,
and
everything
is
connected.
So
that's
all
of
these
reference
projects
so
and
future
work,
so
actually
the
effort
on
bringing
nabla
graph
to
a
mansion.
It's
still
working
play
progress.
I
make
the
pork
and
even
passed
all
the
ut
code
and
the
the
pr
is
like
ten
thousand
of
lines,
but
still
is
the
rifc
is
still
under
discussion
and
review
of
the
manson
community
so
and
in
future
I
want
to
after
it's
merged.
B
B
So
maybe
in
the
future
we
can
leverage
the
serverless
capabilities
or
for,
for
example,
from
the
native
or
dapper.
You
can
do
some
event
driven
thing,
so
maybe
I
can
make
some
example
to
like
you
can
like
add
a
web
hook
when
you
change
something
the
web
will
will
call
a
function
and
the
function
will
help.
You
re
extract
some
metadata,
so
that's
doable
and,
finally
that
when
I
was
doing
this
work,
so
amazing
is
python
based.
B
C
I
am
I'm
really
impressed
by
this.
This
is
fabulous.
I
was
actually
looking
at
this
from
the
direction
from
from
a
couple
different
directions.
I
was
researching
emerson
independently
of
of
my
exploration
of
so
I'm
feeling
like
I
hit
the
jackpot
here.
This
is
beautiful.
Oh
really,
thank
you
cool.
Thank
you.