►
From YouTube: NebulaGraph (virtual) meetup: New Release & Use Cases
Description
In this online meetup, three experts in the industry will introduce what is new in NebulaGraph v2.5.0, how it is used for risk control in fintech companies, and how RocksDB is Used in the Largest Social Network. Welcome to discuss on our Slack: nebulagraph.slack.com
0:00- Intro
====================================
0:02:17- What's new in Nebula Graph v2.5.0
0:14:51- Benchmark 2.5.0 vs 2.0.1
0:16:14- Ecosystem tools overview
0:21:15- Release mechanism for future version
====================================
0:24:40- How Akulaku Uses NebulaGraph for Intelligent Risk Control
0:28:04- Brief introduction to Graph and related use cases
0:29:43- Why NebulaGraph?
0:36:25- Graph processing platform on top of Graph DB
0:41:10- NebulaGraph use cases in Akulaku
0:44:20- Training and deployment of Graph models
0:52:19- Summary
====================================
0:53:27- How RocksDB is Used in the Largest Social Network
0:55:03- RocksDB introduction
1:00:02- Methodology and tool set
1:01:51- RocksDB production uses cases
1:11:25- Why disaggregated storage?
1:13:00- Long-term goal to RocksDB
1:14:16- Challenge of disaggregated storage
1:17:48- Q & A
A
A
B
I
can
see
your
screen.
Okay,
thank
you.
So
thank
you,
everyone
good
afternoon
and
good
morning.
So
this
is
way
google
from
vsoft,
and
I
will
I'm
very
in
the
head
of
developer
developer
advocate
so
in
future.
For
anything,
you
consider
needing
help
just
reach
out
me
from
the
slack
the
forum
or
twitter.
B
B
This
this
mechanism
was
targeted
to
reduce
the
possibility
of
the
om
when
a
large
amount
of
ram
is
consumed
during
the
query,
but
there
are
some
so
it's
actually
kind
of
personally
implemented.
So
for
now
the
current
implementation
is
not
doesn't
fit
the
case
when
the
gravity
is
running
in
the
environment
of
container
container.
B
B
B
The
next
one
is
finding
path.
Now
we
can
finally
support
filtering
the
edge
properties
during
a
find
path.
Query.
So,
as
you
can
see
here
in
in
the
where
clause
we
can
filter
based
on
the
edge
properties
conditions
here,
but
the
still
we
cannot
support
the
detect
the
vertex
properties
filtering.
So
you
can.
B
Next,
one
is
the
subgraph.
For
now
previously
our
subgraph
will
fetch
all
everything,
including
the
the
edges,
the
vertexes
and
all
the
properties
by
default,
without
options
to
disable.
So
now
we
bring
this
option
to
the
user.
So,
with
the
new
parameter
during
the
subgraph
query,
you
can
specify
you
have
to
specify
on
prop.
It
literally
means
you
would
like
to
fetch
all
the
properties,
but
without
this
specification
only
the
topology
of
the
graph
will
be
fetched.
That
is
only
the
vertex
and
edges.
B
B
B
B
B
You
can
now
check
whether
key
a
exists
on
this
variable
to
find
the
last
one
is
the
the
aggregation
function
previously
when
you
query
something
like
return,
the
aggregation
of
the
count
of
of
v
plus
the
average
of
v
so
except
it
should
be
supported,
but
previously
it
will
end
up
with
the
arrow
of
we
don't
support
a
nested
aggregation.
We
know
it's
not
nested,
so
now
it's
fixed,
so
there
are
some
other
enhancements.
That's
not
impacting
our
user
space,
but
it's
somehow
optimal.
B
The
first
one
is
now
the
storage
engine
supports
the
execution
of
the
query
in
a
concurrent
way.
The
background
of
this
change
is
that
previously
in
the
legacy
version
we
actually
we
have
period
that
support
this
kind
of
execution
that
you,
you
don't
run
your
query
in
single
process,
but
it
turned
out.
B
We
thought
and
we
observed
that
it's
not
performance
in
the
oltp
situation.
So
when
you
have
high
concurrency,
the
single
process
performs
better.
So
we
disable
that.
But
in
this
release
we
optionally
introduce
the
possibility
to
user
by
default
is
config
disabled,
but
you
can
enable
it
so
in
in
some
cases
that
it's
not
in
high
concurrency
typical
oltp
situation.
You
don't
have
the
high
concurrency
case
in
that
case,
enable
a
concurrent,
concurrent
execution
of
single
query
will
help
on
the
performance.
B
B
So
I
so
in
some
in
some
cases,
when
you
have
index
multiple
indexes
for
certain
tags
or
edges
it,
it
now
can
consider
all
of
your
more
than
one
indexes
in
some
cases
when
it's
optimal
to
do
so
and
the
rest
two
are
not
impacting
the
users
basic
space
is
good
to
will,
but
it's
a
optimization
one
is
the
listener.
Interface
now
support
full
data
acquisition,
and
next
one
is
the
leader
table
in
menta
d
is
also
reflected.
B
The
last
one
is
the
incapability,
behavior
change,
as
mentioned
before.
Now
you
can
you
have
to
specify
this
prop
this
option
to
get
the
properties
when
you're
fetching
the
sub
graph.
So
this
is.
This
is
a
kind
of
incapability,
because
your
old
query,
fashion,
subgraph
by
default,
the
same
query-
will
not
have
the
properties
being
fetched.
So
this
is
the
major
the
only
major
in
capability
between
2.0
and
2.5.
B
B
For
example,
like
the
exchange
our
tooling,
to
have
to
support
multiple
data
sources
importing
data
to
nebula,
we
have
some
new
data
sources
introduced.
Please
refer
to
the
documentation
and
the
github
repo,
and
also
one
of
the
features
of
exchange
like
the
sst
is
general
available
to
the
core
in
version
2.
Now,
in
this
release,.
B
So,
like
the
clients,
we
didn't
put
the
rust
here,
but
still
we
have
the
repo.
If
you
would
like
to
use
it
just
check
out
our
you
know,
repo
and
regarding
the
visualization
tooling.
Previously
we
only
have
one
tooling
named
nebula
studio
and
now,
with
this
release,
we
come
to
the
version
3.0
and
the
the
other.
Exciting
news
is
that
we
finally
reopen
source
this
tool,
so
just
feel
free
to
check
out
github
repo
and
it
comes
with
same
license
of
the
core
and
another
tooling.
B
That
may
be
strange
to
you,
but
you
may
already
know
that
are
the
explorer
and
dashboard
so
explorer
is
yet
another
data
explorer
tour
which
is
similar,
similar
architecture
of
the
studio.
It's
not
yet
open
source,
but
it's
working.
It
will
focus
on
the
area,
that's
targeting
to
the
end
user,
mostly
on
the
the
graph
expert
or
data
scientist.
So
we
will.
We
bring
a
more
next
level
power
for
the
graph
explorations
on
this
tour
and
I
will
introduce
a
bit
on
that
in
the
following
slices.
B
B
Just
let
you
know,
and
another
interesting
thing
is:
we
introduced,
we
released
a
new
service
called
playground.
So
it's
it's
basically
a
pre-loaded
different
dataset
cluster
that
without
authentication
you
can
use
it.
With
your
browser
connected
your
modern
browser
data
to
internet,
you
can
check
it
from
our
documentation
and
our
our
nebular
website.
There
is
entry
there.
B
So
I
will
give
a
brief
introduction
a
review
of
the
explorer
the
new
touring
you
can
also
play.
Have
we
have
a
playground
similar
to
the
big
one
that
but
that's
based
on
studio.
You
can
play
with
the
explorer
here
so
now
we
introduce
some
next
level
capability
abilities
for
the
user
to
explore
the
graph.
Now
it
looks
awesome
right
so
the
next
one
is
dashboard.
You
you
can
see
we
can
now.
B
B
B
So
basically
the
release
will
be,
at
least
in
the
the
medium
level.
So
we
will
have
the
major
point,
media
point
minor
point
and
in
the
minor
changes
normally
we
don't
break
the
capability
on
the
surrounding
two
or
client
level.
But
when
it
comes
to
the
media
change
you
will
have
to
upgrade
the
clients
and
the
major
level
will
be
a
big
difference
and
in
every
release
of
the
core
part.
After
two
weeks.
B
The
last
one
is,
we
come
back
to
the
repositories
structure,
so
actually
we
are
some
some
of
you.
We
already
know
that
in
the
legacy
in
the
version
one
we
are
using
the
model
repository
only
a
single
ripple
of
the
nebula
ripple
and
we
break
them
down
into
three
separate
some
modules
in
since
version
two
and
until
the
2.5
is,
is
keeps
so,
but
in
the
future
release
like
the
next
cycle,
the
master
we
bring.
We
change
back
to
the
legacy
weight
we
use.
B
A
A
C
So
hello,
everyone
can
you
see
my
screen?
Yep
yep,
yeah,
okay,
and
here
we
start
so
hello,
everyone.
So
today
I
I'm
a
programming
engineer
from
akulaku.
So
today
I'm
going
to
talk
about
how
akula
could
use
nebulograph
for
intelligent
risk
control.
So
first
I
will
have
a
brief
introduction
of
what
we
do.
We
are
internet
finance
company,
an
e-commerce
company.
We
are
leading
player
of
internet
finance
in
southeast
asia,
and
so
basically
we
so
this
is
what
we
do.
We
meet.
Our
major
product
is
an
installment
app.
C
C
Installment
means
when
you
pay,
when
you
buy
something
on
the
internet,
if
you
can't
pay
the
full
bill,
you
can
only
you
can
choose
to
pay
part
of
it
and
you
can
choose
to
the
rest
of
your
bill
to
be
paid
by
our
company,
but
you
have
to
pay
the
interest
and
we
actually
make
money
from
the
interest,
and
that
is
where
our
benefit
from
so
for
a
finite.
We
are
an
internet
company
and,
at
the
same
time,
a
finance
company
for
finance
company.
There
are
two
types
of
risks
for
for
our
customers.
C
We
send
out
bills
once
a
month
and
we
will
encounter
two
types
of
risk
and
two
types
of
customers
for
the
first
type.
They
refuse
to
pay
the
first
bill
and
for
this
kind
of
risk
we
call
it
fraud
because
they
refuse
to
pay
the
bill
on
purpose
and
the
second
second
kind
of
risk
referred
to
the
customers
who
refuse
to
pay
the
last
bill
or
the
third
or
the
fourth,
for
example
the
third
or
the
fourth
bill
in
the
whole
order,
and
that
kind
of
risk
refers
to
the
credit
risk.
C
And
in
today's
conversation
I
will
first
briefly
have
an
introduction
to
graph
and
the
related
use
cases,
and
then
I
will
talk
about
the
choice
of
graph
database
in
our
company
and
then
I
will
talk
about
the
graph
processing
platform
we
built
on
top
of
the
graph
database
and
fourth,
I've
discussed
the
nebula
use
case
in
akulaku,
our
company
and
fifth.
I
will
talk
about
training
and
deploying
graph
models
on
top
of
graph
database.
What
are
the
basic
considerations?
C
C
C
C
So
visualization
is
a
very
important
part
in
our
everyday
usage.
The
second
use
case
is
that
we
can
use
graph
queries
to
implement
complex
queries
for
some
problems,
although
in
the
first
place
they
may
not
seem
to
be
related
to
graph.
We
can
still
translate
these
problems
into
graph
settings
and
implement
the
queries
in
graph
pirates.
In
that
way,
it
may
help
us
to
understand
the
problem
and
to
maintain
the
program.
C
So
for
the
choice
of
graph
database,
actually,
we
have
tried
quite
a
lot
and
we
we
have
actually
tried
three
different
graph
database.
Each
of
them
failed
for
one
or
more
reasons
and
actually
the
reason
they
fail
actually
highlight
the
key
considerations
when
we
are
charging
when
we
are
choosing
graph
database.
C
So
the
first
consideration
is
scalability
and
the
the
first
database
to
to
be
ruled
out
of
the
game
is
neo4j
because
for
the
for
the
community
version
it
only
supports
single
node
deployment
and
the
second
consideration
is
data
import,
which
means
when
we
import
the
graph
data
grab
data.
For
the
first
time,
the
time
of
importing
the
data
must
be
low,
and
the
counter
example
here
is
d
graph,
so
we
tested
d
graph
d
graph
is
actually
a
very
interesting
wrap.
C
So
when
we
test
the
database,
we
can't
even
start
our
test,
because
our
graph
data
is
so
large
that
the
data
import
program
for
dgraph
fails
it.
Never.
It
never
finished,
not
even
slow,
it
never
finished,
and
the
third
consideration
is
of
course,
efficiency,
query
efficiency,
and
that
is
it.
It
also
highlights
the
reason
why
we
choose
graph
database,
because
graph
database
are
optimized
for
certain
types
of
pirates
and
the
counter
example
here
is
general
janus
graph,
so
we
tested
jenna's
graph.
C
The
reason
why
we
chose
it
is
that
it
has.
It
is
kind
of
appealing
to
us
because
it
has
big
data
back
ends.
The
back
end
can
either
be
hbase
or
cassandra
that
gives
us
a
gift.
Give
us
the
impressions
that
well
genescrub
might
be
able
to
hold
the
graph
of
the
huge
graph
data
set
we
have,
and
actually
it
did
it
was.
It
was
able
to
hold
the
data
set,
but
that
is
all
it
can.
C
It
could
do
it
can
hold
the
data
set,
but
you
can't
query
because
the
query,
even
the
pirates,
are
so
slow,
even
two
hop
or
free
hop,
quiet,
and
that
is
a
nightmare,
and
that
is
and
because
of
that,
after
some
experiments
we
are
free
of
all.
Free
database
fail
and
we
come
to
the
network,
and
this
is
and
actually
because
of
what
we
have
suffered
before
in
all
those
database.
We
decided
to
have
a
complete
test
on
it
on
naval,
a
graph.
C
C
This
is
the
diagram
we
we
we
get
after
the
experiment.
We
can
see
that
the
throughput
of
the
of
nebulograph
is
actually
quite
nice.
It
reached
the
maximum
throughput
when
we
have
12
machines
and
at
the
concurrency
of
roughly
100
we
can
reach
a
qps
of
5000,
which
means
5000
batches
of
pirate
per
second,
which
is
well
quite
nice
and
satisfied
our
needs,
so
we
decided
to
well.
Maybe
we
give
a
try
to
this
day
to
describe
database.
C
C
C
And
early
early
july,
I
guess:
yes,
we
have
version
2.5
and
we
decided
once
we
have
the
new
version.
We
decided
to
have
the
same
test
before
we
were
before.
We
actually
start
using
it
and
we
have
basically
had
the
same
setting.
But
we
now
we
have
don't
have
time
to
vary
the
number
of
machines.
So
we
choose
one
setting.
A
C
10
billion
edges
and
the
same
experiment
setup,
and
this
diagram
shows
what
we
have
so
now,
I'm
going
to
talk
about
the
architecture
of
our
graph
platform,
so
only
a
single
graph
database
is
not
it's
not
enough
for
satisfying
our
needs,
so
we
build
a
platform
on
top
of
that,
and
there
is
another
point
I
want
to
highlight
to
satisfy.
Our
need
in
using
graph
processing
and
graph
computations
actually
graph
database
is,
is
not
the
only
option
for
some
use
cases
grab.
Database
is
particularly
suitable,
but
for
other
cases
it's
not
grab.
C
Database
are
especially
suitable
for
cases
when
you
want
to
extract
a
local
sapra,
so
example.
Applications
include
and
hop
features,
deployment
of
ground,
neural
networks
and
label
propagations,
but
there
are
also
some
other
cases
where
still
you
use
you,
you
need
to
use
graph
algorithms
and
graph
implementations,
but
using,
but
you
don't
need
to
use
a
prop
database,
for
example,
for
the
real-time
community
sections
and
a
normally
detections
for
these
application
scenarios.
We
design
our
own
graph
computation
engine,
and
here
is
the
architecture
wheel
of
the
whole
platform.
C
We
have
the
applications
with.
I,
which
I
described
just
now,
and
we
have
platforms
to
support
different
kinds
of
applications
based
on
to
support
the
availability
of
platform.
We
have
different
infrastructures
so
graph
database
and
we
we
also
need
the
help
of
different
big
data
clusters,
like
spark
like
hives,
like
flink
and
hbase.
C
Basically,
you
we
need
to
build
several
data
youtube
around
the
graph
database
and
we
do
need
to
consider
several
issues
when
we,
when
we
are
setting
up
our
graph
database
and
graph
database
clusters.
So
basically
we
have
two
different
modes
of
deploying
the
database.
The
first
most
is
that
we
have
a
cluster
and
we
have
actually
have.
We
have
two
clusters.
C
C
So
as
a
whole,
we're
actually
ensuring
the
high
availability
of
the
production
class
clusters,
which
you
see
on
the
left
and
to
ensure
the
data
consistency
and
for
the
convenience
of
initializing
the
data.
We
need
the
data
production
layer.
As
you
see
on
the
third
layer,
we
need
support
from
height
for
batch
mode,
a
batch
input,
so
we
digest
data
from
hive
and
write
it
to
nebula
using
spark
and
for
real
time
input.
C
C
C
So
when
you
detect
a
suspicious
group
of
fraud,
you
need
to
figure
out
what
is
actually
happening
and
graph
is
a
is
a
tool
that
is
particularly
suitable
for
this
case,
and
the
second
use
case
is
the
calculation
of
device
id.
We
need
to
consider
the
connection
of
features
and,
lastly,
I
will
talk
about
deploying
rap
models
on
top
of
graph
database,
like
label
propagations,
etc.
C
C
C
So,
in
our
case,
when
we
are
dividing
determining
the
id
of
device
device,
we
need
to
consider
the
surrounding
features.
The
features
that
are
related
to
the
devices
and
the
problem
is
problem
here-
is
that
different
devices
may
share
different
features
and
when,
when
we
are
translating
our
rules
of
determining
device
id,
we
come
to
the
conclusion
that
well,
maybe
modeling
features
and
devices
as
node
in
the
graph
will
make
it
easier
to
maintain
and
that's
what
we
did.
We
modeled
what
we
are
going
to
do
in
the
graph
you
see
here.
C
We
model
device
and
related
features
as
nodes,
and
we
implement
our
rules
of
determining
determining
device
ids
as
graphqls.
In
that
way,
you
make
it
easier
to
understand
and
maintain,
but
there
is
a
major
problem
here.
We
need
to
ensure
consistency
when
we
are
doing
update
device
id
update
in
parallel
and
to
do
so,
we
introduce
logs
so
logs
when
we
are
calculating
a
certain
part,
calculating
a
device
id
based
on
a
certain
part
of
the
graph.
C
It
is
the
models
that
based
on
subgraph
extractions,
so
we
start
from
a
node
in
a
center
and
we
extract
a
subgraph
centered
at
a
node.
It
includes
the
first
type
of
models
include
no
related
subgraph
features,
for
example
the
suspicious
nodes,
which,
which
are
three
steps
or
two
steps
away,
and
second
is
label
propagations.
C
So
you
have
some
labels,
for
example,
wrist
labels
or
a
good
labels
in,
and
we
attach
those
labels
to
the
nodes
and
we
propagate
those
labels.
According
to
some
rules
and,
lastly,
and
most
importantly,
the
ground
neural
networks
to
to
deploy
ground
neural
networks
in
real
time
or
nearly
real
time,
we
do
need
graph
database
and
all
these
models
rely
on
subgraph
extractions
using
graph
database,
and
the
difficult
part
here
is
that
the
deployment
usually
require
high
of
a
reliability
of
the
database
and
the
execution
time
must
be
very
small.
C
C
C
C
The
first
example
is
subgraph
extraction
during
application,
so
here
by
by
writing
application-
and
we
mean
that
the
customer
are
common.
The
customers
are
coming
to
our
company
applying
for
quota
limit,
which
means
to
apply
for
the
amount
of
money
they
can
borrow
from
our
company,
and
in
this
scenario
we
need
to
calculate
the
subgraph
features
and
the
required
execution
time
requirement
is
actually
not
very
strict.
C
As
long
as
you
can
finish,
the
whole
process
in
one
minute
is
fine
and
second,
the
size
of
the
subgraph
could
be
large,
because
sampling
is
not
allowed
when
we
are
running
the
model
and
third,
the
change
of
graph,
which
means
the
the
deletions
and
additions
of
edges
in
a
graph.
The
number
of
deletions
and
additions
are
significantly
larger
than
the
number
of
cores
of
models,
so
we
decided
that
we
deploy
the
mode.
C
C
C
So
when
you
so
when
you
calculate
the
score,
we
write
it
to
a
database
and
we
read
the
score
online.
We
read
it
from
this
database.
The
same
database
instance
and
the
database
in
instance,
serves
as
a
barrier
to
separate
online
course
and
a
real-time
computation
of
labor
propagation,
and
the
third
case
is
a
similar
case.
That
is
the
ground
neural
network.
C
But
we
have
a
more
difficult
situation,
because
the
execution
time
of
grant
neutral
network
models
is
usually
significantly
higher
than
label
propagation
and
and
it's
also
more
complex
and
and
also
we
have
the
same
requirements
for
for
the
business.
For
from
our
business
side,
we
have
very
strict
limit
of
execution
time
within
200
milliseconds
for
permit
with
200
milliseconds
and
the
size
of
graph
could
be
large
because
sampling
is
still
not
allowed,
and
we
also
have
the
same
problem
set
the
same
configuration.
C
C
In
this
way,
we
ensure
that
if
there
is
any
delay
in
the
calculation,
we
will
not
influence
the
overall
decision
process
in
production
and
that
delay
could
be
catastrophic
and
also
we
have.
We
have
different
some
other
options.
For
example,
we
can
use
sampling
when
we
are
extracting
a
subgraph,
but
that
should
be
approved
by
the
business
guy,
because
we
produce
model
to
our
anti-fraud
strategy.
C
For
our
friends
and
they
have
to
decide
whether
they
they
are
going
to
use
our
models
or
not
so
lastly,
I.
Finally,
I
will
give
a
brief
summary.
So,
what's
cool
about
nabla
the
first
and
the
most
important
part
of
the
issue
is
that
it's
super
fast
in
data
import,
which
is
much
faster,
that
all
the
other
graph
database
we
tested.
A
D
Okay,
so
thanks
for
doing
the
meetup
invitation,
I'm
churchill
from
facebook
and
I'm
working
on
the
roxdb
team,
so
we
know
that
nebular
use
rustdb
as
its
basic
storage
engine.
So
after
we
listen
to
the
two
presentations
about
the
graph
store
itself
and
how
the
users
use
it,
and
now
we
can
dive
in
and
look
at
its
storage
engine.
Rustdb-
and
here
is
today's
agenda.
D
So
first
I
will
talk
about
how
we
collect
analyze
and
model
the
workloads
of
rustdb
and
facebook
and
it's
it
has
been
presented
as
a
paper
last
year
and
then
I'll
talk
about
the
current
trends
that
we
separate.
The
storage
and
the
compute
clusters
achieved
this
application
and
how
rough
db
is
customized
and
improved
for
those
new
architecture.
D
So,
let's
look
at
the
workload
part.
Maybe
I
can
just
use
several
minutes
to
introduce
about
roxdb,
so
we
know
that
nebula
use
rustdb
as
its
storage
engine
and
all
the
graph
vertices
and
edges
are
stored
as
a
key
valve
house
inside
the
rustdb
and
the
rocksdb
is
our
sm3
based
keyboard
store
and
when
we
do
the
insertion,
all
the
new
health
and
deletions
are
cached
in
the
map
table.
So
those
parts
and
after
the
template
is
full.
D
The
keyword
health
are
sorted
and
flushed
out
as
immutable
ssd
files,
and
they
are
mentioned
in
different
levels.
So
from
level
to
level
the
key
ranges
might
overlap
with
each
other,
but
in
the
same
level
there's
no
overlapping.
Some
new
ssd
files
will
be
merged
with
the
old
ssd
files
in
the
next
level
by
compaction
and
roxdb
supports
very
simple
key
value.
D
Interface
like
get
put
delete
its
return,
merge
and
other
more
advanced
interface
like
delete,
range
or
single
delete
and
the
roughdb
is
an
embedded
storage
engine
and
it
is
widely
used
at
facebook
and
in
other
big
companies.
So
now
I
will
I'll
introduce
a
three
major
use
case
at
facebook.
How
we
use
roxy
b
to
support
social
graph
and
other
data
stores,
so
the
first
use
case
is
udb.
It
is
actually
a
very
large,
maybe
the
largest
mysql
database.
D
So
usually
we
use
my
simple
table
index
number
as
a
prefix
of
the
key,
and
then
we
use
the
primary
key,
for
example
the
vertices
number,
and
also
the
the
other
ps
portion
of
the
raspberry
key
and
stores
other
information
as
a
value
part
and
for
the
social
graph.
We
know
facebook
maintains
the
world's
largest
social
graph
and
we
do
the
graph
partition.
You
can
stick
to
the
papers
about
this
partition
and
each
graph
partition
will
be
stored
in
one
message
table
so
there'll
be
some
interesting
localities
and
the.
D
The
second
use
case
is
zip
db,
so
zp
is
a
distributed.
Keyword
store
based
on
paxos,
so
vp
will
maintain
thousands
of
the
rough
db
instance
and,
for
example,
one
host
will
have
about
100
or
700
of
the
rust
leaking
sensors
and
we
have
the
primary
one
which
will
provide
the
strong
consistency
and
the
secondary
one
will
be
synced
after
that
it
is
pretty
simple.
This
is
no
translation
layer.
D
D
D
So
from
now
we
know
that
rusty
b
is
used
very
well,
including
we
know
natural
exclusives,
but
actually
it's
hard
to
understand
what
is
happening
inside
rustdb,
what's
the
workload
and
how
we
can
improve
the
performance
based
on
the
workloads.
So
we
introduced
a
set
of
methodology
and
tools
to
address
this
issue.
D
So
first
we
have
rossdb
and
we
have
deep
limitations
running
on
top
of
roxdb
and
then
usually
we
just
run
the
db
bench
or
other
benchmarks
and
compile
the
results
with
the
scenario
I
want
to
use
rustdb
and
then
we
can
tune
or
improve
it,
but
without
the
real
world
workloads.
Actually
it
is
pretty
hard
to
get
a
very
accurate
benchmarking.
D
So
what
we
did
is
we
introduced
tracing
framework
for
rustdb.
We
are
able
to
collect
the
trees
and
then
we
have
analyzer
to
generate
a
set
of
reports
from
the
tracing
files
and
based
on
those
characteristics
and
analysis
we
have.
We
can
use
different
mathematical
and
statistics
models
to
get
the
profile
of
the
workload.
And
finally,
we
can
use
this
model
to
rebuild
the
new
benchmark.
So
now
we
have
a
close
loop
of
this
process.
We
can
continue
to
improve
the
benchmark
and
the
better
benchmarking.
D
Moreover,
we
can
replace
a
trace
which
can
be
nearly
the
same
as
a
real
workload,
and
then
we
can
refine
the
model
and
then
improve
the
benchmark.
So
now
I
will
introduce
some
of
the
results
we
collect
from
facebook,
the
social
graph
and
other
use
cases,
and
for
more
information
you
can
see
to
our
papers.
D
So
the
first
question
we
want
to
answer
is:
what's
the
query
compositions
and
what's
the
reach
of
them,
so
we
analyze
them
and
get
the
following
results.
Actually
for
the
udp,
which
is
for
the
social
graph
and
zip
db,
they
have
a
lot
of
get
or
signal
point
lookup
currents
more
than
70
percent,
but
for
the
nuclear
x,
merge
is
heavily
used
because
for
the
uq
tracks,
which
is
like
user
features,
they
have
a
lot
of
updates.
D
When
we
update
a
keyword
panel,
usually
we
just
do
a
get
and
then
update
it
do
a
put,
but
it
will
waste
a
lot
of
time
when
we
do
the
read
so
merge
is
just
a
single
put
special
points.
We
just
store
the
delta
portion
and
what
we
need
to
look
at
in
the
future.
We
just
combine
all
the
merge
information
and
get
the
final
results
and
for
the
one
for
one
rock
db
instance,
we
could
have
different
column
families
and
you
can
see
even
inside
the
same
sim
db
different
column.
D
Families
can
have
different
current
compositions,
for
example,
in
the
udp
use
case.
Object
is
the
vertices
window
of
vertex
in
the
social
graph,
and
we
do
a
lot
of
point
lookup
and
some
of
the
iterators.
We
scan
the
scan
this
breath
the
number.
How
can
we
know
someone
is
connected
with
you
when
you
do
the
scan?
D
Also
for
the
association
and
some
of
the
unrelated
data
of
the
social
graph.
They
have
a
lot
of
point
lookup,
but
for
the
association
secondary
index
they
do
a
lot
of
iterator
because
sometimes
if
we
do
a
graph
search
or
do
some
data
mining
or
the
as3s
on
on
top
of
the
graph
data,
we
need
a
lot
of
scan
and
a
lot
of
iterates
for
directions
we
want.
So
it
has
a
lot
of
iterator
and
have
a
lot
of
single
delete.
D
The
second
thing
we
want
to
know
about
is
the
key
and
value
size
distribution,
so
it
it
influences
the
performance
of
the
roxdb
and
also
we
could
know
whether
we
need
to
do
the
key
value
separation.
We
want
to
do
the
improvement
for
the
compaction
that
flash
and
we
may
be
able
to
improve
the
caching
in
the
memory
so
usually
for
the
key
size.
D
But
you
can
say
in
the
zpdb
case,
since
it
is
a
distributed
cure
store.
We
generate
the
key
based
on
the
rules
we
have
internally,
so
size
just
have
some
fixed
number
and
we
don't
see
any
distribution
here.
This
is
similar
in
the
up
to
x,
maybe
just
like
integer
as
a
p
or
some
of
the
20
bytes
hash
value
as
a
key,
but
for
the
new
db
it
has
a
long
tail.
Distribution
size
is
a
is
also
pretty
interesting
value.
D
Size
distribution
is
wider
than
the
key
size,
distribution
and
also
it
is
closely
related
to
the
data
type
stored.
In
this
use
case
for
the
udb,
the
size
can
be
hundreds
kilobyte
size
of
network
set
level,
because
maybe
someone
wants
to
store
very
long
post
or
if
that
is
a
small
photo.
We
will
install
directly
in
the
graph,
but
if
it
is
very
large,
we
just
store
it
as
a
metadata
and
puts
the
videos
and
north
korea
photos
in
other
storage
and
just
store
the
metadata
here.
D
D
Another
pretty
interesting
things
we
want
to
know
about
from
workload
is
the
locality
of
the
keyword,
health
right.
What's
the
distribution
of
the
heart,
pure
health,
that
was
the
cold
killer
house
and
what's
the
correlation
between
them
in
the
key
space,
what
we
did
is.
First,
we
dump
out
all
the
keyword
health
from
the
instance
and
then
sort
them
as
that's
the
same
order
as
they
are
in
the
db
then
plot
out
how
many
times
they
are
accessed
throughout
the,
for
example,
24
hours
tracing.
D
Then
we
can
see
those
distribution
heat
map
in
the
udp
case.
Remember
that
that's
a
mexico
storage,
so
we
use
a
red
line
to
separate
the
different
mysql
tables
and
we
can
see
some
strong
key
space
locality,
which
means
particular
health
are
usually
stay
together.
You
can
see
those
green
lights,
green
blocks
and
the
cold
killer
house
are
located
together.
So
those
are
the
vertical
region.
D
This
is
a
hot
region,
so
that's
roughly
what
find
out
from
the
locality-
and
it
shows
up
in
all
of
the
use
case
at
facebook
and
which
trigger
us
to
improve
the
caching
efficiency
such
that
read.
Performance
can
be
further
improved
and
we
have
more
information
about
the
workloads,
but
due
to
the
time
constraint,
I
just
keep
them.
You
can
refer
to
the
paper
and
find
out
more
details.
D
Actually,
when
we
use
rusty,
since
it
is
just
a
library,
we
will
have
a
application
process
which
maintains
the
db
pointers
and
call
those
db
apis
if
we
want
to
control
or
access
a
certain
db,
the
application
need
to
expose
some
of
the
api
to
the
upper
layer
such
that
we
can
access
db,
but
it
is
also
also
possible
that
we
can
have
a
separate
threads
running
inside
the
application
as
a
swift
server.
So
this
monitor
can
access
all
the
dbs
registered
in
this
application.
D
D
So
here
are
some
use
cases.
As
I
mentioned,
we,
although
we
have
the
tracing
module,
but
we
cannot
control
that
when
the
db
is
running
so
by
adding
those
monitor
on
module,
we
just
use
a
command
line
to
communicate
with
the
sweep
server
and
then
we
trigger
the
tracing
start
and
the
angle
tracing.
Also,
we
can
manually
trigger
the
flash
compaction
and
also
do
some
adjustments
for
the
db
or
even
change
the
mutable
options.
D
D
It
is
pretty
convenient,
so
this
is
the
module
we
use
to
do
the
tracing
and
also
to
adjust
the
tracing
of
object
options
so
for
the
second
portion,
that's
about
the
current
and
the
future
of
drugs
db
in
terms
of
the
storage
or
the
infra
part.
So
that
is
what
we
change
for
the
desecrate
disaggregated
infrastructure.
D
Why?
We
need
to
disaggregate
the
storage.
So,
first,
in
the
traditional
setup
we
just
run
rustdb
in
a
server
which
has
like
a
four
terabyte
or
10
terabyte
ssd,
and
then
we
do
the
compaction
in
this
server
and
the
applications
just
issue
the
requirements
to
the
rest
of
instance
in
this
server.
But
it
has
a
lot
of
limitations.
For
example,
we
cannot
increase
the
db
size
larger
than
the
ssd
capacity
in
this
server,
and
we
have
imbalance
of
the
cpu
memory
and
storage
usage.
D
So
by
separating
the
storage
with
the
computing
power,
we
could
achieve
better
flexibility
and
improve
the
elasticity
in
the
system.
So
we
can.
We
have
a
shared
story
tool
and
we
just
store
any
data
there
and
access
data
throughout
from
the
remote
storage.
So
that's
what
provides
a
lot
of
flexibility
for
the
ros
db.
D
It
is
kind
of
similar
to
the
remote
sun
we
have
in
the
past,
but
it's
more
flexible.
It
is
kind
of
similar
to
the
snowflakes
data
lake
concept
and
we
can
reduce
the
tcl
of
the
system
and
also
we
could
achieve
better
obstruction
management
and
better
resources
scheduling.
So
here
is
a
long
term
goal.
D
We
have
shared
storage,
which
maintains
thousands
of
ssds
hdd-on
tape,
that's
a
very
complex
storage
pool
and
then
rustdb
primary
instance
will
store
the
data
at
the
remote
storage
and
we
can
have
hundreds
or
have
tens
of
the
rusty
read-only
instance
to
access
the
same
data.
So
we
can
response
the
user
throughout
the
read-only
instance.
D
That's
one
point
last
point
is
in
the
past
when
we
want
to
do
the
analyzing
of
data
of
the
of
this
relative
instance.
We
need
to
do
a
scan
and
dump
out
all
the
data,
something
like
egh
etl
in
the
in
the
data
warehouse.
But
now
we
could
let
those
offline
jobs
directly
reference
to
the
ssd
files
in
the
shared
store
link,
and
then
we
they
all
just
do
the
job
by
themselves.
That's
read
only
so.
D
D
Since
we
need
to
access
all
the
data
through
the
network,
we
will
have
high
average
latency
and
also
the
tail
latency
is
really
low
and
it's
hard
to
predict
and
we
can
potentially
have
different
kind
of
failures
from
network
size
from
storage
size
and
also
we
have
a
complex
software
stack
to
do
the
troubleshooting,
so
the
first
improvement
we
are
doing
is
to
introduce
local
cache
or
the
secondary
cache.
So
when
we
offload
the
storage
part
to
remote
actually
for
the
local
flash,
it
is
not
efficiently
used.
D
We
use
them
as
a
persistent
cache
and
for
each
db
instance,
its
data
will
be
cached
in
the
memory.
Dram
block
cache
first
and
then
will
be
stored
in
the
local
flash
cache.
So
that
will
improve
a
lot
of
performance
because
majority
of
the
data
workload
workloads
have
a
very
strong
locality.
So
when
those
hot
data
are
cached
at
the
local,
the
real
performance
will
be
improved
in
a
very
clear
way.
D
Improvement
want
to
do
is
remote
compaction,
so
the
basic
concept
is
the
priming
of
the
rough
instance
just
issues
a
compaction
command
and
what
the
files
to
be
compacted
then,
at
the
remote
site,
close
to
the
storage,
the
compaction
worker
will
just
gather
the
information
and
do
compaction,
since
all
the
ssd
files
are
immutable,
so
we
can,
it
will
not
influence
the
primary
rusty
instance
performance
after
the
conduction
is
done.
The
results
will
be
returned
to
the
primary
and
update
the
manifest
and
other
metadata.
D
So
this
is
a
kind
of
communication
between
the
primary
and
the
compaction
worker.
The
last
piece
is
about
the
error
handling.
As
I
mentioned
different
from
the
previous
use
case.
We
have,
we
have,
we
have
the
local
and
it
is
pretty
reliable.
Now
we
need
to
go
through
the
network,
so
we
need
to
define
different
arrow
types.
For
example,
some
of
them
can
be
retried,
some
of
them
just
file
corruption.
Some
of
them
might
be
the
system,
corruption.
D
So
we
identified
different
air
types
and
have
a
different
fine-grained
reactions
to
the
arrows,
and
then
we
will
recover
them
at
the
rust
tv
parts.
So
then,
we'll
redo
the
flush
redo
the
connection
and
during
this
process
rustdb
is
not
suspended.
It
can
still
accept
the
read
and
write
query,
so
that's
like
a
best
average
recovery
for
the
ruskdb,
and
maybe
user
will
not
be
aware
of
those
internal
automatic
recovery.
D
Okay,
that's
almost
the
content
that
represents,
and
finally
that's
about
rusty.
We
want
it
to
be
a
fast
and
easy
to
use,
persist,
persistent
killer
store
for
any
workloads
on
any
hardware
platform
and
here's
the
github
and
welcome
to
use
it
and
contribute
to
the
community.
Thank
you
all
and
welcome
for
the
questions.
A
B
B
D
D
Let's
just
assume
that
you
have
non-corrupted
ssd
file
manifest
and
also
for
the
ssd
dump
is
like
we
when
we
have
a
corrupted
ssd
file
or
got
some
data
corruption.
We
use
that
one
to
get
the
detailed
information
of
the
certain
message
file
so
that
for
the
diving
stats
are
debugging
and
also
we
look
at
log.
The
log
is
really
powerful
and
detailed.
D
So
we
can
look
at
the
log
and
get
all
information
about
operations,
so
we
lock
all
the
important
important
things
happens
inside
rusty
b
in
the
log,
and
for
that
part,
as
I
mentioned,
we
have
a
really
complex
and
useful
monitoring
system.
So
for
each
db
instance,
when
it
is
opened,
we
all
know
it
is
online
and
it
will
report
the
db
status
to
us
and
we
are
able
to
if
we
need.
D
We
are
able
to
control
this
db
instead
to
do
some
manual
compression
to
do
some
rear
arrangements
or
do
some
backup
things.
So
that's
like
the
parts
easy
to
use
that
require
to
make
the
db
easier
to
use.
Otherwise,
I
know
that
application
need
to
provide
rewrite
api
from
rust,
be
layer
to
your
layer
and
then
higher
level
application.
All
the
people
can
can
call
those
api,
but
by
using
that
monitor
parts
we
can
just
write
a
swift
command
line,
and
then
that
will
just
connect
to
the
db
and
look
at
it.