►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How We Reduced Write Latency in TiKV? - Liqi Geng, PingCAP
Serving write requests in stable and low average latency is what many distributed databases pursue. So does TiKV, a distributed transactional Key-Value database. After a detailed investigation, Liqi Geng and his team found that TiKV’s write performance might be restricted by many factors. Among all these factors, the raftstore module might be the one that causes delay the worst. In order to optimize write latency, TiKV team plans to lower tail latency, in addition to reducing the average write latency, to make sure the overall latency is consistent and low. In this talk, Liqi Geng will walk you through TiKV’s architecture and share the optimization measures he and his team have used in the raftstore module to reduce the average write latency and tail latency, such as Asynchronous IO and other optimizing trials.
A
A
A
A
A
A
The
customers
can
only
see
the
application
slow
and
they
don't
care
about
whether
they
have
encountered
low
probability
events.
Furthermore,
the
slow
events
may
not
have
a
low
probability,
because
their
latency
may
have
amplification
effects
for
parallel.
Failed
requests,
for
instance
in
the
transaction
mode
of
the
previous
phase,
needs
to
wait
for
all
parallel
provides
to
be
done.
So
the
latency
of
the
right
phase
is
the
maximum
latency
of
all
playwrights.
A
If
the
provided
phase
involved
in
provides
the
probability
that
its
latency
is
longer
than
x,
percentile,
latency
or
price
request
is
one
minus
x
percent
to
earn
its
power
x
percent
represents
the
probability
that
the
latency
of
the
request
is
shorter
than
the
x
percentile
latency,
and
each
request
is
an
independent
event,
so
x
percent
to
us
power
means
the
probability
that
earned
requests.
Latency
are
all
shorter
than
the
x
percentile
latency.
A
A
Next,
let's
move
to
the
right
floor
of
tech
av
before
we
discuss
the
optimization
at
first,
a
request
is
sent
to
scheduler
workers,
which
are
responsible
for
transaction
translators
and
checking
and
transform
requests
to
key
value.
Pairs
to
rough
store.
Rough
store
is
a
consensus
layer
of
tech
uv.
It
uses
rough
consensus
algorithm
to
make
techie
before
tolerant.
A
A
A
A
A
Then,
let's
look
at
the
flowchart
at
the
first
of
each
round.
Store
series
calls
step
to
handle
multiple
messages,
for
example,
for
the
leader
the
source
serial
can
propose,
commands
and
handle
the
append
response
from
from
the
follower
for
the
follower
the
source
serial
can
handle
a
panda
request
from
the
leader.
A
A
A
First,
we
only
consider
single
roughed
group.
Secondly,
disk's
right
rate
is
constant.
Thirdly,
the
request
arrival
time
is
unifying
distributed.
Finally,
the
systems
reaches
a
steady
state.
In
practice,
a
real
system
is
far
more
complicated.
For
example,
there's
always
a
burst
in
request
for
many
reasons,
and
this
correct
latency
is
not
always
the
same.
A
Store
loop
is
a
processing
time
of
one
round
of
source
of
spirit
as
equal
to
massive
duration,
plus
io
duration,
proposed
weight,
method
of
hand,
weight
and
method
of
panel
response
weight
are
all
the
time
the
message
waits
for
the
source
serial
to
process
according
to
the
affirmation
assumptions
as
the
messages
arrive
uniformly
and
they
will
be
processed
in
the
next
round.
Their
average
waiting
time
is
about
half
of
the
store
loop,
that
is
0.5
message,
duration,
plus
0.5,
io
duration,.
A
A
A
Similarly,
when
messenger
panel
response
is
received,
the
proposal
has
to
wait
4.5
hour
duration
more
before
being
committed,
so
the
root
cause
as
both
proposed
weight
and
messenger
append
response
weight,
contain
an
unnecessary
0.5,
io
duration,
but
why
messenger
append
weight
is
not
included
here?
It
seems
to
be
very
similar
to
them.
A
A
A
In
addition,
star
syria
needs
to
use
a
synchronous
io
to
co-operate,
cooperate
with
asynchronous
ready.
Note
that,
although
we
call
it
asynchronous
io
for
now,
we
do
not
use
linux
asynchronous
io
technology
such
as
aio
io
urine.
We
just
simply
move
the
ios
process
to
the
dedicated
io
series
in
the
current
implementation.
A
This
is
a
pr
for
rough
eyes
and
tracking
user
for
techyv.
Now,
let's
look
at
the
flowchart
compared
with
the
old
floor.
Asynchronous
ready
allows
the
save
to
disk
operation
to
be
asynchronous.
After
writing
is
completed.
The
store
series
will
call
unready
unprocessed
ready
to
notify
the
rough
state
machine
for
rockstar.
A
A
In
this
sequence,
diagram,
the
leader,
io
and
follow
I
o-
are
added
to
indicate
the
io
series.
Io
weight
means
a
waiting
time
of
an
io
request,
based
on
the
assumption
mentioned
before.
Io
weight
is
equal
to
half
of
our
io
duration,
since
the
store
series
only
processed
message
and
don't
need
to
wait
for
io,
the
third
loop
duration
becomes
our
message.
Duration.
A
A
However,
in
reality,
the
message
duration
is
not
the
same
in
the
synchronized
version
and
then
synchronous
version
there
is
an
optimization
called
command
batch.
Install
series
batch
will
batch
as
many
proposals
as
possible
into
one
in
the
case
of
the
same
total
number
of
requests,
the
smaller
number
of
proposals,
the
smaller
the
cpu,
had
of
store,
siri
and
jrpc
series.
A
That's
because
of
some
internal
implementation
problems
raised
by
many
proposals
and
rough
messages.
We
are
trying
to
optimize
them
to
mitigate
the
impact,
but
in
my
opinion,
it
can
be
totally
eliminated
back
to
these
equations.
The
effect
of
command
batch
is
very
good
in
synchronous
version.
Due
to
the
long
I
o
duration,
while
it's
not
in
a
synchronous
version.
A
A
A
In
addition,
a
synchronous,
ios
average
store
duration
has
less
jetta
than
masters
in
the
second
figure,
a
synchronous,
io's
99
percent
higher
still
duration,
is
about
one-third
of
masters
in
the
third
figure,
a
synchronous,
io's
99.9
percent,
higher
stall
duration.
As
about
half
of
masters,
next,
let's
talk
about
the
future
work.
That's
it
to
apply
unprecedented
entries,
as
mentioned
before,
store
duration
is
a
time
between
propose
and
commit
of
a
proposal.
A
Actually,
it's
not
accurate.
The
exact
definition
is
the
time
between
when
a
proposal
proposed
to
when
is
committed
and
persisted,
as
shown
in
the
figure,
the
leaders
I
o
encountered,
latency
and
the
two
followers
respond
to
leader
with
messenger
apparent
response.
Earlier
at
this
time,
the
proposal
has
been
committed,
but
not
persisted
to
unleaders,
so
it
cannot
be
applied.
A
A
In
fact,
as
long
as
the
majority
peers
still
exists,
the
committed
entries
will
not
be
lost,
so
applying
unpersistent
entries
in
advance
should
not
break
correctness,
and
it
can
significantly
reduce
the
tear
latency
of
saw
duration,
as
shown
in
the
figure.
The
still
duration
is
much
shorter
than
before.
A
A
A
That's
basically
everything
I
plan
to
cover
today.
If
you
like
to
explore
more
or
dig
deeper
into
techyv
projects,
please
check
out
the
resources
and
listed
here.
You
are
welcome
to
contribute
to
kv
project
and
follow
us
on
twitter
youtube
and
contact
us
by
slack
channel.
Hopefully
you
got
something
useful
today
and
thank
you.