►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Cloud Native Distributed Event Streaming from TiKV - Zixiong Liu, PingCAP
Data processing is evolving to be stream-oriented to enable interconnection between subsystems or microservices, but it could be challenging for distributed data stores, such as TiKV, to stream data effectively. Zixiong Liu and his team have successfully achieved distributed event streaming from TiKV with low computational cost, low latency, and elimination of single points of failure. Now it is possible to produce a stream of updates from TiKV, which, with suitable deduplication, are ordered by their commit timestamps. In his talk, Zixiong Liu will talk about the techniques used in the design of TiKV that facilitate data streaming, and the implementation of the distributed computation performed on the exported TiKV data so that the data can be converted into formats suitable for consumption by third party data solutions.
A
Now,
let's
go
over
today's
agenda.
First
I'll
give
a
brief
introduction
to
techyv
and
then
I'll
talk
about
the
high
level
design
of
our
distributed
event
streaming
project
then
I'll
talk
mainly
two
aspects
of
the
detailed
design
and
then
I'll
also
talk
about
our
use
of
edcp
and
the
higher
bit
high
availability.
We
implemented
with
it.
A
A
This
diagram
shows
the
overall
architecture
of
thai
kv,
so
this
is
a
tech,
heavy
cluster
with
four
machines.
We
use
shared
nothing
architecture
so
that
techie
can
run
on
commodity
hardware,
with
no
special
equipment
needed
each
machine
can
hold
multiple
regions.
The
number
of
regions
on
each
machine
can
be
quite
high,
like
more
than
ten
thousand.
A
A
A
We
use
a
percolator
like
protocol,
but
I'll
not
go
into
detail
of
that.
Our
protocol
requires
a
multi-version
key
value
data
store
and
that's
our
second
layer.
The
third
layer
is
called
reptv,
which
is
where
replication
comes
into
play
and
we
use
left
to
maintain
consistent
replicas
of
regions.
If
you
recall
at
the
bottom,
is
the
roxdb
that
we
use
to
persist
data
onto
the
disk.
A
A
A
A
A
This
is
basically
what
we
mean
when
we
talk
about
event
streaming,
but
only
the
new
data
is
not
enough
because
from
time
to
time,
we'll
need
to
really
historical
data
from
roxdb,
because,
as
a
user
of
because
as
a
user
of
a
change
data
capture
tool,
you
may
want
to
get
a
little
bit
back
into
the
history
and
retrieve
say
a
data
stream.
That
starts
from
five
minutes
before
present.
A
A
A
And
when
we
got
started,
we
first
needed
to
decide
where
we
should
capture
the
data.
We
wanted
to
choose
an
appropriate
abstraction
level
where
we
intercept
the
data
rights.
First,
we
skip
the
transaction
layer.
Implementing
data
capture
at
the
transaction
layer
basically
means
that
we
are,
or
we
would
have
to
record
each
api
calls.
A
A
A
The
high
level
design
is
this:
first,
we
will
receive
data
from
all
regions.
You
can
think
of
this
as
an
as
receiving
from
multiple
streams
simultaneously,
and
then
we
decide
that
and
when
we
decide
that
all
regions
are
done
with
sending
transactions
whose
commit
yes
is
less
than
or
equal
to
some
value.
We
produce
a
watermark.
A
A
This
is
an
overview
of
the
types
of
events
that
we
receive
from
each
region.
We
have
pre-write,
which
has
the
stack
yes
of
the
transaction,
the
key
and
the
value,
and
we
have
commit
which
is
paired
with
prewrite,
and
it
contains
both
start.
Yes,
the
committee
s
and
the
key,
and
we
have
lock,
which
contains
the
cs
and
the
key,
and
we
have
unlock.
A
Now
we
know
the
four
types
of
events:
let's
take
a
look
at
the
first
attempt
of
making
an
algorithm
to
produce
the
final
data
stream.
Here
I
use
the
pseudocode
with
a
syntax,
that's
similar
to
that
of
rus.
Since
ikv
is
written
in
rest,
we
assume
a
data
type,
that's
called
a
tx,
an
event
that
represents
the
items
of
the
stream
that
has
watermarks
in
it.
A
We
have
a
special
name
for
this
watermark
and
we
call
it
the
result.
Yes,
but
quickly.
We
ran
into
a
problem.
What
should
the
result
us
be?
We
didn't
really
have
enough
information
here.
The
locks
are
being
tracked
and
it's
it's
easy
to
know
the
start.
Yeses
of
the
transactions
that
have
been
that
have
placed
at
least
one
lock,
but
the
problem
here
is
that
we
don't
know
if
there
is
a
transaction
that
hasn't
earlier
started.
Yes,
that
is
so
being
processed
yet
to
write
a
lock
carrie
summarize
the
problem.
A
A
In
addition
to
the
function
show
shown
in
the
previous
slide,
we
are,
we
need
to
launch
another
thread,
and
the
reason
is
that
we
we
need
a
barrier
here.
We
need
at
least
two
concurrent
threads,
because
a
barrier
is
needed
to
wait
for
all
pending
rights
to
be
observed
in
this
piece
of
code.
We
first
read
the
latest
logical,
commit
logical,
timestamp
from
td,
and
then
we
enforce
what
we
could
call
a
barrier
here,
so
that
when
we
read
the
logs
here,
all
previous
writes
would
have
already
been
processed.
A
Data,
but
before
getting
into
detail
I'll,
introduce
you
to
another
distributed
component
separate
from
thai
kv.
Thai
cdc
cdc
is
designed
with
three
goals
in
mind.
First,
from
the
management
point
of
view,
we
want.
We
wanted
the
data
streaming
component
to
iterate
separately
from
thai
kv,
and
the
second
goal
is
that
we
aim
to
design
a
fully
distributed
component
to
eliminate
every
single
point
of
failure
in
the
stream
processing
process
and
the
third.
A
A
A
There
are
two
concepts
that
we
need
to
clarify
a
little
bit
spans
and
regions
spans
are
just
the
intervals
in
the
key
space.
The
keys
in
ikv
are
totally
ordered,
so
it's
possible
to
define,
ring
json
and
the
placement
driver
can
be
queried
to
find
out
which
regions
correspond
to
what
span
cdc.
Then,
schedules
grpc
connections
to
take
divisions
with
the
knowledge
of
this
spanned
region,
mapping,
when
regions,
merge
or
split
like
cdc,
will
reschedule
the
effective
span
based
on
the
new
span
to
region,
mapping
and.
A
Third,
to
make
pisces
a
useful
product,
we
made
it
support,
decoding
thaidb
records,
so
it
can
be
used
to
capture
changes
from
type
db.
What
we
have
implemented
is
similar
to
mysql
bin
log,
but
icdc
is
capable
of
writing
the
events
directly
to
various
kinds
of
downstream,
such
as
mqs
f3
files
and
mysql
compatible
relational
databases
such
as
mysql
itself
and
idb.
A
A
This
makes
thai
cdc
a
stateless
service
and
makes
it
very
easy
to
deploy
and
maintain
it.
Second,
we
use
etcd
as
a
consensus
service
to
maintain
the
current
state
of
the
cdc
cluster
that
all
nodes
can
agree
on.
This
is
crucial
because,
if
you
want
to
implement
it,
if
implement
high
availability,
you
need
to
know
which
nodes
are
alive
and
what
they
are
doing.
A
First,
we
add
a
node,
we
add
another
node.
The
first
node
is
now
elected.
The
owner
of
the
cluster.
The
owner
is
the
node.
That's
in
charge
of
actively
manipulating
the
state
of
edcd
to
coordinate
the
nodes
and
the
owner
decides
what
tasks
need
to
be
run
and
assign
the
tables
that
are
to
be
replicated
to
each
node.
That
is
available.