►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Vitess: Stateless Storage in the Cloud - Sugu Sougoumarane, PlanetScale
When Vitess was migrated to run from bare-metal into Google's cloud, it was deployed as a regular stateless application. This meant that a process reschedule resulted in all the local data being wiped. The property of Vitess to survive in such an unforgiving environment made it naturally suited to run on Kubernetes. How did Vitess manage to run on such an environment without losing data, and providing High Availability, Scale and Performance? How are other organizations running Vitess? This session will answer these questions, as well as go into the design principles that prepared Vitess to be cloud-native.
https://sched.co/UaeD
A
A
Hopefully,
I
can
talk
about
why
I
call
this
the
way
it
is
so
how
many
of
you
have
not
been
to
the
keynote
or
did
not
see,
do
not
know
what
Vitesse's
yeah
there
are
a
few,
so
I'll
do
a
quick
rundown
of
what
we
Tessa's
and
then
we'll
get
into
what
this
stateless
storage
is
all
about.
So
with
us
is
what
I
call
is
a
cloud
native
database,
which
means
that
it
runs
very
comfortably
in
kubernetes
without
losing
your
data.
It
scales
massively.
Actually
that's
how
it
was
originally
born.
A
We
when
we
started
developing
it
in
2010
at
YouTube.
It
was
mostly
meant
to
just
scale
massively
and
later
it
became
cloud
native
and
highly
available.
You
can
configure
with
tests
to
have
almost
no
downtime
or
practically
no
downtime,
something
like
five
nines
of
availability
and
it's
the
based
on
my
sequel
means
that
it
uses
my
sequel
underneath
it
also
speaks
the
my
sequel
protocol.
So
if
you
are
already
on
my
sequel
or
if
you
are
familiar
with
my
sequel,
migrating
with
us
is
not
a
heavy
lift.
A
Obviously
we
test
is
graduated
I
assume
many
of
you
know,
and
here
are
some
of
the
stats
so
more
than
even
the
numbers
they
think
that
we
are
most
proud
about
in
with
us
is
who
is
using
with
us
who
adopted
with
us
right
and
this
here's
a
more
extensive
list
of
adopters
that
I
didn't
share
about
in
the
keynote,
which
is
so.
If
you
ask
me,
the
the
coolest
part
of
these
adopters
is
these
are
what
I
consider
technology
leaders
in
the
industry.
A
So
when
these
people
say
that
they
are
using
a
technology,
they
are
kind
of
setting
the
example
that
a
lot
of
others
tend
to
follow.
So
it's
a
pretty
good
stamp
of
approval.
If
you
look
at
some
of
the
names
here,
like
slack
github,
New
Relic
Pinterest,
so
these
are
companies
that
are
looked
upon
when
you
think
about
how
do
you
scale
infrastructure
so
and
some
quotes
from
many
of
these
adapter
adopters?
A
So
slack
basically
is
saying
that
there
is
nothing
like
we
tests
out
there,
the
things
that
we
wanted
to
do
with
our
storage
system.
Only
we
test
could
could
do
it
and
here's
one
from
square
we're
saying
that
somehow
you
can
just
keep
on
scaling
with
tests
and
just
keep
scaling
with
you.
So
that
is
his
testimony
and
here's
another
one
from
Pinterest.
A
It
says
that
the
other
cool
thing
about
Vitas
is
that,
yes,
you
can
do
all
this
leash,
sharding
and
scaling,
but
you
don't
really
have
to
rewrite
your
app
to
be
able
to
achieve
this
scale.
So
those
are
some
of
the
things
that
I
collected,
there's,
actually
lots
and
lots
of
quotes
that
we
can
collect.
But
these
are
some
highlights
that
I
collected
by
looking
around
so
so
that
is
all
overview
about
Vitas
now
I'm
going
to
jump
into
what
makes
Vitas
cloud
native
and
what
this
stateless
storage
means
at
all.
A
So
for
that
there's
two
reasons:
there's
a
historical
reason
why
we
test
is
cloud
native
and
there's
also
a
technical
reason,
so
I'm
going
to
talk
about
the
historical
reason
and
we've
the
technical
details
into
this
story,
so
it
all
starts
in
2010
we
were
at
YouTube
on
in
2010
we
were
on
Prem.
We
had
our
own
data
centers,
even
though
we
were
acquired
by
Google.
We
were
not
part
of
Google
infrastructure,
and
at
that
time
our
infrastructure
was
awesome
was
perfect.
Flawless,
everything
was
fine,
it
was
not.
A
It
was
like
hell
was
breaking
loose,
things
were
falling
all
over
the
place
and
my
colleague
Mike
and
I
decided
to
pretty
much
take
ourselves
out
of
this
firefighting
mode
figure
out
what
we
need
to
do
about
this
thing,
because
it
was
not
going
in
the
right
direction.
It
was
just
getting
worse
by
the
day,
so
we
actually
built
a
huge
spreadsheet
of
all
the
problems
we
have
ever
seen
at
YouTube
and
all
the
problems
we
are
possibly
going
to
face
and
then
possible
solutions.
A
A
So
that's
what
we
initially
built
and
from
that
with
this
was
born,
and
it
was
originally
called
Voltron
by
the
way,
because
it
was
different
pieces
coming
together,
but
later
I
know
you
can't
use
Voltron
as
a
project
name,
which
is
why,
and
we
used
word
Ron
and
we
named
all
our
files,
VT
VT,
VT
right,
so
you
had
to
come
up
with
the
name.
That's
not
Voltron
that
had
VT
in
it.
So
that's
how
it
s
was
born.
Oh
and
the
others.
A
Father
story
is
I,
don't
know
how
many
of
you
have
heard
of
strong
bad
yeah
yeah.
So
he
has
a.
He
has
a
thing
where
it
says
that
the
best
way
to
create
a
cool
name
is
to
actually
find
a
normal
name
and
completely
misspell.
It
I
know
if
you
know
the
Limousin
video,
so
we
said.
Okay,
we
test
mean
speed
in
French,
so
you're
going
to
drop
the
e
and
then
that's
what
that's
why
the
name
is
the
way
it
is.
A
So
it
was
all
fine,
but
in
2013
I
don't
know
you
remember
the
signature.
The
significance
of
this
year
Snowden
happened,
and
there
was
a
directive
that
no
data
shall
be
outside
Google
data
centers,
so
we
had
to
migrate
into
Google
and
until
then
none
of
us
at
YouTube
knew
anything
about
Google
infrastructure.
We
knew
this,
Borg
thing
existed
and
it
does
did
magical
things
and
stuff,
but
we
actually
went
looked
inside
it.
A
It
had
nothing
that
we
could
have
used
needed
for
running
with
tests,
so
Google
Google's
Borg
is
mostly
a
an
environment
that
lets
you
deploy
stateless
applications
at
humongous
scale,
and
if
you
wanted
storage
it
gave
you
api's
like
there's
an
API
for
BigTable.
There
is
an
API
for
Colossus.
There
is
an
API
for
mega
stores
like
all
the
storage,
the
SS
tables
they
had
a
bunch
of
storages
and
as
by
the
way
there
is
no
block
storage
at
Google.
There
is
no
API
for
storing
something
in
a
file.
A
It
doesn't
exist,
it's
not
allowed
and
Colossus
is
the
only
one
and
it's
not
a
file
API.
That
was
very
intentional
design.
So
we
look
at
all
that,
and
he
says
how
do
you
run
my
sequel
on
on
that
right
and
the
only
other
storage
was.
You
could
write
your
local
disk
if
you
are
running
as
an
application,
but
as
soon
as
your
pod
gets
rescheduled,
we
will
wipe
your
disk.
So
that
was
the
deal.
So
that's
when
one
limitation
and
the
other
problem
is
moving
with
tests
into
Google's
ecosystem.
A
They
have
all
these
custom
api's,
they
have
chubby,
they
have
stubby,
they
have
Borgman,
they
have
log
infrastructure,
they
have
throttling
infrastructure.
What
do
all
these
things
have?
They
all
have
api's
so
with
this
is
an
open
source
project
meant
to
run
in
bare
metal?
How
do
you
put
this
into
Google,
so
the
immediate
answer
was,
or
you
should
close
source
it?
A
There
is
no
way
you
can
run
this
in
Google
assets,
because
Google's
API
is
too
custom,
so
we
kind
of
took
that
as
challenge,
since
we
are
not
going
to
do
that,
we
are
going
to
see
what
we
can
do
to
keep
it
as
open-source.
So
what
we
built,
what
we
did
was
just
build
abstraction
layer
after
abstraction
layer
and
I
will
come
back
to
what
that
means.
A
What
that
meant
later
in
the
future.
So
we
built
an
abstraction
layer
for
stubby
we'd,
build
an
abstraction
layer
for
chubby,
so
we
did
all
that
and
manage
to
get
something
deployed.
But
what
about
storage
right,
that
is,
there
is
no
storage
API
of
Google
that
we
could
use.
So
what
we
did
was
we
just
used
local
storage,
but
I,
don't
know
how
many
of
you
were
in
the
previous
talk.
A
Jatin
actually
explained
the
entire
thing
in
like
two
minutes,
but
I'm
going
to
take
longer
to
explain
what
what
we
did
so
we
ran
my
sequel
is
in
this
semi
synchronous
mode,
which
means
that
we
had
one
master
and
lots
of
replicas
and
YouTube
requires
huge
amount
of
read
traffic.
So
we
had
like
AD
200
replicas.
We,
the
semi
sync
mode,
allows
you
to
guarantee
that
a
commit
is
not
complete,
at
least
until
at
least
one
other
replica
had
the
data.
A
So
what
that
means
is
that
if
a
master
goes
down,
it's
guaranteed
that
another
replica
had
it.
So
we
used,
we
relied
on
that,
and
we
said
that.
Okay,
if
that
happens,
which
means
that
it
is
okay,
if
Bob
likes
the
data
of
the
master,
because
there
is
always
one
other
replica
that
has
it.
So
that
is
essentially
how
we
ran
with
us
on
Borg.
So
basically,
we
ran
with
test
as
far
as
Borg
was
concerned
with
this
was
a
stateless
application.
A
All
right,
okay,
so
guess
what
2015
kubernetes
gets
announced.
We
didn't
even
know
we
all
this
was
happening
and
then
we
look
at
kubernetes
says
what
kubernetes
has
ephemeral
storage
it
does
everything
isn't
that
what
you've
been
doing
on
borg?
So
we
go
talk
to
the
kubernetes
teams.
They
say:
hey
I,
think
we
think
with
us
can
run
in
kubernetes.
Already
it
says
yeah
it
looks
like
it
can.
So,
let's
announce
it
so
we
announced
before
kubernetes
1.0
was
released,
that
we
test
is
ready
for
kubernetes.
A
So
March
20th
is
the
date
we
blocked.
July
21
is
when
kubernetes
1.0
was
announced
so
funny
story.
People
believed
us
and
they
said
so
speech.
Labs
are
the
first
company
that
came
and
said.
Okay,
you
think
it
can
work,
say
yeah
yeah
totally,
so
they
they
went
into
production
that
would
they
are
still
in
production.
So
they
are
the
oldest
living
with
us
instance.
They've
been
running
in
kubernetes
since
2016,
so
then
came
HubSpot,
so
stitch
labs
was
actually
a
sharded
environment.
They
were
previously
on
scale
arc
which
went
out
of
business.
A
So
they
said.
Oh,
we
test
can
do
what
scaler
does
so.
Therefore,
we
migrated
to
a
test
so
hops.
What
was
the
first
company
that
actually
used
with
test
because
it
ran
well
on
kubernetes?
They
said.
Oh,
you
have
all
this
orchestration
layer.
You
have,
you
know
how
to
survive.
All
these
part,
takedowns
I,
think
we
like
it
and
the
cool
thing
they
did
was
they
gathered
like
10
terabytes
of
queries
and
they
dumped
it
on
with
us
and
a
whole
bunch
of
them
didn't
work.
A
They
worked
through
each
and
every
one
of
those
and
God
all
of
them
working,
so
they
actually
narrowed
the
gap
of
the
query.
The
query
said
that
we
test
could
handle
too.
They
pushed
it
pretty
far.
It's
still
not
completely
out
there,
but
a
large
chunk
of
the
queries
that
working
with
tests
that
should
work
in
my
sequel
house
part,
actually
got
it
working
and
then,
after
a
while
some
Chinese
people
started
showing
up
in
the
Vitesse
slack.
They
never
identified
themselves,
but
they
were
asking
extremely
sharp
and
intelligent
questions.
A
Who
are
these
people
asking
about
query
plans
they're
asking
about
like
optimization
here
and
there
and
then,
after
a
long
time,
they
reveal
all
we
are
from
JD
comm.
It's
the
you
know
at
that
time
it
was
the
second
largest
online
retailer
in
China.
Now
it
looks
like
they
are
the
largest
now
so,
and
this
thirty
five
million
QPS
information
I
found
out
only
last
week,
I
had
to
really
prod
them.
No,
you
had
your
singles
day
like.
Can
you
give
me
some
good
number?
A
And
finally,
he
told
me
third
I
didn't
believe
it
no
way
say.
Why
should
I
just
say
like
a
few
like
JD?
Does
just
a
few
million
QPS
say?
Oh
no!
No!
No!
This
comes
straight
out
of
our
graphs.
You
can
take
it
to
the
bank.
Okay
I'm,
going
to
announce
it
in
the
keynote.
So
that's
that's
the
story
behind
JD
and
finally,
all
of
you
know
about
nozzle,
where
they
they
were
the
first
ones,
to
prove
that
you
can
actually
move
from
cloud
to
cloud.
A
If
you
are
fully
on
kubernetes
and
running
on
with
tests,
the
one
part
that
I
was
not
clear
about
is
how
they
actually
did
the
pipeline
and
probably
Thursday
when
direct
talks
about
it.
He
I'm
sure
he
will
cover
how
he
did
that.
So
the
test
has
been
running
for
three
years
right.
The
thing
is
that
we
did
not
know
that
kubernetes,
you
are
not
supposed
to
run
storage
on
kubernetes.
A
A
This
is
his
point.
It
says
you
cannot
just
take
my
sequel
and
put
it
in
kubernetes.
You
will
lose
your
data
you're
guaranteed
to
lose
your
data
or
you
are
going
to
have
other
problems,
probably
bigger
than
losing
your
data,
so
I'm
going
to
go
through
some
of
the
scenarios.
Some
of
the
details
involved
in
why
he's
giving
this
advice
is
his
advice
is
sound
and
there
is
a
reason
for
it.
A
So
here's
the
thing
right,
I'm,
the
naive
person,
I'm
going
to
say:
okay
I'm,
going
to
pick
my
sequel
and
put
it
in
a
pod
and
use
local
storage
boom
right.
The
first
thing
is:
what
happens
is
when
your
pod
gets
rescheduled
is
going
to
wipe
all
your
data,
so
obviously
nobody
should
be
running
it.
This
way.
This
is
probably
a
more
sane
deployment
where
you
say:
okay,
I'll
have
mounted
volume
and
then
use
PD
or
EBA
whatever
and
just
run
my
sequel
as
a
stateless
application.
This
will
work
and
this
can
work.
A
There
are
some
people
who
use
this
configuration.
The
problem
here
is,
if
your
part
gets
rescheduled.
My
sequel
when
it
comes
back
up,
has
to
perform
crash
recovery,
and
that
can
take
a
long
time.
I,
don't
know
minutes
hours,
sometimes
depending
on
how
big
your
data
is.
So
it's
definitely
not
good
for
high
uptime.
So
if
you
don't
care
about
uptime,
yes,
you
can
configure
it
this
way,
but
there
is
a
problem.
My
sequel
is
tuned
for
low
latency,
I
ops
and
going
through
a
network.
A
Boundary
is
not
something
it
enjoys
so
you're
going
to
pay
some
price
on
throughput.
Strangely,
it's
not
many
people
care
about
it.
There
are
people
who
can
tolerate
this
and
those
who
can
tolerate
it.
It's
fine.
The
people
who
cannot
tolerate
this
are
typically
people
who
are
already
on
Prem
already
used
to
this
latency
and
they
wrap
has
been
tuned
to
do
that
latency
and
suddenly,
if
you
increase
the
latency
between
the
my
sequel
and
disk,
it's
something
that
most
laps
don't
tolerate.
A
A
How
do
you
put
multiple
so
there
yeah?
The
other
was.
The
other
scenario
is
to
say:
I'll.
Do
the
master
replica
scenario
right
where
I
have
one
master
and
one
replica
and
each
in
its
their
own
parts?
You
could
do
this
either
with
local
storage
or
with
mounted
storage,
but
the
problems
are
similar.
So
let's
say
you
are
going
to
deploy
a
stateful
set
right.
Stateful
set
was
created
for
stateful
things,
that's
a
lie:
stateful
state
has
no
state.
A
You
know
that
the
only
state
in
stateful
set
is
outside
of
stateful
sets,
so
the
the
only
way
a
stateful,
the
main
property
of
a
stateful
set
is
that
you
can
identify
each
part
individually
and
give
give
it
a
specific
traffic.
So
that
is
the
property
of
a
stateful
set.
So
you
can
say
part
zero
is
my
master
and
part.
One
is
my
replica
and
you
can
ask
the
app
server
talked
to
a
part
zero
because
it
is
the
master.
So
we
can
make
that
happen.
You
can.
A
You
cannot
use
kubernetes,
--is
automatic
load,
balancing
because
kubernetes
will
not
differentiate
between
a
master
and
replica.
So
you
have
to
talk
to
individual
parts.
So
the
main
problem
is,
you
cannot
designate
the
master
if
you,
if
you
addressed
it,
the
state
will
set
as
an
entire
entity,
which
means
that
you
have
to
address
each
part.
But
if
a
pod
gets
rescheduled
and
I
say:
okay,
I'm
going
to
failover
my
master
part
1
as
replicas.
You
have
to
tell
the
app,
and
that
doesn't
happen
in
automatically
for
you.
A
So
how
does
this
work
on
Prem,
on-prem
masters?
Don't
go
down.
The
up
time
for
masters
is
like
2
months,
3
months,
6
months.
Here
they
go
down
like
once
a
week
at
least
probably
more
often,
depending
on
how
much
entropy
that
exists
around
here.
Kubernetes
may
want
to
shift
things
around.
So
those
are
the
three
problems
with
using
my
sequel
in
this
scenario,
and
if
you
bring
up
a
new
replica,
how
do
you
initialize
it
that's
another
problem?
It
cannot
in
Coober
in
kubernetes,
just
bringing
up
my
sequel.
A
How
do
you
initialize
the
data
saying
that
ok,
the
data
from
the
master
has
to
be
copied?
It
has
to
be
pointed
to
that
master.
So
all
that
orchestration
is
something
that
you
have
to
write
extra
so,
which
is
the
reason
why
Kelsey
has
been
saying
that
you
cannot
just
take
my
sequel
things
that
are
storage,
that
is
supposed
to
run
on
Prem
directly
into
kubernetes.
A
There
is
lot
of
blue
code
that
needs
to
be
written,
which
is
essentially
what
we
test
does
for
you,
which
is
what
we
had
to
do
for
ourselves
all
right
and
then
there
are
cultural
issues
and
trust
me.
These
are
actually
really
really
horrible,
because
if
you
have
configured
your
system
with
these
long-running
instances
taking
into
a
cloud
environment
is
extremely
different.
A
Like
the
lifecycle,
like
you
expect
things
to
change
at
certain
rate,
but
in
the
cloud
they
change
at
a
different
rate
like
how
long
does
a
process
live,
the
data
size
so
kubernetes
likes
smaller
data
size
in
on-prem,
you
may
have
a
14
terabyte
database
and
kubernetes
likes
to
move
things
around.
You
cannot
move
terabytes
around
with
such
ease,
so
the
culture
of
Vitas
is
to
build
smaller
chunks,
and
so
those
we
can
quickly
move
around.
A
Even
if
kubernetes
asked
reschedule
two
kubernetes
likes
to
reuse
IPs,
that's
a
no-no
in
on-prem
you
IPS,
are
fixed
for
life
and
and
topology
overload
is
another
big
concern.
Many
people
think
that
systems
like
HDD
zookeeper
can
be
used
like
a
regular
data
store.
No,
they
are
used.
They
are
supposed
to
be
used
extremely
sparingly
and
when
you
scale
things
out,
you
accidentally
spam
your
topo
and
bring
it
down.
So
all
these
things
we
had
to
change
in
with
us
so
yeah.
A
It's
not
very
easy
to
convert
a
non
cloud
ready
software
to
be
cloud
ready
because
it
is
not
sensitive
to
these
cultural
differences
that
exist
between
the
two
and
may.
You
may
end
up
spamming
things
that
you
should
not
be
spamming
all
right
so
far,
so
good,
okay,
good
doing
our
time.
So
Kelsey
did
later
reiterate
his
statement
and
he
says
yes,
you
cannot,
but
you
can
use
systems
like
quick
tests
because
they
actually
glue
provide
the
glue
between
kubernetes
and
the
storage.
A
The
app
server
as
far
as
the
app
server
is
concerned,
so
essentially,
if
you
look
at
all
the
problems
in
the
back,
the
proxy
layer
that
we
introduced
at
YouTube
for
solving
a
different
problem
is
also
coming
in
handy
because
that
is
essentially,
you
cannot
run
a
storage
system
without
a
proxy
layer,
because
all
those
problems
can
only
be
solved
with
the
proxy
layers
standing
in
the
middle.
So
we
had
this
VT
gate,
so
app
server
talks
to
VT
gate
mitigate
is
completely
stateless.
A
You
can
scale
it
up
and
down,
depending
on
your
growing
needs
and
as
far
as
the
app
is
concerned,
once
it's
connected
to
VT
gate,
it
thinks
that
it
is
connected
to
it
to
a
humungous
database,
but
under
the
covers
it's
actually
a
whole
bunch
of
mice
equals
all
working
in
their
own
tandem,
like
the
the
way
I
would
describe.
The
philosophy
behind
this
architecture
is
one
is
simplicity.
We
don't
have
any
moving
parts
that
are
not
necessary.
They
are
the
bare
minimum
of
what
we
need
loose
coupling.
A
So
they
are
the
advantage
of
loose
coupling
is
indefinite
scalability.
So
that's
the
reason
why
we
test
can
scale
all
these
parts
are
meant
to
operate
on
their
own
with
very
little
interdependence,
II
between
them
and
somebody
who
are
saying
a
loose
coupling
is
actually
one
of
the
hardest
problems
to
solve
in
computer
science
because
things
tend
to
want
to
come
together.
A
And
how
do
you
draw
the
correct
boundaries
and
keep
the
responsibility
separate
is
where
we
Tessa's
innovation
lies
and
the
last,
but
not
the
least,
is
what
I
would
call
survivability,
which
means
that
no
single
component
here
is
essential
at
a
given
point
of
time.
So
anytime,
we
expect
to
lose
components
here
and
as
they
lose,
every
other
part
of
the
system
knows
how
to
handle
the
fact
that
that
component
went
away.
A
So
those
are
the
three
principles
on
which
this
architecture
was
built
and
I'm
going
to
go
through
the
scenarios
that
I
talked
about
in
the
previous
slides
about
the
problems
that
existed
and
how
they
get
addressed
in
with
test
right,
for
example,
working
backwards
when
I
bring
up
a
new
VT
tablet
as
a
new
replica.
So
what
we
test
will
do
is
it'll
say:
oh
this
VT
tablet,
my
identity
is
I,
am
keyspace
KS,
shard
0.
A
Let
me
go
see
if
there
is
a
backup
so
which
means
that
there
exists
a
way
for
you
to
perform
a
backup.
So
you
could
go
to
VD
tablet
and
say:
please
do
a
backup
of
this
and
MIDI
tablet
will
perform
the
backup
and
store
the
identity
and
necessary
positions
needed
inside
the
back
up
when
the
new
VD
tablet
comes
up,
it'll
find
that
backup
and
restore
it.
A
So
it
has
a
starting
point
and
then
it'll
point
itself
to
the
master
and
says
based
on
the
bin
logs,
give
me
all
the
events
that
have
taken
place
and
it'll
catch
up
as
soon
as
it
is
caught
up.
What
it
does
is
it
advertises
to
the
topology
saying
that
hey
I
am
ready
to
serve
traffic
and
who
is
watching
the
topology,
the
VT
gates,
and
they
discover
that?
Oh
there's,
a
new
VT
tablet
here,
I'm
going
to
start
start
sending
traffic
to
this
VT
tablet.
A
So
discovery
is
handled
by
this
fact
and
what
is
initialization
is
handled.
What
are
the
topic?
Did
we
talk
about
repairing
so
repenting?
Is
the
next
story
like
what,
if
a
master
crashes
like
if
a
master,
the
one
thing
that
we
found
out
is
in
our
original
design?
I
talked
about
cultural
changes
right
in
our
original
design.
When
a
master
went
down,
you
said:
oh,
all,
you
have
to
do
is
save
it
in
the
topology
and
the
mitigates
we'll
all
get
it.
But
topologists
are
not
meant
to
react
that
fast
they
can
they.
A
You
can
be
like
15
minutes
down,
even
in
Google
they
they're.
The
SLA
is
like
15
minutes
before
you
can
get
an
update
from
a
tow
from
a
topology
change.
You
can't
go
15
minutes
without
serving
traffic
right
master
went
done,
I
need
to
react
quickly,
so
what
did
we
do?
Is
every
VT
gate.
That's
connected
to.
Vd
tablet
has
a
back
channel
where
the
VD
tablets
are
continuously
feeding
health
information,
both
about
how
healthy
they
are
and
also
about
who
is
the
master.
A
So
if
a
reparent
happens
as
soon
as
a
VT
tablet
realizes
that
it's
a
master
it'll
tell
the
vita
gate,
I'm
the
master
now
and
we
had
a
situation
where
sometimes
network
partition
right.
There's
a
network
partition.
The
old
cui
T
tablet
see
they
know
I'm
the
master,
there's
two
people
claiming
they
are
the
master.
A
So
this
is
where
packs
comes
to
our
help,
where
we
say:
what's
your
master
timestamp,
what's
your
master
times
time
I'm
going
to
trust
you
because
you
have
a
newer
timestamp
so
that
way
the
system
desire,
greats
and
safely
continues
forward.
So
that's
the
repenting
story.
So,
finally,
as
far
as
storage
is
concerned,
okay,
I'm
good
in
time.
So
as
far
as
storage
is
concerned,
you
can
run
with
us.
Both
using
mounted
volume
or
local
storage
mounted
was
so
people,
don't
trust
kubernetes
enough.
A
They
still
fear
that
there
may
be
this
Black
Swan
incident
where
the
entire
kubernetes
crashes,
so
they
still
have
that
fear.
So
they
still
have
the
fear
that
they
may
lose
all
the
nodes
at
the
same
time.
So
because
of
that,
most
people
that
run
with
tests
still
put
their
storage
outside
they
they
use
mounted
volumes
because
they
trust
the
mounted
volumes
for,
of
course,
that
trust
was
recently
broken
by
certain
cloud
providers.
A
I'm
not
going
to
go
into
that,
but
at
the
end
of
the
day
it
is
software
that
is
running
both
whether
it's
outside
or
inside.
So
you
can
run
with
tests
with
a
mounted
volume.
As
a
matter
of
fact,
you
are
it's
better
to
run
a
mounted
volume
using
with
tests,
because
you
are
so
widely
sharded
that
your
latency
sensitivity
comes
down,
whereas,
if
you
just
a
thing,
you
have
a
single
monolithic,
my
sequel
instance
there,
the
latency
becomes
even
more
significant.
So
and
in
that
case
many
of
these
stories
happen.
A
But
the
case
where
there
are
there
is
a
way
to
run
with
us,
which
is
how
we
ran
it
at
YouTube,
where
we
just
put
it
on
ephemera,
storage,
local
storage
and
semi,
sync
lytic,
like
I,
explained
before,
and
just
play
the
game
of
survival
through
numbers.
You
know
there
is
no
way
Borg
is
going
to
bring
down
all
my
twenty
replicas
at
the
same
time.
So
in
that
case,
then,
there
is
no
need
to
consider
data.
A
As
sacred
can
say,
data
is
disposable
because
I
can
rebuild
it
I
because
there
are,
the
shards
are
small,
200
300
gigs,
the
shards
as
soon
as
a
pod
goes
down.
I
can
rebuild
it
in
like
10
minutes
so,
which
means
that
my
exposure
to
catastrophic
events
is
highly
reduced,
and
there
is
one
more
point
I
wanted
to
make
about
about
this.
Yes,
so
I
I,
it's
possible
that
this
was
covered
in
the
slack
talk.
So
what?
What
is
the
advantage
of
treating
your
local
data
as
ephemeral?
A
A
So
we
can
run
my
sequel
in
crash
unsafe
mode
because
we
never,
we
are
never
going
to
recover.
So
why
make
it
crash
safe,
so
crash,
unsafe
mode
means
that
you
can
use
buffered
I/o
turn
off
all
crash
safety
and
with
buffered
I/o
every
commit
is
in
memory
operation.
So
it's
now
going
to
scream
faster
than
even
a
on
train,
my
sequel
and
we
had
those
kinds
of
numbers.
A
When
you
were
running
at
YouTube
like
we
had
brought
the
latency
down
to
like
really
really
tiny
numbers
like
Maya,
my
sequel
would
react
in
like
microseconds,
which
is
like
memory
right
and
I
am
ET.
Tablet
was
on
the
same
machine
locals
unique,
socket
issue
command
instantly,
get
back
screaming
fast.
This
is
a
subliminal
message.
Nobody
saw
that
message.
A
A
A
It's
called
when
software
engineers
fat-finger
things
and
accidentally
wipe
up
an
entire
column,
never
happened
before
so
at
that
time
you
want
to
go
back
to
your
old
data,
to
restore
it
so
actually
Deepti
who's
going
to
be
talking
tomorrow
has
done
a
feature
where
she
she
implemented
a
point
in
time.
Recovery
where
you
say
I
want
to
create
a
snapshot
of
this
key
space.
This
is
my
cutoff
time,
even
though
the
backups
were
taken
at
different
points
in
time.
They
are
approximately
at
the
same
time
and
that
is
usually
sufficient
for
a
recovery.
A
So
on
top
of
this,
we
are
going
to
add
a
feature
where
it'll
actually
phosphoryl
up
to
the
time
that
you
wanted
the
snapshot
to
be
to
happen.
So
that
part
we
have
not
got
done
yet,
but
you
can
actually
restore
a
key
space
as
of
asada
as
a
fern
approximate
point
in
time
they
accurate
yes,
okay,
cool,
yes,.
A
A
A
Yes,
would
you
consider
adding
relational
databases?
Absolutely
we
are
very,
very
seriously
thinking
about
Postgres.
There
has
been
many
people,
who've
been
asking
about
it.
We
currently
have
our
hands
full,
but
at
some
point
there
will
come
a
time
when
we
are
going
to
have
spare
engineering
bad
with',
and
we
will
definitely
look
at
that.
Yes,.
A
The
question
is:
where
do
you
start
the
backups?
Where
do
they
store
the
backups?
The?
Where
does
the
backups
can
be
stored?
Oh
that's,
the
other,
that's
one
of
the
things
that
I
did
I
forgot
to
cover,
which
is
when
we
moved
from
when
we
moved
into
Borg
and
had
to
build
all
these
abstraction
layers.
We
built
an
abstraction
layer,
for
example,
for
the
topo
server,
which
means
that
we
support
zookeeper.
A
We
support
console,
we
supported
CD
so
that
we
got
for
free
because
we
had
to
build
that
abstraction
layer
in
the
same
way
for
backups,
we
built
some
abstraction
layers,
which
means
that
you
can
back
it
up
to
a
mounted
filesystem
or
you
can
back
it
up
to
s3.
There's
one
more
storage,
Google
Cloud
Storage
is
the
other
one
and
also
there's
different
backup
methods.
There's
the
regular
backup
method,
which
is
stop
my
sequel
and
run
it
and
there's
the
extra
backup
which
can
actually
do
the
backup,
while
the
server
is
continuing
to
serve
queries.
A
A
Good
question,
so
you
are
saying
that
with
semi
sink
replication,
only
one
other
replica
is
guaranteed
to
have
up-to-date
data.
And
how
do
you
make
sure
that
you
don't
lose
any
data
in
terms
of
even
if
you
have
80
replicas?
The
transaction
that
has
committed
just
now
is
is
likely
is
possibly
they're
only
in
in
one
other
replica.
So
this
goes
back
to
Google's
philosophy,
which
is
typically
Google,
says
this.
So
this
has
worked
well
in
Borg.
A
A
It
just
happened
that
we
have
no,
it
has
never
happened,
losing
an
entire
data
center
Google
actually
has
the
freedom
to
come
and
take
down
a
full
data
center,
but
you
have
notice
so
we
get
notice
and
then
we
fail
over
to
another
cell
and
eventually
the
they
call
these
PCRs.
They
come
in
and
take
our
our
self.
But
during
my
years
at
YouTube
they,
an
entire
cell,
never
went
down
without
whatever
without
expectation.
Yes,.
A
Right,
yes,
so
that's
an
excellent
question:
can
you
run
with
us
in
hybrid
mode
I've
been
seriously
thinking
about
that
where
you
run
ephemeral
everywhere,
but
have
one
replica
that
is
writing
to
a
mounted
volume?
That's
completely
viable.
You
need
a
slightly
more
complicated
scripts
to
manage
that,
but
I
think
it's
an
excellent
idea.
Yeah!
A
A
Yes,
sorry
because
I
wanted
to
cover
the
cloud
aspect:
I
skipped
the
rest
of
the
parts.
So
the
way
this
works
is
the
topology
has
information
about
key
spaces
and
shards,
and
these
VT
tablets
register
themselves
and
VTA
mitigates
eventually
discover
them
and
send
traffic
to
them.
The
the
topology
can
be
as
I
say,
read,
CD,
zookeeper
or
console
veetc.
Tld
is
basically
a
dashboard.
It
reads
the
topology
and
displays
what's
in
there
for
you
to
browse.
A
Eventually,
you
can
discover
all
the
VT
tablets
that
exist
and
all
the
VT,
not
the
VT
gates
and
I,
can
explain
why
the
VT
gates
are
not
part
of
the
topology
and
you
can
also
initiate
workflows.
You
can
go,
tell
VT
c
TLD.
I
want
to
do
a
recharging.
I
want
to
create
new
shards.
I
want
to
split
these
two
in
things
you
can
do
you
can
through
VT
c
@e
c
TLD,
you
can
say,
create
a
backup.
So
all
those
commands
happen
through
VT
c
TLD.
A
There
are
a
few
more
components
like
VD,
worker
and
stuff.
They
essentially
are
mini
VT
c
TL
DS.
That
perform
very
specific
tasks.
Oh
that's!
That's!
What
this
this
architecture
is
about,
and
the
only
reason
VT
tablet
exists
is
for
housekeeping
like
it
does
the
backup,
restore
and
and
those
things
that
is
its
main
purpose.
A
Does
Vitas
have
the
ability
to
migrate
entire
data
through
teardown
of
an
existing
cluster,
as
in
is?
Is
this
an
external
data
type
or
internal?
Is
it
like
a
separate
my
sequel
instance,
or
is
it
itself
Oh
within
itself?
It
has
the
concept
of
cells
and
all
cells
are
keeping
up
with
the
master
wherever
the
master
is.
You
can
failover
a
master
from
one
cell
to
another
and
completely
tear
down
that
that
cell,
and
that
happens
all
the
time
at
at
YouTube.
We
take
down
cells.
Sometimes
we
decommission
them.
A
A
Pardon,
oh,
how
do
you
do?
How
do
you
do
multiple
cloud
o
query
across
multiple
shots,
so
that
is
a
challenge
that
is
I
would
say
the
curse
of
a
Charlotte
system.
The
curse
of
a
Charlotte
system
is
that
when
you
make
a
decision
to
say
that
data
a
lives
here
and
data
be
lives
there.
That
means
that
if
you
have
to
join
the
two,
you
are
going
to
go
across
charts.
A
You
can
say
materialized
this
table
from
here
to
there
and
keep
it
up-to-date,
real-time,
you
just
give
it
a
select
query
and
say
do
this,
and
with
this
will
materialize
that
entire
table
here
and
what
it
will
do
is
it
works
just
like
replication.
It
subscribes
to
the
bin
locks
and
watches
the
events
that
are
coming
and
immediately
updates
the
target
table.
So
you
can
do
that
for
filtering.
You
can
do
that
for
recharging.
You
can
do
that
for
aggregations.
You
can
actually
do
real-time
roll-ups
and
it's
extremely
efficient
at
keeping
the
target
table
up-to-date.
A
A
First
question
is:
are
there
is
there
any
recommendation
on
the
number
of
shards?
No,
the
recommendation
is
about
your
shard
size.
So
typically
we
say
about
two
to
three
hundred
gigs
is
the
shard
size
and,
depending
on
how
big
your
data
is,
you
have
as
many
shards
as
you
need.
People
have
the
largest
instances
that
I've
heard
of
have
hundreds
of
shards
and
we
test
has
served
comfortably.
There
are
a
few
people
who
are
going
into
the
thousands
of
shards
range.
A
We
will
see
how
that
goes,
but
I'm
I,
don't
know
of
any
architectural
limitation
that'll
cause
we
test
to
struggle.
If
there
are
too
many
shards
I
think
Raphael
fixed
one
bug
in
one
place
about
parallelism,
but
what
single
shard?
So
if
there
is
any
issues
there
will
be
minor
fixes
that
we
can
probably
make
your
second
question.
I
didn't
get
Oh.
A
Correct
the
question
is:
are
there
any
operational
concerns?
No
you,
the
number
of
shots
doesn't
cause
any
operational
overheads
is
because
the
effective
resource
used
is
the
same
amount
of
resource
used,
is
the
same,
there's
very
little
static,
cost
added
by
the
fact
of
the
existence
of
this
components.
A
This
is
because
my
sequel
is
capable
of
scaling
itself
down,
really
really
well
the
place
where
we
found
where
the
static
cost
of
my
sickle
was
beginning
to
show
up
was
when
your
data
size
was
below
70
gigs,
so
Oh
money,
wise
money,
wise
some
people
have
complained
that
the
TAS
does
burn
a
little
more
CPU
than
usual.
You
have
identified
the
root
cause
as
gr
PC
and
we
have
to
figure
out
what
in
gr
PC
is
causing
that
that
burn
so
we'll
get
to
the
bottom
of
it
very
soon.