►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: etcd - Xiang Li, Alibaba & Wenjia Zhang, Google
This is a Q&A style session where etcd maintainers will discuss frequently asked questions and interesting ideas around the project. We will leave some time for the audience to ask questions, bring concerns, or propose new features! Come and join us to learn more about etcd!
To learn more: https://sched.co/JAo2
A
I'm
from
Alabama,
so
I
work
on
kubernetes
and
sed,
so
pretty
I
work
for
chorus
and
she's
Indian
from
Google,
and
she
also
works
on
kubernetes
and
ICD
okay.
So
today
we
are
going
to
talk
about
a
city
and
this
session
is
Q&A
like
sessions.
So
we
have
some
prepared
questions
and
those
questions
are
frequently
asked
questions.
A
We
will
force
go
through
some
of
these
questions
pretty
quickly
and
we'll
leave
time,
for
you
guys
to
ask
us
questions
so
actually,
besides
out,
there
are
other
familiar
faces
here,
a
few,
a
city
maintainer
in
the
front
line.
So
maybe
they
also
can
answer
your
questions.
So
ask
me:
ask
us
easy
questions
or
hard
questions
or
whatever,
okay.
So
here's
the
fourth
question
yeah.
B
So
we're
gonna
start
with
the
easiest
questions
ever
but
could
be
confusion.
This
is
a
website.
That's
deprecated,
for
official
SD
documentation,
so
you're
taking
pictures
don't
take
pictures
of
this
one.
Also
it
says
it's
the
latest:
it's
not
the
latest.
We
stopped
maintaining
that
website
and
our
new
website
is
right.
Under
the
SED
ripple
documentation.
B
B
B
B
Excuse
me-
and
it
only
makes
sense
if
the
excuse
me,
it
only
makes
sense
if
the
underneath
datastore
is
also
hi
available
as
well,
and
ICD.
Clustering
is
intended
for
high
availability.
So
here
we
go
that
fits
the
needs
for
kubernetes
and
then
the
second
reason
is
that
it's
a
watch
feature
that
I
see
implemented.
We
know
that
one
of
the
core
design
concept
of
core
Nettie's
is
level
triggering
and
software
conciliation
to
do
that.
B
So
every
time
there
is
some
change
to
a
key,
the
Watchers
get
notified
which
exactly
what
coronaries
needs,
and
so
I
showed
this
diagram
of
the
talk
between
API
server
in
communities
and
EDD.
So
they
have
this
watch
stream
established
between
them
and
then
API
server
in
kubernetes
establish
the
watch
in
that
string
and
get
the
watched
event
from
that
stream.
There
is
a
talk
tomorrow
afternoon.
The
last
talk
of
koepka
talking
about
more
details
of
watching,
create
ease,
welcome
to
join.
A
A
So
the
next
question:
what
about
war
files?
You
probably
heard
a
lot
like
okay,
I
said:
II
have
the
concept
of
war
files
and
I
city
needs
to
sync
the
war
files
down
to
the
DS.
So
what
are
war
files?
So
warfare
is
so
firmly
conceptually
just
a
file
on
disk
and
instead
of
file,
there's
a
list
of
entries
and
each
entry
is
action
and,
for
example,
side
foo
equals
bar
or
sec
kubernetes
equals
awesome
and
for
those
entries
in
the
war
file,
that's
an
internal
I
city,
key
lost.
All.
A
So
if
the
clients
and
one
requires
twice,
it
is
over
one,
it
will
go
through
raft
and
other
country
in
the
war
file.
And
similarly,
if
you
send
to
the
server
machine
it
will
get
on
to
the
war
file,
and
this,
if
you
send
to
the
second
one,
it
will
go
through
route
again
and
a
plan
I've
hand
on
the
waffle.
So
you
will
see
that
the
war
file
will
be
the
same
on
all
the
machines
and
all
the
machines
will
add
cute
the
war
file
in
the
same
sequence.
A
B
B
Alright,
alright,
good,
no
matter
if
you
use
that
or
not,
there
are
some
information
for
you
in
this
topic.
Thus
SCE
work
with
cross
region
deployment.
The
answer
is
big,
yes,
not
owning.
That
sed
could
do
that.
It
also
brings
you
for
tolerancing,
prevent
if
improvement
well.
So
this
is.
This
is
supposed
to
be
like
a
representation
of
the.
C
B
So
if
you
do
the
high
latency
the
appointment,
then
it's
possible
that
you
will
have
more
frequently
during
elections
and
you
will
have
heartbeat
timeout.
It
doesn't
mean
you
cannot
do
that.
It
only
means
that
you
have
to
to
some
of
the
parameters
to
adjust
for
your
deployment
and
we
do
have
official
documentation
how
to
help
you,
how
to
do
the
tuning
tunings
and
it's
a
big
topic
of
how
to
upgrade
and
downgrade
tesa
t.
B
There's
two
important
things
that
you
need
to
keep
in
mind.
One
y
is,
you
can
only
do
rolling
update,
which
means,
if
you
have
a
cluster
of
a
couple
of
different
members,
then
you
have
to
upgrade
the
members
one
by
one,
and
you
can
only
do
one
minor
version
upgrade
at
this
time,
which
means,
if
you
have
a
3.1
cluster
and
you
want
to
eventually
upgrade
it
to
3.3,
then
you
have
to
operate
it
to
3.2.
First,
you
have
to
be.
You
know
1
years
old
before
you
get
to
2
years
old,
all
right
now.
B
Let's
take
an
example.
If
you
have
have
a
cluster
of
three
members,
each
member
have
their
own
member
version
and
then
the
entire
cluster
have
a
cluster
version.
The
class
version
is
decided
by
the
lowest
member
version,
which
means
in
this
case
like,
let's
see,
we
start
with
a
cluster
version
of
3.2,
and
you
start
to
do
the
rolling
update
and
to
update
one
of
the
members
to
3.3
and
the
entire
cluster
is
do
3.2,
because
the
other
two
us
do
a
three
point
two.
B
So
all
the
members
can
only
support
3.2
features
and
data
files.
Then
you
update
another
ones.
Do
we
have
the
lowest
version
of
3.2,
not
into
you
successfully
upgraded
all
the
members
to
343
that
the
class
or
version
would
change
to
3.3,
and
we
do
have
very
detailed
documentation
on
how
do
you
upgrade
your
Cassidy
from
3.1
to
2003-2004
and
it
not
only
document
the
detailed
steps,
but
only
like
the
all
the
things
you
need
to
keep
in
mind,
because
change
is
paid
between
different
versions
so
make
sure
you
look
into
that
and
for
downgrade.
B
We
only
support
done
great
with
downtime
at
this
moment,
which
means
you
have
to
stop
everything,
bring
up
your
done
great
sed
with
the
lower
version
and
then
start
everything
again
and
the
downgrade
with
no
downtime
is
on
the
way
and
it
will
be
out
in
v3
point
4
in
which
is
going
to
be
next
year,
and
you
can
refer
to
issue
nine
three
zero.
Six
for
the
detail
of
this
effort.
Any
contribution
suggestions
are
welcome.
A
Ok,
the
next
question
is
what
is
the
database
fell.
You've
probably
heard
that
okay,
besides
of
warfare
I,
said
he
also
has
database
file
right.
So
I
just
mentioned
that
in
the
war
file
it
stores
all
the
actions
that
city
will
take
and
if
you
think
that
ok,
if
I
run
a
city
cluster
for
twelve
four
hours
and
if
I
restart
the
city
node
does
it
need
to
replay
our
logs
for
the
24
hours
right
if
it
needs,
then
you
will
take
a
long
time.
Also
a
new
member
joins
the
cluster.
A
A
A
So
the
next
commonly
asked
question
is:
why
does
a
CD
have
a
data
set
limit
so,
as
I
just
mentioned,
one
new
number
joins
the
cluster
is
needs
to
get
the
entire
database
file
right
and
if
the
database
fell
it's
very
large,
it
will
take
a
long
time
to
downloading
the
file
and
to
open
the
file,
and
you
NASA
t
what
we
really
care.
It's
about
the
mean
time
to
recover.
A
A
If
you
still
under
stream
it,
you
are
good
to
go
and
another
limit
of
the
data
size
is
your
memory,
so
ICD
data
is
actually
mapped,
which
means
that
we
expect
all
your
data
sitting
inside
your
memory
so
make
sure
that
your
memory
is
larger
than
the
database
file
or
ladders
and
the
total
data
you
put
into
a
city
so
that
a
city
can
still
have
a
successful
reads
in
the
next
question
is:
why
does
ICD
require
compaction?
So,
yes,
it
is
three.
A
We
started,
keep
all
versions
of
keys,
which
means
that
if
you
change
the
value
of
a
key,
your
old
value
of
that
key
will
still
be
there.
You
can
still
access
them
by
pointing
to
a
previous
version,
see
right
now.
The
wording
is
three.
You
add
a
key.
You
change
the
key
that
we're
gonna,
be
for,
if
you
still
how
actually,
okay
I
won't
get
the
we're
going.
Three
of
that
key,
you
will
still
see
the
old
version.
This
feature
is
designed
for
configuring.
A
A
So,
but
when
we
start
with
like
keep
all
the
versions,
if
we
don't
do
compaction,
you
you,
you
can
think
that
the
database
size
will
keep
on
growing
and
growing
and
it
never
shrink
right.
So
compaction
is
to
remove
the
old
version
of
keys.
So
when
you
know
that
ok,
those
keys
and
not
natural
anymore,
you
can
compact
them
and
the
nicety
will
start
to
leave
them
in
background.
It
owns
such
a
TDD.
They
will
take
some
time,
but
its
own
block
any
rights
in
the
future.
It
all
it
done
in
bagging.
A
A
Right
now,
I
think
we
matter
the
total
number
of
keys
and
we
measure
the
total
number
of
keys
in
the
body
be
by
hand.
So
if
you
do
the
maths,
you
will
know
there
how
many
keys
and
like
I'm
not
like
it's
not
unused.
We
don't
know
whether
you
are
using
it
or
not,
but
it's
old
keys,
so
you
can
do
the
best
to
total
how
many
old
keys
out
there
right,
but
those
old
kids
maybe
still
be
useful
right,
maybe
we'll
add
a
better
API
to
tell
you
that.
A
A
So
another
question
is
like
why,
as
it
is
sensitive
to
I
all
agencies,
so
ICD
is
sensitive
to
the
network.
I/O
latency,
because
we
use
wrapped
as
the
underlying
consensus,
algorithm
and
draft,
is
a
leader
based
system.
So
the
leader
needs
to
keep
on
standing
harvest
to
the
follower,
and
so
normally
it's
a
follow.
We
receive
it
within
the
timeout
and
followers
stay
in
the
follower
stage.
A
However,
if
the
I/o
latency
is
long
and
the
fuller
can
now
receive
the
heartbeat
in
time,
it
will
try
to
become
a
candidate
which
can
disrupt
the
cluster
stage
and
that's
why,
as
it
is
sensitive
to
I/o
latency.
So
in
3.4
we
are
adding
a
feature
called
promote
to
medicate
it's
problem,
so
the
candidates
node
will
not
disrupt
the
leader,
the
cluster
right,
but
still
it
will
become
candidate.
A
A
We
don't
have
it
now,
but
we
may
want
to
provide
a
to
to
help.
You
basically
is
your
depend
on.
You
are
like
long
trip
time
between
the
two
notes.
We
can
have
a
tool
to
help
you
to
do
that,
so
we
suggest
to
have
the
election
time
out
ten
times
to
your
round
trip
time
and
hardly
time
around
the
round
trip
time
of
your
latency,
because
that's
our
suggested
value,
but
we
don't
have
a
tool
to
automatic
color
there
for
you,
but
we
can
help
you
to
do
that.
So.
A
F
A
Don't
remember,
I,
don't
think
if
there's
a
matrix
for
that,
but
it
it's
configurable
but
I,
don't
think
it's
it's
exposed.
But
when
you
are
reaching
the
time
out,
sed
will
print
out
a
log
saying
that
okay
is
near
the
time.
Outs,
there's
a
warning
in
the
log,
but
we
can
expose
it
as
a
matrix,
but
for
the
Iowa
latency
we
do
expose
the
matrix.
So
no
the
current
latency
between
your
two
notes,
but
we
don't
expose
the
compute
your
config,
it
action
timeout.
A
B
The
performance
is
always
a
very
important
topic
as
more
and
more
people
are
using
in
protection.
We
not
only
want
it
to
work,
but
also
it
better
work
efficiently.
So
there
are
a
couple
of
important
numbers
to
remind
you,
the
two
performance
factors
or
latency
and
throughput
like
anything
else,
and
then
there
are
two
physical
constraints
for
SD
performance.
Networking,
I/o
and
disk
I/o
make
sure
you
understand
the
scenario
in
your
environment
and
then
we
do
have
a
benchmark
tool
in
sed
ripple.
B
Suggest
you
to
oh
yeah
and,
along
with
the
benchmark
tool,
the
there
are
current
benchmark
results
posted
on
that
documentation
as
well.
It
was
based
on
our
certain
set
of
configuration
to
make
sure
you
cross-reference,
with
whatever
the
hot
wall
configuration
that
you
have
and
refer
to
the
result
and-
and
you
know,
reason
the
the
relationship
between
them
and
make
judgment.
What,
if
the
performance,
meets
your
requirement,
and
we
do
suggest
you
like
every
time
you
change
your
environment
or
change
your.
B
How
well
configuration
do
the
benchmark
performance
again,
make
sure
it's
right
for
you,
and
this
is
supposed
to
be
the
last
sites.
But
then
we
do
have
two
extra
slides
I
decide
that
it's
important
enough
to
mention
before
we
get
to
into
the
real
Q&A
session.
This
is
how
you
could
help
with
the
sed
communities.
I
did
this
little
survey
before
we
start
the
talk.
Apparently
there
are
a
couple
people
already
contributing
to
se
and
everybody
else
is
planning
to
contribute
to
sed.
So
we
have
the
contact
here.
B
You
can
email
to
SD
dev
Google
Groups
and
we
had
a
community
meeting
before
bi-weekly
and
we
are
planning
to
reassume
the
community
meeting.
As
we
see
a
lot
of
people
are
more
and
more
interested,
so
it
will
start
since
it's
holiday
season,
we'll
start
right
after
the
holiday
season,
January
8th
at
11
o'clock
in
the
morning
Pacific
time-
and
it
will
be
mostly
me
and
then
there
you
can
go
to
that
link.
It
will
have
like
we.
You
really
have
agenda.
You
can
put
your
agenda
in
there.
B
Don't
think
you
can
only
contribute
that
if
you're
writing
coats
right,
if
you
post
your
use
case,
you
post
questions,
we
know
what's
going
on
outside
there,
everybody
using
a
CD,
different
ways,
everybody
using
cornetist,
different
ways,
so
please
post
stuff
there
and
contribute,
and
with
that
there
are
some
areas
that
you
could
contribute
right
away
like.
We
do
need
a
lot
of
tests
and
well,
if
we
don't
know
your
use
case,
we
don't
know
whether
there
are
tests
we
can
do.
B
And
then,
as
I
mentioned,
there
is
a
new
documentation
website
in
progress,
and
you
can
take
a
look
at
that.
You
can
contact
your
home
for
if
you
want
to
contribute,
if
you
have
comments
and
then
we
do
want
to
have
more
documentation
on
sed
metrics,
we
know
people
might
have
questions
about
which
what
which
metrics
you
want
to
see
which
one
metrics
you
want
to
use
in
which
metrics
you
need
to
monitor
an
alert.
B
B
H
B
A
So
I
just
mentioned
that
we
have
a
know:
I
have
my
phone,
so
we
have
the
database
file
right
now
is
it's
a
p
+
3
+
4,
p
+
3.
It
requires
random,
writes
when
you
merge
the
new
memory
back
to
the
on
these,
and
that
will
cause
a
lot
of
random.
Writes.
I
owe
an
SSD
is
better
for
that.
So
if
we
want
to
solve
this
problem,
we
may
add
some
other,
like
our
SM
based
backhand,
which
handle
sounds
like
change.
The
secret
random
ride
pattern
to
ask.
A
I
I
have
a
question
well
like
cross
region
use
case.
Is
there
any
way
to
rank
the
Horta
like
rank
the
members
that
the
client
is
talking
to
you
by
preference,
because
we
found
that
like?
If
you
have
you
know,
know
their
members
in
a
bunch
of
regions,
and
you
establish,
like
you,
happen
to
establish
a
watch
cross
region
that
it's
much
more
expensive
than
establishing
it
locally,
but
we
still
want
the
clients
to
be
aware
of
all
regions
because,
like
if
the
local
region
member
is
down,
we
still
want
like.
A
Yeah
I
say
to
the
deal
whatever
I
said:
yeah
I,
think
I,
totally
Adam
feature
to
I,
say
decline
to
have
up
power
is
on
different
members.
So
probably
also
want
do
this
for
the
SATs
three
I'm
not
sure
like.
If
we
have
plans
for
that,
because
Kehoe
is
just
rewrite
the
GRP
say
bike
and
the
GRP
see
balancer
floor
I
see
declines.
Probably
after
this
rewrite,
we
can
add
the
feature
you
like
to
see
the
setting
the
power
is
for
the
deployment.
A
J
A
K
Thank
you.
We
have
a
use
case
with
a
regular
each
CD
supporting
kubernetes
in
sometimes
when
we
take
one
of
the
members
for
maintenance.
We
have
seen
that
worker
nodes
in
master
nodes
start
to
getting
some
timeouts,
even
when
they
are
talking
to
each
CD
proxy,
which
should
forward
connections
to
whatever
remember
it's
available.
Have
you
guys
seen
anything
or
hear
anything
like
that
where
each
CD
proxy
is
not
forwarding
necessarily
to
their
write
members?
Maybe.
A
There's
a
bug:
the
proxy
is
not
super
super
reliable.
So
for
us
we
just
use
a
CD
client
directly
and
for
some
of
our
deployments
we
even
have
the
API
server
talk
to
the
s
ad
cluster
just
locally,
so
we
collocates
the
a
CD
so
with
the
EBS
over.
So
we
haven't
like
test
the
proxy
intensively,
but
if
you
hit
a
problem,
probably
crazy
once
in
a
city
repo,
so
we
can
help
to
look
into
the
problem
for
you.
Okay,
in.
K
On
the
same
cluster,
sometimes
I
see
like
a
regular
warning
on
each
CD,
saying
that
it
took
so
long
to
apply
entries
or
something
optimize,
your
queries
or
something.
But
this
is
kubernetes
talkin
to
it
and
I'm
not
gonna,
go
and
modify
the
query,
so
I
will
say
other
than
as
this
message
shows
up
every
time.
The
clusters
it's
getting
beakers,
so
I
just
throw
more
resources
to
each
see.
I
got
to
be
able
to
handle
this,
but
is
there
any
door
other
workaround
that
I
should
do
instead
of
just
throwing
more
hardware
on
it?
K
A
The
problem
is
sometimes
kubernetes.
Try
to
send
very
large
range
queries
to
a
city,
getting
large
number
of
keys
like
thousands
of
keys,
and
that
query
can
take
a
long
time
to
run
so
in
3.4.
We
are
working
out
between
for
a
feature
called
concurrent
read,
so
it
will
not
block.
So
when
you
issue
a
large
rate
twice
a
day,
even
our
blog
other
requests,
so
that
message
will
disappear,
but
for
now
you
probably
need
to
throw
more
recent
versity
to
vertically
skidding
it.
A
This
issue
also
will
be
medicated
at
the
kubernetes
side,
because
in
kubernetes
we
are
working
on
a
feature
called
pagination.
So
when
kubernetes
tries
to
get
a
lot
of
resources,
you
will
get
a
chunk
after
another
chunk.
It
will
now
try
to
issue
a
huge
range
elasticity
to
block
other
requests
yeah.
We.
C
A
H
A
For
the
indexing
problem,
there
are
several
ways
to
solve
it,
so
we
have
been
talking
about
this
for
quite
some
time.
So
first
one
way
is
to
add
the
indexing
to
kubernetes
api
server.
So
in
kubernetes
api
server
with
we
index
everything
have
a
b-tree
or
hash
table,
including
api
server.
The
second
thing
we
automatically
thing
we
can
do
is
to
make
a
CD
understand
portable
fee
encoding.
A
So
when
you
save
stuff
to
a
city,
is
you
register
the
key
space
with
a
probe
of
file
and
acidic
start
decode
the
proper
file,
and
we
can
start
building
index
internally
in
a
city?
So
there
are
two
alternatives
and
draw
enjoy
working
on
this
and
probably
we'll
publish
a
proposal
in
a
city
repo
and
to
see
whether
we
should
do
it
in
kubernetes
or
should
do
it
in
a
city.
But
we
are
solving
this
problem
right
now,.
F
Yeah,
we're
not
sure
which
alternatives
are
best
there's
the
two
that
Shawn
mentioned.
Also
the
idea
of
maybe
having
either
symlinks
or
labels
on
odd
objects
might
be
a
lighter
way
to
do
this,
but
we
will
open
an
issue
on
a
2d
and
anybody.
That's
interested
I
hope
you
jump
in
listen
in
give
us
your
suggestions.
M
A
A
If
you,
your
disk
I/o,
is
slow,
I
said
it
will
be
slow
because,
as
I
said,
so
for
each
write
request
as
it
needs
to
write
sink
down
to
the
disk
through
the
war
file,
if
your
disk
slow,
it
will
be
slow.
A
fast
SSD
is
a
good
enough
matter.
Your
latency
of
a
write
probably
should
be
under
one
millisecond
or
something
for
large
clusters
and
nvme
is
suggested.
H
A
You
have
to
turn
the
time.
Well
you
don't
have
to,
but
you
don't
have
to,
but
you're
through
food
will
be
lower
your
yes,
you
can
turn
to
me
to
see
whether
it
helps,
but
our
suggestion
still
like
choose
a
faster
disk.
Yes,
so
as
it
is
some
case,
the
disk
I
will
the
heartbeat,
but
mostly
I,
shouldn't
I,
see.
A
N
N
A
A
city
requires
alright,
expect
you
to
have
a
memory
size
larger
than
the
total
number
of
total
size
of
your
key
value
space,
including
the
old
keys,
so
say
your
database
size
is
two
gig.
You
have
to
have
for
like
six
three
key
of
memory,
one
gig
more
for
thumb
like
Headroom
right,
so
there's
a
matrix
to
show
that
how
much
disk
space
is
using
by
the
back-end
database.
Well,
if
stat
shows
like
to
get,
you
have
to
have
to
kick
memory.
A
So
one
easy
way
to
matter
whether
you
have
enough
memory
space
is
to
check
the
redial
from
the
ICD
process.
So
we
expect
to
see
the
readout
to
be
very,
very
low,
probably
lower
than
like
10
redial
per
second.
If
you
see
that
number
going
up,
it
means
that
ICD
is
swapping
out
pages
from
disk
to
memory
and
that
hurts
the
performance
of
a
CD.
So
you
know
ababa.
We
have
seen
that
okay,
I
City
is
swapping
pages
into
the
memory
and
spending
four
seconds.
A
O
A
From
two
plays
so
in
we
have
a
in
memory
cache.
It
reads
from
affair.
First,
if
you
hate
here
the
cache
and
there
is
the
database
file,
which
is
on
disk
I
point,
it
is
mm
mapped.
So
if
you
have
enough
memory
those
file,
those
pages
will
steal
your
memory
right
and
then
you
read
from
there.
So
that's
a
tool
place
that
a
raid
can
heat.
O
O
A
If
the
war
file
is
crop,
audacity
will
detect
patterns
and
how
the
okay,
the
warfare
is
crowded
because
you
see
all
CRC
checkpoint,
it
checks,
armed
for
the
database
fell
is
right.
Now
it's
not
fully
checksum.
So
here
maybe
you're
crafting
your
database.
Well,
but
we
add
another
feature
called
hash,
so
I
say
university:
you
keep
on
hashing
the
internal
state
and
compares
through
the
cluster.
If
the
tag.
Okay,
there
is
a
value
or
a
memory
in
the
class
I
have
a
different
cash
than
the
other
two
members.
P
Have
two
questions?
Okay,
so
one
is
I'm
going
to
determine
it
to
tell
what
granularity
at
City
right
performance
is
dependent
on
disk
I/o
latency.
So
is
that,
like
right,
recursive
set
to
have
computed
only
when
we
write
to
memory
and
later
it
can
have
sync
with
disk
or
write?
Requests
is
only
set
to
compute
when
the
data
has
been
dumped
on
the
disk.
A
A
P
P
B
Yeah
yeah
in
the
documentation
it's
very
detailed
recorded
and
then
it
have
documentation
for
upgrading
from
3
point
1
to
2
upon
2
and
I've
documentation
of
operating
from
two
point,
two
to
three
point:
three
and
even
documentation
from
how
to
upgrade
from
a
city
to
two
app
to
these
three.
And
then,
if
you
want
to
upgrade
from
three
point
one
or
two
for
two
point:
four,
for
instance,
then
you
have
to
go
through
like
each
stage
and
in
operating
tire
cluster
to
the
one
minor
version
upper
and
then
and
then
do
it.
Yeah.
Q
A
I
think
I
mentioned
just
mentioned
that,
like
for
ICD,
we
want
to
support
HTV
better
or
like
to
reduce
the
memory
usage
right,
so
better
horizon
and
based
backhand
will
be
better
fit
for
that
use
case,
but
our
I'm
based
a
pole
will
use
much
more
CPU
and
much
more
memory
for
your
normal
use
case.
So
for
small
clusters,
we'll
still
suggest
bode
B,
because
it's
where
lightweight,
but
for
that
our
deployment.
We
are
experimenting
with
our
I'm
based
horizon
based
approach.
N
A
R
F
A
Yes,
so
so
I
city
itself
is
replicating
data
to
all
the
knows
right,
so
normally
you
do
need
to
do
the
back
hub.
So
if
you,
when
you
do
the
back,
he'll
probably
do
bike
house
for
the
data
say
the
data
in
crack
knees,
for
example
for
kubernetes.
When
you
delete
all
your
ports
right,
then
you
want
to
recover
your
state.
Probably
you
won't
recover
it
from
a
backup
right.
So
backup
is
mostly
for
like
saving
your
application,
but
for
a
CD
it's
already
roughly
to
the
data.
So
you
don't.
A
If
you
don't,
if
you
think
that
your
up
killing
is
absolutely
cracking
enough
right,
you
don't
need
to
the
backup
and
bike
hub
does
cost
some
like
IO
I,
also
memory,
because
we
don't
do
it
is
need
to
read
the
data
and
send
to
the
other
node.
So
we
are
having
a
new
feature
called
a
non-voting
members
that
member
will
just
subscribe
to
the
raft
and
keep
on
like
getting
the
new
war
files
from
the
Robbins
and
keep
on
writing
to
disk.
Then
you
will
have
a
continuous
backup.
T
So
I
had
a
question,
so
we
see
in
our
production
that
when
we
have
lots
of
Watchers
watching
a
large
key,
the
size
of
the
value
is
large
and
when
we
update
the
key,
the
absolutely
service
needs
to
send
out
a
lot
of
pies.
During
that
instant
and
then
some
see
we
see
some
network
hiccups
and
even
cost
like
leader,
reelection
and
stuff.
Is
there
any
plans
to
handle
that
more
gracefully,
so.
A
I
think
some
of
the
missions
that
we
have
a
think
how
actually
proxy
that's
a
proxy
is
supposed
to
handle
this
exact
use
case.
So
so
we
have
a
lot
of
waters.
Subscriber
single
key
and
sed
server
now
needs
to
find
out
that
cause
a
lot
of
network
at
all
right.
So
we'll
have
a
proxy,
so
ICD
force
that
will
find
out
to
a
few
proxies
and
then
those
proxies
will
file
to
the
other
clients,
and
this
reduce
the
pressure
on
the
core
ICD
members,
and
this
is
so
try
the
proxy.
A
If
you
hit
an
issue
Krita
a
city
like
issue
in
the
Hat,
a
trip
home
will
help
you
to
fix
them,
yeah
so
I
protein
design
for
this.
But
we
don't
just
we
don't
have
enough
use
case.
For
example,
kubernetes
have
three
clients
right
and
it
doesn't
really
work
for
large,
even
if
it
worked
for
a
large
value
where
they
will
just
send
to
three
clients.
I.