►
From YouTube: The Secret Recipe of etcd - Junho Son, Line Plus
Description
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
The Secret Recipe of etcd - Junho Son, Line Plus
etcd is the heart of a kubernetes cluster.
However, most kubernetes endusers don't care much until the problem occurs.
And as the cluster grows, etcd sends small signals for changing configuration.
In order to detect the signal in advance and configure etcd for the state of the cluster, it is necessary to understand etcd.
In this session, He will talk about how etcd works and monitors, etcd tunning points for cluster conditions, and what user need to do to ensure stable operation.
Presenter Son Junho studied kubernetes at NCSOFT and built and operated a large cluster for web services.
Currently, He is developing and operating a cloud native app deployment service on a kubernetes cluster called Nucleo at Line Plus. He is studying the knowledge of what is needed to run a large kubernetes cluster.
B
B
B
B
B
B
B
B
B
So,,
if
you
check
later,
sd,
is
now
a
distributed.
Key
storage.,
The
origin
of
the
name
is
now
ucc
Unix's,
cd,
folder
and
then
d
in
the
district
system.
Ordering
of
metadata
that
continues
to
change.
It
is
now
a
key
distribution.
Key
value
store
used
when
adjustment
is
required.
So,.
The
big
advantage
is
that
it.
B
Has
the
advantage
of
maintaining
distributed
consistency
of
data
that
is
not
possible
for
data
and
generating
fast
response
and
high
throughput.
What
is
the
role
of
sd
in
First
of
all,?
There
are
three
main
things.,
It,
stores,,
retrieves,
and
replicates
data.
What
kind
of
data
is
the
Cuban
netis
cluster's,
compilation,
information,,
etc.
Then,
the
comment
cluster
az
These
source
objects
Then,.
If
If
you
use
cni
in
,,
Siena's
data
or
mall
outing
information,,
then,
if
you
are
using
a
controller,
user,
define
crd,
etc.
are
all
stored
in
cd.,
You.
A
B
What
happened
is
that
in
order
for
any
action,
such
as
saving
or
deleting
changes
to
take
place
in
the
distributed,
cluster,,
the
consent
of
the
quorum
equal
to
half
of
n
plus
1
of
the
number
of
nodes
constituting
the
cluster
is
required.
If.
You
look
at
the
surface,.
It
is
now
a
quorum,,
but
if
you
think
of
it
as
an
extra
majority,,
it
will
be
a
little
easier
to
understand..
It
is.
B
Now
difficult
to
explain
this
in
words,,
so
I
will
explain
it
through
pictures.
Storing.
The
data
of
the
data
and
maintaining
its
consistency
is
now
the
most
important
thing
in
the
injection
system.
So,
in
this
way,.
The
client
sends
the
leader
an
action
that
now
saves
the
keep
fuel.
When.
The
action
is
requested
to
the
leader,.
The
leader
now
forwards,
the
action
to
all
the
members
to
the
clients.
In.
B
This
way,
I
will
discuss
it
now
and
open
a
group
chat,,
and
what
data
has
come
in
from
the
suckers
and
I
will
save
that
data,
and
I
have
already
saved
it
and
say
to
the
conductor
and
give
me
a
response,,
and
that
response
is
the
current
nl,
because
there
are
3
nodes.
There
are
two
quorums.
So,.
If
you
answer
as
many
as
those
two,
Bryant.
B
Reason
for
the
leader
action
is
that
since
the
lcd
catches
all
changes
through
the
leader
and
then
the
leader
requests
all
changes
to
its
followers,,
the
follower
always
receives
a
heartbeat
from
the
leader.
So.
If
this
Please,,
if
the
leader
has
a
problem
and
ca,
n't,
deliver
the
heartbeat
anymore
If.
The
follower
can't
take
off
the
heartbeat
message
from
the
leader
now:
1
action,
timeout
4,
one
of
the
followers
as
soon
as
possible.
1
action
time
is
over
and
it's
a
tent
date.
B
B
Leader.
Send
your
first
year
Bits
to
your
followers
with
data
and
changes.
Then,
yesterday's
Hotu
Bits,.
If
the
leader
has
a
problem
with
the
current
one
and
won't
be
able
to
tweet
more,,
that
foot
will
now
be
a
Candidate.
I,
send
a
vote
saying
that
I
will
now
become
the
leader,
and
when
the
vote
is
answered,,
Can
Fate
becomes
the
leader
and
operates
again
in
this
way.
B
So,,
by
going
on
top
of
this
lobster,,
you
can
check
how
frequently
redirection
is
occurring.
Now.,
4
And.
The
third
is
to
complete
the
data
consistency
of
the
distributed.
Nodes.
sd
contains
the
log
and
replicates
it.
All.
The
changes
received
from
the
client
in
the
reader
are
now
stacked
in
the
wall.
Log,
and
the
pole
is
now
added
as
an
item
in
the
log,
and
then
the
added
entry
is
added
as
a
member.
B
Then,
after
requesting
two
duplicate,
copies,
Balo
writes
them
all
and
sends
a
response
to
the
leader,
again.
When.
The
response
reaches
as
much
as
the
Koran,.
The
corresponding
data
is
copied,
and
the
large
If
I
explain
this
through
a
picture,.
This
is
now
the
same
as
the
picture
I
explained,
earlier.
In.
B
This
way
4
In
this
way,.
Now
some
actions
or
entries
of
data
are
duplicated
Yes.
So
now
I
explained
the
basic
operation
method
to
sd
So.
Why
us
Then?
We
should
pay
a
little
more
attention
to
the
lcd
cluster
Now.
That's
the
point,!
It's
not
a
big
problem
when
the
worldis
cluster
is
created
at
first
and
simple
workloads
go.
B
B
B
That
happens,
load
is
now
generated
on
the
sd
cross.
Then,.
If
you
go
to
the
large
cluster,
of
course,,
you
will
need
an
sd
tuning,
point.
And.
You
can
see
warning
messages
from
time
to
time..
Now,
I
will
explain
the
failures
and
countermeasures
to
the
LCD
encountered
during
the
operation
of
large-capacity
Koons
Plus.
I
have.
B
Starts
getting
an
error
with
the
message:
These
contents
are
now
sd
server,
mvc
stator
base
l
space
x,
git
first
with
an
error
message:
Huber
netis,
kush,
tieri
or
now
corel
requesting
the
api
server
keeps
getting
error
messages,
so
The
symptom
is
usually
when
this
message,
pops
up,,
the
lcd
cluster,
becomes
a
lead-onie,,
so
it
becomes
a
lead-onie,.
So
it
becomes
impossible
to
change
the
color
of
the
foam
test.
Objects,
create
a
song,,
etc.
B
B
All
of
these
things
were
also
saved
to
the
card,,
but
as
a
result,.
Now
there
are
a
lot
of
g
pro
comments
and
crs,,
and
then
the
history
becomes
like
that,.
So
now
the
length
and
vision
yeast
have
increased.
Because
of
that,
dv
scg
is
now
&.
The
db
size
is
the
default
during
this
period,
and
now
all
of
them
are
used,.
So
some
of
the
member
nodes.
B
B
B
B
B
B
Ok
I
'll
explain
it
in
couple
problem:
t
4
and
the
second
now
the
second
is
now
sd
will
have
lost
the
forum
Literally.
Now
it's
a
case
where
the
node
is
dead
over
Bolt
Torrance
Actually,
we've
never
experienced
it,
but
I
guess
I
thought
that
this
would
definitely
happen,.
So
now
we
have
to
do
something
like
training,,
so
Hana
now
has
one
or
two
unusual
cases,.
So
I
put
this
part
in
to
reveal.
B
Message
that
the
introduction
cd
is
broken
in
the
api
is
removed.
The
cd
does
not
work.,
It
itself
is
impossible
and
it
cannot
be
looked
up
at
all.
So
if
sd
loses
its
steps.
Now,
Raff
consensus
is
impossible,,
so
as
I
said,
r
requests
such
as
append
entry
and
less
action,
are
not
possible..
If
the
dog
dies
or
just
the
process
of
the
node
is
dead,.
It's
2
o'clock
to
live
right,
away.,
It's,
starting
to
build
So,.
Now
there
are
two
ways
to
recover.
When
this
is
lost,,
one
is
sd.
B
B
This
is
one
of
the
methods
below.
In
fact,.
There
are
many
cases
in
which
Mr.'s
mine
is
not
saved
using
v3.,
Planar
and
Calico,,
etc.
that
I
used
are
usually
saved
in
v
toro,.
So
at
this
time,,
if
you
upload
only
v3
backup
from
restore,,
the
keys
of
v2
will
be
revived..
I
can't
do
it,,
so
you
have
to
put
the
above
2
separately
in
the
data
area
and
upload
it.
At
that
time,
I
think
this
part
was
the
point.
Then,
secondly,
sd
is
now
safe,
1.
We,
don't
have
anything
backed
up.
Since.
B
There
is
data
in
one
place,.
You
can
buy
it
through
data
in
2,
hours.
However,.
If
you
simply
upload
it,.
The
problem
is
that
this
is
all
the
information
of
that
member
of
the
column
is
stored,.
So
when
you
upload
one
unit,,
if
the
other
two
do
not
come
up,,
it
does
not
work.
So,
when
uploading,
give
the
option
of
i2
Post
New,
Christer,,
Post,
New,
Classter,
and
now
add
1
plus,,
which
has
one
member
node,
and
then
configure
the
node
again
and
attach
it
in
the
form
of
declaring
the
member.
B
B
B
The
reaction
is
constantly
happening,,
but
it
is
changing
outside
the
leader.
Usually,.
If
you
check
the
symptoms
in
this
case,
now,
the
cd
rater
is
very
high,,
and
now
the
candidate
netis
cluster's
resources
are
searched
or
created,,
or
these
things
slow,
down.
So,
now
check
the
cause.
If
you
do,,
there
are
about
two
big
things.
Now,
sd
has
to
be
a
little
cluster
configuration
and
the
failure
hole
must
be
toner,.
B
B
B
In
order
to
alleviate
this
symptom,,
it's
usually
more
comfortable
to
change
the
disk
to
ssd
now.
I'm
talking
about
it,
but
I'm,
going
to
live.
Yes,,
it's
comfortable
to
change
to
ssd,.
But
since
this
is
not
a
fundamental
solution,,
heartbeat
interval
and
relaxing
timeout
There
is
an
option
called
So,.
You
can
adjust
these
values.
Now,.
If
you
look
at
these
two
values,
on
the
Cours
site,,
the
guide
is.
B
B
Next,
I
will
explain
a
little
more
about
the
tuning
points.
As
I
said
before,.
My
Cuban
netis
cluster
is
sd
So,.
If
you
want
to
use
a
lot
of
metadata,,
it
is
better
to
raise
the
size
of
r
100
yen.
Basically,.
This
is
the
default.
Because
basically,
sd
is
more
than
a
key
veloster,,
so
the
size
is
not
set
that
large.
Yes,.
B
So
there
is
one
way.
Then,.
If
you
say
you
want
to
reduce
the
load
on
sd
as
much
as
possible,
now
developers
just
upload
various
workloads
at
will,.
So
the
history
song
data
is
not
well
managed..
If,
you
use
Flickr
3,.
The
previous
version
has
a
lot
of
gardens.
Then,.
If
you
use
something
like
Mercy
or
Chrome
Tab,,
if
it
fails,,
it
will
continue
to
try
again,.
So
you
have
to
give
more
meat
for
those
things,
and
in
the
case
of
Helm,
you
don't
give
the
revision
history,.
So
Helen
When,
you
ride,.
B
B
B
B
If,
you
do
and
then
register
the
cd
endpoint,
in
this
way,.
The
number
of
MVL
cc
database
sizes,,
then
the
number
of
keys
2,
and
then
what
file
syncs,
As
I
said.
Before,
all
syncs
are
left
as
data.
So
Using
that
data,
you
may
now
like
your
waist
and
do
something.
Then,
in
this
way,.
You
can
create
and
monitor
graphs
using
Prometheus
Grappa
Day.
So,.
You
can
manage
the
LCD
comfortably.