►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Lessons Learned from Operating ETCD - Pierre Zemb, OVHcloud
OVHcloud is the biggest European cloud provider. From dedicated servers to Managed Kubernetes, from VMware® based Hosted Private Cloud to OpenStack-based Public Cloud, we have over 1.4 million customers worldwide. Because of our Kubinception design(using Kubernetes to run Kubernetes), we are putting hundreds of customers in an ETCD cluster. This design is great to easily spawn control-planes for customers, but it is also putting a lot of pressure on ETCD. To keep it healthy while growing up constantly, we had to learn many things about how ETCD works under the hood and how we can operate it efficiently. In this talk, you will have the insights of how we are operating our ETCD clusters. We will tell you our journey to use ETCD, from our observability to deployments and management, what did work and what did not.
A
A
Quick
wins
about
me.
My
name
is
pierre
m.
By
the
times
american
and
his
arms
technical
to
the
soviet
leader
cloud
working
browns
distributed
systems,
ce
avec
spaces.
We
think,
like
it
days,
hadoop
flink
kafka,
pulsât
foundation
flow
to
now.
It's
et
les
giblets
qui
mesh
sit
in
breast,
a
piece
in
love
in
to
local
community
for
developers,
a
duel
that
gives
any
case
where
you
talk
about
your
muean
watching,
say
yes,
beast
in
tingwick
new
position
is
a
key
cloud
and
european
service
platform
company.
A
You
feel
free
to
check
them
out
in
the
waste.
Your
first
housing
units
idi
a
display
sign
of
cubes
and
ecosystems
august
20-.
Just
in
case
you
have,
we
built
de
manage
criminalize
project,
implication
novice
design,
onid
sidi.
Well,
yes,
yes,
move
on
everything.
We
learn
from
production
from
observables
and
titou
configuration
one
type,
soundchecks
quick,
wins
baltovich
cloud:
we
are
global
cloud
provider.
A
Let's
get
it
started
in
the.
What
is
it
if
d
here
strongly
said,
small
consistency
which
various
earlier
the
name
stand?
Ford
slash
is
here
crowd
of
linux,
way
of
using
your
configuration
files,
bationo
distributed
fashion
and
its
cncf
great
white
project.
Where
does
it
so
difficult,
kubina,
10
ecosystems,
souk
you,
we
have
fun
kloster
a
hands
up
weekend.
If
we
want
to
control
full
eye
components,
confess
the
control
system,
manager
of
sky
or
the
devil
serve
great
debate
themes,
yoav
donat
were
in
your
words,
are
running.
A
Is
here
said
he
goodbye
one
of
reason
why
davis,
nice
brain
jogging,
he
is
here
10,
holding
of
intyre
state,
envious,
q,
bert
and
discreet
hours
on
fb
event,
fbi
period,
f2i
to
custom
resources.
Definition'
had
made
a
story
in
tube.
It's
in
the
way
it
is,
it
is
built
up.
There
wants
cube
and
netease
ecosystems.
This
reactive
doctor
face
to
which
is
only
support.
Server
is
capable
of
cooking
everything.
A
Is
here
emissions
management
project
atelier
jazz,
to
build
it
with
it,
something
that
we
call
cube
reception,
un
d'admin,
that
will,
you
think
you
need,
is
to
deploy
cost-benefit
sophie.
We
have
stand
cuba
resist
kloster,
but
in
our
caisse
instead
of
running
wordpress
kafka,
Or,
regniowez
application
warning
I'm,
counting
on
playnow
custom
à
minettes
sort
signal,
you
have
cook
main
one
will
be
happy
ever
in
sound,
and
then
we
have
you,
for
example,
at
custom
hours
you
contemplate,
and
I
don't
know
we
need
you
want
you
back
de
delta
1.
A
Instead
of
the
blind
side
by
side
in
out,
we
decided
to
meet,
you
lie
them
and
use
your
feet.
Sidi
kloster,
forget
of
customers
and
vice
design,
is
pony,
run
off
constraint.
Un
stress
tweed,
sidi
sowie
to
fausto
chad
across
two
years
on
wanted
to
sidi
kloster
under
supporting
each
of
two
towns
average
of
thrones
actions
taken
in
time
and
7d
7.
What
is
your
mind
that
this
number
1
only
nine
lots
compères
du?
What
you
can
see
any
benchmark
beast
one.
You
are
running
doses,
then
brock
road
to
get
you
waiting.
A
His
father
fritschi
switch
tracts,
whif
avis,
qui,
thabite,
chieuse,
c'est
au,
now
that
we
have
the
context
of
diving
so
far
so
how
to
speak
about
observed,
bill
thies
sow
first,
we
need
to
enable
observable
iti,
d'avis
tuf,
like
that
you
need
to
enable
men
each
member
said
here
said:
christophe,
wants
carefully
metrics
together
as
many
metrics
as
possible.
What
noah
he
wanted!
Titou01
have
all
the
crisis
that
we
need
to
be
back.
A
production
will
be
reduced
to
come.
Handsome
observed
bill
thies
in
haiti,
siddiq
lost
you,
man,
2
and
world
wine.
A
Is
you
to
watch
on,
observe
frommer
and
poteau
series
old
weekend
divided
into
fu
lei
us
savior
swan
hills
god,
then,
if
I
pissed
is
used
for
communicates
the
chain
society
of
project
that
you
can't,
you
used
to
develop
a
remote
procedure.
Call
presentation
with
spices
and
50
best
is
cool
draft
undecided,
lac
de
cop
out
of
it
sidi
raf
tizen
consensus.
Ticket
algorithms
knew
too
much
and
a
holding
out
noise
man
d'
state
in
2002
by.
A
Operations
1,
which
value
stores,
which
could
be
bad,
ok,
except
let's
get
started
in
less
observe,
djir
worse
if
it
had
been
from
fiumicino
times,
potente
watch
a
jump
fing
like
to
turn
to
morocco
is
here
that
they
came
to
me
by
death
in
and
out
of
the
most
important
thing
is
care
and
direct.
The
distributor
system
on
you
need
you
pay
attention
to
the
quality
review
network.
Invite
metrics
will
show
your
from
elf
to
violet
trucks
between
de
pire
de
est,
ici
dit
christophe
spears.
So
what
should
fau
et
rock
with
the
metrics?
A
A
So
if
you
want
to
write
united
states
works
the
only
two
unit
or
check
for
first
metrics,
one
of
those
who
master
important
fund
robeco-
it's
you
are
solid
match
at
jean
pellet
s'
will
explain
if
you
leader
is
moving
from
dust
from
one
one
love
to
not.
It
means
that
some
film
he's
going
gloucester
holmes.
You
can
also
used
de
vue
comme
and
line
tube
tu
as
illac
de
thébaux
oil
leader
quiz.
The
option
will
be
organized
on
6.
Then
your
film
formation
of
database
versions,
a
sum
rafting
at
the
nova
spect
of
craft.
A
The
team
portal
2
tons,
isat
de
lydie
ceni
heartbeats-
to
follow
the
attacks
in
too
much
time
to
respond.
Un
nein,
nein
10%
of
the
time
is
because
of
snow
disc
on
est
rock.
So
you
need
to
watch
faux
at
birth
because
there
was
like
the
network,
the
yes,
it's
ok
go
to
me
something
new
christs
hours
will
be
forgotten.
We
have
fully
elected.
A
We
know
kent
class
to
de
new
day
ta1
de
processors
vadim
new
day
help
to
propose
a
because
of
to
click,
title
state
of
raft
to
propose
louis
move
between
different
eight
from
pending
tout
committed
vein
applied,
which
means
that
every
rose
has
now
applied.
Watts
dolly
de
committed
to
you
need
to
pay
attention
to
both
teams.
Metrics
vis,
one
should
be
as
low
as
possible.
One
of
the
railroad
workers
defense
between
dos,
you
have
of
caution
to
pay
attention
to
fail.
A
A
When
you
are,
we
are
trying
to
give
one
value
and
staking
way
too
much
time
racing
by
18,
lynch
and
nicosie
trisha
yearwood
statistics
in
too
much
time
of
course
raft
a
beat
it
sidi
exposed,
ims,
metrics
one
you
are
doing
so
mmes
soccorso
des
strings
hypotonie
tricks
for
clos
evans
balances
first,
is
for
writing
and
how
this
could
fly
v.
Ck
one
is
related
to
reading
d'état.
Forme
follower
under
defeat
is
so
you
need
to
be
on
the
street
hockey.
A
A
I
want,
is
you
know,
baby
boy
and
study
is
that
de
d'efforts
storage
limit
august
17
gigabytes
le
maxi
suggestive
sizes,
ait
ce
haricot,
meme
nature
ready
set
in
there
is
a
mourning
this
boost
or
matrix,
which
will
à
la
youtube
forro
in
type
tracks
of
de
torture,
size
of
view
cailloux
database
jacques
cotta
jumps
the
maximum
in
india
database
I'm.
So
we
want
to
hold
of
things:
bat,
observa
belletti.
Let's
talk
about
his
own
type,
centrex
sidi
allal
klein
to
be
notified
of
every
modifications.
A
Vis
means
that
is
it
sidi
need
to
keep
an
exact
history
for
sale.
What
will
happen
to
mutations?
Sly
will
be
assumed
and
majin
10
history,
nice
to
be
period
period.
Article
compact
and
sheep,
while
we
just
one
out
of
this
qu,
saw
vi
discal
compaction
trial.
I
can't
be
automated
darkly
by
the
database.
A
A
Are
running
period
makes
every
hour
one
thing
you
know
barbie
bolt,
he
said
youssef
on
the
option.
One
opening
fault
switch
cisco
memory,
mapt
files,
this
method
of
opening
phase
means
date
of
under
lyme
operating
system.
Will
cash
wants
files
in
ram
ifia
using
hejira
by
the
data
limits?
Bibeau
l'isf
mais
take
up
sur,
ed
gigabytes,
au
drame,
se
plissent
de
notre
capi
tsy
dit
mais,
maurice
bellot
maxdata
limit
under
the
name.
A
En
plus
we
had
you
and
we
are
one
yet
locally
in
euroleague
fashion,
one
in
portenard,
spect
of
every
thing
des
tab
à
6
aof
come
back
up
ce
we
use
de
dita
von
teese
and
produced
and
how
we
are
using
ve/ebit
tour,
d'avec
les
inspections
de
snapshots,
chinoise
'ft'.
One
thing
she
does
it
of
snapshot:
alocco
one
and
ten
beasts
trigger-
is
fit
because
you
do
not
want
your
boots.
Man
constraint,
aof
funds.
A
A
We
want
your
land
of
kings,
cross
life
stock
on
the
cheeks
cricri
cap.
With
a
film
we
learn
from
the
good
bowlers.
We
observe
ffm
shape
of
happy
feet,
two
eyes,
crook.
In
six,
your
destiny,
everything
chà
country
for
cech
avril,
is
in
the
shape
of
a
disc
of
one
and
rock.
You
drive,
storage,
limited
gigabytes,
a
quick
tips
for
anyone.
Date
is
running
his
village.
A
Cuban
antitrust
and
you
can
divide
a
charles.
We
want
multi.
Bull
is
here
classy
dick
ragnotti
siddiq
the
trick
in
dedicated
machine.
Yes,
it's
6
10
to
the
nba
me
how
to
be
still
young
ram
limits.
You
need
to
one
compaction,
d'ifrane
members,
adolfo
get
to
do
samba
cup,
auto
membership
alteration
still
need
duty.
Lloris,
haiti
will
find
you
and
I
nférieure
zing
est
ici
dit
d'ailleurs
clip
please
des
beaux,
gosses
tout
flags.