►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
There are three high-level requirements to operating an etcd cluster in production:
1 - Each etcd member must be bootstrapped: The etcd binary has to be on the host and the runtime parameters must be defined.
2 - The list of members must be kept up to date: This list must come from some external source of truth such an infrastructure API or the administrator themself.
3 - Periodic backups of the etcd state must be configured.
In this talk, we present etcdadm, a project recently adopted by the Kubernetes Cluster Lifecycle SIG. etcdadm is a command-line tool with a kubeadm-like interface that makes it easy to meet the first requirement by simplifying the bootstrap process. The project’s goal is to also make it easy to meet the second and third requirements by programmatically querying a variety of APIs for cluster membership, and by integrating existing services to store backups. Etcdadm will enable the automated operation of etcd clusters with a user experience analogous to CoreOS’s etcd operator, but without the prerequisite of a functional Kubernetes cluster.
We will explain the etcdadm design in depth, and discuss how our experience running etcd clusters in production informed that design. We will also demonstrate deploying a highly available etcd cluster using etcdadm, recovering the cluster from both partial and complete failures. In the process, we will cover important etcd runtime parameters and caveats of dynamic cluster reconfiguration.
A
Okay
looks
like
it's
down
to
a
trickle,
so
let's
go
ahead
and
get
started.
Thank
you
all
for
joining
us
today.
Welcome
to
today's
CN
CF
webinar
kubernetes
in
production
operating
at
CD
with
NC
d
ADM.
My
name
is
ariel
chatib
I'm,
a
CN
CF
ambassador
and
a
tme
here
at
NetApp
and
I'll,
be
moderating.
Today's
webinar
we'd
like
to
welcome
our
presenter
today,
Daniel
lipid
etske
from
platform,
9
I
got
it
right
again.
A
You
got
it
right
all
right
before
we
get
started
a
couple
of
housekeeping
items
during
the
webinar
you're
not
able
to
talk
show
as
in
attending.
If
you
have
any
questions,
there's
a
Q&A
box
right
down
at
the
bottom
of
your
screen.
Please
feel
free
to
drop
your
questions
in
there
and
we'll
get
to
as
many
as
we
can
at
the
end
of
the
talk
and
the
demo.
B
Great
thank
you
very
much
for
the
introduction.
Ariel
yeah,
like
a
girl,
said
my
name
is
Daniel
Levitsky
I'm,
a
software
engineer.
I've
been
working
on
kubernetes
cluster
lifecycle,
tooling,
for
about
three
years
at
platform,
9
platform
9
helps
our
customers
may
deploy
and
manage
kubernetes
clusters
on-premises
and
in
different
public
clouds,
and
today
I
want
to
talk
about
as
the
sed
ADM
tool
and
its
service
beginning
the
the
reason
why
it
exists.
The
motivations
and
those
are.
B
You
know
the
lessons
that
we've
learned
in
production,
so
to
begin
STD
ADM,
it's
a
CLI
and
it's
meant
to
simplify
a
seedy
operation,
including
disaster
recovery.
Inspired
by
lessons
learned,
running
kubernetes
in
production,
and
importantly,
it's
an
open
source
community
project.
You
can
find
it
on
github,
it's
adopted
by
the
urban
IDs
cluster
lifecycle,
special
interest
group
and
it's
easy
to
install
it's
a
single
goal.
Binary
you
can
go,
get
it
and
we'll
have
binary
releases
coming
soon.
So
why
did
that?
Cd
item
exists?
B
So
when
I'm
talking
about
the
control
plane
or
when
you
think
about
the
kubernetes
control
plan,
think
of
a
group
of
stateless
components
which
is
API
server,
which
is,
of
course,
you
know,
accepting
all
the
requests
and
responding
to
them
and
then
the
controller
manager,
the
controller
manager
where
the
different
controllers
are
running,
for
your,
for
example,
for
your
replica
set
to
your
deployments
or
even
for
nodes
and
then,
of
course,
the
scheduler
which
is
responsible
for
assigning
nodes
to
deposit.
And
finally,
there
is
one
stateful
component,
and
that
is
a
TD.
B
That's
a
distributed
database
and
it
holds
all
of
the
state
of
the
kubernetes
cluster.
Everything
that's
stored
in
the
kubernetes
api.
So
lesson
number
one
is
that
api
uptime
is
critical
without
the
api.
The
cluster
is
a
zombie
I
neglected
to
insert
a
zombie
meme
here,
but
what
I
mean
by
by
zombie
is
that
the
the
workload
that
you
have
running
continue
to
run,
but
anything
that
needs
API
access,
really
can't
make
progress,
and
you
know
in
the
in
the
early
days
of
kubernetes,
that
was
the
the
core.
B
B
When
you
have
an
api
outage,
it's
not
always
easy
to
to
determine
the
cause,
but
as
a
first
step,
you
could
check
these
component
services.
That's
a
coop
cuddle
get
component
statuses
command.
If
the
API
servers
is
responding
at
all,
sometimes
it's,
it
begins
to
crash,
look
at
which
points
or
that's
not
even
useful.
Sometimes
you
just
have
to
go
to
the
API
server
log.
A
B
So
from
those
from
those
lessons
learn
in
production,
we
know
we
want
to
address
or
make
sure
of
a
couple
of
things
right.
We
want
to
make
sure
that
the
API
is
is
up,
as
you
know,
as
much
as
possible,
and
there
are
two
strategies
to
doing
this.
One
is
to
ensure
that
you
can
tolerate
partial
failure,
the
of
the
control
plane
and
that
you
can
reduce
the
recovery
time
from
you
know,
either
a
partial
failure
or
a
complete
failure.
There
are
two
strategies
and
you
need
both
of
them.
B
So
how
do
we
tolerate
partial
failure?
Well,
we
can
deploy
multiple
control,
plane
replicas.
This
is
slightly
easier
because
they're,
stateless
and
if
you
have
some
configuration
set,
you
can
just
you
know,
create
all
the
replicas.
You
want
you'll
need
to
route
your
request
to
the
various
API
servers.
If
you
you
know,
you
may
have
a
load.
Balancer
and
you'll
need
to
update
the
load.
Balancer
records
to
point
to
your
replicas
or
you
may
have
you
may
be
doing
failover,
in
which
case
you
may
be
running
something
like
keepalive
D.
C
B
In
parallel,
it's
relatively
easy
to
do
and
there's
not
a
performance
penalty.
The
API
server
also
is
is
designed
to
be
able
to
run
with
you
know,
multiple,
multiple
copies
it
will
be
able
to
resolve
or
or
yeah
resolve
conflicts
when
there
are
conflicts-
or
you
know,
project
requests
that
are
that
are
maybe
conflicting
with
a
concurrent
request
and
then
the
other
components
like
the
scheduler
in
the
node
or
the
controller
manager
are
designed
to
elect
a
leader
if
there
are
multiple
replicas
running
and
avoids
conflict.
That
way,
so
that's
one
part.
B
The
other
part
is
that
you
need
to
deploy
multiple
TD
members.
If
you
have
multiple
control
plan
replicas,
but
just
a
single
sed
member
and
it's
UD
member
fails
and
the
API
server
is
unable
to
any
of
replicas,
are
unable
to
serve
a
question
more
so
deploying
multiple
SUV
members
is
harder.
It's
a
stateful
component,
and
and
not
only
that,
but
there
is
a
performance
penalty
for
doing
that.
B
So
as
as
the
number
of
members
increases
you,
the
the
what's
called
it
yeah,
the
majority
of
the
SD
cluster,
the
majorities
and
members
need
to
process,
reads
and
writes,
and
so
those
are
of
course
going
over
the
network.
So,
as
you
go
from,
you
know,
one
member,
which
is
handling
everything
locally
to
multiple
members
where
traffic
is
going
over
the
network,
the
performance
does
decrease.
B
So
that's
how
we
would
deal
with
dealing
with
partial
or
handling
partial
failure.
The
other,
the
other
strategy
that
we
needed
is
how
to
reduce
recovery
time
right.
So
we
have
a
couple
options
right.
We
could
write
a
service
to
automate
recovery
now
this
tends
to
be
more
complex
and
less
flexible
right
and
there
are
some.
There
are
some
corner
cases.
B
You
know
if
you
find
that
there
is
an
issue
at
the
very
time
that
you
need
to
recover.
You
know
you
just
don't
have
very
much
time
to
sort
of
investigate.
You
know
what
this
automatic
recovery
services
ISM,
if
it's,
if
it's
doing
something
correctly
and
finally,
I
use
yeah
you're,
trying
to
ensure
that
you
know
that
you're
tolerating
partial
failure
and
be
able
to
recover
this
one
distributed
system
which
is
at
C
D.
But
then
it
turns
out.
B
On
the
other
hand,
you
can
implement
a
manual
recovery
process
and
that
process
can
be
made
simple
with
tooling
importantly,
it
has
no
dependencies
and
it's
easier
to
debug
and
to
patch.
If
there
is,
you
know,
if
there's
a
corner
case
that
you
haven't,
that
you
didn't
anticipate
that
you
need
to
handle
during
this
critical
recovery
phase.
B
So
SCD
ATM
was
designed
with
those
with
those
ideas
in
mind
right,
so
its
goals
are
to
make
it
easy
to
tolerate
partial
ICD
failure
to
make
it
easy
to
have
a
manual
STD
recovery
process
and
to
work
without
dependencies
on
external
services
like
DNS
or
network
storage,
and,
of
course,
as
CDM,
wants
to
compose
well
with
other
tools.
So
you
know,
for
example,
you
can
use
kuba
TM,
which
was
a
huge
inspiration
for
a
TDM
in
order
to
deploy,
control,
plane,
replicas
and
ok
enough
enough
of
exposition.
B
B
In
order
to
do
this,
you
have
to
use
what's
called
the
sed
discovery,
service
or
DNS,
and
update
DNS
records
or
use,
what's
called
a
static
deployment
where
you
configure
all
of
members
initially
and
and
then
start
them
up.
So
the
other
option
is
to
deploy
one
member
at
a
time
then
scale
up
so
sed
ADM.
B
True
the
student
of
the
ladder-
and
the
reason
is
that-
or
there
are
a
few
reasons
right-
one-
is
that
there's
just
one
mechanism
to
understand
the
scaling
mechanism,
there's
no
dependency
on
DNS
or
external
discovery
service,
and
it
it's
easier
to
understand
when
something
goes
wrong.
The
drawback
is
that
you
deploy
members
sequentially,
so
you're
not
deploying
them
in
parallel
and
and
letting
them
sort
of
converge.
B
B
So
this
is
I'm
just
running
a
teeny
here
and
now
we've
got.
We've
got
an
NC
d
service
running
and
I'll
just
query
the
sed
API.
So
let
me
let
me
give
you
a
quick
rundown
of
what's
happening
behind
the
scenes.
So
when
I
run
a
TD
am
in
it,
it
will
generate
the
CA
certificate
and
then
use
that
to
generate
or
to
sign
server
and
client
certificates.
B
The
the
CA
certificate
is,
or
all
these
certificates
rather
are
what
used
to
establish
trust
among
the
different
Etsy
members
and
then
also
client
certificates
are
of
course
used
by,
for
example,
the
kubernetes
api
server
to
be
able
to
talk
to
TD,
so
I
see
the
ADM
will
generate
those
it'll
write,
configuration
and
then
it'll
create
and
start
the
system
B
service.
So
yeah
I've
got
a
single
member
at
GD,
+
you're
up
up
and
running
with.
B
Just
you
know:
a
single
single
command
now
the
the
next
thing
that
I
want
to
do
is
scale
up
my
cluster.
So
in
order
to
do
that,
I'll
want
to
copy
over
the
CA,
cert
and
key,
and
that
is
so.
The
other
members
can
use
that
to
sign
their
server
and
client
certificates
and
as
long
as
the
the
same
CA
is
used
to
sign
the
search,
then
the
the
members
will
will
trust
one
another
certificates
because
they
trust
that
CA.
B
C
B
So
what's
happened
here
behind
the
scenes
is
a
little
bit
more
complicated.
First
of
all,
it's
the
ADM
saw
that
there
was
an
existing
CA
search.
So
it
used
that
to
find
the
server
and
client
certificates.
Then
it's
the
ADM
actually
called
up
to
the
existing
sed
Buster
and
added
this
new
host
using
the
members
API
and
after
it
did
that
successfully.
B
It's
a
great
CLI
tool,
but
one
of
its
drawbacks
is
that
if
you're
using
it
with
a
cluster
that
that
has
TLS
enabled
you'll
need
to
pass
in
either
command
line
flags
or
set
some
environment
variables
so
that
s
unique
uttal,
knows
about
those
certificates
and
where
to
read
them
from
disk
and
if
you're,
using
it
with
the
v3
API
is.
We
are
you'll
also
need
to
set
another
environment,
variable
or
command
line
flag
and
so
sed
cuddle.
B
That
Sh
is
just
a
tiny
shell
script
that
we
bundle
with
at
the
ADM
that
sets
those
environment
variables
and
then
calls
sed
cuddle
for
you,
so
yeah,
just
just
a
little
bit
of
convenience
when
you're
using
that
interactively
so,
okay.
So
now
we've
got
this
two
member
cluster
and
let's
go
and
scale
up
to
three
so
again.
I
just
want
to
show
that
we've
got
the
search
there
and
I
can
use
that
same
command
to
to
join.
I
can
also
switch
the
endpoint
to
a
to
a
different
endpoint.
B
B
Okay,
at
some
point
you
may
want
to
scale
down,
and
to
do
that
there
it's
a
symmetric
interface.
You
run
at
CD
ADM
reset
on
that
member
to
have
it.
Leave
the
cluster
gracefully
I'll
demo
that
and
a
little
bit
when
we
talk
about
handling
failures,
so
now
that
we've
got
our
three
member
cluster.
Okay,
we
want
to
so
now
now
we're
at
the
point
where
we're
able
to
tolerate
partial
failures,
so
I
could
I
could
demonstrate
that,
for
example,
if
I,
if
I
stop
the
sed
service
and
actually
let
me
know.
B
C
B
All
of
them
are
able
to
respond
so
except
read
and
write,
and
now
let
me
take
one
offline
by
stopping
the
service
on
one
of
the
one
of
the
one
of
the
hosts
and
now
here
here
this
will.
This
will
fail
because
I'm
not
I'm,
making
a
connection
to
the
local
sed
member.
But
let
me
let
me
do
that
from
a
different
host
and
then
we
can
see
that
the
the
two
members
are
responding.
B
So
a
minority
failure
is
a
partial
failure
where
a
majority
of
members
are
are
still
available,
as
I
mentioned
before.
In
order
for
you
know
that
you
need
to
accept
any
read
or
write
and
majority
of
the
members
must
be
available.
So
an
example
of
a
minority
failure
could
be
something
like
plain
met.
B
A
planned
maintenance
right,
I
have
to
you,
know,
update
the
kernel
on
on
some
member
and
you
know
restart
restart
the
operating
system,
and
it
will
be
down
for
the
duration
of
that
that
restart
it
could
be
a
network
partition
or
it
might
even
be.
You
know
a
hard
disk
failure
on
on
a
minority
of
the
of
the
members.
On
the
other
hand,
there's
a
majority
failure,
and
this
is
a
partial
or
complete
failure
where
a
majority
of
members
are
not
available.
B
So
we,
you
know,
we've
seen
you
know
where
we
have
multiple
members,
but
they
are
all
backed
by
you
know,
by
the
same
network,
storage
and
at
some
point
you
know
for
for
different
reasons:
right
you
could
have
corruption
or
you
could
have
disk
failure,
and
you
know
one
or
more
members
can
be
affected
by
that,
and
sometimes
all
of
them,
so
I
want
to
really
talk
about
how
to
prepare
for
a
planned
minority
failure.
So
a
plan
minority
failure
is
a
plan
maintenance.
B
So
first
want
to
consider
how
many
failures
your
cluster
can
tolerate.
So
if
you
have
three
members,
you
can
tolerate
at
most
one
sailor
while
still
able
to
accept
reason
right.
Okay,
so
if
we
want
to
you
know
patch
the
patch
to
Colonel
on
one
of
these
hosts,
oh
we
can
do
that.
We
know
that
the
sed
will
continue
to
be
up
and
the
SAR
are
very
important
control,
a
control,
plane
and
API.
Will?
What
can
you
two
continue
to
run?
B
B
So
you
can
actually
take
the
data
on
disk
copy
it
to
a
different
machine
and
then
make
some
changes
to
the
to
the
metadata
of
that
member,
because
the
rest
of
your
cluster
is
up
and
that
API
is
available,
and
then
you
know
that
we
basically
start
running
that
same
member
and
it
with
a
different
network
ID
with
a
different
IP.
It's
a
special
procedure
to
do
that.
But
the
advantage
is
that
there's
less
data
to
catch
up
on
after
migrating,
because
there's
the
bulk
of
the
data
is
already
on
disk.
B
You've
just
copied
it
over,
and
the
third
option
is
to
just
replace
the
member.
So
this
is
nice
because
it
reuses
the
scaling
procedure.
First,
you
scale
up
alright,
so
you
add.
So
if
I
have
three
members
I'm
going
to
add
at
1/4
and
then
I
will
remove
the
member
that
I
want
to
I
want
to
perform
maintenance
on
the
downside
is
that
there
is
more
data
to
catch
up
on
for
that
member.
That's
that's
being
added,
so
you'll
need
to
wait
some
time
for
that
new
member
to
catch
up
to
the
question
and
yeah.
B
So
let's
take
a
look
at
what
we
can
or
how
we
can
do
this
with
with
a
TD,
a
DM
which
makes
a
third
that
third
option
easy,
which
is
to
replace
the
member
and
again.
This
is
so
that
there's
not
you
know
a
special
procedure
to
use.
You
can
just
use
the
same
scale
up
and
scale
down
procedure
so
yeah.
Let
me
actually
take
yeah.
Take
this
this
node
that
that
we
have
and
I.
Let
me
run
at
CD
ADM
reset
on
it,
and
this
will
have
to
have
this
member
leave
the
cluster.
B
So
behind-the-scenes
SED
ADM
called
up
to
the
clusters
member
api
identified,
the
the
local
member
and
then
removed
it
from
from
the
cluster
and
then
cleaned
up.
You
know
on
disk
artifacts,
and
so
now,
if
we
go
to,
if
we
go
and
list
the
current
members
we'll
see,
we
have
one
fewer
ok,
so
that
that
was
the
scaling
down
portion
and
then
for
the
scaling
up,
I'm
actually
going
to
reuse
the
same
member,
so
I'm
going
to
cheat
a
little
bit
but
I
can
I
can
demonstrate
that.
There's
a
there's,
there's
nothing
there!
B
There
is
no.
There
is
no
sed.
You
know
I'm,
not
turning
something
off
and
back
on.
I
really
am
you
know,
scaling
down
and
then
scaling
back
up
so
yeah.
Let's
make
sure
that
we
have
the
certificates
on
this
host
again.
Let
me
just
okay
copy
over
that.
That's
see
a
certain
key
and
now
that
now
that's
there
again,
I
can
run
the
join
command.
B
And
I'm
back
to
I'm
back
to
three
members
in
my
cluster
now
you
know
we
can
pretend
that
that
I,
just
you
know,
went
to
1/4
1/4
machine
and
did
this
but
I
happen
to
use
the
same
machine.
So
that's
how
you
would
prepare
for
a
plan.
Minority
failure
with
with
acedia
I,
relatively
straightforward
is
the
same
as
scaling
up
and
scaling
down.
B
Now.
If
you
have
an
unplanned
minority
failure,
it's
a
little
bit
more
complicated
right
first,
if
you
can
actually
reach
that
member
that
is
unexpectedly
not
participating
in
cluster.
If
you
can
reach
the
member
first
checkup,
the
data
is
on
disk.
Typically
that
data
is
going
to
be
under
Varla
at
CD
and
that's
that's
the
case
for
a
CDM
by
default.
So
let's.
B
Let's
look
at
that,
so
there's
our
there's
all
that
STD
data.
If
it's
there,
if
the
data
is
on
disk,
then
as
long
as
the
FDD
process
is
running
the
you
know,
the
member
should
be
able
to
participate.
So
you
know
you
can
check
the
status
of
the
of
the
STD
service.
Ok,
that
that's
running,
if
it's
not
running,
maybe
restart
it.
If
it's
running
the
data
is
on
disk,
but
it's
still
not
being
able
is
not
participating
in
a
cluster
to
check
for
a
changed
IP.
B
Maybe
the
machine
restarted
and
was
assigned
a
different
IP.
If
you,
if
that's
the
case,
then
you
can
actually
call
up
to
the
members
API
and
adjust
that.
So
that's
not
something!
That's
not
an
operation
in
EDM
supports
natively,
but
you
can
use
SD
cuddle
to
to
do
that
and
then,
if
the
IP
hasn't
changed,
maybe
you've
run
out
of
disk
space.
So
you
can
also
check
that.
Maybe
it's
something
else,
and
in
that
case
you
know
there
are
some.
There
are
different
resources.
B
One
of
them
is
a
great
talk
that
was
given
that
kook
on
Seattle
at
the
end
of
last
year.
On
debugging
a
CD,
so
I
left
the
link
there
in
the
slides
for
that
and
that's
a
good
place
to
turn
now
if
the
data
is
not
on
disk
or
the
member
is
not
reachable
over
the
network,
or
maybe
you
just
don't
have
time
to
investigate
what's
happening,
and
you
just
want
to
you
know
replace
it
so
then
you
can
identify
that
that
fails.
B
Member
by
identify
I
mean
youyou,
should
know
what
the
you
know,
what
the
network
identity
of
that
member
is.
But
what
you'll
need
to
know
is
this.
You
know
what
you'll
need
to
know.
Is
this
this
identification
or
member
ID?
That's
on
the
left-hand
side
here
and
you
can.
You
can
find
that
in
the
member
list
and.
C
B
B
So
this
is
a
case
where
we
have
lost
all
the
members
in
the
in
the
cluster
and
we
we
want
to
be
able
to
get
the
cluster
up
and
running
and
we
have,
let's
say
a
backed-up
snapshot
or
potentially
we
maybe
you
were
able
to
salvage.
You
know
the
hard
disk.
Maybe
we
lost
the
the
machines,
but
we
were
able
to
salvage
a
hard
disk
if
we
didn't
have
a
backup
and
as
long
as
we
have
you
know
on
this
data,
we
can
take
a
snapshot
of
that.
B
Let's
see
I'm
going
to
stop
yeah
I'm
gonna,
stop
at
CD
oops
system
system,
cuddle,
join
them
and
stop
it
see.
I'm
gonna!
Stop
that
it's
a
nice
service
here,
I'm
going
to
stop
the
itsy
service
here
and
also
on
this
post.
Okay.
So
now
we
don't
have
any
of
those
members
up,
but
what
we
do
have
is
the
ability
to
take
a
snapshot
of
let's
say
that
either
either
we
had
a
backup
or
maybe
I
was
able
to
ask
the
hard
disk
of
this
member.
B
B
Looking
for
the
looking
for
the
API
okay,
so
I'll
need
to
need
to
just
use
the
snapshot.
I
took
earlier.
Sorry
about
that,
so
I
have
I,
have
the
snapshot
that
I
took
earlier
and
I
will
use
this
to
first
of
all
copy
this
over
so
I'll
copy
over
the
snapshot
to
a
different
host
and
I
will
I
will
start
a
brand
new
cluster
using
that
snapshot
there.
So,
first
of
all,
let
me
clean
up
I'll.
B
B
B
Okay,
so
now
I'm
joining
and
I'm
recovering
I'm,
creating
a
I'm
recovering
from
this
complete
cluster
outage
by
creating
a
single
member
and
seating
it
with
a
snapshot
so
I
have
Cypress.
Sir
I
preserve
the
data
from
from
the
cluster
that
that
failed
and
now
I'm
scaling.
It
scaling
this
new
cluster
up.
B
B
B
C
B
You
know
we
have
a
two-member
cluster,
but
yeah
we
failed
to
add.
We
fail
to
add
the
third
but
I'll
look
into
that
I
hadn't
I
hadn't
had
any
issues
and
I
was
preparing,
so
I
just
want
to.
You
know,
describe
what's
going
on
behind
the
scenes
so
yeah
the
SDK
DM
will
generate
the
the
CA
in
server
client
certificates
and
then
it'll
initialize
the
data
Directorate
from
the
snapshot
that
we
passed.
It
then
write
the
configuration
and
then
create
and
start
the
system
be
serviced
and
yeah.
B
B
We're
definitely
considering
implementing
some
kind
of
automation
that
invokes
the
sed
ADM
CLI
right,
I.
Think
it's
very
important
to
to
keep
those
separate
so
that
you
can
always
have
a
manual
recovery
process
that
you
can,
for
which
sed
ADM
will
be
the
tooling.
But
if
you
have
you
know,
the
it
sometimes
makes
sense.
If
you
have
the
right
external
API
is
that
you
have
the
right
dependencies
in
place
and
the
ability
to
you
know
rely
on
some
external
mechanism
to
make
your.
B
You
know
your
automation,
highly
available,
which
is
the
case
you
know,
and
often
in
public
clouds,
right
where,
if
there
is,
you
know
problem
with
the
underlying
system
that
is
ensuring
your
recovery
services
highly
available.
You
know
someone
at
a
different
company
will
be
paged
at
3:00
in
the
morning,
but
nevertheless,
if
you
know
there's,
if
there's
an
issue
that
that
brings
your
this
autumn,
you
know
automated
recovery
service
or
automated
configuration
service
for
your
cluster
down.
You
always
want
to
have
a
manual
recovery
process
and
the
tooling
can
make
that
process.
B
Straightforward,
also
we're
thinking
about
implementing
periodic
backups.
So
this
is
also
something
that
would
you
know,
integrate
with
potentially
different
network
storage
or
potentially
different
public
cloud
api's
to
make
it
simpler
for
for
anyone,
that's
that's!
You
know
running
at
the
cluster
to
have
had
backups
periodically,
and
then
we
want
to
improve
the
upgrade
support.
A
B
B
Which
customers
are
most
insistent
on
tha,
I,
guess:
I,
guess
the
what
what
I'm
wondering
is
or
not
what
I'm
wondering,
but
but
when
customers
think
about
I,
think
HJ
for
SED
they're,
just
thinking
about
the
the
potential
you
know
time
to
to
recover
from
a
control
plane
from
an
API
outage,
so
with
hae
TV,
you
know,
and
it
and
or
multiple
control,
plane,
replicas
and
then
multiple
STD
members.
Typically
you
can
tolerate
you
know
one
or
potentially
you
know
two
or
three,
depending
on
your
CD
cluster
size.
B
The
operator,
it's
great,
if
you
have
a
kubernetes
cluster
already
there,
if
you're,
using
that
you
may
just
opt
to
have
a
single
CD
member
and
rely
on
you
know
being
able
to
quickly
reschedule
the
pot,
but
in
you
know
an
on-premises
environment
or
where
you're
really
concerned
about
you
know:
API
downtime,
that's
that's!
When
you
start
looking
at
the
ability
to
tolerate
failure.
A
B
A
B
A
Let's
I'm
sure
Taylor
can
help
out
if
now
I
Gaurav,
maybe
reach
out
on
at
city
admin,
in
kubernetes,
slack
and
and
and
and
I'm
sure
that
anyone
can
provide
an
answer
there
for
you.
Alright,
if
there's
no
EDD
member
connected
to
the
cluster,
then
taking
a
snapshot
is
not
possible.
So
that's
a
it's
more
of
a
statement
rather
than
a
question,
but
I
think
you
get
the
I
think
you
get
to
the
heart
of
it,
which
is
you
did
a
snapshot?
Can
you
gain?
Can
you
can
you
explain
how
that
occurred?.
B
Yeah,
so
what
would
I?
What
I
need
to
do
is
the
take
I.
Think
I
can
take
the
the
the
data
directory
directly
yeah,
rather
than
a
snapshot,
but
I
I.
Think
in
that
case,
yeah
I
would
have
all
right.
I
should
have
had
one
member
running
and
then
taking
the
snapshot
and
and
then
brought
all
the
brought
that
last
number
down.
B
A
And
let's,
let's
continue
down
that
thread
a
little
bit
more
David
all
the
marker,
hopefully
I
said
that
correctly
can
I
copy
that
CD
data
from
disk
directly
without
using
at
CD
cuddle,
Sh
snapshot
save
the
use
case
being
I
had
to
pull
the
disk
out
of
the
server
to
access
the
data.
Furthermore,
could
I
put
the
disk
in
a
new
server
and
use
the
link
so
that
EDD
could
the
data.
B
So
taking
taking
the
day
directory-
and
you
can
start
at
CD
with
that
same
data
on
a
different
machine,
but
it
won't
be
able
to
join
a
cluster
unless,
unless
that
cluster
is,
is
the
its
members
list
is
updated
to
to
the
new
network
identity.
So
the
network
identity
changes
you,
you
won't
be
able
to
to
just
take
that
data
directory
and
copy
it
over.
B
You
also
need
to
make
an
STD
API
call
to
update
the
peer
URLs
and
client
URLs,
and
if
you
do
that-
and
you
configure
sed
correctly-
or
rather
you
can
just
copy
over
with
the
configuration
and
started
to
be
on
that
new
machine.
Then
then
that
should
work.
That's
the
migration
process.
In
fact,
there's
a
link
to
that
in
the
slides.
A
C
B
That's
a
great
question,
so
I
think
I
think
the
question
is
sort
of
you
know:
will
at
CDM
support
an
external
CA
right
where
you
don't
want
to
have
the
CA,
cert
and
key
on
each
of
these
members.
Yes,
it
is,
it
is
possible,
it's
not
something
that
you
know
that's
been
what's
it
hasn't
been
thoroughly
tested,
but
that
that
is
actually.
That
would
be
a
great
a
great
feature
to
have.
B
You
know
for
anyone,
that's
using
a
CDA
DM
as
part
of
like
larger
automation
system,
where
you
know
you're
providing
the
search
externally
or
you're
signing
them
externally.
Something
that
I
think
seda
DM
will
end
up.
Doing
is
what
kuba
diem
has
done
and
that
is
generate
CSR,
so
certificate
signing
request
right
into
the
disk
and
then
allow
those
signing
requests
to
be
to
be
signed.
B
You
know
on
a
different
machine
somewhere
externally,
and
that
would
be
a
nice
option
because
then
you,
you
can
delegate
all
of
the
sort
of
all
the
details
of
you
know
what
the
search
need
to
look
like
what
they
need
to
have
in
their
common
names
and
the
subject
called
names
to
a
CDM,
and
then
you
can,
just
you
know,
rely
on
your
external
states
to
sign
them.
If
you
don't
want
to
do
that,
I
you,
you
should
be
able
to
just
provide
these
signed
search
yourself,
but
then
it
will
be.
A
B
A
great
question
today,
sed
ADM,
only
runs
sed
as
a
system
V
service,
running
system
V
in
a
container,
not
not
not
really
a
great
option,
but
there's
really
nothing
stopping
its
medium
from
supporting
you
know
running
to
be
in
in
a
container
the
the
reason
that
today
supports
you
know.
Systemd
service
is
because
we
wanted
to
decouple
sed
the
lifecycle
from
the
lifecycle
of
you
know,
container
runtime
or
kubernetes
itself,
but
that's
that's
definitely
something
that
we
want
to
support
in
the
future
cool.
B
If
you
can
use
at
sea
operator,
then
you
should
definitely
use
it
now,
if
you
are
using
sed
operator
and
you're
doing
that
without
having
a
kubernetes
cluster,
which
itself
has
a
net
CD.
Please
tell
me
how
you've
done
that,
because
I
would
like
to
do
the
same
so
yeah
I,
see
the
operator
is,
is
meant
to
operate
STV
in
pods
in
a
kubernetes
cluster,
but
it
doesn't
operate
the
TV
that
is
actually
being
used
by
that
cluster.
B
C
B
If
there's
an
option
to
you
know
just
copy
over
the
data
directory,
I
think
that
that
might
work.
You
could
also
take
a
snap
shot
that
might
work
as
well
yeah
there
there,
the
devil
is
going
to
be
in
the
details.
As
far
as
you
know,
making
sure
that
that
the
configuration
is
is
correct.
So
it's
not
not
something
that
the
seda
DM
does
today
doesn't
adopt
clusters,
but
that
potentially
is
something
that
it
could
do
in
the
future.
A
Okay,
cool
and
we
have
where,
at
the
top
of
the
hour,
we
have
time
for
one
more
question
and
then
again,
I'd
recommend
everybody
jump
over
to
at
cdat
and
when
you
burn
any
slack
or
go
to
the
github
repo
and
maybe
issue
a
question
via
nish,
but
Marek
Bartok
asked
great
demo
thanks
for
disaster
recovery,
would
you
recommend
backup,
slash
snapshot
plus
restore
of
EDD
or
dumping
K
eights
kubernetes
resources
with
tools
like
hefty?
Oh,
the
I
guess,
Valero
I
did
not
know.
Ark
had
been
rebranded.
C
B
Think
it
depends
if
you
so
you
know
we're
recovering
so
that
we
can
get
kubernetes
cluster
running
again.
If
the
disaster
that
that
has
you
know,
hit
your
cluster,
if
it
hasn't
affected
your
your
control
plane.
That
is
your
your
stateless
components
that
they're
still
you
know
if
they're
still
running,
and
maybe
they
have
the
same
network
identity.
And
then
your
worker
nodes
also
are
still
running
and
they
have
the
same
network
identity.
B
Then
all
of
those
no
records
are
going
to
be
in
the
SE
state
and
if
you
recover
the
cluster
and
your
you
know
your
API
servers
that
are
still
running
reconnect
after
the
cluster,
then
everything
will
you
know
as
far
as
as
far
as
those
no
records,
they
will
then
be.
The
cubelets
will
be
able
to
to
just
reconnect
and
continue
on.
If
you
use
Valero,
I
I,
don't
think
it
will
preserve.
B
These
are
the
identity
of
the
cluster.
It
will,
you
know,
preserve
some
of
the
configuration
of
your
applications
and
that
III
don't
know
exactly
how
it
would
compare
in
terms
of
recovery
time.
But
certainly
it's
you
know
if
you,
if
you're
able
to
recover
it
at
CD
and
the
rest
of
your
nodes,
are
still
you
know,
still
running
and
still
have
the
same
network
identity.
I've
seen
you
know,
recovery
be
relatively
relatively
seamless,
as
you
know,
as
far
as
all
those
other
nodes
are
concerned,.
A
All
right,
well
Daniel!
Thank
you
so
much
for
your
time
and
expertise
today.
It's
greatly
appreciated
to
everyone
in
attendance.
Thank
you
for
spending
your
afternoon
if
you're
on
the
East
Coast
with
us
and
again,
look
out
for
an
email
with
a
link
to
the
deck
and
the
recording
coming
soon.
Thank
you
and
hope
to
see
you
next
time
all
right
thanks
very
much.
Okay,.