►
From YouTube: Effective Disaster Recovery: The Day We Deleted Production - Rick Spencer & Wojciech Kocjan
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Effective Disaster Recovery: The Day We Deleted Production - Rick Spencer & Wojciech Kocjan, InfluxData
Imagine waking up to an sms, "we lost a cluster." On that day, with a one-line configuration change, we accidentally removed all of the compute from one of our busiest production clusters, causing a multi-hour outage. This presentation will cover the incident from the days leading up to it, to our full recovery, our customers' response to it, and how we implemented changes based on our learnings. It will go into detail about the configuration of our CI/CD pipeline, details about the specific change that caused the outage. Thankfully, we had a disaster recovery plan in place. We will discuss which parts of our disaster recovery plan worked, and critically, the few parts that didn't work. The session will cover a combination of technical and management content.
A
Okay,
hi
everyone.
I
hope
you
can
hear
me
all
the
way
in
the
back,
so
my
name
is
henrik.
I'm
the
moderator
for
these
sessions,
we'll
get
started
here
in
just
a
second
just
want
to
remind
you
once
the
session
concludes
we'll
have
a
microphone
here
in
the
middle
for
questions.
So
please
line
up
and
ask
any
questions.
A
You
might
have
I'll
try
and
get
a
couple
of
questions
from
the
the
virtual
meeting
room
as
well
make
sure
those
guys
are
included
and
after
the
session,
please
don't
forget
to
to
rate
the
session,
and
if
you
have
additional
questions
after
the
session
concludes
and
we're
out
of
time.
Please
continue
those
discussions
out
in
the
hall,
so
we
can
clear
the
room
and
prepare
for
the
next
session
and
with
that,
I'm
going
to
leave
it
over
to
you.
Thanks.
B
Can
you
give
me
a
thumbs
up
if
I'm
perfectly
audible,
great
okay?
So
this
is
the
title
of
our
talk.
It
was
originally.
The
subtitle
was
the
title
of
our
talk,
but
our
pr
person
about
had
an
aneurysm
when
they
saw
that
I
was
submitting
that
talk,
but
I
explained
to
her.
First
of
all,
everybody
at
least
all
of
our
customers
know
we
had
an
outage
and
second
of
all,
everybody
has
outages.
C
B
Okay,
so
who
are
we?
Why
are
we
using
kubernetes?
Why
might
our
experience
be
relevant,
so
at
influx
data
we
view
ourselves
as
a
development
platform
for
writing
time
series
applications.
We
don't
really
view
ourselves
as
a
kubernetes
tools
vendor.
That
said,
a
lot
of
our
customers
use
us
to
monitor
their
kubernetes
clusters,
and
then
we
have
different
companies
that
have
actually
built
sas
solutions
for
monitoring
kubernetes
on
top
of
us.
B
B
It's
built
on
top
of
kubernetes
and
the
reason
it's
built
on
top
of
kubernetes
is
because
we
offer
it
on
the
top
three
clouds
and
multiple
regions
in
all
those
clouds.
So
I
think
we
have
like
12
or
15
production
instances
running
around
the
world
right
now
that
we
manage
and
kubernetes
provided
that
cloud
abstraction
layer
that
we
needed
to
be
able
to
manage
the
same
application
in
all
of
those
different
regions
and
clouds.
C
Right
so
the
timeline
which
is
basically
how
to
delete
your
production
in
a
few
easy
steps.
So,
let's
as
spring
mention,
in
fact
the
cloud
is
a
flagship
product.
It's
a
stateful
kubernetes
based
application
and
we
use
github's
ncicd
to
keep
it
running
and
keep
it
up
to
date.
We
have
multiple
tiers
in
the
application.
Some
of
them
are
stateful.
Most
of
them
are
stateless.
So
one
of
the
key
things
that
we
have
is
the
storage
tier,
that's
using
pvcs
and
keeps
data
in
in
kubernetes
native
volumes.
C
C
So
when
you
run
a
query
against
influx
db,
it
goes
to
our
query:
engine
which
is
entirely
stateless.
It
parses
the
query.
It
sends
the
commands
to
proper
storage
tier
parts
by
knowing
which
charts
to
ask
for
the
specific
data
gets
it
back.
Reconstructs
it
returns.
The
data
and
most
of
our
most
of
our
components
are,
are
stateless
and
we
use
argo
cd
for
deploying
all
of
our
instances
of
infrared
db
cloud.
So
we
have
mentioned
12
to
15
production
ones.
C
C
I'm
suspecting
this
may
be
a
bit
of
the
focus,
but
that's
actually
by
design.
So
this
pr
was
merged,
and
this
is
basically
just
adding
new
data.
There's
like
500
new
lines,
most
of
it
generated
yaml
files.
It
did
not
remove
any
single
thing,
but
as
soon
as
it
got
married
within
minutes,
we
deleted
all
of
our
our
entire
single
production
environment.
C
C
But
what
happens,
but
you
may
not
even
see
it
because
it's
a
tiny
detail
is
that
we
have
a
naming
collision.
So
if
so,
the
ios
was
deployed
as
idp,
and
you
can
see
that
the
first
arrow
points
that
it
should
be
iot
and
it's
idp
I'll
show
it
again
in
a
bit
in
a
better
picture.
The
point
of
this
is:
it
was
really
difficult
to
spot
in
code
review
same
as
it
may
be
difficult
to
spot
for
you
now.
So
what
we
wanted
to,
how
close?
C
C
So
then,
that
repository
is
specified
like
a
namespace
and
defines
all
the
all
the
deployment
stateful
sets
anything
else
that
you
want
deployed
as
part
of
that
application,
and
then
what
we
wanted
to
do
is
wanted
to
add
iocs
alongside
so
we
added
a
new
argo
cd
application.
We
or
we
wanted
to
add
a
new
organic
application.
We
wanted
that
to
point
to
a
different
path
in
the
repository
that
would
have
all
the
objects
for
ios.
C
But
what
really
happened
is
because
we
had
a
typo
and
it
was
idp
and
now
it's
on
the
left
side
in
the
middle
in
red,
because
it
was
idp
instead
of
iox,
which
is
what
we
wanted.
Argot
cd
went
in
and
said,
okay,
I
have
two
definitions
for
the
idp
cd
application,
so
I'm
going
to
take
the
next,
so
I'm
going
to
apply
the
last
one
which
is
actually
pointing
to
the
wrong
git
repository,
so
then
argus.
If
we
apply
that
and
then
the
actual
lcd
deployment
for
that
application
decided.
C
Okay,
I
no
longer
have
idp
to
deploy
and
I
should
be
pruning
everything.
So
I'm
just
going
to
go
ahead,
delete
the
whole
the
whole
production
cluster
and
deploy
this
ios
testing
environment
instead.
So-
and
this
is
a
snippet
from
our
from
our
the
ins
status
page
for
the
incident
once
we
had
the
postmortem
in
place.
This
was
exactly
what
happened
due
to
some
mistakes.
When
we
were
adding
the
ios,
we
should
have
put
io
cd
aws
pro
01u
central
one.
B
I
live
on
the
east
coast
and
I
don't
for
anyone
here
who
manages
a
sas
environment.
You
know
that
you
never
stop
thinking
about
it
and
the
first
thing
you
check
in
the
morning
is
like
what
happened
at
night,
and
so
this
is
what
I
woke
up
to
good
times
this
by
the
way,
if
you're
trying
to
quit
coffee,
this
will
help
you
wake
up
without
coffee.
B
So
I
just
went
straight
to
my
computer
and
just
started
working
on
this,
so
this
is
the
first
part.
This
is
when
the
damage
was
done.
This
is
actually
all
like
right
before
I
got
up
so
the
p.
These
are
all
times
that
are
utc,
but
I'm
based
on
the
east
coast,
so
the
pr
was
merged
and
then
all
our
monitoring
systems
started
to
report
api
failures,
but
we
had
a
bit
of
monitoring
fatigue
at
the
time,
so
we
told
it
like
look.
B
You
really
need
to
wait
a
while,
before
you
start
paging
people,
because,
like
these
bothering
our
staff
with
these
transient
api
failures
is
just
annoying
people
and
it's
not
helping
much.
Then
a
customer
called
in
and
then
another
customer
and
said
hey,
it
seems
like
something's,
not
working,
so
the
support
team
jumped
in
and
said,
customers
are
reporting
a
problem
and
we
looked
around
and
sure
enough.
All
the
alerts
started
firing
at
that
point
and
it
was
clear
there
was
something
wrong.
B
So
in
our
culture,
when
we
deliver
code
that
causes
a
problem,
we
just
revert
it
immediately
and
we
really
try
to
develop
so
that
a
revert
of
code
is
the
path
back
to
stability.
But
this
was
not
a
code
change.
This
was
an
infrastructure
change,
so
the
developer
actually
followed.
That
process
submitted
the
pr.
But
then
the
team
realized
that,
like
just
reverting
a
change
like
that,
since
it
was
infrastructure,
change
is
probably
not
the
way
to
do
it,
but
fortunately
we
also
have
a
culture
of
anyone
being
able
to
stop
the
line.
B
So
we
stopped
all
synchronizations.
We
have
this
thing
that
we
call
internally
the
big
red
button.
Anyone
can
press
it
and
it'll
just
stop
deployments
through
all
of
our
deployment
environments.
Then
the
engineering
team
said
okay.
Do
we
really
need
to
start
really
planning
a
proper
recovery
process
and
by
the
way,
did
anybody
update
the
status
page
and
we
updated
the
status
page
to
alert
customers,
and
then
they
started
calling
everybody
like
hey.
This
is
all
hands
on
deck.
You
know
when
they
really
the
penny
dropped
about
the
severity
of
the
problem.
B
But
the
first
thing
the
team
did
was
to
create
a
deployment
checklist
and
and
double
check
that
list,
and
that
I
was
really
impressed
with
that
everyone's
staying
very
calm
and
instead
of
you
know
panicking.
They
say:
okay,
like
let's
make
a
plan
but
like
let's
double
check
the
plan
before
we
start
stampeding
and
to
make
sure
that
we
don't
make
things
worse
and
I
actually
credit
the
short
time
that
it
took
us
to
rebuild
the
production
environment
to
that
systematic
mentality
that
the
developers
followed.
B
So
then
we
went
through
and
we
started
to
redeploy
services
carefully
and
in
the
proper
order,
and
the
main
thing
was
to
connect
when
we
could
the
stateful
services
to
their
persistent
volumes,
because
that
saved
us
a
lot
of
time
in
terms
of
not
having
to
play
back
backup
data
right.
So
the
data
was
there.
We
could
just
reconnect
it
and
start
using
it.
Then
additional
services,
especially
the
state,
were
stateful.
The
stateless
ones
were
redeployed
in
parallel
with
people
going
and
making
sure
like
did
we
actually
recover
the
volumes
properly
like?
B
If
the
team
thought
you
know,
okay,
the
best
strategy
is
just
to
get
that
from
valero,
but
then
also
we're
like
you
know
what,
when
we
turn
this
back
on
everybody's
telegraph
instances
like
some
of
the
eight
we
we
have
an
agent
that
people
can
use
to
write
down.
Flexdb
those
are
all
going
to
start
writing.
Everybody's
queries
are
going
to
start
stampeding
us
everyone's
going
to
be
trying
to
catch
up.
So
we
scaled
out,
especially
the
ingress
tier,
but
also
also
other
tiers,
anticipating
that
surge
in
traffic.
When
we
came
back
on.
B
B
You
can
push
all
that
work
down
into
our
platform
and
some
people
run
those
every
second
every
minute
every
hour
and
we
realized
when
the
task
system
comes
back
on
it's
going
to
realize
it
wasn't
running
tasks
for
the
past
few
hours,
it's
going
to
try
to
run
all
those
tasks
and
if
it's
trying
to
run
all
those
tasks
and
users
are
trying
to
run
queries
at
the
same
time,
it's
just
going
to
be
a
complete
collision.
You
know,
there's
a
total
traffic
jam
of
queries,
so
we
decided
to
do.
B
Was
before
we
turned
on
queries
again,
we
just
let
the
task
system
run
its
course,
and
fortunately
it
wasn't
too
long.
So
we
let
all
those
tasks
run
and
fail
and
then,
when
the
backlog
was
done
and
the
tasks
were
caught
up,
we
went
ahead
and
turned
on
the
query
service,
and
at
that
point
really,
the
smoke
had
cleared
and
we
were
back
in
business.
So
we
alerted
all
our
customers.
Hey
the
the
service
is
back.
B
So
we
were
able
to
write
an
rca
doc
and
put
it
on
our
status
page
within
an
hour
or
two
after
the
incident,
so
that
people
could
go
back
and
you
know
see
what
really
happened
from
that.
We
got
an
interesting
piece
of
feedback
from
one
of
our
big
customers
who
said
well,
we're
just
glad
it
was
automation
and
not
somebody
sitting
in
front
of
the
terminal,
so
they
actually
was
a
little
bit
almost
confidence
building
in
a
way
that
it
wasn't
somebody
fat
fingering
at
a
terminal,
and
it
was
our
our
automation.
B
B
While
we
had
downtime,
we
did
not
lose
any
data,
and
that
I
mean
if
anybody
tried
to
write
data
while
we
were
down,
they
got
back
a
404
or
500,
or
something
right.
What's
terrible
is
that
they
get
back
at
204.
We
got
your
data,
but
it
didn't
actually
write
the
data
because
then
they're
in
an
inconsistent
state
and
they
don't
know
like
how
their
application
is
going
to
perform.
B
So
in
this
case
they
could
go
back
and
decide
if
they
wanted
to
backfill
during
the
time
that
we
were
down
or
how
they
wanted
to
handle
it.
We
did
avoid
panic
as
as
we
mentioned,
while
we
did
like
immediately
sort
of
knee-jerk
to
that
rollback
attempt.
We
were
able
to
stop
and
create
a
plan.
First
props
to
valero
also
props,
to
our
sre
team,
like
the
week
before
they
had
been
practicing
valero
backups.
B
B
It's
a
it's,
a
very
sharp
knife
which
they
say
a
sharp
knife
is
the
safest,
but
it
cut
us
deeply
in
this
case
and
the
other
thing
was
we
never
anticipated
deleting
a
cluster,
so
our
alerts
were
tuned
to
errors
in
our
code
or
some
user
doing
something
that
we
didn't
anticipate.
But
we
never
imagined,
like
we'd
need
to
be
alerted
that
the
cluster
was
deleted
and
we
had
no
run
books
for
recovering
the
cluster.
So
we
do
now
because
we
wrote
it
over
the
course
of
that
day.
C
Okay,
so
in
terms
of
recovering
some
some,
like
recap,
and
maybe
some
technical
information,
so
as
rick
mentioned,
our
first
instinct
was
to
revert
the
change
like
when
you're
doing
a
lot
of
small
call
changes
and
you're
in
a
microservice
based
architecture.
That's
what
you
often
do
you
make
small
changes
that
aren't
breaking
so
they're
easy
to
deploy
incrementally
and
these
roll
back.
C
So
at
that
point,
as
we
mentioned,
the
team
stopped
and
we
started
creating
a
proper
plan
and
the
goal
was
that
we
want
to
restore
all
the
stateful
guidance
manually
and
then
recreate
the
rest
via
cd,
with
maybe
some
manual
tuning
that
we
will
get
into
in
a
bit.
So
what
we
did-
and
maybe
some
questions
around-
that
it's
like-
why
didn't
we
just
redeploy?
The
storage
there
and
storage
theory
is
the
the
part
of
the
system
that
keeps
all
of
the
time
series
data.
C
So
all
of
your
metrics
for
the
last
10
years,
and
the
reason
for
that
is
if
we
were
to
do
that
and
not
and
not
like
reattach
the
volumes
that
that
we
had
that
we
still
had
we
would
be.
We
would
need
to
fetch
the
data
from
the
cloud
native
objects
or
like
s3
or
google
streamer
storage
system
and
replay
that
data,
and
that
would
take
several
hours
and
keep
in
mind
that
we
were
able
to
recover
in
just.
I
believe
it
was
six
hours
less
than
six
hours.
C
I
think,
even
and
if
we
would
do
that,
this
would
probably
roll
over
into
more
like
10
or
12
hours.
Now
the
other
question
is
maybe
I'll
back
up
and
just
mention
one
thing
so
most
of
our
volumes
in
that
in
that
cluster
were
defined
to
be
in
retain
mode,
meaning
that,
even
if
you
delete
the
pvc,
the
actual
persistent
volume
and
the
actual
underlying
cloud
storage
volume
is
kept
there.
So,
but
that
wasn't
the
case
for
everything.
C
So
for
zookeeper,
that's
just
keeping
the
state
for
kafka
and
it
doesn't
really
have
any
changes
or
topics
in
kafka
and
everything
in
kafka
is
pretty
much
static.
It's
just
a
day-to-day
kafka
that
changes,
so
we
were
able
to
just
restart
it
from
the
hourly
backups
and
we
and
we
decided
that
for
zookeeper.
That's
that's
enough
to
just
have
the
backups
in
place
for
kafka
and
lcd.
We
were
we
we,
since
I
mentioned
we
had
the
persistent
volumes
we
just
restored
them.
It
was
initially
a
manual
process.
C
Then,
once
we
made
sure
that
it
works,
we
could
like
script
that
and
just
run
it.
We
have
a
lot
of
pots
in
in
kafka
at
cdn
storage,
so
it
would
be
painful
to
do
it
manually
and
prone
to
fat
fingering
that
we
didn't
want
once
we
had
that
we
could
recreate
the
stateful
sets
and
then
and
then
by
then
kubernetes
would
recreate
the
parts
with
storage.
It
was
pretty
much
the
same
thing.
C
The
main
difference
is
that
the
way
our
storage
still
works,
it
has
to
index
the
data
at
that
point,
but
basically,
as
soon
as
it
as
every
single.
As
a
storage
spot
wakes
up,
it
re-indexes
the
data
in
the
persistent
volume
in
case
it
was
shut
down
incorrectly,
so
that
have
an
up-to-date
index
and
then
it
can
asynchronously
inject
the
right
ahead.
Log
data
from
kafka
last
question:
why
didn't
we
just
enable
everything
at
once?
As
rick
mentioned?
C
There
are
some
tricks
to
that,
so
we
started
enabling
parts
of
influx
data
as
they
started
to
work.
So
even
before
the
storage
theory
was
fully
restored,
we
could
start
enabling
rights
to
the
system
because
we
because
the
rights
just
put
the
data
in
kafka
and
report
204
if
the
right
to
kafka
was
successful.
So
people
could
start
writing
data
even
though
they
were
not
able
to
create
yet
and
it
wasn't
properly
persisted,
but
the
kafka
with
its
availability
and
replication
allowed
us
to
to
be
confident
with
that
approach.
C
Once
we
had
all
the
stateful
services
in
place,
we
deployed
the
remaining
ones.
We
increased
the
number
of
replicas
and
again
in
terms
of
what
was
enabled
or
what
was
running.
We
started
with
tasks
we
let
all
the
tasks
run.
We
made
sure
that
the
backlog
was
empty
and
then,
after
that
happened,
we
enabled
queries,
and
at
that
point
we
were
confident
to
just
scale
back
and
configure
the
number
of
replicas
to
what
it
is
on
a
day-to-day
basis
or
what
it
should
be
right.
C
So
now,
after
we've
done
that
the
obvious
question
is,
can
we
not
delete
production
again?
Please,
because
it
took
a
lot
of
people
a
lot
of
time
and
everyone
was
had
to
stop
whatever
they
were
doing
at
the
time
or
maybe
like
use
the
text
message
instead
of
coffee
for
waking
up.
So
the
first
thing
is
like.
Obviously
we
don't
want
that
thing
to
be
merged
again
and
just
a
quick
introduction
to
how
we
do
things
we
use
jsonnet,
which
is
a
tool
that
allows
rendering
multiple
well
objects.
C
Basically,
it's
not
kubernetes
specific,
but
we
use
it
for
kubernetes
objects.
Prior
to
the
incident.
We
were
just
writing
everything
to
a
single
yaml
file,
so
the
pr
you've
seen
was.
That
was
just
a
big
yaml
file
that
had
object
with
the
same
with
the
same
name
and
namespace
added
to
it.
So
that
was
the
problem.
We
move
that
to
house
to
have
a
single
object
in
a
single
file
and
the
file
name
is
generated
from
object
properties.
C
So
it's
the
api
version,
it's
the
kind
so
like
a
service
deployment,
stateful
set,
etc,
namespace
or
if
this.
This
is.
If
this
isn't.
If
this
is
not
a
namespace
object,
then
I
think
we
use
a
static
string.
There,
then
the
name
and-
and
that
would
prevent
it,
because
at
that
point,
that
pr
would
show
that
we're
overwriting
an
object
enamel
at
the
preamble
level
and
not
adding
one.
C
But
we
went
a
bit
further
than
that
because,
as
we
include
the
api
version
in
kubernetes,
you
could
have
like
v1,
beta1
and
v1,
and
that
technically
would
be
a
separate
file,
but
at
the
kubernetes
level
would
would
generate
would
override
the
object
anyway.
So
we
use
a
tool
called
config
to
generate
yaml
from
jsonnet
and
we've
added
a
smarter
logic
tool
to
detect
those
collisions
in
there
and
basically,
it's
going
to
refuse
to
generate
the
ammo
files.
C
If
you
have
a
collision
in
there-
and
I
think
that
a
big
upside
from
that
is
now
when
we
review
prs,
we
see
which
object
gets
changed
because
sometimes
when
you
have
a
complex
deployment
state
full
set-
and
you
just
want
to
add
a
variable
another
container
or
something-
it
wasn't
really
clear
what
you
were
editing
and
you
had
to
scroll
up
or
down
a
lot.
So
now
it's
now
it's
more
efficient
in
a
lot
of
other
things,
and
then
we
looked
at.
Can
we
do
something
at
argo,
cd
level?
C
Argos
cd
is
a
great
tool,
but
you
want
to
configure
some
of
the
things
to
make
it
even
better
and
like
make
it
even
safer
to
use
it
because
of
all
the
automation
in
place.
It's
really
easy
to
to
do
something
really
bad
when
you,
when
you,
when
you
make
a
mistake,
so
argo
cd
has
annotations
that
you
can
add
to
objects,
so
basically
any
kubernetes
object
and
one
of
them
is
prone
equals
false.
I
believe
that's
the
annotation.
C
I
mean
that
the
value
for
the
annotation,
which
will
mean
argos
cd,
will
never
delete
the
underlying
object.
So,
if
you
add
it
to
your
stateful
set,
it
don't
remember
the
annotation
name,
even
if
you
delete
it
in
your
yaml
files
or
wherever
argo
cd
is
getting
the
object.
Definitions
from
argo
cd
will
just
leave
it
alone
and
will
just
no
longer
manage
it.
C
One
thing
we
learned
as
well,
and
I
think
this
is
a
valuable
lesson
you
need
to
set
it
for
namespace
as
well,
because
in
one
of
our
first
drills
and
of
testing
the
this
change,
we
noticed
that
actually
argo
cd
deleted
namespace
kubernetes
deleted
all
the
stateful
sets,
so
it
didn't
really
help
without
setting
it
to
the
namespace
level.
We
also
added
annotations
that
make
argo
cd
refuse
to
update
a
resource
that
already
exists
and
has
annotations
that
specified
it's
managed
by
another
cd
application.
C
C
We
also
were
doing
some
exercises
fire
drills
around
things
like
argo,
cd,
after
updating
argo
cd
version,
does
the
annotation
still
work,
because
at
some
point
some
of
the
configuration
settings
changed
in
argo
city
itself
and
we
learned
a
ton
of
new
things
about
what
happens
when
things
go
wrong
when
we
just
basically
did
an
equivalent
of
of
chaos.
Monkey-
and
I
think
everyone
should
think
about
doing
this-
maybe
don't
start
with
production,
but
like
testing.
C
C
We
went
back
and
looked
at
all
the
volumes
across
a
lot
of
environments,
so
this
was
a
relatively
new
environment
that
was
done
when
we
have
all
the
automation
in
place
and
all
of
the
volumes
were
set
to
retain
properly.
But
we
went
back
look
back
at
the
first
environments
that
were
created
manually
and
I
believe
in
some
cases
we
found
some
volumes
that
should
be
retained,
but
weren't
or
vice
versa,
and
we
went
back
and
and
and
updated
all
those
settings
and
one
last
but
very
important
thing.
C
We
looked
at
our
processes
and
made
them
more
consistent
and
made
them
more
easy
to
follow.
So
this
was
our
first
incident
that
involved
a
large
number
of
customers
that
were
affected
by
it.
So
the
first
thing
we
learned
is,
you
need
to
have
a
way
to
list
all
the
customers
that
may
be
affected
by
an
issue
then
make
sure
that
you
have
all
the
valid
contact
points,
because
sometimes
that
may
be
a
different
person
than
the
login
that
someone's
using
have
a
consistent
way
of
conducting.
C
Have
someone
leading
the
the
contacting
effort
and
making
sure
we
know
when
an
incident
is
resolved
and
that
we
have
to
proactively
update
all
the
customers
about
the
incident
status
and
that
they
are
updated
all
the
way
to
it
being
resolved
and
that
we
just
don't
depend
on
someone
refreshing.
Our
influx
status
impact
data
status
page
and
that
being
the
only
way
they
find
out
about
how
things
are,
and
with
that
I
believe.
That's
it.
C
So
I
I
think
that
really
depends
on
what
data
you
have
and
how
much
time,
how
much
time
it
takes
to
recreate
it
and
how
accurate
the
data
will
be
once
you
do
it
in
each
way.
So,
in
the
case
of
zookeeper
that
data
didn't
really
change
often
so
just
reusing
valero
was
fine,
because
once
you
have
kafka
set
up,
the
data
in
valero
doesn't
really
change
often
so
that
was
fine.
C
For
example,
if
we
were
to
use
hourly
backups
for
data
that
our
customers
would
be
writing,
there
would
be
potential
of
losing
the
data
for
the
up
to
last
60
minutes,
for
example.
So
I
think
that's
always
a
case-by-case
choice
when
we
were
designing
which
volumes
to
retain.
We
chose
to
retain
zookeeper
for.
B
Us
there
was
about
three
levels.
The
first
is
it's
like.
If
it's
stateless
valero
doesn't
really
help
just
let
cd
redeploy
that
if
it's
stateful
and
the
data
has
not
changed
since
the
last
valero
backup,
then
valero
can
really
help
you
out.
But
if
it's
stateful
and
the
data
is
constantly
in
flux
like
it
is
for
us,
people
are
constantly
writing.
C
C
I
believe
it
may
have
worked,
but
I
remember
we
had
to
recreate
some
of
the
persistent
volume
objects
or
something
like
this.
So
I
mean
one
thing
that
was
that's
missing
in
here
is
we
use
our
own
crd
and
our
own
controller
to
manage
the
storage
tier?
We
don't
use
stateful
sets
for
that,
and
I
believe
this
is
one
of
the
places
where
we
may
have
to
wait.
We
may
have
needed
to
do
something
slightly
differently.
I
just
don't
remember
the
details
right
now,
but
right.
B
E
C
So
when
we
started
designing
our
cd
process,
that
was
one
of
the
discussions
we
had
do.
We
want
to
commit
the
generated
files
or
have
a
tool
random
and
maybe
also
generate
a
diff
at
the
pr
level
we
chose
to
explicitly
commit
because,
while
this
isn't
the
case
in
most
most
consensus,
but
we're
worried
that
the
version
of
jsonnet
in
argo
cd
may
differ
for
the
version
of
jsonnet
that
they're
tooling
other
tooling
may
be
using,
and
then
you
may
have
those
cell
differences
that
show
up
when
you
apply
it's
just.
C
So
even
with
just
the
numbers,
you
would
catch
that
it's
an
override,
but
often
when
we're
touching
stateful
things,
people
open
their
yaml
files
and
read
them.
It
just
just
seems
more
natural
and
seems
safer
and
that's
what
we
decided
at
a
company
level
and
in
all
the
ci
cd,
that
we
do
for
all
the
tools
that
we
have.
We
commit
the
ammo
files.
F
C
I
can
answer
that,
so
basically
we
I
mean
we're
proficient
with
jsonnet,
so
there
are
ways
in
which
you
can
pick
and
choose
which
objects
get
generated.
It's
relatively
easy
to
just
generate
a
subset
of
the
files
at
the
end
of
the
day
json,
it
is
just
an
array
of
objects.
I
mean
the
the
thing
that
we
render
at
from
those
files
is
an
array
of
objects,
and
you
can
just
filter
on
that
array
of
objects
and
choose
the
ones
you
want,
but.
C
That
you're
like
in
the
heater
or
you
can
just
copy
paste
right.
I
mean
there
are
multiple
ways
to
to
get
it.
It's
not
like
it's
a
closed
system.
You
can
get
like
you
can
json
eval,
and
then
you
can
do
a
lot
of
things
in
in
one-liners
as
well.
So
this
is
beyond
the
scope
of
this,
but
it's
it's
not
difficult.