►
From YouTube: How a Couple of Characters (and GitOps) Brought Down Our Site - Guy Templeton & Stuart Davidson
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How a Couple of Characters (and GitOps) Brought Down Our Site - Guy Templeton & Stuart Davidson, Skyscanner
Skyscanner have been enthusiastic adopters of Cloud-Native technologies and practices, adopting Kubernetes, Helm and ArgoCD as well as a wide range of other open-source technologies. However, adopting these technologies and practices in an existing environment doesn’t come without challenges. In this talk, Stuart and Guy will walk you through the longer-term cultural and technical challenges and benefits brought by adopting a GitOps model, as well as digging deeper into a global outage of Skyscanner’s website and mobile apps and how these approaches both exacerbated the problem but also sped up the time to resolution. They’ll then take the opportunity to explain some of the learnings from the incident with the hope that the insight they gained from this catastrophic situation will help you and your organisation not make the same mistakes.
A
It's
working,
hello,
hello,
hello,
gosh,
there's
a
lot
of
you
welcome!
Thank
you.
Thank
you
for
attending.
Obviously
this
is
the
last
one
between
between
you
and
beer,
so
we're
trying
to
keep
this
as
short
as
possible.
Welcome
to
our
talk!
So
today
guy
and
I
we
we
were
talking
about
how
we
might
pitch
our
talk
about
cube
to
kubecon
and
we
kind
of
agreed
that
the
vendor
talks
are
great
right,
they're,
they're,
good!
You
learn
a
lot,
but
actually
the
thing
that
brings
us
to
conferences
is
the
war
stories
right.
A
A
And
if
you
learn
something
from
it,
then
it
means
it's
less
of
a
failure.
Right
you've
got
something
from
it.
We've
got
something
from
it.
It's
a
data
point
rather
than
a
failure,
but
before
we
go
any
further,
I
need
this
thanks.
Okay,
we
should
introduce
ourselves,
I'm
the
top
one.
My
name
is
stuart
davidson,
I'm
a
director
of
engineering
at
skyscanner,
I
run
the
production
platform
tripe,
which
is
an
entity
a
group
of
squads
that
run
almost
like
a
platform
as
a
service.
So
we
run
all
the
interest.
A
A
Oh,
no,
there
you
go
all
right.
This
is
sky
scanner.
Now
some
of
you
might
not
know
what
sky
scanner
is.
Skyscanner
is
a
website
that
puts
together
a
bunch
of
flights,
hotels
and
car
hire
from
about
1200
travel
partners
across
the
globe,
and
you
want
to
go
somewhere,
maybe
multiple
places
at
once
or
in
a
kind
of
a
holiday
we
put
together
them
all
together
for
the
cheapest
possible
price.
A
Hopefully
we'll
have
lots
of
time
for
questions
as
well.
So
please
think
about
these
questions.
This
is
a
chance
for
you
to
learn
from
us
for
us
to
learn
from
you.
So
this
is
a
deliberate
attempt
for
us
to
share
some
information.
It's
maybe
not
the
best
forum
for
it,
but
there's
beers
after
this,
so
we
can
do
it
there,
sky
scanners,
cloud
native
journey,
so
I
joined
skyscanner
in
2015,
so
it
was
about
seven
eight
years
ago
and
at
that
time
we
had
about
one
mil
just
less
than
one
million
users
a
month.
A
Then
we
got
to
10
million
users
a
month
and
just
before
covert
hit.
We
were
over
100
million
users
a
month,
so
that's
10x
and
then
10x
again
and
it
has
been
a
wild
journey
right.
The
the
the
number
of
pieces
of
infrastructure,
the
number
of
systems
that
you
trusted
in
that
had
never
failed.
In
years
suddenly
became
bottlenecks
suddenly
became
things
that
wouldn't
scale
the
way
you
needed
it
to
or
suddenly
added,
latency
or
or
some
sort
of
I
don't
know
problem
that
you
hadn't
foreseen.
A
So
it's
been
a
really
fun
journey
kind
of
iterating
and
iterating
and
re-establishing
what
you're
going
to
do
and
thinking
about
it
from
almost
the
ground
up
several
times
along
the
way
we
started
off
in
data
centers
like
everyone.
I
guess
we're
what
about
15
years
old
as
a
company,
so
we
were
in
data
centers.
We
had
five
data
centers
across
the
globe
and
yeah
we
made
this
shift
to
the
public
cloud.
We
saw
the
potential
of
the
public
cloud
you
can
see.
A
You
know
we
were
scaling
so
quickly
that
actually
buying
infrastructure
and
getting
it
plugged
in
and
things
like
it
just
wasn't
sustainable
and
we
saw
the
opportunity
of
allowing
our
teams
to
decide
on
their
own
for
infrastructure
and
all
the
benefits
of
is
coding.
That
sort
of
thing.
So
we
made
this
shift
to
the
cloud
now.
I
say
that
in
one
sentence
one
bullet
point
as
if
it
was
a
simple
transition,
but
we
are
still
paying
the
price
of
our
migration
from
the
data
centers
to
the
cloud.
Even
yet
right.
A
So
at
one
point
we
just
decided
to
not
renew
our
data
center
contract.
That's
how
it
took
us
such
a
long
time
to
transition
the
workloads
from
a
data
center
native
into
a
cloud
native
solution,
with
all
the
the
fun
and
games
of
auto
scaling
and
and
and
failure
states
and
such
so
we
lifted
and
shifted
a
ton
of
our
workloads
and
we're
still
paying
that
technical
debt
off
at
the
moment,
but
we're
in
aws
fully
now
so
that's
that's
our
cloud
provider.
A
We
are
fully
aws
at
the
same
time,
around
sort
of
2015
2016.
This
thing
called
docker
was
starting
to
happen
and
it
was
at
the
time
where
people
in
conference
talks
like
this
would
say
who
here's
heard
of
docker
and
people
will
go
wow
and
they
were
like
who's
using
it
in
production.
They
go.
Oh
no,
no,
like
it
was
just
about
that
time,
where
it's
all
quite
exciting,
but
we
hadn't
quite
decided
what
we're
going
to
do
with
it.
A
Skyscanner
decided
to
try
it
out
with
our
ci
solution
right.
So
it's
quite
a
common
kind
of
first
step
to
try
a
container
native
ci
solution.
It's
actually
the
precursor
to
harness.
It
was
a
thing
called
drone
io.
It's
an
open
source
solution,
container
native
and
we
really
started
to
see
the
benefit
of
ephemeral,
build
agents
that
the
squads
could
define
right
instead
of
release
engineers
mucking
around
with
team
city
agents
and
ansible
scripts,
and
things
like
that
squads
could
define
their
own
ephemeral,
build
agents
in
containers
and
do
their
own
builds.
A
A
A
This
was
totally
groundbreaking,
like
it
totally
revolutionized
how
skyscanner
was
doing
things
because
it
used
to
be
that
services
and
squads
would
have
to
talk
to
the
release
engineering
team.
They
would
get
their
ansible
scripts
updated
they
would.
They
would
actually
have
a
release,
engineer
who's
like
have
you
done
your
tests,
etc,
etc.
A
With
this,
it
was
all
self-service
right.
So
squads
could
just
deploy
the
services
without
engaging
with
release
engineering,
and
it
got
to
this
crazy
state
where
we
didn't
actually
know
which
services
were
being
deployed.
It
was
kind
of
a
bit
scary
for
the
release.
Engineering
team
and
our
okr
was
like
get
eight
services
deployed
on
slingshot,
and
we
ended
up
with
like
62
by
the
end
of
the
month
and
our
okr
stats
for
the
for
the
month,
and
the
quarter
were
like
thousands
of
percent.
A
It
was
brilliant,
and
but
this
really
kind
of
progressed
our
our
path
along
using
containers
into
production,
and
it
it
made
a
it
was.
What
was
the
main
it
was
so
important,
because
squads
could
deploy
with
confidence
right.
That
was,
that
was
the
main
takeaway
from
it.
They
could
deploy
with
confidence,
and
it
moved
it
away
from
being
this
really
scary,
really
a
difficult
process
with
lots
of
human
interaction.
A
A
Ecs
took
us
a
long
way
and
in
fact
we
still
use
ecs
to
this
day,
almost
almost
were
out
of
ecs,
but
we
were
finding
that
we
were
having
to
rebuild
a
lot
of
components
that
we
were
seeing
happening
in
the
open
source
community
and
we
decided
to
take
on
kubernetes.
We
thought
this
is
the
way
forward.
Now
again,
there
was
a
lot
of
schedulers
at
the
time,
but
kubernetes
had
quite
a
strong
ecosystem,
as
demonstrated
here.
It's
continued
that
journey
and
we
went
through
many
many
iterations
of
trying
to
get
kubernetes
to
work.
A
Like
I
say,
we
haven't
fully
got
all
our
services
into
kubernetes,
yet
there
are
still
some
on
ecs,
our
first
iteration.
We
got
a
consultant
in
and
we
spent
a
lot
of
money
on
it
and
it
was
at
a
time
again
where
no
one
really
knew
how
to
run
kubernetes
at
scale.
So
they
learned
a
lot.
We
learned
a
lot
and
ultimately
the
solution
that
we
we
ended
up
with
at
the
end
didn't
last
about
it
was
about
a
month
a
month
and
a
half
before
we
started
to
see
problems
as
our
workloads
grew.
A
Then
we
iterated
and
we
started.
We
did
this
typical
skyscanner
thing
of
the
time
because
we
built
so
much
in
ecs.
We
started
to
rebuild
and
build
custom
solutions
on
top
of
kubernetes,
rather
than
leveraging
the
open
source
community.
That
was
one
of
the
main
selling
points
for
kubernetes
and
again
we
went
through
several
iterations
of
kubernetes,
as
kind
of
I
guess
architectures
at
that
time.
A
A
Well,
what
we
thought
was
enormous.
I
mean
we
had
the
keynote
by
cern
right,
they're
enormous.
We
thought
we
were
enormous
at
the
time
and
but
because
it
was
a
single
cluster,
it
meant,
if
any
failure,
the
failure
mode
like
any
sort
of
slight
failure.
The
whole
thing
went
down
and
we
had
one
particular
outage
where
the
business
asked
us
to
slow
down,
stop
changing
things
in
kubernetes
right,
stop
upgrading
kubernetes!
Well,
we
can't
the
the
the
upgrade
path
of
kubernetes.
A
It's
like
a
re-release
every
12
to
14
weeks
right,
so
we
couldn't
slow
down
and
that's
just
the
core
version
of
kubernetes
right.
You've
got
all
the
different
components:
different
parts
of
the
kubernetes
architecture,
so
we
couldn't
slow
down,
and
I
remember
I
was
about
to
go
on
stage
a
github
satellite
and
I
was
about
to
talk
about
our
continuous
deployment
system
and
how
rapid
iteration
reduces
risk
and
like
that's
the
way
we
wanted
to
go
forward
in
skyscanner,
but
paul
gillespie
who's,
our
senior
tech
tech
in
in
production
platform.
A
He
phoned
me
and
he
said,
look
we've
just
had
this
big
outage,
not
this
one
we're
about
to
talk
about.
We
had
another
outage
and
the
business
really
wants
us
to
slow
down,
but
I
couldn't
advocate,
I
couldn't
say:
let's
slow
down
and
then
literally
20
minutes
later
go
on
stage
and
say
we
should
go
fast
because
it
reduces
risk.
So
I
said
to
paul:
there
needs
to
be
a
different
way.
We
need
to
approach
this
in
a
different
fashion.
Is
that
is
there
any
other
way
of
doing
this?
A
So
all
credit
to
paul
and
all
credit
to
guy
and
all
credit
to
guys
team?
We
went
back
to
basics
and
we
started
again
and
we
started
to
reevaluate
all
the
technologies
that
we
were
using
and
actually
which
ones
were
valuable
and
try
and
de-clutter
all
the
different
things
we'd
installed
in
our
kubernetes
architecture.
A
A
A
A
At
the
moment,
what
we
decided
to
do
was
to
to
look
at
the
failure
states
that
we
had
to
work
with
and
create
an
architecture
around
that.
So
in
this
particular
case
this
this
isn't
actually
well
named.
That's
a
cluster,
not
a
cell.
Okay!
Sorry
about
that,
that's
that's
my
fault,
so
these
are
different
clusters,
there's
a
cell
per
region
and
within
a
cell
there
are
many
clusters.
A
Now
this
has
an
amazing
benefit
beyond
being
a
really
resilient
architecture,
one
that's
other
than
the
one
we're
about
to
talk
about,
hasn't
cost
us.
It
caused
us
a
production
incident
and
we
use
spot
instances
a
hundred
percent
in
production,
so
our
entire
infrastructure
is
spot
and
that
has
saved
us
literally
millions
of
dollars.
The
company
has
been
saved
millions
of
dollars
because
of
this
resilience.
If
our
nodes
are
taken
out
because
spot
termination
we've
got
that
resilience,
we
can
deal
with
the
failure
state
really
easily.
A
A
On
the
25th
of
august
there
was
an
engineering
all
hands
and
it
was
a
really
good
question
because
we
moved
about
70
of
our
workloads
into
the
sales
architecture,
and
I
was
asked
the
question
why,
by
by
another
team
that
hadn't
moved
yet,
why
was
it
a
good
idea
to
move
the
sales
architecture?
And
I
said
well,
it's
the
traditional
theory
of
constraints
bottlenecks,
but
we
have
solved
the
compute
problem.
A
It
is
now
the
most
available
resilient
thing,
we're
going
to
make
move
to
a
different
bottleneck
which,
at
that
point
was
traffic
routing
and
the
solution
for
traffic
routing
will
be
based
on
sales.
So
you
should
move,
but
cells
has
fixed
the
availability
problem
at
1553,
not
one
day.
No,
it
was
the
same
day.
A
B
So
how
did
how
did
we
go
from
478
services
running
serving
travelers
to
zero
and
it
good
stories?
Do
it's?
A
no-op
change
just
merge
this
late
on
the
day,
and
this
is
the
change
that
killed,
skyscanner
yeah
so,
as
you
can
see,
removed
some
ginger
brackets
and
what
could
possibly
go
wrong
so
very
quickly
soon
after
that,
rather
than
seeing
our
lovely
homepage
and
being
able
to
search
for
flights,
accommodation,
carhar
people
instead
didn't
even
see
this
nice
error
page.
Instead,
they
saw
this.
B
Which
it's
not
the
greatest
traveler
experience,
it
turns
out,
and
people
get
very
confused
as
to
why
they're
not
getting
access
to
night
for
their
flights
and
very
quickly.
Someone
was
able
to
guess
what
had
happened
and
was
smelling
istio
mesh
because
we
use
istio.
We
have
mtls,
we
use
authorization
policies
and
when
we
deleted
everything,
sdo
started
going,
no
you're
not
allowed
to
call
anything.
B
So
in
terms
of
what
actually
happened,
though,
and
going
back
to
what
stewart
mentioned.
We
have
a
slingshot
or
deployment
tooling
and
for
reasons
when
we
moved
to.
When
we
started
moving
to
kubernetes,
we
went,
we
don't
want
our
developers
to
have
to
relearn
tooling
or
create
completely
change
their
application
specification.
So
we
kept
the
same
slingshot.yaml
format
and
just
added
on
kubernetes
support,
so
they
could
add
a
few
different
fields
and
now,
instead
of
being
deployed
to
ecs,
they
were
getting
deployed
to
kubernetes
for
free.
B
However,
that
meant
we
were
in
a
weird
in-between
state,
and
this
is
where
the
trouble
originated,
because
we
were,
we
had
some
things
which
would
move
to
the
industry
standards
of
githubs,
and
then
we
had
all
of
the
things
that
users
deployed
onto
the
cluster
that
people
care
about
things
like
deployments,
services,
hpas
and
service
monitors.
Meanwhile,
gitops
is
deploying
things
like
sd
objects,
resource
quotas,
but
critically
namespaces,
which
means
that
gitops
is
controlling
something
that
contains
things
which
are
not
get
ops
controlled.
B
So
in
terms
of
how
that
works
for
those
who
aren't
familiar
with
gitops,
I'm
assuming
most
people
will
be,
but
we
have
a
repository
where
people
declare
like
their
service
and
on
board
it
and
say
which
cells
they
want
to
deploy
to
what
how
many
resources,
how
many
pods
they'll
have?
What
resources
look
like
that.
B
This
this
becomes
for
argo,
the
driver
of
what
applications
do
you
need
to
diff
against
the
cluster
and
apply
to
the
cluster?
So
when
this
became
a
invalid
thing,
it
suddenly
is
going.
Oh,
I
don't
need
to
apply
any
objects
to
any
cluster.
Oh
there's
a
lot
of
objects
there
that
I
don't
need.
I'm
gonna
delete
those
now
so
in
terms
of
our
recovery
that
that
meant
that
we
had
githubs
nice,
easy
revert,
the
pr
all
of
our
name.
B
Spaces
are
back,
but
all
of
those
objects
that
people
care
about
like
deployments
and
services
are
not
back,
which
is
a
bit
painful.
So
in
terms
of
the
recovery
we
had
to
first
mitigate,
so
we
got
people
our
nice
error
page
so
that
they
could
at
least
see
you
know
a
traveler
with
a
surfboard.
Instead
of
an
our
back
error,
then
we
had
prioritizing
getting
a
single
region
functional
and
then
also
shed
non-critical
load.
B
So
things
like
price
alerts
for
travelers,
where
we
want
to
prioritize
getting
users,
quotes
for
their
travel
or
information
about
the
tickets
they've
already
bought,
but
price
alerts
we
can.
We
can
do
without
for
a
couple
of
days.
So
this
is
a
graph
of
the
namespaces
and
you
can
see
very
obviously
where
we
deleted
everything
and
how
we
reverted
things
and
classed
argo
cd,
slowly
sort
of
started
going.
Oh
wait.
B
So
in
terms
of
why
that
took
so
long,
that's
that's
the
other
bits
of
restoring
clusters
and
that's
that's
far
tougher
because
it
was
manual
recovery.
We
had
dusty
run
books
that
were
designed
for
not
a
cluster
where
we
deleted
all
the
services
but
kept
the
same
cluster,
but
instead
a
cluster
where
we'd
we'd
just
wiped
out
deleted
the
entire
cluster
spun
up
a
new
cluster
and
restored,
and
we
also
discovered
that
our
runbooks
were
not
easy
to
follow.
B
So
this
is
this
is
what
the
recovery
looked
like
of
the
actual
services.
So
this
is
the
number
of
services
running
on
each
cluster,
and
you
can
see
here.
We've
got
this
like
stepped
recovery,
as
we
did
the
manual
restore
in
each
each
cluster,
so
eventually
get
back
to
a
point
where
we
left
it
overnight.
B
The
the
the
risk
here
came
from
us
crossing
this
chasm
between
we
want
to
get
from
where
we
were
to
using
all
these
standard
tools.
We
want
to
get
all
get
ops.
We
want
to
be
fully
based
in
that
world,
but
we
are
in
an
in-between
state
where
we've
actually
caused
potentially
more
risk
for
ourselves
by
still
being
in
that
state,
where
get
ops
can
cause
a
complete
outage,
but
then
gitops
can't
be
used
to
resolve
all
of
the
outage.
B
Yeah
I
mean
we,
we
provided
argo
cd,
the
power
to
apply
all
these
helm
charts
and
it
used
that
power
and
deleted
everything.
Finally,
templates
templates
with
logic
are
actually
good
like
we,
we
made
a
single
line,
change
to
a
templating
file
and
managed
to
wipe
out
the
entire
cluster.
This
wasn't
caught
pr
time
because
we
don't
have
tests
on
it,
because
it's
just
a
template.
Why
would
you
write
tests
for
that,
especially
when
you've
got
multiple
layers
of
templating?
A
Yeah
yeah
because
you've
just
said:
yeah,
okay,
it's
my
turn.
You
switch
my
mic
on
yeah.
Can
you
hear
me
excellent,
excellent
right
yeah?
I
just
wanted
to
quickly
talk
about
another
part
of
the
less
technical
side
of
this
right.
So
guy
talked
about
the
technical
solution,
the
change
that
was
made
to
the
file
that
was
right
at
the
root.
It
was
the
root
file
of
our
templating
and
the
engineer
that
made
the
change.
A
A
A
A
That
root
file
hadn't
changed
in
like
three
years,
so
it
was
a
failure
state
that
we
hadn't
considered,
but
we
know
that
humans
fail
so
when
it
actually
came
to
the
incident
management
side
of
things,
that's
where
I
genuinely
believe
skyscanner
sean,
because
everyone
that
was
responsible
needed
to
come
to
the
response
of
what
had
happened,
and
that's
not
just
engineers
we're
talking
legal,
we're,
talking,
user
satisfaction,
teams,
we're
like
the
ceo
et
cetera,
et
cetera,
like
everybody
came
into
a
room
and
other
than
the
cxos.
That's
not
fair.
A
A
Now
I
got
a
couple
of
quotes
from
engineers.
Now
the
everybody
was
tired
and
quite
emotional.
So
these
quotes
are
a
little
bit
emotional,
but
I
think
they're
awesome,
like
guy
says
we
actually
because
of
the
architecture.
We
got
to
send
everybody
home
really
pretty
early
on
about
11
o'clock
at
night,
but
people
were
tired
because
it
happened.
A
But
yeah,
like
the
the
positivity
and
calmness,
to
give
us
the
space
to
triage
and
recover
like
these
quotes,
are,
are
to
me
really
inspirational,
and
I
I
share
them
with
you
not
to
beg
us
up,
but
to
try
and
advocate
to
you
the
sense
of
achievement
of
that
blameless
culture
right
so
try
it
try
it
in
your
organization.
Next
time
you
have
a
problem,
don't
look
at
who
just
look
at
what
just
just
leave
it
out.
A
A
D
E
First
of
all,
thanks
for
sharing
failure
stories
are
always
interesting.
So
actually
I
have
two
questions.
The
first
one
is
about
you
mentioned
that
never
you
should
never
configure
global
configurations.
Do
we
have
stages
different
stagings
for
githubs
and
the
second
one
is
about?
Maybe
do
you
have
a
disaster
recovery
procedure
to
maybe
create
a
new
region
at
regular
intervals?.
B
A
So
if
that
were
to
happen,
obviously
there's
a
couple
of
layers
under
the
sales
architecture
in
terms
of
aws
and
such,
but
all
that's
infrastructure's
code.
So
I
think
we
could
do
it
pretty
quickly.
If,
if
forced,
maybe
we
should
run
a
war
game
on
it,
but
yeah.
It's
certainly
something
that
we
theoretically
could,
but
it
probably
wouldn't
be
our
first
option.
F
Hi
yeah
great
talk:
do
you
ever
do
any
dr,
like
drills,
about
like
recovery
from
things
like
this,
because
that
seems
potentially
like
something
that
no
one
ever
drills
for
like
total
outage,
but
you
know
what
happens
if
suddenly
everything
was
to
get
deleted?
How
quickly
can
you
get
back
up
like?
Is
that
worth
even
drilling?
What
do
you
guys
think.
A
A
I
guess
what
we're
trying
to
say
challenge
there
friction
there
right.
We
have
done
subsequently.
We
have
done
far
more
disaster
recoveries
than
we
have
in
the
past,
particularly
security-based
ones.
We've
done,
we've
done
some
really
good
stuff
with
our
security,
tribe
and
gone
okay,
a
malicious
actor
has
come
in
and
switched
this
off.
What
do
you
do
and
some
of
those
kind
of
theoretical
scenarios
you
don't
need
infrastructure,
you
don't
need
the
the
chaos
engineering
or
anything
like
that
to
actually
run
through
that.
A
All
it
was
was
a
powerpoint
deck
and
some
you
know
sugared
up
actors
that
kind
of
went
no
there's
things
around,
and
that
was
actually
a
lot
of
fun
and
we
spent
an
afternoon
doing
that.
So
that
was
pretty
cheap,
but
even
during
the
bat
do
you
want
to
talk
about
the
the
backup
issue
that
we
had
yeah.
B
Yeah,
so
as
as
I
mentioned,
we,
we
had
a
runbook
for
doing
restores
of
xcd.
At
the
time
we
were
using
cops
to
manage
our
clusters,
so
we
we
had
the
lcd
clusters
but
like,
as
I
mentioned,
we
we
had
practiced
restoring
it,
but
only
ever
onto
completely
fresh
clusters
where
we
imagined
the
entire
cluster
had
been
deleted.
B
We
hadn't
imagined
this
as
the
potential
scenario
where
we
were
trying
to
effectively
roll
xcd
back
in
time,
which
did
show
up
some
of
the
some
flaws
in
the
run
book
and
some
assumptions
where
we'd
we
made
incorrect
assumptions.
So
we
had
to
do
some
on
the
fly
stuff
we
have.
We
have
since
moved
to
eks
we're
using
valero.
A
B
Yeah
yeah,
trying
to
on
the
fly,
discover
why
cops
had
created
or
had
not
cleaned
up
old.
I
am
old
backups,
so
we
were
stuck
listing.
Thousands
upon
thousands
upon
thousands
of
objects
trying
to
go.
No
just
give
me
that
I
don't
I
don't
want
you
to
carry
on
listing.
I
just
want
you
to
restore
the
iam.
A
G
B
So
we
yeah,
we
immediately
reverted
the
pr
and
one
of
the
one
of
the
joys
of
the
the
blameless
incident
culture
was
we
had
someone
who
was
not
an
engineer
involved
in
like
the
actual
service
who
took
on
an
instant
commander
role,
so
that
freed
and
they
were
doing
the
coordination
between
squad.
So
that
freed
my
squad
to
do
the
the
revert
and
start
figuring
out.
How
do
we
restore
things
and,
whilst
another,
a
member
of
another
squad,
the
traffic
routine
squad
inside
sky
scanner,
was
able
to
do
the
the
the
effects
of
like?
A
And
that
incident
commander
role
is
really
important
and
there's
a
lot
there's
some
new
incident
management
tools
out
there
that
are
kind
of
slack-based
and
they
drive
this
attitude
of
having
an
incident
commander
who's,
not
doing
anything
technical
and
is
dealing
with
the
comms
to
the
cxos
and
the
legal
department,
because
the
legal
department
had
to
be
told
and
like
twitter,
like
our
social
media
teams,
and
things
like
that.
They
were
dealing
with
that,
so
that
the
technical
teams
could
get
on
with
fixing
the
technical
issues.
B
Because
of
those
those
issues
that
we
we
mentioned
in
terms
of
like
cops,
etc,
it
took
us
it
took
a
it
took
us
time
to.
We
had
to
dust
off
the
run,
but
we
had
to
figure
out
exactly
like
was
a
restore
going
to
actually
fix
all
of
the
services
we
needed
like
we,
there
was.
There
was
a
bit
of
there
had
to
be
a
discussion
in
terms
of
like
what
was
the
best
way
to
approach.
All
of
that
restore
so
yeah.
A
Well,
one
of
my
one
of
the
favorite
things
that
one
of
my
engineers
said
shout
out
to
caitlin.
She
says
before
you
make
a
big
decision
have
a
go,
make
a
cup
of
tea
right,
because
if
you
make
a
quick
decision
in
a
situation
like
that,
you
can
make
it
worse
a
lot
worse
and
there
was
at
one
point
we
were
like
this
is
going
to
take
days
because,
like
guy
says,
git
ops
replaced
all
the
name
spaces.
But
then
how
do
we
deploy
each
service?
And
that
was
actually
going
like?
A
D
H
B
So
we
currently
only
run
stateless
workloads
in
our
kubernetes
clusters.
Everything
staple
was
pretty
much
an
aws
managed
service,
rds
elastic
and
whatever
it
may
be,
and
the
the
one
thing
we
do
is
we
have
prometheus,
but
in
in
those
cases
it
was
we
we
have
those
we
had
some
ebs
volume
snapshotting,
I
think
as
well,
but
again
we,
I
think
we
just
ended
up
discarding
the
data
because
we
went
as
metric
data
like
we
have
thanos
for
long-term
retention.
B
A
Yeah
and
you're
right,
you
know
it's,
it's
the
third
rail
right.
How
do
you
deal
with
stateful
services?
But
in
our
case
such
a
large
proportion
of
our
services
are
stateless,
so
we're
benefiting
from
spot
instances
on
kubernetes
and
saving
a
lot
of
money
doing
it
when
it
comes
to
state
full
services.
We
take
a
different
approach
and
we
we
leverage
aws
far
more.
B
No,
I
I
don't
think
the
cluster
auto
scaler's
ever
done
that
to
us.
He
says
famous
last
words
we
have,
I
think
we
have
had
misbehaving
hpas
there
used
to
be
bugs
in
the
hpa
where
it
would
happily
scale
some
things
down
when
one
metric
said
and
the
other
metrics
were
unavailable,
which
led
to
like,
I
think,
a
prometheus
outage
led
to
some
services
being
scaled
down,
which
was
was
less
than
ideal.
B
J
B
B
Yes,
we
well,
we
we
use.
We
have
a
diverse
range
of
instance,
types
like
we.
We
use
the
cluster
autoscaler
and
auto
scaling
group
still,
so
we
we
do
need
to
revisit
those.
Those
instance
mixes
as
new
instances
are
launched,
add
them
into
the
potential
pool
effectively.
If
we
needed
to.
We
could
change
all
of
our
auto
scaling
groups
to
to
on
demand
instead
of
spot.
A
We've
also
moved
traffic
to
different
regions
right,
so
eu
central
one
had
a
problem.
Last
last
year,
sort
of
october
time,
where
I
I'm
gonna,
I'm
gonna
raise
my
fist
at
cern
again.
I
think
they
were
well.
No,
it
wasn't
certain,
but
every
so
often
cern
will
take
all
the
spot
instances
from
eu
central
one
and
they're
like
where's
ours,
but
yeah.
We
shifted
traffic
to
u.s
one
and
that
that
saves
us
a
lot
of
problems.
A
I
mean
we've
got
a
lot
of
savings
plans
and
things
like
that
that
we
can
set
up
load
groups
where
we're
using
reserved
instances-
and
we
have
done
that
in
the
past,
but
because
we
use
such
a
diverse
range
of
different
spot
instance
types.
We
can
get
away
with
it
and
we
haven't
really
been
in
a
situation
where
we've
needed
to
like
100
percent
move
to
on
demand,
and
we
have
used
on
demand
when,
when
there's
big
peaks
as
well.
F
F
A
There's
one
one
slight
challenge
with
you
using
something
like
ecr,
so
we
use
ecr
as
a
container
registry.
We
think
it's
amazing,
I'm
not
getting
paid
for
anybody
else
to
say
that
we
just
it's
been
so
robust
after
using
other
container
registry
products,
that
we
have
to
run
ourselves
in
the
past.
But
the
one
problem
with
it
is:
it's
got
the
account
id
in
it.
So
if
we
were
going
through
an
issue
where
there
was
a
regional
failure
or
an
account
problem,
and
particularly
with
sales,
each
region
is
a
separate
account.
L
I
have
two
questions
I
think
the
first
one
is
have
you
performed
any
change
in
argo
after
that,
because
we
had
a
similar
situation
like
one
month
and
a
half
ago,
but
it
was
only
on
the
environment
and
it
was
with
the
first
service
that
we
started
our
developers
to
deploy
with
argo.
So
during
the
first
week
of
deploying
argo,
they
changed
the
name
of
the
namespace,
with
same
situation
more
or
less
and
and
they
destroy
all
the
dev
environment
itself.
B
Nice,
so
we
did
disable
auto
prune
and
I
can't
remember
the
name
of
the
cascading
autoprint,
so
it
no.
That
does
mean
when
we
do.
When
we
make
a
change.
That
means
we
have
to
delete
things
or
clean
up
objects.
It
does
mean
we
have
to
at
that
level.
We
have
to
go
in
and
manually
manually
trigger
the
print.
B
We
didn't
we
well,
we
we
put
the
stateful
sites
back
in
place,
but
we
had
that
conversation
about
the
only
stateful
sets
we
had
running
are
prometheus
and
thanos,
and
we
went
well.
Thanos
has
shipped
the
longer
term
metrics
test
three
and
we
we
had
that
conversation
of
going
is
is
metrics.
We
don't
care
enough
to
restore
it
for
time
where
the
site
has
been
down.
We
know
the
site
has
been
down.
We
don't
need
prometheus
metrics
to
tell
us
that,
so
we
we
just
didn't.
L
A
No,
I
mean
we've
we've
war
gamed,
it
we've
done
it
at
a
high
level
with
our
cxos
and
talked
about
all
the
the
situations
and
scenarios
put
them
under
a
bit
of
pressure
about
making
decisions
and
then,
from
a
technical
perspective.
We've
also
talked
about
walking
through
a
scenario
and
then
discovering
more
and
more
problems
than
going.
Actually,
this
might
be
malicious,
rather
than
a
failure
and
and
then
figuring
out.
A
What's
going
on
from
that
and
with
that
there's
a
really
good
game
day,
I
honestly
don't
work
for
aws,
but
it's
a
really
brand
new
new
game
day,
which
the
teams
maliciously
deal
with
one
another.
So
that's
quite
a
good
one.
That's
quite
a
fun
fun
one
to
consider
other
failure
states,
but
we've
never
had
that.