►
From YouTube: 2021-03-30 Delivery team weekly rollbacks demo
Description
Testing out a staging deployment when we have nodes in DRAIN state
B
B
A
B
D
B
B
B
B
B
B
B
E
E
E
D
B
B
B
I
mean
in
theory
that
applies
to
many
things
I
mean
it
should
yeah.
I
kind
of
think
like
looking
at
this
output
from
the
check
command.
It
shows
things
like
upcoming.
Current
previous
makes
sense.
Previous
package
also
makes
sense,
I'm
just
kind
of
curious.
If
there's
anything,
we
could
do
to
make
it
more
obvious
that
the
previous
package
line
is
what
we
want
to
go
in
that
rollback
command.
B
Maybe
one
approach
would
be
to
add
like
a
snippet
of
the
commander,
so
it's
like
previous
package
version
number
and
then
like
slash,
shut
ups,
run,
deploy
rollback
with
that
version
number
in
it.
That
way,
you
can
just
copy
that
entire
command
and
run
it
instead
of
copying.
This
sort
of
template
then
replacing
that
placeholder
with
the
actual
version
and
then
running
it.
B
B
E
A
E
A
A
F
F
E
E
C
C
I
had
a
discussion
item
that
maybe
we
could
jump
to.
We've
got
the
epic
for
11.,
and
this
is
the
pipeline
for
code.
Rollback
we've
been
working
towards
now,
one,
I
think
it's
sure
we
haven't
got
an
sei,
but
I
think
we
can
see
how
far
we
can
get
anyway,
but
the
exit
criteria
for
this
epic
are
sort
of
complete
as
they're
written
and
the
due
date
is
today,
but
actually
I
don't
think
the
epic
is
complete
based
on
other
issues
and
other
things
we've
got.
C
E
E
I
don't,
I
really
don't
want
to
wait
till
an
instant
occurs
for
us
to
test
it.
I
want
to
test
it
when
we're
all
calm
collected.
There's
no
incidents,
we're
all
expecting
this,
it's
okay!
If
we
want
to
create
a
test
package
to
be
safer
sure
I
don't
care.
I
just
don't
want
to
wait
to
an
incident
to
test.
C
E
E
B
I
do
think
if
we
do
a
production
roll
like
the
first
time,
we
might
want
to
pair
that
up
with
a
production
change
lock
just
so
that
we
can
actually
pick
a
day,
knowing
that's
not
good,
knowing
there's
not
going
to
be
sort
of
self-induced
incidents,
because
there
couldn't
always
be
these
sporadic
random
ones,
but
just
we
have
a
day
where
we
know.
Oh,
we're
not
gonna
introduce
a
new
database
migration.
That's
going
to
take
two
hours,
for
example
beyond
that,
I
think
so
far,
yeah
we're
so
far
so
good.
F
Yep
sorry,
so,
basically,
one
that
I
have
been
thinking
about
is
that
we
need
to
roll
back,
and
there
is
a
there
is
a
deployment
in
progress.
So
in
that
case,
what
do
we
do
like
so
far?
Our
options
is
to
basically
cancel
the
the
ongoing
deployment,
but
we
don't
really
have
a
way
to
tell
when
it
is
safe
to
cancel
like
we
can
click
that
button
cancel,
but
at
least
I
have
no
idea
what
is
going
on
with
all
those
logs.
F
Those
are
not
really
readable
for
me,
so
it
will
be
nice
to
have
some
sort
of
tool
that
we
were
talking
about
last
week
to
safely
trigger
that
on
slack
and
then
the
tool
will
analyze
when
it
is
the
safe
time
to
cancel
that
one,
but
well
that
aside,
once
we
cancel
a
deployment
in
progress,
we
need
to
know
when
it
is
a
safe
time
to
do
it
and
then
what
are
we
supposed
to
do?
Are
we
supposed
to
roll
back
immediately?
E
C
B
C
E
E
Yeah
either
the
prairie
job
or
when
ansible
started
the
deploy
would
have
been
like
it's
already
in
drain.
This
is
a
problem,
but
we
didn't
see
that.
So
that's
a
good
thing
and
I
just
did
a
get
server
state
and
it
looks
like
that.
Node
is
in
the
rotation
again,
so
it
looks
like
the
rollback
indeed
completed,
just
like
we
wanted
it
to.
E
A
C
Which
should
in
theory
like
we,
don't
know
how
to
cancel
a
deployment?
But
you
know
that's
not
really
a
blocker
for
rollbacks
right.
We
have
a
slightly
annoying
state
where,
if
we
see
a
deployment
going
through
and
problems
start
to
happen,
we
don't
quite
know
what
to
do
which
we
should
find
a
solution
for,
but
that's
not
really
a
rollback
problem.
E
E
C
What
about,
if
we
thought
about
this
in
the
so
the
goal
we're
going
against
is
improving,
mean
time
to
resolution
right,
and
I
think
euric
has
been
painfully
aware
this
week,
and
I
know
robert
in
recent
weeks
that
this
number
is
quite
high
right.
So
actually
what?
If
for
now,
we
just
made
sure
that
if
we
ever
had
to
cancel
like
in
the
theory
that
we're
talking
about
here,
we
just
cut
it
before
post-deployment
migration
right
so,
oh
wow
everything
looks
terrible,
we're
deploying
on
the
fleet.
At
the
moment.
E
I
think
that
might
be
a
little
naive,
because
if
we
allowed
to
continue
up
until
post-deployment
migrations
and
if
things
are
getting
worse
even
after
say
the
first
two
nodes
deployed,
we're
gonna,
make
it
worse.
Until
we
get
to
the
step
where
we
could
actually
watch
the
current
set
of
jobs
complete
and
then
start
the
rollback
process.
C
E
B
B
A
incident
popped
up
at
people,
I
think
was
skyward.
Actually
hey
can
we,
you
know,
pulse
the
deploy
while
we're
dealing
with
the
incident
and
my
response.
There
is
basically
like
paul's
how
like
and
when
like,
can
I
just
cancel
the
pipeline
or
wait
and
then
the
result
is
okay.
We
have
to
wait
until
I
think,
like
post
deployment.
Migrations
are
done,
reversals
if
gitly
is
done,
but
because
there
was
a
dry
on
it
happened
so
fast.
I
wasn't
able
to
respond
in
time.
B
C
Cool
awesome,
I
add
I
asked
as
well.
This
is
completely
dominated
what
we
were
just
talking
about,
but
just
what's
in
my
mind,
I'll
add
it
to
the
epic,
but
I
checked
up
on
the
qa
tests
and
we
don't
run
them
as
part
of
the
production
deployment
pipeline,
but
there's
no
reason
not
to
have
these
on
the
rollback
pipeline.
So
it
feels
to
me
at
least,
that
a
good
use
of
a
bit
of
extra
time,
following
a
rollback
to
run
these
tests.
B
B
C
C
C
F
E
B
B
E
Wouldn't
touch
canary,
I
think
we
wouldn't
touch
staging.
We
would
probably
make
sure
that
whatever
is
on
staging
make
sure
that
it
does
not
get
promoted,
because
we
probably
need
to
at
that
point
identify
what
needs
to
be
fixed.
What
caused
the
problem
and
get
that
merged
in
place
and
prioritize
getting
that
into
staging
and
proceeding
forward.
C
C
C
We
should
see
how
that
goes,
whether
it's
almost
worth
or
so
it
might
make
it
just
too
complex,
but
with
the
production
rollback
like
with
a
rollback
almost
having
a
reminder
or
something,
that's
like
hey,
don't
forget
canary
or
something,
but
we
can.
We
can
see
how
that
plays
out.
For
now.
I
think
we
can
like
we
just
have
to
follow
the
run
book
super
closely
right.
C
Fantastic
and
then
the
other
thing
that
we
have
on
this
pipeline,
epic,
which
I'm
going
to
propose
moving
out,
is
we
have
quite
a
few
issues
on
here
around
kind
of
almost
like
information
and
posting
of
information.
So
there's
the
issue
which
I
need
to
do
something
with
like
putting
together
a
how
do
we
bring
all
this
stuff
together
into
a
human,
readable
proposal,
and
there
are
various
things
around
information
to
set
up
a
rollback
in
information
that
we've
done
a
rollback.
C
Just
whether
this
those
extra
tasks
should
be
parts
of
the
exit
criteria
for
this
epic
or
whether
we're
actually
saying
that
we
feel
like
production,
rollbacks
will
be
well.
Sorry
we'll
feel
like
a
rollback
pipeline
task
is
kind
of
complete
when
we
actually
feel
happy
with
the
production
rollback
or
whether
we
actually
want
to
go
through
and
call
it
complete.
Once
we
have
all
the
extra
pieces.
So
we
have,
for
example,
mirror
deployment
tracking
on
ops.
E
I'm
going
to
vote
yes,
in
my
opinion,
if
there's
anything
that
we
could
expose
to
make
our
lives
easier
during
a
high
stress
event,
and
I
feel,
like
I
haven't,
looked
at
the
issue,
so
I
don't
know
what
these
are,
but
just
based
on
your
description.
So
far,
they
sound
important
enough
that
I
feel
like
we
should
tackle
them.
C
C
And
then
the
other
thing
that
would
be
worth
doing
is
having
a
write-up
on
the
I'm
going
to
propose
that
we
separate
out
these
two
failure.
Scenarios
that
you
put
together
byron
have
one
the
one
we've
just
done,
one
that
we
haven't
tested
yet
on
a
separate
issue
and
then
have
like
a
summary
of
what
happened
today
on
the
one
we've
actually
just
tested.
C
B
F
E
They're
probably
going
to
be
treated
just
like
we
do
in
when
we
roll
forward,
so
if
they're
maintenance
but
they'll
just
stay
in
maintenance
mode
during
the
rollback
procedure.
If
we
want
to
change
that
behavior,
then
we
need
to
address
that
in
some
way,
but
there
could
be
cases
outside
of
deployments
that
would
lead
nodes
to
being
in
maintenance
mode,
so
it's
probably
wise
to
just
leave
it,
as
is.