►
From YouTube: 2021-04-27 Delivery team weekly rollbacks demo - part 2
Description
Running a dry-run rollback in Production
A
B
A
B
B
B
B
B
Okay,
so
I
think
we've
got
enough
people
to
go
ahead
and
get
started.
I
see
the
recording
is
recording.
So
let's
go
ahead,
get
started.
I
notified
the
on-call.
B
Just
to
validate
our
pre-change
steps,
we
have
the
first
item
already
checked,
which
I
believe
alessio
you
did
earlier
today,
but
just
to
validate
yeah.
I
just
want
to
make
sure
things
are
still
cancelled.
Yeah
the
post,
deploy
is
still
cancelled,
ensure
there
are
no
ongoing
deployments.
There
is
no
ongoing
deployment.
C
B
B
B
Yeah,
that's
fine,
yeah,
take
a
note,
so
technically
the
rollback
command.
We
really
wanted
to
state
that
we
want
to
roll
back
to
what's
currently
listed
as
current,
so
the
fa1
dd,
which
I
need
to
find
the
rest
of
that
package
name
for
so
to
do
that.
I'm
just
going
to
look
at
our
announcements
channel
to
figure
out
what
that
is
supposed
to
be.
D
A
B
B
B
D
Can
I
ask
a
question,
or
is
it
please
please,
how
do
we
check
if
there
is
a
current
deployment
now
or
not
what's
the
command,
for
that?
Is
it
from
chat,
ops,.
D
C
No,
I
wouldn't
no,
I
was
I
mean
I
think
ken
was
asking.
If
there
is
an
ongoing
deployment
and
out
of
reply
status
would
not
tell
you
so.
Okay.
A
C
B
Yeah,
so
this
is
it's
just
a,
I
guess,
a
language
thing
like
from
a
technical
standpoint.
A
deploy
is
still
ongoing.
We
just
paused
it
for
the
purposes
of
this
test,
but
inside
of
g
delivery,
I
just
ran
the
auto
deploy
status.
It
tells
us
there's
an
ongoing
deployment
and
that's
because
we
haven't
finished
the
production
deployment
that
we
were,
that
is
supposed
to
be
rolling
out
and
sure
enough.
Our
last
statement
is
the
production
environment
is
locked
and
that
it's
locked
because
we
didn't
finish
our
last
deployment.
B
So
I
would
answer
yes
to
that
question,
but
for
the
purposes
of
us
in
this
in
this
experiment
we
pause
the
deploy
to
roll
out
so
we're
safe
to
proceed.
B
That's
a
good
thought.
Sorry
meeting
the
last
deployment
to
canary
finished
at
9.
24
am
my
time.
Oh
13
24.
B
A
B
D
D
D
B
E
B
B
E
B
Ensure
the
pipeline
is
running
in
dry
run
mode.
The
easiest
way
that
I
know
to
check
that
is,
we
would
have
checked
mode
set
to
true,
and
that
would
also
be
reflected
in
the
output
of
our
other
jobs
of
this
pipeline
whoops
wrong.
B
B
B
I'm
expecting
to
see
a
play
button
is
the
play
button
going
to
show
up
after
we
get
through
production.
Finish.
A
If
we
are
running
in
check
mode,
I
will
expect
it
to
be
this
like
to
do
nothing
basically.
B
The
reason
why
we
run
this
job
is
because
we
need
to
tell
the
servers
to
update
their
apt
cache,
because
otherwise
they
won't
know
the
package
exists,
because
we're
rolling
back
that
database
should
already
be
populated
with
the
package
that
we're
going
backwards
to
so
hypothetically.
We
don't
need
to
run
this
job
at
all
during
a
rollback.
In
my
opinion,.
B
The
only
other
benefit
I
could
see
us
using
this
warm-up
job
is
that
we
pre-download
the
packages
prior
to
running
the
fleet
during
the
deployment,
so
that
speeds
up
the
fleet
job,
which
you
know
it's
gonna.
B
B
B
B
B
B
D
D
B
D
A
B
The
apt
cache
clean
job,
which
is
one
of
the
finishing
tasks
this
one.
That's
what
technically
removes
the
old
package
from
the
server
since
we're
in
dry
run?
I
don't
expect
this
to
do
anything
and
considering
the
fact
that
the
package
is
actively
running
and
we
haven't
installed.
The
older
version
like
this
shouldn't,
do
anything
yeah
every
deploy.
We
clean
out
the
app
cache,
because
otherwise
our
service
would
fill
out
fill
up
on
the
disk
space.
If
we
don't
do.
B
B
So
anything
to
talk
about
while
we,
while
we.
E
E
One
other
thing
that's
unrelated
specifically
to
this
dry
run,
but
it's
related
to
rollbacks.
So,
just
before
joining
here
I
was
doing
a
bit
of
an
epic
tidy
up,
so
we're
going
to
try
and
get
the
code
the
pipeline
for
code
rollback
epic
to
be
this
basically
and
we
can
consider
it
complete
when
we've
done
a
rollback
in
production
and
it's
all
gone
well.
E
So,
whatever
our
follow-up
test
after
this
is
and
then
I
will
move,
we've
got
quite
a
lot
of
other
issues
which
are
all
around
improving
information
like
dashboarding
commands.
Those
are
kind
of
lots
of
those
pieces,
I'm
just
going
to
move
those
up
onto
the
assisted
rollback,
epic
and
that
epic
will
be
the
thing
that
we
use
to
take
the
rollback
pipeline.
We
have
and
sort
of
fill
in
the
steps
that
we
need
to
go
to
until
we
become,
I
suppose,
confidence.
E
So
it's
filling
in
all
the
gaps
of
getting
from
here
to
the
point
where,
in
the
event
of
an
incident
we
automatically
check,
is
there
something
that
could
be
wrought
back?
If
it
is,
we
would
roll
it
back.
So
all
of
the
bits
that
go
in
between
those
two
things
so
I'll
move
some
stuff
around.
But
hopefully
it
means
that
we
are
quite
close
to
having
the
code
pipeline
epic
completed,
which
is
very
exciting.
E
We
just
have
the
small,
slightly
tricky
task
of
schedule,
assuming
this
all
goes
as
smoothly
as
we're
expecting
slightly
tricky
task
schedule:
a
roll
back
on
production,
which
it's
a
happy
problem,
though
it's
a
heavy
problem
exactly
it
might
be
worth
us
okay,
once
we
get
to
that
stage,
it
might
be
worth
us
really
carefully
checking
incidents
because
we
might
get
lucky
and
there'll
be
a
suitable
incident
that
we
could
just
run
a
roll
back
on,
and
that
would
be
by
far
the
easiest
way
to.
E
Well,
we
end
up
once
we've
done
this
dry
run
test.
We
end
up
in
a
interesting
problem,
which
is
that
we,
the
next
in
order
to
finish
this
task.
We
need
to
do
a
roll
back
on
production.
The
problem
is
really
hard
about
scheduling.
The
test.
Rollback
is
we're
taking
a
fully
working
system
and
potentially
breaking
it,
but
if
we're
in
an
incident
that
risk
is
a
little
bit
different
because
we're
taking
a
broken
system
and
potentially
fixing
it.
E
So
it
kind
of
comes
down
to
confidence,
but
I
think
it
might
be
in
some
ways:
it's
less
risky
right
if
we
actually
have
something
that's
broken
already
and
we're
trying
to
roll
back
to
fix
the
alternative
is,
I
guess,
the
other
safer
way
is
we
set
up
the
rollback
package
and
we
deploy
just
a
very
specific
package
that
we
roll
back,
which
I
think
we
should
aim
to
do,
but
it's
just
there's
so
much
scheduling
around
that
that
it
if
we
could
get
an
incident
and
test
it
on
an
incident.
E
That
would
be
certainly
the
easiest
thing
to
do.
B
I
think
the
type
of
incident
matters
like
if
it's
an
abuse
problem,
then
rolling
back,
doesn't
apply
and
in
fact
that
might
actually
make
things
worse,
because
we're
sitting
here
telling
the
service
to
go.
Do
some
tasks
unrelated
to
processing
a
user
abuse
request.
You
know
I'd
be
wary
of
that
kind
of
style
of
rolling
back
at
that
point,.
B
E
E
E
E
I
know
but
robert's
charts
are
super
amazing
and
you
have
to
ask
nicely
for
them,
but
they
will
be.
They
will
be
part
of
the
deployment
slo.
B
B
B
B
So
let's
go
back
to
our
job
or
pipeline
and
we
should
be
starting
the
fleet,
then
yay
so
get
lab.com
help
currently
running
zero
fa
and
we
wanna
that
should
say
fa1.
When
all
is
said
and
done.
B
E
A
A
E
D
B
B
No,
no,
so
the
reason
we're
doing
a
dry
run
here
is
just
for
an
experiment
prior
to
us
actually
doing
this
in
production
in
real
life,
just
as
a
way
to
catch
any
future
potential
error
scenarios
that
we
may
run
into.
I
think
all
right.
B
B
D
B
B
Doing
so
will
prevent
any
version,
differences
from
showing
up
still
after
a
rollback
is
completed,
and
then
we
could
enable
ourselves
to
utilize
rolling
forward
in
the
future
to
go
to
canary
re-enable
that
validate
the
change
is
actually
working
as
expected,
and
we
could
continue
on
to
production
again.
B
D
B
B
B
B
No,
I
think
this
is
what
alessio
was
trying
to
really
get
out
there.
Is
that
the
normally,
if
any
issues
happen
that
force
us
to
do
a
rollback?
Usually
it's
the
rails
code
base
and
not
the
giddily
code
base
that
causes
the
problem.
So
in
this
particular
case
we
made
it
a
manual
job
because
they
are
relatively
good
at
being
backward
compatible
with
each
other.
So
there's
a
chance
of
just
speeding
up
the
rollback
procedure
by
simply
not
doing
the
giddily
jobs,
which
I
agree
with.
D
So
we
expect
the
gap
between
roll
back
and
roll
forward
to
be
very
minimal
right.
B
Do
we
I
do,
the
quickly
is
a
stretch.
You
know
it'll
probably
take
a
solid
hour
plus
to
actually
perform
the
rollback,
but
yeah.
D
D
B
So
now
we're
just
waiting
on
the
giddily
roll
back,
and
I
guess
after
this
is
completed,
I
could
proceed
forward
with
the
re-enabling
canary
and
then
just
finishing
up
essentially.
B
B
B
B
B
B
B
B
B
So
myra
you
had
the
suggestion
of
ensuring
that
we
maybe
had
a
chat,
ops
command,
perhaps
to
validate
whether
or
not
we
have
ongoing
deployments.
Would
you
mind
creating
an
issue
for
that.
B
Okay,
should
we
always
string
canary,
we
decided
that's
always
a
yes.
B
That
sounds
informational.
We
need
to
add
the
cheaper
and
I'll
create
an
issue
to
address
the
adding
the
prepare
steps
that
alessio
brought
forward
into
our
book
so
I'll
tackle
the
creating
issues
with
these
last
two
and
myra
you've
got
the
action
to
create
an
issue
for
that
one.
So.
A
A
Not
sure
if
it
is
okay,
is
it
actually
wrong
because
we
do
have
since
we
canceled
the
deployment.
B
E
B
Yeah
all
right
so
meyer
and
I
will
create
some
issues
but
as
far
as
I
know,
there's
nothing
blocking
us
from
continuing
forward
with
our
next
actual
rollback.
E
Amazing
excellent:
that's
exciting
stuff!
What
do
you
want
to
do
with
the
deployment
that
we
paused
as
well
scovic?
Do
you
want
to
just
run
the
post
deployment
migrations
to.
B
D
Can
you
explain,
what's
this
retry
post
deployment,
migration
job.