►
From YouTube: 2021-02-02 Delivery team weekly rollbacks demo
Description
Delivery team demo rolling back a deployment on Staging
A
Of
course,
so
this
will
be
the
first
of
our
weekly
rollback
demo
meetings
and
this
week
alessio
has
very
kindly
volunteered
so
just
to
agree
on
the
terms
here
like
alessio
is,
is
leading
the
way
on
the
roll
back,
we're
all
fully
responsible
for
whatever
happens
next,
but
I'm
sure
it'll
be
totally
fine,
but
that's
fine
right,
whatever
happens,
this
is
our
sun
pit
and
we
will
use
this
to
plan
out
the
work
we
need
to
do
so
before
we
start.
We
should
let
some
people
know
right.
A
B
Then
I
realized
after
uric
helped
me
merging
it
that
because
of
yeah,
I
would
say
a
back
a
feature.
A
back.
I
don't
know
in
release
tools.
It
was
not
possible
to
tag
that
build
because
changing
a
file
which
that
is
not
a
code
file.
It
just
gave
me
a
very
small
pipeline
with
only
danger
jobs.
So
basically,
when
it
was
tagging
say
no,
this
is
another
full
pipeline.
I
would
not
tag
this
commit
no
matter
what
so
we
have
a
run-book
for
this
for
tagging,
but
it
didn't
work
as
well.
B
So
I'd
say
I
say:
let
me
check.
What's
the
real
difference
between
the
last
two
deployments,
and
this
is
kind
of
spoiler,
because
we
would
go
through
it
in
the
recording
and
there's
just
one
single
commit
that
is
removing
a
fischer
flag,
a
front-end
feature
flag.
So
I'll
say
we
can
just
throw
back
this
and
go
with
it.
It's
a
very
minimal
change
and,
in
my
opinion,
it's
safe
to
roll
it
back,
but
let's
go
through
it
together.
C
C
C
B
Yeah,
but
that's
the
thing
right,
so
we
are
doing
we.
We
will
go
back
one
version
in
in
staging,
not
just
one
comm.
I
mean
it
happens
that
the
previous
version
deployed
is
actually
one
commit
less,
but
we
will
do
the
the
regular
thing.
So
what
exactly
what
we
would
do,
in
case
of
say,
a
production
outage.
B
B
And
here
we
go,
this
is
staging
here
is
the
list
of
the
employment
with
commits
right?
So
that's
the
thing.
One
hour
ago
we
deployed
this
one
which
is
18
30
80
and
should
be
yeah
exactly
this
one
and
previous
deployment
was
4e59,
and
I
guess
if
we
go
here
in
announcement,
we
should
find
it
as
well
where
it
is
so.
B
B
I
was
hoping
to
find
a
rollback
information
directly
here,
but
it's
not
linked
in
the
in
the
hump
in
the
first
page
in
the
index,
but
I
know
that
is
connected
to
this
run
book
here,
so
just
for
the
sake
of
I'll
just
go
through
the
severity
s1
incident
round
book
now,
yeah,
because
I
know
that
is
here
so
there's
this
remediation
options.
That
tells
us
that
if
we
want
to
do
a
deployment
rollback,
we
should
first
check
the
deployments
contains
no
background
database
migration.
B
Now,
there's
there's
already
an
error
here,
because
this
information
is
not
enough.
We
actually
don't
want
any
type
of
post-deployment
migrations
because
a
single
post-deployment
migration,
unless
it's
just
adding
an
index,
can
we
will
make
impossible
to
run
the
previous
version
of
the
code.
But
let's
go
through
this
because
this
will
give
us
some
information.
B
Okay,
this
tells
us
how
to
check
for
background
migrations,
but
it
doesn't
tell
us
where
to
find
the
list
of
deployments
the
thing
that
I
was
looking
before
so
in
theory,
I
should
be
able
to
run
this
on
my
let's.
Let's
do
in
the
browser
would
be
easier
okay,
so
these
are
the
two
versions
and
yeah
I
did
some
homework.
B
This
is
the
compare
feature
of
the
of
the
two
versions
in
this
case.
It's
pretty
easy
because
we
see
with
only
six
changes
files.
We
can
just
look
at
them
manually
or
we
could
do
something
like
this
db.
Nothing.
I
usually
do
this
structure
sql.
If
I
remember
it's,
the
name
of
is
the
name
of
the
file.
So
if
there's
no
structure
sql
file,
there
are
no
migrations,
but
I
usually
check
structure,
sql
migrations
in
general
and
just
try
to
figure
out
if
something
it
isn't
there.
B
In
this
case,
as
I
told
before,
it
is
just
one
commit
plus
the
merge
commit.
This
is
the
delta,
and
this
is
basically
the
thing
it's
doing
is
removing
a
feature
flag.
There's
nothing
else
yeah.
This
is
the
controller
pushing
that
feature
flag
information.
It
removes
the
tracking
file
and
then
it's
documentation
and
tests.
B
So
I
I
think
we
all
agree
that
we
can
safely
roll
back
this,
it's
kind
of
exactly
the
type
of
situation
where
we
want
to
run
rollbacks
so,
okay,
we
know
that
this
is
not
enough.
There's
not
enough
information
here
for
us
to
to
figure
out
if
we
can
roll
back,
and
so
let's
go
here,
I
would
say
so.
We
check
at
the
background
migrations
also
in
general,
positive
migration.
B
B
B
E
A
B
Okay,
let's
quickly
check
if
the
environment
is
correct,
okay,
so
there's
no
weird
things
like
production,
cannery,
so
yeah.
If
it
breaks
something,
we
it's
still
fine.
Okay,
let
me
double
check
this
one
so
that
we
can
quickly
see
the
variables
nothing.
Okay,
the
version
is
here.
Check
mode
is
false,
so
he's
going
to
do
something.
B
B
In
our
current
rollback
implementation,
we
have
guitar
back,
which
is
after
the
main
fleet
and
is
only
on
staging.
This
is
something
that
I
discovered
recently
so
in
my
mind,
these
things
could
just
be
removed
from
here
and
became
a
manual
job
at
the
probably
at
the
end
of
the
pipeline,
without
any
dependencies,
so
that
if
we
want,
we
can
roll
back
easily
as
well
at
any
moment
in
time,
but
we
will
not
make
guilty
rollbacks
part
of
the
of
the
pipeline
just
to
what
about.
F
B
B
Yeah
so
because
I
I
wasn't
part
of
the
team
when
this
pipeline
was
created,
but
I
think
that,
because
of
all
the
complexity
related
to
migration,
big
changes,
because
when
we
started
this
was
kind
of
a
couple
of
no
a
couple
of
times
a
month,
is
it's
not
enough
once
a
month
or
twice
a
month
or
whatever
you?
You
could
bet
that
there
was
a
post
deployment
migration
in
that
package,
so
I
mean
it
was
too
early
to
think
about
rolling
back,
let's
say
that's.
B
A
A
C
D
C
Question
about
the
assets
job
while
we're
waiting
shouldn't
these
assets
already
exist,
they're
part
of
a
prior
deployment.
Wouldn't
they
already
be
available,
I'm
not
familiar
with
the
with
this
job,
so
you
know
what
is
it?
It
is
my
understanding
that
what
we're
doing
here
is
taking
the
assets
out
of
our
package
and
shoving
them
into
objects.
F
C
Yeah
and
if
this
is
a
rollback-
and
if
my
statement
is
true-
they
should
already
exist
because
I
don't
think
we
clean
out
that
bucket
at
all,
so
we're
just
slowly
acquiring
assets
over
the
course
of
time.
So,
assuming
my
statement
is
true,
we
should
be
able
to
just
simply
remove
the
assets
job
entirely
yeah.
I
think
this
is
something
that
is.
B
B
B
B
A
Be
a
good
one
to
investigate,
though,
like
I
wonder
if
we
could
certainly
do
a
test
one
week
right
where
we
consider
taking
them
out,
because
I
wonder
what
the
following,
like
the
next
deploy,
will
think
of
it.
If
we
didn't
run
them,
but
we
need
to
know
what
a
deploy
checks
for
before
it
kicks
off
again.
B
Yeah,
it
checks,
there's
a
there's,
a
table
which
is
schema
versions.
I
think
which,
as
every
row
has
the
name
of
the
yeah,
the
name
of
the
migration,
not
the
name
of
the
number
of
the
migration.
So
we
just
checked
the
migrations
on
disk
with
the
migration
on
the
on
the
table
and
run
all
the
missing
one.
In
order
so
session.
B
C
B
B
First.
Second,
we
don't
want
because
we
will
break
everything
because
yeah
we
are
waiting.
So
we
can
just
reiterate
this,
which
is,
I
think,
it's
very
important,
so
rail
this
is
specific
to
rails
application,
so
rails
caches
the
table
names
in
memory,
which
means
that
if,
if
you
have
let's
use
version
one
and
version
two
okay,
so
we
have
version
two
running
and
we
are
rolling
back
to
version
one
and
let's
assume
that
it
was
a
migration
adding
a
column.
B
C
B
This
is
no
okay,
so
it's
safe
to
add,
because
just
when
you,
when
you
restart
it
with
the
new
column,
will
be
picked
up
and
the
old
version
can
simply
ignore
it.
It
will
ignore
it
because
you
don't
know
what
to
do.
But
when
you
have
the
old
the
new
version
running
and
you
remove
it,
it
will
crash
because
even
even
the
older
version,
if
it
boots
and
the
table
is,
has
the
new
column.
It
will
cache
that
information.
B
B
E
B
D
C
D
A
A
What
are
the,
what
are
the
kind
of
main
pieces
around
rollbacks
like
what?
What
obviously
there's
the
the
moment?
Someone
asks
you
to
roll
back.
It's
probably
high
stress
right
you're
in
there
incident
zoom
and
you
ask
so
what
do
we
need
to
have
in
place
to
make
aside
from
a
working
pipeline?
But
what
else
do
we
need
to
have
in
place
to
make
rollbacks
not
terrifying
for
iteration?
A
B
I
think
we
need
a
clear
way
to
tell
if
it's
possible
to
roll
back,
so
the
thing
that
we
were
trying
to
run
at
the
beginning
of
this
goal
was
kind
of
yeah
go
here.
Look
at
this
look
at
that
and
then
just
a
lot
of
cut
and
pasting
and
making
sure
that
you're
doing
the
right
thing
and
yeah
this
be.
I
mean
in
now
we're
just
chit
chatting
and
trying
to
break
staging.
B
C
I
personally
would
love
to
see
this
kind
of
practiced
more
often,
maybe
used
in
a
fire
drill.
I
know
we
recently
did
a
hot
patch
fire
drill,
but
it'd
be.
It
would
probably
be
wise
to
have
the
rest
of
the
infrastructure
team
familiar
with
what
this
looks
like
what
we're
looking
for
what
the
pipeline
is
going
to
do
that
way,
they
could
also
chime
in
and
poke
holes
into
our
plane
and
learn
how
all
this
is
put
together
just
to
have
some
familiarity
that
way.
A
A
What
do
you
think
about
measuring
success
of
this
in
line
with
something
like
mean
time
to
resolution,
or
something
like
that?
That's
not
a
number
we've
normally
been
tracking
on.
I
don't
think
we
actually
have
good
data
for
that
right
now,
but
we
could
certainly
put
something
together,
but
does
that
feel
like
the
most
meaningful
numerical
measure
that
we
could
put
in
place.
C
A
I
mean
certainly
true-
I
I
think
we
can
certainly
look
through
historical
data
right
like
we.
We
don't
have
to
look
at
every
single
incident
like
we
already
contribute
to
mean
time
to
resolution
by
picking
revert,
mrs
into
pipelines
efficiently
deciding
how
we
run
stuff
making
decisions
around
baking
time
we
already
kind
of
we
just
don't
surface
this
information
very
widely.
A
I
think
there's
a
great
benefit,
though,
to
us
linking
the
kind
of
the.
Why,
in
that,
although
within
our
team,
definitely
confidence
that
we
could
certainly
have
that
as
a
key
result
right,
maybe
it's
like
confidence
level
within
the
team
for
everybody
running
this,
but
for
people
like
outside
of
even
infrastructure
department,
I'm
not
sure
that
that
really
helps
them.
Really.
I
guess
what
like
see
what
the
value
is
of
doing
this
work.
B
B
A
Yeah,
I
I'm
actually
kind
of
hoping
that
so
with
having
this
weekly
is
that
everyone
can
have
a
week
right
where
we
roll
back
each
week
and
hopefully
gets
less
and
less
time
consuming
as
we
find
things
we
can
safely
remove
from
the
pipeline,
and
it
gets
more
and
more
like
automated
eventually
one
week.
Hopefully
it's
just
press
the
button
right
and
then
everything
happens.
B
So
this
is
the
magic
variable.
So
when
you
say
rollback,
there's
this
variable
here
and
it's
the
one
that
changes
the
the
order
of
the
pipeline,
I
would
suggest
that
we
double
check
the
instead,
the
variable
name
that
our
rollback
feature
the
one
within
the
product
is
using
and
we
standardize
on
that
one
so
that
when
we
are
ready
we
can
start
dog
fooding.
The
the
product
feature.
B
B
B
So
the
asset
took
almost
13
minutes.
So
if
it's
something
that
we
can
safely
remove
will
be
yeah,
I
mean
it
would
be
good
because
the
migration
we
run
in
parallel
with
just
three
minutes
just
a
time
of
running
it,
checking
it
and
it
will
be
a
no
operation.
So
that's
fine.
Definitely
something
to
investigate.
G
C
B
B
B
A
B
Yeah,
but
if
we
want
to
do
this,
I
mean
we
are
just
waiting
for
the
pipelines
to
run.
So,
if
you
want
to
do
this,
I
think
that
we
should
consider
rolling
back
staging
just
by
default.
So
we
do
do
the
roll
forward.
We
we
go
cannery
whatever,
and
then
we
start
rolling
back
staging
so
that
we
are
sure
that
that
package
can
safely
be
rolled
back
or
that
our
procedure
is
still
working.
Something
like
that
right.
A
B
F
A
significantly
newer
version
like
we
deployed
to
staging,
we
are
at
canary,
for
whatever
reason
the
promotion
to
production
is
delayed,
and
then
we
deploy
to
staging
again.
If
there's
then
an
incident
where
we
deployed
to
production,
you
would
have
to
roll
back
two
versions
of
staging
to
be
sure.
If
we
can
roll
back
on
production,
I
I
feel
there
you
end
up
potentially
spending
a
long
time
rolling
back
on
staging.
While
your
incident
is
still
active.
B
A
B
B
Yeah,
well,
what
I'm
thinking
is
that
it's
something
like
if
you
deploy
run
the
full
qa.
So
you
know
that
is
the
smoke
tests
are
passing.
Then
you
roll
back,
and
then
you
redeploy
immediately
right
just
just
for
testing
the
procedure,
not
that
you
want
to
leave
staging
in
in
a
rollback
state
forever.
A
A
F
A
In
a
in
the
right
state
for
us
to
use
it
for
other
stuff,
but
I
do
like
having
some
confidence
in
rolling
back
in
the
in
the
process
and
also
in
the
future.
Definitely
having
a
confidence
in
is
this
package
able
to
be
rolled
back
would
be
quite
cool.
It
almost
feels
like
this
is
a
whole.
Other
environment
though,
like
this
since
staging
is
like
multi-use,
and
this
is
like
a
different
delivery
environment
where
we
actually
are
testing
kind
of
deployments
and
rollbacks
right.
A
Yeah-
and
that
might
make
I
I
definitely
I
think
we
should
put
something
together
for
that,
because
it
will
remove
so
much
of
the
release
manager,
pain
of
the
moment.
Where
someone
says,
can
you
roll
this
thing
back
right?
You
don't
have
to
make
a
decision.
You
can
just
look
at
some
test
results
or
can
tell
you
right.
Yeah.
B
The
thing
should
be
in
sync,
with
our
promotion
step
so
that
we
always
try
to
roll
back
to
what
is
in
production.
In
that
moment,
which
I
think
is
what
you
were
proposing
more
so,
instead
of
just
doing
that,
always
when
we
have
some
so
many
deltas
that
we
never
have
yeah
yeah,
I
completely
agree:
shall
we
take
a
look
if
the
help
page
is
giving
us
the
good
news?
Yes,
have
you
got
dramatic.
B
C
C
B
Yeah,
but
this
will
incr
I'm.
Maybe
we
should
do
something
incremental,
because
I
was
thinking
that
if
we
are,
we
are
in
an
outage
and
maybe
we
are
there's
a
bag
or
something
that
we
are
struggling
with
resources.
We
will
start
removing
more
machine,
I
mean
if
they
are
broken,
they
are
broken,
so
it
doesn't
matter,
but
it
I
don't
think
we
have
a
solution.
That
is
good
for
all
the
situation
right,
because
maybe
you
just
have
something
which
is
broken
and
you
want
to
revert,
but
it's
broken
only
on
a
subset
of
features.
B
B
B
F
B
F
Empty
desktop
yeah,
so
assuming
the
last
deploy
is
still
the
one
that
we
are
reverting.
It
will
be
empty
now,
functionality
wise.
I
think
it's
fine,
because
you
know
it
doesn't
change,
merge
requests
whatever.
I
can
see
there
being
a
weird
case
where
if
there
is
a
delta
and
it
finds
merge
request,
it
suddenly
starts.
F
Are
now
in
deploy
whatever
we
just
reverted
to?
I
think
which
I
guess
technically
is
correct,
because
you
are
reverting
them,
but
I
suspect
that
the
actual
deployment
tracking
you
know
what
we
show
in
the
widget,
not
the
labels
probably
doesn't
like
this.
In
fact,
I
think
it
will
still
keep
showing
it
as
being
in
production
yeah.
This
is.
B
F
B
B
D
E
B
Yeah-
and
I
was
still
thinking
that
in
a
scenario
when
we
do
we
roll
back
immediacy
production,
it
doesn't
really
matter
right
because
you
are
not
waiting
for
for
for
staging
to
to
roll
back
and
wait
for
all
of
this
qa.
You
do
this
on
the
side
I
mean
just
for
checking
if
it's
possible
think
that
marin
and
I
were
discussing
before-
but
it's
it-
it
doesn't
happen
during
the
incident.
G
Yeah
exactly
that's
that's
why
it's
possibly
probably
more
important
to
actually
do
this
automatically
and
have
a
report
so
that,
once
you
actually
do
things
in
an
incident,
you
can
do
yeah
exactly
good
point,
because
all
of
this
is
slow
right
like
this
is
I'm
not
like.
This
is
not
a
take
on
all
of
us
that
this
is
slow.
It's
not
like
this
is
the
best
we
can
do
right
now.
G
B
D
B
A
B
Yeah,
I
don't
know
I
mean
just
the
point
is
that
maybe
someone
wants
to
check
the
feature
flag,
removal
and
we
rolled
it
back
so
either.
We
clearly
say
that
we
are
not
rolling
forward
again
and
we
will
wait
for
the
next
outer
poi.
I
mean
it's
fine,
but
I
just
want
to
make
sure
that
if
someone
is
checking
for
it
and
staging
it,
why
the
future
flags
sit
there.
A
A
Yeah,
okay,
I
agree
that
sounds
good.
I
will
put
together
some
issues
on
that
that
you
can
all
contribute
to
mostly
because
we
haven't
got
working
epic
because
I
haven't
yet
written
at
okr.
So
I
feel
like
I'm
the
blocker
here
I'll
kick
all
those
things
off
tomorrow
morning
and
we
can
put
them
together,
but
yeah
it'd
be
great.
If,
by
next
tuesday
we
can
run
another
roll
back
and
see
how
that
works.