►
From YouTube: Post-deployment Patch (hotpatch) Firedrill
Description
An SRE who has never done the process before will walk through creating the patch and applying it, from this will take a look at ways we can improve the process and the documentation.
https://docs.google.com/document/d/1SrsVEjlsS5NWSUFmbKQCEqOsCabmMyhOj3q6l4M8_h8 (doc is internal only)
A
Okay,
let's
get
started
we're
going
to
be
doing
a
post
deployment
patch
fire
drill,
and
what
this
will
be
is
a
hot
patch
that
is
going
to
be
applied
to
staging
henry
here
has
graciously
volunteered
to
be
the
person
who
applies
the
patch,
and
I
wanted
to
start
off
with
just
a
quick
introduction
of
what
hot
patching
is
and
a
little
q
a
so
first
off.
What
is
a
hotbatch
hot
patch
is
a
way
for
us
to
apply
patch
files
to
our
production
environment.
This
was
done
a
while
ago
before
we
had
kubernetes
clusters.
A
It
was
done
when
we
were
only
running
an
omnibus
and
we
created
it
because
we
needed
a
quick
way
to
get
changes
to
our
production
fleet
and
at
that
time,
that
was
even
worse
than
it
is
now
where
it
just
takes
a
very
long
time
to
build
packages
and
deploy
them
and
rolling
back
is
also
difficult.
So
that's
why
this
process
exists.
It's
gone
through
a
few
iterations.
A
This
is
the
current
iteration.
This
is
probably
the
last
iteration,
because
I
expect
this
process
to
go
away
entirely
once
we
fully
transition
to
kubernetes,
but
for
now
it's
still
something
that
we
use,
and
I
created
this
fire
drill
just
so
that
you
know
everyone
can
be
familiar
with
this
process.
We
can
put
this
recording
up
on
youtube
for
other
people
to
watch,
because
we
don't
patch
as
much
as
we
used
to
and
when
we
do
need
to
do
this.
A
lot
of
times
like
those
brain
cells
have
been
deactivated.
A
B
I
can
say
on
weekends,
because
I
was
once
in
this
situation
that
we
found
a
bug
and
on
weekends,
it's
hard
to
find
enough
people
to
make
a
release
right
and
it's
a
lot
of
more
work,
so
on
weekends.
I
think
it's
more
likely
to
do
a
hot
patch
to
come
over
the
weekend
and
then
on
the
next
monday
to
have
somebody
really
prepare
release
for
the
fixes.
That
was
the
situation
I
was
once
in,
and
I
guess
also
when
we
really
need
to
hurry
up
because
releases
take
long
to
build.
C
D
A
The
most
important
repos
yeah
we
we,
I
can't
think
in
recent
memory
us
using
the
hot
batch
process,
but
it
does,
it
is
private,
so
they're
like
the
patches,
aren't
exposed
anywhere
we
used
to.
We
definitely
used
to
do
this
quite
frequently.
I
don't
think
it
just
happens
as
much
recently
because
it
used
to
be
well
right
now
it
used
to
be
that
we
didn't
deploy
our
security.
Our
security
fixes
get
deployed
to
gitlab.com
before
the
security
release
goes
out.
A
A
A
Yeah
and
that's
and
that's
the
answer-
we
can't
patch
anything
except
the
rails
code
base.
That's
just
the
way
that
the
batcher
works,
so
this
is
limited
to
components
that
run
rails.
Other
things
like
web
shell,
workforce
pages
cast
registry.
Those
can't
be
patched
so
it's
something
to
keep
in
mind.
C
A
We're
you
know
we're
doing
nothing
more
than
slamming
patch
files
on
our
servers
and
hupping
puma.
That's
it
and
that's
how
it
was
designed
a
long
time
ago,
we've
carried
forward
that
design
and
we're
hopefully
going
to
be
deprecating
this
thing.
So
I
don't
think
that's
going
to
change.
How
do
we
initiate
a
hot
patch.
B
E
B
To
get
informed
of
all
of
that,
and
so
my
first
answer
would
be
to
look
for
documentation
because
normally
I
would
have
been
in
situations
that
I
wouldn't
know
exactly
just
have
some
clues.
There's
some
automation
somewhere
so
finding
documentation.
And
then
I
found
that
we
need
to
run
a
chatbot
command
and
then
prepare
a
patch
and
put
it
in
the
right
directory.
A
But
the
whole
thing
is
initiated
with
shout
outs.
This
was
you
know,
implemented
recent
well
recently
like
in
the
last
year.
There
should
already
be
an
incident
open.
If
there
isn't
an
incident
open,
then
you
just
put
a
placeholder
number
for
the
instant
number,
so
we'll
show
this
later,
when
we
do
the
demo
and
where
is
the
hot
patch
documentation,
I'm
just
going
to
reveal
this
since
we
kind
of
already
answered
it.
A
B
B
B
E
B
B
B
Also
something
which
I
always
need
to
look
up
again,
because
I
forget
about
it:
I'm
not
doing
that
often
enough,
but
that's
in
most
cases
necessary.
If
we
do
this
kind
of
hot
patching
right,
we
need
to
identify
an
mr
and
in
most
cases,
just
in
my
experience,
so
that
would
be
an
important
part
to
document
and
train.
B
B
B
And
from
that
I
started
reading,
so
in
this
case
that's
also
a
problem.
I
think
you
do
hot
patches
when
you're
under
a
time
pressure.
So
right
now,
I'm
on
a
time
pressure,
there's
an
urgent
problem.
I
don't
have
many
people
to
help
me
and
I
need
to
figure
out
how
to
get
this
in
as
fast
as
possible.
But
now
I
need
to
read
all
of
this,
but
I
think
it's
still
fair,
but
it's
giving
me
the
background
information
that
I
need
for
that
mostly,
but
it's
still
a
lot
to
read.
B
B
B
A
A
A
A
B
B
B
B
B
B
B
B
B
B
A
So
just
one
comment
here:
you
can
see
that
the
chat
ops
gives
you
the
directories
in
the
response,
the
chat,
ops
command
for
creating
the
hotbatch,
but
you
don't
see
staging
there
and
the
reason
for
that
is
that
staging
is
running
the
same.
I
assume
staging
is
running
the
same
version
as
production
and
canary,
but
let's,
let's
validate
that
with
the
auto
deploy
status.
A
B
E
C
A
B
If
we
go
through
staging
first,
we
need
to
get
into
this
directory
for
sure,
and
then
that
would
work
for
here
and
for
um.com.
I
would
have
thought
either.
We
don't
have
the
problem
there,
because
we
are
in
a
earlier
version
here
or
if
you
also
need
to
patch
there,
then
I
would
have
put
the
same
patch
into
this
other
directory
too
right.
That
would
have
been
my
assumption.
A
E
E
B
But
of
course
it's
just
about
versions
so
yeah.
I
think
I
would
have
gotten
it
from
the
run
book.
What
is
meant
okay,
so
we
have
this
created
for
the
hot
picture.
The
directories
to
go
on
with
that,
I
would
read
how
to
create
the
patch
actually,
and
that
would
mean
I
need
to
do
some
changes
to
gita
by
e
and
create
the
patch
out
of
that
and
that's
proposing
that
I
check
out
the
recent
release
number,
so
I
would
need
to
check
the
number
I
found
in
the
chat
ups
command.
A
Yeah,
that's
the
legacy
name
for
the
it
used
to
be.
We
had
two
projects,
one
for
gitlab
ce
and
one
for
gitlab
e.
That
was
when
we
had
the
split
and
we
would
do
merges
between
the
two.
Eventually
we
collapsed,
gitlab
e
became
git
lab,
and
then
we
had
to
get
lab
foss,
which
is
this.
The
gitlab
project,
with
all
the
non-free
code
removed.
C
D
A
B
B
B
B
A
A
A
A
The
exclusions
here
the
exclusions
here
are
just
for,
if
like
you're
picking,
if
you're
picking
an
mr
or
a
commit-
and
it
has
spec
test
updates,
you
don't
want
to
include
those
in
the
patch
because
it
won't
get
applied.
I
think
because
we
don't
include
spec
tests,
you
know
in
in
our
deployment
package.
B
So
there's
nothing
inside
so
I
can
just
copy
it
into
a
file
called
patch
there.
That's
now,
my
I
don't
know
it's
and
maybe
a
better
name.
Let
me
see,
do
you
have
something
in
the
documentation
for
naming
patches
here
so
numbered?
If,
if
there
are
several
of
them
to
determine
the
order
of
the
patches,
I
patch
okay,
that
should
work
and
give
it
the
issue
number
so
should
we
say
pitchfork
match.
B
B
That's
a
personal
thing
for
me,
I
guess.
But
after
you
have
read
it,
it's
it's
not
not
too
hard
to
do.
I
mean
the
steps
are
not
very
difficult
once
you
figure
out,
which
are
the
right
repositories
and
checking
out
code
and
stuff.
Isn't
that
hard
and
running
the
chat
ups
commands?
So
I
think
it
would
source
me
a
lot,
and
maybe
I
would
be
good
would
take
more
time
right
now
for
that
in
a
real
incident,
but
it
would
work.
So
I
think
that's
okay,
really.
B
B
B
B
B
B
A
B
B
B
B
A
Yeah,
usually
a
developer
is
prepping
the
patch.
Hopefully,
actually
the
developer
drives
this
process,
but
you
might
need
to
drive
it.
The
developer
spits
the
patch
and
it's
best
to
have
another
developer
review
it
if
possible,
because
it's
a
code
change
and
it's
a
code
change,
that's
going
to
go
directly
to
production,
so
normally
there
would
be
another
review.
A
A
These
are
the
other
environments
that
we're
not
changing,
so
I
this
was
kind
of
a
hacky
change.
We
made
recently
that
just
made
it
a
little
bit
easier
by
prepping
these
directories
for
you,
instead
of
having
to
create
them
yourselves,
so
chat
apps
created
these
directories.
These
are
just
the
places.
A
A
A
A
A
So
maybe
this
could
be
improved
there
aren't.
There
are
two
projects:
the
engineering
project
and
the
patcher
project.
Patcher
project
is
where
the
pipelines
actually
run.
If
you
go
to
the
comments
on
the
mr
scroll
down
a
bit,
you
can
see
that
there
are
links
to
the
actual
pipeline
that
runs
on
ops.
A
A
B
B
A
So
this
is
the
dry
run.
We
can
just
take
a
look
at
this.
This
is
doing
a
dry
run
against
staging
it'll.
Do
this,
even
if
you
only
prepped
the
patch
for
production,
in
which
case
it's
a
no-up
since
we're
actually
doing
a
patch
for
what's
running
on
staging,
we
should
see
interesting
output
here
in
which.
A
If
you
see
the
where
it
says
the
following
hosts
will
have
the
patch
applied
these.
These
have
to
all
be
true,
so
the
patch
directory
has
to
exist
that
equals
true
rails
has
to
exist.
In
other
words,
the
you
know
box
we're
deploying
to
has
to
be
running
rails
and
the
patch
has
to
be
a
change
so
the
when
it
runs
it
in
dry
run.
It
will
see
if
it
actually
changes
the
file.
A
So
when
you're
troubleshooting,
if,
for
example,
you
create
a
patch
that
doesn't
seem
to
be
getting
applied,
you
can
look
at
this
debug
output,
and
you
know
you
can
see
the
reason
for
it.
It
could
be
any
one
of
those
reasons
why
it
doesn't
get
applied,
but
more
often
it's
the
patch
structure
doesn't
exist
than
the
past
directory
for
the
version
that
were
that's
running
on
the
box.
A
B
B
E
B
A
So
if
you
scroll
up
to
the
top,
what
it
does
is
it
takes
a
look
at
what
image
is
running
in
the
environment?
Pulls
the
image
creates.
A
new
docker
file
applies
the
patch
file
uploads
the
image
to
dev
and
then
triggers
a
gate's
workloads
by
blind
dry
run
and
when
you
merge
this
to
master
it'll,
actually
do
a
kubernetes
deployment.
A
A
A
A
Apply
we
apply
to
both
staging
and
production
in
the
same
stage,
and
I
think
the
way
this
works
is
that
the
productions
are
manual
and
the
stagings
are
automatic.
So
when
you,
when
you
promote
to
the
deploy
stage,
all
the
staging
jobs
will
run
and
then
you
can
also
then,
in
that
stage
you
can
hit
the
manual
jobs
for
production.
A
If
you
want
to,
you
can
wait
for
qa
to
complete,
but
it's
optional
and
the
reason
for
this
is
there
have
been
some
situations
where
we
desperately
need
to
get
a
patch
on
production,
and
we
can't
afford
to
wait
20
minutes
for
qa
to
complete
on
staging.
So
we
leave
it
up
to
the
sre
and
the
developer
to
decide
on.
B
A
A
So
if
you
were
following
the
runbook
as
it
is
right
now,
you
would
hit
the
the
gprod
prepare
stage,
which
would
then
promote
to
the
deploy
stage,
and
then
the
staging
jobs
would
run
automatically
and
then
you
would
have
to
manually
run
the
production
jobs,
but
I
don't
think
that's
that's
yeah
based
this
is
not.
This
is
not
very
clear
in
the
docs,
because
I
think
this
changed
recently.
So
we
should
update
that.
A
B
B
C
B
C
A
A
B
I
think
after
that
I
would
then
just
I
don't
know,
do
some
cleanup,
I
guess,
writing
issues
and
pinging
people
and
trying
to
make
sure
that
on
monday,
that
would
be
picked
up
as
soon
as
possible
by
release
engineers
and
developers
to
get
out
a
real
fix
and
release
and
monitoring.
If
everything
is
right
and
then
I
would
be
done
for
the
weekend
right,
yeah.
B
A
Yeah
I
mean
normally,
these
changes
are
very
targeted,
so
we
haven't
had
any
problems
where
you
know
things
like
the
application
couldn't
even
run
after
applying
a
patch.
You
do
have
the
health
check.
It's
not
great,
but
it's
something
same
for
kubernetes,
where
we
have
the
same
readiness
check,
so
pods
won't
come,
you
know,
won't
come
up
if
the
readiness
fails
and
then
it
would
just
roll
back
yeah.
Our
best
bet
really
is
the
staging
qa.
That's
the
best
way
we
can
validate
things
and
that
just
takes
a
long
time.
E
I
suppose
the
other
aspect
is
this
would
hopefully
mostly
only
be
used
when
something
is
catastrophically
broken
already,
so
it
is
risky.
But
it's
kind
of
our
last
ditch
like
in
an
ideal
world
we'd
have
like
we'd,
refer
and
deploy
again.
It
would
be
a
better
one
to
go
through
the
full
pipelines
or
something
that
gives
us
a
bit
more
safety.
So
this
one
is
a
it
just
a
lot
of
risk
around
it,
so
hopefully
only
needed
to
be
used
when
things
are
like
s1
level,
broken.
A
Which
is
one
of
the
reasons
why
we
decided
to
start
this
fire
drill,
so
what
we'll
do
is
I'm
going
to
open
up?
Some
issues
to
you
know
fix
some
of
the
things
we
discussed
and
we'll
probably
do
another
one,
but
not
for
probably
at
least
a
month.
I
think
we're
not
going
to
do
these
like
every
every
week
or
anything
like
that
cool.
I
think
we're
out
of
time.
Does
anyone
else
have
questions
before
we
wrap
up.