►
From YouTube: GitLab upgrade plan analysis: 14.8.6 to 14.9.5
Description
In this video, two Senior Support Engineers walk through a plan to upgrade GitLab from 14.8.6 to 14.9.5.
Architecture:
- Single node Omnibus
- Ubuntu 20.04
- JumpCloud for LDAP authentication
- Mailgun for outgoing email
Links:
- The final upgrade plan - https://gitlab.com/bcarranza/upgrade-plan/-/issues/1
- Create a GitLab upgrade plan - https://docs.gitlab.com/ee/update/plan_your_upgrade.html
- Upgrade GitLab - https://docs.gitlab.com/ee/update/
00:00 Introduction
00:37 Architecture and the target version
02:46 Pre-upgrade: Intro
03:17 Pre-upgrade: Version-specific upgrade notes for 14.9
05:18 Pre-upgrade: Checks
08:36 Pre-upgrade: Backups
11:26 Upgrade plan: Upgrade path
12:37 Upgrade plan: Maintenance mode & rest of the upgrade
15:15 Common problems
20:05 Post-upgrade: Checks
24:02 Post-upgrade: Runners
24:50 Rollback plan
A
Okay,
Let's
tickle,
so
all
right,
I
guess:
I'll
start
hello.
Everyone
this
is
my
name,
is
Harish
and
I
have
a
degree
we're
both
senior
sport
Engineers
here
at
gitlab,
and
today
we're
going
to
be
walking
through
an
upgrade
plan
to
dissect
it
and
ask
questions
and
see
how
it
was
constructed
just
so,
we
get
a
better
idea
of
how
to
set
this
up
in
order
to
ensure
you
have
a
successful
upgrade.
So
great.
B
Can
you
see
my
screen?
Okay?
Yes,
all
right
perfect!
So,
as
you
noted,
this
is
a
single
node.
Omnibus
instance.
It's
running
on
Ubuntu,
2004
and
all
of
the
services
are
self-contained,
so
we're
not
using
external
redis
postgres
giddly
Etc.
We
are
using
mailgun
for
outgoing
email
and
we
are
using
jumpcloud
for
ldap
authentication.
The
point
of
this
section
is
to
just
sort
of
call
out
which
services
are
internal
to
the
node,
we're
upgrading
and
which
are
the
prominent
external
services
that
it's
relying
on
for
backups.
B
We
this
instance
is
configured
to
upload
all
the
backups
to
an
S3
bucket
and
that's
to
make
sure
that
if
the
VM
that
we're
upgrading
is
destroyed,
we
don't
lose
the
contents
of
the
backup
and
we
can
still
restore.
So
that's
a
quick
overview
of
the
architecture.
It's
relatively
simple,
but
do
you
have
any
questions.
B
So
right
now
the
instance
is
on
1486
and
what
I'd
really
like
to
do
is
get
it
upgraded
to
the
latest
stable
release
of
gitlab
right
now,
that's
15
7.,
the
upgrade
paths.
Documentation
means
that
we'll
need
to
break
that
upgrade
into
a
couple
different
phases.
What
we're
doing
today
is
just
talking
through
the
first
phase,
which
is
to
go
from
1486
to
1495.
A
Awesome
so
Let's,
let's
go
over
that
that
upgrade
plan.
That
sounds
great
to
me
and
just
to
clarify
by
the
way
you
said:
there's
no
Geo
set
up,
there's
no,
nothing
else.
It's
just
ldap
mail
gun
and
that's
it
right
does
whatever
you
showed
a
moment
ago,
perfect!
So
yeah,
let's
go
go
over
this:
let's
go
over
the
pre-upgrade
checks
and
tasks.
First,
cool.
Let's
see
what
you
put
together
in
here.
B
Cool,
so
these
are
the
things
that
I
want
to
run
through
immediately
before
actually
proceeding
with
the
upgrade.
The
point
here
is
to
make
sure
that
I
have
the
things
I
need
to
perform
any
tasks
during
the
upgrade
and
I
have
information
handy
for
things
that
might
come
up
during
the
upgrade
I
can
talk
through
any
of
these,
in
particular.
If
you
have
questions.
A
B
For
sure
and
I'm
really
glad
that
I
did,
let's
take
a
look
at
those.
There
is
a
note
here
for
14.9.
That
indicates
that
some
of
the
database
changes
could
take
quite
a
while
to
complete
and
so
I
did
use
this
to
inform
how
we
scheduled
the
upgrade
plan,
so
we're
going
to
perform
the
upgrade
at
a
time
where
we
can
sustain
that
amount
of
downtime.
Should
this
come
to
fruition
and
yeah?
A
B
Absolutely
yeah
and
just
a
quick
note,
I
don't
know
if
this
is
obvious
from
the
way
I've
structured
this,
but
I
have
these
check
boxes
here
so
that
I
can.
Actually.
This
is
a
living
document.
This
is
the
upgrade
plan,
but
this
is
also
what
I'll
use
to
conduct
the
upgrade,
and
part
of
that
is
to
keep
this
information
handy.
Should
we
need
to
do
the
copy
and
paste.
A
This
is
it's
definitely
valuable
to
have
everything
together
in
one
place.
However,
anyone
wants
to
do
it,
whether
it's
in
an
issue,
a
Google,
doc
or
I
mean
even
a
text
file.
It's
totally
fine.
As
long
as
you
have
the
steps
clearly
documented
what
you
need
to
do
and
any
potential
problems
you
could
run
into.
So
you
know
what
to
do
if
they
were
to
come
up
exactly
that's
right.
A
B
This
section,
what
I
wanted
to
do
is
combine
the
pre
and
post
credit,
pre
and
post
upgrade
tracks
that
we
have
documented
with
the
things
that
are
unique
to
our
environment.
So
that's
what
I
decided
to
do
is
just
link
out
to
the
docs
for
these
checks,
because
there's
no
need
to
sort
of
write
this
all
over
in
the
issue.
A
B
A
The
secrets,
one
is
something
worth
noting,
because
every
once
in
a
while,
you
may
find
out
that
the
secrets
file
was
was
important.
It
supposedly
instant
was
actually
recorded
to
a
different,
a
different
server.
Then
you
may
find
the
sequence
file
was
imported
alongside
the
backup.
So
you
may
get
an
error
here
on
certain
attributes
from
the
database
can't
be
read
anymore.
So
this
is
a
valuable
check
to
run
in
step,
two
which
will
kind
of
check
just
make
sure
that
the
database
values
can
be
read
and
the
secrets
file
is
compatible.
B
One
note
about
that.
One
of
the
things
I
really
like
about
these
tracks
is:
we
want
to
make
sure
that
the
gitlab
instance
is
healthy
before
we
start
the
upgrade,
because
if
there's
a
problem
that
needs
to
be
fixed,
we
should
fix
that
and
then
upgrade,
rather
than
upgrade
on
top
of
a
broken
instance.
So
that's
what
I
really
like
about
these
checks
that
are
here.
A
A
A
B
Sure
so
I
use,
so
the
point
of
having
that
readily
available
is
twofold:
one,
if
you,
if
I,
want
to
use
the
API
to
turn
maintenance
mode
on
or
off
I,
don't
want
to
have
to
go,
find
a
token
and
all
that,
while
in
you
know
when
I
just
need
to
turn
maintenance
mode
off,
the
other
thing
is,
let's
say,
made,
upgrade
an
encounter.
Some
problem
where
the
web
interface
is
not
quite
accessible,
but
I
need
to
make
some
change
via
the
API
I
can
do.
I
can
do
that.
I.
B
B
A
B
A
B
Cool
one
note
on
the
other
host
S3
bucket
I
may
have
mentioned
this
above,
but
this
command
will
automatically
send
the
backups
to
an
S3
bucket,
so
we've
managed
the
redundancy
there
yeah.
We
definitely
did
talk
about
that.
Yeah
cool
I
will
say
one
note.
The
point
of
recording
the
location
is
so
that
if
someone
else
needs
to
run
through
these
steps,
they
have
the
information
they
need
to
find
the
file,
because
I
wouldn't
want
to
give
someone
else.
B
The
task
of
hey
execute
this
rollback
plan,
there's
not
enough
information
in
the
rollback
plan
for
them
to
execute
it.
To
me,
that's
of
course
a
standard
I
use
to
accessible
back
then
is.
Can
I
give
this
to
someone
else
and
they'll
know
a
you
give
that
to
the
person
who
has
the
bits
needed,
but
B
would
they
know
what
to
do
with
the
information
she
provided
to
them?.
A
Yeah
absolutely
great
idea.
One
more
thing
I'd
like
to
draw
attention
to
is
something
you
said
a
moment
ago,
which
is
the
backup
is
sent
to
an
S3
bucket
I,
think
that's
important
and
we're
pointing
out,
because
you
want
to
make
sure
the
backup
is
accessible
and
if
the
VM
goes
down
or
you
lose
the
VM,
you
don't
lose
the
backup
too.
So
you
want
to
make
sure
that
you
have
that
somewhere
else
and
you
can
pull
it
over
as
needed
if
needed.
Yep.
B
B
A
B
B
A
B
Yeah,
so
yes,
I
actually
have
two
different
ways
of
doing
this.
I
use
insomnia
as
a
rest,
API,
client
and
I
have
requests.
Api
requests
there
stage,
so
I
can
just
click.
Send
and
that'll
use
the
API
to
turn
maintenance
mode
on
or
off.
This
is
just
my
fallback
if
that
doesn't
work
or
I
forget
or
can't
find
it
or
like
whatever.
Ultimately,
I
can
open
a
rails
console
and
paste
this
in
they're.
It's
just,
however,.
A
A
B
Yep,
in
this
case,
our
users
already
know
that
get
kind
of
have
a
heads
up
that
gitlab's
going
to
be
upgraded,
so
this
should
be
enough
for
them
to
say:
oh
yeah,
it's
being
upgraded
and
move
on.
There
is
something
really
interesting
about
this.
Now
that
I'm
looking
at
this
I
wanted
to
call
out
there's
so
much
free
and
post
work.
B
The
upgrade
itself
is
actually
a
relatively
small
part
of
this
and
I
think
that
that's
reflected
in
like
what
we
try
and
do
with
upgrade
assistance
where
we're
focusing
a
lot
on
the
plans
and
making
sure
things
look
right
before
we
proceed.
The
actual
upgrade
part
in
this
case
is
a
much
smaller
component
of
the
overall
plan
and
I
think
that's
kind
of
interesting.
A
B
Right
and
I
do
account
for
that
here,
like
so,
while
the
the
only
three
steps,
this
third
step
conceivably
could
be
a
lot.
There
could
be
some
thing
that
we
didn't
account
for
so
yeah.
It's
one
step.
It
could
be
just
wait,
five
minutes.
It
could
be
a
lot
more
than
that,
but
I
think
that
what
we've
done
should
position
us
well.
A
Excellent
yeah,
so
all
right
before
we
get
to
the
the
post,
upgrade
checks
and
the
verifications.
Can
we
talk
about
a
couple
of
possible
things
that
could
go
wrong?
I
know,
that's
not
really
mentioned
over
here
as
much,
but
it's
something
that
you
and
I
have
run
into
a
few
times
before
with
other
customers
and
I'd
like
to
make
sure
that
we
address
them
to
make
sure
that
in
case
they
happen,
people
know
what
to
do.
A
A
I
believe
we
can,
we
should
be
able
to
look
in
the
UI,
for
once
the
instance
is
up
and
running.
We
can
look
in
the
UI
in
the
main
background
jobs
section.
Let
me
pull
it
up
on
my
instance
and
double
check
the
the
naming
convention,
but
yeah.
We
would
look
it
up
over
there
and
then
there's
also
some
documentation
that
we'll
have
to
find
in
a
second
I
can
link
to
after
this,
but
yeah
it's
valuable
to
have
to
know
what
to
do
in
case.
A
B
I
think
that's
a
really
good
idea.
I'm
going
to
I
just
immediately
update
the
upgrade
plan
to
link
directly
to
this,
because
it's
a
good
call
I
need
to
have
them
handy
since
we're
since
we're
doing
this
live
I'll
start
by
just
linking
to
this,
but
really
I.
Think
the
right
thing
is
to
probably
copy
in
the
commands
to
the
commands
or
path,
either
copy
the
commands
and
or
add
a
link
to
the
path
in
the
web
interface
to
check
all
of
this
to
speed
the
time
to
figuring
out
how
to
recover.
B
A
So
so
that's
great,
so
that's
backward
migrations,
and
that
tends
to
be
one
of
the
bigger
things.
Just
a
little
note
on
that.
We
mentioned
it
before,
but
just
to
kind
of
mention
again.
If
the,
if
there
are
background
migration,
is
running,
make
sure
they
finish
before
the
subsequent
upgrade,
they
have
to
finish
all
the
way
through
and
that's
actually
the
reason
for
this
error
you're
seeing
in
front
of
you
over
here.
A
It's
it
comes
back
by
not
by
those
migrations,
not
finishing,
and
the
next
upgrade
is
started.
So
so
wait,
that's
one
thing.
The
other
thing
I
wanted
to
mention
is
something
we
touched
on
earlier
too,
which
is
the
secrets?
That's
the
other
issue.
I
see
a
lot
yeah
where
maybe
a
secret
file
was
deleted
or
moved
or
renamed
or
just
you
know,
the
entire
git
lab
was
migrated
and
the
sequence
file
wasn't
taken
alongside
it.
So
for
that
we
have
that
check.
That's
in
the
pre-upgrade
checks.
A
B
A
B
No
I
think
I
just
want
to
give
a
giant
plus
one
to
the
things
that
you
said.
I
think
like
yeah
definitely
keep
copies
of
old
versions
of
the
secrets
file,
especially
if
you
wind
up
needing
to
modify
things,
follow
the
docs
I
think
really
actually
follow
the
dogs
wait
for
things
to
complete
contact
us
if
they
don't
finish
properly,
is
Sound
Advice,
but
yeah.
A
B
That's
the
idea,
so
just
one
thing
to
clarify:
we
only
move
to
this
stage
once
we
either
observe
the
command
exit
successfully
or
fix
whatever
happens,
whatever
errors
come
up
during
the
upgrade,
so
at
this
point
like
gitlab
is
up
and
running,
gitlab
is
running
the
new
version.
This
is
all
designed
to
check.
Is
it
doing
all
of
what
we
expected
to
do
with
that
new
version.
A
Yeah
excellent,
so
great:
let's
look
at
the
first
one,
just
post
upgrade
checks,
which
I
believe
is
the
same
thing
as
the
pre-upreme
checks
correct.
So
it's
just
another
link
to
it.
Yep,
it's
a
review.
It's
writing
the
same
checks
again
and
make
sure
that
we're
not
seeing
anything
different
right.
What's
the
idea
excellent!
So,
let's
briefly
talk
about
the
next
few
steps
here
make
sure
the
root
user
can
log
in
that's
valuable
right.
A
Any
user
should
be
able
to
log
in
yep
turn
maintenance
mode
off,
and
you
mentioned
that
you
had
an
you.
Could
change
this
by
the
API.
The
rails
console
are
even
in
the
UI,
but
either
way
just
have
a
plan
to
do
it,
so
you
make
sure
that
that's
done
yep
and
then
make
sure
users
can
log
in
and
that's
a
great
point
you
are,
you
have
ldap
setups.
You
want
to
make
sure
that
users
from
the
local
passwords
and
the
ldap
users
can
log
in
right
exactly
exactly
anything
else.
B
A
Excellent,
so
all
right
great,
so
what
we
have
here,
we
then
we
have
the
products
can
be
cloned,
choose
a
few
different
groups.
That's
a
great
idea
commits
can
be
pushed,
and
just
something
I
want
to
mention
about
all
this
stuff
is
when
we're
checking
all
this.
It's
it's
hitting
different
parts
of
the
gitlab,
like
users
can
log
in
it's
checking,
Puma,
checking
Workhorse
checking
the
database,
make
sure
they're
all
up
and
can
be
accessed,
make
sure
commits
can
be
pushed
it's
checking
Italy
to
make
sure
it's
available
and
can
accept.
A
New
commits
products
can
be
cloned
same
thing
and
different
groups
and
different
projects
in
case
there
are
different
jobs,
that's
able
to
get
to
all
of
them
and
just
checking
a
variety
of
them
and
yeah.
So
it's
just
valuable
to
run
these
checks,
as
it
does
hit
on
many
different
parts
of
gitlab
and
to
make
sure
that
they're
all
accessible
and
running
as
they
should.
B
That's
a
really
really
good
point
about
these
things:
heading
different
components
of
gitlab
as
you're
talking
I,
think.
The
other
interesting
thing
about
these
tracks
right
is
that
the
things
we
do
in
post
upgrade
checks.
Give
us
a
high
level
like
here.
We
get
a
high
level
overview.
Is
that
component
like
up
and
healthy,
and
then
this
sort
of
layers
in
the
more
of
the
like
things
that
humans
expect?
B
A
Exactly
so
great,
so
we
have
a
couple
more
checks
here.
The
git
lab
check,
which
you
mentioned
a
moment
ago,
the
end
VIN
code,
which
checks
to
see
the
version
of
the
different
components
and
make
sure
that
looks
good
too
and
I
like
this
next
step.
You
have,
which
is
hand
it
up
to
the
developer
or
support
the
testing
because,
as
admins
on
the
instance,
we
can
only
do
so
much,
but
the
developer
is
the
ones
that
really
use
it
and
try
to
break
it
right.
Exactly.
A
That's
great
one
more
thing
I'd
like
to
mention
about
this
last
point
you
have,
which
is,
if
you
do
have
Runners,
make
sure
that
they're
that
they're
they're,
the
natural
version
of
the
new
version
will
get
lab
too,
because
I
know,
Runner
versions
have
to
kind
of
keep
in.
You
can
step
with
the
gitlab
versions
exactly.
B
A
B
So
we
had
talked
a
little
bit
before
about
the
specific
risk
of
what
happens
if
that
Google
Cloud
compute
VM
gets
deleted.
So
we
could
almost
take
a
declarative
approach
and
say
like
ensure
that
a
Google
Cloud
compute
VM
exists,
I,
don't
care
whether
it's
like
the
existing,
the
previous
one
or
a
new
one.
B
B
and
then
Spain
gitlab
up
and
ensure
that
it's
working
properly,
and
so
at
that
point,
what
we
then
do
is
tell
the
developers
FYI.
You
should
go
start
your
testing
again
right,
but
also
know
that
you
will
have
lost
any
changes
between
when
we
took
the
backup
and
now
now
we
take
the
backup
and
then
go
into
maintenance
mode
and
start
the
upgrades.
There
really
should
not
be
much
data
at
all
there.
B
Yeah
I
didn't
really
mention
this:
the
DNS
thing,
okay,
so
when
we
restart
the
gitlab
RB,
we'll
have
to
give
it
an
ALT.
In
this
case,
we
give
it
an
alternate
hostname
like
gitlabtest.brid.dev,
get
everything
working
there
and
then
Swiss
Medina's
hostname.
The
point
there
is:
if
the
users
are
trying
to
get
library.get
to
gitlab.readout
Dev,
they
either
get
whatever
state.
The
broken
instance
is
in
or
the
new
working
instance.
A
A
Yep
so
make
sure
you
get
that
as
well
and
one
one
more
thing
alongside
that,
which
really
may
not
matter
as
much
but
for
some
people
it
may
is.
If
you
have
any
other
files
in
Etsy
gitlab,
then
those
should
be
ported
over
too
so
really
that
entire
directory,
but
that's
just
something
to
keep
in
mind.
B
Leave
this
note
here
and
then
say
that
actually,
this
is
not
complete,
because
what
we
should
do
is
either
a
backup,
Olive,
Etsy
or
B
write
down
what
are
the
specific
things
in
Etsy
that
we
need
to
move
over
again
yeah
caring
for,
if
I,
have
to
give
this
rollback
plan
to
someone
else,
make
it
clear
for
them
like
this
is
exactly
what
you
need
yeah,
that's
a
good
point.
Thank
you.
A
A
B
A
A
I
would
create
a
new
backup
after
before,
every
single
upgrade
yeah
that
way,
whenever
you
restore
you're,
not
restoring
something
from
two
weeks
ago
or
from
before
users
were
or
from
even
a
few
days
ago,
with
a
lot
of
changes
that
have
happened
since
then.
This
way
you
have
the
most
recent
version,
and
another
bonus
of
this
is
supposing
there
are
lots
of
migrations,
and
then
you
want
to
wait
until
the
migrations
are
done
and
then
create
a
new
backup.
So
that
way,
you
don't
have
to
wait
for
that.
B
B
How
do
I
say
if,
if
this
plan
is
followed,
the
way
I
intend
it
to
and
meant
to
write
it
out?
Yeah
you'll
get
a
backup
immediately
before
starting
any
upgrade,
and
it's
funny
that
you
mentioned
what
you
did
here:
Rich,
because
I
just
had
a
ticket
with
a
customer
who
is
working
through
a
multi-step
upgrade
situation
and
they
got
a
few
steps
in
and
counted
a
problem
initiated
their
rollback
plan
and
I
was
just
so
happy
when
I
saw
their
email.
B
A
B
No
I
think
this
was
really
good.
The
plan
is
definitely
better
for
our
having
looked
at
it
and
I
think
that's
a
really
I
completely
forgotten
about
the
Etsy
backup
thing,
but
I'm
definitely
gonna.
Add
that
and
then
also
take
a
look
if
there's
anything
outside
of
that
ticket
lab
that
I
may
have
needed,
but
that
falls
in
the
space
of
I'm.
An
expert
in
my
own
architecture.
I
wouldn't
expect
you
to
know
what
those
things
were.
You
know
and
I
actually
think
that
it
should
be
fine
but
yeah.