►
From YouTube: GitLab Production Incident 5158 Corrective Actions
Description
Stan, Matt, Andrew, Jason, Marin and others discuss some corrective actions following on from a production incident: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5158
A
A
So
there's
lots
of
fixes
that
we
need
to
do,
and
some
of
them
are
more
urgent
than
others,
and
I
think,
from
my
point
of
view,
the
the
first
thing
that
we
should
do
is
address
suggestion
of
basically
moving
the
authorized
projects,
workers
to
their
own
queue
or
sorry,
their
own
sidekick
shot,
and
that
will
that
will
be
an
immediate
like
temporary
relief.
While
we
talk
about
the
other
options,
sort
of
the
long
term,
like
really
big
project,
is
getting
rid
of
carrier
wave.
A
A
I
I
would
say,
from
my
point
of
view,
the
reactive
caching
issue,
with
the
that's
basically
masked
all
of
the
alerts,
because
we've
been
ignoring
the
alerts
for
a
month
that
we
should
fix
as
well,
because
we've
been
ignoring
the
all
the
threat
contention
sidekick
issues
because
of
this
known
reactive,
caching,
issue
and
stan
said:
there's
actually
because
I've
seen
on
the
on
the
on
the
issue
for
that
that
people
have
said.
Oh,
this
is
going
to
be
really
hard,
but
stan
suggested
quite
a
simple
solution.
B
B
A
I
will
also
do
this
is
something
like
every
time
and
this
isn't
actually
gcs
issue,
but
it
might
have
helped
us
if
it
was,
and
also
would
have
pointed
us
in
that
general
direction
is
every
time
we
have
a
gcs
issue.
We
talk
about
creating
a
service
in
our
service
catalog
in
our
metrics
catalog
for
gcs
and
it'll
basically
have
two
slis
one
is
like
sidekick
object,
storage
and
the
other
one
will
be.
Registry
object,
storage
and
perhaps
pages
as
well.
A
I
don't
know
if
we've
got
metrics
from
pages
for
its
gcs
operations
and
then,
if
any
of
those
things
go
south
we
can
go
and
look
there
and
one
thing
is
we'd
see
if
it
really
was
gcs
we'd
see
all
three
of
them
dropping
and
in
this
case,
would
only
be
the
sidekick
one.
So
we'd
know
there's
something
else.
That's
the
the
problem,
and
that
should
be
quite
easy
to
do.
B
Right
and
I
wanted
on
that
front-
I
know
our
gcs
support
person
noticed
the
transfer
times
taking
longer,
and
that
was
I
didn't
know
how
to
get
that
data
I
was
looking
at
the
console.
Is
there?
Is
that
metric
available
somewhere
I'd
love
to
have
that
at
least
reported,
because
that
would
help
us
distinguish
between
when
gcs.
A
B
Well,
the
other
issue
I
think,
is
to
investigate
authorized
projects
and
just
figure
out
we
can
make
that
much
more
efficient.
I
think
you
talked
about
the
data
database
bloat
issue,
but
I
feel
like
there
may
be
some
quick
wins
in
either
just
not
updating
everything,
or
at
least,
if
you've
already
updated.
You
know
a
user's
authorizations.
B
You
don't
need
to
schedule
another
job
for
that
user
right,
because
a
lot
of
these
things
are
redundant
instead
of
we
have
7
000
members
per
project,
you
don't
need
to
schedule
that
100
times
you
can
just
do
it
once
it's
bad
enough.
It's
doing
once,
but
you
know
100
7
000
jobs
is
much
better
than
700
000
jobs.
B
C
A
B
Are
we
capturing
that
list
of
like
to
do's
or
issues
we
could
attach
them
to
the
rapid
action,
epic
or
I'm
quite
writing
notes
and
then
we'll
try
to
find
all
the
issues
and
link
them
to
whatever
epic
we
create
okay,
cool?
Where
are
the
notes?
Sorry,
I,
oh
I'm
just
taking
the
miner,
I
don't
know
if
anybody
else
is
taking
those
I
just
I
wanted
to
just
get
this
out
there
right
now
and
then
we
can
put
it
in
the
chat.
C
B
So
that
I
guess
what
I
heard
andrea
said
was
that
number
one
we
want
to
break
up
the
sidekick
pools
like
move
some
workers
to
a
different
pool.
That
sounds
like
it's
probably
pretty
much
an
infrastructure
task.
Do
you?
Are
there
folks
in
development
we
should
be
involving
in
that
piece?
No
okay
and
then
removing
carrier
wave,
which
I
just
looked
at.
That's
that
file
upload
library,
yeah.
B
B
A
B
You
can
change
it.
You
can
change
that
behavior.
You
know.
The
trade-off
is
that
you
might
save
the
entry
in
the
database
before
the
files
persist
in
object
storage,
but
so
there's
some
danger
in
like
something's
there,
but
not
really
there
yet,
but
you
know,
there's
a
the
infrastructure
risk
is
higher,
I
think,
being
a
problem,
so
we
could
change
that
mode.
For
example,
we
could
change
and
say
do
that
and
you
know
we'll
live
with
the
consequences
and
see
what
happens.
D
And
that
that
protects
us
against
against
any
cause
for
for
a
delay
in
in
the
effects,
whether
it's
gcs
or
or
threat
contention.
It's
it's
a
it's
a
more
general
mitigation
effectively
trying
to
minimize
the
duration
of
the
the
the
least
db
connection
from
pg
banner.
That's
great.
That
makes
sense.
Yeah.
A
I
think
another
thing
is
that
this
also
illustrates
like
how
problematic
that
ruby
thread
contention
saturation
metric
can
be,
and
so
we
should
maybe
make
that
like
a
higher
priority.
I
think
at
the
moment
it
just
goes
to
slack.
It
doesn't
alert,
but
the
problem
is
well.
It
will
alert
a
lot
if
we
turn
it
on
now,
so
we
should
fix.
D
D
A
In
in
the
flood
of
alerts,
then
so
we've
got
four
different
alerts
that
are
all
very
similar:
we've
got
replicas
and
the
primaries,
and
then
we've
got
sidekick
and
then
the
synchronous
web
and-
and
I
think
the
problem
is
that
the
synchronous,
the
asynchronous
one,
the
sidekick
one,
is
saturated.
So
much
of
the
time,
mostly
because
of
archive
trace
worker,
that
we,
we
might
have
it
as
a
lower
level
alert
because.
D
B
D
B
A
A
A
C
A
C
A
Out
we
basically
ignore
that
from
the
alerts
and
then,
if
we,
if
we
see
saturation
alerts
for
anything,
you
know
while
that's
getting
fixed,
because
one
of
the
things
is
we've
just
been
ignoring
those
alerts
because
of
a
known
bad
cue,
and
if
we
put
that
in
the
sin
bin,
then
you
know
we'll
get
the
alerts
for
hey.
You
know
everything.
You
know
the
urgent
other
queue
is
is
totally
saturated.
C
Really
hate
to
interrupt
this
discussion,
but
it
would
be
good
to
know
whether
we
can
take
an
action
right
now
on
the
separating
the
shard
out
isolating
the
chart,
because
if
we
have
a
call
it's
in
half
an
hour
to
monitor
the
situation,
it
would
be
best
if
we
can
kind
of
split
off
and
start
doing
that
work.
So
we
can
disarm
that
call
if
possible.
We.
C
D
E
So
yesterday
we
saw
this
problem
where
one
of
the
shards
went
down
to
its
minimum
pod
scale,
because
it
was
struggling
to
do
this
work.
The
cp,
because
we
scale
in
cpu
the
cpu
usage
of
that
shard,
was
so
low.
We
just
scaled
down
to
the
minimum.
If
that
is
the
same
situation
for
this
new
shard,
we'll
kind
of
magically
avoid
creating
havoc
across
all
of
sidekick
and
instead
it'll
be
limited
to
just
this
one
char.
That's
the
sin:
bin
shard
per
se.
F
F
Okay,
we
can
address
that,
as
john
has
pointed
out
when
it
comes
to
the
cpu
and
individualizing
that
shard
that
doesn't
directly
solve
the
problem
of
the
pg
bouncer
being
overloaded
by
a
given
shard
set
or
shard
plural
yeah.
So
you
know
we
may
need
to
have
a
small,
dedicated,
pg
bouncer
for
the
problematic
ones.
The
good
news
is,
we
need
to
put
that
into
place,
but
configuring
that,
throughout
the
application,
consumers
is
actually
very
easy.
A
Yeah
so
my
take
on
that
is,
is
I
I
really?
We
don't
have
a
good
grasp
at
the
moment
of
our
pg
bouncer
pool
sizes
compared
to
you
know,
effectively
we're
taking
500
connections
into
our
postgre
into
our
primary,
and
then
we
divide
that
up
between
the
synchronous
and
the
asynchronous
and
and
the
maths
doesn't
add
up
at
the
moment.
A
Frankly,
I
don't
think
I
don't
know
if
you've
ever
looked
at
this,
but
it
it's
pretty
and-
and
that's
with
two
and
if
you
add
three,
that
that's
a
that's
a
good
project,
but
I
don't
think
it's
a
project.
We
do
now
like
that.
That
needs
proper
consideration
and
a
better
monitoring
and
planning
around,
because
you
know,
I
suspect,
if
you,
if
you
actually
laid
it
all
out
now
we
either
kind
of
quite
far
over
or
worse,
I
think,
we're
probably
quite
far
under
and
we're
actually
sort
of
throttling
ourselves.
A
You
know
if
you
look
at
the
number
of
pg
balances
and
the
you
know
whether
we
over
committing
or
under-committing
pg
bounces
to
postgres.
I
I
think
it's
just
been
done
on
an
ad-hoc
basis.
Like
we've
made
a
change
to
pg
balancer.
Here
we've
changed
the
pool
sizes.
Then
we've
made
a
change
to
postgres
over
there
and
I
don't
think
any
of
those
numbers
add
up
at
the
moment
and
it's
definitely
something
that
we
need
to
fix.
But
I
think
it's
like
here
be
dragons.
I.
D
A
A
Carrier
wave
was
the
thing
that
saturated
pg
bouncer,
but
the
thing
that
caused
the
problem
was
the
authorized
project
which
wasn't
actually
consuming
that
many
connections.
So
you
know
that's
also
something
you'd
have
to
take
into
consideration.
If
you
did
that,
so
you'd
have
to
do
some
sort
of
audit
of
of
the
or
you
you
know
what
you
could
do.
We
have
the
sidekick
attribute
tags
and
you
could
tag
all
of
the
the
carrier,
wave
sidekick
jobs
with
the
carrier
wave
tag,
and
then
we
could
use
a
selector
for
that.
D
D
B
D
A
C
A
C
A
D
D
Okay,
okay,
so
all
right!
So
I'm
I'm
sold
on
this
as
mitigation
for
the
for
the
for
the
trigger
condition
for
the
last
two
days
of
incidents.
I
do
still
think
it's
important
to
address
the
the
connection
pooling
because
that's
a
that's
a
more
general
class,
a
problem,
and
that
is
the
resource
that
caused
this
to
spread
across
a
broader
scope.
D
B
D
A
Yeah
yeah
yeah,
but
but
I
mean
I
do
think
and
that
this
is
something
that's
always
a
low-grade
worry
for
me.
I
do
think
doing
a
proper
audit
of
all
of
those.
You
know
the
sum
total
of
connections
that
we
hold
in
pg
bouncer
and
like
it
sounds
like
a
kind
of
project
that
that
we
do
need
to
do
as
well,
and
then
I.
D
B
B
D
Sure
yeah,
I
I'm
assuming
that
that's
more
work
than
splitting
the
pool
and
if
that's
a
bad
assumption,
then
we
shouldn't
bother
splitting
the
pool
we
should
just
focus
on
fixing
carrier
wave.
I
I
was
going
to
add,
though,
that
we
don't
necessarily
have
to
quarantine
all
job
classes
that
use
carrier
wave.
We
could
potentially
just
focus
on
the
ones
that
tend
to
be
very
high
volume,
because
the.
A
So
one
of
the
things
that
might
actually
help
there
is,
if
you
had
say,
I'm
just
going
to
use
hypothetical
numbers
now
but
say
you
had
10
round
numbers.
We
had
10
connections
for
the
carrier
wave
the
the
scary
carrier,
wave
workers,
then
what
we
could
say
is
the
hpa
only
goes
up
to
like
15
concurrent
sidekick
jobs
for
those
workers,
and
then
everything
else
will
start
queuing
rather
than
timing
out
waiting
for
a
database
connection.
F
So
two
things
is,
I
think
we
should
be
tagging
the
jobs
that
are
using
the
problematic
code
with
the
attribute
as
andrew
suggested.
We.
I
agree,
though,
with
matt
taking
all
of
carrier
wave
and
going
you
over.
There
is
definitely
problematic
if
we
have
two
specific
problematic
background
workers,
the
one
that's
using
carrierwave,
we
can
split
that
off
specifically
and
limit
it
in
the
most
generic
way
and
most
straightforward
way,
as
andrew
has
pointed
out
just
give
it
a
dedicated
set,
keep
its
hpa
low
to
make
sure
it
can't
max
things
out
on
its
own.
F
F
I
I
have
no
idea
what
of
that
is
anyways.
So
whether
or
not
we
actually
do
the
pg
bouncer.
Now
that
I've
heard
further
discussion
and
caught
up
on
some
more
details,
andrew
points
out
rightly
that
if
we
just
limit
the
problematic
set
of
workers
to
their
maximum
number,
we
can't
saturate
the
pg
bouncer
actually
because
they
won't
be
able
to
open
enough
to
saturate
it
right
that
actually
somewhat
mitigates
the
complexity
of
the
mitigates,
the
complexity
of
putting
something
in
place
to
mitigate
the
problem.
We
have
because
there's
code
issue.
D
A
D
F
F
But
yeah
I
mean
we
need
to
have
room
to
stretch,
but
we
do
not
need
to
put
it
so
far
above
our
heads
that
we're
jumping
up
and
down
on
each
other's
shoulders
before
we
hit
the
ceiling
right
there.
We
need
to
be
watching
the
metrics
on
what's
causing
the
actual
load
and
what
are
the
known
pain
points
so
that
we
know
if
we're
hitting
that
tipping
point
where
we've
got
these
backed
up
logs
due
to
gcs
being
slow
thanks
to
carrier
wave.
A
In
this
case,
though,
we
had
that
alert
and
we've
been
ignoring
it
for
a
month
because
of
a
different
piece
of
infra
infradev
technical
debt.
So
we
do
have
the
alert.
That's
why
I
was
talking
about
putting
that
reactive
caching
worker
into
a
sin,
bin
or
or
just
a
cl.
I
mean
we
could
just
change
the
alert
so
that
just
ignores
anything
from
reactive
caching,
oh
no!
You
can't
because
it's
per
shot-
it's
not
per
worker,
but
we
could.
A
B
So
I
guess
who
can
we
get
on?
This
assassin
was
created
two
months
ago
and
it's
in
the
backlog,
and
I
got
confused
with
sas
with
echo
with
junit.
So
this
is
actually
a
different
problem,
but
I
think
sam
you're
on
this
call
is
there.
Is
he?
Are
you
still
in
this
call?
Here
you
jump
off.
Okay,
I
guess
I'll
bring
him
up,
live.
C
C
D
So
this
this
probe
actually
the
the
stand.
This
was
the
probe
that
I
was
kind
of
showing
right
before
we
started
recording
the
the
one
that
scans
through
the
the
list
of
all
of
the
the
job
entries
in
the
in
each
queue,
so
we're
ready
to
so
we've
turned
off
that
that
probe,
I
think,
a
day
or
two
ago
and
we're
ready
to
we're
ready
to
enable
the
new
probe.
So
I
wanted
to.
B
D
E
E
About
creating
a
shard
that
explicitly
uses
that
feature
category
in
its
query.
Unfortunately,
some
of
these
cues
operate
in
catch-all.
Some
of
them
appear
to
work
in.
Where
do
I
document
it,
and
another
shard
like
these
are
multiple,
multiple
shards
that
these
cues
are
spread
across.
So
it
looks
like
urgent.
E
E
D
A
E
E
B
B
B
E
B
Well,
we
can
look
at
our
psychic
logs
and
tell
you
yeah,
yeah
hey.
Could
I
jump
in
with
a
quick
question
unrelated
to
what
y'all
are
talking
about
just
the
root
cause,
the
yeah
the
under
like
what
what's
actually
causing
the
underlying
issue?
Is
there
a
good
summary
of
that
or
I'd
like
to
be
able
to
communicate
that
out
to
like
you
know,
others
sure.
B
A
D
Just
sorry
which
connection
pool
so
this
is
this:
is
the
the
database
connection
pool?
That's
that
all
sidekick
jobs
share,
but
the
the
new
piece
of
information
that
that
that
folks
uncovered,
while
I
was
asleep
is,
is
that
the
is
this
bit
about
the
contention
in
among
the
the
the
thread
contention
on
oh
gosh?
What
is
it
the
so
having
having
a
flood
of
these?
D
In
threads
yeah,
yes,
exactly-
and
this
was
this-
was
implicitly
causing
a
delay
in
processing
other
other
jobs
that
we're
currently
running
on
other
threads
from
from
handling
the
the
storage
io
and
as
a
side
effect
of
the
of
that
of
that
kind
of
recurring
delay.
They
would
hold
their
database
transactions
for
a
much
longer
period
of
time,
which
longer
leases
means
they're,
the
the
pool
becomes
saturated
and
that's
that's.
What
kind
of
led
to
the
propagation
effect.
B
B
Yep,
okay,
awesome
yeah
I'll
capture
that
in
the
issue
and
I'll
share
it
in
the
channel
because
yeah
I
know
folks,
are
interested
and
yeah.
The
other
thing
they'd
be
interested
in
maybe
is
like
what
exactly
we're
doing
and
I
think
stan
you've
got
a
good
list
there
and
we
can
kind
of
update
that
as
we
go
so
I'll
stop
talking.
D
Thanks
for
bearing
with
me
no
that's,
fine
and
the
so
the
the
the
working
the
the
working
theory
is
by
by
moving
the
the
these
authorized
workers,
and
so
these
authorized
projects
workers
to
a
separate
to
a
separate
sidekick
shard.
They
will
no
longer
compete
with
all
of
the
other
job
classes
that
run
in
that
run
on
the
existing
sidekick
shards
and
therefore
are
much
less
likely
to
induce
the
contention
that
that
was
that
was
indirectly
producing
all
of
the
side.
D
D
B
D
B
D
B
G
My
meeting
is
set
to
start
for
us
to
all
go
join
the
incident
room,
I'm
just
watching
the
postgres
overview
for
idle
idling
transaction-
that's
not
happening
yet.
So
do
we
all
just
want
to
go
jump
over
to
that
zoom
and
we
can
go
muted
screens
off
to
do
whatever
we
want
to
do,
but
you
know
keep
charging
forward
scar
back
with
what
you
need
to
do.
I
just
thought
I'd
check
and
see
if
we
were
okay
dropping
this
one
and
joined
in
the
other
one.