►
From YouTube: 2020-03-27 Turning off free pull mirroring in production
Description
Part of https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/248
Related to https://gitlab.com/gitlab-org/gitlab/-/issues/10361
A
A
B
C
A
D
C
From
what
I
remember-
and
this
is
sort
of
kind
of
out
of
date-
facts
but
I
think
it's
a
reasonable
approximation
about
ten
percent
of
projects
are
public,
so
I
don't
normally
use
that
as
a
rule
of
thumb,
for
what
percentage
of
projects
will
be
paying.
It's
not
I'm,
not
saying
how
many
what
percentage
your
projects
are
paying
because
I
don't
know,
and
obviously
that's
a
bit
more
so
simple
about
10%
of
projects
of
public
public
mirrors
will
still
continue
to
work
as
well.
C
B
Myron
Myron
fronted
that
out
and
then
I
gathered
some
numbers
because
most
of
the
mirrors
were
actually
public
mirrors
and
that's
that's
the
only
ones
that
were
keeping
an
eye
on
right
now
because
of
the
weirdness,
with
the
plan
ID
and
stuff.
So
if
we
missing
mirrors
it's
the
paint
like
we
won't
get
alerted
if
paying
mirrors
aren't
being
processed,
which
is
not
good.
There's
an
issue
well.
C
F
C
A
C
C
G
C
B
C
B
F
A
So
it's
cants
the
number
of
workers
that
are
busy
so
and
we're
sorry.
So
you
know
we
have
concurrency
times
nodes
times
processors,
so
we
need
some
set
up
so
say
they
so
they're.
Basically
you
could
have
say
a
hundred
jobs
running
at
the
same
time
and
then
it
looks
at
how
many
jobs
are
running
and
and
basically
it's
a
percentage
of
that.
So
if,
if
you've
got
500
workers
and
there's
250
jobs
running
then
you're
50%
and
we've
kind
of
like
I
spoke
to
Kraig
about
this.
A
It's
not
super
intuitive,
but
we
sort
of
aim
to
be
in
the
band
of
like
23
to
43
percent,
so
basically
10%
either
side
of
33%
and
it's
a
little
bit
counterintuitive
like
if
we
had
auto
scaling
on
those
groups,
we'd
probably
aim
for
50%
and
just
kind
of
keep
it
on
that.
But
33%
is
kind
of
like
a
we
do
it
at
that
level,
because
it
gives
us
some
extra
capacity,
but
obviously
the
threads
will
slow
down.
A
B
A
B
C
B
E
E
C
F
F
C
E
F
Story
is
we
offered
full
mirroring
for
free
for
everyone
for
I
think
a
year
or
something-
and
this
was
an
effort
to
do
CIC
D
for
github
projects,
and
we
always
had
a
due
date
of
22nd
of
March.
And
a
month
ago
we
started
discussing
whether
we
are
going
to
end
up
doing
it
or
not.
Product
wanted
to
postpone
this
deadline,
but
we
it
scalability
like
pool
mirroring,
and
you
know
as
well
right
like
how
much
grief
has
pool
Mearing
caused
us.
It's.
F
So
we
spent
like
the
past
month
discussing
with
product
and
getting
the
approvals
to
actually
shut
this
off.
So
the
decision
was
made
to
do
that,
but
when
we
turned
it
on
on
like
at
the
beginning
of
the
week
or
Sunday
I
think
we
had
an
unintended
consequence
where
everything
was
just
stopped.
Basically,
but
the
idea
here
is
only
people
who
are
paying
customers
and
people
who
have
open
source
projects
will
be
able
to
use
this.
Not
everyone
cool.
H
E
A
That's
that's
exactly
what
you
know
every
it's
pretty
much!
This
is
why
we've
been
so
heavy
on
this
DJ
is
because
every
time
we
run
any
of
that
BPF
tracing,
it's
like
random
pull
Murr.
Is
that
no
one's
looked
at
in
the
last
year,
and-
and
it's
like
you
know
that
that's
exactly
I
haven't
actually
looked
at
that,
but
that's
exactly
what
I
would
expect.
A
A
C
A
C
Yeah
this
is
this
is
kind
of
a
challenging
thing,
because
it's
only
selecting
the
right
mirrors
like
it's
very
hard
to
tell,
because
this
is
the
thing
that,
like
changes
from
point
to
point
like
it's
very
hard,
to
tell
how
many
should
be
processed
to
anyone's
like
how
lumpy
this
is
right.
Now.
Yeah,
basically,
because,
like
the
first
set,
might
have
just
paid
a
bunch
where
there
are
only
a
few
to
be
processed,
and
now
there
are
more
to
be
processed.
C
C
B
C
But
I
haven't
made
my
changes.
Previously
we
had
a
thing
where
we
had
maximum
capacity.
We
could
process
at
any
one
time
and
I
think
we
were
constantly
at
that
capacity,
whereas
now
we
should
be
beloved
a
lot
of
the
time,
so
we
could
potentially
pull
more
in
because
of
that
capacity
that
we've
got
set,
then
we
would
have
done
before.
So
we
might
mirror
some
things
more
often
than
we
did
before,
because
we're
not
mirroring
as
many
things
as
we
were
before,
but
I
don't
know
how
to
verify
that
right
now,.
B
D
B
A
Should
we
should
put
effort
in
to
smooth
that
out
if
we
can
somehow,
because,
like
that
sort
of
lumpiness
has
impacts
downstream
right
on
like
Guinea
and
post
Chris
and
Ritter's?
And
you
know
if
we
can
make
that
smooth
generally
you'll
get
much
better
performance
and
our
hard?
It
is
because
I
don't
have
any
idea
about
that.
Shit-Eating
algorithm,
but
probably
I
also.
C
And
I
this
is
a
classic
sort
of
like
I'm
gonna,
say
this,
but
I
don't
think
we
should
actually
do
it
now.
I.
Also
wonder
if,
like
we're
processing
many
fewer
mirrors,
we
could
make
the
scheduling
simpler,
because
you
know
we
don't
have
to
like
we
basically
before,
like
from
what
I
gather
it
was
the
process
of
like
processing
mirrors
as
fast
as
possible
like
up
to
our
limit,
but
not
beyond
it.
H
A
B
H
But
looking
at
the
overdue
graph,
we
should
turn
through
get
first
and
then
get
to
a
state
where
we
have
lower
load
rights.
So
it
seems
that
the
the
dip
was
first
rescheduled
to
mirrors.
We
weren't
supposed
to
run
so
we
just
DQ'd
those
and
so
I
click
was
very
happy
about
it.
But
now
we're
just
scheduling
more
stuff
that
we
actually
need
to
process
right.
H
A
A
But
Sean
going
back
to
your
original
point
like
I,
don't
know
this
code,
but
it
seems
like
everyone's
a
little
bit
where
the
scheduling
card,
but
it
seems
like
everyone's
a
little
bit
sort
of
like
it.
It
doesn't,
it
seems
like
it
has
very
high
cognitive
overhead
to
actually
understanding
what's
going
on
in
it,
and
maybe
you
know
to
your
point
about
simplifying
it.
Maybe
that's
a
signal
that
it
needs
to
be
simplified.
C
Yeah,
it's
one
of
those
things
where,
like
the
cognitive
load,
is
there
because
we
made
a
lot
of
changes
and
it's
probably
not
a
great
idea
to
change
it.
If
you
don't
understand
why
we
made
the
changes
in
the
first
place,
unless
something
like
this
happens,
where
it's
like
an
external
thing
that
would
like
cause
sort
of
a
rethink
but
I
think
we
could.
C
We
could
consider
looking
at
that
as,
like
you
know,
just
completely
replace
it
by
a
feature
of
like,
like
you
know,
isn't
like
do
the
old,
like
capacity
management,
do
the
new
capacity
management
yeah
from
here,
because
we
have
this
external
thing,
which
should
reduce
the
the
load
I'm
just
trying
to
figure
out
as
well
like.
What's
because.
A
C
C
A
B
D
A
A
B
C
One
thing
I
can
do
which
I'm
gonna
try
and
do
now
is
to
try
and
figure
out
how
many
free
private
mirrors
have
not
been
updated.
Since
we
did
this
because
that's
like
it
doesn't
tell
us
anything
about
what
we
should
be
seeing
in
the
graphs,
but
it
does
tell
us
something
about
like
correctness.
Right,
like
you
know,
is
this:
is
this
actually
working
so
I'm
gonna,
try
and
figure
that
out
right
now.
E
C
Yeah
my
bigger
concern
right
now
is-
and
this
is
because
I
changed
the
query
so
late
top
of
my
right.
It's
correct
now
so
like
if
we
are
currently
not
scheduling
any
mirrors
that
we
shouldn't
be,
and
we
a
scheduling
all
the
mirrors
that
we
should
be.
Then
we
can
like
this
is
lower
than
it
was
before
and
we
could
just
leave
it
running
and
if
it
doesn't
even
out,
we
can
look
into
that.
C
But
if
we
are
scheduling
something
that
we
shouldn't
be
or
worse,
not
scheduling,
something
that
we
should
be,
then
we
need
to
fix
the
thing
again,
which
is
gonna
mean
we
have
to
like.
Do
this
whole
thing
again,
potentially,
if
we're
scheduling
something
we
shouldn't
be:
that's
not
as
big
a
deal
because
someone's
getting
something
for
free
that
they
shouldn't,
which
we
can
you
know
fix
later
on.
But
if
we're
not
scheduling
a
mirror
that
somebody's
paying
for
them,
yeah.
C
Basically,
because,
as
far
as
we're
aware,
like
you
know,
various
well
but
I
have
all
looked
at
this.
My
result
of
this,
like
we
are
pretty
sure
that
this
is
selecting
the
right
mirrors,
but
you
know
maybe
there
was
a
bug
in
the
old
codes
that
were
scheduling.
Some
mirrors
that
it
shouldn't
have
been
sure.