►
From YouTube: 2020-09-15: Gitaly cgroups catchup
B
Yeah
so
to
to
make
sure
I
understand
the
approach.
So,
basically,
what
the
plan
is
to
do
is
instead
of
rolling
it
on
Canary
and
then
on
a
short
basis.
We're
gonna,
configure
the
c
groups
so
quickly
just
create
sold
to
see
groups
and
then
we're
gonna
use
the
feature
flag
as
a
percentage
rollout.
For
example,
one
person,
ten
percent
throughout
the
whole
Fleet
right,
but.
A
A
In
yes-
and
let
me
address
that
right
now,
I
am
totally
planning
on
on
having
so
I
I.
Don't
need
to
have
it
in
production
to
do
the
initial
calibration
and
I'm
expecting
Canary
to
have
different
thresholds
to
have
different
sizing
sizing
limits
because
of
its
and
I'm
planning
on
doing
the
same
analysis
for
for
all
of
them
for
for
all
of
those
categories
that
we
talked
about
before
so.
A
A
B
B
A
A
A
Yes
same
here,
okay
same
here
and
it
it
does,
it
does
at
least
as
I'm
thinking
of
it.
It
does
still
give
us
a
a
rapid
way
to
turn
it
off
and
I
was
I.
Think
I
was
chatting
with
Rachel
a
few
days
ago.
If
memory
serves
about
the
the
consequences
of
turning
it
back
up.
So
if
we're
in
a
pinch
and
I
wanted
to
talk
through
this
with
you
as
well,
just
to
so
Cindy,
you
took
me
on
this.
A
So
if
we've,
if
we've
been
able
to
so
pretend
that
we've
enabled
c
groups,
it's
been
enabled
for
I'm
just
going
to
make
some
stuff
up,
it's
been
enabled
for
an
hour
and
we
start
to
get
some
out
of
memory.
Errors
on
one
machine
say:
file
75.
yeah,
rather
than
try
to
you,
know,
isolate
the
problem
on
that
machine.
I
think
we
just
cut
the
feature
flag
off
globally
and
I
think.
A
B
B
A
A
B
A
A
A
So,
if
there's,
if
there's
so
just
to
kind
of
play
out
an
example
if
we,
if,
after
we've
turned
back
off
the
feature
flag,
say,
there's
a
really
greedy
process
that
wants
to
allocate
you
know
20
Gigabytes
of
anonymous
memory,
and
now
it's
running
it
in
the
the
old
system,
D
managed
C
group
along
with
Italy
and
and
that
at
that
point
Colonel
is
going
to
say
well,
I
don't
have
20.
Gigs
of
you
know.
Free
memory
lying
around
I've
got
to
open
Cash.
B
A
Still
be
able
to
free
up
memory
by
kicking
up
pages
from
that
were
charged
to
the
c
groups,
just
just
in
the
same
way
that
today
it
does
for
for
the
the
old
90
gigabyte
secret,
so
that
behavior
doesn't
change
it's.
It
remains
that
yeah
that
that
remains
perfectly
viable,
and
that
was
the
only
other
gotcha
I
want
I
needed
to
kind
of
you
know
talk
through
to
kind
of
reason
my
way
into
yes.
This
would
be
okay,
too,.
B
Got
it
yep,
I,
think
that
makes
sense
and
I
agree
with
that
I
I,
I
kind
of
feel
like.
Oh,
we
should
have
gone
with
our
original
approach,
just
enabling
Ketone
Canary
and
go
hundred
percent
future
flag
wise,
but
I
do
feel
like
having
a
ripcord
with
feature.
Flags
is
a
lot
safer
in
that
sense,
so
I
agree
with
the
throwout.
A
A
A
I
wanted
to
talk
about
two
two
things:
one
is
expectations
and
for
for
what
the
what
the,
what
what
the
results
are
are
likely
to
look
like
and
two
was
I,
was
a
little
bit
tempted
to
split
it
into
separate
change
requests.
But
there's
there's
so
much
paperwork.
I
figured
just
pile
it
on
to
one
change:
request.
A
A
Yeah
got
it
yeah,
it's
a
it's
a
it's!
It's
really
like
Anonymous
pages
are
like
when
process
you
know
Maliks
a
page,
for
example,
and
it's
it's
kind
of
historical
nomenclature.
So
there's
far
back
to
Pages,
which
are,
as
the
name
suggests,
backed
by
it,
you
know
it's
it's
a
block
from
a
file,
that's
loaded
into
the
into
the
page
cache
which
we
now
call
file
stream
cache
and
then
on
Anonymous
pages
is
everything
else
everything
that's
not
filed,
but
yeah.
A
If,
if
your
program
does
a
read,
Cisco
that'll
load,
the
that'll,
you
know
read
the
file
in
you
know
into
the
file
system
cache
and
then,
if
it
wasn't
already
there
from
disk
and
then
the
the
the
program
is
just
accessing
pages
in
The,
Fast
and
cache
very
similarly,
if,
if
a
file,
if
sorry,
if
a
program
and
Maps
a
file,
then
that's
effectively
saying
within
the
context
of
the
process,
this
virtual
address
space
whenever
I
access
this.
You
know
this
virtual
address.
A
It's
going
to
correspond
to
this
page
of
the
file,
and
this
virtual.
This
virtual
address
corresponds
to
that
page
in
the
file
and
kernel
is
free
to
kick
out
just
like
just
like
normal
file
back
pages,
the
the
kernel
is
free
to
kick
those
out
of
the
file
system
cache
whenever
it
wants.
It's
just
that
whenever
the
process
goes
to
access
that
virtual
memory,
there'll
be
a
address
that
that
corresponds
to
a
page
in
the
file.
Colonel
will
look
at
it.
A
That'll
trigger
a
page
fault
in
in
kind
of
the
normal
control
flow
kernel
will
load
that
page
back
in
from
disk
if
it
wasn't
already
there
and
then
let
the
application
resume
just
as
though
nothing
can
happen,
so
it
works.
It's.
These
are
just
kind
of
different
facets
of
how
the
how
the
kernel
will
will
it
effectively
manage
the
file,
some
cash
yeah,
so
with
risk
with
regard
to
c
groups.
A
A
So
this
is
this
is
actually
why
that
host
is
in
kind
of
an
unusual.
Those
hosts
are
tend
to
be
in
the
unusual
state
of
having
a
relatively
large
number
of
gigabytes
of
free
memory,
because
almost
everything
that
we're
doing
that
touches
that
large
file
system
is
running
inside
that
c
group,
that's
got
a
90
gigabyte
limit
and.
A
Cache
you
know
any
any
Pages,
you
know
any
any
files
that
they
access
get
charged
to
that
c
group,
because
nobody
else
is
actually
accessing
those
pages
and
we
can
load
that
up.
For
example,
if
we
run
I
don't
know
like
if,
if
we
cat
one
of
these,
you
know
large
files
outside
of
that
c
group,
then
that'll
go
in.
You
know
into
the
root
c
groups.
Anybody
I
think
you
get
what
I'm
saying
yeah
yeah.
A
A
That's
gonna
happen
unnaturally
slowly
for
probably
the
next
few
weeks,
because
most
of
the
on
many
of
these
giddly
hosts
most
of
the
faustin
cache
pages,
are
already
charged
to
a
different
C
group,
the
original
C
group,
and
so
our
our
git
commands
that
are
now
going
to
be
running
inside
their
own
per
repo
c
groups.
Would
they
have.
A
A
The
order
of
I
don't
know
like
hours
to
days,
because
we
do
deploys
very
often
anytime,
so
anytime,
we,
we
do
a
goodly
restart
and
the
old
giddly
processes.
Pro
repo
C
grips,
get
destroyed
there,
whatever,
whatever
Frost
and
cash
pages,
had
been
charged
to
those
c
groups.
When
that,
when
those
old
peripro
seed
groups
get
deleted,
their.
B
A
B
A
A
A
B
A
Able
to
use
you
know
as
much
of
the
memory
as
it
wants
to
as
as
as
fast
and
cash
I
kind
of
like
that.
As
a
side
effect,
I
mean
that
wasn't
in
anyone's
mind,
as
we
were
kind
of
planning
the
the
hierarchy
actually
I
kind
of
thought.
It
was
a
little
bit
silly
to
have
a
slash,
giddly
layer
of
the
hierarchy
that
we
didn't.
A
A
B
B
A
Main
thing
I
wanted
to
kind
of
talk
through
was
what
to
expect
in
terms
of
memory
usage
that
gets
charged
at
the
Z
groups,
because
they're
they're
gonna
get
most
of
the
frosting
cash
has
already
been
wormed
by
and
those
those
pages
were
remained
available,
but
they're
charged
to
different
c
groups.
So
what
we'll
see
in
terms
of
the
file
Sim
cash
being
charged
to
the
new
c
groups
is
really
just
the
stuff.
That's
not
already
being
warmed
that
that'll
take
some
time
to
accrue
I'm
guessing
yeah
yeah.
A
B
A
B
That
makes
sense
I
wasn't
aware
of
that.
So
thank
you
for
the
call
out
okay.
So
what
do
you
think,
for
example,
on
day
one,
let's
say
tomorrow
or
Monday
morning
yeah
my
time,
for
example,
we
roll
out
one
percent
on
ten
percent
yeah
with,
like
maybe
an
hour
to
make
sure
everything
works,
fine
and
then
on
Tuesday.
B
B
A
B
A
A
B
A
So
this
is
the
this
is
so
there
were
two
things
I
wanted
to
do
for
for
kind
of
calibrating,
so
the
the
CPU
usage
I
I,
don't
I,
don't
care
about
that.
We're
using
CPU
shares.
It's.
You
know
that
that's
I
figured
we'll
set
up
we'll
set
a
generous,
a
generous
number
of
shares
for
each
C
group.
It
implicitly
supports
burst
Behavior
anyway,
even
if
we
said.
A
Question
right,
so
exactly
exactly
so,
that's
why
I'm
focusing
just
on
the
memory
yeah
perfect,
so
for
the
memory.
So
this
is
this
is
last
seven
days,
I'm
gonna
screenshot
this,
so
you
don't
have
to
like
memorize
it
or
anything.
But
I
just
want
to
talk
very
briefly
so
I'm,
looking
back
at
last
seven
days
for
all
all
yearly
nodes
in
production
that
that
have
that
that
spawned
a
command
through
the
mechanism
that
we're
going
to
be
using
secrets
on
so
each
of
those
commands
gets.
A
Are
you
statistics
captured
and
I
say
each
almost
all
of
them
get?
Are
you
statistics
I'm,
assuming
that
this
is
a
reasonably
large?
It's
not
a
complete
census,
but
it's
close.
It's
close
enough
for
our
purposes.
All
I
really
want
to
do
is
make
sure
that
we
have
that
there's
no
single
command!
That's
going
to
blow
the
budget
in
one
command!
That's
going
to
need
more
memory
than
than
we're
going
to
give
to
the
pro
repo
c
groups.
Then
we
have
to
reevaluate.
B
A
Happily,
this
is
giving
us
the
the
number
of
commands
witnessed
and
the
the
50th
95th
99th
percentile
and
the
actual
Max
is
just
some
bytes
or
yes,
it's
in
bytes,
okay,
so
so
this
is
and
I
got
it
broken
down
by
Shard.
But
we
can
already
see
that,
like
the
that
this
is
I.
I
will
double
check
that
it's
in
bytes,
because
RSS
is
often
reported
in
kilobytes.
But
this
has
to
be
bytes,
because
we
don't
it's
got
to
be
bytes,
but
I'll
double
check.
Did
it
release
bites?
B
A
So
this
is,
if
this
is
in
bytes,
then
it's
70,
79
megabytes
and
that
seems
impossibly
small
yeah,
because
this
isn't
just
Anonymous
memory.
This
is
this.
Is
this
includes
mapped
memory?
So
maybe
this
maybe
this
is
using
kilobytes.
It
was
on
my
list
to
check
anyway,
but
now
I'm
really
really
interested
to
check
it.
Okay,
so
I
guess
I
guess
what
I'm
really
kind
of
getting
at
is.
A
This
is
this
is
RSS,
as
reported
by
the
by
the
process,
not
not
by
the
C
group,
so
this
includes
mapped
files
which
you
know
if,
if,
if,
if
this,
if
this
process
that
used
79
gigabytes
of
Resident
memory,
what
if,
if
it
is
79
gigabytes
it
that
process
almost
certainly
wasn't
I
wasn't
using
that
much
Anonymous
memory.
Some
of
that
would
have
been
file
backed
and
so
at
that
point,
I'd
want
to
I'd
want
to
go.
Look
at.
A
You
know
that
example
command
to
see
what
what
what
was
doing,
what
was
going
on
there.
So
this
is
the
main
thing
I
wanted
to
get
is
what
what
are?
What
are
the
outliers
and
it
looks
like
it
looks
like
we
don't
have
any
in
the
last
seven
days,
any
any
worrying
outlies,
except
for
possibly
this.
You
know
above
the
99th
percentile
in
the
in
default
Shard.
So.
A
A
You
know
over
if,
if
assuming
the
units
or
Kill
the
Lights
here,
anything
that
was
over
20,
you
know
10
or
20
just
to
take
a
look
at
those
those
those
commands
and
treat
them
as
hotliers
and
see.
If
see,
if
there's
any
kind
of
pattern
that
we
need
to
be
aware
of
here.
But
that's
that's
what
I'm
planning
on
to
do
for
the
outlier
analysis,
and
this
is
what
I
wanted
to
kind
of
Reserve.
You
know,
but
maybe
half
an
hour
of
focus
time
for.
A
For
sure,
so
we
so
I
looked
at
the
Epic
and
just
looked
like
the
issue.
I
knew
we
had
an
issue
for
it,
so
I
went
and
looked
for
it
and
I
figured.
That
was
a
good
place
to
quit.
Put
scratch
notes
and
I'll
I'll
add
a
summary
note
to
the
to
the
the
main
epic,
perfect,
yep
yeah,
so
that's
kind
of
okay.
Let's
start
off
the
presses
I
want
to
just
here.
B
B
A
A
B
A
B
B
B
A
B
B
A
A
A
A
B
A
B
A
A
A
A
A
Yes,
exactly
and
I'm
I'm-
probably
going
to
call
it
like.
You
know:
Anonymous
memory,
usage
and
parenthetically
coal
c
groups
RSS,
because
it
will
people
will
misinterpret
it
because,
like
everyone,
has
this
preconceived
notion
of
what
RSS
means
based
on
like
you
know
the
last
30
years
and
having
c
groups
use
the
same
word
for
something
completely
different
is
just
oh
anyway,.
B
A
B
A
A
B
Yes,
thank
you
you're
comfortable
with
this.
So
to
reiterate,
you'll
open
up
the
shift,
merge,
request,
I'll,
look
at
the
canary
scraping
issues,
you'll
also
push
out
the
dashboard
updates
and
I
will
fix
the
relabeling
issue
that
we
have
like
roll
down
tomorrow
and
then
on
Monday
morning.
My
time
I
can
start
with
the
one
percent
and
10
percent
rollout
and
leave
it
for
a
day
and
then
move
on
to
50
and
then
yeah
leave
us
for
two
days
and
move
to
100
and
hopefully
nothing
blows
up.