►
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
So
we're
going
to
chat
about
what
we're
going
to
do
next
about
latency
spike
we're
seeing
on
redis
related
to
the
the
cache
evictions.
When
we
reach
max
memory
yeah,
I
don't
think
we
need
to
repeat
all
of
the
contacts
that
we
discussed
yesterday,
but
now
we're
going
to
like
we've
done
several
investigations
on
the
redis
side
and
on
the
application
side
and
we're
going
to
see
where
we're
going
to
take
it
from
here,
yeah
matt.
You
were
saying
before
I
interrupted
you
yeah.
B
B
Reducing
redis
client
throughput
enough
that
we
have
a
backlog,
accumulate
quickly,
which
competes
for
memory
which
drives
the
eviction
cycle,
and
this
only
abates
approximately
when
enough
clients
get
stalled,
that
the
incoming
request
rate
falls
low
enough
that
that
evictions
can
keep
up.
So
that's
how
the
cycle
ends.
B
B
B
So
let
me
come
up
with
a
different
word
for
it,
the
demand
for
how
much
memory
we
need
to
store
our
keys
needs
to
be
less
than
max
memory
and
the
main
control
we
have
over.
That
is
ttl
the
secondary
control
we
have
over.
That
is
what
redis
instance
do
we
store
these
keys
in
which
leads
us
directly
to
the
the
repartitioning
discussion.
A
Yeah,
I
think,
for
reducing
details.
I
think
we
need
to
because
I
was
looking
at
it
briefly.
There's
some
keys
there
that
really
stand
out
that
shouldn't
like
there
was
one
key
that
was
really
low
hanging
fruit
that
took
almost
two
gigs
of
memory
by
saving
something
for
half
an
hour
when
we
only
need
it
for
a
few
minutes
really.
So,
like.
Okay,
that's
like
low
hanging
fruit,
yeah.
I
think
we
should
find
them
and
fix
them.
Yes,.
B
A
C
B
A
B
C
A
A
A
B
D
B
B
B
A
A
A
Yeah,
it
tries
to
match
keys
according
to
the
prefix,
and
then
it
like
will
try
to
match
them
to
something
that
it
already
knows.
So,
that's
why
these
two
are
like
separate
and
are
very
big,
because
they
basically
contain
everything
that
we
see
below
here,
and
this
one
is
also
not
very
interesting,
because
that
word
is
the
prefix
prefix
of
project
which
we'll
see
later
several
times
so
yeah
this
one.
A
A
B
A
A
A
B
Okay,
yes,
yes,
that
seemed
like
it
was.
That
seemed
like
it
was
the
the
simplest
thing
to
do
and
then
for
if,
if
it's
convenient,
we
can
also,
we
can
also
perhaps
classify
them
by
whether
or
not
they're
it
would
be
feasible
to
to
move
them
to
a
separate
instance,
and
I
think,
as
you
were
describing
a
moment
ago,
going
through
the
the
rails
default
cash
caching
mechanism
makes
that
harder.
So.
A
B
B
D
B
These
for
for
the
cross,
slot
command
keys,
then
moving
that
10
of
the
key
storage
space
out
of
the
the
existing
redis
cluster,
reduce
it
by
enough
to
avoid
the
saturating
max
memory,
and
if
it
doesn't,
then
we
still
have
more
work
to
do
so.
I
guess
I'm
kind
of
I'm
kind
of
thinking
that
that
a
relatively
small
percentage
of
the
of
the
of
the,
as
far
as
I
know,
I
thought
that
we,
I
thought
that
we
have
been
avoiding
the
the
the
cross
slot
design
pattern.
B
A
B
A
B
C
C
C
B
B
And
and
the
redis
x
so
from
from
from
a
design
perspective,
redis
expects
to
have
numerous
keys
and
it
doesn't.
I
guess
what
I'm
working
up
to
is
when
it
gets
to
the
point
of
having
to
do
evictions
redis.
Doesn't
it
doesn't
even
attempt
to
say
I'm
going
to
evict
the
key
with
the
the
the
shortest
remaining
time
to
live?
D
B
C
A
But
looking
at
something
here
that
I
noticed
there's
all
of
these
view
things
that's
quite
a
large
chunk.
That's
also
a
thing
that
I
don't
think
is
currently
covered
by
our
closso
cross
slot
validator,
because,
like
I'm
not
sure,
I
need
to
look
it
up,
but
that's
called
in
a
completely
different
way.
B
A
B
Yes,
so
I
think
that
analysis
is
okay,
so
in
in
terms
of
I
want
to
take
it
back
up
back
up
a
level.
So
in
terms
of
next
steps,
the
ttl
analysis
still
feels
like
the
the
shortest
path
and
the
next
step.
The
immediate
next
step
on
that
was
bob.
You
were
talking
about
re-running
the
the
analysis
to
get
like
a
a
a
finer
grained
distribution
for
at
least
for
for
at
least
the
less
than
one
day
subset.
B
That
will
be
super
fantastic,
and
I
think
that's
that's
probably
going
to
let
us
that
will
probably
lead
to
some
concrete
recommendations
about
which
keys,
I
guess
what
I'm
thinking
of
is.
If
we
imagine
that
we've
got
that
data
set
right
now,
then
then
we
would
be
looking
at
the.
We
would
essentially
be
evaluating
the
semantics
of
what
what
harm
does
it
do
to
reduce
the
ttl
for
each
of
the
the
high
frequency
frequently
occurring
key
keys
and
since.
B
A
B
A
B
The
whole
point
of
doing
the
caching
is
to
avoid
extra
work
and
we
don't
have
a
good
way
to
quantify
how
much
work
we're
avoiding
with
these
cache
entries.
So,
for
example,
one
particular
kind
of
cache
key
is
avoiding
spending.
I'm
totally
making
this
up
say
one
one.
Cash
key
is
avoiding
spending
say
you
know
a
hundred
milliseconds
per
per
key
on
the
primary
db
and
another
type
of
cache
key
is
avoiding
the
same
thing
on
replica
dbs.
B
A
Yes,
also
duration,
right
like
yes,
yes,
but
what
I
what
I,
what
I
was
more
alluding
to
is
like
if
we
have
let's
here
here,
is
idle
time
in
days.
But
imagine
these
are
ours
yeah.
When
you
look
at
this,
then
then
I
I
would
be
inclined
to
say,
like
we
can
set
the
the
default
ttl
to
one
day
and
we
wouldn't
really
feel
anything.
B
A
But
I'm
imagining
that
as
like,
if
we,
if
we
look
at
this
bucket
closer
like
in
more
detail,
then
we're
going
to
have
a
wider
distribution
at
some
point,
which
means
we
can
say
like.
If
we
put
it
there
we're
going
to
lose
30
percent
of
memory
like
right
up
30
of
the
memory,
but
we
still
have
the
70
like
we
we're
actually
yeah.
C
C
A
B
A
B
C
B
It's
it's
not
a
dumb
idea.
The
the
challenge
with
it
is
there
are
a
few
challenges
with
it.
The
the
main
the
main
obstacle
is
we're
because
we're
because
we're
beyond
the
saturation
point,
we
don't
know
how
much
we
would
need
to
grow
it
to
avoid
that
saturation.
All
we
know
is
that
the
demand
exceeds
the
capacity,
which
is
why
we're
having
these
inverse.
B
It
could
be
that
if
we
add
five
percent
more
more
more,
if
we
raise
the
max
memory
limit
by
five
percent,
then
we
would
top
out
at
an
additional
three
percent,
I'm
making
these
numbers
up.
I
don't
actually
believe
this
is
the
case.
If
we
added
50
to
max
memory,
I
would
kind
of
call
it
a
coin
toss
about
whether
that's
enough
to
you
know
level.
A
D
C
B
B
And
we're
definitely
dropping
I
have
to
have
to
attack
this.
I
just
have
to
take
this
on
because
it's
so
germane
to
what
you
were
just
asking.
The
the
inverse
of
that
is
how
much
do
we
need
to
reduce
ttl
to
get
underneath
the
max
memory
and
it's
exactly
the
same
class,
a
problem.
It's
just
easier
to
reduce
the
ttl
than
to
grow
the
machines,
because
that
wipes
the
cache
so
that
that's
the
the
logistics
is
the.
B
C
A
C
The
kind
of
side
effect
of
what
I'm
saying,
which
is
I've,
looked
at
some
of
the
things
it's
clear
that
the
app
decks
for
redis
cash
is
not
good,
but
how
big
is
the
impact?
Is
this
the
thing
that
if
we
work
on
this
for
another
month,
everyone
will
be
okay
or
is
it
the
thing
that
if
we
don't
have
it
fixed
by
next
friday,
you
know
rachel's
gonna
cry,
because
that's
the
other
question.
B
B
Behavior
to
preserve
that
point
in
time,
snapshot
of
all
of
the
entire
memory
footprint
for
the
further
by
this
process,
and
so
the
longer
it
takes
to
make
that
rdb,
the
the
more
pages
will
diverge
over
time
and
so
for
planning
for
capacity
planning
purposes.
We
have
to
have
enough:
yes,
not
necessarily
a
full
double,
but
yeah.
It's
it's!
You
know
the
theoretical
upper
bound
is
twice
the
memory
demand.
So
it's
it's.
It's
a
little.
B
C
B
B
C
B
B
B
This
is
more
important
now
because,
because,
essentially
because
we've
we've
been
working,
bob
in
particular
has
been
working
with
the
stage
groups,
the
feature
teams
to
to
kind
of
own
their
error
budget
and
work
with
them
to
improve
visibility
and
and
part
of
that
kind
of
led
to
uncovering
the
impact
of
this.
On
the
on
the
feature
categories
that
those
stage
groups
own
and.
A
So
the
actual
client
side
impact
is
every
10
or
so
minutes
a
whole
bunch
of
requests
take
way
longer
than
they
should,
and
this
is
especially
visible
for
groups
like
the
people
in
the
create
stage
who
host
like
to
get
the
get
endpoints,
so
the
info
refs,
and
so
on,
like
endpoints,
that
are
hit
by
git
clients,
the
endpoints
that
are
hit
by
ci
and
so
on,
like
the
super
busy
ones.
Those
have
those
noticed
this
the
most.
C
B
Or
yeah
yeah,
so
it
varies
with
time
of
day,
and
so
it
varies
with
workload
which
varies
with
time
of
day.
My
kind
of
anecdotal
observation
is
it
doesn't
matter
you
can
you
can
look
at
the
there
are
graphs
to
show
the
pacing
for
it,
but
it
will
generally
vary
somewhere
averaging
about
10
minutes.
I
think
I
think
I
think
we've
seen
it
as
high
sorry.
My
my
reflection
here
doesn't
matter
the
data.
The
graphs
will
show
what
the
what
the
actual
pacing
is.
B
So
it
it's
a
little
more
so,
as
I
recall
it
was
more
severe
in
in
late
last
year
and
then
we
had
a
low,
and
that
was
probably
because
of
the
the
workload
being
reduced
over
the
over
the
end
of
year
holidays
and
then,
in
january
it
picked
back
up.
There
are
a
few
factors
that
can
drive,
I
mean
essentially
from
redis's
perspective.
B
What
drives
this
is
how
how
quickly
the
the
memory
rises
back
to
max
memory
after
evictions
clear
out
some
some
headroom,
which
is
the
the
undesirable
eviction
burst,
behavior,
that
we're
trying
to
ameliorate
factors
that
influence
that
growth
rates
the
effectively
the
slope
of
the
rise
from
you
know,
back
up
to
max
memory
are
feature
changes
where
we,
where
we
store
more
things
in
cache
user
behavior
that
that
that
happens.
Hidden
points
that
you
know
that
draw
I
mean
changing.
B
Changes
to
to
the
the
cache
keys
can
also
drive
this,
so
there
there
definitely
can
be
influence
from
application,
behavior
and
from
end
user
behavior.
We
don't
have
a
clear
handle
on
on
that,
but
anecdotally.
I've
definitely
seen
the
the
the
the
pacing
of
these
of
how
often
we
reach
max
memory
vary
on
a
time
scale
of
months
going
both
up
and
and
down,
and
I
think
that
was
mainly
workload.
C
B
A
A
D
B
B
A
A
B
Yeah,
no
totally
I
I
was
thinking
I
I
wanted
to
spend
a
little
more
time,
looking
at
the
the
data
that
you've
already
put
together
and
and
maybe
mock
what
what
what
I,
I
guess,
I
I
want
to
sketch
what
what
ideally
I
would
like
to
I'd
like
to
extract
from
from
that.
You
know
in
terms
of
in
terms
of
the
the
subdividing
existing
categories.
So
that's
that's!
That's
it.
A
So
we
look
for
high
adult
time
keys.
Then
we
see
how
to
reduce
the
ttl
on
those
we're
also
going
to
have
an
overall
like
do
the
investigation
of
that
first
day.
If
there's
yeah
an
overall
thing,
we
could
we
spot
like
if
we
that
we
can
reduce
the
overall
ttl
by
x
days
and
see
how
much
we'd
gain
doing
that,
because
that's
a
simple
solution.
A
B
A
A
For
the
very
future
we
need
to,
we
need
to
figure
out,
but
maybe
not
somebody
something
for
our
counterparts
in
the
frameworks
team
to
do.
We
need
to
get
some
consistent,
naming
and
cache
keys,
because
if,
if
all
the
significant
identifying
bits
were
in
the
beginning
of
the
cache
key
like
to
see
where,
where
the
cache
key
is
coming
from,
and
then
the
identifiers
like
project
id
and
user
id
were
at
the
end,
then
this
analysis
would
be
so
much
easier.
D
B
Okay
and
then
in
addition
to
that,
once
we've
selected,
some
keys
that
have
a
high
idle
time
and
collectively
represent
a
large
amount
of
storage.
We
should
spot
check
that
to
to
the
kind
of
assess
to
the
best
of
our
knowledge,
if
any,
before
reducing
their
ttl,
if
any
of
them
are
known
to
to
the
three
of
us
at
least
two
to
be
particularly
expensive
and
maybe
skip
them,
I
don't
have
a
good
idea
about
how
we
would
empirically
test
for
this.
A
Rendered
commit
stuff
that
we
need
to
render
differently
for
certain
users,
but
most
of
it's
the
same.
It's
just
we
copy
everything
and
then
do
it
for
for
each
user,
so
yeah
there
might
be
things
that
we
need
to
optimize
there.
But
I
wonder
if
doing
optimizations
like
that
is
going
to
take
a
bit
too
long.
A
B
No
yeah,
I
think,
if
we
had
a
if
we
had
a
if
we
had
a
point
in
time
snapshot
like
an
rdp
backup,
then
we
could,
I
mean,
most
most
of
the
time
we're
looking
at
the
the
the
names
of
the
keys,
but
we
could
also
look
at
the
do.
An
assessment
of
looking
for
duplicate
values
across
the
whole,
the
whole
database
and
then
identify
what
what
names
those
keys
had.