►
From YouTube: 2022-12-01 Scalability Team demo
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
A
I
wanted
to
talk
a
little
bit
about
some
work
that
we've
been
doing
that
led
to
this
graph.
This
graph
is
the
dominant
forecast
for
reddish
psychic,
read
the
primary
CPU
and
we
have
a
very
nice
forecast,
which
is
actually
a
different
issue.
I
still
I
keep
wondering
when
timeline
will
catch
up
to
the
fact
that
it's
stable
at
40,
but
the
answer
still
doesn't.
A
But
what
I
want
to
talk
about
is
how
we
got
here
and
what
happens
and
this
kind
of
up
and
bottom
up.
There
were
a
couple
optimizations
we
found
and
we
just
started
doing
them
and
then
at
some
point,
I
realized
I
should
make
this
an
epic,
so
there's
actually
an
epic
for
it.
But
it's
just
to
put
those
two
issues
together
and
I.
Think
there
are
some
interesting
lessons
or
I.
I
learned
some
interesting
lessons
from
this
and
I
want
to
share
those.
A
So
what
the
two
issues
in
the
Epic
are
do
something
about
psychic
Chrome
and
do
something
about
duplicate
jobs,
because
they
both
use
redis
a
lot,
and
this
stood
out
because
of
looking
at
the
redis
commands.
Duration
counters,
which
show
time
spent
in
the
main
main
thread
of
red,
is
in
the
in
the
handlers
of
in
the
request
handlers
so
for
ready,
psychic
that
was
showing
that
the
top
commands
were
exec.
A
That's
the
this
one
and
then
the
green
one
is
H
get
all
and
when
I
saw
that
I
thought
that's
funny,
because
neither
of
them
is
about
pushing
jobs
and
Reddit
psychic
is
for
pushing
jobs
to
psychic
processes,
so
that
look
funny
and
I
think
egord
ended
a
traffic
capture
to
figure
out
what
the
age
schedule
was
and
that
turned
out
to
be
the
fault
of
psychic
krone
and
then
Alejandro
worked
on
changing
How
psychicron
does
its
pulling
so
psychicron
does
corn
drops
and
every
psychic
process
periodically
checks
the
list
of
all
cron
drops
to
see
if
they
need
to
run
and
that
code
was
written
in
a
silly
way
where
we
would
pull
way
too
often
and
that's
so.
A
The
actual
rate
of
these
H
get
all
commands
was
very
low,
but
they
were
quite
expensive
and
they
happen
too
often.
So
when
you
see
this
drop,
that
is
where
the
h
capital
commands
went
away
and
then
the
other
thing.
The
exact
calls
were
caused
by
the
way
the
duplicate
jobs.
Middleware
is
using
redis
and
we
had
a
couple.
False
starts
with
feature
Flags
or
making
changes
after
we
iterated
a
couple
times
and
then
from
this
point
on
that
workload
is
gone.
A
Some
of
you
have
known
this
was
going
on
at
the
time.
So
it's
not
you.
What
I
found
interesting
here
is
that
of
the
two.
The
duplicate
job
change
seems
to
have
had
a
bigger
impact,
and
if
we
look
at
the
CPU
graph,
so
October
25th,
roughly
that's
about
here-
is
when
the
H
cattle
change
happens.
A
So
here
we
were
peaking
at
60,
and
here
we
started
peaking
at
55
or
something
and
then
duplicate
jobs
went
away,
and
then
we
went
to
about
40..
So
this
was
roughly
a
five
percent
drop,
and
this
was
roughly
a
15
percent
age
points
drop,
so
the
drop
was
bigger.
Even
though
looking
at
the
graphs,
they
were
not.
A
You
wouldn't
expect
one
to
be
a
three
times
bigger.
A
I
actually
think
the
answer
is
different,
because
here
I
have
a
graph
of
the
request
rates,
and
here
you
can
see
October
25th,
there's
a
drop
in
the
request
rate
because
of
the
H
get
all,
but
the
drop
because
of
duplicate
jobs
is
much
bigger.
A
Because
here
we
see
that
the
total
redis
request
rate
is
speaking
at
around
20,
and
here
it
was
around
40..
A
Exactly
because
every
job
does
this,
that's
that's
why
there
were
so
many
requests
for
this.
So
even
though
these
graphs
were
about
the
same
Heights,
this
is
the
duration.
Again,
the
green
graph
was
caused
by
a
relatively
small
number
of
requests
that
took
a
lot
of
time,
computation
time
on
the
main
threads,
but
because
the
request
number
of
requests
was
slow
low.
The
per
request
overhead
was
also
low,
and
these
were
not
necessarily
that
computationally
expensive,
but
there
were
just
a
whole
lot
of
them
and
they
were
individual
requests.
C
A
A
C
One
somewhere,
I
mean
I,
I,
think
one
one
other
thing
that
we
could
look
at
is
the
CPU
utilization
on
the
main
thread:
Broken
Out
by
mode,
so
that
we
can
see
user
and
system
time
and
see
if
we
see
that
that
corresponding
drop
consistent
time.
So
this.
A
A
Eager
knows
this
very
well,
and
what
is
weird
about
this
is
that
most
of
this
is
IO,
so
this
is
right
IO.
This
is
read
IO
and
the
actual
computational
workloads
of
what
you
think
redis
is
doing
or
what
redis
is
doing
in
user
space.
A
Is
this
small
part
and
the
graph
I
have
here
which
shows
the
durations
is
only
measuring
in
here
in
this
block,
but
it's
blind
to
what's
happening
here
and
what's
Happening
Here
and
what
is
happening
here
and
here
is
proportional
to
the
number
of
requests
we
do
so
another
way
of
explaining
this
is
to
say
that,
because
the
duplicate
jobs,
workload
also
did
a
lot
of
requests.
It
accounted
for
a
large
part
of
this
and
this
so
it
shrunk
this
and
this
Tower.
A
Thank
you
and
for
me
it
was
an
interesting
like
when
I
say
looking
back,
it
seems
obvious,
but
it
was
an
interesting
reminder
of
the
yeah
of
of
the
different
effects
of
and
also
of
how
to
choose
what
to
work
on.
A
B
Do
you
know
about
when
we
introduced
the
the
cookie?
No,
not
the
cookie
starter,
separate
the
wall
locations
where
we
Store
Wall
locations
next
to
you,
the
duplicate
jobs,
and
do
you
know
how
come
we
didn't
notice
that
at
the
time.
A
Okay,
I
need
to
fill
in
the
rest
of
what
we're
talking
about
here.
Duplicate
jobs
sets.
The
middleware
was
originally
written
to
set
just
one
or
two
keys
and
then
at
some
point
it.
The
another
team
worked
on
database
load
balancing
for
sidekick,
so
every
job
carries
wall
locations
in
its
metadata.
A
So
the
idea
is
to
send
as
many
postgres
queries
to
secondaries
as
possible,
and
for
that
you
need
to
know
in
a
in
the
context
of
a
job
or
a
request
like
what
how
up-to-date
your
postgres
has
to
be.
So
this
was
added
later
to
the
duplicate
jobs
middleware,
and
this
was
where
most
of
the
extra
requests
came
from
I,
don't
remember
when
the
code
was
added,
but
another
funny
thing
here
is
that
the
load
balancing
was
actually
not
being
used
because
there
was
a
bug
in
how
the
data
got
consumed.
A
So
we
still
hit
the
overheads
of
the
load
balancing
codes
in
the
duplicate
jobs
middleware,
but
we
were
not
using
it
and
we
found
the
bug
that
caused
low
balancing
to
not
work.
Then
because
Bob
remembered
that-
and
he
pointed
me
at
it-
and
I
happened
to
see
it
because
I
had
to
I-
ran
into
the
same
bug
more
or
less.
When
working
on
on
this
project.
C
A
A
I,
don't
know
when
that
code
got
added
and
I
have
one
graph
up
with
a
one
year.
View
or
another
thing
you
can
see
here
is
that
we
for.
A
Duplicate
jobs,
the
shared
State-
and
that
was
a
bad
bad
idea,
and
the
funny
thing
is
actually
that
here
we
were
not
doing
duplicate
jobs,
and
here
we
are
doing
duplicate
jobs,
so
the
optimizing
them
was
worth
it,
but
where,
in
this
period
the
low
balancing
got
added
I,
don't
know,
I,
don't
see
an
obvious
jump,
it
might
have
been
longer
ago
am
I
yeah
or
it
might
have
been
longer
ago.
Yeah.
B
The
reason
I
ask
is
that
ideally
we'd
respond
to
things
like
that
quicker,
so
we
have
like
so
normally.
We
would
have
seen
that
because
capacity
planning
work
pointed
out
or
something
else.
A
B
B
A
Don't
know
how
long
have
we
had
capacity
planning
like
this.
A
Yeah
we
can
probably
I
could
look
into
commit
history
when
the
database
low,
but
let's
go
and
added
to
the
middleware
but
I.
B
A
Okay,
thanks
for
any
thanks,
everyone
I
was
looking
forward
to
sharing
this.
B
You
want
to
touch
on
your
last
question.
Should
we
remove
the
clubbing
that
we
did
so
we
already
tried
to
move
this
this
workload
once
we
moved
it
to
Shared
State,
and
that
was
a
bad
idea,
because
shared
state
was
more
set
more
I.
A
Can
illustrate
that
for
you
Bob
if
I
share
again,
if
we
go
to
Tim
lands,
this
is
where
we
moved
duplicate
jobs
to
Shared
States
and
we
made
Plumbing
for
that
to
be
able
to
move
the
workload.
And
if
you
see
this
is
the
second
half
of
July
and
then
we
go
to
reddit's
persistent
that
we're
here,
and
that
was
bad
and
I
wonder
if
this
is
because
we
did
duplicates
rights
because
it
starts
early.
A
A
C
A
B
Do
we
decide
that
we're
not
going
to
do
this,
because
it
was
my
understanding
and
even
from
what
you
just
said,
we
would
expect
sidekick
red
is
to
only
do
sidekick,
like
actual
like
the
things
that
come
from
the
sidekick
process.
Gem
yeah
on
the
sidekick
redis
instance.
A
A
Because
duplicate
jobs
would
have
to
work
correctly
across
sonoclusters,
so
they
couldn't
be.
A
B
A
B
A
B
C
Yeah
so
I've
been
working
on
reddit's
cluster
cookbook
this
last
week
and
we
have
a
first
working
setup
on
pre.
So
I
just
wanted
to
give
a
quick
demo
of
that.
C
Here
we
go
so
the
naming
scheme
that
we've
gone
with
and
we
can
change
this
if
needed
is
redis
cluster
rate
limiting.
So
this
is
the
type
this
is
the
name
of
the
service
so
to
speak,
and
then
shard01
and
that's
zero
one
through
zero.
C
Three
and
then
each
shard
has
three
nodes
inside
of
it,
so
there's
sort
of
three
zones
per
shard,
and
so
this
is
running
router
7,
a
custom,
build
of
red
S7
and
we've
implemented
a
gitlab
ready,
CLI
helper
script
that
finds
the
right
secret
so
that
you
don't
have
to
always
remember
this
stuff,
so
we
can
do
ping.
C
One
new
addition
is
that
this
actually
supports
multiple
users.
So
I
can
this.
This
was
added
in
redder
6.,
and
so
we
can
say
who
am
I.
So
this
is
the
console
user
and
I
think
there's
ACL
lists
which
I'm
not
sure.
If
this
just
leaked
our
passwords
Maybe?
C
Yes,
we
we
can
rotate
it
if
needed,
but
yeah
in
in
any
case,
yeah
I'll
rotate
it.
After
this
call.
C
So
yeah
we've
got
like
a
console
user,
redis
exporter
user,
a
replica
user,
that's
used
by
the
other
redis
nodes
to
talk
to
this
redis
node,
and
so
we
get
a
little
more
fine-grained
access
control,
but
what
I
actually
think
is
more
useful
than
that
is
potentially
ability
to
rotate
these
passwords,
which
currently
is
pretty
difficult
to
do
and
also
attribution
so
I,
don't
know
if
all
log
lines
include
this
information,
but
the
the
hope
is
that
if
something
weird
is
going
on,
we
may
be
able
to
tie
that
back
to
the
user.
C
So
then
we
can
run
some
some
redis
cluster
specific
commands.
So
here's
the
dash
cluster
info-
and
so
we
give
it
you
need
to
give
it
a
node,
a
redis
cluster,
node
and
so
I'm
just
going
to
give
the
same,
one
I
could
also
put
localhost
here
if
I
wanted
to,
and
the
display
here
is
a
little
weird.
So
in
some
cases
it
uses
IP
addresses
in
some
cases
it
uses
host
names.
C
This
is
just
Reddit
CLI
being
picky
about
when
to
show
host
names
and
when
not,
but
behind
the
scenes,
it's
it's
using
host
names
for
all
of
this
stuff.
C
Yeah,
and
so
we
can
see,
there's
a
single
key
so
far
and
let
me
try
and
remember
what
the
the
command
was
so
I
think
I
think
it
was
redisclustic.
Well,
actually
we
can
call
cluster
help.
This
should
tell
us,
there's
a
call
command
and
we
can
give
it
host
port
and
then
command
followed
by
arguments.
C
C
A
C
I
not
sure
I
want
to
do
that,
because
then
I'm
going
to
have
to
remove
it
and
removing.
C
Yes
seems
a
little
scary,
yeah.
A
C
Yeah
to
do
for
me
figure
out
the
correctly
passing
arguments
to
to
this
or
I
guess:
yeah
I
mean
I
did
quote
it
here,
so
there
is
some
some
additional
thing
that
needs
to
be
fixed.
C
C
Yeah
yeah
I
can
yeah
I'm
gonna
fix
it
later,
but
but
in
any
case
so
I
can
do
get
through
and
I
know
that
this
is
a
key
that
exists
and
we
can
see
okay,
what
most
of
the
hosts
responded
with
moved
and
then
the
slot
and
then
the
owner
of
the
slot,
and
then
one
of
the
hosts
actually
had
that
key
and
gave
it
back
to
us.
And
so
you
know
we
can
say
Inc
increment
request,
count.
C
And
we
can
sort
of
see
that
counting
up
and
yeah-
that's
that's
pretty
much
it
that's!
That's
kind
of
where
we're
at
and
I
guess
the
The
Next
Step
that
we're
working
on
right
now
is
getting
the
clients
configured
to
actually
talk
to
this
instance.
So
that's
gonna
be
redis.
Sorry,
gitlab
Helm,
chart
updates.
B
How
do
you
want
to
see
like
observability,
dashboards
and
stuff
on
a
redis
cluster
foreign.
B
Because
because
I
got
I
bogged
down
on
the
name,
so
the
first
sentence
you
added
and
then
you've
got
the
cluster.
So
the
cluster
name
in
the
beginning
and
then
Shard
one
two,
three
yes
or
zero
to
two
I.
Don't
remember.
B
So
I'm
wondering
should
those
be
called
shards
or
not,
because
there's
also
an
issue
where
we
want
to
combine
all
of
our
redis
instances
and
and
each
redis
instant
each
thing
that
we
currently
call
a
redis
instance
would
be
a
Shard.
So
we
have
the
services
redis
and
The
Shard
is
plate
limiting
sidekick
cluster
yeah.
C
B
The
alternative
that
I
talked
about,
while
you
were
talking,
is
using
redis
cluster
as
a
alternative
service
short
as
each
shard
in
there,
and
then
that
dashboard
needs
to
be
different.
So
it
accommodates
for
the
different
chart
like
that.
C
Right
but
then
we
need
like
a
deployment,
an
additional
deployment
label
or
something
like
that.
Yeah
I
think
the
easiest
thing
to
do
right
now
and
most
consistent
with
what
we're
doing
is
redis
cluster
rate
limiting
as
the
type
the
service
name
and
then
repurpose
The
Shard
label,
with
a
numeric
Shard,
that's
kind
of
what
I'm
proposing
at
this
point.
B
A
Is
it
possible
to
country
I,
don't
know
how
to
fix
this
is
but
the
the
way
the
presentation
suggests.
These
are
three
separate
things
that
happen
to
be
collaborating
in
redis
cluster.
Although
from
what
I
know
within
reddish
cluster,
it's
actually
nine
things,
and
if
you
would
let
credits
cluster
do
its
thing
and
they
would
move
all
over
the
place
so
you're
using
a
config
setting
to
prevent
that.
A
But
if
we
think
of
them
as
three
separate
things,
we
could
also
have
three
separate
dashboards
and
then
they
would
sort
of
look
like
Sentinel
deployments.
A
A
C
C
So
what
I
would
like
to
see-
and
maybe
Bob
is
gonna
yell
at
me,
because
it's
hard
to
do
is
have
a
Shard
selector
on
the
I.
A
Aren't
we
I
I
wonder
if
we
can
both
so
if
okay,
so
what
bulk
means
by
shards
is
the
red
is,
is
the
center
of
the
poems
we
have
now
and
what
Igor
means
by
shards
is
the
individual
Masters
within
the
this
one
redist
cluster's
appointment?
But
if,
if
we
say
that
we
treat
these
individual
Masters
and
they
replicas
like
individual
things-
and
we
just
forget
for
a
moment
that
they're
joined
together
in
British
clustered
that
you
can
both
and
you're
both
talking
about
the
same
thing
again.
B
All
this
conversation
is
hard,
but
I
think
I
agree
with
Igor
for
now,
like
we
have
some
things
that
are
already
correctly
using
Shard
and
I'm
thinking
about
sidekick,
that's
a
deployment
and
each
deployment
is
a
different
Shard
and
right
now
the
only
thing
like
the
way
we
can
C
different
chart
is
because
we've
adjusted
the
sidekick
dashboard
to
include
some
panels
and
because
we
link
to
different
dashboards,
but
it
would
be
way
handier
if
every
sidekick
chart
had
the
same
kind
of
slis,
which
is
like
throughput
of
jobs
with
what.
B
B
But
that's
that's
what
The
Shard
is
in
sidekick
and
now
what
Igor
is
proposing
and
I'm
I'm
kind
of
on
board,
because
that's
where
we
are
now
is
using
adding
a
new
redis
cluster
rate
limiting
service
and
using
the
the
three
yeah.
The
three
groups
of
three
sir
of
three
redis
servers
below
that
as
a
Shard.
So
every
redis
cluster
service
would
have
three
shards.
B
At
least
three
yeah
at
least
three
shards
got
it
and
for
that
you're
going
to
add
the
the
short
label
on
top
of
the
the
service,
dashboards
and
I
would
reuse
that
for
sidekick,
so
go
ahead,
go
for
it.
Okay,.