►
From YouTube: SIG - Performance and scale 2022-11-17
Description
Meeting Notes:
https://docs.google.com/document/d/1d_b2o05FfBG37VwlC2Z1ZArnT9-_AEJoQTe7iKaQZ6I/edit#heading=h.tybh
A
Okay-
and
please
add
yourself
as
an
attendee
okay
from
the
agenda
today,
we're
gonna
look
at
the
periodic
results.
Now
that
we're
seeing
the
Jobs
pass
again
and
it's
just
the
the
periodic
is
passing
for
the
performance
job,
not
the
dedicated
cluster,
so
we'll
go
over
the
over
the
periodic.
The
dedicated
cluster
is,
has
some
additional
work.
That
needs
to
be
done.
A
We
need
to
Brian
mentioned
that
we
need
to
do
some
upgrades
there
and
it's
running
kubernetes
121.,
so
we
need
to
upgrade
Docker
to
containerd
and
a
few
other
things
and
get
to
a
new
kubernetes,
which
means
a
new
sent
to
us.
So
a
bunch
of
things
need
to
change
to
get
this
back
to
a
healthy
state.
So
all
that's
going
on
we'll
keep
looking
at
the
regular
performance
jobs
that
would
be.
You
know
we
have
enough
memory
to
to
run
everything
so.
A
Let's
just
look
at
one
of
these,
so
we
have
last
time
we
didn't
get
a
chance
that
you
can
look
at
the
changes
that
were
made.
B
B
A
Okay,
so
first
check
is
the
regular
density
tests
and
the
second
check
is
the.
A
A
A
A
C
So
are
those
are
these
values
from
the
instance
type
reconciler.
A
No,
this
is
how
can
I
scroll
this
is
from.
This
is
from
the
this
is
creating
VMS
I
believe
so
without
an
instance
type.
The
the
third
job
is,
with
an
instance,
player
yeah.
C
D
C
Just
wondering
why
why
we
have
different
values
if
it's
the
same,
the
left
and
right
is
the
same.
A
A
Okay,
well,
I,
guess:
I!
Guess
we
should
well
I
guess
we
might
so
like.
Maybe
that's
what
this
is
so
like.
Maybe
that's
what
create
events,
because
I
guess
this
might
include
creating
the
VM
and
then
the
VMI,
so
I
mean
it's
look.
It
look
looks
to
be
roughly
one
to
one
I
mean
I
think
it
would
be.
A
Why
is
my
I?
Don't
know
why
it's
eight
to
one
I
I,
don't
know
about
I,
don't
know
why?
That's
that
that
seems
high,
but
I
I,
don't
know,
that's
something!
We
can
take
a
note
of
here's
another
one.
Like
here's
he's.
Also
like
get
endpoints,
it
looks
like
we
do
two
end
points
for
the
number
of
grades.
I
guess
and
here
looks
like
we're
doing
four
I
mean
that
it
seems
reasonable
this
one,
this
one's
weird,
so
I'm,
gonna,
I'm
gonna
skip
ahead.
Let
me
go
down
to
some
more
interesting
ones.
A
A
A
A
A
C
C
C
Right,
so,
if,
if
on
the
left,
we
are
reconciling
a
virtual
machine
instances
and
then
making
them
in
sync
with
the
state
of
the
underlying
VM,
then
I
would
assume
all
the
state
being
copy
pasted.
This
metric
will
will
go
high
because,
like
as
VM
changes
its
state,
it
will
update
the
the
vmis
and-
and
this
counter
keep
sticking.
A
C
Oh
hold
on
sorry:
I
got
confused,
so
just
to
be
sure
VMI
is
the
board
analogous
and
VM
is
the
replica
set
analogous
set.
A
B
D
A
A
C
So,
theoretically,
all
the
V
VM
controller
does
is
it
takes
the
spec
template
in
the
VM
and
creates
the
virtual
machine
instance
for
it
right
and
then
whatever
it's
the
VMI
state?
Is
it
will
copy
and
update
the
VM
State
accordingly?
A
They
have
slime
in
different
states.
So
like
there's
well
the
VM,
so
the
VMI
has
all
it's
like
it's
it's
running
whatever.
It's
that
has
the
the
runtime
States,
the
VM
I.
Don't
think
the
game
has
any
I
think
it's
just
a.
C
A
I,
don't
like,
if
you,
for
instance,
I
I,
think
if
you,
if
you
start
a
VM,
it's
right,
it
starts
with
Emi
but
I,
but
I.
Don't
think
it
changes
the
vm's
state,
for
instance
like
it
changes
the
object.
The
VM
object
like
it
basically
edits
the
VM
object
and
it'll
mix
it
started,
but
I,
don't
I,
don't
think
it
updates.
A
The
state
I
could
be
wrong,
but
I
think,
but
either
way,
though
this
is
the
VMI
updates,
and
so
the
VMI
is
the
one
that's
like
transitioning
between
the
phases
and
so
then
it
would
then
go
back
and
uptake
the
VM.
A
A
A
I
need
an
example.
This
would
be
on
here.
A
Okay,
there
are
always
here
we
go
yeah,
so
there's
no
there's
no
stay
in
here
like
we,
so
we
say,
run
started
yours.
That
means
it's
going
to
always
start
a
BMI.
C
The
VM
controller
will
like
relay
all
the
state
back
to
the
VM
status.
So
I
just
did
a
cube,
cut,
I'll,
explain,
VM,
dot
status
and
I
see
a
bunch
of
fields,
conditions
created,
ready
restored
in
progress,
snapshotting
progress,
start
failure,
State
change,
request,
volume,
request
and
volume
snapshot
status,
so
I
assume
all
of
these
information
is
relayed
back
and
forth.
C
C
There
is
a
created
field
and
then
there
is
a
ready
field
and
then
there
is
start
failure.
So
if
there
are
failures
in
starting
in
yeah,
these
are
the
three
fields
for
that
that
are
close
to
like
start,
and
then
there
are
conditions.
B
A
Could
be
okay,
I
I,
don't
know
we'll
have
to
think
about
that
one.
So
we'll
assume
that
they're
out
there
somewhere
but
the
anyway,
the
ones
that
we
do
have,
though,.
A
B
A
This
one's
the
I
think
that
one's
that
one's
kind
of
fascinating
I
don't
know,
but
this
one
is
not
quite
a
perfect
relationship.
It
might
just
be
that
we're
getting
some
we're
getting
some
requests
that
are
bringing
some
collisions
or
something
because
it
seems
like
we're
getting
some
through
and
some
of
them.
You
know
they
were
just
you
know,
some
of
them
aren't
or
just
colliding
or
getting
some
storage
Pierre
on
some
of
them.
But
these,
though
this
is
this
is
kind
of
surprising.
You
know.
A
A
A
A
A
D
A
A
Oh,
here's
an
updates
here
we
go.
This
is
what
which
one
is
this,
so
we
do
capture
it
on
one
of
these.
This
is
the
this
is
the
third
one.
So
the
batch
of
running
on
your
videos,
okay,
this
is
an
instance
type,
so
we
do
get
it
on
the
third
one.
Actually
we
don't
get
it
on
the
the
immigration
and
the
second
one
interesting,
there's
a
timing
issue.
Oh
here's
patch.
First
machines
too,
there
we
go.
This
is
good.
A
A
Yeah,
it
should
be
so,
let's
take
a
look:
okay,
create
events,
okay,
so
yeah
100,
100,
pod
crates.
That
looks
good
again
that
create
events.
Okay,
so
I
guess
that
makes
so
create
events.
This
is
I
guess
this
is
the
objects
that
we
created.
So
we
did,
like
instance,
type
objects,
I'm
thinking
and
that's
that's
what
we're
getting
and
then
we're
getting
VM
and
VMI.
So
yeah.
C
I
thought
of
that
as
differently.
What
I
am
thinking
is
there
are
like
Cube
events
right,
which
record
different
state
and
and
race
like
an
object
of
type
event.
I
thought
that
create
is,
for
those
events.
C
A
You
yeah,
how
do
you
create?
How
do
you
like?
How
do
you
get
like?
How
do
you
trigger
these?
That's
why
I
was
thinking
like
okay,
maybe
the
reason
there's
more
is
because
we're
we're
creating
more
objects,
and
then
that's
that's
doing
the
create
of
these
events.
I'm,
not
sure
I.
That's
that's
what
I
had
in
mind.
It
had
something
to
do
with
the
number
of
apologies
that
were
created.
A
A
C
C
No
so
I
verified
that
the
test
test
declined
does
not
the
metrics
generated
from
test
client
is
not
reported
here
so
on
the
right.
You
are
seeing
values
where
the
test
line
is
creating
the
virtual
machine
instance
the
100
of
those
and
then
on
the
left.
That's
why
it's
missing
that
and
on
the
left.
The
VMware
concealer
is
creating
the
virtual
machine
instance
based
on
that
VM
template,
and
that's
why
you
are
saying.
A
B
A
C
C
I'm
sorry
instance,
type
and
virtual
machine
preference.
C
B
C
Oh
I
I
think
one
of
the
patch
call
that
that
the
VM
controller
makes
is
to
remove
the
finalizer
on
the
VMI.
Once.
A
B
B
A
A
A
Give
me
a
Christian
running
panty
on
WOW,
that's
kind
of
interesting
I.
Guess
so
I
wonder
I,
wonder
why
this
was
the
case.
Let's
much
faster
I
guess
mostly
faster.
C
Yeah,
do
we
have
an
idea
of
what
is
the
workload
type
that
is
created
like?
Is
it
exactly
same.
A
It
should
be
I
mean
we
can
take
a
look.
This
is
the.
A
Yeah
look
as
you
can
sing
in
here
too.
His
PR
was
this
one.
B
D
B
C
Yeah
and
then
we
also
need
to
even
out
all
other
things
right
like
if
it
is
cloudy
neat
or
if
it
is
like
how
many
data
volumes
it
has
and
stuff
like
that,
it
I
mean
any
differences.
We
should
probably
try
to
minimize.
A
Yeah,
this
is,
let's
say
so,.
B
A
Yeah,
okay,
so
we
need
to
adjust
that
that'll
make
a
difference.
Okay,
all
right,
so
some
of
these
have
been
I.
I
still
think
these
this
makes
sense,
but
I
think
those
the
creation
time,
because
I
would
I,
don't
explain
it.
I
just
have
less
magnets
to
be
very
allocating,
I,
guess:
okay!
Well,
let's
yeah!
This
will
be
this
probably
an
easy
one
to
fix
all
right,
we'll
keep.
Let's
keep
monitoring
these
I'll
do
a
meeting
of
her
meeting.
A
Let's
see
if
we
can
slowly
chip
away
at
what
the
patches
are,
what
the
updates
are
and
then
these
I
I
mean
these
ones
are
also
what
are
the?
What
is
it
doing?
Why
does
it
take
four
gets
to
Virtual
machinist
types?
Why
is
that
required?
Why
is
therefore
gets
different
machine
preference
in
scale.
C
A
C
B
A
All
right,
the
last
thing
I
did
wanted
to
say,
was
the
the
mem
usage
that
we
talked
about
previously.
D
A
E
A
E
E
Yeah
I
I
think
we
found
the
owner
of
that
cluster,
so
I
think
we're
looking
at
maybe
a
reinstall
of
that
cluster
and
I
want
to
check
with
lubo
from
Cuba
just
first
just
to
see.
If
there's
anything
else,
you
can
do,
but
it
might
be
best
just
to
try
and
reinstall
it
probably
with
something
like
openshift
just
to
have
a
platform,
that's
kind
of
easily
maintainable,
so
I
think
that's
what
we're
looking
at
the
moment,
because
at
the
moment
there
are
those
notes
around
centers
eight.
E
So
yeah
I
just
need
to
check
with
the
owner
of
the
cluster
and
see
if
that's,
okay
and
again
check
the
sixth
scale
as
well.
Just
to
let
you
know
as
well.
A
Sounds
good
cool
all
right
thanks
for
that?
Okay,
then,
let's
go
to
my
message.
This
was
the
this
is
brought
up.
You
brought
this
up
Brian
a
few
weeks
ago
about
how
we
kind
of
ended
up
catching
the
main
increase
in
our
performance.
Job
I
I
think
there
was
good
erase
an
issue
we
had.
We
got
some
discussion,
I
mean
I,
think
maybe
for
next
steps
on
this
one.
A
It
might
be
worth
having
a
mailing
list
thread
and
seeing
what
people's
appetites
are
for,
having
something
that
we
I
mean
like
I
guess,
maybe
two
things
it's
like
one
of
them
is
that
are
the
gate,
is
kind
of
checking
and
I'm,
usually
trying
out
like
we're
kind
of
like
when
the
job
fails.
I
mean
pretty
much
the
only
fires
we've
had
over
the
last
like
five
months
or
so
has
been
then
usage
changes,
and
so
it
is
I
I
think
like
for
the
most
part,
it's
pretty
stable
and
that's
just
been
the
change.
A
So
there's
I
I
think
there's
some
value.
If,
if
we
wanted
to
try
and
make
this
job
something
that
runs
on
a
regular
basis
and
and
had
does
reporting,
it
might
give
us
some
feedback
as
to
both
the
you
know,
the
mem
usage,
which
I
think
is
value
on
that
and
then
also
the
the
actual
performance
if
there
are
actual
performance
changes
on
a
PR
and
I.
Think
that,
like
this,
this
would
be
interesting
as
something
that
I
think
you
know.
E
Yeah
yeah,
it's
funny
like
I,
was
talking
to
a
few
people
about
this
and
and
like
there's
the
view
that
the
memory
usage
will
increase
over
time
as
well
as
as
things
get
more
complicated
or
as
as
more
features
are
added.
E
So
there
is
that
side
of
things
as
well
that
memory
usage
will
just
increase
naturally,
does
that
side
of
it
as
well,
but
obviously
you'd
like
to
have
some
kind
of
view
on
where
the
memory
usage
is
going
as
in
like
where
the,
where
the
kind
of
the
high
memory
usage
is
is
happening.
Of
course,
yeah
I
think
a
mailing
This
Thread
could
be,
could
be
good
to
see
what
kind
of
interest
there
is
from
the
community.
A
All
right,
I
can
take
that
one
I
can
I
can
write
it
a
little
thread
and
we
can
see
what
people's
appetites
are
for
this
and
yeah
I
agree
with
you
that
mem
usage,
you
know,
it'll
increase
and
I
think
what
we
said
previously
is
that
it
would
be
good
to
say
we
know
that
in
this
release
it
increased
whatever
whatever
amount
it
is
and
I
think
that
would
be
whatever
in
a
release
note
or
something
and-
and
this
would
be
a
great
way
to
catch
it
like
we
I,
really
think
like
some
changes,
we've
seen
like
oh
yeah,
we're
clearly
bumping
the
memory
and
there's
nothing
to
say.
A
Okay,
this
is
what
it
affects.
You
know
and
missing.
Big
is
a
way
that
we
can
stay
up
to
here.
It
clearly
affects
the
performance
job.
You
know,
let's,
let's
have
a
release.
Note
that
that's
what
the
change
is
and
maybe
that'll
you
know.
We
bubble
that
up
to
the
release
and
say:
okay,
the
keeper
uses
this
much
more
memory
the
release,
and
you
know
this
is
what
it
you
know.
So
people
know
it
how
can
affect
them.
B
A
Yeah
and
I
think
I
think
even
I.
Think
fixing
like
the
other
thing
is
that
maybe
people
might
complain
a
little
bit,
but
it
doesn't
happen
that
often
that
these
things
come
along
and
then
the
change,
the
fix,
I
I,
think
it's
pretty
easy.
It's
pretty
simple
I
think
we've
documented
it
now
like
four
times,
I
think
pretty
easily.
You
can
show
someone,
here's
how
you
fix
this
and
and
I
think
it's
it'd
be
pretty
simple,
so
I
I,
think
I,
don't
know,
I,
think
there's
a
path
forward
for
this
I.