►
From YouTube: SIG - Performance and scale 2023-03-09
Description
Meeting Notes:
https://docs.google.com/document/d/1d_b2o05FfBG37VwlC2Z1ZArnT9-_AEJoQTe7iKaQZ6I/edit#heading=h.tybh
A
B
Welcome
to
sixth
scale:
this
is
March
9th
2023..
Okay,
the
link
to
the
meeting
notes
is
in
the
is
in
chat.
Please
add
ourselves
an.
B
Okay,
so
first
things
first
I
we're
we've
been
doing
the
the
performance
job.
B
The
periodic
job
results
for
you
for
a
little
while
now,
and
one
of
the
things
that
we
want
to
change
is
eventually
get
away
from
having
to
do
this
and
move
to
publishing
the
stuff
in
graphs
and
then
sort
of
reviewing
it
in
graphs
instead
of
having
to
go
through
the
job.
So
for
now,
just
what
I'll
do?
B
B
So
we'll
continue
to
monitor
this
on
meetings,
but
eventually
we
want
to
get
to
actually
having
something
that
looks
like
this,
which
is
you
can
see,
for
example,
in
this
graph,
basically,
is
looking
at
Gathering
much
of
data
about
different
jobs
and
pro
and
rendering
it.
So.
The
idea
is
that
we
should
be
able
to
do
the
same
thing
with
what
we're
doing
right
now
with
our
jobs.
B
So
Lupo
had
some
ideas
for
this,
so
I
started
a
threaded
keeper
death
with
lubo
and
we
can
follow
up
there
and
then,
hopefully,
in
a
future
meeting,
we
can
have
a
more
in-depth
discussion
if
based
on
what
he
thinks
we
should
do
and
go
from
there.
B
Okay
performance
cluster,
so
I
asked
Brian
to
get
this
information
still
in
discussion
with
them.
To
get
this
I.
Think
I,
don't
know
if
this
is
done
yet
I
don't
think
we
have.
We
have
this
information
and
I'm
still
working
on
getting
the
hardware
respect
so
once
I
have
this
we'll
publish
it
here
and
I
think
what
we'll
do
is
eventually
publish
this
somewhere
as
part
of
like
our
description
of
our
our
performance
cluster.
B
Jobs
are
running
against.
Okay,
let's
talk
about
K
walk
Olay.
Do
you
want
to
talk,
introduce
this
topic.
C
Yeah
can
I
share
my
screen.
A
C
Oh
so
I
just
want
to
take
some
time
today
to
informally
talk
about
a
new
project
that
might
be
very
interesting
for
the
scale
work
that
we
are
doing
in
this
call.
C
So
one
of
the
things
or
one
of
the
key
ideas
behind
this
new
project
kubernetes
without
cubelet
is,
can
we
fake
out
some
of
the
needed
Hardware
so
that
we
can
create
a
fake
notion
of
a
scale
in
the
cluster
and
and
then
see
how
the
control
plane
components
scale
with
respect
to
those
fake
objects?
Still
recently,
an
upstream
project
called
Kwok
was
introduced.
C
This
project
has
two
controllers
right
now,
one
which
that
works
on
the
fake
node
status
and
one
that
works
on
fake
pod.
C
So
here
is
the
blog
post
about
that
some
use
case.
Use
cases
that
are
mentioned
here
is
that
this
is
largely
going
to
be
used
for
testing
like
and
Learning
and
Development.
So
this
helps
us
understand
the
performance
and
scalability
of
control
playing
better
and
same
goes
like
if
we
extend
this
for
the
cubeboard
control
plane.
C
This
could
could
be
leveraged
to
understand
the
scalability
of
the
cupboard
on
control
plane,
obviously,
because
of
the
fake
nature
of
of
the
objects
there
is,
there
is
some
concern
about
accuracy
and
and
functionality
of
of
these
fake
objects.
So,
in
order
for
the
test
to
be
meaningful,
we
should
not
rely
on
the
functionality,
but
rather
rely
on
the
scale
aspect
and
and
create
the
churn
in
the
actual
actual
objects.
C
So
with
that,
I
I
want
to
quickly
walk
through
how
and
introduce
how
this
works.
C
These
controllers
have
a
notion
of
stage
the
the
way
this
works
is
that
in
in
a
stage
there
is
a
template
defined.
These
templates
are
rendered
in
in
the
controller
and
the
rendered
output
is
applied
as
the
status
of
of
the
controller
or
status
of
the
fake
options.
C
The
interesting
part
here
is
that
you
can
combine
set
of
stages,
so
here
you
can
see
that
there
is
a
next
section.
So
after
this
stage
is
applied,
what
is
the
next
stage
that
will
be
applied
right?
That's
what
this
section
defines,
and
there
is
some
kind
of
configuration
parameters
with
respect
to
time
related
configuration
like
some
amount
of
Randomness,
Jitter
and
and
after
how
many,
how
much
time
this
state
with
the
status
will
be
applied.
So
those
are
all
configured
by
this
section
and
the
actual
object
is
configured
by
by
this
object.
C
So,
to
sum
up
the
stage,
this
is
not
actually
a
crd,
it's
something
that
you
would
have
to
Define
at
the
start,
time
of
the
controller.
So
this
is
a
configuration
file
that
is
supplied
to
the
controller
in
the
beginning,
but
in
the
spec
of
this
stage
allows
you
to
configure
what
resource
to
act
on
when
to
act
on
and
what
status
to
apply.
B
So
lay
kind
of
what
you're
showing
here
is
that
so
we
could
take
the
controller
from
KW.
Okay,
we
can
apply
it
to
a
cluster
and
we
should
we
will
be
able
to
launch
a
bunch
of
objects,
and
this
is
going
to
stress
the
control
plan
for
us
and
that's
going
to
give
us
an
approximation
of
of
our
scale.
Is
that.
C
B
And
so
you
would
you,
do
you
know
if
you
do
you,
if
you
need,
is
it
required
for
you
to
launch
fake
pods
on
fake
notes,
or
is
it
like?
What
is
the
relationship
or
is
it
that
they're
independent,
like
we
just
create,
can
create
fake
nodes
to
create
pressure,
or
it's
a
requirement
to
have
to
use
the
two
of
them
together?.
C
So
the
way
it
works
is
you
have
to
use
a
label
selector
to
Define
what
objects
these
controllers
will
act
on,
so
you
can
create
fake
pods
on
on
real
nodes,
but
they
just
have
to
match.
C
The
problem
with
that
is
you'd
somehow
have
to
tell
the
cubelet
on
the
real
node
to
not
act
on
it,
and
that
would
be
little
difficult
to
configure
in
my
in
my
little
experiment.
So
far
what
I
have
observed
is
this
awok
controllers
just
work
on
the
label
selector,
so
they
can
take
over
objects
in
the
in
the
real
node.
But
not
you
would
have
to
somehow
stop
the
cubeletter
or
something
to
make
sure
that
cubelet
and
the
fake
controller
don't
race
against
each
other.
B
C
Yeah,
so
that's
that's
a
bit
of
the
introduction
of
this
stage.
I
wanted
to
take
a
moment
to
go,
look
at
delivered
stages
that
will
help
us.
C
So
this
is
the
default
stages
for
a
node,
so
there
are
two
stages
here:
one
is
the
node
initialize
and
the
other
is
a
node
heartbeat,
so
the
node
initialize,
what
it
does
is
it
initializes
the
node
status,
with
a
set
of
default
resources
and
other
other
status
conditions
like
the
IP
address,
the
hostname,
cubelet
and
point,
and
and
so
on
and
so
forth.
B
Kwok
has
a
controller
and
I'm
seeing
status
here,
I'm
seeing
some
other
fields,
and
since
this
is
a
fake
node,
but
I'm
understanding
like
this,
this
fake
note
is
like
a
real
like
it's.
It
uses
the
node
API
right.
So
how
is
this
able
to
interact
with
status
and
some
of
the
other
fields
that
are
really
meant
for
the
node
controller
and
that
guy
wasn't
not
able
to?
How
is
it?
I
was
able
to
interact
with
those
fields
and
how
is
the
No
Control
and
not
fighting
against
it?.
C
So
I
think
my
understanding
is
that
some
of
the
fields
in
this
status
are
owned
by
the
cubelet.
The
other
fields
are
are
owned
by
the
node
controller.
So
when,
when
you
initialize
the
the
node
this
the
status
that
is
being
applied
only
populates
the
field
that
are
owned
by
cubelet,
so
in
a
way
it
fakes,
the
cubelet
and
once
it
has
learned
that
the
next
part
the
node
heartbeat,
is
what
like
this
is.
C
The
part
I
think
that
is
applied
by
the
the
node
controller
and
because
some
agent
is
already
applying
this.
The
node
controller
is,
you
know,
not
taking
a
look,
but
that
is
one
question.
I
I
have
and
I'm
still
trying
to
figure
out
like.
Why
is
the
node
controller?
Not
fighting,
but
that's
my
initial
guess
as
to
because
there
is
no
cubelet
running
on
it.
Node
controller
is
not
able
to
talk
to
the
cubelet
and
it's
like
not
updating
the
status
so.
B
B
Assuming
then
that
the
node
controller
must
do
something
that
would
be
interesting
because
I
mean
I
I,
don't
see
how
it
would,
and
it
probably
would
want
to
do
something
and
then
maybe
maybe
then
the
the
fake
controller
does
a
bunch
of
stuff
afterwards
or
something
that
I
would
expect
at
least
that
the
node
controller
does
at
least
at
least
something.
But
that's
that's
fine
I
mean
if
it
does
or
not
I
think
it
just
I.
B
C
Yeah
I'll
take
that
as
an
open
question
to
figure
out.
Why
is
this
not
raised
against
the
node
controller?
It's
an
interesting
part.
It
will
help
out
a
lot
in
understanding
how
this
project
can
be
used.
C
Yeah
and
yeah
I
I
wanted
to
complete
the
thought
I
was
running
with.
Is
that
so
the
first
stage,
the
initial
stage
applies
the
phase
equals
to
running
to
the
to
the
node
object.
Then
the
next
stage
you
can
see
here
that
in
the
selector
section
it
says,
match
status,
dot
phase.
So
once
this
fake
object
has
gone
into
running
State
without
delay
of
so
and
so
mentioned
in
the
spec
denode
controller.
Well,
the
fake
node
controller
will
append
these
status
conditions
to
the
to
the
fake
object.
C
So
what
ends
up
happening?
Is
you
stimulate
the
fake
nodes
with
with
some
statuses,
and
you
don't
need
the
actual
hardware
for
it,
and
then
you
can
use
a
similar,
similar,
odd,
yeah
or
letting
your
odds
go
to
running
state.
C
So
if
you
look
at
the
keyword
stack
when
we,
when
a
user
creates
a
VMI,
the
word
controller
sees
that
VMI
and
creates
a
pod
for
it,
the
PO.
What
we
could
do
potentially
is
that
the
BMI
can
be
configured
with
a
fake
with
a
selector
that
selects
fake
nodes
and
the
Pod
created
for
that
by
the
god.
Controller
will
eventually
land
on
those
fake
nodes.
The
Pod
can
go
into
running
State
because
those
are
actually
fake
odds
and
and
not
running
real
compute
behind
it.
C
Although
there
is
an
interaction
between
the
word
launcher
pod
and
the
word
Handler,
the
word
Handler
tries
to
talk
to
the
word
launcher
pod
and
move
moves
forward.
The
state
of
the
vmis
just
like
how
cubelet
does
so
when
I
tried
this.
What
ended
up
happening
is
the
VMI
started.
What
well
the
VMI
was
created.
Word
controller
created
a
fake
pod
for
it
the
part
went
to
running
state,
but
the
VMI
was
stuck
in
schedule,
State
because
there
is
no
word
Handler
running
on
the
fake
node
to
drive
it
forward.
C
So
just
like
how
there
are
these
stages
for
node
and
pod
I
think
we
would
have
to
add
an
extension
for
VMI
to
move
these
stages
forward,
that
is,
to
move
the
BMI
from
scheduled
to
running
State,
and
once
we
have
that
functionality.
C
We
can
leverage
this
to
create
a
constant
number
of
fake
vmis
in
a
cluster,
so
that
adds
the
scale
to
it
and
then
you
can
create
well,
you
can
extrapolate
how
keyword
is
performing
by
actually
running
real
vmis
on
on
real
cluster,
so
this
reduces
the
need
for
real
Hardware
by
a
factor
of
at
least
half
right.
C
C
B
It's
pretty
cool,
so
the
so
fake
hurt
handlers.
One
thing
you've
identified
so
this
we
would
need
a
so
we'd
say
it.
We
call
it
a
fake
VMI,
which
is
really
a
a
fake
for
launcher
running
in
a
fake
pod.
So
is
it
like,
since
it
has
no
run
time,
I
guess
what
we
could
do
is
like
what
we
were
saying.
Is
they
just
kind
of
the
vert
Handler,
since
when
is
the
pathways
right?
B
It's
like
the
launcher
has
a
bunch
of
signals
and
sends
back
to
the
Handler
I
guess
what
we
just
need
to
do
is
have
the
events
we
just
sort
of
faked
the
whole
thing
we
have
as
if
the
events
already
happened
and
and
just
send
the
information,
and
basically
we
just
update
the
object.
This
is
I
think
what
we
do.
Yeah
yeah
yeah
the
other
option
would
be
like
well,
this
is
the
hard
part.
This
is
the
harder
one.
B
Is
that
if,
if
there
was
a,
if
you
had
the
ability
to
run
a
real
vert,
Handler
and
I,
think
this
this
could
be
possible
because
technically
what's
happening
is
the
launcher?
Is
communicating
over
grpc
and
then
there's
a
bunch
of
there's
a
domain
socket
on
there
for
events
with
with
the
guest,
and
so
you
could
communicate
over
the
socket
do
fake
all
events
being
sent
back
to
a
real
word
Handler
and
it
could
do
it
as
well.
There's
there
might
be
two
approaches
in
terms
of
how
we
could
do
it.
B
I
guess
depends
on
the
environment
that
we
like,
if
you
have,
if
you
have,
for
example,
if
you
have
a
node,
a
real
node
and
you
have
a
real
handle,
you
could
do
the
fake,
fake
launcher
method.
If
you
have,
if
you
want
to
go
with
the
full,
fake,
node
and
fake
VMI
method,
you
could
go,
you
could
go
that
as
well.
Yeah,
I
guess
the
difference
would
be
is
like
one
test
of
Our
Land
one
test.
The
word
handle
the
other.
One
are
both
of
them
test
the
control
plane.
C
Correct,
yes,
so
I
I
think
to
elaborate
the
second
part
right,
the
the
one
where
you
have
real
node.
It
tests
the
scaling
capability
of
word,
Handler
Handler,
without
the
limited
without
cubelet
in
the
picture.
So
because
we
have
this
fake
power
cubelet,
we
will
first
have
to
figure
out
how
to
not
race
against
the
cubelet
and
assume
that
cubelet
doesn't
intervene
on
running
that
fake
pod,
and
once
you
have
that
you
can
understand
how
word
Handler
can
scale
independent
of
cubelet.
C
C
The
third
approach
I
had
in
mind
was
that
run
the
run.
The
real
word
Handler
on
a
different
node,
but
ask
it
to
reconcile
World
launcher
pods
on
on
this
of
this
fake.
Now.
B
So
it
could
be
so
like
it'd
just
be
lying,
so
like
it'd,
be,
it
would
be,
it
would
just
have
to
well
okay,
so
there's
some
there
I
might
be
a
little
more
complicated
than
that,
but
you
would
have
to
that
would
be
the
gist
of
it
is
like
you
have
to
sort
of
lie
where,
where
this,
when
a
connection
is
coming
from.
C
C
B
B
B
It
would
be
interesting
to
see
like
you
know,
if
we
can
sort
of
extend
ourselves
a
little
bit
past
the
physical
hardware
and
see
how
what
approximations
we
find
and
since
that
cluster
is
a
little
larger.
So
that's
cool
that
sounds
like
it
would
be
really
cool
to
see.
C
Yeah
yeah
I
think
the
easiest
part
would
be
to
assume
the
fake
node
and
the
fake
part
combination
and
just
try
to
do
the
BMI
objects
from
a
controller.
That
might
be
something
we
can
realize
sooner
than
the
other
parts
sure
so
yeah.
We
can
try
that.
B
Okay,
cool
yeah
thanks
a
lot
of
light
and
I
took
some
notes
here
and
so
next
time
we'll
see
from.
If
we
get
a
chance
that
the
demo
we
can
name
I
wanted
to
show
us
whatever
you
got
for
next
time:
cool.
Okay,
all
right,
I,
don't
have
any
more
topics.
So
is
there
anything
else
and
we
still
got
more
time?
Is
there
anything
else?
People
want
to
bring
up
and
discuss.
C
Hey
Ryan
I
was
a
little
bit
interested
in
getting
an
overview
of
that
CI
health
and
the
the
graph
that
was
that
is
constructed
on
the
the
main
page.
C
I
wondered
if,
though,
the
graph
data
is
coming
from
churning
the
like,
is
it
coming
from
processing
the
the
project
UI
or.
B
A
B
Let's
see,
this
is
a
link
to
a
grafana
void.
What
is
this
free?
We
test
some
words,
so
I
guess
this
is
coming
from
grafana.
A
B
C
Yeah,
the
reason
why
I
was
curious
is
that
it
will
be
good
for,
like,
as
you
said,
for
our
scale
jobs.
It
would
be
good
to
numb
some
kind
of
data
in
the
in
the
graph
format.
So
we
have
this
historical
collection
over
time
and
I
was
curious
like
if
we
can
follow
the
same
method
to
do
this.
So
that's
why
I
was
looking
at
where
to
go,
poke.
B
Yeah,
absolutely
I
I
was
hoping
to
hear
a
little
more
from
Google
because
discussion
and
they
had
an
idea
because
there's
some
automation
here,
like
that's
happening
like
like
how
does
this
get
here?
I-
have
no
idea
like
there's
a
keyword
bot
that
does
this
I,
don't
know
where
the
keyboard
it's
wired
up,
so
that
it
is
able
to
pull
this
data
because
I
I
thought
there
was
a
way
to
pull
this
from
prowl
and
I
thought.
That's
what
was
going
to
be
going
on
here
and
that's.
B
What
I
was
hoping
is
like,
because
we
already
have
like,
like
in
our
jobs
like
right
at
the
at
the
ends
of
each
shot.
We
have
to
like
basically
a
format
like
this:
it's
just
it's
not
in
Json,
but
we
could
easily
make
a
Json
export
it
as
an
object
and
then
and
then
we
basically
have
everything
we
need
and
and
then
we
just
need
to
put
it
to
a
graph.
So
I
mean
we
pretty
much
have
the
hard
part
down,
and
we
just
need
to
figure
out
how
to
wire
this
up.
B
So
yeah
I
don't
know
I'm
hoping
to
hear
a
little
bit
more
from
lubo
and
Daniel
on.
So
we
can
get
get
started
on
this
I
said.
I
would
definitely
love
to
see
this
I
mean
this
is
this
is
great
because
I
mean
thinking
about
it.
Like
you
know,
we
have
a
good
visualization
over
time
like
this
is
something
that
when
we
look
at
our
CI
jobs,
we
we
don't
really
have
that
we
can
like
just
you
know,
look
at
what
we
can
only
look
at
a
few
at
a
time.
B
B
Yeah,
there's
it's
on
it's
on
keyboard,
Dev
and
I
can
tell
you
as
well.
Okay,
okay,
all
right!
Are
there
any
more
topics
anywhere
things
people
want
to
bring
up.
B
All
right,
well,
just
a
reminder:
please
add
yourself:
it's
on
10B,
since
I
we've
had
some
people
talk
with
like
that
yourself,
some
attendees,
because
it's
important
to
see
the
attendance-based
meeting.
Okay,
thank
you.
Everyone,
bye-bye.