►
From YouTube: KubeVirt Community Meeting 2022-01-19
Description
Meeting Notes: https://docs.google.com/document/d/1kyhpWlEPzZtQJSjJlAqhPcn3t0Mt_o0amhpuNPGs1Ls/edit#heading=h.u74oyrl72es0
A
Okay,
so
I've
started
recording
not
quite
going
we'll
wait
until
the
minute
ticks
over,
but
just
wanted
to
put
out
there.
The
kubert
summit
still
coming
up.
We've
got
a
good
number
of
sessions
already,
but
I
was
advised
that
you
know
more,
wouldn't
hurt.
A
So,
even
though
it's
after
the
deadline
for
the
call
for
papers,
we
certainly
would
take
more.
You
can
reach
out
directly
to
any
one
of
us
just
go
to
the
cooper
dev
mailing
list
and
let
somebody
know.
A
A
E
I
think
at
least
the
link
to
the
meeting
notes
is
also
in
the
the
invitation
right.
F
My
name
is
sorry
hi.
My
name
is
abhishek.
I
joined
this
community
recently,
the
cncf
community,
where
I
found
this
this
particular
link
for
keyboard,
so
I'm
very
new
to
this
keyboard.
So
probably
if,
if
someone
can
have
help
me
on
understanding
how
this
cubework
will
help-
and
I
I
have
requested
for
the
access
of
this
link,
can
you
please
approve.
A
Okay,
I'll
look
into
that,
I'm
not
sure
if
I
have
the
the
right
access
you're
trying
to
get
which
link
the
the
one
to
the
meeting
notes.
F
This
box.google.com,
whatever
the
thing
you
open,
I
have
requested
for
the
access.
Oh.
A
So
yeah
join
our
forum.
I
believe
that
gets
you
into
the
google
group
and
that's
open
to
anybody.
So
once
you
once
you
are
on
that,
you'll
also
have
the
calendar,
invite
and
access
to
the
docs.
E
And,
besides
that,
you,
if
you
have
any
questions
you,
we
also
have
a
slack
chat.
That
is,
that
is
in
the
on
the
same
page,
so
feel
free
to
join
and
ask
questions,
for
example,
in
any
of
the
keyboard
slack
channels
that
we
have
there
virtualization
equipment.
I
guess
first
of
all,
maybe
virtualization.
First.
A
Okay,
if
we
don't
have
any
other
new
introductions,
then
we
can
move
on
to
the
agenda.
I
think
we
have
something
about
coveralls.
E
I
I
heard
that
my
audio
volume
is
very
low.
I'm
not
sure
why
that
is.
Can
can
you
still
hear
me
okay
or
is
it
low.
E
Okay,
so
yeah,
okay,
I
wanted
just
I
I
had
a
question
to
ask
regarding
the
coverage
that
we
are
running
on
all
keyword
prs.
So,
first
of
all,
we
had
a
time
I
think
where
we
were
beyond
70
of
coverage
and
this
value
is
slowly
starting
to
drop
again.
So
I
think
we
are
now
near
the
bottom
line
of
70
and
maybe
losing
beyond
that.
E
So
I
was
thinking
about
whether
do
we
want
to
enforce
something
like
a
lower
boundary
of
coverage
value
or
do
we
just
do
we
just
want
to
that
to
happen,
naturally
by
pr
reviewers
that
maybe
tell
people
that
they
should
add
a
test
here
or
test
there,
because
I
think
in
general
coverage
is
just
something
that
is
a
best
practice.
E
I
don't.
I
wouldn't
say
that
that
coverage
in
itself
is
bad,
of
course
not,
but
sometimes
it's
a
little
bit
awkward.
If
you
have
yeah
yeah,
you
have
the
famous
test
test
coverage,
things
that
just
run
things
and
don't
test
anything
or
something
like
that.
So
what
is
the
general
opinion
of
that?
Should
we
should
be
enforcing
lower
boundaries,
or
should
we
just
keep
it
like
like?
It
is
now.
G
So
one
thing
to
to
add
a
little
context
to
what
daniel's
bringing
up
here
we
had
a
pr
this
week
that
was
exclusively
removing
lights
and
and
in
general,
that's
absolutely
good,
because
less
code
like
if
the
code's
just
cropped
and
not
needed,
absolutely
take
it
out,
and
so,
but
unfortunately,
because
it
was
removing
lines
from
the
total.
G
That's
one
of
the
dynamics
that
dropped
our
our
coverage
beneath
that
70
threshold
and
the
coverage
bot
rejected
it,
based
solely
on
the
fact
that
it
had
fallen
underneath
the
the
line,
and
this
is
a
gating
lane,
and
so
that
that
just
begs
the
question
as
to
wait
a
minute.
E
Yeah,
and
also
also
I
was-
was
just
thinking
about
what
what
the
community
thinks
in
general
about
coverage
also,
so
I
want
to
want
to
get
a
feeling
of
how
we
should
be
handling
all
this
coverage
stuff.
D
D
Like
stu
pointed
out,
it
can
get
a
little
hairy
when
we're
doing
changes
that
are
actually
good
in
the
coverage
report
we
enforce.
It
doesn't
necessarily
match
reality,
so
we
also
have
like
lots
of
generated
code
as
well,
so
we
could
have
enormous
amount
generated
code
that
might
not
technically
be
tested.
I'm
not
sure
if
that
happens
or
not,
but
that
would
be
another
thing
that
would
kind
of
put
it
in
like
offset
the
the
percentage
tested
versus
what
was
actually
like
new
logic.
D
I
feel
uncomfortable
with
making
a
hard
requirement
that
we
should
stay
around
a
certain
threshold
or
not
drop
below
a
certain
threshold,
or
something
like
that,
and
I
think
that
it's
kind
of
on
the
reviewer's
part
to
use
this
as
a
tool
use
the
coverage
as
a
tool
to
understand.
D
D
I'm
hesitant
to
do
that.
I'd
be
interested
in
other
thoughts,
I'm
hesitant
because
I
think
there's
going
to
be
situations
where
it
drops
and
that's
okay
and
then
we
have
to
make
a
decision.
What's
our
policy
at
that
point
like
if
we're
saying
that
there's
reviewer
discretion
with
this
unit
test
or
this
coverage
test
and
the
threshold
then
does
it
have
value
to
begin
with
for
us
to
enforce
it?
I
I'm
unsure
yeah,
so.
H
H
D
Yeah,
I
guess
here's
another
scenario.
It
doesn't
always
reflect
even
when
more
test
coverage
is
necessary.
So
let's
say
somebody
makes
a
20
line
change
to
one
of
our
controllers
or
something
like
that.
That
might
need
quite
a
bit
of
test
coverage
depending
on
where
that
is
in
the
code
and
like
what
critical
code
path
that
is
involved
there.
Now,
if
that
doesn't
get
touched,
we're
talking
about
a
fraction
of
a
percentage
difference
in
our
test
coverage
report,
but
that
doesn't
reflect
how
critical
that
one
area
was.
E
I
don't
think
so.
I
think
our
coverage
is
equal
in
that
point,
but
I
just
wanted
to
to
to
get
back
to
david.
What
david
said
about
that?
We
are
also
covering
generated
code,
and
I
think
I
worked
on
that
topic
and
I
just
removed
all
the
generated
files
from
the
coverage
report.
So
I
think
we
should
be
at
least
good
there.
E
Yeah,
I
think
we
are
filtering
by
by
file
extension.
E
So
if
there
is
some,
for
example
like
like
most
of
the,
I
don't
remember
exactly,
but
I
think
there
is
something
like
generated
go
or
something
in
in
the
file
most
of
the
time
and
at
least
those
we
filter
out
when
we
create
a
report,
I
maybe
you
know
better
whether
there
are
other
extensions
that
we
should
take
a
look
for.
D
Yeah,
I'm
not
sure
I
just
know
that
when
we
generate
like
a
new
api,
the
client
code
and
everything
associated
with
that
and
like
the
staging
folder
is
pretty
extensive.
So
maybe
that's
not
included
in
the
coverage.
Maybe
it
is.
I
haven't
looked
at
it
but
yeah.
If
we're
ignoring
some
of
that,
then
maybe
it's
more
accurate
yeah.
E
Okay,
so
so
in
general,
I
think
that
that
we
shouldn't
enforce
any
boundaries
and
just
leave
it
to
the
reviewer.
I
I
was
just
thinking
about
probably
that
maybe
we
are
whether,
when
the
pr
got
merged,
then
it's
too
late
and
if
someone
overlooked,
for
example,
missing
tesla
something
there
could
be
situations
that
the
reviewer,
because
he's
also
just
human,
just
overlooks
the
fact
that
we
didn't
we.
He
forgot
to
ask
that
the
author
of
her
got
to
ask
or
something
but
yeah.
E
I
think
that's
something
that
that
we
then
should
see,
at
least
with
the
coverage
values
decreasing
to
a
certain
point:
yeah.
Okay.
I
think
we
should
leave
that
to
the
reviewer.
D
Yeah,
it
is
a
risk.
What
would
it
look
like
just
as
an
alternative
to
this?
If
we
made
this
required,
this
coverage
report
required
not
to
drop
below
a
certain
percentage
or
whatever
for
us
as
a
reviewer
to
override
that
so
treat
that
as
a
signal
that
has
to
have
a
reviewer
intervention
to
ignore.
E
You
can
you
can
just
override
it
with
with
a
default
override
command,
so
at
least
that
should
be
good.
Then.
G
That's
an
interesting
suggestion,
so
basically
we
would
say
we
would
still
have
a
general
guideline,
but
in
certain
cases
we
could
intentionally
ignore
it.
E
D
E
D
Well,
I
was
just
going
to
say
if,
if
we
do
that,
we
would
need
to
write
down
a
policy
of
how
that
is
handled
like
why
it's
okay
to
ignore
it,
something
to
point
to
that's
kind
of
like
a
contract
that
we've
all
agreed
on
that
this.
D
This
test
is
informational
as
a
signal
to
the
reviewer
that
something
needs
to
be
looked
at,
and
it's
reviewers
discretion
whether
something
needs
to
be
changed,
because
I
don't
want
to
see
it
enforced
where
people
have
to
do
kind
of
silly
things,
just
to
increase
the
code
coverage
for
their
pr
when
it
makes
no
sense.
So
we
need
to
just
be
explicit
about
our
contract
there.
So
there's
no
confusion
about
what
that
failing
test
means
and
what,
whether
it's
okay
or
not
to
ignore
it.
H
It
it,
I
think
that
if
you
put
the
the
check
in
it
will
be
red
and
then
you
ask
someone
maintenance
to
override
the
check.
That's
that's
nasty.
I
think.
Let's
take
it's
complicating
everything
in
my
opinion,
how
how
about
just
putting
a
rule
that
it
cannot
drop
more
than
two
percent
per
per
pr,
or
something
like
that.
Does
this
make
sense.
H
D
I
think
that
was
kind
of
something
like
that
was
what
we
were
talking
about.
I
thought.
H
No,
I
think,
currently
it
was
daniel.
F
H
Can
you
please
correct
me.
E
Yes,
actually
it
was
like
that
and
to
be
clear
on
that,
we
have
two
boundaries
that
we
can
enforce
where
at
first
we
can,
for
example,
impose
a
certain
coverage
decrease
in
the
ndpr
buyer
percentage
that
we
want,
for
example,
if
we,
if
the
coverage
decreases
by
two
percent
of
this
pr,
it
should
fail,
and
also
we
have
the
other
boundary
that
in
general,
if
the
coverage
below
below
a
certain
point,
then
then
it
can
fail.
E
So
we
have
those
two
options
to
to
enforce
those
rules,
but
yeah,
like
you,
said
edward,
it
felt
be
below
70
percent,
and
what
we
did
at
the
moment
is
that
we
just
load
the
the
percentage
to
to
to
give
some
room
for
discussion
and
to
decide
what
we
exactly
want
to
do.
C
H
H
Over
something
like
a
number,
I
don't
know
what
is
the
number
like?
If
we
see
that
we
are
at
70,
then
maybe
it
should
be
60.
I
don't
know,
and
if
we
reach
this
this
number
again,
it
means
that
we
need
to
stop
and
and
check
what's
going
on,
because
we
we
are
dropping
coverage
too
much
and
and
it
must
be
increased,
so
something
like
that
or
maybe
this
needs
to
be
monitored
in
the
long
run.
E
E
I
just
wanted
to
to
put
this
into
discussion,
but
I
would
be
fine
with
something
like
let
let
the
general
decrease
probably
not
be
minus
two
percent
or
something
in
pr,
because
I
think
that
would
be
some
sign
that
there
is
something
definitely
wrong
at
least
and
the
general
rule
would
be
okay.
So
we
we
drop
the
lower
boundary
altogether
and
just
let
the
reviewers
decide.
H
Yeah
and
maybe
just
take
an
action
to
investigate
why
it
dropped
so
much
in
the
last
the
last
month
or
something
we
maybe.
C
E
I
In
my
opinion,
I
have
to
say
that
if
a
reviewer
sees
that
pr
drops
coverage
above
two
percent,
then
I
mean
he
would
likely
not
approve
this
pr
and
see
that
something
is
wrong.
I'm
not
sure.
What's
the
value
of
not
allowing
and
forcing
that
vr
not
to
merge
in
in
such
circumstance,.
H
I
H
It
it
doesn't
fail.
If
you,
I
think,
no,
it
doesn't.
C
H
H
E
E
Okay,
so
if
you
don't
mind,
I
would
just
go
to
the
next
topic.
If
you
want
chandler,
please.
G
E
Okay,
thank
you.
So
another
thing
I
wanted
to
mention
was
that
we
sometimes
have
problems
in
reviewers
not
being
active
anymore,
because
they
have
left
a
project
for
some
reason
or
are
inactive
in
in
another
sense.
So
what
I
saw
what
kubernetes
does
is
that
it
looks
at
least
like
that
that
they,
every
year
or
start
of
the
year,
they
they
run
some
tool
and
to
clean
out
the
owner's
files
with
data
from
the
dev
stats
from
kubernetes,
and
I
was
thinking
about
that.
E
We
could
probably
do
the
same
so
that
there
would
be
some
automated
or
at
least
or
even
a
manual
run-off
of
this
tool
and
just
clean
out
the
the
owner's
files.
G
That's
we
could
one
thing
to
keep
in
mind
is
that
we
do
have
a
published
governance
doc,
which
does
touch
on
this.
So
if
we
decide
to
change
course
that
we'll
have
ramifications
that
need
to
be
actually
documented.
D
I
don't
think
we're
gonna
have
much
pushback
from
removing
people
from
the
reviewers
section
of
the
owner's
files.
I
think
that
we
have
some
policies
in
place
where
some
just
I
think
a
few
people
have
to
help
decide
for
the
owners.
Sorry,
the
approvers,
I'm
not
sure
exactly
what
the
policy
was
so
well.
I
guess
what
I'm
trying
to
say
is.
I
could
see
an
automated
policy
around
reviewers,
which
are
the
people
that
automatically
get
assigned
to
pr's,
so
we
could
do
something
like
if
people
within
the
reviewers
list.
D
I
haven't
done
a
review.
I
don't
know
in
three
months,
then
probably
they
aren't
doing
reviews
anymore
and
I
think
it
would
probably
make
sense
to
remove
them
in
an
automated
way
and
it's
easy
to
re-add
people
as
well,
but
the
approvers
that's
difficult
to
automate.
I
think,
because
that
that
has
some
different
implications.
E
Yeah,
that
makes
sense.
I
think
that-
and
I
am
exactly
exactly
trying
to
aim
at
this-
that
inactive
reviewers
get
at
some
point
somehow,
after
a
certain
period
of
inactivity,
just
removed
and
so
that
we
don't
have
people
that
are
inactive
assigned
reviews
and
that's
exactly
the
problem
that
we're
trying
to
solve
here.
Yes,.
D
Yeah,
I
I
don't
even
know
that
threshold
probably
should
be
just
a
couple
of
months
or
so.
If
and
then
we
could
add
them
back,
they
won't,
but
there's
a
there's,
it's
bad.
If
we
are
auto
assigning
people
who
are
inactive
to
review
things,
because
that
just
means
prs
start
to
pile
up
so
being
aggressive,
there
isn't
necessarily
a
bad
thing.
E
I
think
I
think
what
what
I
was
trying
to
solve
here
was
that
we
have
a
lot
of
ripples
under
the
qubit.org,
and
I
think
this
would
make
sense
to
to
do
this
for
all
repos.
So
at
least
I
think
so
because
I've
seen,
for
example,
not
only
in
cubic
cube
root,
but
also
in
kubernetes.
I,
for
example,
then
the
people
getting
assigned
who
are
no
longer
active
on
the
project
and
yeah.
So
so
I
think
this
would
make
sense
for
the
whole
cubicle.
D
Yeah,
I'm
not
sure,
certainly
the
repos,
you
just
pointed
out
make
a
lot
of
sense.
I
probably
need
to
be
reviewed
removed
from
keyboard
ci
and
like
project
infra,
because
I
don't
do
a
whole
lot
of
reviews
there,
but
I
think
I
still
get
assigned
sometimes
so
those
definitely
make
sense.
D
There
are
some
repos
in
the
keyboard
or
that,
maybe
just
aren't
very
active.
I
mean
there's
a
lot
of
projects
in
there
now,
so
it's
possible
or
foreseeable
that
there's
just
not
a
lot
of
reviews
going
on
at
all
on
some
repo.
So
if
we
had
a
policy
where
people
got
removed
after
not
reviewing
it
after
so
much
time
or
whatever,
it
might
just
automatically
remove
everyone,
because
there
was
no
pr's
for
a
few
months,
I
don't
know
how
so
maybe
it's
a
repo
by
repo
sort
of
opt-in.
D
H
Maybe
it
makes
sense
just
to
send
an
email
to
them
to
the
maintenance
warning
about
this
and
then
asking
them
to
take
action
because
to
nominate
an
approver,
you
need,
you
need
approval,
I
need
maintenance
so
to
remove
one.
I
think
you
also
need
the
same.
It
doesn't
make
sense
that
someone
will
automatically
remove
someone.
E
No,
I
think,
I
think
we're
not
talking
about
approvals,
we're
just
talking
about
reviewers.
At
this
moment,
right
approvers
shouldn't
get
touched.
I
think,
because
we're
just
talking
about
that
that
people
that
are
inactive
are
assigned
actively
reviews
because
they
are
in
the
reviewers
list
and,
like,
like
david,
said
this:
let's,
let's
pile
up,
prs
and
yeah,
I
think
this
doesn't
make
that
much
sense.
So
someone
else
should
the
people
that
created
bi
could
get
a
fair
chance
of
of
getting
reviewed
their
pr
within
time.
H
H
So
so
this
is,
this
mechanism
doesn't
really
work.
I
mean
I
need
to
pick
them
my
myself
and
that's
it.
It
will
probably
work
if
we
have
owner
fries
under
specific
photos,
because
then
it
will
be
more
focused.
E
No,
I
I
completely
understand
what
you're
saying
and
I
have
the
same
problem
most
of
the
time,
but
this
is.
I
think
this
is
a
different
problem
from
from
that
one,
but
I'm
I
was
thinking
about
at
least
but
yeah.
This
is
a
different
problem
that
also
needs
to
get
so
somehow.
Maybe
this
is
also
something
that
we
should
talk
about.
J
Hi,
this
is
something
really
that
I
want
to
bring
up.
I
put
it
in
the
chat
already
it's
the
fact
that
reviewer
is
actually
actually
tied
to
branches.
So
if
you're
in
the
owner
file
of
a
branch,
you
can
still
be
assigned
as
a
reviewer
for
that
range,
I.e,
backboards,
even
if
you're
removed
from
main-
and
I
think
it's
wrong.
I
think
the
owner's
file
from
main
should
be
used
for
every
branch.
K
D
It's
interesting,
I
I
don't
know
what
to
think
like.
I
don't
know
if
any
action
needs
to
take
place
because
of
that,
but
that's
it's
a
good
thing
to
understand.
At
least.
D
A
So
okay
sounds
like
the
conversation's
kind
of
slowed
down
on
that.
Do
we
want
to
move
on
to
the
next
item.
E
Yeah,
I
think
so.
I
think
I
think
that
that
I
have
everything
regarding
opinions
on
that,
and
just
just
my
my
conclusion
from
that
is
that
I'm
going
to
create
the
r
for
a
couple
of
repos
and
ping,
the
maintainers
of
the
repos
and
just
ask
them
whether
we
would
be
good
to
go
with
that.
What
I
bring
up.
A
Okay,
so
the
the
next
item
is
one
that
I
put
in,
I
kind
of
mentioned
this
at
the
top,
but
before
I
started
recording,
so
I
want
to
talk
again,
we
want
to
make
sure
that
everybody
knows
about
the
kubert
summit
coming
up.
It's
in
february
mid
mid
february.
A
It's
a
two-day
virtual
event
and
though
the
call
for
session
proposals
has
closed
as
of
this
previous
weekend,
we
wouldn't
mind
seeing
another.
You
know
session
proposal
or
two.
We
have
enough.
We
think,
but
it's
it's
kind
of
on
the
the
envelope
of
being
enough.
So
if
anybody
else
is
especially
somebody
from
like
a
newer,
maybe
newer
organization
that
is
joining
the
the
cooperate
effort
wants
to
put
in
the
session
proposal,
you
can
reach
out
to
us
directly
either
through
the
dev
mailing
list,
or
you
can
send
it
directly
to
me.
A
So
next
edward.
H
L
D
Yeah
in
general,
we
have
a
guide
when
it
comes
to
our
api.
I
believe
so
backwards
and
compatible
changes
would
be
removing
a
field
from
the
api
or
modifying
the
name
of
a
value
within
the
api.
D
I
think
it
makes
sense
to
maybe
at
least
begin
documenting
what
we
know
causes
backwards
or
even
for
its
compatibility
issues.
H
Like
the
minimum
is
here,
I,
in
my
opinion,
the
menu
is
what
what
are
the
options
that
can
be
deployed
so
in
the
deployment
of
the
upgrade
that
we
can.
For
example,
I
think
we
had
already
this
previously
discussed
that
handlers
can
be
of
a
different
version
at
some
at
a
specific
point
of
time
and
the
controller
and
the
handle
can
be
different
version,
so
it
causes
the
launchers
to
be
of
different
versions
for
different
handlers.
So
that's
this
kind
of
thing.
Yes,.
D
D
Yeah
that
might
be
or
orthogonal
to
the
backwards
compatibility.
Well,
it
is
related
right
because
it
defines
when
we
are
allowing
backwards
and
compatibility
changes
to
occur.
So
it's
the
process
leading
up
to
a
controlled,
controlled
breakage
of
that
sort
of
thing.
H
Sorry
I
had
yeah,
so
we
are.
We
have
a
current
feature
at
the
moment
that
is
affected
by
this.
So
what
we,
what.
H
D
Yeah
makes
sense,
what's
the
what's
the
pr
or
feature
that
you're
kind
of
using
as
the
catalyst
for
needing
this
policy.
Just
out
of
curiosity.
H
It's
I
think
it's
the
the
one
that
it's
about
the
srv
hot
plug.
H
D
Got
it
yeah?
That's
that's
a
really
good
example,
so
I
imagine
that
it's
possible.
I
I
don't
remember
all
the
context
of
this
pr,
but
I
remember
it
now
that
we
have
to
carry
some
old
logic
with
us
for
a
certain
amount
of
time.
Is
that
the
case
here
and
then
we
want
to
be
able
to
to
remove
that
in
the
future.
H
I
think
it's
it's
that
currently
there
is
a
mechanism
that
does
the
hot
plug
as
part
of
the
migration
flow
and,
and-
and
we
are
we
are
turning
into
this
reconcile
loop
and
there
is
the
there
is.
There
are
changes
in
the
communication
between
the
defender
and
bit
launcher,
but
we
at
least
there
is
consequences
in
the,
for
example,
if
there
is
a
new
built-in
old,
build
launcher
or
new
vehicle
launcher
and
hop
on
will
handle
getting
it
matters
here,
because
you
added
the
new
grpc
come
on.
D
Got
it
so
eventually,
what
we
would
like
is
to
depreciate
or
deprecate
duplicate
the
old
behavior,
where
it
was
handled,
probably
like
burt
launcher
or
something
the
hot
plug,
but
there's
a
certain
amount
of
time
that
we
have
to
wait
for
everyone
to
update
and
for
these
apis
with
invert
launcher
to
be
available
for
the
hot
plug,
and
things
like
that
so
yeah
this.
This
policy
would
define.
I
guess
that
timeline
for
when
we
can
remove
that
logic.
D
H
Yeah,
well,
I'm
still
optimistic
that
my
metrics
in
the
end
will
show
everything
is
green,
but
but
yes,
it
may
reach
this
point
that
we
are.
We
need
to
wait
for
support
both
or
something
like
that.
A
E
H
I
Yeah
hi,
so
this
is
an
issue.
I've
met
in
order
to
raise
a
discussion.
I
I
Also,
we
saw
that
the
cpu
utilization
for
these
two
replica
was
very,
very
high
all
through
the
this
period,
so
I
thought
to
basically
leverage
kubernetes
horizontal
pod
autoscaler
in
order
to
auto
scale
these
components.
We
can
start
with
the
virt
api
and
we
can
base
it
on
cpu
usage,
which
would
be
the
easiest,
and
I
just
wanted
to
raise
attention
on
that
and
maybe
raise
the
discussion.
D
So
this
is
the
horizontal
part
out
of
the
scalar,
so
this
means
we
would
be
creating
more
replicas
of
bird
api
to
handle
the
increased
load
from
the
validation
and
mutation
webhooks.
D
On
this
is
to
understand
where
the
scalability
problems
are
actually
occurring
within
our
controller,
so
I
think
before
I
would
look
at
horizontally
scaling.
I
would
look
at
if
there
are
any
really
easy
efficiency
winds
that
we
can
have
within
our
our
component
there.
So
maybe.
D
Something
super
inefficient.
I
I
don't
know
I'd
like
to
understand
that
before
maybe
moving
on
to
something
more
complex
that
might
even
potentially
be
hiding
the
issue.
So
if
we
can
create
more
instances-
and
we
are
just
kind
of
scaling-
an
inefficient
component-
that's
not
necessarily
great,
but
if
it's
in
fact
that
you
know
we're
as
efficient
as
we
can
realistically
or
practically
be
then
yeah
looking
into
horizontally
scaling
would
would
not
hurt.
G
One
thing
that
comes
to
mind
along
what
you're
just
saying
david:
the
lines
of
that
when
you
look
at
a
average
controller
and
there's
a
lot
of
contention
going
on,
we
do
log
a
lot
of
retries
due
to
the
pattern
we
use
of
you
know,
grab
objects
fiddle
with
it
and
issue
an
update
as
opposed
to
patching.
I
know
that
we've
got.
D
I
wonder
so
specifically
we're
talking
about
the
vert
api
component
here
when
we're
doing
the
retries
in
our
revert
controller
loop,
I'm
unsure,
if,
if
there's
a
collision
in
that
update,
if
it
actually
makes
it
to
our
api
or
not
or
if
it's
caught
before
our
api
server
would
see
it.
D
I'm
really
surprised
that
we
would
need
to
go
beyond
two
replicas
for
vert
api
once
we
start
introducing
console
and
vnc
access
or
anything
that
has
a
persistent
connection,
that's
being
forwarded
from
bird
api
to
controller
to
vert,
launcher
pod.
D
That's
when
things
get
kind
of
hairy
with
performance,
I've
seen
so
definitely
in
that
scenario
we
would
want
to
have
the
ability
to
scale
api
for
apis
under
normal
operation,
where
that
isn't
like
a
huge
burden.
If
we
have
problems
with
two
replicas,
I'm
really
curious
where
the
problems
are
like.
If
we
could
do
some
profiling,
for
example,
that
would
be
really
interesting.
We
have
a
prof
profiler.
D
D
I
would
be
interested
to
see
when
we
do
one
of
these
load
tests
or
something
like
this,
where
we're
spending
the
most
time
invert
api,
when
this
is
occurring
to
give
us
an
idea
of
this
is
something
that
we
could
improve
or
not
improve,
and
maybe
that's
in
addition
to
this
discussion
about
horizontally
scaling,
because
it
makes
sense
for
people
to
need
to
horizontally
scale.
G
Environment
yeah.
I
was
just
about
to
touch
on
that
one
of
the
concerns
I
have
with
doing
a
horizontal
pod,
auto
scaler,
based
on
cpu
usages.
It
might
be
too
late
if
you
hit
the
api
server
with
400
concurrent
start
vm
requests.
The
damage
is
done
by
the
time,
you're
actually
rolling
out
new
replicas.
I
think-
and
so
it
might
be
something
where,
if
we
had
a
way
to
be
smarter
ahead
of
time
about
anticipating
that,
we
could
need
that
many
api
replicas
to
have
them
ready.
D
You're
you're
right,
you're
right
and
I'm
looking
at
this,
this
bugzilla
that's
attached
to
it.
D
D
Yeah,
it's
it's
unclear
exactly
where
it's
breaking
down
here,
so
our
web
hook
contacts
yeah,
it
is
our
web
hook.
That's
timing
out.
I
And
by
the
way,
I
I
totally
agree
that
we
need
to
further
investigate
that
that
we
don't
want
to
hide
any
inefficiencies,
but
I
do
think
that
this
have
some
kind
of
a
limit
right.
So
if,
if
we're
using
like
a
thousand
nodes
cluster,
I
guess
the
two
replicas
won't.
Do
it
even
if
we're
very
efficient.
But
I
also
agree
that
we
can
expose
the
knobs
to
hco
or
something
like
that.
I
Instead
of
doing
this
automatically,
although
in
in
the
horizontal
part
of
the
scalar,
we
can
have
a
maximum
and
minimum
replica
count
so
yeah,
but
either
way
either.
D
Yeah,
I'd
definitely
like
to
understand
this
issue
a
little
bit
better,
the
the
bugzilla,
because
there's
there's
potentially
things
that
aren't
even
related
to
cpu
and
memory
here.
So
when
our
web
hook
is
hit
with
an
update,
a
vm
or
a
vmi
that
can
cause
a
chain
reaction
with
our
api.
Like
kubernetes
api
access,
we
can
say
an
update
is
occurring
on
vmi
as
a
result
of
that,
we
need
to
go
off
and
gather
some
information
about
the
larger
system
as
a
whole.
D
D
So
it
could
be
that
we
are
actually
timing
out
not
because
of
cpu
and
memory
usage,
but
because
the
kubernetes
api
server
itself
is
throttling
us,
because
we're
making
two
mirror
requests,
or
things
like
that.
So
something
to
understand
a
little
bit
more
fully
before
we
look
at
an
unmanned
solution.
D
It's
also
interesting,
so
I've
seen
a
lot
of
performance
tests
done
and
this
performance
test
is
different
and
then
it
uses
persistent
storage
where
previous
performance
tests
have
used
ephemeral,
storage,
so
not
using
pvcs.
D
I
wonder
I
have
a
feeling
it
might
have
to
do
with
the
fact
we're
using
pvcs
and
that
maybe
our
web
hook
is
doing
something
inefficient.
Once
pvcs
are
involved
like
introspecting
pvcs,
maybe
it's
not
using
an
informer
when
it
should,
or
maybe
it's
difficult
to
use
an
informer
there
we're
having
to
do
a
live,
get
requests
unsure.
A
So
I
made
an
attempt
to
capture
that
in
a
pithy
note,
so
so
I
heard
the
performance,
profiling
and
check
on
inefficiencies
in
the
components
before
jumping
to
auto
scaling
as
a
recommendation.
D
C
Yes,
let
me
explain:
here's
andra
from
ddesk
how
we
approach
the
same
issue
you
having.
We
have
a
central
api
that
controls
all
the
load
and
then
starts
to
control
in
every
one
of
the
hundred
clusters.
We
have
the
load
not
only
in
this
specific
cluster,
but
across
all
clusters,
because
these
clusters
are
also
across
data
centers
and
for
you
understand
what
we
are
trying
to
achieve.
C
It's
high
availability
and
also
auto
scaling.
It's
something
that
we
are
approaching.
We
are
not
counting
400
percent
of
auto
scaling
of
kubernetes
itself.
We
are
also
controlling
the
load
that
are
coming
to
the
cluster
and
if
something
goes
wrong
with
a
specific
cluster,
we
can
spread
all
over
several
clusters.
B
C
We
have
an
api
that
controls
the
load
and
can,
let's
say,
load
balancing
across
clusters:
okay,
because
a
single
cluster
doesn't
fit
our
needs.
We
have
100
clusters
with
up
to
1250
nodes
in
each
cluster.
C
C
I
can
send
you
a
send
a
link
of
exactly
what
we
are
talking
about
here,
just
one
second,.
C
Yeah
just
put
the
link
here:
okay,
it's
on
the
chat.
C
C
A
Okay,
thank
you.
Yeah
thanks
everybody
for
attending
we'll
go
ahead
and
wrap
it
up.