►
From YouTube: sig-testing weekly
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
B
C
A
Yeah
I
so
I
did
some
work
for
apple
at
some
point
in
the
for
the
retail
stores
and
we
had
to.
They
gave
us
like
a
small
window,
that
we
could
ship
files
to
stores
and
it
was
like
between
two
or
four
p
a.m
or
something.
But
that
means
something
different
in
every
single
time
zone
and
there's
like
there's
like
15
minute
time
zone
and
there's
like
a
leap,
30
minutes
and
yeah.
A
D
Okay,
pardon
the
lack
of
video
I'm
having
some
issues
with
zoom
the
if
we
don't
have
other
stuff
on
the
agenda.
I'd
like
to
discuss
the
working
group,
reliability,
work
proposal,
kind
of
moving
forward
on
reliability.
D
Let
me
okay,
sorry,
I've
been
trying
to
get
zoom
working.
Let
me
add
a
link
to
the
agenda.
A
There's
a
couple
things
that
came
out
of
some
of
the
the
leads
meetings
in
the
community
meeting
last
week.
We
need
to
get
some
onboarding
docs,
not
on
docks,
but
we
need
to
figure
out
onboarding
for
new
contributors
when
it
comes
to
like
prow
and
test
grid
in
general.
I
think
jordan
and
I
were
talking
about
maybe
recording
some
more
videos.
So
if
anyone
is
interested
or
has
ideas
around
that
feel
free
to,
let
us
know.
E
Yeah
we're
continuing
to
we're
planning
to
continue
adding
stuff
to
the
the
proud
documentation
website.
I
know
we
have
a
pr
out
right
now
with
a
detailed
diagram
of
the
functioning
of
tide,
so
we're
working
on
that.
That's
pretty
helpful.
That's.
A
Awesome
I
also
talked
to
sig
release
this
morning
and
the
ci
signal
team
is
also
working
on
their
onboarding
documentation
and
stuff.
So
yeah,
hopefully,
there's
a
good
effort
to
get
people
educated
on
how
this
stuff
works
better.
E
Yeah,
I
think
we
have
some
good
existing
resources
scattered
around
too
there's
like
a
job
cookbook
that
can
be
pretty
useful
to
people
trying
to
use
pro
yeah.
I
think
it's
probably
just
not
not
located
well
so
having
everything
on
the
site
should
make
it
a
lot
more
accessible.
A
D
Yeah,
the
zoom
client
is
not
working
properly
and
I'm
using
the
web
thing,
which
seems
to
have
hardware
problems
so
the
anyway.
We
want
to
move
on
to
the
reliability
discussion.
So
wg
reliability
came
up
with
a
proposal
for
how
to
handle
kubernetes
declining
reliability.
D
A
Okay:
let's
do
that
then
yeah!
So
just
reminder:
this
meeting
is
recorded.
Everything
you
say
will
be
on
the
the
internet.
This
meeting
in
all
kubernetes
and
cncf
meetings,
abide
by
the
cncf
code
of
conduct,
so
please
be
excellent
to
each
other.
D
Okay
cool,
so
we
have
this
kept
proposal
from
reliability
working
group
about
how
to
handle
kubernetes
at
least
lack
of
improvement
in
overall
reliability
over
the
last
couple
of
years.
D
The
as
you
see
from
the
comments,
I
have
some
disagreements
with
approach,
but
regardless
it
does
feel
that
that
clearly
we
as
a
project,
we
need
to
be
doing
something
to
produce
continuous
reliability
improvement,
which
is
not
really
a
thing
right
now,
and
so
I've
been
taking
it
around
to
the
community
meeting
and
to
some
of
the
key
sigs,
which
would
be
release.
D
Architecture
enhancements
in
now
testing
about
what
do
people
have
ideas
of
feasible
steps
that
we
could
take
that
would
produce
reliability,
improvement
over
time,
the
and
both
from
the
reliability
working
groups
proposal
and
for
anything
we
conceivably
would
do.
D
D
D
F
D
D
G
I'd
also
bring
up
that
we've
had
the
long-standing
triage
tool
that
lets
you
drill
into
like
where
test
failure
is
happening.
It's
pretty
powerful.
D
Yeah,
the
and,
and
it
it
may
be,
that
the
primary
problem
is
just
that
people
don't
know
how
to
use
the
tools
that
we
have,
in
which
case
it's
it's
more
of
an
education
campaign
which
would
be
great
honestly,
because
that's
something
that
contributor
experience
could
potentially
handle.
G
Yeah,
if
that
is
the
case,
we
have
some
pretty
excellent
content
that
jordan
put
together.
We
have
it
captured
on
a
doc
and
in
a
video
recording
on
like
how
to
triage
flaky
tests
using
these
tools.
D
Yeah,
okay,
the
so
education
program
there,
something
for
contributor
experience,
maybe
to
put
together
to
make
sure
that
everybody
knows
how
to
connect
the
dots,
but
then
the
other
question
is
you
know
this
sort
of
thing
of,
for
example,
not
not
having
overall
improvement
in
the
number
of
flaky
tests
or
and
and
having
kind
of
overall
decline
in
test
coverage.
D
This
is
not
a
new
problem
for
for
sig
testing.
This
is
a
problem
that,
as
far
as
I
know,
we
have
always
had
as
long
as
we
have
had.
You
know
from
like
year,
two
of
the
project
and
and
seems
like
y'all
would
have
some
ideas
about
what
the
rest
of
the
project
should
be
doing
to
address.
This.
G
Well,
in
the
past
we've
instrumented
coverage
and
we
found
that
we
didn't
have
much
success,
getting
anyone
to
look
at
it.
So
at
this
point
I
think
some
of
our
coverage
tracking
is
just
not
even
fully
functional,
because
there's
no
ins
there's,
I
think,
there's
insufficient
incentive
to
care
about
this
right.
Even
like
the
unit
tests.
We
don't
really
keep
that
dashboard.
Functional
there's
been
some
specific
focus
on
conformance
coverage.
G
D
If
somebody
else
contributed
experience
steering
whoever
was
taking
on
the
hey,
we
will
make
the
rest
of
the
project
care
about
this
or
working
group
reliability.
How
difficult
would
it
be
to
get
those
dashboards
working
again.
G
G
Would
work
we've
implemented
this
in
some
other
places,
or
my
team
has
where
we
basically
have
a
post
submit
record
the
coverage
amount
and
then
you
can
have
the
pre-submit
check
against
the
recorded
amount
and
fail
if
it
dips,
I'm
not
a
huge
fan
of
these.
I
think
it
sort
of
just
drives
checkbox
coverage
like
okay.
I
got
something
to
exercise
the
code
path.
Now
the
coverage
number
will
pass
doesn't
mean
you
actually
have
like
good
comprehensive
tests,
but
it's
something
we
could
consider.
D
Yeah,
because
one
of
the
things
that
was
actually
raised
during
the
community
discussion
was
people
were
suggesting
saying
that
one
of
the
sources
of
problems
is
that
enhancement.
Submitters
are
often
allowed
to
postpone
submitting
e-to-e-tests
and,
of
course,
postponement
sometimes
means
postponement
forever
yeah.
So
I
think
I
think
there
might
be
some
appetite
for
having
more
stringent
requirements.
G
I
think
that
reviewers
checking
for
like
their
concept
is
sufficient
test
coverage
makes
more
sense,
also
say.
Unit
test
coverage
is
probably
like.
We're
actually
think
pretty
good,
where
we
can
be
good
and
there's
other
things
that
just
don't
really
unit
test
and
for
e2e
coverage.
It's
not
super
viable.
Stick
that
in
pre-submit,
because
I
mean
the
performance
penalty
of
instrumenting,
everything
is
kind
of
high
and
it
it
doesn't
directly
correlate.
There's
some
noise
due
to
the
exact
behaviors
that
happen
and
things
like
that,
yeah,
the
it's
useful.
G
That
would
be
the
number
that
I'm
probably
more
concerned
about
myself
would
be
that
we're
exercising
everything
in
e
and
that
as
well,
we're
not
going
to
run
all
of
the
e-s
in
a
given
job,
there's
just
too
many
of
them
it'll
take
too
long.
G
D
Okay,
okay,
I
yeah.
I
was
just
you
know
the
grand
ass
there
yeah.
G
D
You
know
it's
kind
of
all
out
there
right.
We
have
the
visible
parts
of
lack
of
improvement
and
reliability,
which,
in
in
my
experience,
is
primarily
a
flaky
tests,
test
jobs
and
b
upgrade
downgrade
failures,
which
we
see
both
in
our
testing
and
in
the
field.
D
D
G
D
The
well
I
was
going
to
say:
if,
if
downgrade
doesn't
work,
then
then
upgrade
needs
to
work
flawlessly
right.
G
G
Few
folks,
like
me,
are
doing
this
in
our
very
very
spare
time
and
it's
just
not
sustainable,
and
it's
not
great,
tooling
yeah.
I
so
things
like
upgrade
like
the
folks
that
are
approving
the
code
right
now,
really
don't
have
time
to
be
fixing
upgrades
or
there's
no
one
responsible
for
upgrade,
and
often
the
problem
is
actually
with
the
like
upgrading
and
not
necessarily
like
one
of
the
components
and
it's
hard
to
tell.
D
D
Okay,
but
you'd
you'd
agree
with
my
assessment
that
that
one
of,
like
our
sort
of
super
critical
areas,
is
a
kind
of
lack
of
meaningful,
upgrade
testing.
H
D
Speaking
is
speaking
as
somebody
who
works
for
a
vendor.
I
would
identify
this
as
a
critical
area.
H
G
Catching
up
there,
which
works
for
them
but
means
we're
catching
things
late
typically,
so
I
would
add,
I
think,
from
my
point
of
view:
we
have
these
lengthy
release
cycles
with
a
freeze
period
and
we
tag
all
these
betas
and
alphas,
but
I
don't
think
anyone's
running
no.
Those
builds.
D
No
way,
one
of
the
one
of
the
the
to
do's
that's
been
kind
of
perpetually
kept
alive
from
the
bot
since
I
was
released,
lead
actually
almost
two
years
ago
now
is
make
it
easier
for
the
public
to
consume
alphas
and
betas.
G
When
their
tests
are
failing-
because
I
thought
that
was
the
point
of
code
freeze-
is
that
we're
not
like
we're
not
releasing
until
the
tests
are
passing
yeah
necessarily
good
enough
to
be
getting
things
like.
We
have
the
release
blocking
dashboard.
Maybe
more
things
need
to
be
in
release
blocking.
D
Yeah
well,
we've
gone
in
release.
We've
done
this
thing
for
a
number
of
years,
where
we've
said
hey.
If
this
test
is
going
to
be
in
release
blocking,
it
needs
to
be
not
so
flaky
and
you
know
taking
the
job
back
to
the
sig
that
owns
the
job
and
the
saying
we're
not
going
to
fix
it.
D
H
D
I
you,
you
can
see
my
arguments
in
other
people's
as
to
why
focusing
on
this,
as
we're
going
to
block
us
enhancements
is,
is
going
to
be
counterproductive,
which
is
my
opinion,
the
so
the
I
mean
I
do
think
you
know
watching
talking
to
some
of
the
current
release.
Team
ci
people
and
that
sort
of
thing
is
the
release.
Team
has
really
been
led
to
the
idea
that
it's
their
job
to
make
the
test
pass
or
fudge.
D
Instead
of
blocking
the
release,
I
mean,
if
I
compare
to
how
we
behaved
in
the
release
team,
you
know
two
or
more
years
ago
versus
you
know
how
the
release
is
run
now.
Is
the
release
team
has
been
told
to
prioritize
the
timeline
overpassing
the
tests
and
and
I'm
not
sure
that
that
was
a
good
decision
and
the
decision
could
be
revisited.
G
H
Say,
as
the
relatively
new
guy
here,
you've
got
tests
that
are
supposed
to
be
blocking
your
release
that
you're,
taking
out
of
the
way
that
you're
not
gonna,
fix
right
and
then
I
I
kind
of
lost
my
train.
I
thought,
but
it
essentially,
it
sounds
like
some
of
the,
and
that
was
the
other
thing.
Is
that
the
release
the
release
team
was
saying
that
it
was
their
job
to
fudge
or
or
fix
tests
to
make
them
pass,
and
we
started
the
conversation
with
there's
a
perception
that
there's
reliability
issues.
G
I
personally
am
not
sure
I've
seen
like
fudging,
but
I
do
feel
like
they
feel
responsible
for
getting
the
test
to
pass
and
have
done
quite
a
bit
of
chasing
folks
around
trying
to
get
things
fixed
and
it
doesn't
seem
super
sustainable
and
but
like
even
stepping
further
back.
There's
only
so
much
that
we're
actually
blocking
on
to
begin
with,
and
something
like
upgrade
is
not
one
of
those
things.
D
A
G
G
You
need
to
keep
your
job
very
stable
for
a
like
period
of
time
to
show
that
it
is
a
reliable
signal
and
that
the
job
that
the
job
can
be
stable
and
that,
if
it
is
unstable,
we
must
have
had
a
bad
change
and,
as
someone
who's
brought
some
jobs
through
that
it's
really
time
consuming
like
one
of
the
main
things
I'm
doing
when
I'm
getting
the
kind
project
ready
to
be
able
to
be
used
in
rci
is
just
trying
to
create
the
perception
that,
like
this
is
actually
a
reliable
way
to
test,
and
I
have
to
go
running
around
fixing
things,
because
until
it's
a
blocking
job,
people
make
changes
that
subtly
break
the
job,
and
you
have
to
go
through
our
huge
amount
of
changes
and
find
what
broke
it,
get
it
fixed.
G
And
if
you
don't
fix
that
fast
enough,
you
don't
have
a
green
period
on
your
dashboard
that
you
can
point
to
so
you
have
to.
I
would
say
that
you
can
get
that
viable.
You
have
to
really
get
to
a
point
where
you
have
multiple
people
paying
attention
to
this
and
starting
to
already
treat
it
as
sort
of
pseudoblocking
like
if
it
went
red
something's
wrong.
We
need
to
fix
it
and
get
it
stable,
get
it
into
somewhat
blocking.
If
you
have
this
purely
in
post
emit.
G
That
can
be
extra
hard,
because
there's
no
signal
on
a
pr
that,
like
this
was
gonna
break.
This
thing
we've
had
examples
of
this
before,
where
things
that
should
be
totally
viable,
get
knocked
out
and
never
come
back,
we
used
to
test
with
chaops
on
aws
and
pre-submit,
but
we
ran
into
the
billing
account
being
suspended
there
and
it
has
never
recovered.
Since
it
was
a
billing
issue,
it
wasn't
that
it
stopped
functioning.
It
was
we
weren't
able
to
run
ci
because
the
account
wasn't
paid
up
and
once
the
account
was
paid
up
again.
G
We
I
mean
that
it
has
never
come
back
and
we
don't
have
that
coverage
now.
I
think
we're
moving
in
a
direction
where
that's
not
the
sort
of
thing
we
want
we're,
not
trying
to
test
like
on
more
cloud
providers
or
something
like
that
that
isn't
necessarily
what
we
want
in
pre-submit
to
begin
with.
Maybe
we
want
to
get
away
from
all
of
them,
but
it's
an
example
of
like
everything
involved
should
have
been
totally
stable,
but
once
you
lose
being
that
blocking
thing,
the
momentum
to
get
back
in
blocking
is
high.
D
Yeah,
yeah
and
well-
and
the
thing
is
that
you
know
when
we
started
out
with
the
release
team.
We
we
had
the
opposite
right,
which
is
we
had
lots
of
tests
that
would
kind
of
fail
randomly
and
and
their
failures
didn't
mean
anything,
and
so
you
had
the
release
team
scrambling.
D
You
know
on
a
pretty
much
daily
basis,
to
figure
out
whether
the
test
failure
was
meaningful
or
not,
which
would
then
obscure
the
test
failures
that
actually
were
meaningful,
the
so
but
yeah
yeah,
there's
not
a
lot
of
motivation
for
sigs
to
promote
their
own
tests,
because
it's
a
lot
of
it's
a
lot
of
work
and
it
doesn't
solve
a
problem
for
them
necessarily
at
least
not
right.
Now,.
G
Yeah,
there's
a
very
high,
very
high
bar
motivation
there.
It
has
to
be
something
where
you
know
whoever's
paying
you
to
spend
time
on
this,
presumably
because
it
costs
a
really
large
amount
of
time.
I'd
be
surprised
to
see
purely
volunteers.
Doing
these
you
have
to
convince
them
that
you
need
this
to
be
tested
and
released.
Blocking
upstream
I've
known
a
few
examples
of
my
employer.
They've
happened,
but
it's
not
very
common,
because
a
lot
of
times,
like
maybe
just
having
some
test
signal
that
you
can
monitor,
is
good
enough.
G
D
The
I
wanted
to
answer
eddie's
question
by
explaining,
like
one
case
of
where
a
test
job
becomes,
we
won't
fix
it.
That
happened
when
I
was
doing
a
ci
signal
for
the
release
team.
So
this
was
a
while
ago
again
a
couple
years
ago.
D
You
would
have
to
ask
somebody
who's
currently
on
csignal
for
any
contemporary
examples
of
this,
but
really
what
it
comes
to,
and
this
comes
down
to
the
we
used
to
have
both
upgrade
and
downgrade
tests
and
a
whole
battery
of
upgrade
tests
which
still
exist
within
sig
cluster
life
cycles,
test
grid
the,
and
what
happened
over
time
was
that
sig
cluster
life
cycle,
sid
custer
life,
cycle's
pool
of
contributors
became
identical
to
the
kube
admin
contributors
as
in
there
was
not
people
who
were
part
of
sid
cluster
life
cycle
who
didn't
work
on
kubernetes,
and
so
we
had
a
bunch
of
upgrade
tests
that
were
based
largely
on
chaops
and
they
belong
to
sig
cluster
life
cycle
because
upgrade
tests
and
sid
cluster
lifecycle
said
this
is
chaops
we're
not
going
to
maintain
or
troubleshoot
these
anymore,
because
we
don't
care
about
chaops.
D
But
their
own,
so
we
said:
okay,
we'll
swap
in
the
cubid
min,
upgrade
test
right,
because
you
want
to
maintain
coopermen,
which
we
did
for
a
while.
But
it
turns
out
that
with
cluster
life
cycles
workflow
they
were
okay
for
those
with
those
tests
going
red
for
weeks
at
a
time
which
was
obviously
not
acceptable.
If
they're
going
to
be
master
block.
D
So
we
ended
up
without
upgrade
tests
at
all
in
in
master
blocking
the
I
mean,
that's
an
example
of
sort
of
thing.
That
happens.
I
mean
often
it's
not
so
much.
You
take
a
job
that
is
clearly
owned
by
a
sig
to
that
sake,
and
they
say
we're
not
going
to
fix
it
a
lot
of
times
what
happens?
Is
you
take
a
job
to
a
sig
that
ostensibly
owns
the
job
and
they
say
well.
G
It
for
the
gce
tests,
I
also
for
the
team
that
was
like
okay.
We
need
upgrade
tests
to
cover
this.
This
thing
that
we're
writing,
and
so
they
put
a
bunch
of
effort
into
getting
the
like
upgrade
upgrade
tests,
because
another
bit
of
thing
is
that
those
other
tests
we're
talking
about
that
there
isn't
any
special
upgrade
testing.
It's
test
upgrade
test
and
it's
just
the
usual
tests.
We
actually
had
some
tests
that
try
to
do
things
across
the
upgrade.
G
I
don't
have
a
application
running
or
something,
but
those
are
bad
from
early
in
the
project
not
very
worked
on
and
horribly
tied
in
tube,
like
the
gce
clusters
and
things
it
needs
to
be
able
to
talk
to
like
the
deployer
to
like
run
the
upgrade,
and
it's
not
aware
of
many,
though
getting
those
running
again
itself
has
some
unlimited
value,
because
no
one
wants
to
touch
those
tests,
and
so
the
activation
energy
to
get
good
upgrade
testing,
I
think,
is
even
further.
G
A
I
was
again
I
was
talking
to
jordan
last
week
and
like
we,
I
still
think
that
there's
an
onboarding
problem,
I
think
even
me
as
a
maintainer
and
lead
for
the
past
two
years,
still
struggle
when
I
have
to
debug
a
test
which
I
spent
yesterday
doing,
and
you
know
it's
not
a
knock
against
the
tools.
It's
not
a
against
our
documentation.
A
It's
just
that
like
like
take
the
terms,
for
example,
pre-submit
and
post,
submit
right,
like
those
mean
nothing
to
most
people
who
aren't
working
at
google
right
like
that,
is
not
a
term
I've
ever
had
in
any
of
the
six
companies.
I've
worked
at
before
right,
we've
called
it
something
different,
and
you
know
that's
just
like
little
things
like
that,
like
when
people
see
that
they're,
you
know
they
get
a
a
failing
crowd,
job
right.
Most
people
just
slash,
retest
and
move
on
right.
They
don't
know
what
to
do.
A
There's
no
actionable
steps
there,
other
than
hey,
run,
slash
retest
to
run
me
right.
So
it's
it's
a
bigger
problem
and
it's
definitely
the
the
lack
of
resources
and
people
that
we
have
to
put
on
making
it
better.
So
I
know
paris
is
putting
I
I
got
a
sneak
peek
of
a
deck
that
paris
is
writing
right
now
for
the
the
cncf
but
yeah
we
need.
We
need
more
dedicated
resources
and
individuals.
D
D
The
so
the
okay,
if
folks,
have
other
ideas,
observations
commentary
on
this.
Please
comment
either
on
the
cap
or
on
the
thread
I
opened
up
on.
D
D
So
that
your
thoughts
get
shared
with
the
rest
of
the
project
because,
like
I
think
one
of
the
reasons
why
sig
testing
is
under
resourced
is
because
people
expect
it
to
just
work
and
don't
really
think
about
the
people
who
are
in
this
sig,
which
which
happens
with
a
lot
of
things
right,
they're
focused
on
their
code
and
they're
like
well.
Somebody
else
is
taking
care
of
the
test.
H
The
no
that's
wrong
and
any
in
any
way
you
look
at
it.
It's
the
the
person
that
authored
the
test
to
the
test
belongs
to
them
forever
and
ever
and
ever.
H
They've,
been
down
from
the
test
should
go
with
the
code
and
the
code
should
go
with
the
test
and
they're
one
in
the
same,
and
they
should
never
part
the
one
of
the,
I
guess
general
observation
is
the
scope
of
the
group
is
into
the
like
the
space
of
like
governance
and
process
and
ownership
versus
just
straight
test
tool
development.
So
we,
but
we
built
these
tools,
so
you
can
measure
in
some
aspect
the
quality
of
your
of
your
code.
H
G
We
help
run
the
like
infrastructure
and
tools
for
the
test.
Okay,
yeah,
I
one
of
my
backbone
things
is,
it
would
just
be
so
expensive
to
do
but
someday.
I
would
love
to
see
this
sig
renamed.
Everyone
thinks
we
write
tess
and
nobody
here
wants
to
write
tests,
we're
not
going
to
fix
the.
G
We
have
tools
that
can
let
you
know
that,
but
you
like
you,
have
to
opt
into
being
alerted
we've
done
campaigns
to
get
cigs
to
set
up
like
an
email
group
that
gets
these
or
things
like
that,
but
no,
the
main
group
that
actually
actively
tracks
that
is,
there's
a
subgroup
of
the
release
team.
Okay,
yeah
signal
group
that
is
focused
on
just
the
release
blocking
signal.
Those
folks
will
actually
go
chase
people
down
trying
to
get
tests
fixed,
but
for
the
entire
project
at
large.
H
You
know
I
don't
I,
I
completely
think
you
should
not
do
that.
I
think
if,
if
and
I
guess
it-
and
I
also
don't
know-
is
there
like
a
central
execution
environment?
That
is
the
the
gatekeeper
for
all
of
this
that
this
all
runs
in
and
if
it
doesn't
pass
there,
it
didn't
pass
period.
G
Yeah,
so
the
vast
majority
of
of
ci's
run
on
our
own
homegrown
infrastructure
prow,
and
we
also
have
a
tool
test
grid
that
gives
you
display
of
the
your
results
and
it
can
email
you
on
failures.
We
and
we
have
some
of
the
ci
like
prevents
things
from
merging.
Okay.
G
But
now
I
like
that
it's
been
it's
there's
not
much
that
we're
really
using.
We
have
a
we,
so
I
worked
on
a
program
for
here
you
can
like
run
your
conformance
tests
and
upload
results
and
if
you
get
them
running
like
continuously
against
the
latest
kubernetes
code
and-
and
this
is
stable,
then
like
I'll
help,
you
push
like.
Let's
get
this
into
release
blocking
because
it
was
hard
to
get
even
like
our
base
conformance
sets
in
because
then
people
are
like.
Oh
well.
G
H
G
But
getting
on
third
party
to
pro
to
provide
like
reliable
enough
results,
has
not
been
very
successful.
G
I
think
it's
important
that
our
contributors
are
enabled
to
go
fix
these
things,
the
things
that
are
release
blocking.
That
does
actually
happen.
The
problem
is,
there's
a
lot
more.
That
probably
should
be
in
there
and
it's
hard
to
scale
that
up.
H
Right
but
well
it's
it's
it's
you're
kind
of
damned.
If
you
do
because,
then
you
leave
it
in
the
sorry,
I
shouldn't
use
bad
words
on
my
being
recorded,
but
anyway
you,
you
can
scale
a
little
bit
more
effectively
when
you
start
involving
third
parties
as
well
right
because
then
you're
not
it's
not
all
on
you
to
scale
up
every
single
platform.
That's
out
there.
G
Oh
sure,
but
I
mean,
as
a
community
third
parties
can
come,
help
us
scale
what
we
have
and
it's
when
we've
had
like
third
parties,
totally
run
their
own
thing
and
just
submit
results
yeah
they
haven't.
It
actually
hasn't
been
much
of
a
scale
up.
It's
usually
like
one
or
two
people
from
the
like
company
or
whatever,
and
then
they
kind
of
flake
out
on
us
and
we
lose
contact
and
we
don't.
A
Eddie,
I
just
wanted
to
give
a
quick.
I
don't
know
I
I
want
to
talk
through
what
I
worked
on
yesterday
right.
So
this
was
the
issue
it
started
failing
27
days
ago
right,
I
got
pinged
by
the
release
team.
I
don't
know
when's
the
first
comment
by
us
on
here
so
seven
days
before
we
noticed
it
right.
A
We
as
a
sig
are
not
looking
at
all
the
you
know,
tagged
issues,
it's
just
way
too
much
github
noise
right,
I
have
almost
all
github
emails,
muted
and
then
so
we
take
a
look
at
this.
We
dig
into
the
result
there's
a
problem
with
these,
the
the
page,
stable.txt
and
stable1,
so
those
are
for
the
current
stable,
commit
on
master
and
then
back
one
version.
A
So
this
would
be
123
branch
right,
so
those
weren't
there
they
didn't
get
created
by
some
release,
script
that
should
have
created
them
right
so
boom,
like
steven
augustus,
figured
that
out
once
it
got
to
him
like
what
is
that
six
days
later
and
then
another
seven
days
later
or
so,
we
got
pain
that
it
was
still
failing,
even
though
that
those
things
were
there-
and
this
is
where
I
came
in-
to
fix
this-
and
this
was
a
failure,
because
I'm
curious,
like
I'll,
tell
you
what
the
problem
was,
and
you
tell
me
where
the
hole
was
so
we
added
a
feature
to
124
that
service
account.
A
A
A
The
fix
here
was
to
back
port
a
test
change
from
124
to
123
that
didn't
wait
for
it
in
a
certain
way.
You
can
see
the
the
the
commit
that
I
backported
cherry
picked
there
right.
So
where
was
the
hole
here?
Where
should
this
have
been
caught
on
the
the
pre-submits?
The
post
submits,
not
master
blocking?
Was
it
bad
test.
G
I
I
think
the
problem
here
is
again
that,
on
the
whole,
the
community
isn't
paying
attention
to
upgrade
tests,
because
otherwise
we
probably
would
have
thought
we
had
to
make
this
test
change.
We
we
need
to
backport
it
like
when
we
made
the
change
regardless
of
ci,
but
it's
this
isn't
on
the
brain
and
yeah,
I
mean
we're
not
gonna,
be
able
to
put
everything
that's
in
release
blocking
into
pretty
cement.
G
Honestly,
we
have
too
much
and
it's
too
flaky
in
pre-submit
today
I'll
say
to
the
same
thing:
the
releasing
did
with
reducing
the
number.
There
is
a
lot
to
be
said
for
having
reliable
signal
and
less
of
it
just
because
that
means
that
people
pay
better
attention
to
failures
when
they're
expecting
things
to
pass,
but
in
like
pre-submit,
when
I,
when
I
send
a
kubernetes
pr,
I'm
just
like
tests
are
failing
again
stupid
tests,
like
I'm
not
really
thinking
did
I
break
something,
I'm
thinking
these
tests
aren't
reliable.
This
is
annoying.
H
G
G
But
you
know
no
one's
actually
doing
this
right
now
and
the
problem
with
doing
that
is
then
no
one's
gonna
go
look
at
where
we
do
run
all
the
flaky
tests
and
say,
oh,
but
I
can
fix
this
one
and
put
it
back
because,
like
why
is
that
your
problem?
It's
just,
I
think,
most
of
the
folks
that
would
want
to
do
something
like
this
are
just
under
a
deluge
of
things
to
deal
with
already
and
it's
hard
for
this
to
rate.
A
G
G
We
have
tests
that
test
before
and
after
we
have
some
tests
that
are
based
on
testing
around,
but
that's
in
very
limited
use,
I'm
not
sure
if
anything's
actually
running
today
in
ci
and
then
in
this
case
it's
a
cluster,
that's
running,
multiple
versions
which
isn't
strictly
an
upgrade,
but
is
like
one
of
the
main
motivations
for
testing.
That
is
that
you
know,
if
I
upgrade
a
node
is
there
is
my
cluster
is
still
going
to
work
yeah
like
so
I
can
have
a
rolling
upgrade
yeah.
G
This
has
been
a
really
useful
discussion,
but
I'm
looking
at
the
clock-
and
I
want
to
let
andrew
get
to
his
topic-
yeah.
D
I
Sure
so
yeah
I
wanted
to
spend
some
time
and
ask
about
the
current
state
of
cubetest
cubetest2,
and
did
you
see
a
cloud
provider
so
for
some
context?
I
As
many
of
you
know
we're,
you
know,
there's
this
long-running
effort
to
remove
the
entry
cloud
provider
code
and
I
think
we're
like
we're
at
a
point
where,
like
we've,
externalized,
everything
like
the
csi
plug-in
the
controller
manager,
the
credential
provider
and
the
cubelet
everything's
kind
of
externalized
and,
like
the
final
stretch,
to
really
convince
ourselves
that
we're
okay
to
remove
things
is
to
have
all
our
tests
running
with
the
externalized
version
of
the
cloud
providers
in
the
test,
jobs,
and
so-
and
this
is
kind
of
like
related
to
the
staffing
problems
right
like
I-
wouldn't
expect
anyone
in
sick
testing
to
like
help
or
take
on
the
work
of
like
converting
those
jobs
so
like.
I
I
think
it's
kind
of
on
our
sig
to
like
kind
of
leave
the
effort
and
like
see
what
we
can
do
to
slowly
convert
those
jobs.
I
So
yeah
like
I
want
to
be
mindful
of
staffing
problems
and
try
to
get
a
sense
of
like.
Where
would
be
a
good
place
to
start
to
slowly
convert
tests
to
use
external
cloud
providers
without
you
know
destabilizing
any
existing
jobs
or
anything
like
that,
and
I
did
notice
yeah
like
there
was
some
effort
around
keep
test
2
and
it
has
like
a
legacy
mode
and
a
standard
mode
to
use
external
provider.
But
I
wasn't
sure
what
the
recommended
approach
for
testing
external
cloud
providers
is
on
gce
at
least.
G
I
don't
know
if
we
can
a
whole
lot
about
that.
I
can
tell
you
cube
test.
2
should
be
in
a
pretty
stable
state,
now
it's
being
used
in
ci
for
e2
tests
and
should
be
fine,
but
that's
in
that's
one
of
those
projects
where
hi
I'm
like
the
only
active
approver
right
now.
I
it's
very
high
on
my
to-do
list
to
change
that,
but
even
still,
it's
probably
not
going
to
have
a
super
active
investment
and
there's
going
to
be
a
really
high
expense.
G
I
don't
think
it
the
scope
of
moving
all
of
the
things
that
use
the
entry
gce
provider
to
not
moving
them
is
enormous,
like
that's
most
of
our
ci,
and
it
also
raises
some
interesting
questions
because
we're
like
not
trying
to
test
this
out
of
tree
provider
or
the
provider
at
all
we're
just
trying
to
test
kubernetes
in
most
of
those,
and
I
think
we
get
to
cheat
today
on
the
stability
of
that
by
like
well
everything
blocks
on
it
and
it's
it's
in
the
kubernetes
repo.
G
So
it
it
better,
be
passing
it's
not
gonna
like
you,
can't
fundamentally
break
the
cloud
provider
and
merge
your
pr,
but
once
it's
out
of
tree
that
gets
closer
to
where
we've
been
with
cops,
where
folks
need
to
be
convinced
that
this
is
like
reliable,
that
there's
a
stable
version
that
we're
using
for
kubernetes,
ci
or
else
there's
going
to
be
a
push
to
remove
it
everywhere.
I
Because
you
so
the
I'm
fairly
confident
that
if
we
were
to
reconfigure
the
ci
jobs
to
not
use
an
external
cloud
provider,
but
just
like
configure
them
without
a
provider,
a
lot
of
the
tests
would
break
or
not
work.
There's
like.
G
So
we
actually
have
a
lot
of
tests
that
will
work.
Fine,
I
mean
kind,
doesn't
configure
a
a
cloud
provider
at
all,
but
there
are
tests
that
are
covering
things
that
only
work
with
that
and
then
there's
just
the
bulk
of
ci
is
running
on,
like
you
know,
a
cluster
using
disposable,
gc
evms,
and
so
we
we
just
need
the
cloud
provider
to
have
the
cluster
yeah
like
I,
I
kind
of
like
we
just
won't
even
have
a
cluster.
If
we
don't
have
a
provider.
I
So
I
guess
what
I'm
trying
to
do
is
like.
I
don't
want
to
be
in
a
state
where,
like
when
we
remove
the
cloud
providers,
we
also
disable
or
remove
a
bunch
of
tests,
and
so
I'm
trying
to
get
to
a
state
where,
like
when
we
get
there
eventually,
at
the
very
least
like
there
are
a
set
of
jobs
that
have
been
converted
to
user
external
providers.
So
those
tests
can
continue
to
run
as
opposed
to
like
just
turning
them
off.
G
I
guess
what
I'm
thinking
is,
so
I
think
most
of
the
tests
that
we
should
be
running
in
kubernetes
to
test
kubernetes
should
work
without
any
particular
provider,
except
for
the
part
that
we
need
clusters,
and
I
don't
think,
there's
been
enough
push
to
say,
like
oh
we're,
going
to
run
all
these
with
like
a
single
node,
cubatum
or
kind,
or
something
like
that,
where
we
don't
need
that.
As
soon
as
we
get
into
like
multiple
vms,
I
mean
we
need
an
act.
G
We
need
a
provider
just
so
that
we
can
like
bootstrap
a
cluster
together.
I
think
we
can
even
get
to
a
state
where
the
bulk
of
what
we're
covering
in
pre-submit,
if
we
set
aside
like
node,
which
is
with
different
unrelated,
will
work.
But
there
are
other
things
like
scale
testing,
where
absolutely
not
that
you're
never
going
to
have
that
or
it
doesn't
actually
care
about
the
provider.
G
But
it's
never
going
to
work
without
having
a
bunch
of
like
real
vms
running,
with
substantial
compute
capacity
that
we're
not
going
to
simulate
by
like
just
sticking
kind
on
a
huge
vm
or
something
and
like
you're,
making
things
yeah.
Those
things,
I
think
are
still
pretty
important
and
a
huge
portion
of
ci
like
even
if
we
were
like.
I
don't
care
about
that.
G
G
That's
like
our
default.
Our
default
choice
for
testing,
kubernetes
and
ci
is
to
create
a
gce
cluster
because
that's
where
we
have
lots
and
lots
and
lots
of
compute-
and
there
are
a
lot
of
things
where
you
know
we
haven't
made
this
huge
push
like.
Oh
we're,
just
gonna
run
them
all
like
kind
or
something
we
could
go
one
of
those
directions
like.
G
G
I
would
probably
start
by
picking
something
like
the
conformance
gce
suite
and
trying
to
make
sure
that
that's
running,
but
I
think
this
we're
gonna
have
to
have
conversations
beyond
even
like
r2
cigs,
for
example,
with
release
about,
like
you
know,
should
this
be
release
blocking?
G
What's
the
motivation
yeah
like
you're,
making
a
decision
there's
also
like
a
general
direction.
Question
like,
for
example,
in
pre-submit.
Like
do
you
know?
Do
we
want
providers.
G
And
if
so,
like
I
mean
do
we
want
to
go
back
to
having
multiple,
and
if
we
do,
I
mean
do
we
want
that
to
be
like
a
plugin
having
lots
of
them
or
or
do
we
if,
if
we
could,
I
mean
scale
is
a
question,
but
if
you
scale
we
could
head
back,
we
could
head
in
the
direction
of
just
like
no
providers
and
pre-submit
either
of
the
any
of
those
options
is
going
to
take.
G
I
think
a
pretty
large
investment,
and
if
this
is
one
of
the
things
I've
been
trying
to
vote
people
about
that,
I
think
is
going
to
be
the
hardest
part
of
actually
finishing
provider
extraction.
Is,
I
don't
think
we're
ready
for
there's
just
a
huge
amount
of
ci?
G
That's
just
like
I'm
just
going
to
use
clusterup.sh
and
create
a
gc
cluster
and
as
someone
who
has
gotten
out
of
tree
deployment
tool
to
like
pass
nci
it
it's
really
hard
to
get
to
the
point
where
people
like
trust
the
tool
and
keep
things
functional
and
don't
just
break
you
with
their
pr's.
And
then
it's
blocking
and
we
as
a
project
cheat
a
lot
by
having
the
gce
and
the
cluster
scripts
in
the
repo.
So
that,
if
you
change
anything
in
your
pr
like
you
will
wind
up
inevitably
fixing
those
like.
G
If
you
remove.
If
you
remove
a
flag
from
a
component
or
something
like
that,
you
have
to
change
those
scripts.
It
is
going
to
be
very
different
when
none
of
the
tools
live
in
the
repo
except
sort
of
cubanum.
For
now
and
there's
even
talk
of
moving
that
out.
I
Yeah,
so
I
think,
like
one
thing
I
want
to
make
clear,
is
I
am
kind
of
willing
to
like.
I
understand,
like
there's
a
ton
of
work
on
this.
I
am
willing
to
kind
of
roll
up
my
sleeves
and
like
just
just
go
into
it
and
start
doing
some
of
the
work,
but
I'm
trying
to
figure
out
like
the
best
place.
I
To
start
and
like
you
made
a
distinction
between,
depending
on
a
cluster
configured
with
a
provider
versus
like,
depending
on
the
test,
deployer
like
a
provider
for
the
test,
deployer,
and
it
feels
like
things
can
be
like
tackled
separately
or
maybe
like.
We
start
one
job
that
is
configured
with
an
external
provider.
G
G
I
don't
want
things
in
pre-submit
that
like
actually
care
about
having
like
a
gce
provider
or
something
like
that,
I
I
just
want
to
be
testing
kubernetes,
and
that
sounds
like
something
I
should
be
testing
in,
like
the
in
the
various
cloud
provider
repos
and
maybe
even
it's
a
common
thing-
they
all
need
to
test
and
we
should
have
a
common
test,
but
I
still
don't
like
care
about
that
blocking
my
ci,
but
there
are
the
other
things
where,
like
I
am
actually
just
testing
kubernetes
that
do
depend
on
it
and
even
for
those
just
the
question
of
like
which
one
are
we
using
and
like?
G
How
do
we
keep
it?
Stable
I've
had
some
meetings
about
this
in
the
past,
but
they
haven't
really
gone
anywhere.
I
can
dig
up
some
of
that
stuff
and
follow
up
with
you
on
that.
I
I
think
I
would
start
by.
G
I
believe
we
actually
already
have
ci
for
this
provider
doing
some
basic
tests
with
cube
test
two.
I
would
take
a
look
at
those
and
see
what
shape
they're
in
and
then
probably
next
step
is
like
I
mean:
where
do
we
want
those
like?
Do
we
want
to
get
that
into
like
release
blocking,
or
I
think
the
question
remains
about?
How
do
we
version
like,
like
what
version
are
we
using
and
right
now,
I
think
they're
using
it
out
of
tree
cluster
scripts?
I
Makes
sense?
Okay,
so
then
yeah.
That
sounds
like
a
good
next
step,
though,
like
looking
at
the
current
jobs
that
run
extra
providers.
If
you
can
send
me
a
link
to
those-
and
I
will
see
like
what
shape
they're
in
and
yeah,
I
think
that's
a
good
starting
point
is
just
stabilizing
those
and
seeing
if
they
actually
more
can
validate
what
we
want.