►
From YouTube: Kubernetes SIG Testing - 2020-12-01
Description
A
Hey
everybody
today
is
tuesday
december
1st.
This
is
the
kubernetes
sig
testing
bi-weekly
meeting.
I
am
your
host
aaron
of
sig
beard.
This
meeting
is
adhering.
We
adhere
to
the
kubernetes
code
of
conduct
in
this
meeting,
which
basically
means
we're
going
to
be
our
very
best
selves
and
be
kind
to
each
other.
A
A
B
A
It
out
sure
I
don't
have
it
handy
right
now,
but
yeah.
This
is
something
we're
pretty
interested
in.
I
think
I
feel
like
there's
a
lot
of
stuff
in
the
testing
for
repo,
that
it
would
be
helpful
to
try
and
separate
some
things
out.
So
we've
talked
about
the
idea
of,
for
example,
trying
to
move
proud
into
its
own
repo,
or
the
other
thought
was
moving
config
files
into
their
own
repo.
A
Some
of
those
things,
I
think,
are
kind
of
tangled
up
with
the
way
that
prow
auto
deploys
itself,
but
I'm
just
trying
to
describe.
I
think,
like
the
challenge
that
the
challenge
is
for,
why,
like
it,
it
hasn't
been
done
already
not
to
say
that,
like,
I
think
anybody
here
if
they
were
willing
to
could
draft
a
plan
on
how
to
do
that.
Migration
and
we'd
be
supportive
of
moving
forward
with
that.
A
C
C
C
Sorry
yeah,
so
what
we
do
is
that
we'll
have
to
extract
sorry
we'll
have
to
pull
down
the
images
from
docker
hub
or
any
other
registries,
which
has
the
images
that
is
being
used
for
e2e
test
re-push
them
to
a
registry.
That's
typically
local.
That's
just
like
a
registry,
that's
accessible
by
the
air
gap.
Cluster
then
configure
that
to
the
registry
to
the
local
registry.
D
D
A
Yeah,
I
sorry
for
taking
so
long
to
first
respond
on
the
issue.
I
started
trying
to
investigate
what
we
do
with
images
today,
because
so
basically,
I
feel
like
folks
from
vmware
got
real
interested
in
air
gap
testing
a
number
of
years
ago,
and
I
felt
like
I
don't
think
I
was
a
I
was.
I
didn't
review
the
particular
piece
of
code
that
added
this
registry
configuration
to
e2b
test,
but
it
seemed
like
that's
likely
where
that
came
from,
and
I
thought
that
was
sort
of
sufficient.
A
E
A
A
My
only
knit
is
like
because,
like
wilson
was
saying,
like
you
know,
looking
at
the
the
docker
library
images
that
we
depend
on
we're
going
to
want
to
move
those
to
a
different
registry-
and
there
are
a
couple
other
gcr
dot,
io
registries-
that
we
want
to
move
to
case.gcr.io,
and
so
that
makes
me
start
thinking
about.
Like
do
we
need
to
version
this
registry
list,
and
so
I
just
want
to
be
like
mindful
of
attempting
to
create
any
kind
of
api
around
the
etv
test
thing
right.
A
So
if
we
think
of
it
in
just
like
registries,
that
you
have
to
search
or
place
that
that
feels
a
lot
easier.
So
I
was
trying
to
propose
a
way
where
we
could
probably
support
both
the
old,
like
variable
names,
that
you
have
to
inspect
the
code
to
know
and
the
registries
that
you
could.
You
could
just
get
by
looking
at
the
existing
list
of
images
because,
as
we
add
images
for
new
tests,
you're
gonna
have
to
look
at
that
list
of
images
anyway,
to
figure
out
what
you
need
to
mirror.
A
It
also
sent
me
down
this
long
rabbit
hole.
Apparently
I
discovered
that
our
storage
e2e
tests
do
some
crazy
stuff,
where,
like
they
load
in,
manifests
and
then
dynamically
patch
images
in
the
manifests
on
the
fly
based
on
the
registry
names,
so
that
was
that
was
fun
anyway.
Like
sorry,
my
tldr
is
like
I'm
supportive
of
it.
I
dropped
an
issue
in
the
comments
describing
why,
and
I
see
your
your
follow-up
questions
wilson,
so
I'll
I'll
respond.
There.
A
F
If
everyone
can
see
this
guy,
it
should
just
be
the
this
metrics
issue
over
the
thanksgiving
break.
I
took
all
of
the
or
a
good
portion
of
the
metrics
in
our
metrics,
like
subdirectory,
or
the
queries
and
added
them
as
data
sources
to
data
studio
and
then
just
put
together
like
a
little
sample
report
for
things
like
yeah
our
commit
consistency.
F
This
is
a
little
bit
more
human
readable
and
I
just
wanted
some
people's
input
on
the
issue
of
what
metrics
they
might
want
to
see
from
the
metrics.
What
might
be
useful
and
where
this
should
live.
It's
embeddable-
and
I
was
thinking
that
maybe
triage
like
a
new
page
added
to
triage,
might
be
a
good
spot
for
it.
But
I
was
hoping
I
could
embed
it
in
just
the
metrics
readme
itself,
but
it
seems
like
github
doesn't
support
iframes
as
inside
the
markdown,
but
it
does
automatic.
F
F
B
F
B
B
F
B
E
A
So
you
you
can
do
that
today,
yeah,
if
you
have
a
so
like.
I
did
this
with
my
personal
account
before
I
worked
for
google
I
just
signed.
I
just
like
signed
up
for
google
cloud
or
something,
but
I
never
actually
set
up
any
building
information
so
that
I
could
use
the
free
tier,
yeah
you're
able
to
query
up
to.
I
think
it's
up
to
a
terabyte
of
data
from
gubernator
without
having
to
like
or
from
from
bigquery
without.
B
Having
to
pay
anything
okay,
yeah,
okay,
so
alrighty,
so,
okay
right,
I
I
didn't
even
think
about
cost,
but
but
yeah,
but
let's
yeah.
I
might
I've
tried
to
do
that,
but
I'm
I
may
have
there
could
be
user
error
there.
You
know
on
that
so,
but
but
I
was
getting
permissions
this
year
when
I
tried
to
run
queries
and
query
editor.
E
A
A
B
A
I
would
really
love
for
people
to
get
curious
and
explore
that
data,
because
I,
I
think,
there's
a
lot
of
useful
stuff
there.
This
used
to
be
available
via
thing
called
telodrome,
which
was
basically
a
grafana
dashboard
that
did
not
query
bigquery
directly
but
queried
something
else
that
bigqueries
were
periodically
dumped
into.
A
So
that's
where
my
question
of
anonymous
access
came
from
because
we
used
to
have
anonymous
access
there.
The
the
other
thought
I
I
have,
I
guess,
would
be
the
ability
to
either
change
time
ranges
on
the
the
metrics
dashboard
or
have
certain
dashboards
that
are
updated
more
frequently.
I
guess
because
the
way
I
personally
used
telodrome,
I
don't
know
if
anybody
else
did
was
one.
A
A
I
want
to
be
able
to
surface
the
the
issues
that
should
most
urgently
be
dealt
with
like
top
flakes
and
stuff
or
tests
that
suddenly
start
you
know
their
duration,
suddenly
changes
or
their
failure
rate
suddenly
changes,
or
something
like
that,
and
so
today
people
go
to
test
grid
to
look
for
that
sort
of
stuff
which
is
updated
like
every
15
minutes.
20
minutes,
it
kind
of
depends
on
how
frequently
the
updater
runs,
and
I
would
love
to
see
like
the
dashboards
generated
from
those
bigquery
metrics
to
update
at
a
similar
cadence.
A
A
G
G
So
it's
a
lot
more
atomic
in
that
sense.
So,
even
if
one
image
fails,
the
others
should
still
function
properly,
which
is
great,
and
this
prequels
that
I
linked.
Basically,
we
need
that,
because,
in
order
for
docker
build
decks
to
properly
build
multi-arc
images,
it
has
to
call
that
register
that
sh
with
the
persistent
flag.
This
is
like
it
really
has
to
do
that,
and
we've
included
qmo
and
camera
binaries
into
the
gcp
docker
gcloud
image,
which
is
being
used
by
google
cloud
builder
to
build
the
test
images.
G
So
now,
when
you're,
actually
using
that
image
to
to
build
old
images,
it
has
all
the
necessary
dependencies
and
requirements
to
build
all
the
images,
including
windows
ones,
including
arm
images,
s
390x.
So
on
so
forth,
I've
tried
it
multiple
times,
even
double
rasa
systems,
with
nothing
on
them
after
reboots.
So
I
had
no
lingering
cpu
flags
set,
so
it
worked
for
me.
G
G
G
A
A
It
kind
of
we
passed
test
freeze,
and
so
at
the
moment
we
should
really
be
looking
to
either
fix
broken
tests
or
fix
regressions
that
have
been
introduced
or
revert
stuff,
and
I
feel
like
I.
I
appreciate
that
rob
is
here
on
behalf
of
the
ci
signal
like
I.
I
don't
think
we
should
be
changing
the
images
that
are
used
for
tests
at
this
point.
A
H
H
H
A
A
H
I
I
This
isn't
a
major
thing
it
just.
It
did
block
us
from
from
releasing
more
coverage
this
release,
but
I
would
love
to
know
what's
the
best
practice.
As
we
start
to
add
supporting
test
infrastructure
to
our
images
when
we
need
to
bump
this
release,
was
it
just?
Was
this
a
one-off
and
we
don't
have
to
worry
about
it
too
much
or
is
there
a
process
going
forward
where
we
would
need
to
do
something?
I'm
not
sure
what
that
is.
A
Could
you
please
like
create
another
projob
for
it,
so
it
uses
the
same
shop.
It
doesn't
try
to
refresh
hey
this
says
it's
using
master,
so
I
should
go
and
use
head
for
master
right.
That
also
depends
upon
how
long
that
proud
job
crd
stays
in
that
kubernetes
clusters
registry,
which
I
think
it
times
out
within
48
hours.
At
this
moment,
it's
sort
of
dependent
on
like
how
much
traffic
pro
is
getting
and
stuff
different
pro
installations.
Have
it
tuned
differently.
A
A
A
But
I
think
we
as
a
community
would
really
appreciate
that
ability,
because
we
are
getting
a
lot
more
dependent
upon
post,
submits
and
it's
kind
of
there's
kind
of,
like
the
broader
question
of
like
making
sure
post
submits,
are
reliable
and
flake
free.
Like
we've
already
got
feta,
fatabot
set
up
to
go
and
just
spam
retest
on
pr's
that
have
flaky
jobs
right
and
just
keeps
rerunning
the
pre-submits
until
they
pass.
A
A
Yeah,
so
I
don't
want
to-
I
want
to
stitch
while
I'm
going
to
search
for
stuff,
but
I
just
want
to
comment
on
that
real
quick.
So
there
is
an
open
issue
somewhere
in
test.
Infra
that
talks
about.
Could
we
make
post
submit
reporting
better?
So,
like
you
know
how,
when
you
update
a
job
config,
and
you
merge
that
pr
you'll
see
a
comment
from
prow,
that's
like
hey.
I
deployed
these
yaml
files
which
equate
to
that
job.
A
J
Oh
sorry,
one
additional
comment
to
this
topic
of
possums
and
actually
also
periodic
reporting,
because
they
have
the
same
problem.
What
we've
been
doing
in
openshift
is
we've
been
using
a
slack
reporter
that
will
basically
for
a
given
set
of
jobs
report
to
a
slack
channel
of
ours.
If
they
fail
because
for
success,
we
don't
really
care,
there's
nothing
to
do.
A
G
A
Yeah,
so
I
don't
know,
I
personally
would
be
less
interested
in
solving
this
problem
specifically
for
image
building
jobs,
and
I
try
to
look
at
it
at
the
abstraction
level
of
like
how
do
I
trigger
a
periodic
or
a
post
submit
on
demand
and
I'm
yeah,
I'm
not.
I
guess
I
don't
have
an
idea
off
the
top
of
my
head
there,
but
I'm
not
sure
like
adding
a
slash,
build
command
would
make
a
lot
of
sense.
A
We
already
have
the
trigger
plug-in,
which
is
what
you
can
use
to
slash
test
whatever,
and
it's
a
pre-submit
job
name.
I
guess
we
could.
We
could
talk
about
looking
into
just
adding
the
ability
to
trigger
post,
submits
or
ci
jobs
from
that,
but
I'm
sure
we'd
want
to
take
a
good
long.
Look
at
how
we
authorize
that
stuff.
J
Yeah,
the
same
also
via
this
v1
authentication
group
setting
you
can
have
on
the
jobs
but
yeah
the
the
issue
we
also
have
is
that
often
people,
especially
for
post
submits,
want
the
latest
version,
because
not
only
the
sha.
Actually,
it's
also
the
job
configuration
itself
because
of
that
change,
it
also
won't
get
picked
up.
J
A
I
K
There
is
an
centralized
issue
for
putting
pre-submits
in
test
grid
for
almost
all
repos
and
modules
and,
with
the
exception
of
a
short,
allow
list,
this
is
enforced
with
our
test
grid
like
repo
specific
testing,
some
users
of
test
grid
like
to
have
their
pre-submits
up
and
some
think
that
it's
noisy
and
flaky
and
that
they
don't
want
it.
Aaron
commented
on
this
thread
kind
of
explaining
why
we
would
want
to
have
pre-submits
visible
on
test
grid.
A
B
A
A
B
B
H
That
I
mean,
do
you
mean
like,
like
failure
rate
or
something
it's
a
little
hard
to
like
I've,
tried
to
monitor
that
in
the
past
it's
a
little
hard
to
poke
around
for
like
piers
that
just
don't
build,
and
sometimes
people
are
spamming,
pairs
that
don't
build,
and
then
the
failure
rate
goes
through
the
right
and
doesn't
really
mean
anything.
So
the
heat
map
on
product
h.I.o
is
pretty
helpful
because
you
can
visually
filter
out
really
short
runs,
but
I
don't
think
there's
any
like
hard
tool
for
like
how
you
would
monitor
the
head.
A
H
That
is
not
that's
over
some
minimum
run
time
and
maybe
spot
check
those
the
quick
failures
to
make
sure
that
they
are
just
build
fills-
and
I
do
this
like
just
annually
like
I
did-
vary
sometimes
when
things
have
been
pretty
stable,
I'm
not
checking
as
often,
but
if
we've
you
know
in
the
past,
we've
had
a
rougher
patch.
It
might
be
something
I
do
daily
here
or
some
more
than
daily,
okay
kind
of
like
ti
signal,
but
just
for
just
for
what
I'm
working
on.
A
What
what
introduced
the
regression
or
what
broke
things
or
often
times
there
are
questions
about
like
hey,
did
something
in
the
test:
infra
change
that
was
there
some
environmental
issue
that
caused
this
job
to
start.
Failing
generally
in
those
scenarios,
you
want
to
start
looking
at
like
okay,
so
can
we
pinpoint,
when
this
job
over
time
started
to
fail
or
like
started
to
look
a
lot
worse
and
so
for
jobs
that
do
report
to
test
grid?
A
I
can
just
go
to
pastries
and
take
a
look
at
that
view,
but
for
jobs
that
don't.
I
have
to
know
that
I
could
also
go
through
and
click
on
go
to.
Products.I
owe,
and
I
could
search
for
the
pre-submit
job
name
and
then
I
could
click
onto
one
of
those
runs,
and
then
I
could
click
on
job
history
and
then
I
could
start
to
like
scroll.
B
H
Have
one
for
blocking
kubernetes
pre-submits?
I
I
found
it's
not
very
useful
for
two
reasons:
one
on
test
grid,
it's
harder
to
filter
out
the
noise
of
like
oh,
this
is
just
a
failed
build
from
like
code
that
doesn't
build
you're
gonna
need
to
click
through
to
the
logs
anyhow,
and
I
guess,
because
you
can't
see
the
overall
time
on
the
on
the
main
grid
view
and
the
other
one
is
the
test.
H
H
A
So,
just
to
finish
my
thought
there,
like,
like
my
two
conferences,
it's
all
good
like
I
don't
want
to
be
the
only
one
speaking
I
really
do
appreciate
I
just
want
to
so.
My
principal
concern
is
like
we
force
all
post,
submits
and
periodics
to
report
to
test
group,
but
we
don't
do
that
for
presidents.
A
A
A
H
I
found
it
really
hard
as
well
to
get
test
grid
alerting
working
well
for
pre-submit
because
of
the
fact
that
you're
going
to
see
pass
unfail
just
from
pr
noise.
But
I
do
think
that
something
like
grant's
work
on
the
on
the
business
dashboard
is
where
we
might
be
able
to
say.
Okay,
let's
have
people
other
than
heroic.
Looking
at
how
reliable
our
appraisals
are
and
have
actual
metrics,
instead
of
trying
to
like
squint
at
a
job
or
something.
K
Yeah,
I
can't
really
recommend
setting
up
alerts
for
pre-submits
in
general,
which
is,
I
guess,
kind
of
where
my
confusion
on
adding
them
to
test
grid
came
from,
but
adding
them
to
test
grid
doesn't
necessarily
require
an
alert.
It
doesn't
set
up
an
alert
for
you.
It
doesn't
set
up
stale,
alerting
or
any
of
those
kinds
of
things
that
wouldn't
necessarily
make
sense
in
a
pre-submit
context.
It
just
kind
of
displays
it
in
the
test
grid
format
for
up
to
15
days
back
or
I
guess
until
it
runs
out
of
memory.
K
So
like
from
that
perspective,
I
can
see
like
kind
of
a
contrabex
reason
for
adding
them
like
adding
those
pre-submits
like,
if
you
add
the
pre-submits
and
have
a
test
that
requires
them
to
add
a
pre-you
know,
add
them
to
test
grid.
Then
you
can
just
say:
oh
every
proud,
job's
on
test
grid
they're
all
there.
K
H
And
not
not
all
jobs
are
going
to
happen
and
for
jobs
that
don't
it
it
is
like
not
actually
significantly
more
useful
than
just
like
native
display,
which
will
search
for
the
job
name.
Probably,
actually
I
don't
know.
Oh,
I
do
also
want
to
add
a
quick
assignment
in
a
perfect
world.
I
actually
have
a
reporting
enabled
first
imprisonment
like
to
me
personally,
because
I
would
like
to
know-
and
that's
a
little
bit
less
to
just
be
like.
H
H
But
if
that
was
a
thing
I
I
would
absolutely
use
it
and
I'm
trying
to
use
it,
as
is
not
surprisingly,
it's
not
very,
not
very
helpful
tuning
up
the
number
of
jobs
and
the
reasons
it
helps
a
bit
to
filter
out
for
it
because,
like
you,
can
have
an
alert
for
when
it
just
starts
failing
continuously,
which
does
happen
occasionally
like
an
infrastructure
outage
or
something.
But
you
need
to
turn
it
pretty
high,
so
it
takes
a
while
to
notice.
A
A
A
I
just
want
to
put
out
throw
out
there
the
idea
that
well
we'll
probably
be
meeting
again
in
two
weeks,
but
then
I
think
the
next
meeting
after
that
will
likely
fall
really
close
to
the
new
year
when
I
suspect
a
number
of
people
will
be
available,
so
we
might
want
to
consider
canceling
that
so
I
was
thinking
either
like
next
meeting
or
the
first
meeting
that
we
have
in
the
new
year.
We
could
kind
of
get
together
and
talk
about
sort
of
what
our
what
our
goals
are
for
the
year.