►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Intro: Testing SIG - Aaron Crickenberger, Google & Steve Kuznetsov, Red Hat
The Kubernetes test infrastructure schedules and executes all of the tests which validate pull requests and repositories in the Kubernetes ecosystem every day. This presentation will explore how job authors can make use of new job configuration options to write jobs that execute identically inside and outside of the test infrastructure while requiring a minimal set of configuration to be provided. Jobs defined in this manner have reproducible behavior for local debugging and do not require job authors to adopt test-infrastructure-specific frameworks or workflows.
To learn more: https://sched.co/GrbJ
B
A
So,
as
always,
I
think
sig
testing
is
the
most
fun
cig
of
all
time.
We
have
created
a
world
where
robots
emoji
reacts
to
other
robots
in
a
bizarre
festively
of
human
interaction.
We
are
making
the
future
that
we
deserve,
so
this
was
actually
an
old
old
screenshot
where
we
had
our
merge
robot
and
it
was.
We
wanted
to
celebrate
when
the
bots
merged
instead
of
humans
merging,
but
it
since
turned
into
a
fun
game
where
lots
like
to
fight
each
other,
because
we
like
creating
BOTS
and
automation.
A
You
may
know
sig
testings
work
from
literally
every
PR.
You
have
submitted
to
any
of
the
kubernetes
repos,
the
most
common
workflow
that
we
have
concentrated
a
lot
of
our
efforts
on
our
work
is
the
work
flow
for
kubernetes
kubernetes,
where
we
need
to
do
the
pretty
bog-standard
things.
I
think
a
lot
of
open
source
projects
have
to
do,
which
is
make
sure
that
the
person
is
actually
allowed
to
submit
code
to
the
project
we
use
the
CLA.
Some
projects
used.
Ecos
we've
got
to
make
sure
the
PR
actually
pasts
just
a
couple
tests.
A
We
want
to
make
sure
that
the
PR
has
been
approved
so
like
a
humans
merged
or
a
human
has
looked
at
it.
The
PR
actually
has
to
be
mirja
Buhl's,
so
it
doesn't
need
to
be
repay,
stand,
there's
no
merge
conflict
and
we
want
to
make
sure
that
the
merge
commit
must
actually
pass
tests
right
before
we
commit
the
merge
commit.
A
This
is
because
it
is
a
high
velocity
project
that
merges
on
the
order
of
20
to
40
pr's
a
day
and
the
tests
take
anywhere
from
60
to
90
minutes
to
run
because
they
involve
spinning
up
entire
kubernetes
clusters.
This
is
actually
an
old
screenshot
because
it
does
not
include
the
job
that
spins
up
100
node
kubernetes
cluster,
for
each
PR
that
touches
scalability
things.
A
A
The
project
started
pretty
much
just
with
github
and
CI,
and
the
CI
would
write
its
results
back
to
github,
and
that
was
fine,
except
it
wasn't
really
that
way:
I'm
kind
of
dramatically
oversimplifying
and
the
over.
In
the
early
days,
we
actually
had
three
different
CIS.
We
were
using
Travis
and
shippable
and
an
internal
Jenkins
instance,
but
this
is
basically
what
it
started
as
if
you
ignore
all
the
stuff
that
we
don't
have
anymore.
A
A
This
was
about
as
much
visibility
as
we,
the
open-source
community,
had
into
what
our
testing
infrastructure
did,
but
it
was
fine.
It
was
a
pretty
easily
understandable
model.
Prowl
is
super
awesome
and
super
complex,
but
I'm
just
gonna
float
flow
through
these
diagrams.
To
tell
you
that
there
are
other
talks
that
show
this,
and
there
are
links
you
can
go
read.
We
didn't
have
docks
on
prowl
for
a
while
and
one
of
our
one
of
our
campaign
tree.
A
It's
coal
recently
spent
a
lot
of
time
trying
to
make
human
readable
Docs,
including
one
titled
I,
don't
care
how
all
this
works.
I
just
want
my
PR
to
merge
a
lot
of
the
jobs
end
up
using
a
library
called
cube
tests
which
tries
to
standardize
the
common
flow
you
might
have
for
testing
things
related
to
kubernetes,
where
you
want
to
stand
up,
but
kubernetes
cluster.
You
want
to
configure
things
pretest.
You
want
to
run
your
tests.
Do
some
things
post
tests
and
then
probably
tear
down
the
clusters?
A
The
post
test
might
be
like
gathering
artifacts
from
the
cluster,
so
that's
prowl.
We
then
kind
of
realized.
Maybe
it
would
be
a
good
idea
to
actually
see
all
of
the
artifacts
that
we
collect
from
prowl
and
put
them
in
a
nice
world
readable
place
instead
of
hidden
behind
the
walls
of
Google
or
hidden
across
a
couple
different
CI
systems,
and
so
we
updated
all
of
the
prowl
jobs
to
standardize
writing
their
results
into
GCSE,
Google,
Cloud
storage,
and
then
we
made
prowl
start
reading
from
that.
A
Instead
of
from
individual
CI
systems,
artifacts
kind
of
look
like
a
bunch
of
log
files-
and
you
know,
logs
for
every
single
node
and
master
under
tests,
the
J
unit
files
JSON
things
like
that,
not
super,
pretty
pretty,
but
really
informative
for
a
single
run.
We
then
kind
of
realized
that
we
wanted
something,
maybe
a
little
bit
prettier
than
just
looking
at
raw
GCSE
views,
so
we
created
a
thing
called
goober
nadir,
which
parses
the
logs
and
shows
a
couple
more
results.
A
This
is
not
the
greatest
screenshot,
because
it
shows
all
the
tests
passing
which
warms
my
heart,
but
doesn't
necessarily
show
all
of
the
features
available
to
use
such
as
showing
the
exact
failed
tests
aligned.
You
can
copy-paste
to
reproduce
the
failed
test
things
of
that
nature.
Again,
it's
great
as
a
one
shot,
but
not
super
great
at
history.
So
we
next
realized
we
needed
to
use
something
to
display
the
history
of
all
the
jobs
that
were
running
and
all
of
the
test
results
that
they
produced.
We
called
this
test
grid
I.
A
Camera
and
I
got
to
see
something
that
looked
like
this
and
I
wondered
why
that
was
something
that
could
only
be
seen
by
a
select
few
people
and
the
test
infrastructure
team
in
or
the
engineering
productivity
team
in
Google
responded
and
made
this
available
to
the
world
such
that
anybody
can
contribute
test
results.
They
they
landed
GCS,
they
get
scraped,
they
get
displayed
via
test
grid,
and
you
get
graphs
like
this.
A
That
show
you
every
row
for
every
test
case
that
a
given
job
runs
every
column
for
every
run
of
that
job,
and
you
can
kind
of
as
a
human.
Do
some
quick
pattern
matching
to
get
a
feel
for
like
when
the
job
didn't
run
for
a
little
bit
or
the
fact
that
there
doesn't
seem
to
be
any
one
particular
test
that
consistently
fails
more
than
any
other
test.
Although
we
do
have
the
tests
sorted
by
that,
so
that's
test
grid.
We
then
realized.
A
We
should
probably
make
sure
that
kubernetes
works
on
more
than
just
one
cloud,
so
we
started
adding
tests
for
other
cloud
providers.
Today
we
spin
up
clusters
in
at
least
these
three
cloud
providers
on
our
own,
and
we
are
happy
to
exercise
kubernetes
and
more
cloud
providers
just
so
long
as
we
don't
have
to
be
the
ones
to
pay
for
it
Google
initially,
by
the
way
when
I
say
we
I'm
talking
about
we,
the
kubernetes
community,
not
Google,
my
current
employer
but
Google.
A
My
current
employer
did
at
one
time
pay
to
stand
up
clusters
in
AWS
because
they
felt
it
was
very
important
to
prove
that
kubernetes
worked
across
different
cloud
providers.
The
situation
we,
the
community
art
in
today
is
that
there's
an
infrastructure
working
group
that
is
funded
by
the
CN
CF
and
the
CN
CF
provides
credits
to
us
the
project
to
stay
in
the
clusters
in
AWS.
A
We
then
realized
that
goober
nadir
is
great
tester,
it's
great,
but
wouldn't
it
be
even
better
if
we
could
store
all
of
this
data
about
our
tests
in
some
kind
of
database
that
we
could
query
because,
as
human
beings,
we
were
getting
kind
of
tired
of
every
day
scanning
dashboard
after
dashboard
after
dashboard
looking
at
test
after
test,
so
the
kubernetes
extract,
transform
and
load
executor
engine
I
forget
what
it
stands
for.
Kettle
was
responsible
for,
scraping
test
results
out
of
this
world,
readable,
GCS,
buckets
and
storing
it
inside
of
a
bigquery
database.
A
The
bigquery
database
is
publicly
readable.
Anybody
in
this
room
could
write
a
big
query
to
run
against
it
today.
It
would
cost
you
zero
dollars
to
do
that
up
until
you
had
queried
about
one
terabytes
worth
of
data,
which
protip
don't
experiment
with
queries
against
the
all
builds
dataset,
but
you
can
get
by
just
fine
querying
the
day
or
week
data
set,
which
it
is
in
fact
what
we
did
with
a
proud
job,
so
just
another
job
executed
by
prowl,
but
instead
of
spitting
up
clusters
in
GCE
and
gke,
it's
running
queries
against
bigquery
dumping.
A
Those
results
into
GCS
as
JSON
files,
which
are
then
parsable
by
a
triage
dashboard.
That's
that
starts
to
show
the
count
of
failed
tests
and
failed
jobs
across
every
single
job
that
we
run.
It
shows
you
useful
things
like
the
top
five
most
recent
occurrences
of
different
failures
and
really
what
I
say:
failures
I
mean
failure,
clusters,
the
idea
being
that
the
same
test
case
might
be
failing
across
multiple
different
jobs.
A
But
some
of
the
text
is
slightly
different
because
it
happened
at
different
times
or
it
used
different,
slightly
different
pod
names,
because
pod
names
have
that
random
UID
on
them.
So
this
would
be
responsible
for
taking
a
look
at
all
of
the
clusters
or
all
of
the
failure
text
across
all
of
the
tests
kind
of
clustering
it
together
in
the
the
failure
text
that
looked
most
similar
and
then
ordering
it
by
what
happened
the
most.
A
This
lets
you
see
in
general,
like
are
things
getting
worse
over
time,
and
is
this
test
failing
just
in
my
job
or
consistently
across
a
set
of
jobs?
What
does
the
behavior
look
like?
That's
triage,
so
we
realized
we
kind
of
had
a
good
handle
on
what
was
going
on
with
our
tests,
but
we
didn't
actually
know
about
the
operational
health
of
the
components
at
play
here.
A
So
we
started
using
a
pretty
common
stack,
I'm
sure,
you're,
aware
of
the
Prometheus
scraping
metrics
from
some
of
our
micro
services
in
prowl
dumping
that
data
into
an
influx
DB
instance
and
then
being
displayed
by
Griffin
ax.
In
this
case,
we
called
that
velodrome,
so
I
showed
this
screenshot
only
to
show
you
that
there
are
graphs
and
they
look
kind
of
like
a
graph
on
a
dashboard.
A
We
don't
have
to
spend
too
much
time
on
what
they
are,
but
these,
let
us
know
things
like
how
healthy
is
the
project
right
now
in
terms
of
merge
velocity.
How
long
has
it
been
since
something
merged?
This
was
a
nicer
way
than
having
to
scan
dashboards,
continue
to
scan
a
bunch
of
different
tests,
your
dashboards.
A
We
then
realized
now
that
we
have
the
ability
to
dump
data
into
influx
DB
and
display
it
that
we
could
probably
get
a
lot
more
useful
data
out
of
bigquery,
and
so
we
wrote
some
metrics
jobs
again,
just
a
prowl
job
that
runs
the
bigquery
command-line
tool
and
dumps
data
into
in
flux,
TB,
which
we
display
on
dashboards.
This
allowed
us
to
identify
things
like
what
are
the
pull
request,
jobs
that
are
failing
the
most
or
flaking
the
most
this
week
and
then
for
the
flaky
estab.
A
What
are
the
to
flake
iasts
test
cases
when
I
say
flaky
I
mean
this
is
a
job
that
both
passes
and
fails,
even
though
the
commit
hasn't
changed
at
all.
This
is
a
clear
indicator
that
it
some
environmental
or
flake
condition,
rather
than
the
failure
of
the
code
itself,
but
since
we
did
find
failure
rates
also
kind
of
a
useful
approximation,
we
have
this
graph
at
the
bottom,
which
just
showed
on
a
daily
basis.
A
What's
the
failure
rate
for
each
individual
job,
this
allowed
you
as
a
consumer
of
the
project
or
a
developer
on
the
project,
to
understand
that.
It's
not
just
you.
The
e2e
cops
AWS
job
is
actually
failing
a
hundred
percent
of
the
time
or
is
starting
to
fail
more
than
it
used
to.
This
was
the
sort
of
thing
that
allowed
us
to
quickly
identify
things
like
quota
issues
or
too
many
tests
being
added,
causing
the
performance
to
kind
of
malfunction
and
identifying
new
flakes
or
anti-patterns
this
job.
A
This
dashboard
also
shows
like
yep
over
10,000
jobs,
running
on
a
daily
basis
and
roughly
what
the
pass
and
failure
rate
is
for
all
of
those
you'll
notice.
This
is
a
very
old
image.
There
are
some
jobs
that
have
been
continuously
failing
for
over
a
hundred
days.
It's
up
to
400
now
for
some
of
our
jobs.
A
So
that's
metrics
and
visibility
into
what's
going
on.
I
am
almost
out
of
time
here,
so
I'm,
just
gonna
whiz,
through
the
fact
that
we
did
some
more
things
for
scalability
like
instead
of
scraping
GCS.
We
started
streaming
data
out
of
GCS
to
that
database
that
we
were
querying
to
make
all
those
dashboards.
We
started
using
kind
of
like
a
resource
pool
manager
called
Vasquez,
which
would
pre
spend
up
Google
cloud
projects
for
us
rather
than
having
to
be
very
diligent
about
maintaining
them.
A
We
could
just
check
something
out
of
a
pool,
use
it
and
then
throw
it
away
and
trust
that
that
would
work.
We
started
using
a
build
cache
called
greenhouse
for
build
system.
We
call
basil
this
allowed
us
to
drastically
reduce
the
amount
of
time
we
spent
building
the
code
for
periodic
jobs
over
time
and
then
finally,
we
learned
fun
things
like
did.
A
You
know
that
github
API
really
is
only
designed
to
let
you
talk
to
it
once
per
second,
who
knew
we
found
out
that
eventually,
instead
of
trying
to
maintain
in-memory
caches,
it
was
just
a
lot
easier
to
create
an
external
proxy
or
cache
to
get
such
that
we
haven't
had
to
run
into
our
5,000
API
token
limit
5,000
tokens
per
hour.
All
that
much
which
of
these
things
are
open-source,
like
which
of
these
components.
Could
you
use
today
if
you
wanted
to
everything,
but
that
black
box
and
that
black
box
is
test
grid?
A
We
want
to
open-source
test
grid
in
order
to
better
help
us
understand
how
best
to
open-source
it.
We
would
appreciate
people
who
are
interested
in
the
using
test
grade
themselves
to
comment
on
this
issue.
This
is
going
to
help
us
identify
like
the
potential
use
cases
which
will
help
us
identify
exactly
which
components
we
should
open-source
and
how
we
should
architect
it,
because
the
way
that
this
works
right
now
kind
of
relies
on
some
magic,
google,
internal
infrastructure
and
we'd
like
to
figure
out
how
best
to
decouple
which
parts
first
finally,
I
wanted
to.
A
The
Elder
and
James
Nunnally,
the
two
founders
of
kind,
better,
have
a
yo
dog
slide
in
their
talk,
and
just
in
case
they
didn't
I
made
sure
I
had
one
in
mine
kind
is
basically
about
running
kubernetes
in
containers
in
containers,
because
you
know
dog
I
heard
that
you
appreciate
containers
I
won't
go
into
this
too
much
either
because
we
have
a
link
to
their
talk
and
it
was
yesterday.
So
unfortunately
you
missed
it,
but
trust
me.
B
Awesome
thanks
Aaron,
so,
as
was
very
obvious
to
probably
all
of
you
that
entire
system
is
incredibly
complicated
and
especially
if
you're
starting
up
a
new
repository
or
maybe
you're
thinking
about
adding
a
new
test
going
face
to
face
with
everything
that
existed,
there
is
pretty
pretty
frightening
and
I.
Think
a
lot
of
people
often
have
a
question
of
well.
You
know,
Circle
CI
is
easy.
You
know,
Travis
is
easy.
Why
don't
I
use
those
systems
instead
and
so
I
wanted
to
really
quickly
walk
through?
What
does
it
look
like
to
actually
write
a
job?
B
Why
might
you
think
about
adopting
proud
instead
of
those
other
systems
and
yeah
like
we
just
what
does
it
look
like
and
how
easy
is
it
actually
so
one
of
the
largest
reasons
that
you
might
think
about
adopting
prowess
Peschel
ii,
if
you're
adding
a
new
repository
to
the
kubernetes
or
gor
the
community
cigs
org
is
you're.
Gonna
have
consistent
UX
with
the
rest
of
the
projects,
and
so
developers
can
interoperate
between
different
repositories.
They're,
not
learning
new
new
tools,
they're,
not
learning
new
commands.
You
know
it's
simple:
they
like
that.
B
B
One
of
the
things
that
Aaron
called
out
earlier
is
that
we
managed
automatic
merging
for
pretty
much
147
repos,
and
one
of
the
things
that
we
try
to
enforce
when
we
merge
automatically
is
that
your
code
is
not
only
tested
with
the
latest
version
of
the
code
in
your
PR,
but
it's
also
tested
against
the
leis
latest
version
of
the
branch
that
you're
merging
into
and
that
helps
keep
everything
healthy.
That
isn't
actually
something
that
github
knows
how
to
track
and
most
of
their
systems
don't
track
that
either.
B
Also,
at
this
point,
we'd
like
to
think
it's
kind
of
simple
to
write
a
new
job
and
as
soon
as
you
do
that
you
actually
buy
into
all
of
that
complication.
That
aaron
was
talking
about.
So
you
get
the
gubernatorial
IDing.
You
get
the
test
code
overviews.
You
know
you're
going
to
be
pulled
into
all
of
the
alerting
and
the
metrics
that
are
coming
out
of
that,
like
detection.
So
it's
going
to
be
a
lot
easier
for
the
developers
on
your
project
to
operate
with
that
job.
B
One
other
thing
that
you
might
notice
as
your
project
grows,
is
that
circle
CI
or
Travis
at
the
free
tier
will
start
throttling
your
builds,
and
that
means
that
pr's
are
going
to
start
waiting
longer.
It's
gonna
be
harder
to
actually
get
the
merge
velocity
that
you'd
like
right
now,
if
you're
adopting
prowl
there's
no
throughput
limit,
so
you're
gonna
be
able
to
test
as
your
project
rose
without
any
issues.
B
B
B
So
if
it's
a
PR
or
if
it's
a
batch
of
PR
is
everything
merged
together,
everything's
checked
out
correctly,
when
things
are
being
run,
the
working
directory
is
your
repo.
There
is
an
environment
variable
called
artifacts,
it's
a
directory
in
the
container.
If
you
put
anything
in
there,
it's
going
to
be
slipped
up
into
GCS,
all
the
metadata
that
the
other
external
tools
need
to
integrate
with
your
job.
B
That's
going
to
be
generated
for
you,
so
really
we'd
like
to
think
that
when
you're
providing
us
with
this
job,
we're
handling
everything
on
the
infrastructure
side,
and
then
you
just
get
to
think
about.
What's
in
your
job,
if
you
start
integrating
with
us-
and
you
find
oh,
you
know
if
something's
missing
go
ahead
and
let
us
know
like
we'll
be
happy
to
add
more
stuff
in
there.
We
do
support,
you
know,
checking
out
private
repositories
using
SSH
I
think
using
yeah,
let's
just
say
using
SSH
for
now,
but
yeah.
B
So
we're
hoping
that
you
know
all
the
infrastructure
pieces
are
there
for
you
so
super
quickly,
just
to
look
through
an
example
of
a
job.
We're
gonna
be
looking
at
a
job
that
verifies
generated
code
in
our
tests
and
for
a
repo,
we're
gonna
look
at
how
is
it
actually
written?
What
does
it
look
like
in
in
practice
and
some
of
the
integrations
that
we
got
for
free
just
by
checking
in
a
couple
lines
yeah?
B
B
You
know
whatever
image,
has
your
runtime
dependencies
and
we're
running
a
script
from
inside
of
the
repo
on
every
PR,
and
if
we
open
a
pull
request
to
test
in
for
today,
this
job
is
are
going
to
run
it's
going
to
be
reported
just
like
every
other
job.
You're
gonna
see
it
at
the
bottom
of
your
PR
is
a
status
if
it
fails.
You'll
have
that
comment
that
lets
you
know
hey.
This
is
failing
for
this
commit.
You
know
here
are
the
logs.
B
If
you
are
writing
the
job,
and
you
want
to
look
at
the
health
of
the
job,
you
know
in
a
sort
of
short
timespan,
you
can
look
at
our
proud
overview,
page
called
deck
there.
You're
gonna
see
every
single
run
of
that
job
in
the
last
day
or
two
you'll
see
what
the
revision
was
for
it.
You
know,
how
did
how
long
did
it
take?
Did
it
fail
or
succeed?
This
helps
you
get
like
a
quick
bead
on
whether
the
job
is
healthy,
especially
useful.
B
If
it's
something
that
new
that
you're
debuting,
also
one
of
the
free
integrations
that
you're
going
to
get
is
test
grid.
So
the
metadata
that
is
generating
this
page
is
automatically
created
for
you
by
prowl.
This
is
a
really
simple
job
which
either
fails
or
succeeds.
So
it's
not
the
greatest
example,
but
you
know
if
your
job
is
giving
us
more
information.
You'll
have
that
sort
of
larger
grid
that
Aaron
was
showing
for
the
other
job
and
super
quickly.
These
are
links
that,
hopefully,
if
you're
interested
you
can
click
in
your
own
time
right
there.
B
If
you
want
more
information
on
that
I,
just
looking
at
the
talk
from
yesterday
by
Ben
but
yeah,
the
the
bare
minimum
that
we're
gonna
need
is
just
a
container
image
runtime
and
then
the
command
to
run
in
there.
So
because
we
want
to
have
some
time
for
questions.
These
are
some
of
the
other
talks
that
suggesting
is
involved
in
a
cubic
on
this.
B
If
you're
interested
in
some
of
the
more
in-depth
look
at
how
the
whole
process
then
works
together,
:
Katherine
are
giving
a
talk
at
3:40
and
if
you
want
to
contact
us,
there's
a
whole
bunch
of
different
ways
we're
on
slack.
We
have
a
mailing
list,
absolutely
reach
out
to
us,
and
hopefully
we
have
a
bunch
time
for
questions.
A
A
Yeah,
this
has
been
over
the
evolution
of
the
project.
There
definitely
some
pieces,
I
have
left
out
I
left
out
a
thing
called
munch
github,
which
was
around
for
quite
a
long
time.
It
was
supposedly
the
thing
that
was
going
to
save
us
from
the
crazy
days
of
checking
either
shippable
or
Travis,
or
an
internal
Jenkins
instance,
depending
on
which
one
of
the
three
just
happened
to
pass
at
that
time.
That
seems
no
I,
see
maybe
a
hand
coming
up.
A
A
A
The
biggest
test
of
this
would
be
what
we
like
to
call
code
thaw
where,
during
the
release
process,
there's
a
bit
of
a
buildup
of
unmerged
PRS
and
almost
every
time.
We
would
have
to
do
this.
What
the
old
thing
we'd
wait
to
do
code
thought
over
the
weekend
so
that,
like
no
new
PRS
would
land
and
we
just
kind
of
watch
a
tail
off.
B
A
A
A
People
who
are
not
using
prowl
I
wish
I
could
give
you
a
firm
answer
to
that
I.
Think
then
the
Eldar
was
the
author
of
greenhouse
and
Jeff
Grafton
is
the
other
Basel
expert
that
we
have
I
would
say
likely?
Yes,
although
it's
unclear
to
me
how
much
sense
it
makes
in
the
face
of
Basel
attempting
to
have
more
in
the
way
of
remote
execution,
do
you
know
any
more
than
that's?
Do
you
not.
A
Definitely
one
of
those
projects
that
got
to
a
place
where,
like
it's
good
enough
for
us,
and
we
would
love
to
see
if
it's
good
enough
for
other
people
too
you'd
like
most
of
the
boxes
in
this
are
all
sub
directories
of
tests.
Infra
kind
was
actually
our
first
sub
project
that
we
started
and
tested
for
and
then
broke
out
into
its
own
repo.
A
D
A
D
A
A
A
A
I
would
like
to
see
it
be
a
project
that
a
bunch
of
people
other
than
just
core
kubernetes
contributors
actively
contribute
to
we're
thrilled
to
see
this
community
like
we've
got
I,
don't
know
how
many
of
you
use
Jenkins,
X
or
realize
like
how
thin
of
a
wrapper
it's
becoming
over
proud
these
days,
but,
like
even
cloudy,
seems
to
be
thrilled
to
kill
off
Jenkins
in
favor
of
something
like
prowl.
We
love
this
support.
We
are
ready
to
like
drive.
This
help,
see
this
drive
forward
and
not
be
the
bottleneck
on
it.
B
Red
Hat
our
proud
employment
is
at
similar
scale
to
the
kubernetes
running
fewer
jobs
in
throughput,
but
supporting
a
similar
amount
of
people
and
so
I
think
for
us
too.
It's
important
that
you
know.
There's
decisions
that'll
be
me
made
about
the
direction
of
the
project
are
sort
of
independent
I
think
we
can
break
it
out
pretty
soon,
like
we've
done
so
much
legwork.
Recently,
it's
super
close.
You
have
more
questions,
but.
A
So,
just
two
more
things
to
tag
onto
that
to
me
personally,
the
two
pieces
of
the
prowl
experience
that
aren't
quite
super
a
hundred
percent.
There
are
the
thing:
the
uber
Nader
and
the
test
grid
test
good,
because
it's
not
quite
open
source
uber
Nader,
because
it
is
an
app
engine,
app
I
think
the
ideal
vision
of
the
future
would
be.
You
get
a
kind
of
a
nice
consolidated,
opinionated
stack
of
components
running
on
it,
nice
cluster,
if
he
so
desire.
We
also
want
to
head
in
that
direction.
A
We
do
kind
of
have
about
three
years
worth
of
project
legacy,
carrying
that
we
have
to
carry
forward.
So
the
idea
to
replace
goober
nadir
is
something
called
spyglass,
which
is
a
proud
component
which
would
allow
you
to
view
arbitrary
artifacts,
and
then
we
are
also
working
on
the
historical
component
that
goober
nadir
provides
like
some
things.
I
left
out
or
a
lighted
over
is
that
goober
nadir
provides
a
handy
dashboard
of
given
a
single
pull
request.
What
is
the
history
of
all
of
the
runs
of
the
various
jobs?
For
like
each
of
the
revisions?
A
To
that
pull
request
is
you've,
pushed
up
to
a
rebase
or
or
fix
different
things,
and
then
test
grid.
Obviously,
I
feel
like
it's
just
that
much
longer
historical
view
that
you
don't
really
get
with
the
proud
UI
that
you
have
today.
The
the
UI
we
have
today
is
called
Dec
and
it
just
kind
of
shows
a
list
of
the
past
end
days
worth
of
jobs
that
have
been
running
and
then.
A
Finally,
the
other
piece
is
like
we
tied
a
GCS
and
a
bunch
of
other
cloud
services
so
that
it
was
really
easy
for
us
to
not
have
to
write
those
parts
ourselves.
So
I
have
no
idea
what
it
would
be
like
to
run
these
things
on,
like
in
air-gapped
cluster.
That
doesn't
have
these
cloud
services
available.
Like
would
you
you
know,
stand
up
a
safe
cluster
and
then
create
sort
of
an
adapter
that
all
of
these
things
would
talk
to
instead
of
GCS.
A
Maybe
and
I
have
no
idea
like
what
performance
characteristics
or
limits
you
would
hit
there
versus
the
other
things.
And
finally,
the
github
component
of
it
is
something
we
are
very
interested
in
attempting
to
abstract
away
today.
In
fact,
if
you
hop
into
the
prowl
channel
on
slack
we're
talking
about
you
know
who
are
the
folks
that
are
actively
working
on
this
sort
of
stuff
right
now.
Okay,
next
question:
just.
D
The
last
one,
so
we
already
adopted
it
so
we're
operating
it
now.
The
the
next
stuff
we
have
on
our
list
is
to
start
monitoring
right,
the
whole
private
infrastructure,
and
do
you
have
some
resources?
You
can
open
up
information
like
I'm,
maybe
templates
for
the
profile
dashboards,
whatever
that
you
use
for
really
monitoring
the
problem
for
structure
yeah.
B
It's
early
so
I
think
in
the
velodrome
directory.
There
are
all
sorts
of
dashboards
that
are
being
used
right
now
to
monitor
prowl
and
the
infrastructure
for
it,
and
then
I
did
a
lot
of
work
last
year
to
make
sure
all
of
our
logs
are
structured
in
standardised,
so
any
sort
of
like
log
ingestion
that
can
handle
structured
data
and
do
queries
across
it
will
handle
like
logs
from
the
aggregated
services
pretty
well,
but
yeah
I
would
take
a
look
at
vélodrome
yeah.
A
The
important
call
to
help
I'll
put
out
there,
as
if
anybody
has
experience
working
with
a
graph
on
ax
Prometheus
influx
DB
deployment
on
kubernetes.
That's
kind
of
like
get
ops
based
I
would
love
to
talk
to
you
and
see
how
we
could
use
that
instead
Velodrome
also
a
vestigial
piece
of
code
that
was
actually
custom
engineered
and
really
we're
just
using
it
now,
because
it's
the
deployment
in
flux
and
Prometheus
and
Grif
Lana
that
hasn't
yet
broken
on
us.
So
why
bother
spending
the
time
REE
like
reinventing
it?
It.
A
Okay,
I
think
we
are
at
time.
Thank
you,
everybody
for
your
questions.
One
last
thing:
I
just
wanted
to
call
out
I
came
from
another
room
that
was
like
they
were
basically
two
non
white
dudes
in
the
room,
I'm
thrilled
to
see
that
the
attendance
for
this
room
is
slightly
different
and
if
there's
anything
that
sig
testing
can
be
can
do
to
be
more
welcome
to
that
I
am
super
open
to
criticism.
So
thank
you
very
much.
Everybody
thanks
for
attending.