►
Description
Given that there are many ways to create Kubernetes clusters and many environments used to host them, those tasked with maintaining a cluster are often left wondering whether it is ‘correct’. Is it properly configured? Does it work as it should?
Sonobuoy, an open source diagnostic tool, aims to address these questions by running upstream Kubernetes tests (and your own custom plugins) to generate reports that help you understand the state of your cluster. Its diagnostics provide a customizable, extendable, and cluster-agnostic way to generate clear, informative reports about your cluster, regardless of your deployment details.
Sonobuoy is the underlying technology powering the Certified Kubernetes Conformance Program, which was created by the Cloud Native Computing Foundation (CNCF) and is used by every Certified Kubernetes Service Provider.
In this webinar we will have John Schnake, tech lead for Sonobuoy, go over what Sonobuoy is, how you can use it in your own cluster in just a few minutes, and how you can get involved with the project.
A
Hi
everyone
and
welcome
to
today's
webinar
and
today
we're
going
to
cover
sonobuoy
validating
your
kubernetes
configuration
with
me
today.
I
got
John
and
who's
the
son,
a
boy
tech
lead
here
at
VMware,
so
John
will
go
over
what's
on
the
boy
is
how
it
works
and
how
to
get
things
running
inside
your
own
environment.
What
to
look
out
for
when
it
comes
to
the
results
and
how
you
can
actually
gather
a
ton
of
information
using
sort
of
way.
If
you
have
any
questions,
you
are
free
to
raise
your
hand
within
the
the
webinar
here.
A
B
B
So
we're
gonna
start
with
an
introduction
about
what
sana
buoy
is,
who
would
use
it
and
why?
What
really
sana
buoy
accomplishes
and
we're
going
to
talk
about
conformance,
it's
always
a
key
factor
of
when
people
think
of
sana
buoy.
What
it's
used
for
is
getting
those
conformance
results
for
their
cluster.
B
So
first
sauna
buoy
is
a
diagnostic
tool
that
makes
it
easier
to
understand
the
state
of
your
cluster
by
running
a
set
of
plugins.
This
includes
the
kubernetes
conformance
tests,
its
customizable
and
extendable,
so
that
you
can
run
it
against
any
cluster
and
get
the
information
that
you
need
so
cluster
operators.
It's
really
two
main
reasons
cluster
operators
would
want
to
run.
This
is
one
they've
just
stood
up
a
new
cluster
and
they
want
to
make
sure
that
it
is
in
a
good
state
before
they
start
putting
important
workloads
on
it.
B
So
you
might
run
an
entire
conformance
suite
just
to
verify
the
cluster
is
in
the
correct
state
and
ready
for
production
workloads.
Secondly,
if
there
starts
to
be
problems,
sonna
buoy
is
a
way
for
you
to
run
all
those
tests
again
and
see
exactly
what
is
failing
right.
There
may
be
specific
functionality
that
is
causing
problems
and
sonamoo.
We
can
help.
You
gather
all
that
information
as
quickly
as
possible.
B
Now,
service
providers
as
well
would
want
to
ride,
runs
on
a
buoy
so
that
they
can
verify
that
a
cluster
with
their
services
continues
to
function
in
a
conformant
way
in
the
proper
way,
and
specifically
they'd
want
to
know.
Does
it
pass
conformance
is
when
you
can
say
that
you
passed
the
conformance
test.
You
get
a
nice
certified
kubernetes
stamp
like
this
from
the
CNC
F.
B
The
CNC
F
develops
a
certified
kubernetes
program
to
try
and
create
the
idea
of
what
it
means
to
be
a
conformant
workload
or
a
conforming
cluster,
and
what
that
really
means
is
it's
promising
that
if
your
workload
works
on
this
cluster,
then
you
move
it
to
another
cluster
that
is
also
conformant.
It
should
still
be
able
to
work
and
that's
important
when
you're
moving
workloads
between
clusters
that
may
be
in
completely
different
environments.
B
It
is
important
to
note
that
that
certification
to
be
sort
of
certified
kubernetes
is
version
specific,
because
the
tests
that
define
conformance
are
constantly
in
a
state
of
change
so
as
kubernetes
adds
more
and
more
features
to
alpha
or
beta
or
GA
right.
They're
gonna
rely
on
certain
fundamental
behavior
that
should
be
the
same
between
all
clusters,
so
sana
Bui
takes
that
hard
part
away
from
you
as
well,
because
we
look
at
your
cluster
and
run
the
appropriate
set
of
tests
for
you
to
prove
conformance.
B
So
let's
talk
a
little
bit
more
about
how
sana
Bui
relates
to
the
upstream
community.
Since
we've
talked
about
conformance
a
lot
of
times,
people
get
confused
about
where
those
boundaries
exist
and
they
are
a
little
bit
fuzzy.
To
be
honest,
because,
even
though
the
CNC
F
is
a
standalone
organization
and
there's
the
kubernetes
organization
and
code,
there's
the
Sun
a
buoy
code,
those
seem
like
clear
lines,
but
the
developers
do
move
back
and
forth.
B
So,
for
instance,
the
cloud
native
computing
foundation
defines
what
it
means
to
be
conformant
they're,
the
ones
that
give
that
stamp
of
conformance
for
a
certain
version
of
kubernetes,
but
they
don't
actually
write
the
tests.
They
don't
own.
The
tests
that
define
conformance
all
the
tests
live
in
the
actual
kubernetes
repo.
So,
as
new
versions
come
out,
the
tests
keep
getting
put
into
that
repo.
Likewise,
the
kubernetes
repo
and
the
test
infra
repo.
B
They
are
the
ones
responsible
for
actually
building
the
test
image
that
is
bundling
up
all
those
tests
with
a
few
tools
to
run
them
and
that's
published
on
every
release.
That
is
part
of
the
upstream
kubernetes
community.
Asana
bui
doesn't
own
those
tests
either,
but
we
run
them.
So
we
define
a
plugin
which
is
really
just
a
description
of
how
we
want
to
run
those
tests.
B
So
we
define
the
plug-in
and
then
we
utilize
that
test
image
from
the
upstream
community
and
the
CNC
F
points
to
sana
Bui
as
a
preferred
method
of
running
those
tests,
so
that
everybody
can
have
a
simple
way
of
gathering
that
data
that
they
need
and
handing
it
off
to
the
CNC
F
for
review.
But
again
the
developers
kind
of
live
in
between
all
of
these
organizations.
For
instance
I'm.
B
All
the
way
from
the
sauna
buoy
side,
all
the
way
to
the
upstream
side
and
I
help
describe
on
the
CN
CF
website
how
to
run
sauna
buoy
for
those
results.
So
let's
talk
about
the
structure
of
sauna
buoy,
it's
at
its
heart,
a
pretty
simple
application,
and
that
there's
only
four
major
components
to
consider.
There's
the
CLI
that's
out
here:
that's
the
part
that
you
actually
interact
with,
and
it
produces
a
bunch
of
the
amyl
that
we
throw
into
the
cluster
to
actually
run
all
the
work.
The
main
thing
that
it
creates.
B
The
first
thing
to
note
is
the
aggregator
pod.
Now
this
aggregator
is
a
sauna
buoy
image.
It's
really
just
a
minimal
image
with
sauna
buoy,
the
CLI
put
into
it
and
sauna
movies
running
in
an
aggregator
mode
in
this
mode.
It's
looking
at
your
configuration,
and
by
that
I
mean
the
configuration
of
your
test
run.
What
plugins
do
you
want
to
run?
How
do
you
want
to
run
them
those
sort
of
things
and
then
it
for
every
plug-in
it
launches
a
pod?
B
That
is
a
combination
of
two
things,
your
plug-in,
which
is
the
thing
that
actually
does
the
work.
That
runs
the
test
that
gathers
the
logs
and
a
sauna
buoy
worker.
Now
the
sauna
buoy
workers
is
responsible
for
the
communication
back
to
the
aggregator.
That
communication
has
done
over
HTTP
and
sauna.
Buoy
handles
all
of
this
in
clear
communication
itself
for
you,
so
the
only
thing
that
the
plug-in
has
to
do
is
its
own
work
and
then
transmit
those
results
to
the
worker
and
since
it's
local
within
the
pod,
that's
usually
a
very
simple
thing.
B
You
can
do.
A
simple
bash
script
is
enough
to
communicate
with
the
worker.
Okay,
the
two
plugins
that
we
provide
in
sauna,
buoy
itself
or
the
into
end
tests.
Those
come
from
the
kubernetes
repo
again,
but
then
also
a
systemd
logs
gatherer
and
those
kind
of
exemplify
the
two
different
types
of
plugins
that
exist.
The
end
end
tests
are
a
what
we
call
a
job
plug-in.
They
run
once
in
one
pod
and
submit
the
results.
B
There's
the
systemd
logs
gatherer,
which
is
what
we
call
a
daemon
set
plugin.
That
runs
once
on
every
node.
In
the
cluster,
so
that
can
help
you
gather
data
about
you
know
what
resources
and
capabilities
each
machine
might
actually
have
or
gather
those
nodes
specific
logs
on
every
single
node.
You
can
probably
find
a
lot
more
imaginative
uses
for
that
too.
If
you
just
want
to
run
something,
you
know
based
on
the
number
of
nodes
that
you
have,
so
you
those
are
the
two
ideas
that
you
run
once
or
run
once
per
node.
B
Okay.
So
let's
do
a
basic
demo
of
some
of
the
features
of
sauna
GUI,
the
main
entry
point,
our
main
command
is
something
we
run
once
we
do
that,
you
can
see
it
creates
a
bunch
of
objects,
like
I
said
it
creates.
The
amylin
throws
that
into
your
cluster.
To
actually
start
the
run
now
by
default.
What
sauna
Bui
is
gonna
do
is
run
the
end-to-end
tests
and
run
them
specifically
in
a
way
that's
gonna
test
for
conformance,
so
those
test
runs
usually
take
about
an
hour.
B
We're
not
gonna
wait
for
those
to
finish
or
anything
in
this
demo,
but
then
we
also
run
these
systemd
logs
gatherer
so
that
you
can
see
what's
happening
on
each
node.
If
there
are
problems.
Okay,
now,
once
the
run
is
going,
you
have
a
couple
of
options
about
how
to
monitor
that.
You
can
do
sauna
buoy
status,
which
will
list
all
the
plugins
that
you're
running
and
their
current
status
running
or
complete,
and
then
it
just
says
in
a
human
readable
fashion
that
it's
still
running
more
or
less
be
patient.
B
You
can
also,
if
you
want
to
dig
more
deeply
into
it
because,
like
I
said,
the
end-to-end
tests
take
up
to
an
hour
to
run.
Usually
so
maybe
you
want
to
know
how
are
those
going
you
know
are
their
failures.
Yet
what
current
test
is
it
on?
Is
it
hanging
you
can
do
sauna
Bui,
logs,
asana,
Bui
logs
is
looking
at
the
container
logs
for
every
container
that
we've
launched
in
that
namespace.
So
all
of
this
is
coming
from
the
end-to-end
tests
and
you
can
use
that
correlate
with
how
many
tests
you're
expecting
to
run.
B
Then
four
out
of
the
3,000
tests-
this
is
an
important
thing
to
point
out
as
well.
It's
like
I
said
there's
the
idea
of
conformance.
That
is
a
subset
of
all
of
those
tests
that
exist
upstream.
So
you
only
need
to
pass
those
204
conformance
tag
tests.
The
other
ones
deal
with
a
lot
of
other
upstream
concerns.
Scalability
optional
features,
different
storage
providers
and
things
like
that.
B
B
B
You
really
just
need
a
small
section:
that's
describing
the
plugin,
so
here's
the
end-to-end
plugin
and
it's
a
job,
so
we're
gonna
run
it
once
and
then
there's
some
details
about
what
image
to
run
the
command
all
these
sorts
of
container
customizable
options.
But
this
is
what's
loaded
by
that
aggregator
pod
order,
B
again
pods
with
a
Sun
sauna,
GUI
sidecar.
B
B
B
B
B
We
wait
from
waiting
okay,
so
now,
if
we
do
sauna
buoys
status
now
you
can
see
that
this
column
has
been
filled
and
not
only
is
the
plug-in
completed.
We
can
actually
see
that
it
passed
and
for
a
lot
of
users
like
in
their
CI
system.
That
might
be
all
that
you
really
need,
but
you
can
also,
if
you
ask,
for
the
status
in
the
JSON
format,
you
can
see
additional
information
such
as
this
past,
one
test
in
it
skipped
3584
and
there's
also
some
extra
information
about
the
results
tarball.
B
Sauna
buuck
status
also
mention
that
you
can
use
retrieve
to
download
that
tarball,
so
the
Samba
we
retrieved
command
actually
echoes
the
name
of
the
file
that
it
downloads,
so
I
typically
always
store
that
in
a
variable,
so
that
I
can
immediately
use
it
to
check
the
results.
There's
this
command
sign
ugly
results.
B
This
provides
just
a
simple
human
readable
report.
Now
here
it
looks
pretty
similar
to
just
that
JSON
data,
but
if
there
were
failures,
as
we'll
see
later,
it
provides
additional
functionality
to
dig
deeper
into
those
and
now
locally.
You
do
have
a
tarball
that
you
can
open
up
and
again
when
we're
taking
a
deep
dive
into
those
results.
I'll
show
you
what's
all
inside
of
that
tar
ball
and
we're
gonna
clean
everything
up
for
our
next
demo.
B
B
Since
the
worker
is
as
a
sidecar,
we
add
a
mount
that
is
shared
between
the
plugin
and
the
worker
so
that
we
can
share
data
over
just
the
local
file
system.
So
once
the
plugin
produces
some
output
and
can
be
a
single
file,
it
can
be
a
tarball.
It
could
be
a
whole
website.
It
could
be
anything
you
want
to
do.
Maybe
you're
doing
image,
processing
tests
or
something
like
that,
whatever
you
produce,
all
you
have
to
do
is
write
the
path
to
that
object.
B
So
once
the
plug-in
reports,
those
results,
the
aggregator
actually
looks
into
them
and
actually
counts
up
the
successes,
the
failures
so
that
that
sana,
Bui
results
command
and
the
status
can
tell
you
more
useful
information
right.
It's
not
just
a
black
box
anymore,
like
it
was
a
few
releases
ago.
This
is
a
relatively
new
feature.
B
So,
let's
take
a
look
at
a
custom
plugin
from
start
to
finish,
I'm
going
to
talk
about
how
you
can
build
a
plugin,
run
the
plug-in
and
then
view
the
results
from
that
custom.
Plugin
now
inside
the
hep
D
asana
Bui
repo,
though
we
do
have
an
example,
plug-in
just
called
command
runner,
which
is
a
very
simple
3
file,
plug-in
right.
One
of
these
is
even
the
readme,
so
it
is
very
simple
what
it
does
is
it
takes
all
the
arguments
that
the
container
has
and
it
runs
them
in
the
shell.
B
B
The
example
code
down
here
shows
you
from
start
to
finish
all
the
code
that
was
required
to
go
from
this
repo
to
actually
running
the
plugin.
So
I
want
to
show
you
each
of
those
files
right.
We
need
an
image
first
and
foremost,
so
we're
gonna
be
running
a
kubernetes
container,
so
we
build
the
image
and
we
push
the
image
to
an
accessible
location.
That
docker
file
can
be
very,
very
simple
in
this
particular
case,
we're
starting
from
Debian
we're,
adding
in
and
some
useful
tools
right.
B
If
this
is
gonna
be
running,
commands
I
wanted
it
to
have
some
tools
in
there
so
like
we
have
put
cube
control,
we
put
curl
and
some
net
tools
in
there,
and
then
we
have
one
simple
script:
okay,
so
get
get
stuff
in
my
container
run
the
work
now
that
running
script
is
very
simple
as
well
right.
This
is
all
of
my
program
logic,
but
really
the
key
point
that
you
should
be
aware
of
is
this.
This
one
function
is
what
is
satisfying
the
contract
with
sonically.
B
This
is
what
makes
it
really
able
to
be
run
with
sahn
ability,
and
that
is
whenever
you're
done
it
calls
this
function.
It
tours
up
a
directory
and
it
writes
the
name
of
that
tor
bowl
in
a
file
called
done
and
that's
it
then
sauna
Bui
knows
to
grab
that
tar
ball
and
report
it
back
to
the
aggregator.
Everything
else
is
bash
specific
about
handling
the
arguments
and
running
them,
which
is
not
important
for
this
demo.
B
So
now
we
have
an
image
and
it's
available
for
use,
but
we
have
to
figure
out
how
to
tell
Sanne
Billy
to
run
it.
If
you
remember
in
that
sauna,
Bowie
gin
output,
there
was
a
plug-in
definition
of
some
sort
that
mentioned,
which
image
to
run,
how
to
run
it.
What
are
the
environment
variables
and
that
sort
of
thing
it
used
to
be
that
you
had
to
copy
and
paste
that
and
edit
it
each
time
you
wanted
to
make
a
plug-in
or
write
that
boilerplate
from
scratch.
B
B
So
this
produced
this
all
the
animal
that's
necessary
to
feed
into
sauna
buoy
as
a
plug-in.
It
is
a
job
plug-in.
Here's
the
name
here
are
some
arguments:
I
want
to
use
this
image,
which
I've
so
you
know
presumably
published
somewhere
and
that's
it
more
or
less.
Okay,
all
you
don't
have
to
worry
about
any
other
detail
other
than
what
image
you
want
to
run
and
how
you
want
to
run
it,
and
so
I've
actually
published
the
image
somewhere
and
I've
saved
that
animal
to
a
file.
B
B
But
now,
instead
of
just
running
this
basic
sauna,
Bui
results
command.
We
can
still
leverage
that
all
the
functionality
is
the
same,
but
we
just
have
to
specify
that
we're
wanting
the
plug-in
hello
world
and
it
has
to
what
it
calls
still
tests
or
two
things
that
have
passed
and
if
you
recall
I
said
sauna,
Bui
is
trying
to
be
aware
of
the
types
of
results.
You're
processing
here-
that's
just
data.
It's
just
output
into
a
few
different
files.
It's
not
j-unit!
B
It's
really
not
tests
that
pass
or
fail,
but
it's
still
useful
because
we
can
specify
we
can
actually
get
that
data
out
without
opening
up
the
tarball
by
using
a
detailed
mode
here
for
every
single
one
of
those
files
of
results
that
it
found
it's
gonna
print.
The
file
results
with
a
prefix,
letting
you
know
which
file
it's
printing
right
and
that's
optional.
You
can
remove
that
so
that
you
can
pipe
it
to
other
tools
or
save
it
to
a
file.
B
But
you
can
see,
we
ran
were
a
command
that
said
hello
world
and
we
did
another
one
that
ran
cubed
control
cluster
info
from
within
the
cluster,
and
so
this
is
powerful.
You
can
do
this
to
gather
all
sorts
of
other
custom
data
or
run
your
scripts
and
if
you
did
it
as
a
daemon
set
you'd
run
that
on
every
node
make
sure
that
every
node
can
reach
the
API
server
or
check
what
the
network
settings
are
on
every
node.
You
know
whatever
your
custom.
Scripts
of
interest
are
now
this
particular
plug-in.
B
Had
plenty
of
power
behind
it
already
right,
we
just
wanted
to
run
a
script.
It
didn't
need
any
sort
of
privilege
status.
One
of
the
thing
that
certain
plugins
run
into
is
that
they
have
a
problem
because
they
need
to
run
in
a
special
environment.
They
need
to
be
in
a
secured
or
privileged
security
context,
or
they
need
to
be
in
the
host
PID
space
and
that
used
to
be
a
blocker
for
plugins,
but
we've
recently
added
a
an
ability
to
specify
any
pod
options
that
you
want.
B
So
if
we
do
something
to
ijen
plug-in
like
we
had
done
before,
but
we
add,
another
option
show
default
pod
spec,
there's
this
new
optional
section
of
the
plug-in
definition
which
it
will
take
all
of
these
pod
options
for
when
it
launched
as
your
container
right,
we
still
need
to
put
our
sauna
buoy
sidecar
into
the
pod
so
that
it
reports
the
results,
but
all
the
other
pod
options
that
you
need.
You
can
specify
here.
B
So
if
you
need
to
specify
host
PID,
post
IPC
or
something
else
right,
all
of
that
is
available
now
to
you
right.
This
can
be
very,
very
powerful.
Also
if
you
need
other
containers
right,
maybe
you
don't
have
a
single
container
test,
but
maybe
you're
running
a
database
in
one
container,
and
you
know
another
application
logic
specific
thing
in
a
container.
You
can
add
all
of
those
there
if
you
need
we're
just
gonna
add
into
that
our
sidecar
worker
and
you're
still
gonna
report
results
in.
B
B
Now
the
demos
that
we
have
been
doing
so
far
we're
assuming
everything
went
well
and
I
hope
it
does.
It's
really
nice,
but
everything
goes
well,
but
when
you
do
have
failures,
you
need
to
know
how
to
dig
deep
into
the
logs
of
these
runs.
Today
knows
what
those
problems
are,
and
it's
not
really
does
a
few
things
to
help
you
so
I
want
to
show
you
what
those
are
we're.
B
Gonna
first
identify
failures
and
I'm
going
to
look
through
that
tarball
a
little
bit
more
to
see
what
else
can
you
discern
about
the
cluster
and
then
specifically
cuz?
This
is
a
very
common
question
in
our
slack
channel
is
what
do
we
do
at
the
end?
What
how
can
we
tie
this
back
to
the
upstream
code
because
often
times
you
do
have
to
go
all
the
way
back
to
the
code
to
diagnose
the
real
cause
of
the
problem?
So
let's
take
a
look
at
how
we
can
do
that.
B
Command
and
here
I
have
a
tarball
from
a
failed
run,
just
as
an
example,
and
you
can
see
when
you
do
have
a
failure.
The
basic
behavior
is
just
to
list
the
names
of
the
tests.
Okay,
this
is
useful,
so
that
you
can,
you
may
even
just
know,
since
you
launched
your
cluster,
why
that
test
failed
right.
Maybe
you
were
experimenting
with
a
certain
configuration,
configuration
object
and
now
that
feature
failed
right.
B
That
might
be
all
the
information
you
need,
but
if
you
need
additional
information,
you
can
dig
a
little
more
deeply
into
it
again
by
using
the
detailed
mode
here.
This
is
going
to
be
running
details
for
every
test
right.
So
if
you
run
this
right
now,
it
will
actually
run
details
for
3,000
tests,
which,
if
that's
what
you
want
there
you
go.
B
That's
probably
not
gonna
be
useful
for
you
in
this
case,
but
the
point
is:
is
that
since
it's
giving
all
that
detailed
information
on
every
test,
you
can
also
pipe
that
to
other
tools
like
JQ,
so
that
you
can
filter
for
whatever
it
is.
You
need
whether
it's
based
on
features
or
based
on
the
tag
of
the
test
here,
if
we
just
want
to
see
the
ones
that
have
a
failed
status,
it's
really
just
one
test.
This
is
a
bunch
of
detailed
information,
asana
bui
pulled
from
the
logs
for
you.
Well,
this
is
one
object.
B
Here's
the
name!
It
failed,
and
this
was
the
file.
If
you
want
to
look
at
the
XML,
that's
where
we
pulled
all
this
information
from,
and
it
has
a
failure
message
and
it's
just
captured
system
out
all
this
comes
from
the
upstream,
but
because
it
was
reported
and
j-unit
format,
it's
able
to
integrate
pretty
seamlessly
with
sauna
buoy.
B
Now
you
may
just
be
able
to
look
at
this
information
and
figure
out
why
it
failed
personally,
I
usually
open
up
the
logs
if
I
get
to
this
point,
so
this
gives
us
an
opportunity
to
open
up
that
tar
ball
when
you
open
it
up.
You
see
data
about
your
hosts
metadata
about
the
run
itself,
information
about
the
plugins,
the
pod
logs,
the
resources
and
even
some
basic
data.
Just
about
that
at
the
API,
a
server
itself
they're
going
to
go
through
a
few
of
these.
B
Not
all
of
the
data
is
gonna,
be
covered
in
this
webinar,
but
hopefully
enough
to
help
you
debug
some
of
the
most
common
types
of
problems
that
you
run
into
first,
the
place
I
always
look
is
gonna,
be
in
the
plugins
right,
there's
folders
for
each
of
the
plugins
that
you
ran
we're
digging
into
a
failure
for
the
ET
plugin
I
want
to
be
looking
at
the
global
results
ring.
It
was
just
run
once
so.
They're
considered
global
results,
and
here
you
can
open
up
the
logs.
B
B
Okay,
so
there's
some
sort
of
problem
running
this
test
in
my
environment,
but
you
could
tell
by
the
name
there
were
all
sorts
of
feature
tags,
and
it's
no
surprise
to
me
that
this
test
fails
on
my
my
system,
but
if
this
doesn't
work
for
you
and
you're
still
kind
of
scratching
your
head
about
why
it
might
fail,
you
can
try
and
correlate
it
with
other
things
that
are
going
on
in
the
cluster.
At
the
same
time,
so,
like
I,
said
we
always
gather
the
systemd
logs.
B
Maybe
there
is
just
a
systemic
problem
on
your
node.
If
you
want
to
dig
into
those
here,
you
go,
this
is
what's
actually
happening
in
those
systemd
logs.
If
you
still
can't
find
the
problem,
I
go
to
pod
logs
and
after
the
run
occurs.
After
all,
the
plugins
say
they're
done,
somebody
goes
and
does
a
number
of
other
queries
against
the
cluster
that
are
only
aimed
to
help.
You
understand
the
results
right.
So
if
you
get
a
failure,
you
need
to
dig
into
it.
B
Even
if
it's
a
transient
problem
right,
you
can't
afford
to
look
into
it
later.
You
want
to
gather
those
logs
now
you
want
to
see
what
the
state
of
the
cluster
was
when
that
test
failed.
So
by
default
we
always
gather
the
pod
logs
for
where
we
ran
the
test.
So
this
is
the
hefty
of
sana
Booey
names
space,
but
that's
customizable.
B
You
can
specify
that
you,
for
instance,
want
to
grab
all
the
cubes
system
logs,
that's
another
very
useful,
one
that
you
can
do
and
that's
just
part
of
sending
setting
configuration
options
when
you
do
your
sauna
GUI
run,
but
this
is
useful
because
for
one
you
can
see
the
aggregator
logs.
This
is
useful
because
it's
kind
of
the
brain
behind
the
whole
song
on
a
buoy
run.
If
there
was
a
problem
launching
a
plug-in,
you
should
see
that
message
here
right.
B
That's
a
common
situation,
if
there's
a
permission
problem
right,
oftentimes
the
credentials
that
you
are
using,
aren't
powerful
enough
to
do
everything.
Maybe
it's
trying
to
doesn't
have
permission
to
create
namespaces
or
doesn't
have
permission
to
create
certain
role
bindings
in
the
cluster
oftentimes
you'll
get
that
message
here.
If
it
does
run
everything-
and
this
is
not
providing
clues-
you
can
dig
into
the
logs
of
your
job
right.
We
eventually
cleaned
up
the
run
and
the
plug-in,
but
we
gathered
all
the
logs
before
we
did
that.
Okay.
B
So
here
this
is
just
a
text
file
of
results.
We
already
captured
those
in
our
plugin,
though
so
this
is
not
particularly
useful
in
this
case
right.
Some
plugins
don't
provide
their
logs
as
part
of
the
results
table,
although
so
this
can
still
be
very,
very
vital,
but
we,
you
also
get
to
see
the
container
logs
for
the
sauna
GUI
worker.
This
is
really
useful
if
you're
having
a
problem
submitting
the
results
back
to
the
aggregator
right,
maybe
there's
some
network
problem
and
the
connections
keep
getting
reset.
That
would
show
up
here.
B
Another
option
for
you
to
dig
into
is
the
resources.
This
is
gonna,
just
be
digging
it
to
all
the
JSON
metadata
that
exists
existed
in
sed,
describing
the
objects
in
the
cluster.
So,
for
instance,
you
can
look
at
all
the
pods
and
the
heftier
sauna,
GUI
namespace
there's
a
lot
of
information
stored
on
these
objects.
That
can
be
useful
in
debugging.
B
It
can
show
you
the
exit,
statuses
or
the
termination
and
reasons
for
the
different
containers,
and
it
can
also
just
help
validate
that
you
ran
what
you
thought
you
ran,
so,
for
instance,
this
was
running
my
a
container
and
I
can
see
the
image
here,
but
maybe
I
wasn't
sure
what
image
version
I
was
using.
Why
I
can
check
that
here?
In
addition,
I
can
check
what
was
that
environment
variable
did
I
really
run
the
test
that
I
expected
to
run
so
on
and
so
forth.
B
There's
a
lot
of
information
here
that
can
often
point
to
the
problem,
but,
let's
just
for
sake
of
argument,
say
that
still
doesn't
solve
your
problem,
you're
still
having
mysterious
timeout
or
EOF
error,
often
times
you
do
get
to
that
point.
Where
for
the
end-to-end
tests
you
just
have
to
go
back
to
the
upstream
code.
Now,
when
that
happens,
if
you
look
in
the
logs
there's,
actually
a
lot
of
important
information
about
what
version
of
the
tests
are
running,
what
version
of
the
API
server
it's
running
up
against?
B
You
can
find
those
kind
of
sprinkled
throughout,
but
this
is
usually
where
I
grab
them
from.
When
it's
running
things,
it
will
tell
you
full
log
lines
about
what
file,
what
line,
but
it's
important
that
you
don't
just
go
to
the
kubernetes
repo
and
go
to
this
file
in
line
that
won't
be
useful
because,
like
I
said,
those
tests
are
constantly
in
a
state
of
flux,
lots
of
contributions,
updating
tests
or
adding
new
tests.
So
it's
important
that
you
do
one
of
two
things
you
either
go
to
this
tag.
B
B
So,
for
instance,
if
you
go
to
the
kubernetes
kubernetes
github,
the
tags
are
available
from
right.
Here
you
just
be
sure
to
switch
over
to
to
be
looking
for
its
tags
type
in
the
one
you
want,
or
if
you
want
to
get
fancy
you
can
type
in
the
sha
here
just
in
the
URL
as
part
of
the
path
right
tree
and
then
the
short
straw
will
get
you
to
where
you
want
now.
B
Once
you're
there
you
can
plug
in
the
full
file
name
and
path
or
just
navigate
it
through
the
the
github
API
or
web
UI.
If
you
want
I
would
recommend,
if
you're
at
this
point,
that
you
clone
the
repo
so
that
you
can
use
your
IDE
and
jump
back
and
forth
between
functions
a
little
more
easily.
But
in
this
particular
case,
I
was
looking
for
line
347,
and
this
is
the
thing
that
was
failing.
I
can
dig
into
each
one
of
these
things.
How
is
it
initializing?
The
pod
is
that
the
problem?
B
B
Ok,
so
I
want
to
talk
a
little
bit
about
our
road
map.
First,
the
things
that
we've
recently
knocked
off
of
our
bucket
list,
so
to
say,
we've
we've
made
it
easier
for
you
to
run
tests
in
your
air
gapped
environments.
So
if
you
weren't
aware
to
run
the
kubernetes
tests
upstream
or
on
your
cluster,
it's
not
just
a
single
test
image.
That's
required,
because
those
tests
themselves
are
launching
a
lot
of
containers
and
pods
and
doing
all
sorts
of
things.
So
this
command
was
made
to
help,
simplify
and
grab
all
those
images
you
need.
B
You
could
pull
all
the
test
images
locally.
You
can
push
them
to
behind
some
sort
of
available
registry
so
that
even
in
your
air
gapped
environment
you
can
grab
and
run
those
tests.
Then
we've
tried
to
make
writing
and
using
plugins
more
simple
you've
seen
how
you
can
generate
a
plug-in
definition.
Now
from
the
CLI.
You
don't
have
to
write
that
boilerplate
code
anymore,
and
then
you
run
those
plugins
just
from
the
command
line.
Now
you
don't
have
to
edit
yamo
directly.
B
In
addition,
we've
made
writing
more
powerful
plugins
possible
prior
to
making
the
pod
spec
completely
customizable.
Any
feature
that
you
wanted
surfaced
had
to
be
wired
through
sauna
buoy
entirely
so
implementing
post
PID
would
have
been
one
feature.
Implementing
host
IPC
would
have
been
a
separate.
B
Every
single
pod
field
would
have
been
a
new
feature,
a
new
PR,
a
new
option,
but
in
one
fell
swoop
all
those
options
are
now
available
to
you,
so
you
can
run
plugins
in
exactly
the
environment
you
want,
then
we've
tried
to
make
reporting
the
results
for
even
custom
plugins
simplified
by
the
sauna.
Buoy
results
command
again.
I
really
want
to
emphasize
that
this
is
extensible.
B
There's
a
lot
of
custom
production
environments
where
the
Indian
tests
just
simply
won't
run
because
of
assumptions.
The
framework
is
making
and
we're
trying
to
work
with
the
upstream
community
to
simplify
and
streamline
that
process
and
then
a
very
requested
feature
or
status
updates.
The
end
end
test
in
particular
can
run
for
hours
at
a
time
and
Sue
might
your
custom
plugins.
We
won
away
for
through
the
CLI
you
to
see
the
progress,
how
many
tests
are
completed.
B
If
you
see
that
you're
taking
10
minutes
per
test,
there's
probably
some
timeout
issue
and
you're
not
making
the
progress
that
you
think
there's
no
point
in
waiting
three
and
four
hours
to
get
those
results.
So
that's
gonna
be
very
useful
when
that's
implemented,
probably
around
the
time
for
v17,
so
that
corresponds
to
kubernetes,
be
1.17,
and
after
that,
our
priorities
in
the
future
gets
a
little
more
fuzzy.
But
we
do
know
that
we're
looking
at
supporting
Windows
nodes
and
running
tests
on
those.
This
is
the
big
thing.
B
The
upstream
community
is
working
on
in
kubernetes,
getting
all
the
conformance
tests
to
support
different
types
of
nodes
and
having
windows
parallel
and
they're,
trying
to
work
out
what
conformance
means
on
those.
Likewise,
we
will
have
lots
of
work
to
support
those
windows
nodes,
so
you
can
definitely
pitch
in
and
help
whether
it's
lots
of
small
bug
fixes
or
helping
us
figure
out
what
and
how
to
run
the
tests
for
all
these
windows
related
workflows.
B
If
you
want
to
clone
or
fork
or
submit
a
PR,
you
can
follow
us
on
Twitter.
We
often
release
information
on
our
upcoming
features
and
releases
and
our
brand
new
website
that
features
our
all
our
documentation,
as
well
as
blog
posts,
that
kind
of
walk
through
the
new
features
as
we
introduce
them.
So
again,
sana
Bui
is
doesn't
own
all
those
upstream
parts,
whether
you
have
a
question
about
conformance
or
tests.
B
We
don't
own
those
but
we're
happy
to
help
a
lot
of
people
that
work
on
sana,
buoy
or
USANA
buoy,
we're
all
working
in
the
same
area
right
we're
trying
to
run
the
tests.
Often
the
conformance
tests,
often
for
the
goals
of
getting
scenes
you
have
conformance.
So
whatever
your
question
is,
hopefully
we
can
help
and
we'll
see
you
there.
Thank
you.
A
B
Question
so
can
you
run
sauna
Bui
against
a
kind
environment?
You
absolutely
can
and
that's
actually
what
all
these
test
results
that
I
was
doing
for
my
demo
are
from
so
I
have
a
git
clusters:
I
have
a
single
node
cluster
right
now
running
and
and
that's
how
I
run
conformance
tests
I,
don't
think
a
single
node
cluster
will
not
pass
conformance,
but
that's
not
kind
fault
to
be
conformant.
A
Right,
I,
don't
see
anything
here,
I
want
to.
Thank
you
all.
Thank
you
so
much
John
for
for
doing
this
presentation
and
I
want
to
thank
you
all
for
joining
today.
Please
join
us
on
slack
and
say
hi.
We
will
be
uploading.
This
recording,
if
you
missed
anything,
we'll
be
uploading
it
to
YouTube
shortly
and
with
that.
Thank
you
all
so
much
for
joining
today
and
have
a
great
rest
of
your
week.
Thank.