►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Intro: CNCF Kubernetes Conformance WG - Dan Kohn, CNCF & Aaron Crickenberger, Google
To learn more: https://sched.co/IExs
A
Maintainer
track
intro
session
for
kubernetes
cncf
conformance
working
group
to
judge
the
audience
here.
How
many
of
us
have
been
involved
in
certifying
at
least
one
of
the
79
certified
kubernetes
distributions.
A
Okay,
so
we
are
going
to
do
the
this
as
an
intro,
but
we
really
are
available
to
go
into
more
detail
and
then
tomorrow,
there's
going
to
be
a
very
interesting
deep
dive
session,
which
is
going
to
be
much
more
of
a
working
session
of
the
working
group
to
decide
you're
on.
A
Okay
to
decide,
particularly
around
windows
conformance,
so
this
has
been
in
the
three-year
history
of
the
cncf.
I
would
argue
the
single
thing
that
has
had
the
most
impact
on
the
cloud
native
ecosystem
and
it's
something
that
I've
personally
been
extremely
involved
in
and
I'm
very
proud
of,
and
so
we
will
walk
through
it.
A
Okay,
yeah,
so
the
certified
kubernetes
program
is
a
conformance
program.
It's
actually
a
little
unusual
compared
to
most
conformance
programs.
There's
no
third
party
lab
that
you
send
things
to
there's
no
big
fee
that
you
pay.
A
There's
there's
an
open
source
software
conformance
suite,
which
is
a
subset
of
the
test,
suite
already
developed
in
kubernetes,
and
it
is
a
way
for
vendors
to
certify
that
their
implementations
of
kubernetes
have
not
broken
broken
any
critical
apis
that
their
specific.
As
we
say,
if
you've
seen
a
roadrunner
cartoon,
acme
kubernetes
engine
still
implements
all
the
functionality
of
vanilla
kubernetes.
A
So
we
launched
this
just
a
little
over
a
year
ago
and
we
are
now
up
to
79
certified
kubernetes
platforms.
A
Advance
is
you
fall
back
and
if
we
can
just
take
a
moment
here,
taylor
can
you
stand
up
please
so
for
everyone
who
always
wonders
what
happens
when
I
yeah
thank,
you
submit
my
results
to
github
and
who
looks
at
those
it
is
all
taylor
and
she's
the
one
who
does
fantastic
work.
Very
politely
saying:
oh,
you.
C
A
B
B
A
Of
sarcasm
in
her
voice,
but
I
really
would
compare
this
program
to
almost
any
other
conformance
or
certification
program
in
history,
maybe
up
there
with
bluetooth
and
wi-fi
in
terms
of
the
breadth
of
the
industry
that
we've
been
able
to
pull
in.
That
is
literally,
everyone
has
signed
on
to
this
program
and
that
there's
some
pretty
clever
aspects
built
into
it
that
really
discourage
people
from
falling
off
that
the
release
train.
A
So
yeah.
We
won't
do
every
piece
here,
but
the
key
parts
are:
you
need
to
be
a
member
of
cncf
or
a
non-profit
organization.
So
if
you're,
just
a
community
distro,
that's
totally
fine,
you
don't
have
to
pay
anything.
You
sign
a
form.
It
has
a
lot
of
legalese
in
it,
but
it
is
hopefully
understandable.
It's
like
three
pages
long
and
then
you
upload
the
test
logs
from
your
conformance
results.
A
So
one
of
the
key
conditions
is
that
it
has
to
be
the
current
release
or
the
previous
one,
and
if
we
have
time
tomorrow,
I
think
we
might
consider
relaxing
that,
where
we're
considering
making
it
the
past
two
releases
because
to
try
and
be
a
little
bit
more
realistic
about
where
people
are
the
the
there's,
these
ideas
built
into
it
there.
A
This
is
a
little
more
complicated,
but
the
idea
behind
it
is
to
say
that
we
don't
want
you
to
be
able
to
take
a
super
old
version
of
kubernetes
and
say:
okay.
This
is
now
conformant
and
just
stick
there
and
having
done
a
conformed
version,
we
don't
want
you
to
just
be
able
to
stick
there
either
and
say:
oh
see,
we're
conforming
forever,
and
so
we
have
a
carrot
and
a
stick,
and
the
carrot
is
the
both
the
certified
kubernetes
logo,
but
also
the
ability
to
use
the
word
kubernetes
in
your
product
name.
A
A
But
you
actually
do
need
to
do
those
new
versions
and
offer
them
for
sale,
and
so
it
really
is
trying
to
avoid
this
fear
of
people
and
then
something
that
we've
actually
seen
in
other
programs
in
the
past
and
then
maybe
the
the
most
clever
and
kind
of
most
subtle
aspect
of
the
of
the
conformance
program.
A
That's
had
the
biggest
impact
is
that
it's
not
just
an
assertion
where
you
can
say:
okay,
well,
fine
in
theory,
somebody
says
that
they
pass,
but
I
could
have
just
copied
somebody
else's
results
and
posted
it
up
there.
A
A
There
is
a
it's
a
universal
consensus
among
this
working
group
that
if
a
vendor
had
claimed
conformance
and
was
not
actually
conformant
that
they
had
changed
something
in
the
production
environment
that
they
were
now
failing,
a
test,
a
test
that
the
that
the
sig
architecture
I'll
say
felt
was
a
valid
test
of
kubernetes,
conformance
that
we
would
decertify
them
and
they
would
then
no
longer
be
able
to
use
the
trademark.
And
if
that
happened,
it
would
actually
be
pretty
big
news.
A
So
this
is
not
like
highlighted
on
the
web
page
or
anything,
but
I
think
is
just
a
really
critical
aspect
of
how
the
program
was
designed.
A
Yeah.
I
think
I
hit
this
so
for
cncf
members,
it's
free
of
charge,
if
you're
a
little
startup,
it's
seven
thousand
dollars
to
become
a
cncf
member.
If
you,
for
whatever
reason,
are
opposed
to
being
a
cncf
member
you're,
welcome
to
just
pay
and
certify
it's
the
same,
seven
thousand
dollars
either
way
and
free
for
for
non-profits
and
I'll.
Take
a
question
and
I'm
going
to
hand
it
off.
A
It
is
a
certified,
so
you're
certified
by
product
and
any
member
of
cncf
is
welcome
to
certify
any
number
of
products
free
of
charge,
and
we
have
at
least
a
couple
cases
huawei
and
ibm
in
particular,
where
they've
certified
both
their
public
cloud
product
and
their
private
cloud.
I
think
there's
one
or
two
others
where
they've
certified
two.
A
A
D
A
The
the
operating
system
is
decoupled
from
kubernetes,
so
you
can
run
kubernetes
on
old
versions
of
rel
in
particular,
but
to
sort
of
get
at
what
you're
asking
with
the
question.
We,
if
you
certify
a
version
and
then
you
keep
certifying
at
least
one
new
version
per
year,
the
old
versions
remain
certified.
So.
A
D
B
I
feel
like
it's
a
super
legit
use
case
and
my
my
suggestion
might
be
to
defer
that
discussion
until
we've
had
time
to
really
think
it
through
in
the
lts
working
group
as
part
of
the
kubernetes
project,
because
I
think
this,
the
the
cncf
here
is
really
about
making
sure
that
the
qualification
and
certification
process
makes
the
most
sense
and
meets
users
needs,
and
I
think
the
actual
set
of
requirements
and
set
of
distinct
users
still
needs
to
be
hashed
out
there,
because
we
do
care
very
much
about
that
regulatory
concern.
B
B
B
C
For
the
gpu
use
case
in
particular,
I
think
that
we'll
probably
solve
that
with
a
with
a
optional
profile
to
it,
to
avoid
basically
that
coupling
that
you're
trying
to
that,
you
don't
want
there,
and
I
think,
just
in
general,
just
to
echo
what
aaron
said
the
work
group
does
care
about
that
use
case.
C
So
probably
the
best
thing
to
do
is
if
you're
interested,
you
know
just
join
the
mailing
list
and
participate,
and
let
us
know
if
something's
kind
of
falling
through
the
cracks
we're
certainly
not
trying
to
cut
that
out,
and
so,
if,
if
we
are
inadvertently,
then
we
can
revise
that.
I
think
yeah.
So
I'm
going
to.
B
Hand
it
off.
Thank
you,
yeah
thanks.
You
just
set
that
on
the
table.
I
guess
so.
How
do
you
actually
run
the
conformance
tests?
There
are
two.
I
know
the
way
to
leave
me
with
the
easy
stuff
dan
there.
There
are
two
different
ways
that
are
documented
and
you
could
maybe
use
a
couple
different
ways,
but
the
idea
is
the
tests
themselves
are
the
same
tests
that
the
project
uses
to
test
itself.
They
are
authored
by
the
project
they
are
owned
by
the
kubernetes
project.
B
B
This
should
be
pretty
easy
and
then
inside
of
the
kubernetes
directory,
you
are
making
just
the
binaries
you
need
to
make,
which
is
the
end
to
end
test
binary
ginkgo,
because
that's
what
we
use
to
run
the
tests
and
then
kubectl,
because
there
is
some
client
interaction
with
the
server.
There
are
some
environment
variables.
Just
to
set
up
like
the
path
to
your
kubernetes
config
and
your
your
con
and
saying
yes,
we're
running
a
conformance
test,
and
then
I've
got
you
two
options
here
to
run
the
conformance
tests.
B
The
first
one
is
the
most
boring
option
and
I
think
it's
actually
what
the
the
second,
what
the
sauna
boy
path
does
and
that's
just
runs
all
the
tests
in
serial.
So
it
runs
every
test
with
the
text
conformance
in
it,
and
it
takes
about
one
to
three
hours,
because
it's
running
every
single
test,
one
after
the
other
and
some
of
those
tests
are
exercising
things
like
exponential
back
off
and
reaching
maximum
delay.
B
Another
perhaps
faster
way
of
doing
it
is
to
run
the
tests
sort
of
twice
once
you
run
just
the
tests
that
have
to
be
run,
serially
because
they're
disruptive
and
then
you
also
run
in
parallel
the
rest
of
the
tests
right
and
that
method.
It's
probably
larger
numbers
now.
But
when
I
did
this
a
couple
months
ago,
it
used
to
be,
I
could
run
in
parallel
like
in
six
to
eight
minutes.
B
That's
basically,
what
slaughterboy
does
soniboy
is
the
command
line
that
sort
of
orchestrates
all
of
this.
But
behind
the
scenes,
it
you
know,
schedules
a
pod
and
a
couple
other
things
to
watch
that
that
pod
is
doing
what
it's
supposed
to
and
the
pod
does
all
that
gobbledygook.
I
showed
you
earlier.
This
is
really
great.
It's
super
user
friendly!
It's
basically
just
these
two
lines.
It's
super
easy.
B
So
it's
important
to
me
that
we
use
we
sort
of
support
both
of
those
use
cases,
but
this
is
an
extremely
concise
way
of
doing
it.
The
other
thing
I'll
mention
about
this
is
that
haptio
built
on
a
buoy.
They
also
built
the
image
that
has
all
the
gobbledygook
packaged
up
in
there
because
they
needed
to
do
it.
Super
expediently
we'd
really
like
to
see.
B
If
we
can't
have
the
project
actually
own
this
image
instead,
because
if
the
project
owns
the
definition
of
what
the
tests
are
sure
seems
like
the
project
could
package
up
the
definition
of
all
of
those
tests
into
an
image
we
had
work
started
on
doing
that
in
113.
I
actually
don't
know
authoritatively
whether
or
not
it
landed
by
default
as
part
of
113.
B
So
maybe
it
did
if
you're
watching
this,
maybe
you
can
go
verify,
but
I
can
guarantee
it'll
be
done
by
114,
since
I'm
leading
that
release
process,
question
yeah
we're
doing
questions
in
the
middle
of
this.
D
B
It
runs
the
same
e2v
test
binary
inside
yeah.
That's,
I
suppose
the
other
thing
is
sauna
boy.
Only
cuts
releases
like
on
release
boundaries,
whereas
cubeup
kind
of
recompiles
everything.
So
that's
how
we
use
it
to
keep
track
of
conformance
tests
as
they
are
added
along
the
way,
whereas
sauna
buoy
tends
to
be
released
like
right
on
a
release
boundary.
B
That's
why,
for
example,
the
cube
test
way
of
standing
up
a
cluster
is
what
got
the
first
certification
for
112,
because
sanovo
had
to
wait
a
while
to
cut
the
appropriate
image
for
112
certification.
B
So
sauna
boy,
the
the
utility
itself,
I'm
not
sure
I
want
to
be
an
upstream
dependency,
but
the
image
that
it
schedules
to
run
the
conformance
tests
is
what
we
we
did.
We
started
work
on
that,
like
the
the
code,
dims
actually
landed
some
of
the
codes
code
to
to
build
that
image,
but
fully
integrating
it
into
everything
and,
as
part
of
the
set
of
release
artifacts
that
we
we
provide
with
kubernetes,
I'm
not
sure
if
we
actually
got
that
far
by
default.
B
I
also
wanted
to
talk
a
little
bit
about
the
progress
we
have
made
in,
like
what
does
conformance
actually
mean
so
conformance.
We
define
as
a
set
of
tests
that
interact
with
the
cluster
as
an
end
user
would
so
that
it's
a
set
of
end
user
behaviors,
visible
as
an
end
user,
would
see
them
and
we've
been
working
on
trying
to
improve
the
set
of
behaviors
or
use
cases
that
we're
exercising
and
so
part
of
that
is
well.
How
do
we
actually
add
the
tests?
Part
of
that
is,
how
do
we
know
like?
B
B
B
B
So,
after
that,
we
decided
to
find
ways
to
measure
whether
or
not
we
were
headed
in
the
right
direction.
We
didn't
actually
have
a
good
way
of
measuring
behavior
behavior
is
a
pretty
abstract
concept.
So
what
if
we
instead
focused
on
api
coverage
and
also
look
at
something
like
code
coverage
code
coverage,
you're
pretty
familiar
with
from
unit
testing,
but
what
if
we
could
do
the
same
thing
with
the
end-to-end
tests
that
we
run
as
conformance
tests.
C
C
B
That
says,
we
guarantee
this
is
really
really
fast
and
super
performant,
and
there
are
certain
use
cases
that
supports
that
a
raspberry
pi
cluster
doesn't
but
at
present
no
pat,
okay.
So
just
the
the
process
to
add
tests,
I
think
I
tried
to
sort
of
walk
through
some
of
the
requirements
the
little
asterisk.
There
is
because
I
have
lots
of
slides
on
very
detailed
examples
of
each
of
these
requirements,
but
around
all
of
that
you,
you
write
an
end-to-end
test
for
your
feature.
B
This
is
just
kind
of
what
you
should
do
during
development.
Anyway,
then,
you
should
document
what
that
end-to-end
test
does
yes
developers,
you
must
actually
document
your
code
and
then,
after
it's
demonstrated
to
meet
the
requirements
of
like
being
well
written
and
not
doing
anything
in
a
flaky
manner
and
using
the
appropriate
set
of
features,
propose
it
to
sig
architecture
that
it
be
promoted
to
conformance.
So
why
sig
architecture,
sig
architecture,
is
the
special
interest
group
that
most
appropriately
defines
what
kubernetes
is
they
have
a
number
of
sub-projects
around
like?
B
How
do
we
add
new
enhancements
to
kubernetes
through
the
cap
process?
How
should
we
organize
the
code
in
the
code
base
through
the
code
organization
subproject
and
also
what
are
the
full
set
of
things
that
make
the
most
sense
for
conformance
the
list
of
people
who
can
actually
make
that
decision
is
small
right
now
and
we
are
looking
to
grow
it
and
part
of
growing.
B
It
is
going
through
this
process
a
lot
and
having
people
review
these
tests
and
make
sure
we've
got
all
the
requirements
written
down
to
the
point
where
it's
not
just
like
a
very
thorough
review
by
a
person,
but
going
through
just
a
pretty
quick
and
easy
checklist.
Did
you
still
have
a
question,
patrick.
D
B
So
basically,
the
the
idea
was
some
of
the
tests,
as
written
today
are
like
a
couple
hundred
lines
of
code.
That's
super
verbose
because
that's
just
the
way
goes
formatting
works
and
it
can
be
difficult
for
someone
who
is
non-technical
to
at
a
glance
read
that
test
and
quickly
grok
what
it's
trying
to
do.
So
what
if
we
instead
had
like
a
one
to
three
sentence,
description
that
tried
to
describe
like
what's
the
setup?
B
What's
the
logic
being
executed
in
rfc,
2119
compliant
language,
you
know
the
one
that's
like
must
in
all
caps
should
in
all
caps,
and
those
actually
mean
things
like
use
that
kind
of
language.
So
it
sounds
really
formal,
but
does
actually
use
well
understood
definitions
for
those
things,
and
then
from
that
we
can
use
a
tool
that
generates
a
set
of
documentation.
That's
like
these
are
all
of
the
tests,
and
this
is
what
they
do,
so
you
can
kind
of
get
close
to
that.
What
does
conformance
actually
do
thing?
B
B
It's
in
the
conformance
requirements.
It
actually
specifies
the
format
that
you're
supposed
to
do
the
the
comment.
So
it
says
something
about
where
the
test
name
is
supposed
to
go.
What
release
the
test
was
added
in
and
then
what
the
description
is
supposed
to
be
so
here
in
graph
form,
is
the
number
of
conformance
test
cases
that
we
have
added
to
the
kubernetes
conformance
test
suite
over
the
past
year.
This
is
a
little
out
of
date
because
it
was
before
113
was
cut,
but
it's
basically
the
same
number
of
tests.
B
It's
cool
how
it
goes
up
into
the
right
here.
It
is
in
text
form
and
I'm
basically
just
showing
this
to
demonstrate.
How?
If
you
want
to
read
these
slides
later,
because
they're
posted
online,
you
can
see
all
the
exact
tests
that
we
added,
but
I
was
pretty
happy
with
the
fact
that
the
text
actually
overflowed
the
slide.
We
added
so
many
in
113
that
was
really
great
and
the
bulk
of
the
tests
were
around
cube.
B
Ctl
client,
which
was
also
great
end
users,
do
often
use
coupe
ctl
cube
cuddle
if
you
will
to
interact
with
kubernetes
clusters.
B
However,
that
progress-
maybe
doesn't
look
so
great
if
you
look
at
the
number
of
conformance
tests
relative
to
say
the
number
of
tests
that
we
use
to
block
the
release,
and
this
isn't
even
actually
the
full
number
of
tests.
This
is
just
the
tests
that
run
in
one
of
the
30
or
so
jobs
that
could
potentially
block
the
release.
These
are
all
of
the
tests
that
are
non-disruptive,
non-flaky
non-slow,
don't
don't
exercise
a
feature,
that's
optional
or
beta,
and
you
can
see
there's
a
lot
more
of
them,
so
it
could
be
like
what.
B
It
turns
out,
if
I
look
by
api
coverage
as
one
of
the
proxies
for
behavior,
that
those
release
blocking
tests
don't
really
add
a
whole
bunch
more
coverage.
It's
like
it's
an
entire
extra
10
percent,
which
means
maybe
there
is
some
room
to
grow.
Maybe
there
are
some
features
to
grow,
but
it
also
means
like
there's
a
whole
other
40,
that's
not
being
covered
right.
Now,
all
right,
this
graph
is
kind
of
colorful.
Let's
actually
explain
it
a
little
bit
more
than
that,
so
each
of
the
colors
signifies
a
color
of
the
rainbow.
C
B
A
release
so
the
red
one
is
the
latest
release
and
then
the
cooler
you
get
the
older
you
get
and
the
release
and
then
going
down
the
graph.
We
can
see
that
the
very
top
entry
is
the
api
coverage
for
every
single
api
endpoint
available.
So
a
by
api
endpoint,
I
mean
a
path
and
a
verb
like
post
to
api
v1,
pods
foo,
then,
instead
of
focusing
on
all
api
endpoints.
What
if
we
just
focused
on
the
stable
api
endpoints,
which
are
unchanging
and
is
one
of
the
requirements
for
conformance
and
then
beyond
that?
B
What
if
we
focused
on
the
stable
core
api
endpoints?
So
core
means
things
like
pods
things
like
name
spaces,
things
that
might
end
up
in
the
nucleus
layer
of
kubernetes.
If
you've
seen
that
architecture
diagram
they're
the
things
upon
which
all
the
other
pieces
of
kubernetes
are
really
built
like
you
can't
really
have
a
deployment
without
a
replica
set,
you
can't
really
have
a
replica
set
without
pods.
You
can't
really
have
services
without
end
points,
things
of
that
nature.
B
The
lower
three
tests
are
the
release
tests
instead
of
the
conformance
tests.
So,
like
I
said,
I
don't
think
they
provide
a
ton
of
additional
coverage.
The
other
thing
I
wanted
to
denote
with
the
colors
is
that
yay
progress
as
you
go
down
as
we
release
more
things,
we
get
more
coverage,
we
are
making
progress.
This
is
good.
B
B
This
is
generated
by
api
snoop,
a
tool
written
by
hippie
hacker
over
here,
contracted
by
cncf
over
there
and
the
sunburst
graph
sort
of
shows
if
the
inner
ring
is
the
level
the
outer.
The
the
second
ring
is
the
api
group
and
then
the
ring
beyond
that
is
all
of
the
different
names
or
verbs.
B
B
B
So
if
we
look
at
potentially
all
of
the
api
coverage
from
every
potentially
qualifying
test
so
like,
let's
assume
all
of
the
release
tests
qualified
that's
sort
of
the
delta
that
we
would
get
with
this
graph,
it
looks
better
it's
up
to
43.3
percent,
but
we
still
have
a
lot
to
cover.
We
still
actually
need
to
write
more
end-to-end
test
cases,
so
how
you
can
help
you
can
help
contribute
tests
that
exercise
pod
functionality,
because
I
jammed
three
talks
into
one
in
this.
B
I
guess
I
kind
of
lost
the
flow
here,
but
suffice
to
say
one
of
our
priorities
from
a
conformance
perspective.
As
we
look
to
improve
coverage,
is
we
want
to
make
sure
that
the
pod,
the
fundamental
building
block
of
kubernetes,
is
what
we
have
the
most
coverage
on?
You
can
change
your
network.
You
can
change
your
run
time.
You
can
change
your
storage,
but
a
pod's
got
to
work
the
same
way.
B
Every
time,
so
we
got
to
make
sure
that
all
of
those
behaviors
you
expect
from
pods
from
their
the
container
probes
that
monitor
them
to
the
way
volumes
work
within
them
to
when
they
start
and
when
they
die
and
their
the
sidecar
stuff
works.
All
okay,
like
we
have
to
make
sure
all
of
that
works.
You
could
also
help
by
categorizing
the
existing
set
of
tests
that
are
labeled
with
conformance
some
of
them
don't
meet
the
requirements.
In
that
finely
detailed
list
I
listed
out,
you
could
run
conformance
tests
against
your
own
kubernetes
cluster.
B
You
might
find
there
are
places
where
it's
not
working
in
your
cluster
and
we
can
have
a
discussion
about
whether
that
means
your
cluster
is
configured
incorrectly
or
maybe
it
shouldn't
be
a
conformance
test.
Great
example
of
this
is
those
folks
who've
had
to
run
conformance
tests
in
air-gapped
environments.
That
is
a
totally
valid
use
case.
We
had
to
figure
out
how
to
make
that
work.