►
From YouTube: Adventures in Conformance - Aaron Crickenberger, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Adventures in Conformance - Aaron Crickenberger, Google
What is Kubernetes, really? Documents may say one thing, but running code will always win. Our community thus defines what Kubernetes is via conformance tests that exercise functionality expected of any given Kubernetes cluster. Last year the CNCF introduced the Certified Kubernetes Conformance Program that uses these tests to certify whether a given Kubernetes distribution or platform is behaving as expected. What have learned as over 60 vendors have certified their Kubernetes? This talk will walk you through the progress we’ve made over the past year in improving the fidelity of conformance tests, and how they are integrated into both upstream and downstream development. The audience should leave this talk knowing what guarantees conformance tests provide, how they can use them, and how they can contribute.
To learn more click here: https://sched.co/FuKh
A
All
right
we're
gonna
go
ahead
and
get
started.
That
was
really
fun.
Usbc
connectivity
is
great
so
because
of
that,
my
talk
is
going
to
be
five
minutes
shorter
when
I
practiced
this
it
took
me
50
minutes,
so
apologies
if
I
go
really
fast
through
some
of
this.
So
welcome
to
my
talk,
adventures
in
conformance
I'm
Aaron
frickin,
burger,
I
work
for
Google.
As
of
a
couple
months
ago,
I
worked
for
Samsung
SDS
before
that
I
started
working
on
kubernetes
prior
to
its
launching
1.0.
A
Around
the
same
time,
Janet
Kuo
started
I
started
the
cig
testing
group
I
started
the
conformance
test
framework
around
version
1.2
I'm,
currently
a
member
of
the
steering
committee.
But
the
reason
I
want
to
talk
to
you
today
is
as
a
member
of
the
kubernetes
community.
So
when
I
say
we
I'm
not
talking
about
my
employer
I'm
talking
about
all
of
us,
you
are
all
members
of
the
communities
community
as
well,
so
I'm
going
to
talk
about
what
we
want
and
what
we
should
do.
A
lot.
A
Okay,
so
Dan
talks
during
his
keynote
about
how
kubernetes
has
crossed
the
chasm
of
uncertainty
early
on.
We
realized
there
was
this
primordial
soup
of
container
orchestration
implementations
and
we
needed
one
of
them
to
win
to
become
the
de
facto
standard.
In
order
for
this
to
happen,
the
kubernetes
project
chose
to
prioritize
moving
fast
and
intentionally
take
on
technical
debt.
In
order
to
do
so,
this
meant
that
very
often.
A
A
Since
we
have
crossed
the
chasm
we
are
now
trying
to
make
kubernetes
boring,
as
Janet
said
in
her
icky
note.
So
a
lot
of
this
involves
defining
extension
points
and
extracting
functionality
out
of
Nettie's.
So
this
is
why
we
now
have
container
networking
interfaces,
container,
runtime
interfaces
and
container
storage
interfaces,
we're
also
working
to
extract
the
cloud
provider
functionality
outside
of
core,
and
as
we
do
this,
we
are
focusing
on
stability
and
reliability
of
kubernetes
as
a
whole.
A
We
don't
want
to
add
new
features
while
we're
doing
this-
and
it's
important
through
all
of
this-
that
we
focus
on
portability
of
workloads
to
make
sure
that
if
I
write
an
application
and
deploy
it
to
one
kubernetes
cluster,
that
it
works
just
as
well
on
another
kubernetes
cluster.
This
is
how
we
grow
and
mature
as
a
project.
A
Finally,
once
we
may
make
kubernetes
boring,
we
want
to
keep
it
boring,
and
this
is
where
conformance
comes
into
play.
Conformance
is
the
boring
thing
that
allows
cluster
operators
to
know
is:
is
my
cluster
a
certified
kubernetes
cluster?
Is
it
conformant
that
I
deploy
it?
The
right
way?
Is
it
configured
correctly?
It
will
allow
application
developers
to
know
is
my
application
is
my
workload
actually
portable?
A
A
But
really
a
lot
of
people
think
about
kubernetes
as
a
set
of
behaviors
or
functionalities.
I,
don't
care
that
I
have
a
deployment
I,
don't
care
that
I
have
an
ingress
controller,
I
care
that
I
have
an
application
which
will
automatically
scale
in
response
to
demand
so
that
traffic
will
route
from
the
outside
Internet
to
my
pods
through
a
service
and
the
pods
will
be
scaled
via
a
horizontal
pod,
autoscaler
and
the
whole
thing
will
be
managed
by
a
resource
quota
and
limited
in
the
range
all
that
to
say
I.
Don't
care
about
pods
I!
A
Don't
care
about
replica
sets
I,
don't
care
about
ingress,
I
care
about
the
behaviors
that
are
visible
to
an
end-user,
and
this
is
what
conformance
means
it
means
we
want
behaviors
that
end-users
can
see
that
work
on
any
given
conformant
cluster.
It's
important
that
I'm
talking
about
behavior
and
functionality
here,
not
just
API
interaction.
There's
a
lot
of
behavior
and
functionality.
I
want
from
kubernetes.
That
is
more
than
HTTP
requests
and
responses.
It
is
running
a
container
and
seeing
that
container
do
things
right:
data
someplace,
making
requests
to
other
services.
A
So
it's
important
that
we
look
beyond
just
api's
another
way
of
thinking
about
this
is
architectural
II.
We
like
to
break
kubernetes
up
into
these
layers
and
the
bottom
layer
has
many
different
components
which
can
vary,
including
the
container
runtime,
we're
the
different
kinds
of
networking
or
the
different
kinds
of
volumes
we
can
use.
On
top
of
that,
sits
the
nucleus
layer
where
things
like
config
maps
and
pods
will
live
fundamentally,
if
those
don't
work,
all
of
the
other
layers
on
top
can't
work
either,
and
when
we
focus
on
conformance.
A
A
A
Is
that
true?
What
what
do
we
mean
when
we
say
that
so
I
want
us
to
as
a
community?
You
need
to
add
more
tests
to
exercise
more
functionality,
so
that
conformance
covers
meaningful.
Behavior
I
want
us
to
focus
on
pod
functionality,
because
the
pod
is
the
fundamental
abstraction
of
kubernetes
I.
Think
Brendan
Byrne
said
this
well
in
his
keynote
when
he
talked
about
how
people
cared
about
where
the
pods
ran
not
necessarily
about
the
nodes
or
anything
we
want
to.
I
also
talked
about
measuring
functionality
and
covering
functionality,
that's
difficult
to
measure
directly.
A
I
don't
have
time
to
go
over
all
of
these,
but
please
feel
free
to
ask
at
the
end,
but
essentially
these
are
requirements
that
tests
work
across
all
kubernetes
clusters,
so
they
must
be
using
features
that
are
standard
and
default
and
must
work
for
all
providers.
They
ideally
should
be
capable
of
running
in
parallel.
They
should
not
require
direct
access
to
Internet,
sorry,
direct
access
to
network
interfaces
or
run
as
root
and
I.
Think
importantly,
here
we
should
require
that
the
tests
work
without
internet
access.
A
We
think
an
air-gapped
compare
Nettie's
cluster
that
doesn't
talk
to
the
Internet
is
just
as
conformant
as
a
kubernetes
cluster
that
lives
in
the
cloud
and
is
soaked
in
the
Internet.
Finally,
we
want
to
make
sure
that
that
test
is
consistent
and
is
signed
off
by
the
architecture,
special
interest
of
kubernetes,
to
make
sure
that
the
test
describes
what
we
think
kubernetes
to
be.
A
You
can
see
that
these
bars
go
up
and
to
the
right.
That's
great
I
joined
Google
and
started
working
on
this
around
version
111,
so
I
like
to
think
that
I
had
an
impact,
and
each
of
these
bars
corresponds
roughly
to
a
quarter
so
version
1:9
was
December
of
last
year.
Version
113
is
the
release
of
kubernetes.
We
are
working
on
right
now
and
I
think
we
are
trending
in
a
good
direction.
A
If
you
want
to
know
what
the
specific
tests
aren't
here,
they
are,
but
I
don't
have
time
to
walk
through
them.
Really.
The
point
of
these
slides
is
to
show
what
those
bars
look
like
as
text
and
to
brag
about
that.
I
can't
even
fit
the
tests
on
this
slide.
There's
so
many,
but
really
when
I
compare
the
number
of
conformance
tests
to
the
number
of
tests
that
we
run
to
define
to
as
release
blocking.
There
are
many
more
tests
that
are
release
blocking.
A
We
run
hundreds,
if
not
thousands,
of
tests
to
define
what
a
kubernetes
releases
and
many
of
these
tests
do
include
features
which
are
optional
or
do
require
root
access.
It
seems,
though,
that
there
might
be
some
functionality
from
those
tests
that
we're
not
covering
with
conformance
tests,
and
we
have
room
to
improve.
A
A
The
things
we're
working
on
for
improving
the
requirements
for
conformance
involved
tagging
the
tests
that
do
things
that
are
wrong
so
that
we
can
talk
about
those
identifying
other
constraints.
We
might
want
to
add
to
those
requirements.
So
today,
I
believe
we
want
to
say
that
conformant
clusters
must
have
multiple
nodes
so
that
we
can
sufficiently
exercise
deployments
and
replica
sets
things
that
offer
zero
downtime
deployments
our
rollout
and
we
want
to
develop
profiles
which
have
optional
features.
So
we
may
see
a
profile
on
top
of
additional
conformance.
A
That
says
this
cluster
can
run
Windows
containers
or
we
may
say
that
this
cluster
has
cloud
provider
functionality,
I,
don't
care
which
cloud
you're
in
I
believe
your
cloud
has
a
load
balancer
as
a
service
that
you
can
use
so
I'm
about
to
show
you
a
bunch
of
data.
Just
take
my
word
that
if
you
two
can
do
this,
this
is
how
we
create
clusters
and
run
them
in
Google's
CI
or
in
the
kubernetes
CI.
A
This
is
a
pull
request
that
I
submitted
to
the
conformance
repository
to
make
sure
that
I
got
this
certified,
as
conformance
before
giving
this
data
to
you
all
of
the
results
that
we
run
as
these
conformance
tests
are
posted
on
a
website.
We
have
called
test
grid
where
you
can
see,
we
have
conformance
results
for
Google
cloud.
Openstack
VMware
is
submitting
results
up
there.
This
is
proof
that
the
conformance
tests
are
run
every
hour
of
every
day.
A
Here,
I
have
clicked
through
and
chosen
the
dashboard
for
the
conformance
tests
for
version
113
of
Google
Cloud
I'm,
clicking
on
a
specific
result
and
I'm,
seeing
the
results
from
this
test.
The
results
that
I
care
about
our
j-unit
XML
file
and
some
log
files-
we
don't
log
files
from
all
of
the
components
as
well
as
all
of
the
nodes.
A
Using
these
audit
log
file
I,
can
use
API
snoop
a
program
developed
by
the
CNC
F
to
compute
API
coverage.
Api
snoop
can
parse
kubernetes
open
to
API
spec
to
find
all
of
the
possible
end
points.
An
end
point
here
is
an
HTTP
verb
and
a
path
so,
for
example,
posting
to
a
pods
endpoint
in
namespaces
to
create
a
pod.
That
is
a
single
endpoint.
We
can
then
parse
the
kubernetes
audit
log
to
find
all
of
the
endpoints
that
have
been
hit
as
a
result
of
the
conformance
tests.
A
Using
this
we
can
answer
questions
such
as
what
are
the
total
endpoints
that
we
can
cover
and
what
are
the
actual
endpoints
that
we
hit?
So
here
is
another
graph
with
shiny
colors.
The
red
is
the
latest
release
and
you
can
see
each
release
has
had
more
API
coverage
by
percentage,
so
we're
making
great
progress.
You
can
also
see
that
there's
been
a
lot
of
increase
in
coverage
for
version
112
and
beyond.
Behind
this
white
box
is
the
API
coverage
offered
by
those
release
tests.
A
Those
big
blue
bars
any
idea
how
much
additional
coverage
those
provide
it
turns
out.
It's
only
about
10%
more,
it's
not
a
whole
lot.
I
also
just
quickly
want
to
explain
the
reason
those
bars
jump
up.
So
is
we
discovered
that
there
are
deprecated
end
points
for
watching
that
you
don't
need
to
cover,
because
conformance
tests
shouldn't
be
exercising
deprecated
features.
A
A
So
this
is
a
graph
produced
by
API,
snoop
I
call
it
the
Sunburst
graph.
The
inner
ring
denotes.
The
different
API
levels.
Green
is
stable,
blue
is
beta,
red
is
alpha
version.
1/9
of
kubernetes
was
the
first
version
that
had
audit
logging
turned
on
by
default.
So
this
was
the
first
release
of
kubernetes
that
we
could
use
API
snoop
to
measure,
and
you
can
see
we
had
about
24
percent
total
API
coverage.
A
By
version
112,
you
can
see
the
impact
we
had
by
removing
those
endpoints.
We
also
moved
a
number
of
end-to-end
tests
that
were
run
just
for
the
kulit
and
turned
them
into
conformance
tests
for
the
whole
cluster.
This
was
also
the
first
release
where
we
could
parse
the
user
agent.
From
the
end,
the
user
agent
was
provided
by
end-to-end
tests,
so
we
could
see
which
API
coverage
was
coming
from
end
to
end
directly.
Those
are
the
darker
colors
and
which
API
coverage
was
coming
from
the
kubernetes
cluster
directly.
A
Thus
far,
we
have
promoted
even
more
tests
and
that's
great,
but
if
we
look
just
at
the
API
coverage
from
the
end-to-end
test,
it
seems
like
we're
not
making
a
lot
of
progress.
This
is
why
we
think
that
it's
important
to
not
just
measure
the
API
coverage
coming
from
running
the
tests,
but
also
how
kubernetes
as
a
distributed
system,
behaves
while
running
tests.
If
we
look
at
the
API
coverage
that
could
be
available
to
us
by
running
the
release
tests,
we
could
have
a
fair
amount
more.
A
By
setting
the
user-agent
to
not
just
the
name
of
the
program
hitting
kubernetes
but
also
the
test
names,
we
can
answer
the
question
which
endpoints
are
hit
by
which
tests
which
endpoints
are
hit
by
the
most
tests
here.
I
can
see
that
many
endpoints
are
hit
by
all
of
the
tests
all
at
once.
These
endpoints
probably
don't
mean
anything
to
me.
So
let
me
instead
look
at
the
end
points
that
aren't
hit
by
all
the
tests.
A
Take
my
word
for
it,
the
endpoints
that
are
hit
the
most
involved,
creating
and
scheduling
pods,
and
this
should
not
be
a
surprise
and
users
want
to
create
in
scheduled
pods.
So
that's
how
we
write
a
lot
of
our
tests.
I
wanted
to
look
at
a
specific
test
called
proxy
version.
One
should
proxy
through
a
service,
and
pod
I
won't
worry
with
the
details.
This
is
clearly
very
easy
to
understand,
so
instead,
I
will
walk
through
what
the
test
does
in
terms
of
API
access.
A
This
is
the
reason
that
all
of
the
endpoints
get
hit.
All
of
the
time
we
create
an
API
client
that
explores
every
possible
resource
that
the
API
offers
I.
Don't
care
about
this?
This
looks
more
useful.
I
creates
the
namespace
I,
create
my
kubernetes
resources
and
then
I
start
hitting
the
proxy
endpoint,
but
it
seems
that
I'm
only
using
the
get
verb
to
hit
the
proxy
endpoint.
A
So
the
lesson
learned
by
looking
at
the
way
a
single
test
interacts
with
the
API
is
that
there
are
six
different
HTTP
verbs
that
this
test
is
not
using.
This
test
is
also
only
hitting
slash
proxy
instead
of
slash
proxy
slash,
myth,
slash,
sub
path
right
away.
I
found
13
additional
API
10
points.
I
could
cover
by
writing
new
tests.
A
Some
of
the
next
things
we're
going
to
do
with
API
coverage
include
looking
at
the
open,
API
spec
from
a
live
kubernetes
cluster
to
get
API
extensions,
such
as
scaling
policy
and
metrics
dynamically
configuring.
The
audit
log
policy,
so
that
we
don't
have
to
ssh
to
the
cluster
to
get
the
audit
log
and
the
thing
I'm
most
interested
in
is
covering
more
than
just
verbs
and
paths,
but
also
parameters
and
the
fields
of
all
resources
on
the
bodies.
A
So
looking
at
line
coverage,
many
of
you
probably
know
that
golang
supports
instrumentation
for
unit
test
line
coverage
or
code
coverage,
but
the
problem
is
this:
only
gathers
code
coverage
for
unit
tests
and
it
only
outputs
data
of
when
the
unit
tests
finish
what
if
we
made
kubernetes
a
distributed
system
of
unit
tests,
what
if
each
component
of
kubernetes
was
just
a
unit
test
and
then
what?
If,
instead
of
writing
the
code
coverage
data
commedia,
once
kubernetes
shut
down?
What
if
we
wrote
it
to
disk
every
five
seconds?
A
In
this
way,
we
could
deploy
kubernetes
gather
the
coverage
data.
Before
we
run
our
end-to-end
tests
run
our
end-to-end
tests
gather
the
coverage
data
after
and
then
do
some
mouths.
We
wrote
a
tool
called
go
fridge
to
allow
us
to
compare
to
merge
all
of
the
code
coverage
from
the
various
components.
Take
code
coverage
from
three
different
couplets,
a
scheduler
or
a
controller
manager
and
API
manager
merge
it
up
into
two
four
and
after
and
then
dip
the
two
finally
exclude
a
couple
packages.
A
We
know
we
don't
care
about,
such
as
a
cloud
provider,
since
this
is
conformance,
and
now
we
have
a
coverage
file
that
any
tool
that
knows
how
to
use
coverage
files
can
use.
So
some
of
you
may
know
that
go
generates
an
HTML
report
that
shows
which
lines
of
code
are
covered
on.
The
right
is
code
that
is
covered
by
the
conformance
test.
There's
a
lot
of
red
there.
It's
not
covering
very
much
on
the
left
is
that
same
code
but
being
covered
by
the
release
tests.
A
Maybe
this
is
a
file
we
should
look
at
more,
but
looking
at
files
seems
very
boring
and
difficult
to
traverse
and
search.
So
what
if
we
instead
wrote
our
own
report
that
allowed
us
to
traverse
a
hierarchy,
dough
fridge.
Does
that
go
fridge
allows
us
to
see
that
looking
at
the
lines
of
possible
executable
code
today
we
cover
28.5%
kubernetes,
that's
a
lot
of
red.
A
A
Further
I
should
point
out
that
the
release
tests
only
cover
33.5%
of
kubernetes.
So
again,
there's
a
lot
more
tests.
We
could
be
writing,
but
it's
not
all
red.
If,
oh,
if
we
look
at,
we
can
start
to
compare
code
coverage
over
time
and
you
can
see
this
is
difficult
to
read
right
now.
We
are
working
on
ways
to
reduce
the
noise.
A
It's
not
all
bad,
though
you
can
see
the
code
for
the
couplets
to
interact
with
docker
is
mostly
green.
Conformance
has
a
lot
of
interaction
with
it,
as
does
the
end-to-end
tests,
also,
here's
an
area
or
an
example
of
gap
in
testing
that
we
identified
that
we
could
not
find
with
API
coverage
on
the
left
is
the
code
that
executes
liveness
and
readiness
probes
against
containers
on
the
couplet.
It's
green.
A
A
Finally,
just
for
grins
I
tried
taking
the
code
coverage
file
and
upload
it
to
a
service
called
code
kaavo
to
see
if
I
could
generate
any
pretty
graphs.
They
made
this
tree
map
here,
it's
all
red,
so
it
doesn't
do
any
of
the
filtering
that
our
other
report
did.
It's
making
me
feel
like
we
could
write
our
own
version
of
this
that
maybe
even
linked
directly
to
the
files,
so
we
could
go
from
the
Big
Red
Square
to
the
directory
to
the
individual
files.
A
So
I
also
feel
like
that
test
grid.
All
the
green
squares
that
I
showed
you
earlier.
We
could
put
our
code
coverage
data
there
as
well
to
see
how
it
changes
over
time,
so
how
you
can
help
I
really
really
think
we
as
a
community
need
to
be
writing
more
tests
that
exercise
pods
and
look
at
their
behaviors.
You
can
help
us
categorize
the
existing
tests.
There
are
many
tests
that
seem
to
be
slow
or
flaky
that
aren't
categorized.
As
such,
you
can
run
conformance
tests
against
your
own
kubernetes
cluster.
A
Many
of
you
probably
use
kubernetes
today
and
who
knows
if
your
Callister
is
actually
conformant.
If
you
are
doing
useful
stuff
and
your
cluster
fails
those
tests,
we
should
understand
what
we're
not
testing
that
works
for
you.
Finally,
you
can
participate
in
discussion
about
this
at
the
CN
CF
conformance
working
group
or
at
the
architecture
sig
details
of
which
I
have
linked
in
these
slides,
which
you
can
find
online.
A
So
I
have
a
bunch
of
links
if
you
download
these
slides.
If
you
want
to
contact
me,
I
am
at
spiff
XP
on
all
the
things:
twitter,
github,
slack,
SoundCloud,
tumblr
whatever
and
there's
my
email
address.
I
also
wanted
to
congratulate
than
the
elder
for
his
work
and
helping
me
put
this
presentation
together.
Kathryn
did
the
code
coverage
tools,
hippie
hacker
and
Zack
Mandeville
made
API
snoop
they're,
giving
a
presentation
as
hippies,
giving
a
presentation
about
that
next,
which
I
highly
encourage
and
that's
it
any
questions.