►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: CNCF K8s-Conformance WG - Zefeng Wang, Huawei & Srinivas Brahmaroutu, IBM
For end-users, kubernetes conformance tests are recommended "first to run" against a new deployed kubernetes cluster, to check everything is fine. However, it's still not convenient for especially newbies to run in some closed environment or debug when some tests fail. This presentation will share experiences on running and debugging conformance tests, as well as ongoing sub-projects relevant to k8s conformance WG, etc.
To learn more click here: https://sched.co/FuLS
B
A
And
today
we
will
talk
about
various
pin
points
around
submitting
your
certification
and
running
these,
the
conformance
test.
Along
with
that,
we
will
also
talk
about
what
conformance
means
and
how
you
promote
a
test
into
a
conformance
test
suite
so
that
you
know
yeah.
At
the
end
of
the
day,
we
maintain
a
set
of
conformance
tests
that
matters
to
to
have
conformance
among
various
providers
of
kubernetes,
so
yeah.
B
B
One
issue
is
that
it
takes
a
long
time
to
to
run
performance,
and
we
know
that
from
from
the
releases
and
updates,
there
are
more
and
more
tests
right
now,
so
it
typically
takes
can
take
up
to
three
hours
to
pass
the
whole
conformance
test.
Even
if
everything
goes
fine
and
if
there's
any
error
occurred
during
the
testing,
it
may
takes
more
time
because
some
of
the
in
a
test
implementation
we
there
will
be
logic
to
retry
the
test
to
to
to
avoid
some
accidentally
fail.
So.
B
So
if
it
takes
more
than
one
hour
to
to
finish
the
test,
they
it
will
be
killed.
Currently
it's
already
extended
to
three
hours,
but
if,
in
your
environment,
it
still
takes
a
very
long
time,
you
can
try
to
extend
that
time
too.
So
you
can
wait
until
all
these
tests
finished
and
get
a
full
report
of
the
furthering
test
then
start
to
work
on
fixing
the
issues
and
also
during
during
during
the
debugging
and
the
testing
you
can
use
the
argument
focused
to
to
run
specific
tests.
B
Another
one
is
about
the
network
issue.
This
is
the
issue
that
everyone
would
meet
in
China
because
we
GTR
is
is
plugged
by
the
Great
Firewall
or
in
some
case
there.
The
cluster
may
be
offline,
not
connected
the
Internet.
So
so
you
may,
you
may
not
able
to
to
run
a
conformance
test
because
many
of
the
test
that
depends
on
the
image
that
that
is
on
GCR
and
that's
not
possible
to
walk
around
with
a
land
image
Mira,
because
the
yama
was
generated
by
the
test,
so
so
how
to
deal
with
this
issue.
B
B
All
these
images
required
to
the
tool
to
the
host
the
darker,
the
demon.
Then
you
can
start
round
or
your
image,
because
it's
already
loaded
with
our
original
image
repository
and
the
tag.
So
that's
what
about
the
most
the
issue
we
face
in
China
and
if
you
have
any
other
questions,
you
can
just
feel
free
to
reach
and
reach
me
after
this
introduction.
A
All
right,
Thanksgiving,
so
I'll
talk
briefly
about
what
conformance
tests
are.
Essentially
conformance
tests
are
nothing
but
ete
tests.
Eta
test
framework
is
written
in
ginkgo,
so
you
have
the
describe
block
context,
block
and
set
of
tests
which
are
under
8
blocks
right.
So
we
we
try
to
cover
very
serious
of
kubernetes
through
ete
tests.
Some
of
those
the
subset
of
the
e2
is
tests
or
conformance
tests,
confirm
its
tests
are
supposed
to
cover
the
functionality
aspects
like,
for
example,
parts
back.
A
They
can't
be
existing
parts
back
tests
already
or
there
may
be
some
that
are
missing
the
gaps.
That
means
you
need
to
write
new
tests,
ET
tests
and
promote
them
into
conformance
tests.
We
should
also
cover
API
coverage.
Like
Erin
said
yesterday,
we
have
audit
logging
to
to
trap
through
the
API
snoop,
what
not
the
tools
to
trap
all
the
api's
that
are
called
when
a
test
is
executed,
so
we
do
know
what
all
the
api's
that
are
covered
and
we
can
identify
the
gaps
and
add
new
tests
to
to
fix
the
deficit.
A
A
Given
this
situation,
we
do
not
have
conferment
stress
testing
all
the
functionality
all
the
code
is
covered,
so
the
conformance
tests
keep
growing.
For
that
reason.
From
for
every
minor
release,
as
well
as
new
functionality
that
gets
added,
should
we
should
make
sure
that
new
conformance
tests
are
added
to
that
new
functionality?
That's
added
for
each
of
the
releases,
so
the
sum
of
the
coverages
of
the
core
features
should
include
things
like
note:
pod
volumes,
networking
queue,
control,
security,
etc.
A
There
are
strict
criteria
for
us
to
measure
what
should
be
part
of
the
core
and
conformance
test
should
be
written
to
to
make
sure
that
all
the
tests
that
are
written
should
work
on
practically
all
the
providers,
all
platforms,
so
a
conformance
test
is
an
existing
ETA
test
that
will
be
changed
to
a
conformance
test
or
we
write
a
new
test
right.
So
there
are
certain
criterias
that
we
have
to
look
for.
The
test
has
to
be
testing
a
ga
feature.
A
A
It
should
not
rely
on
internet.
That
means,
if
there
is
any
need
for
you
to
download
any
images
that
has
to
be
pre
done,
and
it
should
not
have
any
special
binaries
that
your
kernel
requires
or
your
cubelet
requires.
It
should
be
also
stable
and
consistent.
There
are
tests
that
are
marked
flaky
today
and
there
should
not
be
part
of
the
conformance
test
really
just
because
there
is
no
guarantee
that
it
would
run
consistently
on
all
the
platforms
on
all
the
cloud
providers.
A
A
I'll
show
you
in
a
bit,
and
we
also
have
a
conformance
test
to
document
that
we
generate
for
every
minor
release,
so
new
tests
are
added.
So
the
document
is
changing
for
every
minor
release,
so
we
generate
one
document
for
every
minor
release.
For
example,
1.10
has
a
document.
1.11
has
a
document,
so
on
so
forth.
A
That
document
will
list
out
all
the
tests
that
are
part
of
that
release,
as
well
as
each
of
those
tests.
You
have
a
general
description
of
what
the
test
does,
so
you
don't
have
to
go
to
the
code
and
figure
out
what
the
test
does
and
also
a
link
to
the
source
code
so
that,
if
at
all,
you
need
to
know
more
details
of
the
test,
you
can
go
into
the
source
code
and
check
what
the
test
is
doing.
A
We
maintain
washing
involved
information
along
with
the
test,
so
that,
if
you
are
running
the
test,
you
know
when
the
test
was
introduced
into
the
conformance
test
suite
and
also
if
a
test
is
modified,
that
will
be
part
of
their
versioning
information
along
with
other
documentation.
We
add
to
the
test
so
that,
if
something
fails
you
know
is
this
test
was
the
same
during
the
entire
release
from
110
111
to
112.
A
There
are
this:
we
use
this
document
to
measure
coverage.
I
can
I'll,
show
you
how,
because
now,
if
the
metadata
we
have
about
the
test,
name
tells
you
that
what
are
all
the
parts
back
test?
What
are
all
the
part?
What
are
all
different
types
of
tests
like
volume
tests?
So
you
know
the
functionality
that
you're
testing
and
you
know
the
gaps
by
reading
through
the
documentation
and
also
it
helps
you
in
debugging.
A
A
Now,
there's
no
automation
to
that
process.
We
are
working
on
that.
I
have
some
ideas
on
that.
So
for
now
it
is
manually
generated
and
it's
pushed
as
a
PR
to
CN
CF
website,
but
we
talked
about
having
it
as
part
of
the
released
or
ball
itself,
so
that
you
know
it's
easier
for
people
to
access
it
locally.
I
can
generate.
One
I
will
show
you
how
to
generate
one
right.
A
Right,
so
all
the
guidelines
for
conformance
is
written
in
a
nice
document
that
will
show
you
what
it
constitutes
for
writing.
A
conformance
test
right,
like
I
described.
What
are
the
criteria
for
for
a
ETA
test
to
become
a
conformance
test
I'm
quickly
going
through
this
document,
and
there
is
a
skew
policy
that
how
long
this
conformance
test
should
be
should
be
supported.
Basically
on
what
releases
it
should
be
run
like,
for
example,
you
are
writing
a
conformance
test
from
now
on,
one
1.13,
it
should
be
run
on
the
master
as
well
as
1.13.
A
A
All
the
conformance
related
work
is
is
reviewed
by
architecture
board,
so
there
is
a
backlog
here
that
will
show
you
with
which
face
various
items
related
to
conformance
or
and
if,
if
you
can
see,
if
there's
an
implemented
test
there,
you
can,
you
can
see
how
the
code
has
has
been
changed
to
make
it
a
conformance
test.
There
is
a
dashboard
kubernetes
where
you
can
see
what
are
all
the
tests
that
are
run
as
part
of
the
conformance.
A
A
The
process
is
described
out
here
under
ciencia
conformance
website.
There
is
a
process
for
you
to
to
to
get
certified,
how
you
submit
a
PR,
VT
or
conformance
test
results
along
with
that
the
documentation
for
each
of
these
releases
is
under
here.
That's
what
the
question
was.
There
is
a
document
for
one
levin.
There
is
a
document
for
one
to
all,
that's
missing.
We
will
add
that
soon.
A
A
Save-
and
it
will
generate
these
document
I'm
talking
about
for
for
that
release
for
that
source
code.
I
have
generated
the
document
already,
and
the
conformance
document
looks
like
this.
So
if
you
look
at
this
document
as
an
end
user,
you
have
all
the
information
about
the
conformance
test
and
based
on
that
key
words.
They're,
like
scheduler
tests,
you
will
be
identifying
what
all
the
tests
that
are
run
in
that
particular
area,
so
that
you
can
you
can
you
don't
have
to
go
into
the
source
code?
A
But
if
you
want
to
really
go
into
the
source
code,
you
can
click
on
the
test
itself
and
you
can
look
at
the
source
code
itself
of
the
test.
As
I
said
there,
that
documentation
is
part
of
the
test
itself
right
above.
There
is
a
release,
information
test,
name
and
a
brief
description
of
the
test
along
with
more
documentation.
If
you
want
to
add-
and
normal
ginko
test
is
an
it
block,
it
will
convert
it
into
a
conformance
it
block
so
that
we
can
make
it
a
conformance
test.
Okay,
question.
C
A
C
D
Don't
enforce
that
requirement
right
now.
I
would
like
to
close
that
gap
within
the
next
quarter
or
to
you,
but
right
now
it
was
done
like
as
a
fix-it,
a
bunch
of
Google
engineers
got
together
and
did
it
in
a
week,
but
it
was
only
sort
of
an
ad-hoc,
informal
thing,
so
I'd
love
to
see
more
of
that
finished.
E
A
A
F
B
Apparently,
some
of
the
tests,
you
know
that
Cuban,
a
nice
API
implement
as
I'll
say
a
synchronized.
Sometimes
you,
you
call
an
API
and
they
it
it's
soon
written
success.
But
if
you
want
to
check
the
results,
it's
the
application,
whether
it's
running
running
well
or
or
for
job
you
you,
you
need
to
wait
until
the
job
is
complete,
that
in
the
in
the
testers
there
are
some
code
that
will
repeat
to
all
the
status
of
the
application
to
check
if
it's
successful.
Currently.
B
B
B
F
D
Often
run
in
serial
so
one
after
another,
there
are
currently
about
two
hundred
and
something
different
conformance
test
cases,
which
means
we
have
to
wait
for
the
first
case
to
finish,
and
then
the
next
one
and
then
the
next
one
two
hundred
times
there
are
many
many
of
those
tests
that
could
be
run
in
parallel.
This
is
how
we
do
testing
for
most
of
kubernetes
when
we're
doing
pre
submit
tests
for
ball
requests
and
I.
Don't
have
the
numbers
in
front
of
me.
D
It
used
to
be
that
you
could
run
all
of
them
in
parallel
in
something
like
six
to
eight
minutes,
but
then
there
were
some
remaining
tests
that
you
still
had
to
run
in
serial
because
they
require
complete
access
to
the
entire
cluster.
For
example,
when
you
are
testing
some
things
like
scheduling,
algorithms,
you
need
to
know
the
state
of
the
entire
cluster
what
all
nodes
are
like
in
order
for
you
to
exercise
this,
there
are
also
some
tests
that
are
disruptive
that
we're
not
sure
we
want
to
remain
in
conformance.
F
D
You
would
need
to
fill
up
your
cluster
entirely
to
exercise
the
positive
and
negative
conditions
of
those
test
cases.
Another
reason
that
some
of
these
tests
take
a
long
time
is
that
there's
some
hard
coded
constants
for
things
like
back
offs
and
timeouts,
so
one
example,
I
can
think
of
is
verifying
that,
when
a
container
has
to
restart
that
it
not
restart
super
super
quickly
at
each
time,
it
backs
that
delay
off
exponentially
until
it
reaches
some
max
delay.
D
That
delay
is
five
minutes
and
the
amount
of
time
it
takes
to
restart
and
restart
and
restart
until
you
hit
that
maximum
delay
is
about
27
minutes.
I
did
not
include
that
test
in
conformance
because
I
don't
think
a
27
minute
test
is
valid,
but
I
do
think.
We
have
some
tests
that
do
similar
patterns
that
wait
for
a
while,
and
it
could
be
that
we
want
to
reprove
them
reproduce
the.
B
D
So
at
the
moment,
performance
echoes
are
not
considered
part
of
conformance.
We
do
have
the
there
is
discussion
ongoing
about
the
idea
of
optional
profiles,
because
conformance
is
supposed
to
work
for
any
kubernetes
cluster
pretty
much
everywhere.
So
it
should
work
from
like
a
bunch
of
raspberry
PI's,
all
the
way
up
to
a
massive
data
center
with
thousands
and
thousands
of
nodes.
D
It
could
be
we're
going
to
find
that
there
are
some
performance
guarantees
we
want
to
make
and
you'll
have
to
use
certain
qualified
Hardware
before
you
can
meet
those
s
clothes,
and
we
may
say
that
you
have
a
scalability
profile
that
she
could
stamp.
On
top
of
your
cut
your
cluster
being
conformant
that
it's
also
scalable
up
to
some
limit,
but
right
now
the
kubernetes
project
is
still
taking
its
time
in
more
granularly,
defining
what
those
SLO
s
are
supposed
to
be.
D
But
really
conformance
is
all
about
what
an
end
user
experiences,
and
so
we
feel
like
we
need
to
get
a
little
more
granular
with
things
like
deployments
and
services
and
endpoints
to
understand
all
of
the
various
use
cases,
because
we
are
seeing
a
variety
of
people
try
to
use
kubernetes
at
different
scale
than
just
a
bunch
of
pods
and
a
bunch
of
no
it's
the
most
common
one.
I
see
is
people
scheduling
a
lot
of
jobs,
and
by
did
you
have
something
you
wanted.
G
B
G
B
G
C
G
My
impression
is,
we
can't
do
it
on
with
DNS,
so
it'll
need
to
be
something
like
Kade,
SC,
n
dot.
It
shouldn't
be
g
CR,
it's
g
CR
it.
It
should
be
the
other
way
anyway.
It
rather
than
GC
r
KS
that
io
it
should
be
some
sort
of
C
n
dot
in
what
sig
do
you
has
responsibility
for
the
container
registry
today
and
would
want
to
manage
this
I
love,
picking
random
I.
D
Think
I
would
say
that
the
sake
release
is
probably
the
sake
best
in
charge
of
that.
However,
given
that
Google
donated
nine
million
dollars
to
the
CN
CF
to
help
Ronnie
infrastructure,
the
community
has
put
together
a
working
group
to
work
on
that
process
and
I
would
imagine
the
people
most
actively
involved
are
in
that
working
group.
It
meets
on
Wednesdays,
but
I
don't
have
the
time
off
the
top
of
my
head.
It
is
a
friendly
time,
though,
that
it
was
intended
to
work
Asian.