►
From YouTube: Intro + Deep Dive: Kubernetes Conformance WG - Dan Kohn, CNCF, William Denniss, Aaron Crickenberger
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Intro + Deep Dive: Kubernetes Conformance WG - Dan Kohn, CNCF, William Denniss, Aaron Crickenberger, Google & Hippie Hacker, ii.coop
We will start the session with a high level introduction of the Certified Kubernetes conformance program. Topics will include how to run the conformance test suite, how to certify your company's platform, distribution or installer, and how to get maximum benefit out of the program for your company. Following the the introduction, there will be a technical working session by members of the working group to discuss current topics, likely including the in-development Windows container conformance profile.
https://sched.co/MPl1
A
Everyone
thanks
for
joining
our
session
today,
a
cubist
confluence
where
we've
combined
the
intro
in
the
deep
dive
this
time
so
first
kind
of
15
to
20
minutes
will
be
sort
of
will
be
introductory
and
we're
gonna
gradually
add
deeper
and
deeper.
So
apologies!
If
you
here
for
the
deep
dive
section,
you
can
just
turn
out
for
a
little
bit
and
and
if
you're
more
interested
in
the
intro
topics,
no
one's
going
to
be
offended.
A
B
Hi
this
is
the
key.
The
money
slide,
as
we
said
so.
My
name
is
Dan
Khan
I'm,
executive
director
of
CMC
F,
and
we
launched
this
conformance
program
back
in
September
of
2017
and
I'd,
say
that
it
has
exceeded
our
wildest
expectations
in
terms
of
getting
the
entire
industry
to
buy
into
this.
So
it's
84,
certified
kubernetes
partners
today
and
there's
remains
huge
amounts
of
activity.
If
you
follow
the
KS
conformance
repo,
we
get
people
registering
new
ones
and
occasional
ones
falling
off,
and
so
it's
a
very
active
program.
B
I
think
that
conforms
working
group
deserves
a
lot
of
credit
for
coming
up
with
a
program
that
is
both
doesn't
depend
on
a
lab
that
each
vendors
is
able
to
do
the
conformance
test
themselves.
But
this
sort
of
subtle,
powerful
aspect
of
it
is
that
any
user
can
come
along
later
and
confirm
that
that
product
is
still
conformant
and
so
there's
a
distributed.
B
Crowdsource
aspect
to
the
enforcement
that
I
think
is
subtle,
but
but
really
quite
powerful
and
important,
and
you
know
the
program
continues
to
evolve
so
much
chocolate.
Just
last
week
figured
out
that
there
was
missing
step
than
the
checks
that
we
were
doing,
but
no
one
it
hadn't
come
up
yeah,
so
we
fixed
it
so.
B
When
we
created
the
program,
we
had
the
specific
idea
of
profiles
in
mind
and
we
haven't
done
a
profile.
Yet.
I'll
also
mention
that,
if
you
think
of
the
analogy
to
Android
today,
we're
only
certifying
the
phones,
we're
not
certifying
the
Android
applications
brought
this
up
again
with
our
governing
board
on
Monday
and
had
absolutely
no
interest
in
doing
the
other
side
of
certification.
So
remains
something
that
we
could
do
in
the
future.
But
there's
just
does
not
seem
to
be
a
business
need
for
it
or
have
it.
Our
membership
does
not
seem
interested
it.
B
So
with
that
I'm
gonna
hand
it
back
over,
but
really
excited
to
be
part
of
this
I
think
it's.
The
very
last
thing
I'll
say
is
of
all
the
CMC
have
programs
a
chip
on
obviously
being
a
huge
one
and
community
certified
service
providers
and
lots
of
other
things
that
we've
done.
Our
landscape
and
such
I
think
that
this
certified
kubernetes
program
has
probably
had
the
biggest
overall
impact
on
the
cloud
native
ecosystem
in
terms
of
preventing
fragmentation
and
ensuring
that
there's
one
platform
that
everyone
can
work
with,
Thanks.
A
A
Certified
kumys
it
is
a
software
performance
program
and
more
or
less
vendors
can
just
certify
with
the
CNC
F
that
their
community
products
conform
to
the
community
test.
Suite
kind
of
consists
of
two
parts,
a
technical
test
suite
that
people
run
in
their
cluster
and
then
a
legal
aspect
to
it
where
they
submit
the
test
results
and
they
give
us
a
shiny
badge
on
the
right.
A
So
why
did
we
actually
do
this?
In
the
first
place,
the
the
driving
reason
is
really
to
guarantee
Portability
and
interoperability
as
as
best
we
can.
So
this
is
more
or
less
the
promise
of
committees.
We
wanted
to
make
sure
that
this
was
actually
going
to
be
delivered.
We
want
to
make
sure
that
if
you
can,
you
know,
deploy
your
application
just
fine
on
GK,
that
you
can
take
it
and
run
it
on
pram
or
run
somewhere
else.
A
Another
one
was
predictability,
so
when
there
are
differences,
we
sort
of
recognize
that
there
will
be
differences.
We
want
users
to
have
an
easy
understanding
of
what
to
expect
when
they're
migrating.
So
this
is
Dan
mentioned
profiles
briefly.
This
is
where
the
profile
concept
may
come
in
in
the
future,
where
there
may
be
some
differences
between
clouds.
A
Like
some
of
that
sport,
it's
a
Windows
and
some
that
don't
every
one
of
these
things
to
kind
of
be
obvious
and
discoverable
and
the
last
one,
but
but
not
the
least
actually
is
that
we
wanted
to
make
sure
that
that
users
would
always
get
upgrade
and
take
advantage
of
all
the
latest
features
of
communities.
So
I
think
one
of
the
potential
failure
conditions
for
the
project
actually
as
a
whole
like
kubernetes,
is
everyone
just
locks
in
it?
A
Like
one
point
ten
or
something-
and
you
know
one
point-
ten
becomes
like
the
version
that
everyone
uses
and
then
you
know
all
the
great
stuff,
all
the
all
the
great
work
that
everyone's
doing
basically
goes
to
waste,
because
everyone's
just
using
one
ten.
So
one
of
the
things
that
we
tried
to
build
in
this
program
was
kind
of
incentivizing
people
to
stay
up
to
date
and
make
sure
that
that
users
can
ultimately
get
the
benefits
of
the
communities
community
that
everyone's
working
so
hard
on.
A
So
looking
at
it
by
different
classes
of
users,
if
you're
an
end-user
conformance
helps
you
avoid
the
vendor
lock-in,
and
so
you
can.
You
can
verify
that
the
environments
honoring
the
API
contract,
as
Dan
said
like
one
of
the
really
advantages
of
this
program,
is
that
you
can
run
this
test
suite
yourself
and
just
verify
that
your
own
vendor
is
actually
conforming
if
you're
a
vendor,
and
you
can
add
the
conformance
to
just
to
validate
that
you're,
actually
delivering
real
Kupa
daddies
to
your
users
and
if
you're,
a
nice
fee.
I
think
this
is.
A
This
is
a
very
useful
one
here,
if
you're
shipping
software
to
customers
that
they're
then
running
if
they
encounter
any
problems
which
they
will
I'm
sure
and
they'll
complain
to
you.
One
of
the
great
things
that
components
is.
You
can
actually
ask
your
customers
to
first
verify
that
their
clusters
are
conformant
before
you
spend
like
recent
weeks,
like
debugging
their
environment.
So
it's
kind
of
a
sanity
check,
like
you
know,
easiest
environment
actually
Humanity's
before
really
get
the
details.
A
Why
bother
certifying
see
anyone
can
just
run
the
test?
Suite
certification
is
an
extra
step
after
that
point,
I've
also
an
agreement
with
the
CNC
F.
So
why
would
you
do
that
extra
step?
Well,
one
is
you
know
you
get
to
provide
that
assurance
to
your
customers
that
it's
real
good
notice
and
you
get
that
cool
badge
and
last
but
not
least,
this
is
extremely
powerful.
Actually,
you
get
to
add
the
word
kubernetes
to
your
product,
so
Cuba
nice
is
a
trademark
owned
by
the
CNC
F.
So
normally
you
wouldn't
be
able
to
have.
A
B
A
A
What
is
the
canaries
API
that
we're
testing
that
is
governed
by
Sagarika
texture,
one
of
the
special
interest
groups,
and
then
we
have
the
components
workgroup,
which
is
the
people
here,
all
those,
although
there
is
a
lot
of
overlap,
we
actually
kind
of
run
the
process
itself
and
train
our
policy
and
things
like
that
all
right.
So
let
me
do
a
quick
demo
just
to
show
you
how
easy
this
is
and
I
think
like,
even
if
your
user.
A
This
is
it's
kind
of
valuable,
because
you
can
just
you
can
just
test
this
on
your
own
cluster,
so
to
get
started.
You
need
to
download
sooner
boy
and
rather
than
unless
you
like,
a
glutton
for
punishment,
love
building
go
code.
You
could
just
grab
one
of
these
pre-built
binaries,
which
I
did
earlier
and.
A
B
C
A
A
So
yeah
you
run
sooner
boy
status
and
just
proving
the
cross-platform.
You
know
the
the
great
thing
about
docker
is
as
a
developer
like
I
can
build.
Linux
containers
and
I.
Really
really
operating
system
doesn't
matter
anymore,
which
I
think
is
one
so
yeah
more
or
less.
That's
it
I
know.
That's
all
I
need
to
do,
and
this
will
actually
take
like
three
hours.
So
I'm,
not
gonna
I'm,
not
gonna,
I'm,
not
gonna,
boil
with
that.
But
what
uh-huh,
but.
A
Sixty
minutes,
I
think,
is
a
bit
optimistic.
It'll
eat
at
let's
say
order
of
magnitude,
yes
Saburo.
So
what
you'll
be
looking
for
is
a
user?
Is
you
can
just
run
the
center
boy
status
and
once
that
running
command
there
says
complete,
then
then
you're
good
to
go
at
any
time.
You
can
also
see
the
locks,
so
I
can
check
those
out.
A
A
Results
alright,
and
you
see
there
are
two
files
there.
Let's
inspect
the
you
doing
long,
so
I
just
ran
this
this
morning,
a
basically
that
exact
command
I
showed
you
earlier
around
this
morning.
This
is
what
happens
once
you
wait:
the
necessary
60
plus
minutes
so
lotta
logs.
The
important
thing
here
at
the
bottom
200
past
no
failed,
so
that
basically
means
that
this
cluster
is
conforming
to
the
best
of
our
knowledge.
A
Of
course,
as
you'll
hear
later
in
the
deep
dive,
there
are
certainly
some
gaps
in
this
that
we're
trying
to
fill,
but
for
now
that's
what
we
have
alright.
So
that's
the
technical
side
of
it,
then,
when,
if
you're,
if
you're
an
actual
vendor-
and
you
have
a
distribution
or
a
a
platform
or
an
installer
of
kubernetes,
you
can
basically
just
take
these
those
two
files
on
the
disk.
Here,
the
you
do
a
vlog
and
the
XML
package
that,
up
with
a
little
bit
of
metadata
about
your
service,
submit
that
to
our
repo.
A
You
do
need
to
be
a
CNC
at
remember
and
that's
it.
You
can
be
certified,
so
I
hope,
I've
demonstrated
just
how
easy
it
is
to
actually
run
these
tests
and
your
whether
or
not
your
user
just
going
to
validate
your
vendor
or
whether
you're
a
vendor,
and
you
actually
want
to
certify.
And
then
you
can
respect
the
results
here
so
say
for
113.
A
E
Everybody
I
am
Aaron
of
Sig
beard,
also
known
as
Aaron
Frankenberger
I'm,
the
co-founder
of
cig
testing
I'm,
also
kubernetes
steering
committee
member
I'm
actively
interested
in
seeing
what
we
can
do
to
improve
the
state
of
conformance
testing.
So,
as
william
alluded
to
earlier,
there
are
potentially
gaps
in
conformance
testing
we'd
like
to
make
sure
that
when
we
say
something
is
certified
kubernetes
that
that
is
meaningful,
that
we've
adequately
exercised
enough
of
kubernetes
functionality
to
give
users
the
guarantee
that
all
of
their
workloads
are
portable.
E
So,
even
though
this
is
the
intro
part,
I'm
gonna
just
dive
deep
right,
okay,
because
I
have
to
bounce
out
of
here
soon
for
another
panel.
So
here
this
is
sort
of
an
update
of
a
talk.
I
gave
in
Keep
Calm
Shanghai
called
Adventures
and
conformance
where
I
sort
of
laid
out
a
bunch
of
charts
and
data.
I've
reduced
it
down
a
bit
to
talk
about
the
number
of
tests
we've
added
on
a
quarterly
basis
and
to
show
that
hooray,
it's
always
been
going
up
into
the
right.
E
E
So
what
do
pots
do
on
a
capelet?
Let's
look
at
how
we
run
the
things.
Let's
see
how
we
make
sure
that
things
are
active
and
ready
things
of
that,
and
so
we
spent
most
of
our
time
picking
up
tests
that
were
already
in
our
end-to-end
tests,
corpus
and
turning
them
into
conformance
tests.
This
largely
involved
making
sure
that
the
tests
exercise
behavior.
That
was
conformant,
so
was
it
the
sort
of
behavior
we
expected
to
work
across
all
clusters,
or
were
the
tests
being
a
little
too
white
box
about
it?
E
We
also
had
to
make
sure
that
the
tests
were
written,
the
correct
way
to
make
sure
that
they
didn't
exercise.
They
didn't
look
for
the
functionality
in
API
fields
that
weren't
guaranteed
to
change
between
versions
so,
for
example,
anything
that
looked
for
the
way
that
pods
were
scheduled
or
the
way
that
pods
started
up
by
looking
for
events
that
were
generated
by
that
is
not
really
a
conformance
test,
because
events
are
not
actually
a
versioned
API
objects,
they
can
change
at
any
time
across
different
kubernetes
versions,
so
any
tests
written
that
way.
E
A
lot
of
great
test
113
was
awesome.
I
was
in
full
swing
here.
Also
I
had
to
talk
about
performance
at
Shanghai,
so
I
was
really
incentivized
to
make
sure
we
pushed
extra
extra
hard,
but
really
the
bulk
of
the
tests
here
are
Cube
CTL,
cube
control,
QB,
cuddle
or
cubicle,
depending
on
your
pronunciation
of
choice,
tests
and
the
reason
all
of
these
tests
got
promoted
in
113
is
because
I
went
poking
around
Y
sauna
boy.
E
The
the
tool
that
William
introduced
was
using
end-to-end
test
image
that
was
built
by
hefty
o,
rather
than
an
end-to-end
test
image
that
was
built
by
the
kubernetes
project
and
it
turns
out
the
kubernetes
project
didn't
have
one
of
those
in
test
images,
so
we
started
working
on
that
and
113
and
why
did
hefty
as
image
exclude
queue,
control
tests
from
conformance?
Well,
it
turns
out
there
was
something
about
the
way
that
those
tests
worked,
that
they
didn't
really
at
kubernetes,
1/8
and
1/9.
E
They
didn't
function
well
inside
of
the
cluster
right.
The
way
sauna
boy
works
is
it
takes
a
container
that
has
all
the
end-to-end
tests
and
then
it
schedules
it
inside
of
the
tester
sorry
inside
of
the
cluster
and
then
so.
It's
sort
of
like
the
cluster
tests
itself.
You
know
the
call
is
coming
from
inside
the
house.
I
sometimes
feel
like
a
more
authentic
and
user.
E
Experience
is
one
where
the
tests
are
run
from
outside
the
cluster,
because
I
generally
interact
from
my
clusters
in
the
real
world,
or
usually
from
my
laptop
or
something
and
the
cube
control
tests
worked
just
fine
there.
But
since
we
now
live
in
a
world
where
scheduling
the
tests
inside
of
your
cluster
is
far
simpler
and
sonna
boy
is
the
document
of
way
of
doing
conformance
tests.
We
were
sort
of
beholden
to
figuring
out
what
was
the
problem.
Why
couldn't
these
tests
work
inside
of
the
cluster
and
it
turns
out?
E
We
figured
that
out
back
in
like
110
111,
something
like
that,
but
I
found
this
regular
expression
that
was
still
excluding
the
cube
control
tests.
So
we
now
live
in
a
world
where
the
regular
expression
for
what
tests
should
run
is
owned
by
upstream.
The
image
that
runs
the
tests
is
owned
and
built
by
upstream.
So,
as
a
result,
we
got
way
more
then
114
happened
and
I
was
the
release
lead
for
114
and
somehow
I.
E
Have
these
this
hope
or
dream
that
like
because
I
was
the
release,
lead
I
would
have
the
power
to
push
through
as
much
stuff
as
possible,
but
that
also
apparently
meant
I
didn't
really
have
a
whole
lot
of
time
to
focus
on
conformance
and
also
some
of
you
may
have
heard
that
114
was
the
release
of
kubernetes.
That
said,
that
Windows
Server
notes
are
now
GA,
and
so
it
turns
out.
E
We
spent
most
of
our
time
in
1:14
talking
about
what
G
really
meant,
especially
in
the
context
of
conformance
and
you'll,
see
that
one
of
those
tests
up
there
has
a
little
tag
called
Linux
only
and
that's
because
we
decided
to
handle
the
difference
in
behaviors
or
functionality
or
what-have-you
between
Linux
nodes
and
Windows
Server
nodes
by
sort
of
triaging
off
functionality
that
can
only
be
exercised
on
linux
nodes
as
linux.
Only
and
this
is
still
like
a
thing
we're
going
to
have
to
work
through
right
windows.
E
Server
nodes
are
not
conformant,
but
they
do
pass
a
bunch
of
end-to-end
tests
and
they
are
supported
enough
to
be
called
GA,
but
we
still
need
to
go
through
the
exercise
of
figuring
out.
You
know
what
are
the
behaviors
that
we
would
expect
to
work
on
nodes,
even
if
they
may
be
implemented
in
slightly
different
ways.
So
the
canonical
example,
I
tried
out
here
is
file
system.
Permissions
yeah,
but
both
operating
systems
have
file
for
filesystem
traditions
to
say
what
you
can
read
and
write,
but
they're
specified
in
different
formats.
E
There's
no
way
I
can
take
a
Linux
file
system
permissions
and
apply
them
to
a
Windows
server
node.
There
also
a
number
of
things
around
a
host
networking
and
the
way
that
DNS
works
that
require
that
you
know
we're
gonna,
have
to
break
that
functionality
up
and
start
to
chunk
it
up
into
two
lists
about:
what's
the
functionality
that
will
just
never
ever
ever
ever,
work
on
Windows
server
nodes
and
what's
the
functionality
that
will,
but
just
not
yet
it
could
be
because
of
how
the
tests
are
written.
E
E
Rather
than
talking
about,
and
also
all
of
the
other
architectural
concerns
of
Cades.
We
also
kind
of
finally
created
a
backlog
and
a
project
board
with
issues
for
all
of
the
things
that
we
are
currently
working
on.
Everything
is
kind
of
consistently
labeled
these
days.
If
you
want
to
work
on
anything
related
to
conformance,
you
look
in
github
and
you
look
for
anything
that
has
the
area
conformance
label,
and
so
you
can
see
we
actually
have
conformance
tests
that
are
coming
from
a
variety
of
areas:
apps
and
network
and
storage
and
node
and
stuff.
E
E
E
This
graph
is
a
little
dense,
but
again
what
I'm,
trying
to
show
you
is
everything
is
going
up
and
to
the
right,
which
is
always
great,
and
the
yellow
line
is
the
one
we
care
about
the
most,
because
this
is
coverage
of
api's,
which
are
considered
stable
or
GA.
It
won't
change
and
they're
also
part
of
the
core
namespace,
so
they're
all
the
resources
of
kubernetes
that
we
expect
to
exist
absolutely
everywhere.
There
are
not
extensions,
they're,
not
optional,
it
will
always
be
present,
and
so,
at
least
from
an
API
perspective.
E
If
I
look
at
all
of
the
endpoints
available
on
a
cube,
radedi
server
and
all
the
verbs
I
can
throw
at
those
endpoints
we're
heading
over
50%
of
them
with
the
beta
of
115
that
was
recently
cut,
so
good
job,
everybody,
yes,
so
this
is
basically
API
snoop.
That
said,
the
reason
I
didn't
show
you,
the
pretty
shiny,
API
snoop
graphs
is
just
for
consistency,
I'm
using
the
same
numbers
from
Shanghai
from
an
older
version
of
API
state.
E
But
it's
the
same
thing,
so
this
last
graph
I
just
kind
of
wanted
to
talk
about
like
this
is
sort
of
the
story
of
cig
architecture.
Getting
involved
in
the
story
of
us
actually
ramping
up
our
efforts
to
improve
performance,
so
the
yellow
bar
is
any
issue
or
pull
request
in
github.
That
has
the
label
area
conformance
applied.
That
has
been
updated
within
that
time
period,
and
so
you
can
see
that
we
are
starting
to
create
and
update
and
talk
about
and
merge
and
discuss
exponentially
more
issues.
E
Every
quarter,
which
is
great
and
encouraging
that
you
know
we
are
actually
starting
to
build
momentum
here.
A
much
lower
bar
in
red
are
all
the
pull
requests
that
are
that
have
the
label
area
conformance
that
are
merged,
pull
requests
related
to
conformance,
can
be
merged
in
multiple
repos
across
the
project.
You
know
we
could
be
updating
Doc's
that
describe
what
conformance
is
and
why
it's
meaningful.
We
could
be
updating
some
of
the
test,
jobs
that
automatically
run
conformance
tests
and
post
them
to
test
grid.
E
We
could
be
actually
merging
new
conformance
tests
or
we
could
be
promoting
existing
tests
to
conformance.
Finally,
the
lower
blue
line
of
the
blue
line.
There
is
all
of
the
pull
requests
that
actually
touch
the
conformance
directory
inside
of
kubernetes
kubernetes.
These
are
merged
PRS,
not
long-standing,
open
PRS,
but
these
are
PRS
that
either
are
promoting
a
conformance
test
or
promoting
in
a
dentist
into
conformance,
or
they
may
be
PRS
that
are
touching
the
tooling
related
to
conformance
tests.
E
So
we
have
tons
some
tooling
that
it
walks
and
Lintz
the
source
code
in
kubernetes
of
all
of
its
end-to-end
tests
to
make
sure
that
they
are
or
are
not
applicable.
In
conformance
and
the
thing
that
kind
of
bugs
me
is
we,
you
don't
really
have
a
lot
of
the
blue
ones.
The
blue
ones
are
kind
of
the
goal
here
and
so
I'd
like
to
kind
of
figure
out
how
we
can
convert
all
of
our
you
know,
enthusiastic
inspired
discussion
and
conversation
about
conformance
if
you
actually
merging
things
that
improve
the
state
of
conformance.
E
The
fact
that
we
are
building
up
the
momentum
seems
great
and
makes
me
think
that
perhaps,
if
I
like
wait,
another
quarter
or
two
we're
gonna,
see
like
that:
yellow
bar
convert
into
a
much
larger
blue
bar
down
the
line,
but
I
think
I,
look
forward
to
strany
and
John
and
some
other
folks
talking
about
you
know.
How
can
we
do
this
in
a
scalable
way,
because
I
think
ultimately,
there's
still
some
review
bandwidth
bottlenecks
that
we
have
and
I
think
we
could
benefit
from
documenting?
E
You
know
how
better
to
onboard
folks
who
wants
to
help
out,
with
conformance
now
that
we
have
this
reoccurring,
conformance
definition,
some
project
within
save
architecture,
for
people
who
show
up
what
do
they
do?
How
can
they
help
out?
How
can
we
push
stuff
forward?
The
other
thing
I'll
say
about
the
way
that
group
is
working
right
now
is
I
feel
like
there's.
An
awful
lot
of
time
spent
taking
a
look
at
existing
pull
requests
and
just
opportunistically
saying
why?
Don't
we
try
promoting
this
to
conformance
or
why?
E
Don't
we
try
promoting
that,
to
conformance
just
picking
up
a
lot
of
chaff
to
show
like
look
we're,
trying
we're
really
trying,
but
if
we
don't
actually
grow
the
pool
of
reviewers?
Who
can
say
yes,
that's
ready
or
no
that
isn't
that
just
clogs
our
queue
and
actually
wildly
increases
the
latency
of
getting
anything
actionable
done.
So
I'd
really
like
to
figure
out.
You
know
how
can
we
make
sure
that
when
we
pick
something
up,
but
it's
actually
going
to
be
meaningful
as
opposed
to
getting
lost
in
going
back
and
forth?
E
I
didn't
have
time
to
collect
the
stats
on
this,
but
I
I
suspect
that
a
lot
of
that
yellow
bar
that
you
know
comments
and
discussion
on
any
open
issue
or
PR
is
actually
a
lot
of
back
and
forth
about
why
this
isn't
working
this
potato
way.
So
I'm
sure
a
lot
of
this
represents
the
discussion
we've
had
about
Windows
versus
Linux
and
how
to
work
on.
All
of
that
we
are
making
progress
on
all
that
I'm
encouraged
to
see
that,
but
anyway,
that's
sort
of
the
state
of
today.
E
Most
of
the
stuff
I
said
is
not
in
the
speakers
notes.
So
please
watch
the
recording
if
you
missed
this
I,
just
figured
I'd.
Give
you
some
pictures
to
look
at
while
I
talked
these
are
the
sources
of
the
data
that
I
showed
you.
The
last
two
links
are
spreadsheets
that
are
accessible
to
all.
The
first
is
the
repo
I
use
to
build
the
data
and
generate
those
graphs
and
stuff
or
coupe
con
China.
That's
it
Thanks.
B
F
D
This
graph
here
is
from
our
in
the
end
conformance
tests.
It
is
for
our
release
blocking
jobs,
so
this
is
what
we
use
to
stop
broken
releases
from
making
it
to
our
end
users.
So
this
is
one
way
of
defining
if
it
gets
passed
here.
People
are
consuming
it
and
I've
limited
with
some
of
our
filters
over
here
that
we're
not
looking
at
anything
hit
by
our
end-to-end
test
suite.
So
all
of
these
95
endpoints
we're
not
touched
by
our
intent
test
suite,
and
this
is
zoomed
in
to
just
stable.
D
So
this
green
area
here
I'll
move
down,
so
you
can
see
the
maybe
have
to
zoom
it's
just
slightly.
The
legend
down
here
shows
that
the
green
is
stable.
So
not
only
is
this
things
that
are
used
during
our
release
blocking
jobs,
it's
the
stable
things
that
are
used
by
a
release
blocking
jobs,
and
if
we
were
going
to
try
to
prioritize
endpoints,
I
think
things
that
are,
we
used
they're
a
release
blocking
jobs
that
are
not
tested
or
probably
a
good
list.
D
D
So
if
you
wanted
some
quick
insight
as
to
what
we
might
should
prioritize,
this
first
link
is
not
a
bad
list.
It's
not
prioritized
yet,
but
that's
okay.
We
can
start
looking
at
other
metrics
to
to
prioritize
that
stuff.
I'm
gonna
go
back
to
our
slides
now,
so
this
is
the
stables.
What
can
we
do
if
we
knew
this?
We
could
prioritize
writing
tests.
D
Our
next
question
is
going
beyond
stay
a
little
bit
stable
core,
and
we
asked
a
different
question
about
our
stable
core
stuff.
That's
already
tested,
but
it's
not
yet
conformant,
because
these
are
those
easy
upgrades
that
we
saw
that
nice
list
of
things
in
113
you're
like
ooh,
we've
got
tests,
that's
just
magnums
conforming,
but
we
we
do
need
to
inspect
those
and
ensure
that
they
are
so
we've
decided
to
pull
those
out
stable
core.
D
That's
a
good
list
of
endpoints
that,
if
we're
actively
using
them
during
the
release
blocking
job
and
their
their
tests,
it's
probably
a
good
place
that
we
should
look
for
the
using
of
a
conformance,
so
I'll
zoom
in
on
this
list.
I'll
click
and
launch
will
add
to
the
sidebar
at
some
point
these
exist,
but
these
are
30
different
end
points
and
if
you
mouse
over,
you
can
see
where
they
are
or
scroll
down
forget
for
how
scroll
down
there
we
go.
There's
the
end
point
list
you
get
the
idea
and
then
Rico.
D
D
So
this,
what
we're
asking
here
is
not
stuff
around
the
ete
test,
but
the
specific
user
agents
that
are
core
to
kubernetes.
This
is
poor
kubernetes
talking
to
itself
that
we
don't
hit
at
all
during
our
tests.
So
we'll
click
on
this
next
link
and
you'll
notice.
Over
here
there's
a
regular
expression
for
Kubb
dash
star
and
if
I,
if
you
look
at
what
is
that
batch
well,
will
drop
down
the
match.
D
This
is
Kubb
API
server,
coop
controller
manager,
coop
scheduler
and
cout
proxy
in
there
they're
likely
others
that
you
can
come
up
with
your
own
regular
expression
to
search
inside
these
audit
logs
and
see
what
was
of
interest.
This
is
not
focusing
on
core.
This
is
all
the
endpoints,
but
there's
a
hundred
and
twenty-five
endpoints
that
are
being
hit
by
the
cluster
itself
that
are
primed
for
writing
some
tests
of
some
type
and
likely
since
we're
talking
about
the
core
components
of
kubernetes
itself.
These
are
high
important
endpoints
to
add
test
coverage.
D
For
so
again,
you
can
see
that
list.
These
three
questions
give
us
a
prioritized
list
of
endpoints
and
so
far
this
journey
that
we've
been
in
on
its
kind
of
focused
on
that
metric.
We've
seen
the
information
provided
by
Aaron
on
we're
going
up
into
the
right
and
and
that
that's
great
and
I
and
I
and
I
think
this.
We
need
to
continue
to
get
to
a
point
where
we
have
a
hundred
percent
coverage
on
a
least
stable,
core
and
stable,
particularly
the
components
that
our
own
software
uses
within
communicating.
The
API
server
go
ahead.
D
F
D
D
What
I'm
super
interested
in
is
seeing
what
analysis
our
community
can
do
now
that
we
can
easily
define
where
these
audit
logs
are
coming
from
by
just
going
into
the
API
snoop
and
creating
a
PR
you
can
modify,
which
jobs
that
the
data
is
coming
from
and
when
you
have
inside
your
PR
without
even
asking
us,
there
will
be
a
net
Liffe
ID
job
and
you
can
browse
through
and
say.
Well,
we
read
X
out
here
for
our
stuff
and
find
some
interesting
things
with
what
we
currently
have.
D
There
are
questions
that
we
can't
answer
yet,
even
if
you
were
to
submit
PRS
against
the
different
things
or
give
different
links
to
API
snoop
and
that's
what
I
wanted
to
kind
of
see
what
we
could
do
next
and
do
better.
So
these
are
additional
questions
that
we've
thought
of
and
then,
after
that,
I
want
to
see
what
other
questions
we
can
come
up
with.
D
So
that's
an
obvious
next
step,
another
one.
The
utility
conversations
on
is
the
fields
and
parameters.
So
when
our
applications
are
using
the
kubernetes
api,
there's
many
different
fields
and
parameters
within
each
of
the
objects
and
kinds
that
we
care
exploring.
If
we
knew
which
ones
were
used,
we
could
identify
and
prioritize
the
optional
and
mandatory
fields
and
have
these
patterns
of
usage.
These
would
encourage
us
to
write
tests
for
things
that
are
heavily
used.
Another
ask
that
I've
seen
is
what's
changed
since
last
week.
D
We
have
these
nice
bars
up
into
the
right,
but
as
we're
doing
test
coverage,
it's
not
just
about
how
many
endpoints
did
we
do.
What
is
the
new
name
of
the
test,
but
what
new
parameters
are
recovering?
What
new
behaviors
are
recovering?
Who
was
involved
in
the
changes
this
last
week?
We
can
measure
many
things
there,
so
this
is
identifying
coverage
and
parameter
and
and
field
coverage,
but
also
who
is
doing
the
work,
because
I
think
we
need
to
acknowledge
the
the
people
doing
the
heavy
of
the
hard
yards
here.
D
Those
are
the
three
general
direction
questions
we
had,
but
I've
kind
of
given
some
ideas
on
how
you
can
explore
our
current
theta
and
I
couldn't
go
if
you
want
to
I,
can
show
you
how
to
change
but
we're
playing
from
release
blocking
for
further
relief
for
the
CI
jobs.
So
what
queries
we
might
do?
What
are
the
data
sets
we
might
pull
from
and
what
other
questions
we
might
ask?
G
Sorry
one
thing
we've
talked
about
before
is
the
doing
the
field
introspection,
so
we
can
tell
more
what
those
resources
look
like,
because
there's
a
lot
of
different
behaviors
of
an
individual
resource,
but
even
taking
that
a
step
further
I
wonder
if
there's
a
way
we
can
look
at
interactions
between.
So
if
there's
a
single
task
that
creates
multiple
resources
and
then
multiple
configurations
of
those
different
resources-
I,
don't
know
if
it's
useful,
but
maybe
we
can
at
least
get
an
idea
of
coverage
of
some
of
those
interactions
and
behaviors.
From
this
we.
D
Thought
hard
about
that,
since
we
had
those
discussions
and
and
they
the
eye
on
a
piece,
the
obtainment
have
thought
about
extending
how
we
capture
those
audit
logs
right
now
we're
just
analyzing
the
entire
end
and
sweets
when
they
run
during
release
blocking
jobs
and
the
like.
But
some
more
interesting
stories
might
be
looking
at
the
documentation
that
we
have
and
taking
that
little
snippet
and
riding
the
coop
Codel
commands
that
go
through
and
perform.
D
The
expected
behaviors
that
we
have
written
in
our
documentation
and
having
those
audit
locks
processed
via
API
seem
to
say
this
is
what
we're
doing
here
and
we.
You
know
this
dude,
that's
a
good
test
to
write.
It
doesn't
write
the
test,
but
it
gives
us
this
identification
of
behaviors
based
on
flows
and
I.
Think
at
some
point
in
their
future
will
be
be
presenting
this
distich
archer
or
other
people
to
for
inclusion.
H
H
That
is
outstanding,
but
in
general
this
is
not
really
a
confinements
related,
but
the
general
idea
that,
during
our
discussion
in
Seattle
components
deep-dive
session,
we
realize
that
we
should
have
validation
test
weights.
They
are
something
like
what
we
have
today
for
node
node
conformance.
There
are
suite
of
tests
for
node
conformant.
Similarly,
validation
Suites
are
set
of
ete
tests.
They
are
grouped
based
off
of
a
specific
functionality
and
usually
the
six
are
responsible
for
for
driving.
This
work,
for
example
like
six
storage
they
have
CSI
today.
H
If
you
want
to
test
CSI
functionality,
you
need
to
have
set
of
procedures
how
to
setup.
This
is
not
expected
to
be
a
core
functionality
of
kubernetes,
but
the
CSF
functionality,
for
example
right
how
you
setup
your
driver
and
how
you
define
your
storage
class
or
what
not
so
well
dishes
Suites
will
be
written
by
six
storage,
with
a
set
of
documentation.
How
to
set
up
to
run
that
suite.
H
These
validation
Suites
can
shrink
or
grow,
and
they
don't
have
any
deprecation
policy.
They
don't
belong
to
conformance.
They
do
not
require
any
certification,
so
that
means,
as
you
evolve
your
feature
or
functionality,
there
may
be
more
tests
added
or
functionality
changes
from
release
to
release.
You
can
change
the
tests
or
delete
some
of
the
tests
that
are
relevant
to
that
functionality
and
six
are
responsible
for
for
defining
the
functionality
as
well
as
how
to
run
them.
H
I
H
Promoted
one
of
the
conformance
tests
into
conformance
test
weight
which
we,
which
is
part
of
the
core
functionality,
so
similar
things
can
happen
to
validation
tests
and
it
brings
us
closer
to
promoting
two
conformance.
That's
the
basic
idea.
So
once
you
have
validation
Suites,
we
know
exactly
how
to
group
the
functionalities,
and
then
we
can
think
whether
that
behaviors
belong
to
the
core
functionality.
So
is
it
a
promote?
So
the
idea
is
also
that
we
want
to
document
them
properly.
H
So
we
expect
the
defenders
to
run
validation,
test
Suites,
so
they
it
has
to
be
much
more
user-friendly.
That
way,
you
don't
have
to
look
at
the
code
to
understand
what
the
test
does,
but
generally
the
documentation
should
help
and
the
the
proposal
is
Brad
wrote
the
proposal,
the
PR,
for
how
validation
tests,
which
should
work
and
I'll,
actually
show
the
PR
in
a
second,
but
to
continue
on
this.
What
validation,
test
documentation
should
look
like
similar
to
conformance
test
documentation.
H
H
Think
of
it
I
mean
the
sig
will
define
how
to
run
the
validation
test
suite,
but
generally
today,
a
component
is
a
ginko
focus
or
conformance.
Similarly,
you'll
have
ego,
focus,
CSR
validation,
test
or
focus,
not
valuation
test.
What
so
that's
a
whole
idea?
This
is
there's
a
PR
still
I
referred
in
the
slide
deck
post
shortly
that
open,
open
and
feel
free
to
discuss
your
viewpoints
on
this
PR.
This
PR
is
ready
for
review
final
reviews.
Actually,
if
I
may
I'll
also
talk
about
the
project.
H
H
See
LCF
project
work
board,
so
this
is
how
complex
it
is
all
the
conformance
tests.
Today,
all
the
components
work
you
can
see
here.
I
mean
there
is
a
notion
that
conformance
test
suite
is
pretty
static.
It's
not
gonna
change,
but
it
is
not
true.
Basically,
a
lot
of
conformance
tests
are
being
added
right.
So
to
get
a
better
view
for
end-user,
they
can
go
to
the
project
code
word
for
components
and
there
is
a
process
how
the
the
lifecycle
of
a
PR
that's
having
it
under.
H
What's
Becca,
rich
and
as
the
PRS
there
are
a
lot
of
peers
coming
in
the
parts
vector
is
right
now
and
they
are
going
to
the
in
progress
and
in
review
where
the
six,
as
well
as
conformance
folks,
are
reviewing
these
peers
and
eventually
they'll
go
into
needs.
Approval
column
where
the
actual
conformance
leads
will
approve
the
PR.
That's
when
the
conformance
test
goes
into
that
release.
H
Talked
about,
we
need
to
write
new
you
to
you
test
to
fill
the
gaps,
and
then
we
have
to
then
promote
that
test
into
the
into
the
conformance
the
next
release
cycle,
so
so
to
automate
this.
There
are
a
couple
ways
we
are
working
on.
Those
are
again
outstanding
PRS
as
soon
as
there
is
an
issue
for
example,
in
this
case,
you
can
assign
the
issue
to
the
project
board.
So
there
is
a
pro
plugin
that
is
under
works.
H
H
So
that's
where,
when
you
tag
this
just
my
testbed,
so
as
soon
as
you
tag
the
PR,
the
PR
will
show
up
right
here
in
the
project
board.
There
are
ways
we
are
automating
this
process
that
it
it
at
least
shows
up
for
us
to
to
start
working
on
it
and
triaging
it
and
moving
it
through
that
lifecycle.
I
showed
you,
that's
all
I
have
any
questions.
A
G
Thank
you.
So
one
thing
we've
been
talking
about
a
lot
in
the
meetings
is
around
how
we
measure
coverage
and
API
Snoop
is
one
way
we
do
that.
But
part
of
the
problem
is
that
we
haven't
really
documented
all
of
the
things
we're
trying
to
cover,
so
you
can
have
tests.
But
if
you
don't
know
the
universe
that
you're
trying
to
test
you
don't
know
how
much
of
that
university
is
covered.
So
I
wanted
to
talk
a
little
bit
about
how
we
identify
that
and
how
we
target
our
tests.
I
G
The
tests,
which
includes
identifying
what
they
should
be
executing
them
the
results,
evaluation,
the
results,
reporting
and
then
eventually,
some
sort
of
evaluation
of
workload
portability.
So
all
these
have
been
brought
up,
I
think
by
previous
people.
I
wanted
to
talk
really
just
on
the
test,
development
and-
and
just
briefly,
I
I,
tend
to
think
of
this
in
these
four
stages.
For
the
development
of
the
test,
one
is
just
identifying
the
behaviors.
We
want
to
test
classifying
them,
and
this
gets
into
the
validation
Suites.
Some
of
them
get
get
malady.
G
These
aren't
even
necessarily
conforming
behaviors,
just
just
in
testing
in
general,
developing
the
tests
to
cover
those
things
and
measuring
the
coverage
against
our
list.
So
the
couple
of
things
I
wanted
to
point
out
here
or
discuss
the
things
in
green
are
the
things
we're
actually
doing
right
now
as
part
of
this
conformance
program,
so
we
are
promoting
existing
end-to-end
tests.
G
Maybe
we
are
also
writing
some
new
tests
that
should
probably
be
green,
but
as
far
as
is
identifying
the
behaviors
we're
just
looking
at
the
existing
end-to-end
tests,
we're
not
really
documenting
the
the
behaviors
from
a
more
holistic
point
of
view,
at
least
not
beyond.
What's
out
there
right,
all
of
that's
out
there
in
the
docs,
all
of
its
out
there
there's
a
lot
embedded
in
the
API
schema
and,
of
course,
the
code
and
and
really
there's
still
a
lot
of
stuff.
G
That's
just
in
individual
people's
heads
on
how
all
this
stuff
should
work,
and
so
what
I'm
proposing
is
that
we
we
make
a
sort
of
we
step
back
a
little
bit
make
a
concerted
effort
to
to
come
up
with
how,
where
I
come
up
with
new
ways
to
identify
the
tests
and
new
ways
to
document
in
a
machine-readable
way
what
those
tests
are
and
then
that
can
flow
through.
We
can
classify
them
into
their
future
associated
features
so
and
then
we
can
classify
which
of
those
are
actually
should
be
conformance
related.
G
The
the
people
who
need
to
review
and
approve
whether
a
given
behavior
should
be
a
conformance,
behavior
or
is
or
is
not
required
for
conformance
I,
think
is
a
different
and
very
small.
Select
group
of
people
who
have
very
low
bandwidth
because
they're
so
committed
to
other
things,
and
so
part
of
the
goal
I'd
like
to
achieve
is
separating
the
review
process
for
the
identification
of
those
behaviors
from
the
review
process
of
we
have
a
test
for
it
and
that
test
accurately
tests
that
behavior
I
think
a
lot
more.
G
People
can
do
that
second
thing,
and
that
second
thing
actually
is
a
lot
of
work
right.
It's
so
you
can.
You
can
state
a
behavior
if
you
go
to
the
API
documents
and
you
just
pick
out
a
field,
there's
a
statement
on
the
behavior
of
what
that
field
does,
and
that's
pretty
easy
for
someone
to
say
yes
or
no.
That
should
be
part
of
conformance,
whereas
actually
reading
the
test
that
evaluates
that
behavior
and
and
in
different
circumstances
and
making
sure
it
works
and
follows
all
the
conformance
test
guidelines.
It's
a
substantially
more
more
work.
G
So,
in
any
case,
this
is
kind
of
really
was
just
intended
to
start
the
discussion
or
continue
the
discussion
that
we've
had
what
I
would
like
to
come
out
of
may
be.
I
guess
I
expected
this
to
be
last
presentation
and
more
discussion,
and
so
what
I
would
hope
we
can
come
out
after
the
deep
dive
after
this
part
of
it
is
some
consensus
on
an
approach,
and
at
that
point
you
know
we
can
start
to
formalize
it
into
something
more
in
the
cap.
So
any
questions
on
this
or
comments
Jago.
J
Proposing
generally
is
to
flip
the
model
from
someone
goes
off
and
does
a
lot
of
work
to
clean
up
and
propose
that
a
conformist
test
be
merged
into
the
white
gold
list
of
conformist
tests,
and
only
then
do
folks
in
Sigma
architecture
say.
Actually
no,
this
isn't
quite
right.
That's
not
what
we're
trying
to
focus
on
right.
I
J
G
G
What
I'm
suggesting
is
that
we
should
also
be
trying
to
do
that
upfront
work
to
decide
on
what
our
end
goal
is
and
if
we
can
identify
right
now
we
don't
know
we
could
keep
running
into
estás
until
the
end
of
time,
and
we
don't
know
if
we've
covered
everything,
because
we
don't
have
any
real
way
to
measure
that
we
can
use
some
after
the
fact
things
which
we're
trying
to
do
with
API
snoop.
But
how
do
you
measure
this
is
kind
of
what
I
was
getting
at
before?
If,
if
pada
pada
networking
right?
G
That's
not
that's
something
we
require,
but
it's
not
something
you
can
measure
in
a
test
that
just
checks
an
API
field
right.
So
there's
there's
just
it's
something
somebody
has
to
think
of
to
to
put
a
test
down
on
and
I
think
that
we're
not
formalizing
that
process
of
thinking
and
documenting
what
others
are.
We
haven't
been
some
umbrella
issues
and
things
like
that,
but
it
it's
just
very
difficult
to
track
whether
we've
covered
them
so.
J
G
Like
is
fact
I
would
I
would
probably
go
the
other
way.
I
would
start
from
our
API
schema,
because
the
schemas
already
very
it's
already
machine
digestible
and
now
we
can
go
through
that
and
we
can
identify
I
mean
actually
some
of
the
globin
guys
did
this.
They
went
through
and
did
it
by
hand
right.
Okay,
let's
go
see
the
IPS
give
me
here's
all
the
different
API
fields
in
pods
back
and
we're
gonna
need
tests
for
every
one
of
these
and
some
of
them
have
tests
today,
some
of
them
don't
a
lot
of
times.
G
There's
interactions
between
them
that
aren't
tested.
So
you
can
get
a
start
by
going
through
that
API
schema
and
just
spitting
out
a
bunch
of
tasks
you
still
have
to
see
which
ones
are
covered
and
which
aren't
you
can
also,
and
partly
you
can
do
that
through
the
API
snoop.
You
can
see
what
what's
been
covered,
especially
if
we
add
the
field
introspection
there,
so
I
think
there's.
That's
that's
like
to
me.
The
the
API
Snoopy's
gets
into
the
measuring
average.
G
The
way
we
can
identify
baby
but
I
think
it's
just
got
to
be
an
active
process
of
we
need
to
have
you
know
a
concerted
effort
to
put
create
some
tooling
around
the
API
schema
to
go
through
the
docs
and
to
write
these
things
down.
I
love
a
suggestion
that
would
purely
automate
it,
but
I
have
yet
to
hear
one
so.
J
We
we
did,
there
is
a
PR
that
died,
a
death
of
lagging
into
oblivion,
where
we
tried
to
do
it,
data-driven
go
through
all
the
api's
and
and
validate
that
we
can
create.
We
can
do
all
crud
operations
on
all
objects
right.
The
value
of
that
was
kind
of
unclear,
but
it
might
be
worth
at
least
looking
into
that.
B
I
B
Obviously,
there's
no
payload
and
there's
no
intelligence
to
it,
but
I
never
quite
understood
why
it's
a
bad
as
a
step
to
have
what
you'd
call
extremely
shallow
testing
as
opposed
to
the
much
deeper
testing
that
that
we're
focused
on,
in
particular
in
a
scenario
where
we're
worried
about
vendors.
Turning
off
EP
eyes
and
I
mean
very
explicitly
when
we
launched
the
we
had
major
vendors
flatten
members
of
cm
for
good
internal
reasons
that
turn
them
back
on,
so
that
they
could
pass
the
certification
test.
And
it
seems
to
me,
like
the
incredibly
silly
naive.
J
J
G
I
think
that
gets
to
sort
of
there
there's
III,
agree.
I,
don't
see
that
as
it's
not
harmful,
they
have.
That
is
relatively
easy
to
do
right,
but
it
doesn't
make
it
a
false
sense
of
confidence.
If
it's
your
only
signal,
but
if
you
got
so
it's
not
sufficient.
We
all
know
that
we
all
agree
with
that.
The.
B
B
G
D
And
we
spent
some
time
Mada
adapting
the
existing
kubernetes
documentation
in
a
form
that
would
be
executable
and
when
you
go
through
and
follow
the
steps
defined
in
the
documentation
it
against
the
living
cluster.
It
performs
these
operations
and
if
you
go
through
and
actually
test
with,
coop
cuddle,
just
simple
non
programmer
interactions
to
to
document
that
it
we've
thought
about
a
way
to
turn
on
the
API
snoop
the
audit
logging
before
and
then
we
go
through
and
it
executes
these
these
behaviors.
D
G
I
think
that
would
be
really
interesting
that
that's
sort
of
like
the
documentation
isn't
identified
and
then
there's
the
widget
that
we
talked
about
the
executable
Docs
we're
talking
about
there
and
that
would
be
sort
of
like
as
you're
going
through
and
identifying
these
tests,
rather
than
just
adding
them
to
some
list
somewhere.
You
could
actually
convert
them
into
some
executable
test.
Write
that
in
there
I
think
there's
value
in
that
I
think
that
you'll
probably
find
a
lot
of
the
examples.
G
Don't
work,
but
the
my
only
real
concern
there
is
that
those
are
getting
scattered
everywhere
throughout
the
documentation
and
there's
not
like
one
place.
Maybe
you
can
have
tooling
that
pulls
them
all
together,
but
those
are
very
user
oriented.
And
how
do
you
say
this
sequence
of
five
steps
is
one
conformance
test
to
me.
That's
a
little
bit.
It's
a
great
end-to-end
task,
maybe
I'm
not
sure
if
it's
one
conformance
if
it
gets
the
conformance
tag,
but
it's
something
it's
debatable
in.
D
G
G
So
I
guess,
where
I,
where
I
would
want
to
go
with
this
as
I
get
the
feeling
of
the
room
here,
I'm
happy
to
start
working
on
trying
to
formalize
this,
some
more
and
I
mean
I.
Think
a
big
problem
we've
had
in
this
group
is
staffing
and
I
know.
We've
got
some
contractors
to
help
us
some
things.
I
think
we
need
to
have
a
little
more
structure
around
guiding
them.
So
that's
what
I'm
hoping
to
do
here
and
you
know
any
other
to
me.
J
Think
it
would
be
great
to
identify
some
big
themes
in
addition
to
the
stable
core,
I
would
propose
security
as
a
topic
where
portability
of
security
policy
would
be
important
to
end-users,
and
this
is
just
becoming
more
of
a
concern
as
there
are
more
vulnerabilities,
exposed
and
publishing
best
practices
and
how
to
have.
How
do
we
ensure
that
those
best
practices
can
be
adopted
by
end-users
and
all
of
these
conformant
clusters?
J
G
That
that
makes
sense
one
thing,
speaking
of
areas
of
focus
like
one
of
the
things
areas
where
there's
a
likely
variation
between
vendors
or
between
distributors,
like
with
all
the
plugins
and
I'm
wondering
I,
was
working
thinking
about
what
you
had
with
yours,
to
see
us
CSI,
testing
and
I.
Guess:
I
wonder
if
we
should
have
some
sort
of
mock,
but
maybe
they
already
do
and
ICS.
Maybe
the
CSI
guys
already
have
it.
G
Something
like
that
and
but
I
wonder
about
the
the
the
conformance
testing
right
now
runs
to
does
end
antastic.
This
might
get
into
the
how
we
actually
get
the
tests
like
when
we
run
those.
Maybe
this
is
already
I'm,
just
not
quite
familiar
enough,
but
if
I
create
my
cluster
I
guess
if
you
create
your
cluster
ahead
of
time,
so
you
create
with
whatever
CSI
driver
and
whatever
CNI
driver
you
want,
and
then
you
run
those
tests
and
we
see
what
happens
so.
I
guess
covered.
H
Yeah
part
of
the
problem
right
now
for
for
the
workout
incontinence
is
that
behaviors
are
hard
to
pick
from
from
just
the
API
documentation.
So
we
need
to
actively
start
building
that
so
that
we
can
have
a
healthy
pipeline
of
work
because
there
are,
like
you
said
we
have
enough
of
people
to
work
on
generating
the
EDA
test
cases,
but
or
initially
it
is
a
lot
easier
for
us
to
go
through
all
the
details
and
see
what
can
be
promoted.
But
as
we
go
further
further
I'm.
G
We
need
to
split
this
because
it's
Windows
and
that
parchment
reaction
windows
and
that
part
doesn't-
and
we
should
split
it
up,
and
it's
like
all
this
pre
work
that
that
it's
actually
writing
you
either
he
tests
is
what
it
comes
down
to
I,
don't
know,
I
I,
think
if
we
have
that
driven
list
or
that
prioritized
list,
we
can
look
at
what
what
tests
already
meet
those,
but
a
lot
of
them
do
stuff
that
you
they
were
written
without
conformance
in
mind,
so
they
do
stuff.
That's
true.
H
J
J
So
in
the
security
case,
maybe
we
can
go
to
the
security
sig
and
say
this
is
the
validation
suite?
Can
you
you
know
our
back?
What
does
it
mean
for
our
back
to
work?
Are
there
already
our
back
tests
that
could
be
promoted
and
have
them
go
through
the
exercises?
The
subject
matter,
experts
and
that
seems
to
work
much
better,
especially
as
kubernetes
gets
more
states
rights
rather
than
federal
government
ties
they.
They
do
have
a
lot
more
context
than
folks
trying
to
go
across
the
whole
service
area.
Yeah.
C
F
C
G
H
A
A
G
A
G
I
B
G
A
A
A
J
Mechanism,
yeah
yeah,
it's
just
formalizing
the
process
so
that
it
doesn't
get
missed.
I
know
it
stopped
about
the
two
that
are
on
my
radar
right
now
or
the
extensibility
features
CR
DS
and
getting
those
two
GA
and
ingress,
which
is
early
tricky
one
and
those
getting
those
two
GA
by
the
end
of
2019
part
of
that
discussion
and
plan.
All
along
has
been
the
conformance
tests
as
part
of
that,
and
so
folks,
who
do
a
lot
of
the
reviews
are
keeping
that
in
mind.
J
We
also
do
scale
testing
when
it
goes
to
stable,
which
never
happens
before
that
and
I.
Think
getting
all
of
the
required
api's
to
stable
or
GA
means
that
they
get
backwards,
compatibility
guarantees
and
it
means
they
get
scale
tested.
So
this
is
really
important
to
our
bigger
customers
and
I'm
sure
everyone
else's
yeah.
B
B
B
G
I
G
A
A
If
I
mean
if
the
test
goes
with
the
feature,
that
should
be
okay
but
I
actually
agree
with
you,
I
think
these
tests
should
exist
in
the
beta.
In
fact,
I'd
like
to
see
sooner
boy
have
kind
of
an
optional
like
beta
option
where
you
you
get
like
the
definitive
conformance
result
right
and
then
you
kind
of
get
like
a
tentative
result
for
features
that
are
coming
down
the
pipeline.
So
if,
ideally,
if
these
tests
doing.
A
News
you'd
be
like
oh
this
one's
family,
which
is
your
cue
to
go
to
the
community.
Well,
it
either
like
check
to
see
why
it's
failing
and
legitimate
reason
fix
that
reason.
If
you
think
the
buggies
in
the
test,
which
does
happen
sometimes,
then
you've
got
a
window
to
actually
go
to
the
community
and
say
hey
I'm,
seeing
a
probably
this
tester
also
seen
this,
and
so
then
we
have
some
notice
and.
G
A
Actually,
in
distribution
like
I
do
I
would
like
to
see
this
in
the
kit
proposal,
like
maybe
we
can
incrementally
start
adding
these
things,
but
I
like
to
say
actually
like
why
we
promoting
used
to
bait
it
without
test
like
to
me
that
doesn't
actually
make
sense,
no
for
I
kind
of
get.
But
then
I'd
like
to
see
the
the
beta
promotion
has
the
test
and
then
it's
that
same
test,
you
know
getting
finalized
I.
G
G
A
I
C
J
G
F
First
comment
about
this,
so
basically,
I
was
like,
like
I
used
to.
There
was
a
platform
which
was
conformant.
That
I
would
really
love
to
see
that
at
least
no
conformant
desert
will
go.
They
become
confirmed
this
next
version
and
run
them
in
my
cluster
in
advance.
So
I
won't
have
some
surprises,
because
we
we
had
some
that
was
back
in
kubernetes
when
I
didn't
want
conformance
tests.
When
you
have
proxy
environment
variable
and
like
because
of
weight,
we
had
to
get
notion.
F
Okay,
it
doesn't
feel
clear
when
you
have
protein
bar
variable,
but
in
the
next
version
of
commodities
it
will
work.
So
we
have
to
change
our
CI
just
give
the
desert
not
another
I
would
really
love
to
test
it.
A
little
bit
advanced.
Okay.
This
will
be
conformance
test.
It's
fine
with
my
cost
I
can
raise.
It
is
a
Cabarrus
community
in
advance
and
not
after
the
fact,
my
lime
table
day
to
next
your
Shannon
Terron
eaten.
J
F
J
Just
one
one
other
come
in
on
this:
if
folks
are
not
aware
as
part
of
the
st.
cloud
provider,
we
work
with
sync
testing
to
open
up
test
grids
such
that
providers
can
run
the
confirmes
test
suite
in
their
own
environments
and
submit
the
results
back
to
test
grade
and
show
up
as
part
of
an
overall
dashboard
of
all
the
conformance
tests
that
run
on
all
of
the
environments.
And
this
is
how
cloud
providers
get
early
warning
that
someone
else
inadvertently
broke
their
implementations
or
that
a
conformist
test
doesn't
work
on
some
upcoming
version.
I.
J
The
proposal
here
that
I
agree
with
is
to
consider
the
e
de
tests
that
are
being
proposed
for
a
beta
API
in
the
context
of
conformance
and
what
would
be
the
additional
work
to
promote
that
to
conformance.
We
pushed
that
discussion
upstream,
rather
than
waiting
until
someone's
hopes
and
dreams
are
ongoing,
GA
and
116,
and
then
they
get
feedback
that
it
needs
a
conformist
test,
and
this
isn't
gonna
cut
it,
because
that's
a
very
discouraging
process
to
go
through
is
that
right.
John,
I.
A
Agree,
we
should
definitely
bring
it
aboard,
but
at
the
same
time
also
that
that
risk
is
also
a
huge
motivation.
But
but
I
agree
with
you.
Okay,
so
I
hate
I
hate
to
cut
it
short
just
get
into
a
good
discussion.
But
hopefully
hopefully
everyone
can
take
this
energy
forward
and
and
we
can,
we
can
improve
all
the
work
collaboratively.