►
From YouTube: Extending Kubernetes or: How I Learned to Stop Worrying and Trust the Spec - David Zhu, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Extending Kubernetes or: How I Learned to Stop Worrying and Trust the Spec - David Zhu, Google
Kubernetes Storage is moving to a new model for volume plugin development with the new Container Storage Interface (CSI) specification, and engineers from multiple companies are now implementing CSI drivers. However, switching to this new paradigm is not without its fair share of challenges. This talk explores some of the important lessons learned from writing a production-grade CSI driver and discusses some of the challenges that come with conforming to an actively evolving spec. Attendees will learn about dealing with concurrent development across multiple projects, working with different interpretations of the same specification, and getting user traction on a new project when the current alternative “just works.” These issues become more relevant as Kubernetes moves towards an increasingly decoupled architecture.
To learn more: https://sched.co/GrUU
A
Hi
everyone,
my
name
is
David
and
today
I'm
going
to
be
talking
to
you
about
extending
kubernetes
or
how
I
learned
to
stop
worrying
and
Trust
the
spec.
So
in
this
talk,
I'm
going
to
be
talking
about
specifications
in
general,
but
I'll
be
pulling
examples,
and
lessons
learned
from
my
own
experience,
working
with
the
container
storage
interface,
which
is
a
storage
related
specification
that
we'll
discuss
a
little
bit
later,
but
just
as
an
example
for
the
lessons
that
we
learned
so
before
we
start
a
little
bit
about
me.
A
Sorry,
a
little
bit
about
me.
I
am
a
software
engineer
at
Google
I'm,
a
current
cig,
storage,
contributor
and
I've
been
working
on
and
with
the
container
storage
interface
since
before
0.1,
and
we
actually
hit
a
pretty
big
milestone
recently,
where
we
released
version
1
with
kubernetes
113
I'm.
Also,
a
developer
for
the
GCP
persistent
disk,
CSI
driver,
so
I've
kind
of
worked
on
both
ends
of
the
specification
and
I'll
talk
a
little
bit
more
about
what
that
means.
A
I
have
to
show
you
what
it
was
like
in
the
dark
ages
of
the
now
ancient
1.9
super
important
to
note
that
this
is
probably
the
system
that
you're
still
using
now,
and
it's
not
actually
that
ancient.
This
is
maybe
three
months
ago
or
a
year
ago
anyway.
So
to
motivate
this
talk,
I
want
to
talk
a
little
bit
about
what
storage
looks
like
in
kubernetes
right
now.
A
So
basically,
what
happens
is
storage
vendors
that
want
to
provide
their
storage
in
kubernetes
to
kubernetes
workloads,
kind
of
write,
special
bits
that
make
their
storage
available,
and
this
is
actually
a
screenshot
of
the
kubernetes
codebase,
and
you
can
see
that
all
of
these
vendors,
all
of
these
driver
vendors,
build
their
code
directly
into
kubernetes,
so
that
ends
up
in
the
kubernetes
cube
controller
manager,
as
well
as
the
cubelet
okay,
and
so
in
this
paradigm,
actually
being
a
maintainer
for
kubernetes.
It
was
super
hard
and
I.
Think
Michelle
said
it
best.
A
Who
is
a
kubernetes
storage
guru
by
the
way
where
she
says
code
for
storage
plugins
that
are
not
supported
on
your
platform
are
still
a
risk,
and
so
going
back
to
how
everything
is
built
into
the
same
binary.
A
good
example
of
this
would
actually
be
this
pretty
nasty
bug
we
had
a
while
back
where,
if
you
tried
to
provision
a
like
a
volume
for
a
storage
provider
that
was
not
on
our
system,
it
would
actually
go
into
this
storage
providers
code.
A
This
vendor
code
would
actually
look
for
config
that
did
not
exist,
because,
obviously
we
don't
actually
support
that
vendor
and
it
would
seg
fall
and
you
may
be
thinking
what
what
seg
faulted
well
kubernetes
seg
faulted.
This
was
actually
the
cube
controller
manager
that
was
running
this
code
and
would
just
blow
up
if
you
tried
to
use
code
that
really
shouldn't
even
be
in
there,
because
we
don't
support
that
vendor
at
all
right.
So
that's
one
of
the
problems,
but
on
the
other
end,
so
being
a
storage
vendor
was
also
really
difficult.
A
A
So
here's
a
quote
from
singh
who
works
on
open
sts?
What
she
said
is
if
we
found
a
critical
bug
in
crew
or
in
our
plugin
after
the
kubernetes
release,
we
don't
really
have
a
way
to
release
the
plug-in
separately,
and
so
this
comes
down
to
the
same
problem.
I
have
a
bug
I
immediately,
because
I'm,
a
great
developer,
I'm,
seeing
I,
find
make
a
bug,
fix
and
I
immediately
submit
it.
Well.
First,
you
have
to
get
through
the
kubernetes
PR
process
right.
You
have
to
get
through
all
the
reviews.
A
All
the
approvals
which,
if
you've
worked
on
kubernetes,
you
already
know,
is
quite
a
lot
to
ask
and
then
once
that
happens,
you
can't
actually
release
a
new
version
of
your
plugin
at
all.
You
have
to
wait
for
kubernetes
to
then
give
another
like
dot
X
release
before
your
bug
is
fixed,
so
obviously
that's
not
ideal
as
well.
A
A
So
I
didn't
really
cover
that
last
bit
too
much,
but
as
a
kubernetes
approver,
you
had
to
review
and
maintain
code,
that's
all
built
into
the
codebase
that
you
may
not
have
a
lot
of
contexts
or
expertise
on
so
hopefully
I've
convinced
you
that
we
need
something
to
fix
it,
and
what
we
came
up
with
was
the
container
storage
interface.
Okay,
so
I'll
talk
a
little
bit
more
about
what
this
is
the
way
I
see
the
container
storage
interface
might
not
be
exactly
what
everyone
sees,
but
I
see
a
few
story.
A
A
few
sidecar
containers,
some
storage
drivers,
so
those
these
are
equivalent
to
those
entry.
Vendor
directories
that
you
saw
earlier
some
pieces
inside
a
container
Orchestrator.
In
our
case,
we
care
about
kubernetes,
but
also
this
is
supported
by
mesosphere
and
cob
boundary,
as
well
as
the
specification
itself,
which
is
kind
of
like
an
RPC
layer
that
vendors
and
cloud
provide
our
container
orchestrators
can
both
code
to
and
kind
of.
Most
importantly,
it
lets
us
move
the
storage
provider
code
outside
of
crouppen
okay.
A
So
what
does
it
look
like
exactly?
It
kind
of
looks
like
this.
This
is
by
no
means
like
a
complete
diagram.
I
just
picked
a
couple:
random
storage,
vendors
here
and
I-
think
maybe
there
are
more
container
orchestrators
that
support
it,
but
we
have
this
RPC
boundary
in
the
middle
and
we
have
people
consuming
it
from
almost
two
different
worlds
right,
and
this
is
gonna
motivate
some
of
the
lessons
learned
that
I'll
talk
about
later.
So
what
is
the
specification?
Look
like
exactly
well
here
at
pages
1
through
3
I'll.
A
Give
you
a
little
bit
of
time
to
read
through
it.
Alright,
so,
hopefully
now
we're
all
experts
on
the
specification.
So,
let's
move
on
I'm
just
kidding
by
the
way.
I.
Don't
expect
you
guys
to
know
anything
about
the
specification
as
I
alluded
to
earlier.
This
talk
isn't
actually
about
the
container
storage
interface
right.
It's
actually.
What
I
want
to
do
is
just
share
some
challenges
that
we
faced
and
some
of
the
learnings
that
we
had
from
building
an
ecosystem
around
this
specification.
A
So
the
problem
here
are,
you
may
remember
this
diagram
from
three
slides
ago.
Basically,
what
you
may
have
noticed
is
that
quite
a
lot
of
people
care
about
the
specification
and
what
it
does
cuz
there's
people
on.
You
know
this
there's
I
four
over
here,
but
there's
actually
probably
more
something
like
30
vendors
right,
then
there's
sorry,
then
there's
container
orchestrators
on
the
other
side.
So
a
lot
of
people
kind
of
care
about
what
this
specification
is
and
there's
a
lot
of
concurrent
development
across
multiple
projects.
A
A
Sorry,
I
accidentally
skipped
a
couple
pages,
so
one
of
the
most
important
things
that
we
did
in
order
to
keep
everyone
on
the
same
page
and
to
keep
all
the
developers
kind
of
happy
and
understanding
was
using
a
transparent
and
develop
an
open
forum.
So
that's
the
octocat
from
github.
So
what
we
actually
have
is
the
specification
and
all
of
the
development
and
changes
that
go
into
the
specification
are
on
github
and
so
people
interact
with
it
like
just
like
they
interact
with
kubernetes.
They
submit
pull
requests,
people
can
review
it,
etc,
etc.
A
A
So
here's
a
quick
example:
it's
a
little
bit
hard
to
see,
but
this
is
a
change
that
I
made
from
the
very
early
days,
pre
0.1
and,
as
you
can
see
it
actually
already
has
it
had
a
hundred
plus
comments
and
about
five
reviewers,
and
the
power
of
this
is
actually
as
a
developer
as
a
storage,
vendor
developer
or
a
co
developer
when
you're
looking
at
this
call,
which
is
called
node
stage
volume
if
you're
like
what
does
this
do?
Why
did
they
put
this
in
here?
This
seems
absolutely
insane.
A
You
actually
can
go
back
on
github
and
take
a
look
at
the
original
proposal,
as
well
as
all
the
discussion,
explanation,
questions
and
changes
that
were
made
to
specification
in
the
process
of
getting
this
change
merged.
So
there's
a
full
record
of
the
discussion
and
let
me
show
you
a
little
bit
more
recent
change
as
well,
so
this
is
adding
snapshots
into
CSI.
So
this
comment
has
this:
PR
had
about
200
plus
comments
and
ten
plus
reviewers,
and
these
people
actually
represented
people
from
all
different
sorts
of
storage,
vendors
and
multiple
container
orchestrators.
A
So
all
of
them
kind
of
wanted
to
make
sure
that
the
specification
change
met
their
needs.
They
all
wanted
to
understand
what
the
change
meant
to
them.
It's
so
and
after
this
process
of
going
through
this
pull
request
merged
process.
Actually,
all
of
these
or
many
of
these
people
by
the
end,
were
completely
on
board
and
actually
become
advocates
for
your
specification
change
and
that's
invaluable.
A
When
you're
trying
to
get
people
on
board
and
for
those
of
you
that
actually
paying
a
lot
of
attention,
you
may
have
noticed
that
this
PR
was
actually
closed
and
not
merged
and
there's
a
reason
for
that.
Well,
it
turns
out
that
there
were
actually
so
many
comments
on
that
github
pull
request
that
github
was
failing
to
serve
that
website
to
people
that
were
trying
to
look
at
it
and
so
Shing.
What
Shing
had
to
do
was
actually
open.
Another
PR
that
went
through
review
again
in
order
to
get
this
feature
merged.
A
A
So
next
I
want
to
talk
a
little
bit
about
some
of
the
lessons
learned,
wet
on
developing
an
ecosystem
to
help
people
use
the
specification.
So
the
problem
here
is
that
the
specification
isn't
actually
just
an
RPC
boundary.
It's
actually
a
large
body
of
text,
kind
of
defining
the
usage,
different
error
codes
and
the
control
flow
of
what
happens
when
you're,
using
this
specification.
A
So
as
with
any
text
as
with
any
large
body
of
text,
people
from
different
backgrounds
and
different
mindsets
can
read
the
exact
same
words
on
the
page
and
actually
come
to
very
different
interpretations
of
the
specification.
So
how
did
we
solve
this
problem
or
partially
solve
this
problem?
Well,
we
actually
developed
this
thing
called
CSI
sanity,
which
what
that
is
is
kind
of
a
light
conformance
test
suite
just
to
mean
make
sure
that
the
basic
specifications
are
met.
So
this
is
by
no
means
a
full
end-to-end
test
suite.
A
So
this
is
a
real
fake
example
of
what
CSI
sanity
test
failures.
Look
like
in
my
case.
This
RPC
call
is
returning
an
error
code
3,
which
means
not
found
when
it
should
actually
be
returning
an
error
code
5,
which
is
invalid
argument.
Ok,
so
this
might
actually
seem
really
inconsequential,
not
super
important,
but
I'd
actually
like
to
dig
into
it
a
little
bit
more
and
prove
to
you
that
it
actually
is
very
important
that
we
agree
on
whether
it
should
be
not
found
or
invalid
argument.
A
So,
for
this
was
a
real
thing
that
happened
actually
by
the
way
for
the
specification
and
a
real
argument
that
we
had,
but
on
a
strict
definition
of
the
spec
for
certain
cases,
it's
actually
fairly
easy
to
make
the
argument
either
way
right
like
if
you
wanted
to
return,
not
found.
You
could
make
that
argument.
A
If
you
want
to
make
invalid
argument,
you
could
also
make
that
argument
on
strict
definition
of
the
specification,
so
different
drivers
were
actually
implementing
the
same
case
with
different
error
codes,
but
CSI
sanity
was
able
to
kind
of
catch
that,
and
what
I
noticed
was
that
previous
error
it
had
some
error
and
I
was
like.
Why
is
that
happening?
And
this
is
kind
of
important,
because
when
we
have
an
RPC
boundary,
small
inconsistencies
can
actually
cause
different
trigger
different
behaviors
upstream
right.
A
So
in
this
case,
if
my
driver
was
to
return
invalid
argument,
maybe
the
person
or
the
container
Orchestrator
consuming
it
on
the
other
end,
interprets
that
as
a
user
error
and
surfaces
it
to
the
user
for
them
to
change
their
argument.
Whereas,
if
we
return
and
not
found
error,
maybe
kubernetes
then
interprets
it
as
needing
to
create
an
object
or
something
in
response
to
that,
to
make
sure
that
we
could
actually
find
that
object
all
right.
So
it's
possible
that
these
small
inconsistencies
can
make
very
large
changes
upstream.
A
So
what
I'm
trying
to
say
is
that
conformance
tools
maintain
consistency?
Okay,
so
if
you
want
consistency
across
all
interpretations
of
your
specification,
maybe
you
should
make
a
conformance
tool
and,
lastly,
I
want
to
discuss
one
more
thing,
which
is
lessons
about
getting
adoption
for
your
extensions
and
so
the
kind
of
problem
here
is
that
a
lot
of-
and
this
will
be
shared
with
other
extension
mechanisms
as
well,
but
a
lot
of
kubernetes
extensions
are
actually
going
to
be
and
in
our
case,
is
a
replacement
for
existing
functionality.
A
Okay
and
this
pattern
might
pop
up
again,
so
we
currently
support
the
old
way
of
doing
things
while
also
supporting
container
storage
interface,
and
so,
as
a
user.
It's
it's
kind
of
difficult
to
get
users
to
move
to
the
new
way
of
doing
things.
This
extensions
way
when
the
alternative
just
works
right.
So
we
want
users
to
go
from
this
state
of
kind
of
confusion
to
a
happy
user
right,
and
how
are
we
going
to
do
that?
Well,
we're
still
in
the
process.
A
A
Get
added
to
the
container
storage
interface,
for
example.
Snapshots,
so
snapshots
are
pretty
important
functionality.
A
lot
of
people
really
want
it
and
we're
adding
it
to
CSI,
but
not
to
entry
plugins
and
currently
this
is
in
alpha.
So
in
the
future.
More
features
like
this
will
be
added
where
we're
adding
new
functionality
to
CSI
and
not
really
adding
it
to
the
older
version,
and
hopefully
that
will
drive
user
adoption.
That
will
add
enough
value
to
see
a
side
that
people
will
move
over.
A
Another
thing
that
we're
doing
that's
currently
in
flight
is
a
new
test
framework
for
CSI.
That
kind
of
lets
you
plug
your
own
driver
in
to
existing
kubernetes
e
to
e
tests
for
storage,
so
that
every
driver
can
now
run
the
kubernetes
e
to
e
suite,
and
these
are
tests
that
the
kubernetes
maintainer
x',
have
kind
of
decided
are
a
set
of
things
that
we
want
volume
volumes
in
workloads
to
do
so.
You
get
all
the
benefits
of
the
same
entry
tests,
except
you
can
run
them
on
any
CSI
driver.
A
So
currently,
I'm
sure
other
vendors
are
also
running
tests,
but
a
lot
of
times,
they're
kind
of
hard
to
find,
and
they
may
not
be
the
same
set
of
tests.
So
with
this
new
framework,
what
we
hope
to
do
is
have
everyone
running
the
same
set
of
kind
of
kubernetes
e2e
tests
and
if
you
pass
all
of
them
it's
at
least
a
basic
understanding
of
yes,
we
support
kubernetes.
A
So,
if
you're
one
of
those
people-
and
you
want
people
to
find
your
driver,
add
it
up
there
and
of
course,
I
can't
actually
attribute
all
of
the
success
of
CSI
to
those
two
previous
examples
that
I
just
made
but
I
think
we're
getting
pretty
good
adoption
so
far,
and
it's
generally
a
good
sign.
But
the
point
here
is
that
adding
value
accelerates
adoption.
A
So
I
just
want
to
reiterate
again
that
these
lessons,
although
they
come
from
my
experience
with
CSI,
they
can
actually
be
taken
and
applied
to
most
arbitrary
extensions
or
specifications
right,
they're,
actually
quite
general
and
I'm
sure
these
patterns
will
pop
up
more
in
the
future.
So
the
three
points
I
really
want
to
drive
home
or
that
having
an
open
development
process
really
gets
everyone
on
the
same
page,
you
in
the
process
of
actually
making
changes
to
your
specification.
If
it's
open
and
transparent,
you
can
actually
build
a
group
of
Advocates
for
that
change.
A
That
can
then
go
and
like
spread
the
gospel
about
it
afterwards.
If
they
were
involved
in
the
process,
I
also
want
to
talk
remind
you
that
the
RPC
definition
often
isn't
enough
right.
We,
we
might
need
tools
that
exist
external
to
the
RPC
definition,
to
enforce
consistency
and
to
make
sure
that
everyone's
on
the
same
page
when
they're
developing
their
drivers
also,
we
want
to
give
a
compelling
reason
for
people
to
move
to
this
new
paradigm.
We
don't
want
to
force
people
over,
but
when
there's
enough
value,
add
they're
going
to
move
themselves.
A
A
That's
support
it
as
well
as
a
whole
bunch
of
processes
that
help
it
make
that
help
make
it
successful,
and
so
the
title
of
this
talk
is
actually
a
little
bit
misleading
I
think
it
should
have
actually
been
this,
but
it
rolls
off
the
tongue
a
little
bit
less
again.
My
name
is
David.
You
can
contact
me
by
email
or
follow
me
on
github
and
I'm
here
to
answer
any
questions
you
have.
Thank
you.
A
Yes,
so
what
we
have
with
the
new
volume
plugins
is
that
the
method
of
deployment
is
that
they're,
actually
all
containerized
inside
of
pods,
and
so
in
order
to
achieve
that,
we
deploy
the
pods
with
the
sidecar
inside
and
the
sidecar
does
the
like
viewing
of
the
kubernetes
objects
and
that
sidecar
object
makes
the
calls
inside
the
same
pod
to
the
driver
itself
right.
So
the
RPC
layer
actually
exists
for
some
calls
between
the
sidecar
and
the
driver.