►
From YouTube: Building and Maintaining a Client Library - Stories From the Trenches - Jussi Nummelin, Kontena Inc.
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Building and Maintaining a Client Library - Stories From the Trenches - Jussi Nummelin, Kontena Inc.
As Kubernetes is a pretty complex system, so is the (REST) API of it. On the first look things feel pretty simple, you authenticate with a token or SSL client certificate and POST few resources to the correct API endpoint. How hard can that be? As we’ve been developing our own Kubernetes distribution and some additional tooling for Kubernetes, we’ve also build a new client library for the API in Ruby, k8s-client (https://github.com/kontena/k8s-client). While building and maintaining the client library we've stumbled on few, well, odd things on the API and faced some technical challenges also. In this session we'll go through some the most interesting findings and how we've tackled them. These topics include for example the complexity of merge-patching resources, empty vs. null vs. non-existing values, PUT vs. PATCH and API compatibility.
https://sched.co/MPXW
A
A
I'm
gonna
start
by
by
going
through
a
bit
of
the
deserters
background.
Why
why
we
actually
started
to
build
build
a
new
new
client
library
for
kubernetes,
we'll
go
through
some
authentication
bits
on
there
from
the
client
purely
from
the
client
point
of
view?
How
it's
a
well,
it's
a
bit
hard
or
at
least
a
lot
harder
than
it.
A
A
Unfortunately,
less
not
really
enough
time
to
go
through
every
single
one
of
them,
because
we
trust
me.
We
quite
a
few
of
them
will
go
I'll,
try
to
select
the
sort
of
most
important
ones
if
you're
thinking
about
writing
a
client
robbery
or
if
you
have
to
have
to
work
with
the
sort
of
low-level
API
things
with
with
kubernetes.
Of
course,
if
you're
just
using
tube
GDL
apply,
then
this
well.
This
might
not
really
apply
to
you,
but
but
it
might
help
you
and
to
actually
understand
how
things
work
with
cubes
EDL,
how
they
do.
A
So
yeah,
at
least
in
in
in
our
cases
when
we
started
the
journey,
we
really
really
underestimated
the
airport,
what
it
actually
takes
to
build
a
a
truly
working
client
library
for
kubernetes,
but
on
the
other
hand
it
was
really
a
good
learning
exercise
for
us
too.
So
now
we
really
understand
how
Cube
API
works
and
why
it
works
as
it
as
it
does
and
then
in
the
end,
it
really
is
now
in
sort
of
enable
for
us
and
our
teams
to
to
build
different
things
on
top
of
and
using
kubernetes
api.
A
A
A
And,
of
course,
basically,
everything
I'm
talking
about
this
open
source
and
and
and
the
main
one
being
the
Kas
client
that
we
have
so
it's
under
Apache
2
license
first
cluster,
it's
also
the
open
source
and
it's
the
sort
of
main
user
of
the
of
the
client
library.
On
top
of
the
client
library,
we've
also
created
another
sort
of
deployment
tool
called
Mordor
which
helps
you
manage,
manage
resources
and,
and
this
complex
set
of
resources
and
and
and
those
are
just
mainly
mainly
as
an
example
of
the
users
of
the
client
library
we
built.
A
So
to
really
understand
how
how
and
why
things
work
in
the
APS
they
do.
First,
we
had
to
really
understand
how
the
request
post
processing
goes
into
the
API.
So
every
single
request
on
the
API
service,
of
course,
authenticated
there's
a
lot
of
different
plugins
that
you
can
plug
on
the
configure
on
the
API
server
and
basically
the
first
one
successfully
ordinay
and
the
user
passes.
Then
the
identity
forward
to
the
later
stages
authorization.
A
A
The
last
step
is
actually
validating
the
resources
so
that
the
port
you're
trying
to
create
is
actually
a
valid
port
and
then
only
after
that,
we
store
it
on
there
on
the
HDD
data
store
on
the
API
and
then
when
it's
on
the
HDD
and
then
available
on
the
API.
Then
there's
a
lot
of
other
controllers.
Actually
creating
reports
and
replica
sets
and
whatnot.
A
Authentication,
the
first
step
in
there
in
the
request
policy
mitts
well
from
the
API
server
point
of
view.
It's
pretty
pretty
straightforward
from
the
client
point
of
view.
It's
it's
actually
not
on
the
API
side,
it's
basically
three
ways
which
you
can
authenticate:
client
certificates,
tokens
or
HTTP,
plain
authentication,
of
course,
for
tokens.
There
are
a
couple
of
variations,
but
it's
still
essentially
a
token
of
negation,
so
pretty
pretty
straightforward.
A
The
top
left
corner
is
from
Google
and-
and
you
actually
see
that,
there's
a
configuration
to
call
the
external
tool
to
some
JSON
path
to
the
output
of
the
tool
and
I.
Only
after
that,
you
get
the
token
to
be
passed
on
the
API.
Well,
it's
not
that
difficult
to
support.
We
can
pull
it
off.
Well,
then
came
eks
which
uses
pretty
much
the
same
sort
of
style
and
and
an
idea.
We
call
an
external
tool,
we
get
to
talk
and
we
pass
it
today
API.
A
But
if
you
see
the
configuration
is
totally
different
on
the
cube
convict
so
yeah,
let's,
let's
build
a
support
for
that
to
you.
Well,
on
the
right
hand,
side
those
are
the
most
typical.
Once
you
see,
there's
a
plane
token
already
on
there
on
the
cube
convict,
we
just
pass
it
on
to
the
API
or
there's
a
client
client
certificate.
We
put
it
on
there
on
the
request
and-
and
it
just
works
as
it
should.
A
One
of
the
oddities
we've
seen
also
is
that
in
some
cases
to
token
in
the
cube
convict
file,
it
can't
be
or
can
well,
it
might
be,
or
might
not
be
already
base64-encoded
and
that's
a
pretty
difficult
thing
to
figure
out
whether
whether
I
have
something
that's
already
pays
encoded
or
not.
So
we've
had
to
do
some
workarounds
for
that
too,
and
then,
of
course,
the
configuration
is
somewhat
different,
whether
you're
running
out
of
cluster
and
my
CLI,
or
whether
I'm
running
this
thing
on
a
port
within
the
cluster.
A
So
we
try
to
support
everything
and
then,
of
course,
from
from
purely
if,
if
I
see
myself
as
user
of
this
library,
the
user
shouldn't
really
have
to
deal
all
about
this
complexity.
So
what
we
actually
be
now
able
to
do
is
that
so
we've
tried
to
hide
all
the
nasty
details
from
the
users
of
the
library.
So
just
okay,
let's
fly
and
order
config
and
I
get
a
client
instance
back.
So
that's
pretty
pretty
neat.
A
A
So
of
course,
as
we
developers
always
do,
we
start
with
a
sort
of
simple
naive
approach
and,
of
course,
that
that's
what
we
did
do
so
to
be
able
to
create
a
resource.
You
do
a
post
on
the
API
server
to
the
even
path
of
the
resource
and
and
voila.
You
have
a
resource
on
the
on
the
API.
Then,
when
you
need
to
update
it,
you
get
it.
You.
A
A
The
local
merging
can
be
actually
problematic
when
you
have
deep
structures
and
arrays
and
everything
in
your
llamo,
so
that
might
become
actually
pretty
bit
complex.
So
we
started
to
think
that,
okay,
why?
Why
can't
we
just
basically
load
the
llamo
file
from
the
disk,
a
port
and
why
not
just
put
it
to
put
it
back
to
the
server
sis?
Well,
it
turns
out
that
the
what
you
have
in
your
local
llamo
and
if
you
compare
what
you
have
on
the
server
side,
then
not
even
close
to
being
the
same
resource.
A
So
there's
a
lot
of
defaulting
happening
in
the
in
the
API
where,
when
the,
when
the
objects
are
more
shot
from
Jason
to
golang
and
back
to
Jason,
to
duet
CD,
and
also
remember
that,
on
the
on
the
Rico
stages,
we
have
dis
admission
controllers,
which
can
actually
mutate
your
objects.
So
what
you
have
on
local
disk
and
what
you
have
on
server,
they
might
or
might
not
be
the
same
thing.
A
Of
course,
you
should
have
pretty
much
same
attributes,
but
but
usually
on
the
server
side,
you
have
a
lot
more
default
values
and
and
and
and
those
sort
of
things.
So
essentially,
when
you
load
up,
they
load
up
the
llamo
form
a
disk.
You
try
to
put
it
that
server.
Just
says
that
no
can
do
can't
accept
this
not
valid.
A
The
problem
in
that
is,
is
that
the
strategy
used
to
merge
things
together
is
actually
decided
on
the
server
side.
So
the
result
might
be
different
if
I'm
updating
a
port
versus
if
I'm
updating
a
custom
resource,
so
the
logic
can't
be
different
and,
for
example,
if
I
have
a
port
existing
already
on
server
and
on
the
local
one.
I
rip
replace
a
a
container
in
a
pot
and
do
this
strategic
merge
on
the
server
side.
A
A
What
we
actually
ended
up
doing
is
pretty
much
what
chip
CDL
does
with
when
you
do
a
cube,
GDL
apply.
So
essentially,
it's
a
sort
of
a
three-way
merge
on
the
client
side.
So
whenever
we
create
a
resource
with
with
a
client
now
we
record
the
last
applied
configuration
in
an
annotation.
So
basically
we
record
what
was
what
we
saw
last
time
when
we
loaded
the
thing
from
a
llamó
file,
then
we
get
back
the
resource.
A
We
look
at
the
annotation
that
this
was
the
last
used
configuration
and
then
we
do
basically
a
sort
of
a
pre-emerge,
and
then
we
patch
it
with
that
with
specific
Jason
patch
content
type
on
the
server.
So
essentially
it's
a
pretty
much
saying
what
what
keep
CDL
does
on
own
appliance
as
mentioned
well,
keep
CDL
doesn't
actually
use
Jason
patch,
but
it
does
pretty
pretty
similar
thing
anyway.
A
So
the
chosen
patches
is
basically
as
a
thief
between
what
we
had
and
what
we
have
from
the
local,
what
we
have
on
the
server
and
what
we
had
on
there
on
the
disk
and
that
thief
is
sent
with
that
with
the
application
changing
pads
errors
that
the
or
content
type
the
nice
thing
with
Jason
patch
is
actually
that
it's
defining
in
an
RFC.
So
there's
not
much
room
for
interpretation.
It
should
work
as
it's
specified
cause.
A
Well,
it's
an
RFC
well
turns
out
that
we
would
have
liked
to
be
able
to
do
some
interpretation
in
that.
But
that's
that's
another
discussion.
Maybe
so,
essentially,
a
adjacent
patches
is
a
set
of
operations.
Remove
this
key
from
them
from
the
document
from
the
resource.
Add
this
key
and
with
this
value
to
the
document
replace
this
key
with
this
value
in
the
document,
so
it's
pretty
pretty
pretty
actually
discreet,
describing
what
the
end
result
should
be.
A
Just
to
note
that
all
this
complexity
on
the
on
the
client-side
is
probably
gonna
be
away,
go
away
in
some
timeframe.
So
now
in
wonder,
14
the
server-side
apply
is
on
on
alpha
stage,
so
it's
under
a
feature
gate
that
you
have
to
specifically
go
and
enable
on
your
API
server.
So
it
basically
means
that
all
this
merging
logic
and
all
this
pain
is
actually
handled
on
on
the
on
the
API
server
side,
which
is
super
super
cool
and
a
nice
for.
A
Therefore,
the
client
libraries,
but,
of
course,
as
a
fully
generate
client
library,
we
just
cannot
jump
in
the
into
using
that
sort
of
a
thing
because
well,
you
cannot
go
and
enable
that
on
e
KS
or
GAE
and
of
course
our
library
should
work
with
that.
Well,
older
version
of
copper,
nice
I
think
it
still
works
with
1.7
kubernetes.
Even
if
anybody
is
using
that
old
old
version.
A
One
thing
we
we
sort
of
struggled
a
bit
with
is
also
this
empty
versus
null
versus
undefined
thing,
which,
if
anybody
is
using
javascript,
you
probably
bump
into
similar
things
and
so
I
have
a
I,
have
a
resource,
a
port
where
I
don't
have
labels
I,
don't
have
ports
or
anything
defined
that
that's
fine,
then
I
create
another
port
where
I
define
labels
as
an
empty
struct.
I
define
ports
as
an
empty
array.
A
When
you
post
those
two
ports
into
the
into
API
server,
they
end
up
being
the
same,
which
is
okay.
I
can
live
with
that.
Well,
as
it
turns
out,
it's
a
well,
they
are
the
same.
Well
then,
the
until
they
really
actually
are
not
the
same
anymore.
So
as
we
are
using
this
jason
jason
patch
to
add
and
remove
these
keys,
so
we
can
do
a
json
patch
and
Adam
key
as
an
empty
value
to
the
API.
Well,
the
APS
is
here
done
it's
there.
A
Okay,
then
we
try
to
add
another
value
with
the
same
key
and
it
says
that
no
can
do.
It
is
not
no
such
key
here
like
okay.
What
if,
but
it
turns
out
that
well,
the
API
server
just
basically
filters
out
all
the
empty
valleys,
so
it's
like
a
bit
of
a
limbo
whether
whether
something
is
empty
or
missing
or
nil,
or
it's
just
not
there.
So
empty
is
not
there
in
in
kubernetes.
A
Which
is
actually
pretty
pretty
problematic
because
in
some
cases,
simply
might
actually
mean
something
like,
for
example,
in
in
network
policies.
Empty
actually
does
kind
of
mean
something,
and
then
in
your
custom
resources.
Well,
we
don't
know
what
the
resource
is
empty
actually
might
mean
something.
A
So
with
with
Jason
Pat
you
you.
Basically,
you
have
two
options.
You
can
either
drop
all
the
empty
values
or
empty
keys
from
there
from
that
sort
of
the
last
applied
configuration
annotation,
which
we
record
on
that
on
their
objects
or
then
you
can
deal
with
the
empty
values
when
you're
actually
sort
of
calculating
the
diff
between
these
two
different
versions
of
the
of
the
resource.
A
What
we
ended
up
doing
in
the
indicators
client
is
we
actually
drop
empty
values
from
the
annotation,
where
we
record
the
last
applied
configuration
and
that
that
has
been
working
really
nicely
for
us,
at
least
in
in
in
in
this
version
of
the
client
but
I'm
sort
of
pretty
sure
that
we'll
hit
some
problem
with
that
too.
You
know
in
some
time
frame.
A
All
right,
just
a
quick
sip
of
water
to
keep
my
voice
going
I've
been
on
a
booth
duty.
There
all
whole
conference,
so
I'm
a
bit
lost
of
lost
in
voice.
So
sorry,
okay,
so
how
does
the
client
actually
know
to
which
an
HTTP
endpoint?
It
should
post
or
patch
the
resources
so
on
so
in
in
the
API?
There's
this
discovery
mechanism,
so
there's
a
specific
API
endpoint
where
you
can
call,
and
you
basically
get
get
all
the
API
information.
The
API
understand
these
sort
of
resources,
and
these
are
the
API
endpoints.
A
You
should
be
using
to
access
and
control
those
resources,
and
the
same
mechanism
is
of
course
used
for
C
IDs.
So
when
you
create
a
new
or
sort
of
registering
a
new
custom
resource,
JP
I
will
then
on
the
discovery.
Api
tell
that
okay
there's
this
new
type
of
resource
use
these
endpoints
to
control
those
those
resources
too.
A
So
it's
an
essentially
gives
you
a
map
of
for
these
coins,
use
that
endpoint.
In
a
nutshell,
from
the
client
point
of
view,
you
have
to
think
whether
whether
you
wanted
to
do
the
client
to
really
have
a
static
mapping
of
certain
types
of
resources
to
the
certain
endpoints,
or
do
you
really
want
it
to
be
fully
dynamic?
A
I
mean
with
dynamic
I
mean
that
the
client
actually
understands
and
and
is
able
to
reload
all
this
API
discovery
information
from
the
from
the
API
server,
and
then
one
thing
you
have
to
think
about
also
is
caching
whether
you
want
to
cache
that
information,
somehow
cube
GDL,
for
example,
caches
a
lot
of
that
information
on
disk
in
in
your
home
directory
dot,
cube
dot,
something
and,
of
course,
in
caching.
Well,
there's
two
hard
problems
in
computer
with
computers
and
I
think
one
of
them
was
cache
invalidation
if
I
remember
correct.
A
So
what
we
decided
to
do
is
no
caching,
because
well
that
just
created
opens
up
a
Pandora's
box,
always
so,
whenever
you
boot
up
put
up
to
the
caters
client,
it
basically
does
the
API
discovery
on
the
fly,
and
it's
just
able
to
work
with
that
work
with
any
any
custom
resource
on
the
fly.
It
also,
of
course,
does
mean
that
there
is
a
sort
of
a
performance
hit
on
that,
because
it
has
to
do
their
discovery.
Api
calls,
but
in
our
cases
say
it
hasn't
been
too
bad,
at
least
so.
A
We
did
a
a
sort
of
a
experimentation
on
that
then,
and
actually
we
we
ditched
all
that
and
and
we
the
client,
doesn't
understand
anything
about
the
resource
kinds.
It
just
understand
that
it's
a
resource
so
in
in
in
our
case,
and
especially
now,
we'd
see
a
lot
of
C
or
D
is
coming
into
the
picture.
It's
super
easy
for
us
and
with
the
client,
because
well
we
don't
have
to
do
anything
to
work
with
CR
this.
It's
just
another
resource.
A
And
it's
a
it's
it.
One
of
the
sort
of
main
building
blocks
behind
that
is
is
a
ruby
gem
called
recursive
open
struct,
where
you
can
basically
have
a
ruby
hash
and
it
sort
of
a
objectifies
that
so
that
I
can
actually
call
just
plain
object
methods
on
any
given
resource,
whether
it's
a
known
pod
or
some
new
CRD
just
works
the
same
way.
A
So
just
as
an
example
by
to
create
a
CRD
resource,
you
just
create
a
resource.
Put
all
that.
Basically
what
you
see
on
a
on
any
given
llamó
file,
you
define
that
what's
Davey
Iverson
watch
the
client
and
and
all
the
other
needed
details
as
a
ruby,
hash
and
voila.
You
have
a
resource.
You
can
just
now
go
and
create
it
on
the
on
the
API.
You
don't
have
to
go
and
generate
the
stops
or
or
anything
for
the
custom
resources.
You
don't
have
to
figure
out.
A
How
do
I
actually
now
register
discussed
some
resources
and
api's
into
the
client
library
or
anything
like
that?
So
no,
no,
no
yuck
shaving
anymore,
and
then,
when
you,
if
you,
if
you
want
to
just
go
and
get
some
custom
resource,
you
load
up
the
client
with
them.
One
single
order,
config
method
call
define,
what's
the
what's
the
API
of
that
or
the
API
version
of
the
resource
here,
you're
getting
the
type
of
that
kind
of
the
resource
and
the
name
of
the
resource
and
voila.
A
So,
especially
for
us,
this
sort
of
generic
resource
has
been
has
been
working
really
well
and
hopefully,
hopefully
someday.
Somebody
comes
with
something
similar
for
that
for
the
official
go
library
to
I'm
I'm,
not
that
super
guru
and
go
whether
that's
actually
doable
or
not,
but
maybe
somewhere.
Somebody
after
the
session
can
enlighten
me
whether
whether
it's
possible
or
in
it.
A
A
B
You
I'm
all
ready
to
understand
that
the
the
client
that
you
created
is
not
compatible
with
the
control
clan
because
what
I
mean
by
that
is
in
order
to
be
able
to
do
your
patch
much
as
you
want
to
put
some
extra
labels
on
you,
the
things
that
you
are
deploying.
So
if
I
had
deploy
something
with
cube,
city
and
I'll,
try
to
do
a
patch
with
your
new
client
that
will
not
work
well,
it.
A
A
Did
there's
a
special
flag
on
coop
serial?
They
does
save,
save
configuration
or
something
which
actually
does
it
so
yeah
for
in
in,
in
our
main
use
cases
we
we
always
use
the
same
client.
So
for
us
it
it
works
pretty
pretty
nicely,
but
it
should
work
with
cube
CDL.
You
just
have
to
remember
to
use
this.
That's
just
save
thingy
can't
remember.
What's
the
flags
full
name
test
too
many
flags
on
cube
city
also.
C
A
So
so
that
the
simple
reason
why
we
didn't
use
cube
serial-in
behind
is
that
they
are
the
tooling
that
for
the
for
the
distribution,
the
main
thing
that
we're
working
on
is
ruby
tooling.
So
we
didn't
wanna
sort
of
from
Ruby
to
fork
out
to
cube
CDL
and
fork
back
and
and
all
sort
of
things.
So
it's
it's
still
sort
of
entry
and
then
works
within
the
same
process.
So
that's
the
main
reason
what
we
didn't
do
it.
A
A
Our
projects
are,
are
the
main
users
and
but
but
I
I
do
know
few
other
other
users
also
and-
and
we
received
couple
of
sort
of
external
PRS
also.
So
it's
a
it's
great
in
that
sense
and
but
of
course,
we'd
be
more
than
happy
to
see
more
using
using
of
our
tooling
and
more
PRS
and
issues
and
everything
so
so.
F
F
So
if
anybody
looking
for
a
PI's
inclined
and
want
to
help
me
so
I
use
it
everywhere,
but
there
is
this
kind
of
always
the
question:
how
much
like
high-level
functionality
you
want
to
add
and,
for
example,
was
previously
some
high
level
functionality
for
scaling
and
so
on.
So
I
want
to
remove
it
or
partially
already
removed
it,
because
it's
harder
to
maintain.
Do
you
get
any
high
level
stuff
and
we.
A
Really
try
to
avoid
it
and
there
has
been
actually
a
couple
of
issues
where,
where
people
are
asking
that
that
features
like
okay
I
have
this
deployment,
maybe
I
would
have
a
flag
where,
where
there's
some
force
flag
to
force,
the
deployment
actually
happen
every
single
time,
whether
I
change
anything
or
not.
But
no.
E
A
Not
really
because
we
sort
of
points
to
that,
keep
CDL,
does
it
and
try
to
be
compliant
with
that,
and
and
and
also
the
fact
that
we
had
a
discussion
with
can't
remember
who
it
was,
but
about
the
servers
write
a
polite
thing
coming
anyway
up
so
you
know
some
time
frame
will
be
able
to
ditch
all
the
clients
that
logic,
hopefully
anyway.
So
we
didn't
want
to
go
that
route
because
that
that's
then,
when
other
set
of
resources
and
everything
and
it
sort
of
complex
applies
a
bit.