►
From YouTube: Building a Software-Defined-Network Using K8s API Machinery and Controllers - Matteo Olivi
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Building a Software-Defined-Network Using K8s API Machinery and Controllers - Matteo Olivi, Independent
Kubernetes controller pattern and API machinery are being used to build systems other than Kubernetes itself. As a result, they are put to the test with new, original challenges, and strengths and weaknesses emerge. This talk describes the lessons learned building an SDN with Kubernetes controller pattern and API machinery, as well as some interesting and general problems that arose: locking and allocation of IPs without multi-object transactions, enforcement of invariants between API objects of the same kind, managing a dynamic set of informers on API objects of the same kind to eagerly and efficiently disseminate guest - host mappings. The audience will also learn how to apply the controller pattern to their own problems and how doing so automatically handles for them the (hard) details of building a reliable control plane, freeing them to focus on the specifics of their problems.
https://sched.co/ZenR
A
Hello,
everyone
and
welcome
to
building
a
software-defined
network
using
kubernetes,
api
machinery
and
controllers.
My
name
is
matteo
libby
and
I'm
a
recent
computer
engineering
master's
graduate
from
the
university
of
bologna.
This
talk
is
about
a
project
I
worked
on
while
I
was
an
intern
at
ibm
jointly
with
my
background
supervisor
mike
mike
spritzer.
The
project
is
on
github
and
we
hope
to
contribute
soon
to
the
kubernetes
examples
repo.
A
So,
let's
set
the
agenda
before
we
move
forward,
we're
going
to
begin
with
a
short
background
on
dxlan
and
the
kubernetes
controller
pattern,
and
then
we're
going
to
dive
into
the
api,
of
course,
into
its
architecture
and
finally,
we're
going
to
discuss
some
technical
problems
that
we
had
to
solve
and
that
we
believe
have
a
relevance
beyond
just
our
proof
of
concept.
Sdm,
so
vxlan.
A
The
x9
is
a
protocol
for
overlaying
virtualized,
ethernet
networks
on
top
of
an
underlay
of
ip
networks,
and
it
provides
isolation
between
different
virtual
networks,
as
well
as
the
illusion
to
resources
within
the
same
vehicle
network
of
being
located
within
the
same
physical
network.
Even
though
that's
not
true
and
it
achieves
this
via
vxlan
encapsulation.
A
So
what
happens
if
two
communicating
parties
exchange
data
and
are
within
the
same
virtual
network
is
that
the
packets
or
ethernet
frames
that
they
exchange
will
make
use
of
their
virtual
addresses,
but
those
have
no
meaning
on
the
underlying
network.
And
so
what
happens
that
transparently,
their
traffic
is
intercepted
and
encapsulated
into
vxlan
packets.
That
will
be
routed
over
the
underlay
network
and
that
will
make
use
of
the
ip
addresses
in
use
on
the
underlay
network
rather
than
a
virtual
ones,
and
the
problem
that
cost
solves
is
that
of
dynamically
configuring
vxlan
overlay
networks.
A
So
next
introductory
topic,
the
kubernetes
controller
pattern
in
kubernetes.
We
have
api
objects
and
api
objects
have
two
sections:
the
spec
that
describes
the
desired
state
of
the
api
object
and
the
status
which
is
a
report
on
the
observed
state
and
the
kubernetes
control
plane,
consists
of
a
series
of
control
loops
that
or
controllers
as
they're
called
that
constantly
listen
for
notifications
on
api
objects
and
react
to
them
by
asynchronous
by
asynchronously
processing.
A
The
api
objects
and
processing
typically
begins
by
reading
the
desired
state
of
the
api
object,
modifying
the
real
state
of
the
real
world
to
drive
it
towards
the
desired
state
and
then
by
writing
back
the
new
state
of
the
real
world
into
the
status
of
the
api
object,
and
these
controllers
are
not
built
from
scratch.
But
by
using
standard
building
blocks,
collectively
known
as
api
machinery
and
by
following
a
more
or
less
standard
design
pattern
known
as
the
kubernetes
controller
pattern
and
a
really
important
building
block
is
the
informer.
A
An
informer
is
an
abstraction
to
receive
notifications
on
all
our
subset
of
api
objects
of
a
certain
type
from
the
api
servers,
and
it
also
maintains
a
local
cache
with
the
most
recent
version
of
each
api
object,
for
which
a
notification
was
received
and
reception
of
one.
Such
notification
triggers
execution
of,
what's
called
a
notification
handler,
which
is
a
small
function
that
typically
puts
a
reference
to
the
api
object.
A
The
notification
is
about
in
a
work
view,
and
these
references
are
usually
the
namespace
names
of
the
api
objects
and
references
are
extracted
from
the
work
queue
by
a
pool
of
worker
threads
that
use
them
to
retrieve
the
whole
api
object
associated
to
the
reference
from
the
cache
of
the
informer
and
then
do
the
actual
processing
on
that
api
object.
Two
things
are
worth
mentioning
here.
The
first
one
is
that
informers
are
eventually
consistent.
A
So
if
an
api
object
goes
through
a
sequence
of
states,
there's
no
guarantee
that
there
will
be
a
notification
for
each
intermediate
states.
Rather
those
might
be
skipped,
and
so
the
resulting
behavior
is
level
based
and
second,
the
work
queue
provides
implicit
locking
on
the
references
in
it.
So
I
don't
have
time
for
the
details,
but
what
this
concretely
means
is
that
we
have
the
guarantee
that
at
any
given
time,
there
will
be
at
most
one
worker
processing
a
given
object
reference.
A
Okay,
we've
concluded
the
background.
Now
we
can
look
at
cos
itself
and,
let's
begin
by
its
api
users,
interact
with
cost
just
like
they
do
with
kubernetes,
by
creating
updating
and
deleting
api
objects
and
cos
defines
three
custom.
Api
object,
types
served
by
custom,
api
servers
and
these
custom
types
are
the
subnet.
The
network
attachment
that
you'll
see
abbreviated
to
n
a
for
the
rest
of
the
slides
and
the
ip
lock
we'll
talk
about
ip
logs
later
in
the
talk.
A
Now,
let's
focus
on
the
first
two,
and
in
this
slide
you
see
examples
of
them
and
on
the
left,
you
see
a
subnet
api
object
and
a
subnet
describes
an
ipv4
subnet
within
a
vxlan
virtual
network,
and
so
its
spec
has
a
field
called
vmi,
which
contains
a
unique
numeric
identifier
of
the
vxlan
virtual
network.
The
subnet
is
part
of,
and
it's
packed
and
as
subnet
spec
also
is
a
field
which
describes
the
pool
of
ip
addresses
allocatable
to
resources
within
that
subnet
subnet
also
have
a
status
field
more
on
that
later.
A
On
the
right
you
see,
a
network
attachment
api
object
and
a
network
attachment
describes
a
network
interface
within
a
vxlan
virtual
network,
and
its
pack
has
a
subnet
field
with
the
name
of
the
subnet
api
object.
The
network
attachment
is
part
of
and
the
cost
control
plane
will
choose
an
ip
address
from
that
subnet
and
assign
it
to
the
network
attachment
by
writing
it
into
its
status,
guest
id
field
and
the
cost
control
plane.
A
So
we've
seen
its
api.
Now
we
can
look
at
its
architecture.
Oh
I'm
sorry,
I'm
talking
about
cos
and
it's
very
similar
to
kubernetes
architecture.
There's
a
group
of
api
of
custom
api
servers
that
serve
the
custom,
api
objects
and
those
custom.
Api
objects
are
stored
in
a
dedicated
lcd
cluster
and
are
implemented
by
a
group
of
custom
controllers
that
are
built
not
from
scratch.
But
by
using
the
standard,
kubernetes
api
machinery
building
blocks
that
I've
mentioned
before,
such
as
informer
and
work
you
and
worker
threads
and
more
precisely.
A
There
are
three
custom
controllers
and
you
see
two
of
them
on
the
top
right
of
this
slide,
which
are
the
subnet
validator
and
the
item
controller.
Those
are
single
tones,
but
more
on
them
later
and
then
on
every
underlay,
node
or
worker
node,
as
we
call
them.
There's
a
cos
agent
called
connection
agent,
which
is
responsible
of
implementing
the
network
attachments
that
are
bound
to
its
node
in
the
way
I
described
in
the
previous
slide,
and
it
does
so
by
interacting
with
a
local
openviswitch
daemon.
A
Finally,
cos
currently
gets
deployed
as
a
kubernetes
workload,
so
all
of
its
components,
including
api
servers
and
ncd,
run
within
pods
managed
by
a
kubernetes
cluster
via
deployment
or
a
demon
set.
However,
conceptually
nothing
forbids
you
from
deploying
costs.
As
a
standalone
system,
so
okay,
we've
concluded
a
high
level
overview
of
costs.
A
Creation
of
that
subnet
should
fail,
and
the
only
way
you
can
get
this
behavior
is
by
validating
subnets
before
they're
actually
created
or
by
created.
I
mean
persistent
question,
and
so
the
validation
logic
should
reside
either
in
the
api
servers
or
in
a
validating
webhook,
and
it
would
look
something
like
what
you
see
in
this
slide.
A
So
when
a
subnet
s
that
does
not
exist
yet
is
validated,
you
would
start
by
retrieving
all
the
existing
subnets
in
the
same
vxlan
virtual
network
as
s,
and
then
you
would
compare
them
one
by
one
to
s
and
if
at
least
one
conflict
is
found,
creation
of
s
should
fail.
Otherwise
it
succeeds,
and
what
I
mean
by
conflict
is
that
s
and
other
subnet
mutual
existence
lead
to
ovulation
of
an
invariant
such
as
because,
for
instance,
because
their
pool
of
ip
addresses
overlap.
A
So
this
is
good,
but
is
not
enough,
because
when
s
is
validated,
it
does
not
exist
yet,
but
is
only
compared
to
already
existing
subnets.
So
what
might
go
wrong
is
that
two
conflicting
subnets
are
valid,
created
and
validated
more
or
less
at
the
same
time,
so
they're
validated
in
parallel,
and
so
when
either
one
of
those
two
conflicting
subnets
is
validated.
A
A
There's
one
last
subtlety
that
we
have
to
mention,
though,
about
the
subnet
validator
and
the
subtlety
sub
tool
is
that
when
the
validator
validates
subnet
s
and
it
retrieves
all
the
other
existing
subnets
with
in
the
same
vxn
virtual
network
as
s,
it
uses
a
direct
list
against
the
api
servers.
And
this
might
sound
odd
because
remember
that
the
validator
has
an
informer
on
subnets
with
a
local
cache,
and
so
it
would
be
much
more
efficient
to
retrieve
all
the
other
existing
subnets.
A
From
that
informal
cache,
however,
there
are
edge
cases
where
correctness
would
be
violated,
and
these
slides
shows
an
example
so
assume
that
the
validator
retrieves
subnets,
not
from
the
api
servers
with
a
direct
list,
but
from
the
cache
of
its
informer
and
assume
that
two
conflicting
subnets
are
created
more
or
less
at
the
same
time,
and
also
assume
that
two
validators
are
running.
At
the
same
time.
Now
before
I
said
that
the
validator
is
a
singleton,
and
so
it's
managed
by
a
deployment
with
a
replica
field
of
one,
but
that
is
no
hard
guarantee.
A
There
could
be
short
transients
because,
for
instance,
because
of
a
network
partition
when
more
than
one
validator
is
running
at
the
same
time,
and
if
this
happens,
the
gist
of
our
problem
is
that
there's
no
guarantee
on
the
order
with
which
api
objects,
including
subnets,
are
added
to
informer
caches.
So
what
might
go
wrong
in
this
scenario?
A
Okay,
we've
concluded
discussion
of
the
first
of
our
technical
problems.
The
second
problem
is
that
of
that
we're
going
to
discuss
is
that
of
assigning
virtuality
addresses
to
network
attachments
while
avoiding
ip
collisions
and
once
again,
there's
a
dedicated
singleton
custom
controller.
For
this,
the
ipam
controller
and
the
iphone
controller
is
in
charge
of
assigning
addresses
to
network
attachments,
and
it
keeps
an
informer
on
subnets
and
it
processes
each
subnet
by
initializing.
A
Only
if
acquisition
of
the
lock
is
successful.
An
interesting
part
is
in
how
the
lock
is
implemented.
A
lock
is
implemented
as
an
ip
lock
custom
api
object.
An
ipl
object
represents
a
lock
on
a
virtual
ip
address
within
a
vxlan
virtual
network,
and
the
interesting
part
is
in
its
name
so
for
an
ip
lock
on
a
virtual
ip
address
x
within
a
virtual
network
that
has
a
vni.
A
Okay,
we've
also
done
we're
also
done
discussing
the
second
technical
problem.
Now
we
can
look
at
the
third
and
final
one,
and
this
is
the
problem
of
dynamic
filtering
in
informers,
but
first
we
have
to
understand
why
we
need
this.
So
to
recap
on
every
worker,
node
or
underlay
node,
there's
a
cos
agent
called
connection
agent
that
is
responsible
for
implementing
the
local
network
attachments,
that
is
the
network
attachments
that
are
bound
to
its
underlay
node
and
it
implements
them
by
creating
for
them
a
linux
network
interface
and
configuring,
a
local
openviswich
virtual
switch.
A
However,
if
we
want
those
local
network
attachments
to
be
able
to
communicate
with
remote
network
attachments
that
are
in
their
own
virtual
network,
that
alone
is
not
enough,
and
so
we
say
that
if
on
a
node,
there
is
at
least
one
local
network
attachment
with
a
vni
x,
vnix
is
relevant
to
that
node,
and
what
this
practically
means
is
that
a
connection
agent
on
a
node
needs
to
be
notified.
Besides
of
local
network
attachment.
Oh
I'm,
sorry,
let
me
take
a
step
back.
A
A
A
Where
that
remote
network
attachment
b
is
implemented,
but
the
virtual
switch
alone
doesn't
do
anything.
So
it
can
only
know
this
mapping
if
the
connection
agent
on
its
node
on
ace
node
has
made
it
aware
of
its
mapping
by
appropriately
configuring
it,
and
so
the
question
becomes.
How
does
the
connection
agent
on
ace
node
know
the
mapping
between
these
vni
and
virtual
addresses
to
the
ip
address
of
the
underlay
node?
Where
b
is
implemented?
A
And
the
answer
is
that,
well,
this
mapping
is
natively
stored
only
in
the
api
object
of
the
remote
network,
attachment
b,
and
so
the
bottom
line
is
that
the
connection
agent
on
ace
node
to
properly
configure
the
virtual
switch
on
its
node
must
also
be
notified
of
the
remote
network
attachment
b,
even
though
that
network
attachment
is
remote
remote,
like
I
said
so.
To
recap:
we've
introduced
the
notion
of
relevant
dni
with
respect
to
node
and
we've
established
that
to
properly
configure
the
virtual
switch
on
its
node.
A
It
changes
over
time
so,
for
instance,
if
at
one
time
there
are
no
network
attachments
that
have
a
vni
x
on
a
node,
the
nix
is
not
relevant
on
that
node,
but
as
soon
as
the
first
network,
attachment
with
vnix
on
that
node
is
created,
vni
x
has
become
relevant
from
that
node
and
when
all
the
network
attachments
that
have
vni
x
on
the
node
are
deleted.
Vni
x
ceases
to
be
relevant
from
that
node.
A
So
more
concretely,
our
problem
is
ensuring
that
a
connection
agent
on
a
node
is
notified
only
of
the
remote
network
attachments
that
have
a
relevant
vmis
and
none
of
the
other
remote
network
attachments,
because
it
doesn't
care
about
those-
and
this
is
important
because
we
could
also
say,
oh
we're
not
going
to
do
that.
We're
just
going
to
keep
a
unique
informer
on
all
remote
network
attachments,
and
it
will
be
the
connection
agent
itself
that
manually
filters,
the
remote
network
attachments
on
the
basis
of
whether
the
vmi
is
relevant
or
not.
A
But
this
would
never
scale
because
it
would
mean
that
the
api
servers
would
have
to
notify
all
of
the
worker
nodes
of
all
of
the
network
attachments,
even
though
most
of
those
notifications
wouldn't
be
would
never
be
needed.
And
so
first
I'm
sorry.
Our
first
question
was
since
we're
using
formers
to
receive
notifications
and
informers
support
filtering
of
api
objects
on
which
they
receive
notifications
on
the
basis
of
the
fields
of
those
api
objects.
A
So
if,
on
a
node,
there
are
p
relevant
dni's
the
connection
agent
on
that
node
will
have,
besides
of
an
informer
on
local
network
attachments,
one
informer
for
each
relevant
bni
that
specifically
filters
in
on
that
specific
vni
and
as
soon
as
a
new
vmi
becomes
relevant.
The
connection
agent
starts
an
informer
dedicated
to
that
vni
and,
as
soon
as
a
vni
ceases
to
be
relevant,
the
connection
agent
stops
the
informer
for
that
vni.
A
But
with
our
design
there
are
potentially
a
lot
of
informers,
one
for
each
relevant,
vni
and
so
worker
doesn't
know
from
which
informer
to
retrieve
the
network
attachment
that
it
must
process.
And
so
the
solution
that
we
found
was
to
simply
have
each
connection.
Agent
maintain
an
in-memory
map
that
associates
each
namespace
name
to
the
informers,
where
the
network
attachment
that
has
that
namespace
name
is
stored
and
this
way
a
worker
that
pops
a
namespace
name
from
the
work
queue
begins
by
consulting
this
map.
A
Now
this
works,
but
there's
one
subtlety
that
we
have
to
be
careful
with
and
the
subtlety
arises
because
there
could
be
transients
during
which
the
same
network
attachment
is
stored
at
the
same
time,
into
more
than
one
informer
cache.
This
problem
arises
because,
like
any
other
api
object,
network
attachments
can
be
updated,
but
as
a
result
of
these
updates,
their
vmi,
their
virtual
network,
can
change.
A
So
for
this
network
attachment,
three
notifications
will
be
received
at
the
node.
In
our
example,
first,
there
will
be
a
creation
notification
at
the
informer
for
vnix,
because
that
was
the
first
vni
of
the
network
attachment
and
then,
when
the
vni
of
the
network
attachment
changes
from
x
to
y,
there
will
be
two
notifications.
A
But
the
problem
is
that
there's
no.
The
problem
is
that
these
three
notifications
are
delivered
to
two
different
informers
and
there's
no
guarantee
on
the
order
with
which
notifications
are
delivered
to
two
different,
two
different
informers.
And
so
what
might
go
wrong
is
that
out
of
the
three
notifications,
the
last
one
to
be
received
is
not
the
creation
notification
for
the
new
vni
of
the
network
attachment,
but
is
the
deletion
notification
for
the
old
dni
of
the
network
attachment
x?
A
And
so,
if
this
happens,
the
first,
the
two
first
notifications
that
are
received
are
the
two
creation
notifications
at
the
informer
for
vnix.
Another
informer
for
dni.
And
if
this
happens,
it
means
that
before
the
last
deletion
notification
is
received,
the
network
attachment
in
our
example
is
stored
at
the
same
time,
into
two
different
informers,
the
one
for
dnax
and
the
one
for
vni,
and
so,
if
a
worker
pops,
the
namespace
name
of
the
network,
attachment
in
our
example
during
this
transient.
When
that
worker
reads
the
map
from
namespace
name
to
informers.
A
It
finds
out
that
the
map
is
ambiguous,
that
it
points
to
two
different
informers
at
the
same
time,
and
so
the
worker
doesn't
know.
What's
the
informer
that
contains
the
network
attachment
which
it
should
process,
and
we
found
that
for
a
worker
that
is
in
this
situation,
it's
enough
to
simply
drop
the
namespace
name,
that
he
just
popped
and
give
up
on
processing
that
network
attachment,
and
so
a
worker
in
this
situation,
simply
pops
a
new
namespace
name
and
moves
on
to
processing.
A
The
information
that
the
network
attachment
in
our
example
is
stored
at
the
informer
for
vnix,
because
it's
no
longer
there
and
it
will
re-add
to
the
queue
the
namespace
name
of
that
network
attachment,
and
so
a
new
worker
will
pop
that
name,
his
name
from
the
queue
and
when
that
new
worker
consults
the
map
from
namespace
name
to
informers,
all
the
ambiguity
will
will
have
disappeared.
The
map
will
universally
point
it
to
the
informer
for
vni,
and
so
the
worker
will
be
able
to
complete
to
complete
processing
of
the
network
attachment
okay.
A
So
this
concludes
discussion
of
the
last
technical
problem
that
we
had
to
solve
and
also
concludes
my
talk.
So,
in
summary,
we
prototyped
an
sdn
using
kubernetes
api
machinery
and
controller
pattern
to
test
and
demonstrate
that
those
are
good
and
general
enough
building
blocks,
not
just
for
kubernetes
but
for
other
cloud
control
planes
as
well
and
in
the
process.
Three
interesting
challenges
emerged
and
we
had
to
be
creative
to
find
solutions
to
them,
but
we
believe
that
both
those
challenges
themselves
and
our
solution
have
a
relevance
beyond
just
cause,
and
so
this
is
it.
A
A
A
A
And
before
that
question
adriano
petsuto
asked
if
we
have
actually
implemented
the
last
controller
we
talked
about,
which
is
the
connection
agent,
the
one
with
multiple
informers
and
the
answer
is
yes,
we
implemented
it
and
it's
on
github,
open
source
I'll
make
sure
to
post
a
link
to
it.
On
slack
after
the
q,
a.
A
So
the
the
resources
within
a
given
virtual
network
can
communicate,
can
communicate
with
other
resources
within
the
same
dxn
virtual
network,
but
not
with
those
on
other
vx
and
virtual
networks,
and
you
can
have.
This
is
what
vxlan
does
and
you
can
have
multiple
use
cases
for
this.
Of
course,
one
really
big
use
cases
is
that
of
implementing
multi-tenancy
in
a
cloud
data
center.
So
the
idea,
let
me
adjust
the
angle
of
the
camera.
A
So
that
could
be
a
use
case.
This
way,
each
separate
canon
would
be
isolated
from
the
others
and
another
characteristic
of
vxlan,
which
makes
it
suitable
for
our
data
centers.
Is
that
it's
very
scalable
in
terms
of
the
number
of
different
virtual
networks
you
can
support.
I
don't
remember.
The
precise
number,
I
believe
is
16
mega
different
virtual
networks,
which
again
obviously,
is
really
desirable.
A
If
you
want
to
implement
a
cloud
data
centers
cloud
data
center,
where
you
potentially
have
a
lot
of
different
tenants,
so
christopher
gods
asks
do
you
have
plans
to
continue
your
work
on
this
project,
eventually
taking
it
to
production?
And
if
this
is
the
case,
if
you
put
any
thought
into
where
the
cluster
that
runs
the
operator
would
live?
A
Okay,
so
yes,
we
don't
consider
me
and
the
other
author
don't
consider
this
project
to
be
complete.
As
I
said,
it
was
my.
The
other
order
was
my
supervisor.
While
I
was
an
intern
at
ibm,
and
we
worked.
This
we've
been
also
working
on
this,
since
I
got
back,
although
not
at
a
very
sort
of
satisfying
rhythm,
because
we
also
have
other
things
to
do.
Obviously.