►
From YouTube: Connecting and Testing Virtual Network Topologies on Kubernetes - Rich Renner & Gage Orsburn
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Connecting and Testing Virtual Network Topologies on Kubernetes - Rich Renner & Gage Orsburn, One Source Integrations
Today's complex, agile networking setups demand the same CI/CD and automated testing support as user-facing applications have. But you can't simulate an entire networking topology with Kubernetes. Or can you? The declarative approach of Kubernetes makes it simple to recreate and orchestrate distributed applications for continuous integration. In this session we are going to talk about how we used L2 device plugins to wire up the different parts of our applications. And how KubeVirt allowed us to even test pre-existing virtual network appliances in the very same setup. This setup enables entire enterprise topologies to be modeled and validated before released into production.
To Learn More: https://sched.co/GrR9
A
Our
business
is
network
testing,
Network
automation.
Enterprises
will
hire
us
to
do
things
like
data
center
refreshes
or
advising
on
next
phases
of
technology
for
their
on-prem
infrastructure.
One
of
the
challenges
that
we've
had
in
that
business
model
is
being
able
to
provide
certainty
around
automation,
strategies
or
tooling
that
we
recommend.
A
So
today's
agenda
we'll
go
through
the
problem,
hopefully
give
those
that
haven't
been
a
part
of
a
enterprise
network
change
window,
an
understanding
of
what
goes
into
that,
at
least
at
a
high
level
talk
about
what
we
wanted
to
achieve
with
this
project,
as
well
as
some
of
the
key
community
projects
here
at
the
cloud
native
conference.
That
enabled
us
to
to
do
this.
A
A
A
Because
of
this,
the
recommendation
from
the
community
was
to
say
tell
me
more
about
what
you're
trying
to
solve
the
end
of
the
day
and
then
I
can
understand
where
you're
trying
to
go
so,
if
you're
an
enterprise-
and
you
have
a
let's
say-
data
center
network
and
when
network
and
a
few
branches
or
several
branches
in
certain
cases,
the
only
metric
that
your
users
measure
you
buy
is
how
available
you
are
they
don't
ask
you
what
version
you're
running
on
your
routing
platform?
They
don't
ask
you
how
how
good
is
your
throughput?
It's
just.
A
In
this
case,
in
our
Acme
Incorporated
here
we've
been
ax
asked
to
update
an
access
control
list
so
pretty
simple
request:
I
have
SFTP
clients
and
HTTP
clients
on
the
right
and
the
Internet
and
the
SFTP
server
that's
being
targeted
on
the
left,
so
we
need
to
make
a
firewall
rule
which
will
allow
that
and
secure
the
firewall
from
the
perspective
of
not
allowing
other
types
of
traffic
through.
So
let's
go
ahead
and
get
started,
it
seems
to
make
sense
to
put
the
access
rule
in
the
outbound
direction.
A
Make
it
secure
so
I'm
gonna
have
to
deny
something
looking
across
Internet
there's
a
lot
of
default
rules
that
say
well
just
an
isle
IP
that
seemed
good
and
I
want
to
make
sure
that
I
can
turn
in
a
good
change
report.
At
the
end
of
this
change
request,
so
I
test
can
443
get
out
perfect
how
about
telnet,
something
that
I
wasn't
asked
to
enable
okay
can't
get
out
perfect.
So
close
it
down
we're
good!
A
A
So
that
was
a
contrived
example.
Obviously,
an
experienced
firewall
engineer
wouldn't
make
that
mistake,
but
the
idea
is
to
give
you
an
idea
of
some
of
the
challenges
and
things
that
are
creeping
up,
that
you
might
not
necessarily
think
about
when
given
a
problem
to
solve.
Thank
you
solved
it,
but
in
reality,
a
lot
of
other
things
at
play.
The
second
part
of
the
problem
is
that
this
isn't
a
new
goal.
You
know
vendors
have
been
producing
network
appliances,
platforms
operating
systems
for
years
and
their
customers
have
said.
A
The
biggest
problem
that
our
company
is
run
into
is
that
most
of
those
can't
scale
for
various
reasons.
You
know
and
know
no
harm
no
foul,
but
that
was
a
sticking
point.
You
know
and
pretty
much
everything
we
tried
right
around
the
time.
Gage
joined
the
team,
I
started
hearing
a
lot
about
kubernetes
and
when
I
told
him
well,
I
need
to
make
300
devices
to
model
this
customers
network
boy
he's
like
will
here,
look
at
this
planet
scale,
so
I
read
it
and
it's
like.
A
A
The
most
important
thing
that
we
want
to
do
is
provide
that
assurance.
Like
we're
tarmo
earlier,
a
company
is
only
as
good
as
the
availability
or
a
network
team,
as
only
as
good
as
its
availability,
so
that's
0.1
and
then
various
online
posts
and
other
surveys
and
things
of
the
like
say
that
the
network
process
is
archaic,
a
lot
of
other
words
being
thrown
around.
A
A
A
In
addition
of
that,
while
customers
are
asking
for
it
and
the
answers
is
being
replied
or
the
vendors
are
providing
solutions,
rather,
the
ecosystem
could
use
some
growth.
I
mean
there's
just
a
lot
of
tools
that
will
help
you
produce
changes
in
your
production
environment,
but
there
doesn't
seem
to
be
as
much
focus
around
the
validation
and
the
the
pipeline.
A
That
gets
you
to
that
production
page
and
the
first
thing
that
we
noticed
was
that
we
have
a
network
topology
that
looks
a
very
particular
way
and
taking
advantage
of
kubernetes
is
going
to
swing
that
topology.
The
routers
are
going
to
be
all
over
separate
notes,
so
gonna
solve
that
problem.
Somehow.
B
So
I'm
gonna
hit
on
that
first
point
in
that
previous
slide,
that
Ridge
hit
about
network
appliances
being
hard
to
containerize.
Our
solution
that
we
came
across
for
this
is
a
Red
Hat
upstream
project
product
project
called
Q
Bert
get
repose
on
the
slides
along
with
a
demo.
It's
a
I'll
just
give
like
a
beat
brief
kind
of
description
of
it.
B
Cubert
kind
of
allows
us
to
do
some
cool
things
with
converged
infrastructure
so
that
we
can
put
kubernetes
or
OpenShift
on
bare
metal
and
still
run
our
vm
workloads
as
we
need
there.
There's
some
cool
talk
and
cool
stuff
going
on
in
the
community,
for
instance,
we
there
there
was
an
issue
and
there's
been
some
examples
out
there
about
putting
kubernetes
on
top
of
cube
roots,
so
you're
kind
of
running,
like
kubernetes,
on
top
of
kubernetes,
real,
real,
interesting
stuff.
B
B
Their
they're
running
products
like
legacy
applications
that
they're
just
not
ready
or
able
to
containerize
just
yet.
This
is
especially
true
for
network
appliances.
A
few
vendors
have
containerized
their
appliances.
Many
are
still
VMS
it's
great
when
the
vendors
already
have
containerized
their
appliances,
because
it
we're
running
kubernetes
kubernetes
is
great
at
orchestrating
containers.
So
we
don't
like
there
there's
no
action
for
us
to
do
there.
B
B
B
But
there
are
some
instances
where
we
want
to
be
able
to
power
off
or
our
virtual
router
and
then
power
it
back
on
and
it
kind
of
have
that
same
state
or
you
know,
have
a
router,
that's
kind
of
fixed
in
it
in
an
in
a
state
without
us
having
to
provision
it.
So
example
of
that
is
bridge
earlier
mentioned.
B
Another
cool
thing
that
we've
done
is
we
we
use.
Qbert
Qbert
has
a
like
bridge
like
little
bridge
mechanism
that
it
uses
to
pass
the
the
IP
address
from
the
pod
directly
to
the
VM,
and
we
kind
of
hooked
onto
that
and
add
it
in
like
dhcp
options.
So
we
can
leverage
some
some
stuff
like
automatically
bootstrapping
our
routers
with,
say,
SSH
config
or
say
we
want
to
be
testing
like
ztp
stuff,
like
that,
you
can
kind
of
see
an
example
of
this
in
the
slides
and
that's
kind
of
kind
of
all.
B
B
So
in
this
picture,
we've
used
a
cube,
verb
or
kubernetes,
manifest
to
kind
of
deploy
a
service
provider
topology
in
its
own
namespace
after
we
actually
deploy
it.
As
you
can
see,
we
we
have
this
new
object
called
a
VMI,
which
is
a
virtual
machine
instance.
We
got
a
we
got.
We
got
a
virtual
machine
instance
for
every
one
of
our
routers.
They
each
got
their
own
IP.
B
B
We
kind
of
already
touched
on
the
the
TFTP
we're
using
that
to
bootstrap
routers
as
we
deploy
a
topology
OC
serve,
is
open,
connect
server,
we're
typically
using
that
whenever
we
want
to
allow
ingress
into
our
kubernetes
cluster,
and
we
can
leverage
things
like
goo
varieties
and
network
policy
to
kind
of
like
lock
that
down
and
control
what
what
can
access?
What
the
last
thing
there
is.
The
metadata
server
and
it's
if
you've
ever
used,
ec2
I,
believe
there
is
a
like
or
AWS,
which
most
of
you
probably
have.
B
There
is
a
like
169
169,
something
address
that
you
can,
that
you
can
query
and
it'll
give
you
it'll
give
you
metadata
and
information
about
your
about
your
servers,
and
this
is
useful
for
stuff,
like
like
say
whenever
we're
installing
OpenShift
on
a
bit
on
AWS.
The
open,
shipped
installer
actually
uses
this
to
gather
information
about
my
nodes.
What
we
typically
use
it
for
or
well
this
this,
the
service
kind
of
mirrors
that
and
what
we
typically
use
it
for-
is
to
provide
us
kind
of
like
a
dynamic
inventory,
so
that
we're
not.
B
So
that
we're
not
having
to
like
re
like
mess
with
static
files.
Every
time
we
deploy
a
new
topology,
usually
we're
testing
topologies
pretty
rapidly
and
we're
trying
different
combinations
of
different
things,
and
it's
it's
a
lot
to
keep
track
of
that
static
file.
So
we
try
to
do
it.
We
try
to
do
as
much
dynamically
as
possible
and
with
that
I'm
gonna
give
it
back
to
rich
to
talk
a
little
bit
about
how
we
conquer
some
of
the
C&I
and
networking
problems.
To
accomplish
this
sure.
Thanks.
A
Gage,
just
to
recap,
I
guess
from
the
perspective
of
the
network
engineer
at
this
point:
I
now
have
the
ability
to
provision
as
many
routers
as
is
required.
I
can
also
set
up
the
initial
bootstrap
of
those,
so
I've,
already
probably
saved
a
day
in
my
change,
preparation
or
validation
efforts.
So
big
thanks
to
the
Qbert
guys,
if
anybody's
here
definitely
a
key
component
to
what
we're
trying
to
achieve.
A
So
this
is
our
first
time
talking
to
cube
Khan
and
in
watching
all
of
the
other
videos
on
the
internet.
It
seems
like
it
is
unspoken
rule
that
you
have
to
quote
Kelsey
Hightower
or
have
a
tweet
from
him
or
something
so
he
doesn't
talk
about
Network
automation,
change,
control
and
see
ICD
pipelines
in
that
context
very
often,
but
he
does
mention
that
multiple
interfaces
is
hard.
A
He
is
not
wrong
it.
It
was
a
problem
that
we
thought
we
would
solve
in
a
week
and
I
think
it
took
several
months
before
it
was
actually
all
all
settled.
So
why
is
this?
Why
is
this
hard
well
by
default?
If
I
were
to
spend
up
at
kubernetes
cluster
and
spin
up
all
these
routers?
That
would
work
except
they
would
only
be
provisioned
that
single
communities
allocated
address
an
interface.
So
some
of
the
appliances
actually
would
not
even
boot,
as
a
result
of
that,
so
multi
seems
to
be
a
pretty
popular
network.
A
So
the
mechanism
to
get
more
interfaces
associated
in
a
particular
pods
in
network
namespace
is
an
annotation
I
think
that
this
is
just
due
to
the
youth
of
the
project
and
eventually
will
be
part
of
like
the
actual
spec
and
not
be
a
custom
resource.
But
for
the
time
being,
we
have
to
use
this
network
attachment
definition
to
let
the
provisioner
know
or
the
orchestrator
cubelet
in
this
case,
like
what
is
the
interface
connectivity
and
in
some
cases
addressing
that
we
want
to
use.
A
A
This
is
very
simple:
it
just
means
that
I
have
a
network
interface
that
is
already
ready
to
be
leveraged
by
a
pod
and
it's
in
the
root
namespace
at
the
host.
This
is
a
common
theme
that
we'll
see
throughout
these
I
tried
to
simplify
the
example.
I
mean
there's
plenty
of
talks
in
past
Q
cons
around
the
network,
name,
space
and
and
CNI
provisioning
process.
So
just
to
give
you
some
context
here,
the
router
one
is
our.
A
You
know
virtual
router
name,
space
or
a
pod,
just
like
any
other,
and
we
highlight
when
we're
on
the
inside
the
outside
of
the
network
namespace.
So
like
I
said,
the
host
device
means
I.
Gotta
already
have
that
pre
provisioned
and
it's
job
is
just
to
put
it
inside
the
bottle.
That's
not
too
bad.
It
becomes
difficult
to
predict
what
topologies
come
in
next,
especially
as
we
go
to
GA.
A
This
one
tricked
me
I
thought
that
point-to-point
meant
like
very
simple
point-to-point,
pipe,
no
interference,
but
what
it
actually
means
is
it
expects
that
there's
gonna
be
an
IP
address
allocated
to
the
pod
interface,
at
which
point
it
gets
routed
to
the
host
that
wasn't
optimal,
because
we
have
no
idea
what
the
interface
IP
address
is
of
that
router
and
you
know,
hopefully
the
customer
is
leveraging
some
sort
of
configuration
management
tooling,
once
they
have
their
router
bootstrapped.
You
know
to
assign
that.
A
So
what
enable
will
take
advantage
of
that
Mac
VLAN
again
is
kind
of
a
potential
option,
notice
that
we've
added
another
Ethernet
interface
here.
That's
just
a
demonstrate
that
if
we're
gonna
enslave
this
Ethernet
one
in
this
case
to
be
taken
advantage
of
by
the
pods
being
provisioned,
it
basically
means
that
I'm
pushing
my
problem
like
outside
of
kubernetes,
we're
looking
to
take
advantage
of
multiple
clouds
and
cost
optimized
in
that
way.
A
So
we
have
no
idea
what
the
environments
gonna
be,
nor
visibility
and
access
to
make
changes
so
that
one's
out
this
is
the
one
that
again
we
thought
we
were
good
to
go.
We
started
running
down
the
road
with
a
bridge
plugin
and
there's
an
option
with
the
bridge
CNI
plug-in
where
you
can
operate
an
l2
mode
only
so
for
us.
That
seemed
like
a
good
deal,
because
we
could
get
back
to
the
days
of
getting
our
virtual
routers
and
appliances
into
a
common
Linux
bridge
trusted
well
used
across
several
solutions
and
off
we
go.
A
So
even
if
you've
deployed
it-
and
you
know
how
everything's
connected
and
you
trust
us,
a
common
utility
used
by
most
operators
and
administrators
is
well.
What
is
my
neighbor
like?
What
is
the
protocol
telling
me
that
I'm
connected
to
well?
It
turns
out
that
we
started
to
get
a
list
of
these
hacks
on
the
left
side
of
the
screen
there,
the
first
of
which
was
spanning
tree
any
time.
We
were
running
something
that
talks
back
tree.
That
Linux
bridge
was
interacting
with
that.
A
So
we
took
a
cluster
down
one
time
by
not
knowing
that
and
eventually
found
out
that
we
could
disable
it,
which
then
again
causes
the
table
cluster
down
for
different
reasons.
So
that
was
a
bad
bad
start,
I
guess
in
general
we
kept
pushing
forward,
though
we
thought
well,
you
know
we
can
manage
it.
We
can
figure
it
out.
We
eventually
found
this
lldp
forwarding
hack,
which
is
something
that
had
to
be
ran
every
time.
A
So
eventually
we
had
a
problem
that
we
couldn't
solve,
which
I'm
actually
thankful
for,
because
already
we
had
so
many
problems
with
the
solution
that
we
should
it
ditches
it.
But
I
don't
know
what
that
last
one
was,
but
I
put
it
there
just
for
remembrance,
we'll
fill
it
in
one
day
if
we
ever
come
around
to
it.
So
as
I'm
sure
you
notice
on
the
original
slide,
some
foreshadowing,
we
had
to
write
our
own
CNI
plugin
for
this
product,
and
this
is
some
information
around
how
we
did
it.
A
Why
we
did
it
also
if
anybody
is
interested
in
it,
this
github
Lincoln
and
the
community
around
seeing
eyes
is
they're
doing
a
great
job,
I
mean
I.
Think
I
learned
what
I
see
and
I
was
at
the
beginning
of
this
process
and
I
was
able
to
write
one
at
the
end,
not
because
I'm
all
that
good
at
going,
but
they
pretty
much
have
written
it
for
me
and
just
left,
stubs
and
I
just
had
to
fill
it
in
so
a
very
easy
process.
A
A
Step
one
you're
responsible
for
putting
this
interface
in
the
network.
Namespace
we've
already
talked
about
that.
This
is
a
simplified
version
of
the
configuration
but
I
mean
of
the
implementation,
but
again
very
easy.
I
mean
step.
One
was
done
before
lunch
and
we're
thinking
that
we
should
have
done
this
a
long
time
ago.
A
Okay,
step
2
make
any
necessary
changes
of
the
hoses.
The
part
we've
been
waiting
for
basically
I
just
want
to
take
the
other
end
of
that
D
eath
pair
in
the
hostname
space
and
put
it
in
whatever
forwarding
I
guess
container.
In
this
case,
a
linux
bridge
that
I
choose
and
again
fairly
easy
implementation.
They
left
a
nice
stub
area
for
us
to
fill
that
in
and
then
I
finally
got
to
send
this
picture.
A
A
Let's
say
that
I
did
win
the
battle
and
we
updated
the
kubernetes
or
the
cubelet
base
container
and
I'm
allowed
to
use
OVS
vs
CTL
in
it
the
biggest
problem
that
we
hadn't
tackled
and
didn't
know.
Really
how
to
tackle
was
when
a
customer
defines
their
a
topology.
It
looks
like
this
and
they
say:
I
have
router
one
who
was
connected
to
router
two
on
these
interfaces
and
somehow
we
got
to
figure
out
how
to
put
that
on
the
kubernetes
cluster.
A
So,
in
the
background,
I'm
feverishly,
let's
say
trying
to
figure
out
how
to
write
open,
V
switch
database
transactions,
the
other
guys
on
the
team
are
figuring
out
how
to
program
these
nodes.
So
the
first
example
is
when
I
have
a
router
on
a
single
node
and
luckily
its
neighbor
also
gets
provision
on
that
same
note,
so
our
CNI
is
working
perfectly
and
provisioning
the
eath
pairs,
putting
them
into
the
host
namespace,
putting
them
into
the
correct
bridge
and
the
same
process
happens
for
router
two.
A
So
the
first
thing
to
know
is
that
by
default,
the
open
V
switch
is
a
broadcast
chamber.
If
you
don't
set
it
to
mode
secure
or
fail
mode
secure,
it
means
that
allows
all
flows
until
you
tell
it
not
to
allow
the
flow-
and
we
wanted
the
reverse
of
that,
because
we
wanted,
as
close
as
possible,
represent
a
network
topology,
the
way
that
it
is
in
real
life.
A
A
It's
a
little
bit
cryptic
at
first
and
I.
Think
the
only
way
to
really
say
is
that
open,
V
switch,
an
application,
just
like
any
other,
has
an
internal
index
that
it
references
these
interfaces
by,
and
so
that's
how
this
flow
gets
programmed.
It's
saying
that
on
this
bridge,
if
it
comes
in
port
21,
send
it
out
port
11,
so
pretty
simple
and
matches
what
we're
looking
to
do
almost
exactly
so.
A
So
local
node
programming,
some
challenge
not
too
bad.
Going
back
to
the
scale
aspect.
We
don't
strive
to
have
a
lot
of
virtual
routers
on
one
node.
There's
a
lot
lot
easier
ways
to
achieve
that.
So
now
we
gotta
figure
out
how
to
stitch
these
things
together
on
remote
nodes.
So
in
this
case,
I'm
gonna
have
to
use
an
external
interface
either
one,
and
the
biggest
point
that
I
want
to
make
in
this
case
is
that
you
have
several
options:
you'll
have
to
use
VX
LAN.
A
We
chose
that
for
scale
purposes
and
all
this
required
is
just
selecting
the
remote
node
and
saying
hey,
there's
going
to
be
a
key
for
each
flow,
and
that
is
going
to
be
how
you
determine
similar
to
VLANs
anybody's
familiar
with
with
those.
So
you
do
this
VX
land
tunnel,
just
call
it
the
highway
in
both
directions
and
then
just
like
before
you
program
a
flow
down
which
says,
if
I
come
in
this
port
set,
this
VX
Lang
attribute
ID
and
then
send
it
to
this
port.
A
Likewise,
in
the
reverse
direction,
just
like
before
doing
the
other
side.
So
now,
let's
try
to
understand
how
this
frame
is
gonna,
get
from
gig
one
on
router,
two
to
router
one
when
I
first
leave
the
interface
I'm
a
standard,
Ethernet
frame
and
I'm
boarding
based
on
the
ve
pair
that
exists
when
I
hit
the
host
I'd,
be
para
to
an
open
flow
rule.
A
Lookup,
which
says
if
I
came
in
this
port,
send
me
to
this
port,
which
has
instructions
that
say
put
me
in
a
VX,
LAN
header
and
send
me
to
some
IP
address,
which
happens
to
be
our
other
node.
This
is
where
we
depend
on
the
external
environment
like
I
mentioned
earlier.
We
don't
always
have
the
ability
to
you
know
no,
but
that's
gonna
look
like,
and
so
likewise,
in
the
reverse,
sorry
same
idea
on
the
other
side.
A
Okay,
so
now
we've
finally
done
everything
we
came
to
do
it's
time
to
have
fun.
Let's
make
a
network
see
ICD
workflow,
we
didn't
pick
any
one.
Particular
product
tool
set
or
otherwise
I
think
ansible
is
probably
the
most
popular,
but
some
things
to
leave
you
guys
with
is
that
once
you're
able
to
provision
a
network
topology
reliably
bootstrap
it?
However,
you
want
all
those
other
discussions
actually
become
pretty
easy,
because
the
community
and
and
the
networking
environment
you
know,
is
ready
to
receive
any
of
that
configuration.
A
A
And
we've
had
a
lot
of
success
with
this,
some
of
our
internal
testing
and
now
for
customers.
You
know
guys
that
our
network
engineer
has
been
doing
it.
Their
whole
lives,
don't
know
anything
about
kubernetes
they're
able
to
start
using
things
like
ansible,
because
now
they
have
an
environment
that
is
in
their
production
environment,
to
test
against.
A
We've
been
able
to
use
a
piece
or
API
subscription
from
a
particular
vendor
that
when
a
vulnerability
comes
in
the
model
that's
saved
and
all
the
testing
gets
kicked
off
with
an
automatic
upgrade
and
then
returns.
The
report
as
good
as
your
testing
is,
which
is
the
caveat,
but
some
things
we're
doing
next
and
are
continuing
to
work
on
now
is
integrations
of
vnfs
a
lot
of
work
in
the
community
around
producing
new
cloud
native
oriented
vnf
and
finally,
with
respect
to
the
data,
that's
required
to
train
things,
to
classify
and
logs
and
network
behavior.