►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
High Performance Networking with Kubevirt - Doug Smith, Red Hat & Abdul Halim, Intel
When we’re talking about legacy workloads running in Kubernetes as Virtual Machines, we’re not talking about “any old workload” -- we’re usually talking about performance workloads. The KubeVirt community, Red Hat & Intel have come together to enable high performance networking for these workloads. Need hardware-accelerated networking in a cloud-native environment? Using KubeVirt (for virtual machines on top of Kubernetes) and recent improvements in SR-IOV networking integration with Kubernetes, we'll talk about gaining performance networking for your apps, and for legacy workload migration. You’ll get the opportunity to explore the mechanics of running pods that are also connected to SR-IOV NICs. We’ll also look at how KubeVirt builds on top of and extends the base Kubernetes platform support for SR-IOV networking into the world of legacy virtual machines, all managed by kubectl.
https://sched.co/MPcw
A
Thanks
Brian,
so
today
we're
gonna
talk
about
some
high-performance
networking
using
Qbert.
My
name
is
Doug
Smith,
as
brian
introduced,
I'm
from
the
Red
Hat
office
of
the
CTO
in
my
particular
team.
We
work
on
networking
and
we
also
work
with
strategic
partners
of
Red
Hat,
specifically
looking
at
NFV
situations
and
I'm
joined
today
by
Abdul.
A
So
let's
just
talk
about
Qbert
a
little
bit
and
really
Cooper
is
a
way
to
take
your
virtual
machine
workloads
and
run
them
under
kubernetes.
So
this
is
something
that
you
can
do
in
order
to
prevent
yourself
from
having
to
say,
maintain
two
separate
platforms:
you're,
seeing
a
lot
of
benefits
out
of
using
kubernetes
for
your
for
DevOps,
and
you
might
have
workloads
that
just
aren't
containerized
yet
right.
So
we
might
ask
the
question:
well,
can
I
just
containerize
everything
and
the
thing
is
sometimes
that's
just
not
really
a
possibility.
A
Sometimes
maybe
your
vendor
has
shipped
you
software,
that's
already
a
virtual
machine,
and
now
you've
got
to
run
it,
and
do
you
really
want
to
have
to
have
another
platform
when
you've
got
a
lot
of
forward
progress
here
in
order
to
run
your
containerized
workloads?
Maybe
you
need
kernel
customizations
and,
as
you
know,
your
containers
are
utilizing
your
hosts
kernel
and
maybe
you
just
like
having
some
pets,
maybe
they're
part
of
the
family,
and
you
just
want
to
keep
them
around.
A
So
here's
the
thing
now:
we've
got
this
like
crazy
world
here,
where
we've
got
containers
and
VMs
side-by-side
right
and
we're
gonna
be
able
to
take
those
and
build
them,
deploy
them
maintain
them
all
in
the
same
environment.
We're
gonna
be
able
to
keep
our
same
workflow
that
we
know
and
love
under
kubernetes
and
we're
going
to
be
able
to
migrate
these
workloads
to
potentially
be
containers
in
the
future.
If
you
want
or
don't
just
keep
them
as
keeping
those
virtual
machine.
A
So
how
do
we
actually
do
this,
and
the
thing
is
it's
really
not
that
complex?
It's
just
an
addition
that
you
use
with
kubernetes
you're,
just
gonna
use
the
same
kind
of
workflows.
You're
gonna,
create
images,
so
you're
gonna
store
in
a
registry
that
are
gonna,
be
your
container
images
that
are
also
you're.
Gonna
spin
up
your
virtual
machines
from
you're
gonna
create
gamma
little
pod,
specs
and
you're
gonna
deploy
those
and
it's
gonna
look
almost
just
like
you
would
for
any
other
workload
that
you
have
running
in
queue
and.
A
A
So
let's
take
a
look
at
what
Qbert
virtual
machine
looks
like
and
really
you've
got
your
node
you've
got
your
container
run
time
running
on
it
and
then
you're
gonna
have
a
daemon
set.
It's
got
your
vert
handler
and
then
that's
gonna
spin
up
a
pod.
That
is
your
VM
and
it's
so
a
pod,
of
course,
is
kind
of
a
logically
grouped
set
of
containers.
You've
got
kind
of
a
sandbox
namespace
that
these
containers
live
in
and
you're
gonna
have
one
that's
your
launcher.
A
A
A
So
we
assign
an
IP
address
to
it
right
and
it
connects
to
our
default
network
and
when
I
say
default,
network
I
mean
our
network
that
we
get
our
pod
pod
connectivity
from.
So
this
is
something
that
we're
always
gonna
want
to
have.
You're
gonna
have
other
services
that
you're
gonna
consume
on
your
control
point,
be
it
maybe
you're
gonna
get
metadata
from
a
database
or
something
like
that.
You're
gonna
want
to
connect
to
the
rest
of
your
cluster,
as
you
normally
would
so.
A
You
might
assign
that
with
DHCP
and
you've
got
here
on
the
left,
this
kind
of
definition
of
your
interfaces.
We
got
your
default
interface
and
your
default
Network
and
those
are
connected,
and
then
what
we're
going
to
do
is
we're
going
to
plumb
another
interface
into
into
the
virtual
machine
right
and
into
the
pod,
by
definition,
since
we're
running
in
kubernetes,
and
so
we
have
this
other
network.
A
So
this
is
going
to
be
your
data
point,
so
this
is
where
you're
gonna
have
your
high-performance
needs,
there's
something
that
you're
going
to
do
on
this
this
network.
So
in
our
demo,
in
this
case
it's
a
voice
over
IP
application.
Yours
could
be
anything
that
your
heart
desires,
so
we
have
on
the
left
here,
our
llamo
that
defines
this
secondary
other
interface
and
this
other
network,
and
that's
really
all
that
you
have
to
do
in
order
to
enable
this
Qbert
is
managed
by
an
operator.
A
A
A
So
one
of
the
particular
standards
that
Qbert
in
implements
is
something
that
was
created
by
the
network,
plumbing
working
group,
which
is
a
group
that
was
formed
that
cube
con
2017
in
Austin,
and
it's
a
number
of
people
that
were
interested
in
looking
at
the
kind
of
multi
networking
problem
in
kubernetes.
To
answer
the
question:
hey:
how
do
I
attach
multiple
networks
to
a
pod,
so
the
network
plumbing
working
group
came
together
and
they
created
a
de
facto
standard
for
a
CRD
customer
resource
definition
in
kubernetes.
A
That's
called
the
network,
attachment
definition
and
it
essentially
specifies
how
do
I
express
my
intent
to
a
attach
an
additional
interface
to
to
a
pot
and
Cubert
uses
this
in
order
to
accomplish
what
we
showed
in
those
previous
diagrams,
with
attaching
that
interface,
specifically,
the
technology,
that's
used
to
execute
that
custom
resource
is
multi,
CNI
of
which
I
am
a
maintainer
and
motifs
is
a
meta
plugin.
So
it's
a
CNI
plug-in,
that's
able
to
call
other
CNI
plugins.
A
So
let's
talk
about
a
device
plugins
for
just
a
moment.
So
in
order
to
use
SRV
interfaces,
we
use
an
S
ROV
device,
plug-in
device.
Plugins
give
the
kubernetes
schedule
awareness
of
physical
resources
that
have
been
utilized
on
particular
note.
So
in
the
case
of
SRO
V,
we
have
physical
functions
and
then
we
have
virtual
functions
and
when
you
use
a
virtual
function,
this
is
a
resource
that
can
be
exhausted
on
a
particular
host.
So
you
go
and
create
your
VM
it's
in
a
pod.
A
It
uses
an
SRO,
the
interface
and
the
device
plug-in
lets
the
kubernetes
scheduler
know
hey.
This
resource
has
been
used
up
so
that
way
this
works
at
scale.
So
you
schedule
workloads
on
the
node
until
it's
full,
then
the
kubernetes
scheduler
lets
is
told
by
the
device.
Plugin
hey
this
node,
it's
full,
don't
oops,
don't
schedule
anymore
anymore,
pods
here
and
so
previously,
there's
been
like
a
SRO
VC
and
I
plugin,
that's
still
used
and
that
works.
A
B
So
this
virtual
switch
or
Linux,
please
introduce
some
additional
overhead
which
may
be
affect
your
performance
workload,
so
you
can
more
latency
and
less
throughput,
so
this
may
be
not
sufficient
for
the
many
use
cases,
for
example
high
performance
and
FB
use
cases,
and
we
need
really
first
so
here
we
see
so
ideally,
we
would
like
to
have
if
we,
if
you
like,
to
remove
this
overhead
we'd
like
to
have
a
direct
NIC
assignment
to
the
HP
IBM.
But
problem
with
that
is
we
have
limited
number
of
I/o
port
in
a
system.
So
that's
problems.
B
The
scalability.
So
here
here
is
where
they're
sorry
we
they
come
here
to
help.
So
we
can
see
that
in
a
but
it's
our
we,
we
have
a
way
of
one
device
appear
to
be
multiple
PCI
device,
so
each
of
this
device
can
be
discovered
and
configured
independently,
so
so
each
VM
now
can
get
a
BF
directly.
So
as
I
we
introduced
this
concept
of
physical
function
and
virtual
functions.
B
So
now
we
will
see
okay.
How
do
we
enable
this
asari
BC
and
is
RB
networking
and
kubernetes?
So
initial
work
was
done
by
$0.10
creative
disaster.
We
see
an
eye
plugin
to
enable
syb
networking
in
kubernetes.
So
we,
when
you
start
looking
into
that,
we
discovered
there
are
some
shortcomings
that
distilling
a
plugin
didn't
handle
properly
the
device
accounting
that
which
we
observe
free
error,
which
ones
are
not
free.
So
we
started
extended
this
plugin
and
added
this
BF
management
capabilities.
B
So
now,
CNI
plugin
can
allocate
a
BF
which
one
is
fear,
which
one
is
not
News
and
the
one
is
use.
The
one
is
not
free.
You
don't
touch
that
one.
So
you
also
wanted
to
have
this
V
F
to
be
used
in
the
DPD
application,
so
we
also
added
abilities
so
that
VF
can
be
attached
to
a
DVD
capable
driver,
but
problem
with
that
was
that
we
now
we
were
replacing
the
existing
default.
It
it's
zero
interface.
B
In
a
part,
we
want
to
maintain
the
kubernetes
control
plane
traffic
to
the
it's
your
interface
and
we
want
to
add
the
additional
dark
plane
through
their
soybean
etwork.
So
for
that
multis
there
to
help.
So
we
we
started
working
on
the
Maltese
way.
Maltese
was
a
way
to
introduce
additional
interface
in
there.
So
now
we
can
add
the
data
plane
network
through
a
so
ABC
and
I
plugin.
So
by
looking
into
this
picture,
you
may
think:
okay,
that's
I
mean
CNI.
Plugin
is
enough
to
solve
your
I've,
been
African
problem
in
Cuban.
B
It
is,
but
we
fit
into
some
limitation
here
when
a
what
is
it
so
there's,
as
dog
already
touched
into
that
that
scheduler
is
not
error
of
this
resources,
so
CNI
plugin
is
not
a
place
that
it
can
communicate
to
the
kubernetes
api
we
also
wanted
to.
If
we
want
to
ensure
the
really
high
performance
capabilities,
we
want
to
ensure
that
this
behaves
are
coming
from
the
same
no
more
circuit
of
your
CPU
and
memories.
So
this
is
also
cannot
be
addressed
in
the
CNI
plug-in
itself.
B
Then,
if
we
want
to
add
this
bf
in
a
using
is
a
user
Express
driver
where
we
couldn't
do
this,
as
if
you
see
group
isolation
through
the
CNI
plugin
as
well,
so
for
that
we
started
looking
with
possible
solutions,
so
device
plugin
was
the
perfect
fit
for
this.
All
of
this
problems,
just
the
new
Maryland
mint,
is
still
not
possible
because
we
are
still
working
on
with
the
kubernetes
upstream
to
get
auto
project
manager
KP
to
be
March
once
that
goes
in
there.
B
We
can
address
the
new,
my
alignment
as
well,
so
this
is
just
a
brief
overview
of
how
our
device
plugin
works
in
kubernetes,
so
device
plugin
is
alone.
When
it
is
it
just
it
implements
the
device
plug-in
API
and
the
it
responsibility
of
this
is
to
discover
the
devices
and
then
cubelet
can
make
a
request
for
a
device
allocation
for
a
workload
and
then
device
plugins
and
required
configuration
random
configuration
for
the
for
those
devices.
B
B
So
here
is
the
pod
spec,
so
we
have
the
device
plug-in
that
discovered
and
then
update
the
node
status
as
a
you
can
see
the
highlighted
purple
writing
there.
That
resources
are
this
resources.
I
have
discovered
for
this
node
and
this
can
update
it
to
the
kubernetes
api.
Then
we
create
the
net
attachment
CRD
for
maltese
to
say
that
we
are
going
to
use
this
PF
using
this
resource
and
that's
going
to
use
this
RB
CNI
plugin.
B
So
this
is
the
overall
flow,
so
you
have
a
part
coming
in
with
the
so
every
network
resource
requirement
in
the
poly
spec
and
then
Hewlett's
does
the
once
the
part
is
scheduled
to
this
node
cubelets
does
the
resource
allocation
through
the
device
plug
in
and
then
multis
during
the
network
network
creation
multis
attach
those
bf
into
their
part.
So
here
are
some
links
for
the
reference
anyone
would
like
to
contribute
to
this
project
this.
B
So
we
also
have
some
demo
going
on
in
inter
booth,
so
you
can
see
the
live.
Demo
have
the
whole
orchestration
worse
in
there
we
device
plug
in
in
kubernetes,
and
we
also
have
some
benchmarking
results.
Anyone
want
to
interested
to
see
that,
so
you
can
welcome
to
come
along.
So
with
that
I
will
hand
over
to
Doug.
B
A
So
just
quickly,
I
just
wanted
to
take
a
look.
Some
of
the
roadmap
for
SRO
V
and
some
of
the
things
that
to
me
are
the
most
important
things
are
functionality
that
we
need
to
be
able
to
push
all
of
the
performance
that
we
can
out
of
here.
So
you
can
bring
up
all
these
tools
you
get
all
of
it
working,
but
we
still
need
some
forward
progress
on
CPU
management
and
numa
alignment.
A
So,
with
those
two
pieces
in
place,
we're
gonna
be
able
to
crunch
all
of
the
performance
that
we
can
out
of
these
with
some
of
the
newer
hardware,
you
might
be
able
to
get
pretty
close,
but
you're
gonna
be
able
to
really
push
out
all
of
that
performance
if
you
with
all
of
your
hardware,
if
you've,
if
you've
got
these
in
place,
so
we're
working
towards
that.
So
let's
take
a
look
at
a
demo.
Let
me
get
this
up
here.
A
You
can
use
the
single
interface,
you
can
just
use
TCP
HTTP
type
type
traffic,
but
in
the
networking
world
often
we
have
the
kind
of
traffic
that
just
doesn't
fit
that
mold,
quite
as
well
so
I
kind
of
want
to
highlight
that
so
think
about
the
void
workload
in
terms
of
what
you
might
actually
put
on
there.
So
I
have
a
background
in
telephony,
so
this
is
of
course
interesting
to
me,
but
you
might
substitute
it
with
just
about
anything.
That's
there.
So
let's
take
a
look
at
this
particular
demo.
A
So
what
I'm
showing
here
is
that
we've
got
this
three
node
cluster.
Only
one
of
these
nodes
has
the.
Let's
see
one
of
these
nodes
has
the
s
ROV
devices
here,
so
that's
the
one
with
the
non-uniform
and
so
I'm
just
running
LS,
PCI
and
I'm,
just
showing
that
we've
got
this
world
thirty-two
different
vfc
vailable
here.
A
So
what
we're
gonna
do
is
we're
gonna,
just
spin
up
a
C,
Rose
VM.
So
that's
like
kind
of
a
classic
type
of
virtual
machine
workload
for
a
test
scenario,
so
kind
of
just
two
quick
steps
here.
There
may
even
be
more
efficient
ways
to
do
this,
but
just
a
cute
cuddle
create
from
IMO
spec
will
look
at
the
details
of
one
for
SRB
in
a
second
and
then
I.
A
Just
use
kubert's
vert
cut
all
command
to
start
this
VM
then
I
just
watch
it
come
up,
doesn't
take
long
at
all
and
then
what
I'm
gonna
do
is
I'm
gonna
console
into
this
VM
and
there
we
go
just
like
you're
at
any
VM
console
and
I'm
gonna
run
IPA
here
and
something
I
just
wanted
to
point
out
and
sort
of
your
vanilla.
Virtual
machine
is
when
I
run,
IPA
I've
just
got
a
loopback.
A
Oh
he's
got
a
loopback
and
you've
got
your
East
zero
here,
and
in
this
case
this
is
kind
of
your
like
typical
flannel
address
that
you
would
have
assigned
to
it.
So
that's
what
you
that's,
what
it
looks
like
to
just
spin
up
any
VM
now,
let's
take
a
look
at
the
specification
for
a
for
a
workload
that
is
going
to
have
srl
be
connected
to
it.
So
here
we
see
in
the
spec
we've
got
this
special
kind
virtual
machine
instance.
A
That's
how
we
specify
that
this
is
going
to
be
enabled
by
Cubert,
and
we've
got
a
number
of
things
here
that
are
relevant,
one
of
which
is
your
image
here.
So
just
like
any
any
kind
of
spec
for
our
workload
in
kubernetes,
you're
gonna,
say
hey.
This
is
an
image
I'm
using,
but
in
this
particular
case
it's
kind
of
our
disk
image
for
the
VM
right.
A
So
this
one
is
a
fedora
at
the
end
and
we're
gonna
specify
here
in
the
networks
that
we
have
we're
using
Malta's
for
our
additional
network,
so
we're
still
connected
to
the
default
network.
But
then
we
are
going
to
specify
that
we
have.
This
network
name,
which
is
s
RI
LV
net
and
this
maps,
to
a
configuration
that
we
have
stored.
So
let's
take
a
look
at
these
particular
network
attachment
definition.
A
So
here
I'm
saying:
let's
get
these
custom
resources
that
we
have
for
network
attachments,
which
are
essentially
going
to
be
our
configurations
that
we
have
for
each
of
our
additional
interfaces
and
we
have
one
name:
s
ROV
net,
just
as
we
it's
best
fide
in
the
spec
for
the
4q
pert,
and
let's
describe
this
here.
Quick
and
we're
gonna
see
that
there
is
it's
a
kind
network
attachment
definition
and
we
have
a
config
here-
and
this
is
a
CNN
configuration,
that's
packaged
as
part
of
the
content
of
this
custom
resource
the
things
granted.
A
It's
all
condensed,
JSON,
which
isn't
always
the
nicest
to
read
but
I
think
the
stuff.
That's
the
most
important
for
now
is
that
we've
got
a
type
of
srl
V.
That
means
that's
the
binary
that
CNI
is
gonna,
call
on
disk,
and
then
we
have
the
name
s
ROV
net.
So
this
mapped
from
what
we
had
in
our
in
our
gamal
spec.
What
we
have
here
as
our
name
as
our
arena
here,
and
we
see
that
it's
there,
so
that's
our
mapping
of
which
configuration
do
we
want
SRO
V
net.
A
A
So
let's
take
a
look
at
the
pods
that
we
have
running
and
we
can
see
that
there
are
those
VMs
running
as
pods.
You
can
check
them
out
with
Q
cuttle,
just
like
any
other
and
I'm
gonna
go
ahead
and
describe
this
node
so
that
we
can
take
a
look
at
the
allocated
resources
on
this
particular
node.
So,
on
this
node
we
can
see
that
we've
gotten
this
type
of
resource,
that's
Intel,
comm
/s
rlv,
and
we
have
two
requests
for
that
type
of
resource.
A
A
So
that's
what
we're
gonna
do
is
use
cube,
cuddle
to
describe
the
node
and
look
at
our
allocatable
resources
and
we're
gonna
see
that
we
have
32
possible
sro
VVS
right
there,
just
like
we
had
in
the
beginning
of
the
demo
or
I,
showed
hey
here's
the
LSP
CI
on
this
particular
node.
So
all
the
stuff
is
being
tracked
for
you.
All
you
have
to
do
is
request
it
and
it
does
all
the
difficult
work
for
you.
So
now,
I'm
going
to
console
into
one
of
these
s,
ROV
MVM
run
IPA
and
we're
gonna
see.
A
A
Now
that
we've
got
all
of
those
running
what
I'm
gonna
do
is
I'm
going
to
go
into
one
of
those
I'm
gonna
use
asterisk
to
create
a
void.
Call
and
I'm
gonna
originate
a
call
from
one
machine
to
the
next,
and
here
we've
got
a
ton
of
debug
output
from
it.
But
what
we
can
see
here
is
that
we
have
sent
and
an
invite
to
the
called
party
and
it
contains
an
SDP.
A
That's
our
media
description
and
we
have
received
a
response
to
answer
that
call
from
the
call
party
and
they
begin
the
call,
and
then
we
see
the
media
coming
back
and
forth
between
these
two
machines
to
actually
complete
the
call.
So
we
have
created
a
data
plane
and
we
have
originated
some
VoIP
traffic
over
that.
A
So
that
is
the
demo
here
and
lastly,
you
can
go
ahead
and
run
through
this
entire
demo
on
your
own,
go
to
Doug,
btv
comm
or
follow
this
QR
code
and
I've
got
the
step-by-step
instructions
to
be
able
to
do
that.
You
don't
have
to
be
a
super
kubernetes
expert.
In
order
to
do
it,
you
don't
have
to
be
a
super
networking
expert
to
do
it.
It's
fairly
simple
and
thanks
to
the
hard
work
of
the
cuber
guys
with
the
operator,
the
stuff
is
pretty
easy
to
get
going.
A
C
A
So,
first
first
service
discovery,
so
let
me
just
make
sure
to
clarify
it.
There's
when
I
think
of
services
and
I
might
be
thinking
this
from
a
network.
Plumbing
working
group
perspective
is
something
that
we
have
been
exploring
and
I
certainly
wouldn't
invite
anyone
to
join
us
with
the
network.
Plumbing
working
group
is
the
interfaces
that
we
create
that
are
additional
to
your
default
Network,
our
sort
of
side
side
car
network,
so
they
are
what
we're
looking
at
doing
is
making
those
interfaces
complete
parallel
equals
to
the
interfaces
that
are
available.
C
A
B
E
A
Will
I'll
do
I'll
defer
to
the
kubera
guys
on
the
on
the
storage
side
of
it,
but
on
for
the
services
in
particular,
is
one
of
our
goals
for
this
next
cubic
on
us
is
to
figure
out
exactly
how
we're
gonna
handle
the
service
abstraction,
and
we
actually
had
a
discussion
with
a
group
here
at
cube
con
yesterday
and
I
believe
that
we
have
a
good
path
forward.
There
is
we
have
some
proof
of
concept
code
out
there
that
works
quote
unquote.
A
We
need
to
further
gel
the
kind
of
specification
for
it
and
also
with
some
recent
advancements
and
how
service
objects
work.
Thank
you.
We
think
that
we
have
a
path
forward
for
that.
One
I
believe,
though,
that
with
storage
that
you
should
just
be
able
to
amount
volumes
into
the
pods
like
you
do
any
other
work
load
in
kubernetes,
so
you
can
make
your
dynamic
volume
attachments
and
whatnot,
so
you
can
use
other
whatever
storage
engine
you
like.