►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Performance Optimization – Rook on Kubernetes - Mark Darnell & Ryan Tidwell, SUSE
Storage systems like Ceph require maximum performance from their underlying platforms, historically relying on custom silicon or bare metal access. Rook, a containerization of Ceph, will potentially require optimization of the network stack in order to match Ceph’s bare-metal performance capabilities. Join us as we walk the audience through the measurement, discover, and optimization processes, leveraging different architectures and CNI plugins to optimize Rook/Ceph on K8s.
https://sched.co/ZekO
A
A
So
as
an
introduction,
why
did
we
do
this?
Put
simply
souza
has
been
a
long
time
player
in
the
open
source
space.
We
are
involved
in
kubernetes,
we're
involved
in
ceph,
software-defined
storage
and
we're
rolling
products
out
that
will
integrate
those
technologies
via
rook.
In
order
to
integrate
them.
You
need
to
ensure
that
you
have
a
reasonable
network
stack
in
terms
of
performance
to
tie
those
together
and
the
driving
component
to
do.
That
is
a
cni
plug-in.
A
So,
let's
talk
about
kubernetes
networking
enough
to
ensure
that
once
again,
we're
working
from
a
level
playing
field
within
the
audience,
kubernetes
networking
works
in
a
couple
of
ways.
One
is
what
you
would
call
kind
of
the
standard
approach,
and
this
uses
a
set
of
technologies
that
are
part
of
both
user
space
and
kernel
space
in
linux,
you'll
create
things
like
v
eth
pairs,
you'll
have
mac,
vlan
and
ipvlan
you'll
talk
to
a
physical
interface.
A
A
A
It
it
kind
of
bypasses
a
chunk
of
security
that
you
would
typically
expect
to
have.
So
while
performance
tends
to
be
the
best
because
you're
not
running
through
any
of
that
overhead,
that's
listed
there
in
the
first
bullet
point,
you
do
have
a
problem
in
the
sense
that
you
may
be
less
secure,
so
souza
as
we're
rolling
these
products
to
people.
A
We
want
to
make
sure
that
we
provide
the
best
possible
performance
while
not
compromising
security
and
that's
part
of
what
these
benchmarking
methodologies
are
for
that
we're
working
through
I'm
going
to
go
ahead
and
turn
this
over
to
ryan
he's
going
to
do
some
of
the
heavy
lifting
here
and
describe
a
lot
of
the
work
that
he's
done
in
this
benchmarking
environment.
So
ryan.
Over
to
you.
B
What
I
wanted
to
show
here
was
to
kind
of
set
a
level
playing
field.
The
the
the
first
bottleneck
you
might
think
of
would
be
the
the
hard
drive
or
the
ssd
itself.
So
the
first
thing
we
did
was
just
benchmark
the
performance
of
these
ssds
that
we
had
in
our
storage
nodes.
These
are
the
same
ssds
and
the
same
on
the
same
motherboard
and
identical
configurations
across
the
cluster.
B
So
we
we
first
benchmarked
that
with
testing
right
and
re-dialogues,
I
want
to
just
point
out
something
on
this
slide.
That
might
be
useful
as
we
useful
to
remember.
As
we
go
on
with
the
with
the
right
benchmarks.
We
noticed
that
we
start
off
kind
of
slow
at
1k
and
2k
block
sizes,
and
we
believe
this
is
just
a
function
of
not
aligning
to
native
block
sizes
things
like
page
tables
sizes
and
that
sort
of
thing
so
keep
that
in
mind
again.
B
B
Get
this
is
just
a
snapshot
of
the
software.
You
know
the
kernel
versions
of
rope
things
like
that.
The
two
bullet
points
I
want
to
call
out
here
are
at
the
bottom,
calco
and
psyllium.
These
are
the
two
cni
plug-ins
that
we
tested
we
grabbed
psyllium
and
tested
that
we
do
a
lot
of
work
with
psyllium
here
at
souza.
B
This
is
also
interesting
from
a
performance
perspective
because
of
the
the
bpf
data
path
involved.
Calco
is
also
another
popular
tool
that
people
use
and
both
support,
different
encapsulation
modes,
and
I
want
to
call
your
attention
back
to
psyllium
and
just
mention
that
we're
going
to
be
testing
with
both
encapsulated
network
traffic
with
vxlan
and
we're
going
to
be
testing
a
mode
called
psyllium
direct
mode
where
we
don't
encapsulate
pod
traffic
for
calico,
we're
going
to
use
ip
and
ip
encapsulation
and
vxlan.
B
B
So
we
have
three
nodes
that
are
dedicating
as
storage
nodes
and
a
single
node
that
we're
using
as
our
client
node
now
there's
a
couple
things
I
want
to
call
out
here
at
the
top
in
that
top
box.
You'll
see
a
pod
now,
if
you're
not
familiar
with
file
file
as
a
benchmarking
tool
that
we
can
use
to
run
synthetic
workloads
on
a
storage
system,
whether
that's
a
physical
device
or
something
that
we
mount
through
nfs
or,
in
this
case
ceph
we're
going
to
use
that
to
gather
our
data.
B
So
we
have
a
pod
there,
that's
acting
as
our
client
and
that's
where
our
file
benchmarking
tool
is
going
to
run.
I
also
want
to
call
attention
to
the
networking
technologies
real
quick.
We
have
100
gig,
ethernet
networking
available
to
our
storage
nodes
and
a
bonded
pair
of
25
gig
mix
available
to
our
client
code
and
we'll
get
back
to
the
bonded
pair
of
25
gig
mix.
That
is
an
important
part
of
the
story
that
we'll
talk
about
later.
B
A
B
A
A
B
Now
that
I
want
to
point
out,
this
is
something
that
your
cni
configuration
and
your
network
configuration
isn't
really
going
to
influence.
That's
going
to
be
a
pretty
static
cost
in
the
system.
The
the
next
bullet
point
is
network
transport
latency.
Now
this
is
the
latency
that's
going
to
be
incurred
by
emitting
a
packet
on
the
network
having
it
traverse
the
your
topic,
rack
switch,
maybe
a
couple
of
top
rack
switches.
Maybe
a
router
depends
on
your
on
your
saddle.
B
B
The
key
point
here
is
less
network
latency
equals
more
ions,
so
the
more
we
can
drive
down
that
network
latency
the
more
efficient
we
can
be
with
providing
storage
to
clients,
because
we
talk
in
terms
of
iops
and
we
think
about
storage
systems,
we're
going
to
express
these
results
in
terms
of
iops
going
forward.
So
that's
just
a
note
to
set
the
terminology,
and
now
let's
move
on
and
talk
about
the
methodology,
so
scientific.
A
B
B
We
want
to
optimize
the
base
system
make
that
as
fast
as
we
can
so,
we've
got
jungle
frames
or
a
maximum
transmission
unit
or
mtu
of
9000,
and
we
want
to
use
the
fastest
disks
we
can
so
the
disks
are
faster
than
the
network.
We
don't
want
the
network
to
be.
We
don't
want
the
disc
to
be
the
bottleneck
here.
We
want
the
network
to
be
the
bottleneck
so
that
we
can
change
parameters
on
that
and
see.
B
Configurations
influence
the
performance
of
the
system
as
I've
been
alluding
to
disk
access.
Time
is
effectively
going
to
be
a
constant
here,
regardless
of
the
cni
plug-in,
and
then
we
have
a
single
rbd
client
running
on
a
dedicated
node
and
that's
where
we
measure
the
iops
we're
taking.
Latency
and
peak
bandwidth
demand
measurements,
and
then
we
run
through
this
whole
process,
swapping
out
the
cni
plug-in
or
the
cni
configuration
and
re-running
the
tests
by
changing
that
variable.
B
B
When
performing
a
read
benchmark
with
the
single
client,
we
observe
a
20
to
25
overhead
when
using
encapsulation,
so
both
psyllium
and
calico,
when
encapsulating
pod
traffic
are
incurring
pretty
significant
overhead.
Now
I
mentioned
psyllium
direct
mode
earlier.
This
is
a
mode
that
psyllium
runs
in
there's,
there's
a
similar
configuration
for
calco
that
that
we
didn't
test.
But
if
the
principle
still
holds
it,
we're
not
encapsulating
traffic
and
we're
not
paying
those
overheads,
and
so,
as
you
can.
A
B
B
B
B
We
observed
a
five
to
fifteen
percent
overhead
when
encapsulated
psyllium
direct
mode
still
stacks
up,
favorably,
again
we're
seeing
roughly
two
percent
overhead
for
that
psyllium
direct
mode,
really
competitive
with
host
networking,
and
then,
when
we
look
at
those
larger
block
sizes
in
one
meg
to
the
eight
minute
block
sizes.
This
is
where
things
got
a
little
interesting
and
we
had
to
dig
a
little
deeper
into
what
was
happening.
B
B
B
The
key
here
is:
we
want
to
figure
out
how
we
can
balance
traffic
really
nicely
across
these
mix.
Again
we
have
225
gig
mix.
This
is
a
lot
of
bandwidth,
let's
make
the
most
of
it.
So
how
do
we
balance
this
and
it
turns
out
it's
these
hash
policies
that
matter
so
bringing
this
back
to
the
environment,
our
initial
round
of
benchmarks.
We
observe
that
poor
utilization
on
that
on
that
bond.
B
Well,
we
had
to
tune
these
bond
settings
and
it
turns
out
that
the
parameter
that
yielded
the
most
dramatic
results
was
tuning.
This
transmit
hash
policy
parameter
on
the
bond.
Now.
Let
me
just
preface
this
by
saying
these
are
the
settings
that
yield
the
yield
of
the
most
dramatic
performance
gains
in
our
cluster,
so
for
host,
networking
and
psyllium
with
vxlan
and
selenium
with
direct
mode.
These
are
the
configurations
that
we
tested
by
with
the
tuned
hash
policy.
B
We
had
to
tune
them
a
little
different
because
they
have
different
characteristics
in
the
way
that
they
set
up
the
data
path.
So
you
know
with
host
networking,
we
use
this
layer,
3
plus
4
hash
policy,
which
what
that
does
is
that
just
takes
into
account
your
source,
ip,
your
desk,
ip,
your
source,
port
and
your
destination
port
feeds
those
into
the
hash
algorithm
and
determines
which
nic
packet
goes
out.
Based
on
those
parameters
for
vx
slant
for
sewing
with
vxlan,
we
had
to
use
the
end
cap
three
plus
four.
B
A
B
Outlet
packet
now
psyllium
direct
mode.
We
use
the
layer,
two
plus
three
hash
policy,
which
takes
into
account
the
the
source
mac
address
the
destination
mac
address
and
then
the
source
ip
and
the
destination
id
these
settings
yield
the
most
dramatic
performance
gains
in
our
cluster
keep
in
mind.
These
are
very
specific
to
our
cluster
factors
such
as
cluster
size,
bonding
mode
hardware,
capabilities,
they're,
all
going
to
call
for
different
settings,
and
you
may
need
to
tune
things
differently
if
you're
going
to
use
bonds.
B
This
is
an
indication
that
these
sorts
of
bonding
settings
will
matter
if
you're,
using
if
you
have
multiple
clients
using
the
network
and
putting
packets
out
there
and
stressing
the
network
now.
Another
thing
to
point
out
here
is:
you
may
not
be
using
bonding.
This
may
not
be
common
on
your
client
nodes.
That's
where
our
the
bonding
in
our
setup
was,
but
you
may
be
employing
bonds
on
your
storage
nodes
and
the
same
principle
applies.
B
B
So,
let's
finish
up
talking
about
bonding
what
what
what
to
know
about
bonding
again
this!
This
may
not
be
something
that
you
do
in
your
environment,
but
if
you
do
something
to
pay
attention
to
and
there's
there's
a
lesson
here
that
that
we
learned,
which
is
question
your
assumptions,
we
we
just
assumed
that
we
had
you
know
a
bonded
pair
of
25
gig
mix.
That's
you
know,
50
gig,
pipe
right
that
that
should
be
plenty
of
bandwidth.
There
should
be
a
bottleneck
there.
B
So
it
turns
out
if
you're
going
to
use
bonding
bonding
modes
matter,
you
want
that
balance
of
traffic.
Again,
these
are
sorts
of
configurations
that
are
going
to
be
specific
to
your
environment
and
factors
such
as
your
cni
configuration
your
scale,
hardware
capabilities,
they're,
going
to
drive
your
choices
here.
B
As
far
as
general
recommendations-
I
I
some
of
these
may
seem
seem
basic,
but
you
know
we
again.
These
are
learnings
and
things
that
were
confirmed
for
us.
Overlays
and
encapsulation
are
going
to
limit
your
iops
because
they're
going
to
introduce
latency
in
the
network,
so,
where
possible,
try
and
avoid
encapsulating
pod
traffic
with
vxlan
or
ipnip
or
gre
pick
your
encapsulation
protocol
that
that's
going
to
add
overhead.
So
we
want
to
avoid
that
where
we
can
bandwidth
demands
of
a
single
client
are
going
to
be
highly
correlated
with
block
size.
B
B
Tuning
these
things
can
yield
significant
performance
gains,
and
it
may
not
always
be
obvious
where,
where
your
bottlenecks
might
be
so
again,
question
everything.
This
experience
that
that
we
had
with
bonding
is
a
perfect
example
of
that.
So
with
that,
I
want
to
hand
it
back
to
mark
to
wrap
things
up.
A
Thanks
ryan,
I
really
appreciate
all
the
solid
work
there
and
we
believe
it's
going
to
set
the
stage
for
some
great
follow-up
work
that
we
want
to
present
at
following
kubecons.
So,
let's,
let's
drive
one
specific
point
home.
Once
again,
we
were
really
doing
this
work
in
order
to
determine
look,
there
are
multiple
cni
plug-ins.
You
can
use
in
multiple
network
paths
that
you
can
use
in
order
to
set
this
three-level
sandwich
of
seth,
rook
and
kubernetes.
A
A
A
So
it's
an
excellent
thing
to
kind
of
set
the
stage
for
where
your
system
is,
but
it
may
not
be
what
you
want
to
actually
deploy
in
production
next
iops
matter,
and
this
this
one
is
one
of
those
obvious
statements.
I've
realized
that
storage
people
look
at
iops
unnecessary
network
latency
is
going
to
hurt
iops
so
as
a
system
architect
ensure
that
you
design
your
architecture
to
minimize
network
latency.
A
Next
thing,
ceph
actually
implements
policy
enforcement.
Why
does
why
does
this
matter?
Well,
you
have
policy
enforcement
that
can
happen
at
the
ceph
layer.
You
have
policy
enforcement
that
can
happen
at
the
cmi
layer
and
you
will
have
some
storage
people
tell
you
hey.
I've
got
this
at
ceph.
I
don't
necessarily
need
this
in
my
cni
layer,
one
of
the
things
we
did
not
benchmark
here
was
implementing
additional
policy
in
the
cni
itself,
and
different
cni's
have
strikingly
different
performance
levels.
A
One
of
the
things
we
did
not
look
at
was
multis
with
sr
iov
adapters
using
virtual
functions.
Both
ryan-
and
I,
with
our
network
background,
believe
that,
fundamentally,
that
should
not
matter
that
host
networking
should
be
the
primary
or
the
best
performance
case
that
you're
going
to
be
able
to
generate.
A
We
would
like
to
test
that,
though
next
thing
calico
is
doing
some
extensive
work,
integrating
bpf
into
their
environment,
so
that
would
begin
differentiating
now
between
calico
and
celium,
and
using
the
same
underlying
technology
which
would
be
bpf
and
xdp.
We
believe
that
would
be
a
valid
test
case
to
start
looking
at.
In
addition,
we
would
like
to
improve
our
in-line
instrumentation
in
terms
of
making
sure
we
can
actually
timestamp
from
the
time
a
packet
leaves
fio
and
actually
exits
a
nick
on
the
box.
A
A
Finally,
we
have
three
nodes
in
a
cluster
and
one
device
driving
the
workload,
while
that's
great
for
setting
baseline
benchmarks
over
time.
We
need
to
add
more
clients
and
more
storage
nodes
in
order
to
have
cross
load
interference,
so
we'll
go
ahead
and
open
the
floor
for
q,
a
I'm
going
to
rip
through
this
and
we'll
figure
out
how
this
works
with
our
with
our
online
format
right
now.
Thank
you
for
your
time,
appreciate
your
attention
and
look
forward
to
hearing
back
from
everyone.
B
Okay,
everybody
thanks
for
joining
happy
to
take
some
questions.
We've
had
a
few
good
ones
come
through
the
chat,
while
the
presentation
was
airing
so
dive
into
a
couple
of
those
one
question
that
kind
of
popped
to
the
top
here
was:
how
would
I
improve
latency
if
I'm
using
nfs?
My
answer
to
that
would
be
everything
that
we
talked
about
here
applies
to
nfs
as
well,
so,
whether
you're
using
rbd
or
nfs,
these
same
principles
are
gonna
apply.
B
A
B
I
should
be
able
to
get
amazing
performance,
because
I
have
all
this
fast
networking
available
to
me
and
the
reality
is
at
least
in
this
setup.
Latency
or
bandwidth
was
not
the
driving
factor
for
performance
and
what
it
really
turned
out
to
be.
Were
these
different
configuration
parameters
that
we
discussed
in
the
talk,
cni
plug-in
and
the
way
you
configure
it
actually
makes
a
huge
difference,
and
what
we're
hitting
there
is
that
latency,
both
in
how
we
access
the
network
and
in
the
kernel
as
we
use
different
cni
configurations.
B
If
I
tune
it
right,
it
turns
out
the
really
the
way
to
use
all
that
bandwidth
was
actually
to
push
larger
block
sizes.
Now,
of
course,
as
you
scale
up,
and
you
start
using
more
as
you
have
more
clients
using
the
cluster,
regardless
of
the
block
size,
as
you
scale
that
out
you're
going
to
make
better
use
of
that
bandwidth.
A
B
B
B
B
B
B
Nice
for
filling
your
pipe,
but
not
for
iops
yeah,
absolutely
the
the
one
four
meg
that
definitely
fills
the
pipe
puts
stress
on
the
network
and
kind
of
the
reason
I
scaled
up
there
is,
I
actually
profiled
the
characteristics
of
different
block
sizes,
just
as
something
to
explore
to
kind
of
see
what
the
impact
on
the
network
was,
and
it
turns
out
that
that
turned
into
a
sort
of
a
proxy
for
scale
in
the
system.
It's.
B
But
you
know
it:
it
puts
more
network
traffic
out
there,
it's
more
for
the
system
to
handle,
so
it's
sort
of
a
proxy
for
scaling
up
with
multiple
clients.
Obviously,
as
we
talked
about
in
the
talk
just
having
a
single
client
out,
there
is
not
realistic
and
that's
one
of
the
things
that
we're
looking
at
is
doing.
A
B
Exercise
in
a
up
environment
where
we
are
using
more
realistic
block
sizes
using
a
more
realistic
number
of
clients
and
a
more
realistic
configuration.
This
exercise
is
kind
of
a
first
pass
to
build
kind
of
a
basic
understanding
of
what
the
key
factors
we
need
to
pay
attention
to
with
regard
to
performance
on
so
yeah.
Thanks
for
that.