►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Storage and Networking: Rook on Multus - Sébastien Han & Rohan Gupta, Red Hat
Networking plays an important role in the world of baremetal cloud storage. I/O on devices are dependent on network latency. Rook-Ceph now makes use of multus to better utilize host network interfaces for Ceph Public and Cluster Networks. Having dedicated networks for storage boosts performance. This talk will introduce Ceph public and cluster networks and their use cases, the requirement of dedicated networks in baremetal environment, HostNetworking vs Multus, defining network-attachment-definitions, and how Rook-Ceph utilizes it. It will also include a demo showing an already running cluster utilizing multus networks and we will compare the performance with and without multus. By the end of the talk, you'll know the perks of using private and cluster networks, when to use them, and will be able to create a basic networking setup for Rook-Ceph.
A
A
Obviously
there
is
the
deployment
burden,
but
this
could
just
be
the
responsibility
of
the
storage
team
and
how
about
day,
two
operations
if
you
want
to
increase
the
capacity
of
the
cluster
like
who
is
responsible?
Who
is
actually
managing
that
storage
for
you?
If
you
have
to
go
and
ask
for
like
a
different
team
to
increase
the
capacity,
and
how
is
the
storage
going
to
be
allocated
when
these
are
all
the
the
questions
you
and
just
to
add
to
that,
we
also
we
we
might
be
relying
if
you're,
not
like
relying
on
external
storage.
A
You
might
be
relying
on
providers
in
many
services
like
ebs,
but
this
means
also
that
you
have
been
the
login
too,
and
also
cloud
providers
comes
with
their
own
set
of
issues.
You
might
be
limited
to
the
number
of
pcbs.
You
can
request
for
each
nodes
and
things
like
this,
so
there
are
also
limitations
when
you
use
the
bear
cloud
provider
managed
services
too,
and
rook
is
essentially
the
response
to
that.
A
So,
if
you're
familiar
with
the
operator's
pattern,
then
you
would,
you
should
know
that
typically,
operators
are
logical
entities
are
responsible
for
bootstrapping
software
and
managing
them
and
automating
their
deployments,
their
configuration
and
the
upgrade
so
essentially
the
entire
lifecycle.
This
is
what
rook
does.
A
First
rook
is
a
cnc
evaluated
project
since
last
october,
and
then
what
I
just
like
I
said
now,
storage
is
being
provided
from
within
the
kubernetes
cluster.
It
is
almost
just
like
another
app
that
runs
as
part
of
kubernetes
and
what
we
gain
from
that
is.
We
really
have
this
homogeneous
experience,
regardless
of
the
platform
you
might
be
running
on,
so
your
kubernetes
cluster
may
be
bare
metal
or
maybe
running
on
the
cloud,
for
example.
A
Then
what
is
ceph
so
ceph
is
also
open.
Source
sap
is
a
distributed,
storage
software
defined
solution
and
is
really
unique,
unique
in
a
sense
that
not
only
it
provides
you
many
storage
interfaces,
but
it's
extremely
scalable
with
no
single
port
of
failures.
It
was
really
built
for
for
scale
and
one
of
the
most
unique
features.
As
I
said,
it
provides
access
and
it
really
allows
you
to
consume
storage
through
several
interfaces.
A
It
also
exposes
a
shared
file
system,
either
using
a
native
driver
or
using
fuse,
and
it
also
provides
object.
Storage,
like
native
object,
storage.
It
has
gateways
that
are
necessary
compliant
if
your
app
is
designed
to
interact
with
s3,
then
ceph
throughout
its
gateway
can
just
understand.
All
of
that
and
your
app
all
of
the
southern
is
compatible.
It's
like
having
amazon,
s3
in-house.
A
All
of
these
layers
support
snapshots
cloning,
your
replication
and
the
list
goes
on
honestly.
We
don't
really
have
time
to
discuss
all
the
goodness
of
self,
but
this
is
just
like
a
highlight
of
what
seth
is
really
capable
of
and
last
but
not
least,
have
his
robbers
battle
tested
for
more
than
10
years
has
seen
thousands
of
production
deployments
going
through,
like
multiple
terabytes
to
essentially
petabytes
of
storage.
A
A
little
bit
more
now
about
like
that.
Now
that
we
know
about
group,
we
know
about
seven,
how
do
they
fit
together?
What
is
the
actual
architecture
you
get
when
you
run
rooks
f,
so
essentially
it's
divided
into
three,
but
also
optionally,
four
pieces
that
we'll
see
so
rook
as
mentioned
is
d
operator.
The
brain
that
owns
the
management
of
management
means
deploying
configuring
and
upgrading,
and
we
have
the
ceph
csi
driver,
which
is
a
csi
driver
that
allows
you
to
dynamically
provision
storage
so
effectively
what
it
does.
A
It
provides
persistent
storage
to
your
applications
more
like
not
more
recently,
but
coming
soon,
we
will
have
a
self
cozy
driver.
Actually
we
already
have
it,
but
it's
really
better
alpha
at
the
moment
and
what
it
does
it's
essentially
similar
to
csi,
but
for
object.
So,
instead
of
claiming
for
storage
through
either
a
block
or
a
file
system
type
of
interface,
then
you
could
claim
bucket
and
they
would
put
objects
in
it.
Just
like
you
would
like,
if
you
interact
with
s3,
for
instance,
and
then
we
have
steph
as
the
data
layer.
A
B
A
Bucket
provisioner,
so
on
the
very
left
we
we
have
a
traditional
claim
with
rwo
or
we'd
write
once
so.
Essentially,
the
app
is
claiming
one
block
device.
This
block
device
is
being
attached
to
the
node
and
comes
through
a
storage
class
that
currently
is
rbd,
which
represents
the
router's
block
device
and
in
the
middle
we
have
files,
but
this
could
also
be
true
for
the
block
layer,
because
all
the
of
all
the
storage
interfaces
for
himself
do
support
all
the
wx.
A
All
of
these
interfaces
are
supported
by
all
the
interfaces
of
the
driver,
so
the
app,
for
example,
can
consume
files
with
the
rwx
fashion,
which
is
read,
write
many,
which
means
I'm
writing
this.
I'm
mounting
this
storage
multiple
times
and
I'm
accessing
it
reading
and
writing
to
it.
At
the
same
time,
from
various
nodes
and
on
the
very
far
right,
we
have
the
object
functionality
which
comes
with
a
bucket
claim
today,
so
you
would
be
creating
an
object
bucket
claim.
B
A
Self-Cluster
replication
network,
which
essentially
represents
the
internal
replication,
because
ceph
is
a
replicated
storage.
So
once
you
once,
the
client
writes
once
then
that
data
will
be
just
duplicated
like
two
times
more
so,
which
gives
you
replica
three,
for
instance,
and
this
network
is
mostly
responsible
for
replicating
data
or
backfilling
it
if
you
lose
a
node
and
like
those
types
of
internal
operations,
of
course,
seph
works
well
with
a
single
public
interface.
A
This
is
an
example
of
the
topology
that
rohan
will
walk
you
through
during
the
demo,
but
typically
this
is
like
all
the
next
networks
that
will
be
present
in
the
demo,
and
this
is
like
more
or
less
a
traditional
step
deployment
regardless
if
it's
full
of
kubernetes
or
not
at
the
very
top.
We
have
the
rook
operator
that
is
connected
to
the
sdn
network.
So
I
find
that
it's
a
default
network
that
you
get
from,
for
example
like
kaliko.
A
It
is
a
default
interface
that
all
your
pods
are
connected
to.
Then
we
have
the
second
network,
which
is
the
most
interesting
part
of
the
third
one
which
are
actually
being
provided
by
multis,
and
we
have
so
the
staff
public,
as
well
as
the
sub
cluster.
We
can
see
that
the
oasis
at
the
very
bottom
are
all
connected
throughout
the
cluster
network,
but
then
all
the-
and
they
are
obviously
also
connected
to
the
public
facing.
A
If
we
look
at
the
very
top
right,
we
see
a
user
writing
data
in
its
application
and
it's
using
rbd,
and
then
we
can
see
that
the
traffic
goes
through
the
sdn,
the
the
primary
default
sdn,
as
well
as
the
rook
public
interface.
As
mentioned
earlier
now,
I'm
just
going
to
hand
it
over
to
rohan
to
walk
you
through
the
networking
with
lotus
you're
on.
B
Except
so
currently,
rook
has
support
for
three
networking
methods.
It
has
the
traditional
pod
network,
where
the
pods
use
the
default
kubernetes
networking,
which
has
been
created
by
the
administrator.
Then
we
have
the
host
networking
in
this.
The
pods
utilize,
the
host
network
and
the
network
interfaces
on
the
host
are
visible
inside
the
spot.
B
So
this
is
a
basic
network
attachment
definition
that
uses
public
network
that
uses
a
macbool
and
type,
and
it
is
linked
to
ens
5
on
the
host
so
for
every
network
attachment
definition
that
is
which
is
using
macbool
and
type.
We
will
have
to
pick
up
a
interface
on
the
host
and
we
will
have
to
refer
it
here
in
master
so
that
it
utilizes
that
interface.
B
And
similarly,
we
will
create
a
cluster
network
attachment
definition,
and
how
do
we
tell
rook
about
this
networks?
So
in
the
rook
spec
we
have
a
network
section.
We
specify
that
we
are
using
provider
as
multus.
Then
we
have
selectors
for
the
public
and
cluster
network,
so
the
public
is
pointing
to
rook
sap,
slash
rook
public
nw.
Here.
B
A
Thank
you
ron.
So
now
we
are
going
to
dive
a
little
bit
into
some
of
the
performance
analysis
that
we
that
we
run.
What
we're
really
trying
to
see
here
is
not
so
much
how
many
nodes
we
had,
what
how
many
cpus
we
had.
What's
what
was
the
amount
of
ram?
What
type
of
disks
we
we
had,
because
at
the
end
of
the
day,
this
doesn't
really
matter,
because
what
we're
really
trying
to
determine
here
is
what
is
the
performance
boost
that
we
can
get
by
adding
more
interfaces?
A
How
much
performance
can
I
get
if
I
add
more
interfaces,
because
right
from
like
from
the
beginning,
it
sounds
really
obvious,
like
yeah,
I'm
adding
more
network
interfaces,
I'm
getting
more
throughput,
so
I'm
going
to
get
more
performance,
but
what
is
like
in
in?
What's
that
dimension
at
what?
What
type
of
scale
are
you
going
to
get
more
performance
in
the
in
the
setup
that
we
had?
A
We
we
had
three
interfaces,
just
like
I
mentioned
earlier,
and
you
also
see
that
throughout
the
demo
we
have
one
interface
for
the
default
sdn
network
and
we
have
a
second
one
in
each
pod
for
the
public
network
as
well
as
a
third
one
for
private
network.
The
private
network
is
only
part
remember
of
the
osds,
because
they
are
the
ones
writing
reading,
replicating
backfilling
storage,
so
that's
the
they
all.
They
only
need
it.
Only
the
osds.
A
So,
what's
up
with
the
random
rights
in
terms
of
iops
and
bandwidth
like
how
what
type
of
boost
did
we
get,
we
kind
of
got
a
similar
boost.
So
if
we,
if
we
focus
for
a
moment
on
the
on
the
left
diagram,
the
random
writes
for
iops
per
node
with
and
without
voltage,
so
the
blue
line
is
with
multis
and
the
red
line
is
without
voltage.
A
A
So
again,
the
blue
line
is
with
multis
where
the
or
the
red
line
is
without
multiples,
and
we
can
see
that
really
adding
this,
like
those
extra
networks
to
ceph,
is
really
helping
when
running
client
ios,
and
we
can
see
that
this
additional
replication
network
for
dedicated
for
replication
is
really
helping
and
we
kind
of
see
a
22
gain
at
the
end
of
the
spectrum
as
well.
So
it's
really
something
like
I
mean
it's
a
really
good
performance
boost
in
terms
of
random,
read
again
for
iops
and
bandwidth.
A
So
it's
it's
always
difficult
to
just
consume
those
25
gigs.
And
then
we
have
reads
we're
just
served
from
memory.
Then
yeah,
you
don't
see
that
much
of
an
improvement
so
now
I'll
just
like.
Hang
it
over
again
to
rohan
who
is
going
to
walk
you
through
the
demo
and
a
cluster
that
we
have
set
up
for
you.
B
B
So
in
the
network
we
have
specified
the
provider
as
multus
and
the
cluster
network
is
set
to
default.
Slash
cluster
and
public
network
is
set
to
default,
slash
public.
So
when
I
am
specifying
here
default,
it
is
the
name
space
where
the
network
attachment
definition
exists.
So
the
cluster
network,
attachment
definition,
is
in
default
namespace.
So
we
use
it
to
use
it
here
like
this.
So
if
the
network
attachment
definition
was
in
rook
step,
so
we
would
have
written
rooks
appear
instead
of
default.
B
A
A
A
A
Using
the
ip
address
management
whereabout
is
preferred.
It
really
allows
you
to
have
a
smooth
deployment.
Experience,
get
a
unique
ip
address.
Of
course,
the
entire
cluster,
and
without
using
dhcp
service
at
all,
there
is
a
performance
benefit
from
separating
interfaces
than
just
using
a
single
one,
and
all
of
that
is
available
since
the
rook
release
1.7,
obviously
we'll
keep
on
adding
more
improvements
over
each
releases,
so
stay
connected.
A
So
well
thanks
everybody
for
for
your
kind
attention
if
you
want
to
reach
out
to
us.
These
are
our
email
addresses,
and
here
are
some
links
for
resources.
You
might
be
interested
like
all
the
projects
that
we
referenced
over
this
presentation.
Some
presentations
are
available
throughout
those
links,
so
thanks
again
and
enjoy
the
conference.