►
From YouTube: Dynamic multi-cluster management with Rook for cloud native IaaS providers for the private clouds
Description
Presented by: Joachim Kraftmayer
Over the last few years, we have been gaining experience with Rook in production. One of our challenges was to implement dynamic resource management between 50+ Ceph clusters. Kubernetes events dynamically and fully automatically distribute loads and capacity between Ceph clusters. This is done by removing single or multiple Ceph nodes from Ceph clusters while ensuring data integrity at all times. In the next step, the released Ceph nodes are integrated into other Ceph clusters as needed.
A
Okay,
first
I
have
a
short
introduction
for
myself.
You
have
some
I
mean
it's
the
first
time
for
me
in
the
U.S
since
since,
since
a
long
time,
I'm
more
focused
on
Europe,
so
I'm,
the
self
ambassador
of
the
dark
region
I'm
the
foundation.
Member
of
the
board
I
was
previously
Susie,
Enterprise
principal
consultant.
It's
also
I'm
working
for.
We
worked
for
Redhead
as
an
Enterprise
consultant
in
Europe
as
well.
A
A
Sorry,
is
this
better
now,
okay,
good,
so
the
company
is
founded
in
2010
focus
on
infrastructure
and
platform
as
a
service
based
in
in
Munich,
and
we
are
proud,
proud
to
be
the
member
of
lion,
Foundation
self
foundation
and
also
Cloud
native
Foundation,
so
Dynamic
multi-cloud
cluster
management,
I.
First
of
all,
I
want
to
give
some
key
facts,
because
this
product
already
started
in
2020
and
it's
all
and
it
will
I
think,
will
go
on
for
additional
two
years.
A
So
so
the
main
the
main
keys
are,
they
have
no
vendor
lock
and
open
source,
scalability
and
and
so
on.
Pure
IPv6.
Everything
is
in
this
project
is
kubernetes
driven
only
micro
services
are
the
the
workload
and
also
in
the
used
in
the
core,
and
it's
a
project
that's
coming
from
from
the
public
cloud,
and
it's
going
back
into
the
on-premise
private
Cloud
so,
and
it
has
a
moment
somehow
a
bit
more
than
200
petabytes
of
ram
footprint
in
production.
A
So
so
the
motivation
for
50
or
even
more
safe
clusters,
and
why
we
need
this
Dynamic
multi-class
management
solution
is
we
have
different
project
requirements,
so
we
have
exclusive
use
cases.
We
have
shared
use
cases,
we
have
failure
domains.
We
have.
We
have
different
availability
zones,
data
centers
regions,
we
have
security,
isolation,
I
mean
the
talk
before
we
have.
We
spend
a
lot
of
time
in
investigating
how
how
the
staff
CSI
is
doing
encryption.
A
How
how
Rook
is
doing
this
and
also
the
underlay
is-
is
completely
built
or
based
or
rely
on
on
Seth.
So
for
the
virtual
for
the
station
stack
for
for
out
of
band
management
for
all
the
services,
you
need
to
run
a
cloud
in
the
kubernetes
way
you
yeah,
we
have
so
so
much
different
requirements.
Even
this
nice
database
as
a
service
is
even
something
very,
very
good
talk,
I
think
at
least
an
hour
on
it.
A
So
when,
when
we
come
to
this,
to
the
point
that
yeah
that
the
project
or
even
the
user
requirements
constantly
changing,
even
I
think
we
talked
today
about
this-
that
the
car
can
a
client,
explain
you
what
kind
of
performance
and
what
kind
of
capacity
he
really
needs,
and
how
is
the
outlook
for
the
for
the
next
two
years?
It's
quite
hard
for
them
even
to
somehow
report.
A
Okay,
we
have
at
the
moment
this
IO
or
throughput
needs
and
that
will
grow
over
perhapsible
shrink
next
year,
even
when,
when
some
of
them
update
their
software
stack,
then
they
also
have
not
the
real
yeah
information.
How
this
will
impact
the
back
end
as
well,
and
so
out
of
out
of
this
it
comes.
It
comes
to
a
point
that
we
can
do
some
estimations
that
we
Acclaim
resources
for
specific
customers
for
specific
projects
for
even
for
the
internal
infrastructure
as
well,
but
we
normally,
we
are
not
never.
A
We
are
never
our
right,
it's
just
an
assumption,
and
then
we
have
to
adapt
it
afterwards
again
and
for
this
we
implemented
this
Dynamic
multi-cloud
multi-cluster
management
that
we
somehow
can
create
elastic
back
end
that
can
adapt
dynamically
to
the
demands
of
of
the
application
stack
above.
So
what
does
what?
Does
it
mean?
A
A
A
Yeah,
so
once
again
is
informing
us
what
staff
should
do
Seth
is
taking
care
about
what
how
it
should
train
safe
nodes
and
inform
all
the
kubernetes
that
it
will
be
possible
or
even
not
possible
to
to
drain
them,
and
then
it
will
give
it
back
and
kubernetes
takes
it
into
account
and
bring
it
to
the
next
cluster
again.
So
I
think
that's
not
on
this
slide,
but
I
will
explain
it
later,
but
with
an
example
in
the
upper.
So
we
don't.
We
also
changed
some
parts
of
Rook.
We
also
refactored
Rook.
A
At
a
certain
point.
We
also
said
okay,
who
is
responsible
for
what
kind
of
management,
and
that's
just
one
example
that
we
said.
Okay,
we
should
delegate
the
responsibility
responsibility
to
scaling
the
safe
monitors
back
to
kubernetes,
because
when
Rook
was
introduced
there
was
only
a
deployment
available
to
manage
replica
sets
and
I.
Think
two
years
later,
then
the
state
full
set
was
introduced
to
kubernetes,
and
this
one
can
take
over
or
can
guarantee
that
the
self-monitors
will
be
scaled
one
in
the
right
way.
A
A
So-
and
this
is
the
the
part
I
was
talking
previously
before
we
get
a
kubernetes
event,
adding
nodes.
We
get
we're,
adding
the
nodes
to
to
the
crush
root
staging
doing
the
the
OSD
test,
doing
doing
the
benchmarking
doing
the
classification,
because
we
have
different
fast
and
slow
device
or
Hardware
storage
classes
in
kubernetes
there
and
then
waiting
or
checking.
A
And
also
example,
how
this
is
implemented.
Is
that
how
you
remove
a
node
from
from
our
existing
root
cluster,
that
you
say,
Okay
I
have
the
oval.
The
kubernetes
controllers
told
you
okay,
we
need
a
fast
or
slow
note
from
this
cluster.
Please
free
it
up.
Then
Rook
is
verifying
the
the
user
capacity,
the
use
capacity
and
is
say:
okay,
we
have
enough
capacity
in
this
cluster.
We
can
give
it
back
to
to
you,
then,
the
next
step,
it's
very
to
the
verification
of
quality
of
service
requirements.
A
A
We
try
to
find
the
boundaries
of
the
existing
cluster
that
we
initially
deploy
and
then
we
can
somehow
create
an
estimation
how
many
RBD
images
of
this
specific
storage
class
is
100,
iops
or
200
iops
and
50
50
megabyte
of
throughput
we
can
deliver
and
and
when
we
remove
a
node
in
this
case,
then
we
also
have
to
reduce
the
the
images
that
we
can
deliver
with
a
specific
quality
of
service
requirement
in
the
next
step.
Also,
we
verify
verify
can
be
still
can
be
still
handle
the
failure
domain
across.
A
If
you
have,
for
example,
a
clustered
deployed
across
bootable
availability
zones
and
your
capacity
is,
you
could
remove
a
node
on
multiple
nodes,
capacitive
wise,
but
perhaps
you
cannot
even
compensate
a
failure
of
of
one
a
set
anymore.
Then
it
would
be
also
taken
into
account
in
this
process
and
perhaps
it
would
be
also
denied.
A
A
So
so
what
what
we
also
did
in
this
time
was
we
added
the
extended
Rook
that
kubernetes
is
more
Seth
is,
is
yeah?
It's
a
data,
awareness
or
I've,
always
always
said
leader
at
the
end,
who
makes
a
decision?
If,
if
you
can
how
you
how
you
can
manage
a
ceph
node
in
Rook,
we
also
removed
some
some
features
like
the
PG
Auto
scale
or
disabled
them.
A
What
you
also
did
we
extended
the
health
states
of
of
Seth?
At
this
point,
we
have
to
differentiate
between
data,
replication,
Delta,
Data,
health
and
and,
for
example,
the
the
status
of
the
service
we
have
to
we
have
to.
We
also
have
to
suppress
for
this.
Some
some
health
states
of
self,
for
just
a
simple
example
is,
is
the
crash
warning
that
we
have
to
suppress,
because
it's
not
Management
in
a
setup
like
this.
So
what
we
did
is
okay.
A
A
We
we
extended
also
Seth
for
Safari,
improved
with
this
additional
recovery
options,
so
that
we
also
can
interact
with
Mr
osds
if
there
is
really
a
problem-
and
we
also
extended
the
RBD
metadata
in
safe
itself
for
for
this
project,
so
that
you
can
store
different
metadata.
That's
not
that's
not
not
in
a
direct
context
of
Sephora,
even
RBD.
A
That's
an
example
for
for
the
metadata
is,
if
you
have,
if
you
want
to
have
a
router,
if
you
want
to
use
it
for
a
router
environment,
and
you
want
to
add
RBD
images
to
a
virtual
machine
and
you
want
to
have
it
always
in
the
again
in
the
right
order.
You
need
something
like
a
a
worldwide
name
that
that
you
really
can
identify.
Okay,
that's
that's
my
disc
at
this.
A
A
A
Three
new
regions
are
built.
They
are
building
up
at
the
moment
and
because
of
this
also,
the
data
center
fail
domain
is
still
in
progress.
So
and
also
now,
after
around
about
two
years
decision
was
made
that
we
don't
will
support
voter
machines.
It's
a
standard,
kubernetes
PB
PVC
stack
because
it
makes
the
handling
much
more
complicated
and
the
cefsys
I
is
also
somehow
a
limiter
to
use
all
the
set
features
that
are
really
available
and
I
think
with
the
smartnex.
A
A
The
one
idea
was
to
implement
or
to
directly
use
the
nvme
over
over
February
client
of
of
the
smartnic
and
directly
connected
to
to
the
project
of
surf
with
nvme
over
fabric
as
a
security,
isolation
and
also
and
also
height,
even
the
informations
of
of
the
ceph
client
for
for
clients
that
have
direct
access
on
on
the
hardware
as
well.
So
what
so?
A
What
we?
What
we
achieved
is
that
from
a
client
perspective
is
you
have
a
hardware
node,
you
see
disks
in
the
hardware
node,
but
but
in
reality
it
is
just
a
mapping
of
the
smart
Nick
on
on
this
Hardware
node,
where
you
have
yeah
an
RBD
client.
That's
this
is
this:
is
the
config
in
the
in
the
key?
That's
just
taking
care
of
this
as
well,
but
it's
it's
one
way
to
go,
but
it's
it's
not
not
decided
if
or
it's
not
100
production
ready,
good
and
yeah.
A
If
you
have
questions,
perhaps
I
did
a
little
bit
a
bit
wrong
at
the
beginning,
but
it's
also
quite
complex
project
at
the
moment.
It
and
I'm
involved
in
this
since
now
more
than
two
years,
and
it's
it's
really
really
hard
to
compensate
every
single
to
compress
this
in
in
in
30
minutes.
So
if,
if
you
have
any
questions,
if
then,
please.
A
Sorry,
maybe
I
missed
this.
You
said
you're
not
using
the
CSI
driver
for
the
utilizations
to
take
as
a
software
for
normal
kubernetes
cluster
so
for
client
kubernetes
cluster.
So
we
will,
we
will
use
it
or
we
are
using
it,
but
we
also
have
we
have
a
virtualization
stack,
that's
consuming
the
self
cluster
and
the
decision
I
think
was
made
yes
well
last
week
that
we
will
not
use
it
anymore,
that
we
have
to
re-implement
it
at
the
end.
A
A
We
are
talking
about
ten
thousands
of
football
machines
and
it
brings
no
no
benefit
for
the
underlay
that
you
handle
it
like
this
and
there's
also
I
mean
there
are
a
lot
of
other
services
as
well,
and
one
one
is
just
taking
care
already
about
the
quality
of
service
and
how
many,
how
many
images
are
allowed
to
to
be
created
on
one
thing
self
cluster,
and
this
can
take
also
over
the
part.
That's
normally.
This
fcsi
is
doing
I
mean
it's.
A
A
A
Yeah,
correct
yeah,
partially
I
mean
we.
We
have
to
remove
a
lot
of
stuff
of
this.
We
we
and
we
have
to
also
extend
our
services.
There
I
mean
that's,
not
not
the
main
part
of
of
us
as
a
storage
consultant,
it's
more
or
less
the
kubernetes
experts
who
who
do
the
decision
and
we
just
support
them,
how
how
they
should
do
it.
A
Yeah
I
think
they
they
try
to
to
stay
with
kubernetes
with
the
Holy
Stein
and
and
follow
all
of
this,
and
then
they
realized
after
a
while
that
it
is
not
sufficient
enough
and
we
have
we
have.
We
have
different
other
problems
as
well.
How
many
pots
you
can
deploy
on
one
on
one
server,
one
physical
server
to
get
rid
of
this
limitations
as
well,
but
it's
everything
more
we
need
is
related,
yeah
yeah.
A
A
A
A
As
an
example
yeah,
we
are
still
on
this
topic
with
safe
namespace
implementation
in
the
boot
cream
or
I.
Think
that's
open
since
three
years.
So
so
we
will
see
if
this
sorry,
it's
it's
not
I
mean
the
ticket,
for
this
is
I,
think
three
years
old.
Something
like
this
at
the
last
time,
when
I,
when
we
requested
or
when
we
asked
for
this,
then
they
just
ask
us
for
what
customer
what
could
be
the?
What
is
the
customer
who
wants
to
have
this
yeah.