►
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Your Manila CephFS Share Backups Belong to S3 - Robert Vasek, CERN
Backups. Boring and mundane, until you lose your application data and need it back. Our Kubernetes users at CERN make extensive use of CephFS-backed storage managed by the OpenStack Manila service. Streamlining and automating the process of backups gives them a chance to prepare so that–should a disaster strike–they can recover. In this talk we will delve into the work we have done to make Velero, CephFS, Manila and an S3 store cooperate together, and bring an application back into life. Expect code snippets and demos. By the end of this session, you should have a clear overview of how each component contributes to our current backup and restore workflow in Kubernetes, and how you can integrate this setup into your clusters too.
A
A
So
this
is
the
agenda
for
the
session.
First,
we'll
have
a
look
at
what
kind
of
storage
we
want
to
actually
backup
after
that,
we'll
briefly
describe
the
individual
components
right
after
that.
We'll
have
a
look
exactly
how
we
want
to
carry
out
the
backups.
What
is
the
workflow?
We
would
like
to
follow
after
that?
We'll
have
a
peek
into
the
future.
A
That
still
needs
to
be
done
because,
as
you'll
see
later
on
in
the
in
the
presentation,
there
are
still
couple
of
blocker
issues
that
we
are
facing
and
those
need
to
be
resolved
first.
So
I
really
take
this
whole
talk
more
as
a
project
update
rather
than
a
showcase
of
a
final
ready-to-use
product,
and
in
the
end
we
will
conclude
with
the
summary
so
quickly
about
cern
itself.
It's
the
european
organization
for
nuclear
research,
it's
located
in
geneva,
switzerland
at
the
swiss
french
border.
A
A
To
try
to
answer
those
questions
cern
has
built.
These
very
large
machines
called
particle
accelerators.
The
one
you
can
see
on
the
photo
here
is
the
largest
of
its
kind.
It's
called
lhc,
it's
of
a
circular
shape,
27
kilometers
in
circumference,
and
it's
placed
in
a
tunnel
100
meters
below
ground,
and
in
this
in
this
tunnel,
there
are
two
beams
of
protons
being
traveling
in
opposite
directions
and
being
accelerated
very
close
to
the
speed
of
light,
and
they
are
made
to
be
collided
with
one
another
at
at
another
kind
of
huge
machines.
A
This
is
this.
One
is
called
a
detector
that
is
then
able
to
monitor
the
aftermath
of
the
collision,
and
this
is
basically
what
the
physicists
then
study
and
analyze.
At
cern,
those
detectors
generate
huge
amounts
of
data
and
even
after
all,
the
filtering
is
done.
It's
still
tens
of
petabytes
per
year.
That
needs
to
be
stored
and
analyzed,
and
for
that
we
have.
A
We
are
running
private
cloud
based
on
openstack.
This
is
a
pretty
recent
screenshot
of
our
graphone
dashboard.
You
can
see
we
have
around
460
000
physical
cores,
87,
000
vms
running
on
almost
58
000
hypervisors.
We
are
also
running
around
360
kubernetes
clusters.
We
have
openshift
as
well,
but
kubernetes
leads
by
far
in
in
the
numbers
for
storage.
We
have
those
kinds
of
numbers
block
devices
around
3.8,
petabytes
file,
shares
almost
900
era
and
object
storage
at
around
48
terabytes.
A
A
So
this
is
a
lot
of
data,
but
for
the
purposes
of
this
presentation
we
will
only
focus
on
the
file
shares.
So
that's
around
900
terabytes,
but
not
all
of
this
amount
needs
to
be
backed
up,
because
pretty
big
chunks
of
this
amount
is
used
by
ci
by
qa
by
testing
environments
and
development
environments.
So
we
don't
really
need
to
backup
those.
A
So
we
did
some
accounting
and
by
some
estimates
we
have
around
65
projects
that
are
deploying
159
production,
kubernetes
clusters
and
those
then
store
around
74
terabytes
of
manual
sfs
storage.
So
that's
basically
the
kind
of
numbers
we
are
looking
at.
So
I
mentioned
manula
sfs
shares
it's
in
the
title
of
the
presentation
as
well.
So
what
exactly
are
those
suse
is
scalable
distributed
storage
system
that
we
rely
on
heavily
it's?
A
It
offers
three
interfaces
for
this
storage
in
the
same
package,
object
store,
called
blog
devices,
rpds
raiders
blog
devices,
and
then
shared
file
systems
called
cfs,
and
this
is
the
last
one
is
what
we
are
focusing
on
right
now.
Then
we
have
the
openstack
manual,
which
is
shared
file
system
service
for
openstack,
and
it's
the
is
the
point
of
where,
basically,
the
shared
storage
systems
are
able
to
interact
with
the
openstack
cloud.
A
A
It
also
provides
multi-tenancy
quota
management,
all
very
important
features
that
we
rely
on
then.
Lastly,
we
have
csi
it's
the
container
storage
interface
and
it
forms
this
well,
it's
an
interface
between
some
storage
system
and
the
container
orchestrator,
for
example,
kubernetes,
and
it
allows
storage
vendors
to
write
their
own
storage
drivers
and
those
drivers
are
called
csi
drivers
and
basically,
it
forms
like
a
middleman
between
the
storage
and
the
orchestrator.
A
A
A
What
sort
of
information
is
needed
to
mounted
share
in
case
of
surface?
This
is
the
monitor,
ips,
the
root
part
of
the
volume
and
cfx
credentials,
and
those
are
then
forwarded
to
this
ffs
csi
node
plugin,
which
is
a
completely
separate
csi
driver,
and
it
has
the
tool
the
tools
needed
to
actually
carry
out
the
mount
on
the
node
and
then
expose
it
to
the
consumer
port.
A
A
So
it
consists
of
six
steps
crossing
the
application,
creating
a
snapshot
unquestioning,
creating
a
volume
from
the
snapshot
then
backing
up
this
intermediate
volume
and
removing
removing
it,
along
with
the
snapshot.
So,
let's
break
down
those
steps,
one
by
one
classic.
The
application
serves
two
purposes.
First,
one
is
that
it
stops
or
pauses
the
application
for
from
processing
any
further
requests.
A
So
it
needs
to
flush
those
caches,
and
then
you
can
take
a
snapshot
of
the
volume.
Otherwise
you
could
get
inconsistent
data.
This
is,
of
course,
very
application.
Specific
and
not
all
apps
need
that
some
applications,
don't
some
applications
would
even
consider
this
disruptive
because
they
might
prefer
availability
and
not
being
posed
rather
than
having
consistent
state
on
the
disk
being
snapshotted.
A
So
we
have
created
a
snapshot.
Now
then
step
number
three
unquestioning
the
application
means
resuming
resuming
it
and
making
it
available
to
process
the
requests
again
and
now
we
can
actually
think
about
backing
up
the
data
that
we
have
just
snapshoted.
The
problem,
however,
is
that
you
can't
really
backup
a
snapshot.
A
A
A
A
A
A
A
A
A
A
And
well,
because
of
a
bug
or
incorrect
handling
of
of
snapshots
names,
we
are
basically
unable
to
access
snapshot
data
within
this
dot,
snap
directory,
so
we
are
sort
of
stuck
at
this
point.
There
are
likely
already
patches
for
for
this
issue
upstream
and
it's
big
being
worked
on,
but
right
now
my
nose
is.
A
I
cannot
implement
this
feature,
but
as
soon
as
this
is
done,
we
can
move
to
the
next
step,
which
is
then
choosing
the
the
correct
backup
tool
to
to
carry
out
the
backup
procedures,
because
what
we
are
doing
is
implementing
the
functionality
needed
by
the
csi
drivers.
The
other
part
of
the
equation
is
to
actually
have
some
some
tool
to
actually
execute
the
workflow
of
of
the
backup
and
that's
one
of
the
tools
you
can
see
up
here.
A
A
A
So
the
rule
is
basically
that
if
the
back-end
storage
system
is
able
to
upload
the
data
to
the
backup
location
by
itself
without
by
itself,
and
in
the
background
without
you
having
to
deploy
some
sort
of
external
tool
within
in
your
kubernetes
cluster,
that
does
the
copying
for
you.
So
as
long
as
you
don't
need
this
tool,
you're
fine-
and
this
is
the
case
for
ebs.
Google,
press,
10,
disks,
azure,
managed
disks
and
similar.
A
A
A
It's
around
8,
gigs,
and
but
still
this
is
expected
because
rustic
needs
to
do
the
deduplication,
and
that
takes
a
lot
of
memory
to
keep
the
indices
around.
So
it's
understandable,
but
the
problem
is
that
the
node
on
which
the
rustic
pot
is
running
might
not
be
large
enough
to
carry
out
the
backup
operation
in
at
which
point,
or
it
just
runs
out
of
memory.
It
is
killed
which
brings
us
to
the
next
issue
that
failed.
Backups
stay
failed.
There
are
no
retries.
A
So
when
a
when
a
rustic
pod
goes
out
of
memory,
it
is
killed
and
then
it
is
restarted
because
valero
deploys
it
as
a
demand
set.
So
it
is
restarted
right,
like
any
other
demand,
support
would
be
and
you
would
expect
that
rustic
would
be
able
to
continue
where
it
left
off
before
it
was
killed
due
to
running
out
of
memory.
But
this
is
not
what
we
saw.
A
What
we
observed
was
basically
it
just
got
stuck
until
the
backup
action
failed,
backup
action
timed
out,
at
which
point
the
valero
controller,
just
move
on
to
the
next
backup
item,
and
this
all
was
done
silently.
We
didn't
see
any
errors
only
after
the
whole
backup
job
finished
and
and
that's
when
we
could
see
the
the
issue,
but.
A
A
A
A
And
from
what
I
could
see,
they
are
planning
to
improving
their
csi
snap,
shooting
capabilities,
hopefully
also
providing
support
for
the
backup.
Workflow
we've
seen
a
couple
of
slides
earlier
after
that
they
are
considering
adding
alternatives
to
rustic.
For
example,
copia
and
I'd
like
to
point
out
that
rustic
is
by
no
means
like
a
bad
tool.
Our
theft
team
uses
it
uses
it
in
their
internal
operations
on
millions
of
files,
and
it
works
just
fine.
It's
just.
A
Those
are
very
much
just
preliminary
benchmarks
on
a
small
data
set
in
very
controlled
conditions,
so
not
really
something
to
be
taken
very
seriously,
but
it
gives
an
idea
of
what
things
might
look
like
once
the
once
valero
receives
this
copia
support.
So
what
we
had
was
a
volume
with
about
1.5
million
files.
This
was
just
uncompressed
copies
of
the
linux
kernel
and
it
was
the
same
linux
kernel,
so
majority
of
the
files
were
well.
A
A
Yet,
but
it's
being
addressed
so
if
you
need
back
snapshot,
support
as
well,
you
you
have
to
wait,
and
so
do
we,
if
you
don't
and
your
application
is
not
that
sensitive
to
have
in
maybe
inconsistent
state
on
the
on
the
disk
and
that
being
backed
up
then
you're
good
to
go,
but
also
being
mindful
of
the
limitations
that
we
have
experienced
so
pretty
large
memory,
consumption
and
scaling
issues,
but
in
the
end
they
are
all
being
addressed.
So
we
are
looking
forward
to
that
all
right.