►
From YouTube: Hands On with Rook: Ceph & Kubernetes - Maxime Guyot, Root Pi & John Studarus, Packet Host
Description
Hands On with Rook: Ceph & Kubernetes - Maxime Guyot, Root Pi & John Studarus, Packet Host
This is a hands-on tutorial walking through the use of Cephia via Rook, a storage orchestration service for Kubernetes. Each attendee will be provided a deployed Kubernetes cluster on bare metal and will walk through setting up Ceph via Rook across the bare metal SSD resources and how that storage is presented to Kubernetes clusters. We will then run through some scaling up/down of the underlying storage infrastructure as well as failing storage devices to showcase recovery.
About John Studarus
JHL Consulting
Cloud Architect
Greater San Diego Area
Website
Twitter Tweet LinkedIn Connect
For the last twenty years, John has been providing technical management services building and evaluating complex distributed systems across the telecommunications, pharmaceutical, and financial services industries. He's a graduate of the University of California, San Diego and Carnegie Mellon University.
Recently John has been developing the software ecosystem to support applications running at tower based edge locations. This has revolved around testing and modifying cloud and container-based open source software, such as Kubernetes and OpenStack, to easily deploy and utilize the bare metal compute, network, and wireless infrastructure across these edges.
John runs a number of CNCF and Open Infrastructure meetup groups across Southern California, volunteers as an Ambassador for the OpenStack Foundation and serves on the Carnegie Mellon Information Networking Institute (INI) alumni board.
About Maxime Guyot
Root Pi
Cloud Consultant
Maxime is a cloud architect and engineer passionated with IT and open source technologies. He specializes in Software Defined Infrastructure using Openstack, Kubernetes, and Ceph. He's a contributor to Open Source projects such as Kubespray and Service Catalog. For fun, he likes to build CI systems across as many public clouds as he can get accounts.
A
B
A
A
You're
gonna
have
access
to
two
physical
machines,
one's
a
slightly
smaller
machine
that
just
has
a
routes,
file
system
and
another
one.
That's
got
a
root
file
system,
along
with
a
number
of
nvme
SSD
drives
attached
to
it.
We're
going
to
walk
you
through
everything's
available
up
on
github
you're
welcome
to
go!
A
Take
the
repo
you
know
clone
its
welcome
pr's,
and
but
there
are
a
number
of
steps
you're
going
to
go
through
we're
going
to
do
everything
from
scale
up
scaling
up
an
environment,
work
environments,
doing
the
basics
of
installing
Brooke
we're
gonna
do
an
upgrade
of
Ceph
within
the
root
ruck
environment.
We're
just
going
to
do
some
failures.
We're
gonna,
create
some
volume
claims
we're
going
to
actually
start
up
an
application
as
well.
So
we
got
quite
a
bit
to
do
or
even
gonna
do
some
Prometheus
and
crow
fauna.
A
So
you
can
monitor
the
environment.
So
we've
got
a
bunch
to
do
like
I
said.
If
you
don't
have
a
laptop,
no
problem,
we're
going
to
everything
we're
gonna.
Do
you
me
going
to
be
able
to
watch
up
on
the
screen?
If
you
want
to
run
this
later,
please
let
us
know
happy
to
leave
the
instruction
infrastructure
going
for
you.
First
step
is
up
on
etherpad
here
the
pad
dots,
F
comm,
slash,
P,
slash,
rook
workshop,
that's
the
starting
spot!
So
start
there.
A
A
Just
come
back
markdown
that
you
broke,
that
one
will
just
rebuild
it
and
then
put
your
name
down
by
another
environment
like
I,
said:
I,
think
we
got
eighty
environments
going
so
there's
there's
plenty
to
go
around
and
if
you
get
yourself
in
a
bind,
you
can't
forget
how
to
undo
it
just
move
forward.
So
with
that,
hopefully
everyone's
got
a
lab
environment.
We're
gonna
go
through
and
start
with
step
one
yeah.
It
looks.
A
To
the
github
and
follow
through,
if
you
got
any
questions,
I'm
gonna
be
walking
around,
raise
your
hand
I'll
come
by.
Tell
me
your
question.
I'll,
probably
repeat
the
question
out
loud,
so
we
got
to
record
it
as
well.
Most
of
the
time.
If
people
run
into
a
question,
everyone
else
has
got
the
same
question
as
well,
so
I
might
as
well
repeat
it
out
loud
and
we'll
go
from
there.
Okay,
without
any
further
ado.
Alright.
D
Let's,
let's
get
started
so
I'll
start
with
the
first
lab,
it's
kind
of
a
presentation
of
the
environment.
So,
like
John
said,
we
have
two
VMs
or
two
physical
servers,
one,
that's
what
we
call
a
t1,
small
and
other
one.
That's
a
c2
medium.
The
first
one
just
has
a
root
root.
Filesystem
and
the
second
one
has
some
misses
these
available
that
we
will
later
on
use
for
blue
store,
OSD
devices
kind
of
setup
of
how
you
access
the
environment.
Is
we
have
those
lab
masters?
D
You
find
the
IP
of
your
lab
master
in
the
inter
pad.
So
that's
this
IP
over
there
and
from
there
you
have
the
SSH
keys
and
IP
addresses
and
all
the
SSH
configs
to
access
your
your
nodes
that
are
specific
to
your
yo
lab.
So
for
everybody
they
are
called
node
1
and
node
2,
let's
configure
in
your
SSH
config
with
the
correct
IP.
So
you
just
need
to
do
association
or
one
in
your.
You
know
you
can
access.
So,
let's
get
started
and
ssh
to
the
lock
master.
I
took
lab
number
one.
D
A
D
Right
so,
let's
verify
that
each
end,
looking
into
your
your
lab
master
and
then
from
there
well
verify
the
unstable
inventory
like
to
have
a
look
at
what
the
machines
are.
So
we
have
this
command
over
here
on
civil
inventory
list,
so
we
can
see
we
have
node
1
and
node
2
that
show
up
in
to
our
on
simple
inventory.
We
use
dense,
able
to
provision
all
the
labs
right.
D
A
D
Right
so
before
we
move
to
the
rooks
specific
things,
we're
just
gonna
do
some
verification
of
the
equipment
setup.
So
we
have
set
up
already
communities
on
those
two
machines
and
what
we
will
do
in
the
labor
labs.
We
will
set
up
a
rock
on
top,
so
phonon
will
just
verify
that
kubernetes
is
behaving
correctly
and
all
that.
So
we
have
those
comments
here
that
will
install
nginx
deployment
into
into
kubernetes.
So
we
can
just
copy
paste,
keep
CTL
apply,
which
will
download
the
config
file
and
create
a
nginx
deployment.
D
Then
we'll
do
a
cube,
CTL
expose,
which
will
create
a
community
service
which
is
the
way
to
do
the
service
discovery
within
kubernetes
and
from
that
point
on,
we
can
do
cube.
City
Li
get
pods,
we
can
see,
we
have
our
nginx
deployment
pods
that
are
up.
We
have
two
of
them
and
they
are
both
in
ready
state.
So
1/1
means
they
are
ready
and
they
are
running
right
now.
So
that's
good
news
means
our
community
is
working
just
fine
there.
D
Then,
let's
verify
the
networking
so
we'll
keep
city
I'll
get
services
to
find
the
IP
to
control
C.
First
find
the
IP
the
cluster
IP
of
our
nginx
deployment
nginx
service.
So
we
have
the
IP
there.
We
can
do
a
curl
of
that
IP
and
then
we
get
some
HTML
blob
back.
So
we
have
the
load
balancing
working
within
communities.
D
Then
we'll
just
run
the
delete
commands
to
clean
things
up,
since
we
don't
need
this
nginx
for
work.
This
is
just
to
test
things
right.
So
let's
delete
this
as
well.
Now
we'll
keep
CTL
get
pods
to
verify
that
things
are
deleted.
So
now
it's
terminating
state
and
now
it's
gone.
Okay
and
I'll
verify
to
the
service.
It's
gone
as
well,
so
you
can
see
that
the
nginx
deployment
service
is
gone,
but
we
still
have
the
kubernetes
service,
which
is
the
internal
service
discovery
service
for
accessing
the
communities
API.
So
this
is
normal.
A
We
had
one
question
of,
and
I
run
this
on
my
own
cluster.
Yes,
all
these
instructions
are
gonna,
be
on
github
you're.
Welcome
to
run
on
your
own
infrastructure.
For
today
we
recommend
using
our
infrastructure
because
it's
set
up
with
multiple
drives,
so
we
can
run
multiple
sd's
and
your
cluster
might
not
have
as
large
environment
as
we're
making
available
for
you
today
so
use
the
infrastructure
we've
given
you
for
today,
and
then
you're
welcome
at
your
own
leisure
at
another
time
to
to
fall
through
these
instructions
on
your
own
infrastructure,
yeah.
D
If
you
do
this
on
your
own
cluster,
your
drive
layout
may
be
a
bit
different.
Here.
We
have
a
mix
of
file
store
for
the
first
server
and
bluestone
for
the
second
server.
So
then
you
will
have
to
like
adapt
to
your
environment
and
your
servers
alright.
So
now
that
we
have
verified
that
Trinities
is
all
good,
we'll
start
to
deploy
Brooke.
So
this
is
the
kubernetes
like
the
recuperator
4sf
and
that's
just
two
commands.
D
We
have
the
first
one
here
that
will
install
the
common
components
of
roofs
things
like
role-based
or
back
policies,
and
things
like
this
to
make
sure
that
the
rook
has
the
appropriate
permissions
to
to
do
its
job,
and
then
we
have
the
second
command,
which
creates
the
the
rook
pod
and
then
will
be
able
to
do
a
cube.
Ctl
get
pods
watch
and
we
will
see
that
currently,
our
rook
operator
pod
is
a
container
creating
state.
D
Now
it's
running
it's
starting
to
create
other
pods,
and
things
are
starting
to
do
to
happen
in
to
bootstrap
there.
So,
let's
see
and
wait
for
things
to
be
fully
running,
we
need
everything
to
be
in
ready.
1/1
state
and
in
running
status,
so
that
means
application
is
ready
to
go,
takes
a
few
seconds
or
a
few
minutes
to
download
the
containers
and
stop
them
making
sure
they
are
healthy
and
ready
all
right.
D
We
can
verify
that
the
deployment
is
up,
so
this
is
the
deployment
resource
in
activities
ready
state.
We
see
one
instances
up-to-date
and
one
is
available,
which
is
what
we
expect
all
right.
Good,
let's
go
on
to
the
next
next
lab
and
now
we'll
set
up
a
safe
cluster.
So
ruk
is
like
the
control
plane
to
be
able
to
create
safe
clusters,
which
we
did
in
just
now,
and
I
will
create
our
safe
cluster.
D
On
top,
we
are
starting
with
a
safe
cluster
that
uses
just
the
node
1,
so
it
will
be
a
single
node,
safe
cluster
and
in
a
later
lab
we
will
expand
our
single
node
safe
cluster,
to
use
the
second
node
so
just
to
explain
a
little
bit
of
context.
We
have
this
come
in
here,
cube,
CTL,
create
which
will
create
a
safe
cluster
object,
and
then
we
can
do
the
watch
command
to
see
what
ruk
is
going
to
do.
So
we
have
our
at
the
top.
Here
we
have
our
safe
cluster
object.
D
D
D
Shell
old,
probably
both
of
them
all
right.
Let's
see
this
is
easy
to
increase,
and
this
is
that
that's
bigger
all
right
all
right.
So
now
you
have
the
first
one
up.
The
second
one
is
created
as
well,
so
we
can
see
them
they're,
creating
this
third
month
after
that
is
gonna,
create
the
manager
pods.
D
D
And
we
know
that
Luke
is
done,
creating
this
cluster
once
the
state
goes
from
creating
to
created
and
that's
the
way
a
rook
will
tell
us
that
okay
I'm
done
with
this-
and
this
is
ready
to
go
so
I'm
just
waiting
for
that
to
happen.
Basically
right,
so
rook
prepared
or
discovered
that
there
is
one
good
good
device,
that's
available
on
the
node
1
and
we
have
configured
it
to
use
file
store
for
that.
D
So
it's
popping
up
always
d0
and
using
that
as
the
first
ways
D
for
the
environment
and
now
that
cluster
is
created,
so
that's
good
and
we
can
see
how
foci
as
well.
So
we
are
ready
to
go
we'll
examine
the
the
cluster.
So
this
is
like
the
parameters
we
have
set
into
the
environment.
There.
Yes,
perfect,
created
health.
Ok,
we
can
do
a
LS
of
Varley
Baruch.
This
is
the
location
where
rooks
stores
some
persistent
data,
so
that,
if
you
would
reboot
your
server,
the
data
would
still
be
there.
D
You
have
things
like
keyring
information
in
there,
save
that
conf
and
all
those
things
are
and
are
there
and
all
persisted
on
a
specific
server
that
hosts
the
mons,
the
ways
DS
and
all
that
all
right.
So
are
we
all
that
created
static
credit
state
health,
ok,
yeah
I
see
some
nods
good,
wait.
Let's
not
lock
the
screen,
so
I'll
move
over
to
lab
12.
D
So,
as
you
can
see
so
far,
we
did
a
lot
of
cube,
CTL
commands,
so
that's
the
kubernetes
control
command
and
so
how
we
mainly
interact
with
rook.
But
if
you
want
to
run
some
say
standard
or
be
de
or
safe
comments
or
a
dos
commands
you're
going
to
need
a
container
for
that,
that's
what
we
call
the
tool
box.
So
we
have
a
comment
for
that
and
we'll
just
do
that.
D
It's
gonna
create
a
new
container
that
includes
all
the
safe
tools
of
CLI
tools
like
our
or
videos
and
all
that
and
has
appropriate
permission
to
run
any
command
in
yourself
cluster.
So
this
is
more
for
debugging
purposes
of
all
labs
like
this,
it's
nice
to
be
able
to
run
whatever
you
want,
but
in
production,
maybe
it's
better
to
not
keep
this
toolbox
container
around.
Maybe
you
want
to
remove
it
and
just
create
it
when
you
need
it.
D
So,
let's
verify
that
that
container
is
ready.
Yes,
one
one
ready
running,
then
we'll
use
this
command
to
get
a
terminal
into
that
container.
So
we
are
doing
a
cube,
CTL
exact
in
interactive
mode,
to
be
able
to
get
a
shirt
a
shell
there
and
and
to
avoid
confusion,
we're
just
gonna
change
the
prompt
from
the
custom
prompt
to
this
safe
toolbox.
D
That
way,
we
know
that
okay,
here
we're
in
the
toolbox
and
we
can
differentiate
things
so
I'm
gonna
open
another
window
so
that
I
can
go
back
to
the
toolbox
easily
later
on
and
switch
between
the
node
one,
where
we
have
the
cube
CTR
access
and
the
toolbox
where
we
have
the
staff
access
rain.
So
I'll
just
create
a
new
tab
for
that.
D
So
here,
yes,
we
can
run
safe
status.
We
see
health,
okay
as
well.
Three
months
called
ABC.
We
see
that
there
is
one
manager,
one
OSD
that
is
up
and
we
have
100
gigs
of
row,
more
less
storage,
available.
I
think
this
first
server
is
like
hundred
20
gigs
good
device.
We
can
do
safe
with
this
status
as
well.
D
We
just
have
one
OSD,
so
it's
not
super
exciting,
so
mode
one
and
there
is
20
gigabyte
used
since
we're
using
the
file
store
back
in
there.
It's
counting
the
operating
system
data,
as
used
in
the
staff
as
well
the
SF
DF
about
those
DF
and
all
the
comments
you
might
be
familiar
with
on
self.
You
can
run
there
alright,
so.
D
So
now
that
we
have
our
SEF
cluster
up,
we
are
gonna,
try
to
use
it.
So
let's
create
a
volume
into
it
and
for
this
in
kubernetes,
it's
well
you
before
creating
a
volume,
we're
gonna
need
to
create
a
cool
first,
because
now
we
don't
have
a
pool.
If
we
go
to
the
toolbox,
we
could
do
those
endless
pools.
There
is
no
pool.
We're
gonna
need
a
cool
phone
to
continue.
So
this
is
what
we're
going
to
do.
D
Next,
we
create,
what's
in
a
root
terminology,
is
called
a
safe
block
pool
and
that
we
have
configured
as
replicated
type
with
a
replication
factor.
One
for
now
we'll
also
create
a
cube,
Aneta
storage
class
and
that's
necessary
for
communities
to
be
able
to
use
our
storage
later
on
and
the
orbit
drive
and
go
ahead.
So.
A
We
had
one
set
of
ego
eyes
that
said:
hey
wait!
A
second
I,
don't
see
the
same
number
of
containers
running
on
my
environment,
all
right!
Keep
in
mind
that
there
are
two
notes
in
this
cluster,
so
the
scheduler
might
take
some
of
the
bonds
and
put
it
on
the
second
node
down
as
a
post,
so
on
the
first
node.
So
that
way
beep.
Why
am
I
seeing
slightly
different
deployments
than
what
you're
seeing
up
on
the
screen
here?
A
D
Right
so
now,
I'll
go
back
to
the
toolbox
container
and
do
rio
celeste
pools,
and
we
see
that
now
we
have
a
pool
created
so
humanity,
communities
and
rooked
when
to
talk
to
to
staff
and
say:
hey
and
you
look
cool,
replicate
a
pool
size
one.
So
this
is
what
we
have
done
now:
let's
create
a
volume
in
that
pool,
so
we'll
do
what's
called
a
purses
and
volume
claim
which
will
create
a
persistent
volume
which
will
create
a
safe
volume
or
be
the
image
into
that
specific
pool.
D
We
have
just
created
all
right,
so
we
have
that
there,
let's
do
our
BD
l,
s-
p.
So,
let's
list
all
the
urban
images
into
a
given
pool.
So
we
go
back
to
the
toolbox.
For
that
and
now
we
see
we
have
PVC
created
our
volume
created
with
this
name,
PVC
and
and
some
ID
I
think
we
can
do
chips
each
and
get
PG,
for
instance,
and
we
see
the
same
thing
from
the
kubernetes
point
of
view
a
list,
so
two
gigabyte
volume
we've
just
created
right.
D
See
all
right
and
I
see
like
here
says:
you
can
also
verify
in
a
safe
large
board,
but
the
dashboard
is
in
the
next
next
lab.
So
you
can
ignore
that
step
and
welcome
back
to
that
once
we
install
all
that
dashboard
all
right.
So
now,
we've
done
some
CLI
things
and
the
toolbox
is
nice,
but
you
already
want
to
to
have
a
look
at
the
safe
dashboard.
It's
new,
ish
I
guess
it's
kind
of
useful
to
have
a
visual
view
of
things.
So,
let's.
D
So
we
are
going
to
need
to
expose
the
dashboard,
so
we
can
access
it,
otherwise,
by
default
it's
accessible
only
from
within
the
cluster
and
from
your
laptop
you're
outside
the
cluster.
So
we
need
to
do
the
cuba
native
service
with
a
type
node
port.
So
this
is
what
we've
done
in
this
in
this
lab.
But
you
know:
production,
environment,
you're,
gonna
want
to
enable
HTTPS,
have
some
SSL
certificates
and
all
that
stuff.
So
not
port
might
not
be
the
best
choice
for
production
scenario,
but
for
a
lab,
it's
good
enough.
D
So
this
is
what
we're
going
to
do
another
one
and
then
we'll
we
have
this
snippet
that
just
fetches
the
IP
and
the
node
port,
which
is
generated
randomly
and
tells
you
where
you
should
connect
all
right.
So
have
this
address
and
just
copy
paste
that
into
my
browser,
I
have
an
HTTPS
warning.
Like
I
said
this
is
self
signed
certificate
so
now
do
let's
encrypt
or
whatever
is
a
provider
you
choose
in
production
and
we
should
arrive
on
our
own
sub
dashboard
brilliant.
D
This
is
that
14.
Well,
it
asks
us
to
login.
The
username
is
admin
and
the
password
is
randomly
generated
for
each
lab,
so
you
need
to
run
a
specific
commands.
The
air,
that's
gonna,
fetch
the
random
password
that
you
have
in
your
lab
and
it
will
be
different
for
everybody.
So
just
keep
me
log
in
all
right.
So
now,
I'm
in
the
dashboard-
and
this
is
the
standard
session
dashboard-
think
was
luminous
ish
released
and
we
can
see,
we
have
three
monitors.
One
is
the
up.
D
D
Yes,
we'll
just
keep
the
dashboard
window
open
throughout
the
workshop,
because
it
can
be
nice
if
you
want
to
check
things
see
what's
going
on
later
on,
we
are
creating
some
applications
and
all
that
stuff,
it's
nice
to
have
the
dodge
ball
around
and
since
the
URL
is
not
so
easy
to
remember,
just
keep
the
tab
open.
Now
that
we
have
things
up,
we
can
just
we'll
start
to
create
an
application
and
we
took
a
wordpress
as
an
example.
So
it's
a
wordpress
kind
of
blog
with
a
my
sequel
database.
D
Wordpress
has
a
volume,
my
sequel
has
another
volume
and
they
get
provision
on
kubernetes
and
we
have
staff
running
in
the
same
cluster.
So
it's
kind
of
a
hyper-converged
example.
In
a
in
your
scenario,
you
might
have
some
some
notes
of
duty,
dedicated
to
storage
and
some
notes
that
are
dedicated
to
run
containers.
But
here
we
just
have
two
nodes,
so
we'll
download
the
manifests
firsts.
D
D
D
I
can
also
show
the
volume
mount
aspect
all
right
and
then
the
volume
gets
mounted
involve
our
HTML.
So
this
is
where
the
20
gigabyte
William
ends
up
being
and
for
my
sequel,
it's
20
gigabyte,
William
as
well.
You
can
see
it
here
that
then
gets
mounted
into
volume,
my
sequel,
so
this
is
very
sim
standard
things.
Let's
create
our
my
sequel,
pods
all
deployments,
then
we
will
create
our
WordPress
one.
We
can
keep
an
eye
on
the
the
persistent
volumes.
D
It
was
a
bit
too
too
slow,
but
if
you
are
fast
enough,
you
can
see
it
going
from
pending
state
to
bound
state,
so
pending
state
is
when
it's
just
created
but
not
created
in
the
root
but
not
created
in
safe.
Yet,
but
now
it
has
been
created
in
self
very
quickly,
so
everything
is
ready
to
go.
I
can
refresh
the
report,
I,
don't
even
that's
what
the
volumes
they
show
up
immediately.
All
right
now,
we'll
check
our
pods.
D
So
we
have
one
pod
for
WordPress,
one
pod
for
my
sequel,
both
ready
and
running,
so
they
have
their
volume
attached
already
and
we're
ready
to
go.
We
can
check
the
deployments
we
created
so
again,
one
deployment
for
WordPress
one
for
my
sequel.
This
is
not
an
AGA
type
example,
but
so
you
have
one
pod
for
each
deployment.
D
Then
we
can
go
with
the
services,
so
we
have
one
service
for
our
communities
like
I
mentioned
before
this
is
the
communities
API?
This
is
not
something
we
have
created
in
this
example.
This
is
just
part
of
how
humanities
works.
Then
we
have
one
service
for
the
WordPress
web
app
and
another
service
for
the
my
sequel
database.
So
the
WordPress
web
app
will
talk
to
the
my
sequel,
service
and
I.
D
D
D
Access
the
application-
yes,
so
everything
looks
good
on
the
WordPress
deployment.
Let's
try
to
access
our
WordPress
deployment,
so
we
have
those
commands
that
will
find
the
IP
and
the
specific
IP
in
a
specific
pod,
where
your
WordPress
application
is
available.
You
paste
that
into
your
web
browser-
and
in
here
you
are,
you
are
into
WordPress
safe
on
something.
D
D
D
D
D
So
basically,
from
the
applications
perspective,
the
OSD
went
down
and
then
rook
detected
that
something
went
wrong
with
that
OSD
and
we
started
the
container
because,
while
it's
supposed
to
run
right,
this
is
a
pot
that
is
supposed
to
run.
This
is
how
route
configures
those
things,
so
it
monitors
that
they
are
running
and
if
they
crash
like
for
whatever
reason
somebody
runs
a
pickle
by
accident,
for
instance,
rook
tech
on
humanities
and
rook
take
care
of
restarting
it
and
bring
it
back
to
the
expected
state
which
is
running.
D
So
we
might
have
our
tool
box
over
here,
safe
status
for
all
this
make
up
and
we
are
still
at
health.
Ok,
even
though
we
lost
OSD,
and
it
came
back
up
what
one
second
later.
So
this
shows
a
bit
the
benefits
of
using
rook
and
and
all
that,
otherwise
you
would
have
maybe
system
D
or
whatever
process
manager.
You
have
handling
those
things
all
right.
So
now
we'll
do
another
type
of
failure.
It's
let's
imagine
that
we
will.
D
We
have
an
application-level
failure,
so,
like
a
database
trailer,
we
will
just
delete
the
my
sequel
pod
with
this
command
so
we'll
tail
to
Trinity.
Is
this
very
spot
that
we
are
created
from
a
sequel,
let's
delete
it
and,
let's
see
what
happens,
all
right
says
what
press
my
sequel
being
deleted.
The
port
goes
to
terminate,
terminate
terminating
state
and
then
humanities
creates
another
one.
So
it
shows
a
little
bit
the
benefits
of
of
resiliency
for
the
application
as
well,
not
just
for
surf
surfboards.
A
D
D
So
if
that
pod
or
container
gets
scheduled
on
to
some
other
server,
because
it's
more
appropriate
at
that
point
in
time
to
a
schedule
8
somewhere
else
into
the
cluster,
then
it
will
detach
the
volume
or
really
volume
from
that
first
server
and
then
attach
it
to
the
other
location.
So
this
is
really
like
fully
automated
they're
kind
of
like
what
you
would
expect
in
a
cloud
provider
with
block
storage
services
instance.
So
this
is
really
what's
happening
behind
the
scene.
D
We
can
do
another
one,
if
suppose
I
don't
know
if
it
will
move
to
a
node
1,
but
not
two
as
well.
All
right
so
note
2
is
much
bigger
and
not
one.
So
that's
probably
why
it's
sticking
to
not
do
it
doesn't
want
to
good
another
one?
Yet,
all
right,
so
everybody
got
that
one
working
with
a
Mexico.
Coming
back.
All
right,
so
we
don't
need
to.
D
The
WordPress
part-
this
is
just
for
the
demo,
so
well,
we'll
just
tear
things
down
kind
of
what
I
mean
for
those
who
are
interested.
You
make
this
watch
with
a
cube,
CL
get
PVC
which
will
list
the
precedent
volumes
we
have
created
in
our
cluster
and
then
we'll
run
the
delete
commands
over
there,
and
you
can
see
that
humanities
detects
immediately.
This
volume
is
not
needed
anymore.
D
We're
gonna,
terminate
it
so
first
we're
gonna
stop
the
container
then,
once
the
container
is
stopped,
we're
gonna,
detach
the
volume
and
then
we're
gonna
clean
up
the
volume
from
the
safe
side
and
like
free
up
the
resources
for
somebody
else
to
be
using
them.
So
this
is
done
father,
my
sequel,
won
and
now
for
the
WordPress
one,
alright
perfect.
So
now
WordPress
is
completely
gone.
We
can
verify
with
a
dashboard
now
there's
just
this.
D
First,
two
gigabyte
volume
created
at
the
beginning,
all
right,
oh
lab,
30,
all
right,
so
ruk,
it's
very
much
focused
on
safe.
Oh
it's,
the
like
stable
one
of
the
first
feature
that
Brooke
has
also
has
support
for
other
capabilities
like
database
provisioning.
I
think
it
has
cockroach
DB,
which
will
demo
just
now.
We
have.
There
is
a
cassandra
edge,
FS
and
FS,
and
a
couple
of
other
features
it.
We
just
don't
have
enough
time
to
cover
everything.
D
So
we
take
Coker
GB
Cooper
view
is
like
distributed
database
shorted
across
different
servers
and
has
the
same
wire
protocol
as
PostgreSQL,
so
most
likely
it.
So
you
can
use
any
PostgreSQL
library
or
PostgreSQL
client
to
connect
to
cockroach
DB,
which
is
kind
of
neat
on
that
front.
So
this
is
like
a
separate
operator
of
rope,
so
it
has
to
be
installed
additionally,
because
not
everybody
wants
everything
to
be
installed
by
default.
So
let's
copy-paste
this
keeps
it
really
create
the
convertible
operator.
D
D
We
give
it
volume
template
which
will
use
the
Ceph
provided
storage,
so
Cockrell
gb
will
use
storage
provided
by
SEF
and
Brooke
will
manageable,
the
safe
part
and
the
cockroach
DB
part.
We
ask
for
cluster
of
three
nodes
and
then
we
define
a
couple
of
ports
for
access,
cache
levels
and
miss
kind
of
things.
All
right.
A
D
So
the
access
modes-
you
have
kind
of
three
choices
there
and
it's
it
selects
the
way
you
can,
how
many
people
can
use
this
volume.
So
this
this
one
which
is
readwrite
once
is
the
one
we've
been
using
so
far,
and
that
means
that
only
one
container
can
use
the
volume
at
any
point
in
time
it
being
in
read
or
in
the
right
mode.
So
there
is
no,
it
blocks
people
from
having
two
containers
trying
to
write
to
the
same
volume,
for
instance,
or
one
would
write
100.
D
So
this
is
that
what
read
right
once
means,
then
you
have
read
right
many,
which
means
that
any
number
of
container
could
read
and
write
to
the
same
volume
in
parallel,
and
there
is
read
many
right
ones,
which
means
one
container
can
write
to
the
volume
and
as
many
as
you
want
can
read
the
volume.
Since
here
we
are
using
obd-ii.
D
B
D
It's
there,
which
might
not
be
a
great
idea
to
use
it
right
because
you
can
do
it
doesn't
mean
you
should
do
it
great
great
tip
there.
All
right
wait
locked
again
so
now
that
we've
had
a
look
at
the
manifest,
we
can
just
apply
it,
so
it
keeps
it
well
create
the
file
and
then
let's
describe
the
objects
we
just
created.
So
this
is
more
like
a
more
human
readable
way
to
display
the
same
information
as
what
was
displayed
in
the
yamo
once
it's
applied.
D
So
if
you're
not
very
friendly
with
the
animal,
you
have
the
describe
comment
that
can
be
useful
all
right.
Let's
do
a
watch
yep.
So
now
we
see
that
Luke
created
three
pods
zero
one
and
two,
and
now
they
are
already
running
and
ready
to
go.
It
also
created
three
too
wide
to
fit
in
the
screen.
Three
volumes
of
one
gigabyte
each
so
and
those
volumes
are
then
attached
to
each
pod.
Of
course,
all
right
now
I
said
4gb
I
could
PostgreSQL
database
PostgreSQL
like
database.
D
Let's
go
into
one
of
the
pods
and
we'll
try
to
run
some
sequel
commands
to
kind
of
flip.
It
around
this
database
cluster
so
similar
to
the
toolbox.
We
just
run
a
shell
into
it
when
I
have
this
sequel
commands,
so
we
can
list.
The
databases
currently
is
just
a
system
database.
So
let's
create
one
which
will
call
example.
D
Alright,
we
have
created
the
database
called
example.
Let's
use
that
example
database
now
we
will
create
a
table
with
an
example
called
customers
which
has
a
first
name
last
name
and
some
ID
all
right.
We
have
a
table.
We
will
insert
some
data
in
said
table
right
and
now,
let's
do
a
select,
and
we
have
a
bunch
of
data
in
our
table
now
brilliant.
D
We
can
do
some
more
complex
selects.
So
it's
cuckoo
RGB
has
like
the
fullest,
equal,
specs
and
I'll.
Remember,
which
version
of
it,
but
you
can
do
aggregation
functions,
account
sums
all
that
stuff.
So
it's
not
just
a
key
value
database.
It's
a
full
relational
database,
drawing
so
I'll
support
it
and
all
that
all
right.
So
it's
not
a
cookers
DB
workshop.
So
let's
exit
that
what's
kind
of
nice
with
cooker
GB
is
that
it
comes
built
in
with
a
dashboard.
So,
just
like
the
SAF
dashboard
by
default,
it's
internal
to
the
cluster.
D
Only
so
we
need
to
do
a
node
port
to
be
able
to
access
it
from
outside.
So
let's
do
that
and
then
we'll
run
the
snippet
that
outputs,
the
random
port
that
need
to
use
all
right
over
there,
and
now
we
can
reach
the
cockroach
GB
console.
So
it
lists
the
three
different
pods
that
are
part
of
this
cluster.
D
With
like
memory
usage,
different
version
which
is
kind
of
nice,
you
can
see
information
about
the
capacity
that's
used
in
your
cluster.
Of
course,
this
is
very
empty.
Cluster
I
just
created
what
sixth-century
in
one
table,
or
instance,
it's
kind
of
empty.
We
have
some
metrics,
so
this
is
like
built-in.
It
collects
some
metrics
about
sequel
agencies,
amount
of
queries
that
are
on
memory
usage,
CPU
usage,
all
that
stuff,
so
I
invite
you
to
have
a
look.
D
It
replicates
the
data
across
the
speak,
containers,
of
course,
and
the
three
volumes-
and
we
can
see
our
databases.
This
is
the
system
database
below,
and
example,
database
here
with
our
customers
inside.
You
can
see
our
schema
and
all
of
that
all
right,
so
everybody
got
to
the
cockroach
DB
dashboard.
D
You
see
some
nodding,
yeah
cool,
so
let's
move
on
to
lab
40.
So
so
far
we've
been
playing
with
a
two
node
kubernetes
cluster,
but
a
single
node,
safe
cluster.
That's
not
really!
You
know,
that's
not
really
cool.
We
want
to
have
a
proper,
proper,
more
multi
node
cluster
right,
so
this
is
lab
40.
So
first,
let's
have
a
look
at
the
the
node
two
how
it
looks
like
so
here
we
do
fdisk
minus
L,
which
lists
all
the
block
devices
available
or
not.
D
D
All
right,
so
we
can
see
that
Luke
has
already
created
a
discover
pod
on
the
second
node.
So
we
have
like
the
first
one,
which
is
on
node
one
and
the
second
one.
That's
one
two
and
those
pods.
They
are
there
to
collect
information
about
what
blog
devices
are
available
on
a
given
server.
So
Luke
does
like
blog
device
discovery
in
your
cluster.
A
D
1,
so
the
node
1
has
one
block
device
right,
but
it's
already
formatted
for
the
boot
file
system
or
the
root
filesystem.
So
we
can't
really
like
reformat
it
and
and
use
it
for
for
that.
So
what
we've
done
there
is
we've
configured
things
to
be
in
file
store
mode.
If
you
know
look
from
tree
luminous
versions,
there
was
the
file
store
mode
and
now
there's
the
blue
store
mode,
which
is
much
better
in
terms
of
performance.
D
So
here
we
have
a
bit
of
a
mix
of
both
from
node
1
file
store
and
no
we
will
do
blue
store.
So
we
can
have
a
look
at
that
how
we've
configured
this
with
cubes
ETL
edits,
F
cluster,
which
will
allow
us
to
edit
our
cluster,
and
we
can
see
that
in
the
node
list,
so
nodes
there's
a
list
which
contains
node
1
and
for
that
server.
We
have
said
to
use
this
directory
Valley
Brook
and
when
we
said
these
directories
it
means
use
file
store.
D
Now,
let's
add
another
node
or
let
me
go
more
here
and
we
will
do
add
another
element
to
that
list
and
we
will
call
it
node
to
the
UML
in
the
day.
Yamo
is
very
picky
with
indentation,
so
be
careful
to
have
your
own
dashes
at
the
same
level
for
node,
1
and
node
2
and
make
sure
you
have
the
same
spaces.
D
D
Let's
see,
I'll
reduce
the
font
a
little
bit
all
right
so
and
then
we
can
see
that
the
rook
is
doing
things.
So
we
can
see
that
our
cluster
is
in
a
updating
state,
so
it
means
that
rook
is
doing
something
with
this
cluster
and
that
thing
that
is
doing
it's
currently
preparing
the
oh
s
DS.
So
we
see
that
there
is
this
OSD
prepare
not
to
running
on
node.
That
is
the
new
node.
We
are
adding
to
our
cluster
and
we
are
preparing
those
OSD
so
for
blue
store.
D
You
still
need
a
little
bit
of
partitioning
for
the
rocks
DB
and
when
the
journals
and
finish
so
it
prepares
things
there
and
then
it
pops
up
the
ways
these
once
they
are
prepared.
Just
creates
the
ways.
Do
you
wanna
see?
What's
the
three,
so
we
can
see
them
they're
good,
and
we
can
see
now
that
our
cluster
is
back
created,
health-focused,
8,
perfect,
a
health
warned
now,
because
we
we
just
added
some
OSD,
so
there's
some
a
little
bit
of
rebalancing
happening.
Probably
let's
resize
this
down
all
right.
D
Yes,
so
it
will
run
one
container
for
new
block
device,
let's
exit
this,
and
so
our
first
pool
we
created
was
replica
one.
So
now
that
we
have
two
servers,
let's
do
replica
right,
so
chips,
ETL,
edits,
F
block
pools
and
we
end
up
in
a
text
editor
again,
we'll
just
say
size
tools
are
the
size,
one
we
save
and
I
said.
D
Then
we
can
go
to
our
toolbox
container,
which
was
over
there.
You
do
a
safe
watch
and
we
don't
have
so
much
data
right,
so
it's
very
quick
to
rebalance,
but
for
a
second
you
could
see
that
it
was
self
health.
Health.
Warn
you
now
it's
back
to
health,
okay,
with
the
decorated
stage
being
resolved
all
right.
A
D
All
right
so
yeah,
so
the
question
for
the
recording
is:
how
does
it
work
with
a
different
OS,
this
being
enough
different
sizes
right?
So
let's
do
you
in
a
safe
toolbox.
You
have
with
the
status
and
we
will
see
indeed
that
there
is
the
OSD
0
that
was
100
gigs
115
gigs
and
we
have
always
the
one
400
gigs
OSD
2
as
well:
400
gigs
and
third
OSD
200
gigs.
So
the
first
one
is
on
node
1,
the
last
3
or
not
right.
So
those
ways
this
should
have
different
weights.
D
So
let's
have
a
look
at
yourself.
Oh
is
the
tree,
and
so
this
is
the
tree
structure
of
how
the
OS
these
are
organized
in
the
in
the
Ceph
tree.
So
we
have
the
root
which
is
called
default
by
default.
Then
you
have
one
step
below
you
have
a
node,
so
we
have
a
host
node,
1
and
node.
One
has
a
leaf
called
OSD,
0
and
then
note
2
has
3
leaves
or
is
the
one
was
D
2
and
D
3
and
each
of
those
have
a
weight
associated
to
them.
D
You
can
see
that
node
host,
OS
D
0
has
a
weight
of
0
13,
which
is
small
as
the
amount
of
terabyte
it
has.
0
13th
or
vice
and
OSD
1
has
a
weight
0.4,
which
is
400,
gigabyte,
modest
and
the
same
for
the
two
same
size
right.
But
as
these
three
has
mo
has
the
weight
0
2
because
it
has
200
gigabytes
available,
so
safe
sets
different
weights
there
and
it
is
in
the
commercial
grade
M.
D
The
weight
can
be
whatever
number
you
want.
It's
a
float,
I
guess.
So
if
you
have
a
photo
right
drive,
it's
gonna
be
four.
If
you
have
a
14
terabyte
drives
like
it's
only
14,
and
then
you
can
adjust
those
weights.
If
you
have
some
preferences
and
thing
like
this
all
right,
so
we
can
get
going
with
the
object
storage
part.
D
So
we
will
create.
Let's
see,
there's
not
one.
We
will
create
an
object
store,
which
is
basically
another
set
of
tools,
because
in
rgw
you
need
several
tools,
one
for
metadata,
one
follow
Ockman
for
data,
and
then
we
will
watch
safe.
All
rook
create
different
things
there.
It
takes
a
little
bit
of
time,
the
whole
bunch
of
tools
to
create
a
bunch
of
containers
to
create
we
create
the
reduce
gateway,
pods
or
gateways,
and
there
will
need
to
be
a
little
bit
patient
right.
D
D
B
D
D
No,
the
dashboard
doesn't
have
the
object,
gateway
configure,
yet
all
right
so
to
be
able
to
use
our
reduce
gateway.
We're
gonna
need
some
users,
because
we
want
to
upload
some
objects
to
it
right
and
download
them
and
create
buckets
and
all
that,
so
there
is
a
object,
store
user
manifest
for
that.
So
we
do
that.
We
tell
to
work
to
create
an
object,
a
user
and
we
will
try
to
fetch
the
API
key
is
associated
to
this
newly
created
user
and
we
have
an
access,
key
and
private
key
or
seeker
key.
D
So
this
is
used
for
the
s3
API
access
to
to
rgw.
So
we
have
those
values
there
and
I
will
install
a
client
here,
I'm
using
the
mini
mini
o
client,
which
is
kind
of
nice
for
accessing
it
through
API.
So
let's
download
that,
let's
make
it
executable,
let's
verify
that
works
right.
We
have
MC
working
then
we'll
need
to
configure
the
client
to
point
to
the
correct
end
point:
that's
the
API
credentials
to
it,
and
we
say
we
see
this
added
my
store
successfully
brilliant.
D
Alright,
then
next
step
is
to
do
make
bucket,
which
is
abbreviated
to
MB,
so
to
create
a
new
bucket,
which
we
will
call
my
buckets,
and
you
should
get
this
bucket
created
successfully
message
after
which
we
will
list
all
the
buckets
into
my
store.
We
can
see
our
bucket
perfect
and
we
will
do
MC
CP,
which
will
copy
data
from
our
machine
into
the
rtw
endpoint.
Here
I'm
just
copying
data
from
/tmp,
because
there
is
a
bunch
of
files
available
there
there's
a
few
hundred
megabytes.
D
You
know
you
could
copy
over
whatever
file
you
want
to
your
rgw
and
you
can
see
that
it
was
quite
fast
all
right
now,
let's
do
a
list
of
the
objects
available
in
that
bucket,
and
it
just
so
happens
that
this
is
the
locations
where
the
temporary
installation
files
for
queue
manatees
are
located.
So
we
just
copied
cube
ADM
hyper,
cube,
calico,
CDL
and
CNI
plugins
into
our
GW,
but
could
be
any
file
all
right.
D
So
it's
nice
to
have
a
CLI
access
to
our
GW
OS
3
API,
but
we
can
also
enable
HTTP
access
right.
So
let's
do
that.
We're
gonna
enable
read-only
access
to
to
our
bucket
so
that
people
from
the
internet
can
download
objects
from
our
bucket.
So
this
is
this
policy
download.
Unfortunately,
it
says
error,
but
it's
ok,
so
it
says
error
and
then
later
ok,
but
it's
fine.
Just
you
should
get
this
message.
This
is
what's
expected:
200!
Ok,
all
right!
So
don't
be
scared.
D
If
you
have
this
and
then
we'll
list
the
policies
associated
to
our
bucket-
and
we
can
see
that
my
bucket
has
the
policy
read-only,
which
is
fine,
which
is
what
we
want
and
currently
our
bucket
is
our
our
GW
service
is
exposed
only
through
a
cluster
IP
which
again
is
internal.
So
we
need
to
create
a
node
port
again
to
have
the
external
access
and
again
this
in
real
life
you're
going
to
want
this
to
be
over
HTTPS
with
SSL
encryption
and
all
that.
D
But
here
we
are
doing
plain
HTTP
and
again
we
will
get
the
node
port
number
and
URL
that
we
can
use
to
access
our
bucket.
So
this
is
the
one
for
my
lab,
but
it's
gonna
be
different
for
your
lab
I
copy
paste
that,
in
my
browser
and
here
I
get
an
XML
output
of
the
s3
API
that
lists
all
the
objects
that
are
available
in
this
bucket.
So
if
I
zoom
in
a
little,
we
have
a
calico
CTL,
which
is
available
there
as
an
as
a
file
on
an
object.
D
C
D
In
the
self
dashboard
I
in
the
safe
dashboard
yeah
in
the
self
dashboard,
it's
not
enabled
one
would
have
to
enable
the
reduce
gateway
part
of
it.
Yeah
yeah
this
one.
This
looks
like
it's
not
enabled
by
default,
so
some
something
needs
to
be
fixed
there,
but
otherwise
we
through
cube
CTL.
You
can
see
the
containers
so
currently
we
have
three
instances
for
semi
redundancy.
If
you
know,
if
you
will
have
more
than
two
like
a
proper
cluster,
it
would
be
distributed
across
your
nose.
Okay,.
C
D
B
D
D
We
need.
We
need
to
keep
the
same
storage
class.
So
this
is
in
case
you
would
have
several
safe
cluster
within
one
given
kubernetes
cluster
or
different
routes
into
your
OSD
tree
or
subtree.
But
currently
this
is
just
just
the
one
cluster
we
have.
So
we
keep
this
readwrite
once
so.
Since
we
are
using
the
RVD
back-end,
we
need
to
keep
the
access
mode
to
read,
write
1,
so
exclusive
access
to
a
volume
per
one
container,
and
then
here
we
can
set
whatever
size
we
want.
D
So
let's
put
12
gigabytes,
for
instance,
and
then
we
can
cube.
Ctl
apply
eff,
this
file
and
we
should
get
this
workshop
1
or
whatever
name
you
have
set
created
all
right
now,
let's
list
the
PVCs
and
see
if
we
can
find
our
workshop
1
purses
and
volume
claim.
Yes,
we
see
it
in
a
default
namespace
workshop
1,
so
it's
already
created
and
safe
12.
You
buy
it
good
and
we
might
be
able
to
see
it
in
the
block
device
there
as
well.
D
So
we
should
see
something
with
12
yeah
12
gigabytes,
perfect,
so
everything
is
created
there
and
this
is
like
if
you
would
create
volumes
manually.
This
is
kind
of
how
you
would
go
about
interacting
with
the
rook
side
of
things
rather
than
interacting
with
the
safe
aspects
like
or
really
create,
and
all
that
all
right,
so
we
can
move
on
to
the
lobby
et
I.
Think
yeah,
oh
good.
We
got
10.
D
A
D
35
right
all
right,
so
liability
is
all
about
monitoring
because
okay,
it's
fun
to
run
safe,
but
you
need
to
keep
an
eye
on
things
right.
You
don't
want
things
to
to
break.
Even
you
wouldn't
know
if
they
self-heal
later
on.
You
still
want
to
keep
an
eye
on
capacity
to
see
when
you
need
to
order
more
stuff.
You
need
to
keep
an
eye
on
your
performance.
That's
not
gonna!
Oh
here,
alright.
So
when
you
drink
is
super
important.
D
So
this
is
what
we
cover
now
and
for
that
we're
gonna
use
the
Prometheus
operator,
which
is
basically
Prometheus
as
a
service
inside
kubernetes,
so
kind
of
like
Luke,
is
safe
as
a
service
inside
kubernetes
from
with
us
operator,
is
prometheus
as
a
service
inside
humanities.
So
it's
a
one
line
installation
for
that
operator
as
well.
Oh
yeah
right!
So
we
have
our
operator
created
and
we
need
to
wait
for
all
the
thoughts
to
go
running.
So
this
one
and
it's
running
so
that's
good.
D
Things
for
monitoring
safe
to
tell
to
the
operator
hey.
We
have
this
safe
thing
there
lets
you
know,
monitor
that,
and
this
might
also
be
worth
checking
that
everything
is
still
running
good.
Now,
let's
check
that
URL,
where
prometheus
will
be
available.
So
this
is
the
URL
I
put
that
in
my
browser
and
I
am
able
to
access
Prometheus
all
right.
It
doesn't
tell
you
so
much.
It's
very
simple
interface
Prometheus
is
like
a
time
series
database.
D
It's
not
the
place
where
you're
intended
to
like
graph
things
and
all
that
this
is
more
for
like
querying
the
database
directly.
So
it's
not
supposed
to
look
amazing,
and
so
there
we
can
explore
the
interface
right.
So
we
have
the
alerts
superbly
to
use
you
can
do
you
can
set
alerts
based
on
metrics
to
say
if
my
metric,
my
storage
usage,
is
above
this
or
that
percentage,
then,
let's
trigger
an
alert
and
to
send
an
email
or
a
slack
notification
or
whatever
you
want.
D
So
here
we
have
a
bunch
of
things
that
are
defined
like
if
there
is
something
that's
near
fool
or
no,
this
down
in
your
cluster,
these
kind
of
things.
So
everything
is
good
in
our
cluster.
That's
nice!
We
can
check
the
targets,
so
this
is
the
URL.
Well,
metrics
are
available
from
within
the
cluster,
so
this
basically
talks
to
the
set
manager
and
has
the
Prometheus
module
enabled
in
a
safe
measure
to
collect
metrics
and.
D
Do
we
have
any
been
just
a
graph
aspect
of
it?
I
think
so
it's
closed
and
so
on.
The
graphs
I
can
give
an
example
of
a
query
like
if
you
would
be
the
right
on
your
query.
You
could
want
to
know:
let's
increase
the
size
there,
the
your
total
usage
of
your
safe
cluster,
so
the
metrics
is
in
bytes
so
to
convert
that
to
more
human
readable
unit.
We're
gonna
do
10:24
^
3
to
convert
that
to
that's
gonna,
be
gigabytes.
D
Now,
let's
execute
that
and
we
get
this
reply
from
the
server
it's
an
HTTP
metric
from
this
instance.
So
it
comes
from
the
rook
self-manager
in
a
rooks
F
namespace
and
which
comes
from
this
board
in
this
service
and
the
value
is
23,
so
we
know
we're
using
23
gigs
of
our
staff
cluster
and
we
could
well
our
Prometheus
server
hasn't
been
running
for
very
long,
so
we
don't
have
a
lot
of
history,
but
now
it
just
keeps
track
of
this
and
we
have
the
graph
over
time.
D
So
you
can
see
when
it's
going
up
or
down
and
take
the
appropriate
actions
to
buy
or
the
Commission
equipments
based
on
that
alright,
so
the
UI
is
very
basic.
We're
gonna
need
something
better
than
that.
We're
gonna
need
graph
Anna.
So
a
graph
Anna
is
like
a
visualization
tool
for
for
data.
So
let's
install
that.
D
So
we
have
this
come
in
there.
It
will
install
the
rivana
and
it's
already
with
a
service
type
node
port.
So
it's
already
pre-configured
to
be
externally
available
right.
Let's
collect
graph
Anna
password,
which
is
quite
long,
and
then
we
might
want
to
wait
for
the
graph
and
a
pods
to
be
ready.
So
that's
good
and
we'll
get
the
graph
on
a
URL
well
copy.
The
URL
in
my
browser,
I'll
copy
the
password
as
well.
D
First
load
is
a
bit
few
seconds.
Then
the
username
is
admin
and
the
password
you
just
decoded
it
so
put
that
log
in
now.
I
don't
want
to
keep
that
then
we'll
add
a
data
source
because
by
default
graph
and
add
just
doesn't
know
about
Prometheus.
So
we
need
to
tell
to
graph
an
aware
prometheus
is
so
a
data
source
click
on
Prometheus
and
then
we
will
copy
paste
paste,
which
is
the
internal
URL
of
from
of
Prometheus,
so
that
graph
Ana
can
access
Prometheus
from
inside
the
cluster.
D
So
this
URL
will
not
work
in
your
browser,
but
it
will
work
for
fragra
file.
We
put
that
in
there
save
in
tests.
Alright
data
source
is
working,
that's
good!
Now
we
will
like
go
and
import
some
dashboards,
so
you
go
over
the
plus
at
the
top
left
corner.
You
just
don't
click
on
it.
You
just
put
your
mouse
on
it
and
then
you
go
to
import
and
we're
gonna
import,
dashboard,
28
42!
So
we'll
paste
that
in
and
we're
gonna
select
our
data
source
to
be
Prometheus.
D
Again,
we
don't
click
on
the
thing.
Yes,
we
select
the
data
source
import
and
the
last
one
is
53
42,
so
those
numbers
are
basically
just
IDs
of
upstream
Prometheus
dashboards
provided
by
safe,
safe
project,
so
that
you
can
directly
use
quite
nice,
looking
dashboard
out
of
the
box
without
having
to
make
it
yourself
right.
D
So
we
have
those
dashboards
available
and
it
takes
a
few
minutes
for
Prometheus
to
collect
enough
data
for
the
graphs
to
show
up
and
in
the
first
few
minutes
is
gonna,
say
yeah.
Something
doesn't
work
because
doesn't
have
enough
history
to
to
show
the
data
on
a
graph
and
also
be
a
bit
patient
and
then
you'll
you'll
have
now
it's
fine.
You
have
enough
data
that
it
shows
up.
So
it
shows
this
is
a
nice
dashboard
you
can
put
in
your
TV
at
the
office,
monetary
or
your
clusters.
D
It
says
the
the
health
state
of
your
cluster,
how
many
monitors
you
have
in
the
current
quorum
number
of
tools,
the
capacity
the
used
capacity
available
capacity?
How
many
ways
is
up
down
out
in
average,
latency
and
all
that
stuff?
So
maybe
we
can
zoom
in
our
last
five
minutes
or
something
yes,
when
we
see
some
graphs.
D
Of
course
you
know
it's
up
to
you
to
keep
this
running
and
then
maybe
in
one
hour
you
have
a
lot
more
data,
you
create
some
more
pods
or
something,
and
it
could
be
more
interesting.
So
this
is
the
cluster
dashboard.
Then
you
have
what
the
pools
dashboard,
let's
put
five
minutes
as
well,
so
you
can
select
different
pools
and
see
information
users,
information
for
this,
so
that
pool
how
many
objects,
how
many
I
ops
this
so
that
will
have
and
OSD
dashboard
here
you
can
select
a
specific
OSD.
D
So
let's
look
at
OSD
home,
what's
its
utilization
percentage?
What's
the
latency
and
the
space
situation
we
have
there.
So
this
is
very
useful,
complement
to
the
Ceph
dashboard.
It's
very
focused
on
monitoring.
You
can
read
igg
in
to
make
custom
queries
and
have
a
historical
view
of
things
all
right
and
then
what
do
we
have
left?
The
last
big
one
I
think,
is
the
safe
upgrades,
because
it's
kind
of
nice
to
run
safe
and
you
have
the
monitoring
in
place
and
all
that.
D
D
All
right,
so
first
I
will
do
the
watch
so
yeah,
okay,
maybe
not
all
right.
Let's
just
do
it.
I
did
then.
So
we
will
do
cube.
Ctle,
DTF
cluster.
We
go
over
to
the
speck
image
definition
and
there
we
will
just
edit
the
the
self
image.
So
instead
of
saying
to
use
version
13
we
will
use
version
14
and
it's
that
simple.
We
just
save
that
and
now
Luke
will
start
to
upgrade
the
cluster.
So
I'll
try
to
put
this
in
fullscreen.
D
All
right,
so
we
see
that
our
staff
cluster
at
the
top
is
now
in
updating
state
and
rook
has
started
to
update
one
of
the
months.
So
one
of
the
money's
already
running
notice,
because
notice
is
14
and
13
was
mimicked,
so
we
started
with
a
safe
mimic.
Now
we
already
have
one
month
running
notice
and
a
rock
we'll
wait
for
that
month
to
be
happy
and
healthy
before
upgrading
the
second
one
and
then
the
third
morning
in
the
managers
and
then
the
OSDs
as
well.
So
this
is
really
like
one
one
command
chip.
D
D
D
And
then
under
manager
being
upgraded
to
version
14,
so
otherwise
you
would
have
to
you
know:
do
this
manually
go
into
the
proper
server
and
great
things
and
it's
very
tedious
process.
Now
it's
really
cool
that
yeah,
it's
taken
care
of
for
you
as
a
safe
operator.
Its
upgrades
can
be
a
bit
scary.
This
is
already
a
convenient
all
right
now
that
we
have
the
manager
available
and
ready
I
think
it's
gonna
touch
the
always
DS
right.
Yes,
always
diesel
next
brilliant.
D
So
we
have
always
the
zero
running
version
14
now
so,
once
it's
started
and
ready,
it's
gonna
go
on
to
the
next
two
Sdn,
and
the
next
way
is
the
in
the
next
OSD
in
the
next
OSD.
It's
fully
automated
upgrade.
Basically
so
it's
a
turning
of
the
the
OSD
and
then
turning
it
back
on
and
it's
the
status
of
the
cluster
is
still
health.
Ok,
so
it
sets
the
current
flat
the
proper
flag
to
say
that.
D
Ok,
this
is
expected
to
happen
and
we
don't
trigger
rebalance
every
time
right
because,
especially
in
large
clusters
or
hard
drive,
space
clusters
can
be
super
expensive
or
performance
wise,
expensive.
To
do
rebalances
during
upgrades
right,
so
this
is
the
we're
just
restarting
it,
and
you
see
it
text
what
I
know
30
seconds
per
OSD
to
to
upgrade,
so
you
wouldn't
have
time
to
do
rebalance
anyway.
D
D
A
E
E
E
D
I
will
have
to
check
the
dog
exactly
I,
think
it's
like
allow
volume
resize
or
something
at
it,
so
expansion
and
also
in
the
cube
API
server
I.
Think
you
need
to
set
a
flag
to
enable
that
and
we
don't
have
that
enable
in
the
lab.
So
I
was
just
it
did
let
you
yeah.
So
this
is
what
I
suspect,
like
I've
done
it
and
into
another
cluster.
Not
in
this
lab.
D
We
have
to
set
something
on
the
API
server
to
enable
that
feature
which
was
flagged
as
beta
or
something
it
is,
and
then
you
need
to
set
the
flag
on
the
storage
class.
Then
you
delete
the
pod
where
the
volume
is
attached.
It's
gonna
detach
the
volume
from
that
container
and
then
way
it
attaches
it
back.
It
sees
that.
Well,
this
volume
is
bigger
now,
so
it's
gonna
do
a
filesystem
resize
which
what
with
whatever
tool,
is
applicable.
D
If
you
have
selected
the
X
FX
for
whatever,
so
this
is
gonna
take
a
little
possibly
a
little.
While
and
then
it's
gonna
mount
the
volume
for
that
container
and
then
it
makes
it
available
to
be
noted
that
this
works
on
in
one
way
right,
you
can
only
expand.
You
cannot
like
resize
down
the
volumes
with
RBD,
though
some
things
are
needed
in
the
Trinity
size
for
it
to
work,
and
we
haven't
covered
those
in
the
lab.
So
maybe
it's
a
good
addition
to
cover
right
there
for
sure.
That's
definitely
a
thing.
A
So
I
think
that's
that's
everything
we
got
for
the
lab
a
couple
of
administrative
things.
If
you
want
us
to
keep
your
lab
running,
please
on
the
ether
pad
put
down,
please
keep
the
lab
running
and
send
us
a
note.
Otherwise,
if
you
don't
we're
going
to
go
through
and
shut
down,
because
it's
physical
computers
running
it's
going
to
be
used
for
other
things,
if
you
got
suggestions,
send
us
a
note.
You
know
we
wanted
you
to
we're
planning
on
doing
this
again
to
the
next
cute
cube
con
in
San
Diego.