►
From YouTube: Rook, Ceph, and ARM: A Caffeinated Tutorial - Federico Lucifredi & Sébastien Han, Red Hat
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Rook, Ceph, and ARM: A Caffeinated Tutorial - Federico Lucifredi & Sébastien Han, Red Hat
We picked a single-board computer with a dual-core ARM cortex A-53 and 2GB of RAM, put eight of them together, and built a low-power Ceph cluster not much bigger than the hard drives themselves. Sounds interesting? Come see us do it live right in front of you! After we review the design choices and architecture of the cluster, we will use it to showcase the use of Rook, the CNCF's incubation project designed to be a cloud-native storage orchestrator for Kubernetes, turning storage technologies into a service completely transparent to the user. A comprehensive examination of uses and open problems will complement our review of Rook's architecture. We will deep-dive into what it does well, what it does not do (yet), and what are the operational trade-offs of using a storage operator. With live access to a running cluster, we will showcase Rook in action as we discuss its capabilities.
https://sched.co/MPfn
A
B
B
Let's
start
from
basics,
for
those
of
you
who
do
not
know
self
is
an
open
source
distributed
software-defined
storage
solution
that
allows
users
to
access
data
through
different
interfaces,
including
object,
block
and
file,
or,
as
we
like
to
think
of
it.
Saif
is
the
Linux
of
storage
and
not
just
because
of
the
open
source
aspect,
but
because
there
are
many,
many
technically
fascinating
things
to
surf's,
beautiful
and
invincible.
B
B
Higher
level
interfaces
provide
block
storage
in
the
RBD
library
here
or
object
storage
in
the
form
the
rgw
demon
deliver,
which
delivers
the
AWS
s3
protocol
on
top
of
the
underlying
storage
core.
One
of
the
things
that
is
fascinating
about
SEF
is
its
data
placement
approach
requiring
no
gateways
but
finding
data
algorithmically.
In
short,
if
you
know
how
distributed
hash
tables
work,
you
already
know
what
concepts
are
is
built
around
and
what
makes
it
look
more
like
a
peer-to-peer
networking
system
than
traditional
storage.
B
So
why
is
storage
in
kubernetes
hard?
Well,
let's
cut
to
the
chase,
and
let's
say
that
as
micro
services
are
decoupling
infrastructure
and
application
logic,
the
developer
is
no
longer
need
to
worry
about
infrastructure
that
is
great
for
developers,
but
the
operators
still
need
to
figure
out
how
to
make
things
work.
Even
if
it's
behind
the
scenes,
the
kubernetes
architecture
is
very
dynamic
environment
with
containers
being
created
and
destroyed,
based
on
healing
and
need
and
load.
B
Statelessness
is
close
to
the
heart
of
the
paths
and
Ketel
metaphor
that
we
like
to
use
for
clouds,
but
many,
if
not
most,
of
the
useful
applications
today
are
stateful,
and
that
brings
the
need
for
external
network
storage.
The
simplest
solution
to
this
problem
is
to
have
apps
running
inside
the
cluster,
with
their
storage
residing
in
an
external
storage
setup
such
as
a
public
cloud
or
worse.
B
Yet,
the
legacy
storage
appliance
that
you
already
have
in
your
data
center
using
a
traditional
storage
appliance,
usually
kills
the
scalability
that
you've
won
on
the
compute
side
by
building
a
cloud
and
cripples
its
economics.
But
this
is
not
simply
about
bashing
the
limits
of
appliances,
even
if
you
are
using
a
uniformly
scalable
software-defined
storage
provider
like
SEF,
instead
of
an
appliance,
you
have
this
picture
of
storage
as
something
that
exists
outside
of
your
cloud
outside
of
your
cloud
fabric,
usually
underneath
it
notably.
B
This
is
how
OpenStack
does
storage
and
there
are
some
well-known
limitations
with
this
approach.
The
most
notable
one
here
is
that
it's
not
portable.
If
you're,
relying
on
external
storage,
you
need
to
have
it
already
accessible,
as
you
are
deploying
your
cluster,
because
it
is
not
built
in
kubernetes
and
when
you
want
to
move
your
application
to
a
different
cloud
provider
migrating
between
Amazon,
Microsoft
and
Google,
which
is
realistically
all
your
options
or
even
moving
between
public
clouds
and
on-premise.
You
need
to
worry
about
how
that
external
storage
is
going
to
go
there
with
you.
B
So
one
approach
to
solve
this
problem
is
to
run
storage
on
top
of
kubernetes,
rather
than
underneath
it
as
modern
software-defined
storage
like
seth
is
homogeneous
would
compute
in
that
it
uses
servers
from
your
favorite
iron
vendor
of
choice
as
building
blocks.
This
is
relatively
easy
to
pull
off.
You
can
gamble
your
way
to
success
by
deploying
our
storage
as
just
another
kubernetes
application,
maybe
a
special
one,
but
not
that
special,
so
that
is
building
block.
One
here
is
the
next
bit.
B
Another
aspect
we
need
to
discuss
before
we
tackle
rook
is
how
kubernetes
interfaces
with
storage-
in
other
words,
drivers,
storage
in
kubernetes,
has
multiple
abstraction
layers
ultimately
intended
to
deliver
workload
portability
at
its
most
basic.
The
storage
interface
in
kubernetes,
peer
storage
volumes
to
requests
coming
from
pods
in
the
most
sensible
of
these
variations
using
a
remote
network
based
storage
system
like
SEF,
delivers
dynamically
allocated
persistent
volumes
to
claims
made
by
pods.
B
This
is
done
in
a
way
that
is
decoupled
from
the
local
implementation,
while
requiring
no
knowledge
of
the
local
platform
by
the
user.
The
administrator
can
still
and
probably
should
still
have
local
knowledge.
This
allows
a
workload
to
move
from,
let's
say,
gke
to
Amazon
and
we're
not
going
to
go
over
all
of
the
details
of
the
storage
implementation
in
kubernetes,
but
others
have
done
that
work
for
us.
B
Saad
Ali
of
Google
has
a
very
nice
talk
that
is
easy
to
find
on
YouTube,
covering
the
fundamentals
of
PBS
and
claims
and
how
they
interface
with
each
other
and
the
IBM
FSS
FCI
team
also
has
a
series
of
short
videos
detailing
that.
In
painstaking
detail,
so
looking
at
a
slightly
higher
level,
the
first
set
of
kubernetes
volume
drivers
were
so-called
in
three
drivers
maintained
as
part
of
the
kubernetes
codebase
itself,
and
here
is
how
the
entry
drivers
connect
remote
storage.
B
This
approach
enabled
more
rapid
development
of
kubernetes
itself,
as
did
not
require
a
consistent
storage
API
to
be
developed
in
three
plugins
enabled
dynamic
provisioning
of
user
volumes.
Volume
drivers
conveniently
attached
pods
to
storage
automatically,
regardless
of
which
nodes
are
actually
hosting
the
pods
all
the
while
providing
sufficient
abstraction
to
enable
this
workload.
B
Of
course,
the
remote
storage
in
this
picture
could
be
AWS,
CBA,
Google,
Cloud,
PD
or
anything
else
that
has
an
entry
driver,
but
obviously,
in
this
session
we
like
that,
so
the
example
is
stack
based
in
the
New
World
in
abled
by
the
container
storage
interface
volume.
Plugins
are
maintained
out
of
tree
well.
This
requires
the
well-defined
API
that
we
dodged
earlier
and
Couples
API
changes
with
kubernetes
releases.
It
has
an
amazing
feature
for
operations.
Storage
can
once
again
be
treated
as
a
workload
to
be
containerized
and
deployed
on
kubernetes
cluster,
just
like
any
other.
B
So
first
we
talked
about
doing
it
with
the
storage
itself.
Now
we
can
actually
do
it.
Also
with
the
storage
drivers,
storage
plugins
can
now
be
provisioned
simply
by
calling
your
cube
control
cube
cattle
with
a
llamó
file.
The
cover
Natives
team
and
third-party
vendors
are
responsible
for
separate
discrete
components
and
security
access
does
not
exceed
the
permissions
that
are
actually
required
for
the
past.
These
components
run
inside
card
containers,
sharing
network
and
storage
access
with
the
main
workload
container.
B
You
can
see
here
from
the
massive
errors
that
clearly
there
is
an
API
at
play,
but
we
don't
really
need
to
understand
all
that
unless
we're
writing
a
driver.
The
bits
that
is
interesting
to
understand,
however,
is
that
all
this
abstraction
is
not
changing
the
performance
from
the
previous
picture,
because,
once
the
storage
is
attached
to
a
pod,
the
interface
is
direct.
There
is
no
intermediation
of
the
data
exchange,
so
maybe
allocating
the
storage
looks
a
little
bit
more
complicated
but
performance
wise.
B
It's
exactly
the
same
thing,
some
more
details
on
implementation
here,
probably
too
much
detail
frankly,
but
one
bit
of
Driver
lore
worth
knowing
beyond
what
I
just
mentioned,
and
something
that
is
not
in
this
slide
is
that
there
are
these
things
called
flex
volume
drivers
which
are
popular
with
work,
so
they
are
worth
mentioning
flex
volume
drivers
are
another
out
of
three
effort
which
is
still
maintained
by
now,
but
now
considered
already
legacy
and
not
really
being
enhanced.
The
flex
volume
drivers
make
exact
calls
to
a
file
on
the
local
machine
operationally.
B
This
may
be
limiting,
as
you
need
special
access
to
deploy
these
files.
So
what
I
was
just
saying
about
the
CSI
that
you
can
deploy
the
drivers
as
a
llamó
is
not
really
true
of
flex
drivers
unless
you
have
special
access,
which
actually
is
not
possible
in
certain
public
cloud
environments,
notably
Google
Cloud.
B
B
The
integration
here
was
completed
with
rook
1.0
and,
as
the
user
does
not
need
to
be
privy
to
surfs
internal
implementation,
detail,
an
application
could
move
from
your
private
data
center
to
a
public
cloud
without
adjustments,
provided
you
manage
that
pesky
data
gravity
aspect
on
your
own.
By
the
way
volume
claims
and
kubernetes
have
access
modes
which
are
known
as
our
way
or
the
blacks.
That's
another
interesting
detail
about
drivers
are
wo
and
rwx
are
the
most
popular
choices.
Rwx
is
used
to
share
data
between
iPods,
multiple
containers.
Rwo
of
is
the
default
choice.
B
Providing
storage
in
exclusive
access
to
one
in
SEF
RW
is
delivered
by
our
BD
to
ensure
maximum
performance
while
surface
delivers
the
sheer
rwx
mode
when
it's
required
and
now
we're
finally
reaching
what
we
are
all
here
to
talk
about,
which
is
project
work
rock
is
the
CN
CF
storage
operator
of
choice
and,
as
of
today,
it's
most
solid
integration
with
is
with
storage
as
delivered
by
itself.
A
kubernetes
operator
functions
by
letting
the
site
administrator,
define
a
target
state
and
then
working
its
way
toward
achieving
and
maintaining
that
state.
B
For
a
kubernetes
hosted
workload,
he
doesn't
do
it
in
a
vacuum.
Brook
does
just
that
for
safe
storage.
As
soon
as
rook
is
deployed
in
Burnet,
his
cluster,
a
complete,
safe
environment
is
running,
and,
while
my
and
while
one
might
be
forgiven
for
thinking
that
suffers
long
solved,
the
issue
of
persistent
resilience,
a
resilient
and
remote
storage.
B
The
combination
of
the
two
projects-
safes
advanced
storage
technology
pairs
with
rooks
high
usability
to
deliver
what
is
the
perfect
combination
for
developers
and
operators
hosting
workloads
in
kubernetes
environment,
and
on
that
note,
I
will
hand
over
to
my
colleague,
Sebastian
Han,
to
dive
deeper
into
the
architecture.
I
think
that
we
need.
We
need
the
other
microphone
on.
Please
all.
C
Right,
quick
question
before
I
start
how
many
of
you
knew
Seth
before
rook
and
how
many
got
to
know
Seth
with
rook,
okay,
it's
good
that
actually
I
was
actually
thinking
that
we
will
have
to
do
more
introduction
of
stuff,
but
actually
the
audience
is
actually
really
familiar
with
this
app.
So
it
is
actually
a
really
good
thing.
So
we
have
two
big
themes
right.
The
first
one
is
installing
staff
as
part
of
the
as
part
of
kubernetes
cluster.
C
So
in
in
this
scenario,
Luke
is
actually
responsible
for
installing
configuring
scaling,
upgrading,
safe
and
managing
its
entire
lifecycle,
just
just
just
like
an
application
running
on
kubernetes,
that's
the
first
part.
The
second
one
is
actually
providing
the
system
storage
to
applications,
and
this
is
done
by
a
CSI
so
root
not
only
deploy
staff
as
an
application
in
communities,
but
also
helps
you
providing
using
that
storage
for
your
application.
C
So
the
root
operator
actually
implements
the
operated
pattern.
It's
actually
a
very
popular
concept
in
the
kubernetes
word.
Basically,
an
operator
is
just
an
entity
that
runs
in
cuban.
It
is
so
that
it
will
extend
kubernetes
knowledge
landscape.
So
if
Cuba
latest
doesn't
know
something,
then
by
an
operator,
then
we
will
be
able
to
instruct
Cuba
nurses
that
you
know
how
to
deploy
that
cluster.
You
know
many
more
things
and
all
of
those
things
are
being
unbiased.
Yard
is
so
customer
shows
definitions.
C
C
So
now
that
you're
familiar
with
SAP
community
storage
in
kubernetes
and
in
ruk,
more
generally,
we're
gonna
dive
into
books,
architecture
a
little
bit
more.
So
what
you
see
at
the
very
bottom
here
and
what
we
see
at
the
very
button
or-
and
this
is
again
theme-
number
one-
infinite
number
two
right
so
team
number
one
again
is
deploying
safe
as
an
application
in
cuban.
It
is
so
always.
These
are
objects
towards
demons
responsible
for
basically
replicating
objects
for
adding
objects,
scrubbing
objects.
C
We
have
months
basically
the
brain
of
the
cluster.
They
come
with
with
three
or
four
is
just
five.
Sorry,
it's
always
an
even
number,
an
odd
number.
Sorry
they
maintain
the
map
of
the
cluster,
but
they're
not
in
a
data
path,
and
then
we
have
all
the
demons
like
MDS
writers,
gateway
and
MGR,
so
the
MDS
is
basically
the
metadata
server
that
exposes
the
disability
filesystem
of
Ceph,
where
read
as
gateways
the
rest
endpoint
where
you
can
consume
data
at
rest,
the
manager
is
just
a
pluggable
interface
that
you
can
use
to
extend
safe
capabilities.
C
So,
like
it's
a
it's
a
pluggable
mechanism
where
you
can
add
more,
we
can
think
add
things
like
like
metrics
on
the
second
theme.
Here
we
have
what
so,
obviously
the
rook
operator
the
main
thing
responsible
for
all
the
things
we
do
so
the
bootstrapped
and
the
maintenance
of
staff
and
the
connection.
So
the
connection
is
actually
being
done
by
agents
here,
so
we
call
them.
Agents
is
actually
really
generic,
but
agents
in
this
case
actually
represent
CSI.
So
at
the
very
top
here
we
we
have
your
app
and
the
claim
of
your
volume.
A
C
Going
back
to
this
desired
state
that
data
was
mentioning
earlier,
so
this
is
a
really
simple
example
of
a
CR
that
you
will
inject
inject
into
cube
unit
is
so
that
you
can
deploy
step
cluster
and
what's
actually
really
interesting,
as
you
can
see,
is
that
with
within
29th
of
Yama's,
then
you
can
actually
get
a
cluster
up
and
running.
So
that's
actually
in
terms
of
user
experience
that
that's
actually
really
good.
So
if
we
look
a
little
bit
so
we
have
the
namespace,
but
we
don't
care
much
about
this
at
this
point.
C
So
if
you
look
quickly
at
the
spec,
what
we
can
see
is
that
we
we
only
have
to
provide
a
relatively
few
information
here,
so
we
have
the
image
which
is
actually
critical
because
basically
States
what's
the
version
we
want
to
use.
So
these
images
are
provided
by
the
docker,
what
we
provide
them
as
the
second
safe
team
they're
available
in
the
docker
hub
and
what's
really
important
about
that,
is
that
these
images
are
built
every
day.
C
C
These
images
should
be
the
ones
that
that,
if
you
wanna,
if
you're,
looking
at
the
playing
safe
in
containerize
environment-
because
these
are
let's
say
golden
five
images
provided
and
maintained
by
the
surf
team-
the
non-count
is
just
it's
actually
something
that
I
would
like
to
add,
because
if
you
are
looking
at
the
playing
staff,
you
want
it
to
be
really
easy.
So
you
wouldn't
really
have
to
choose
how
many
months
you
want
to
run
should
be
able
to
tell
you
that
and.
A
C
We
can
obviously
did
you
can
always
ask
you
as
well
if
you
want
to
take
advantage
of
the
staff
dashboard
so
that
you
can
visualize
visualize
and
interact
with
the
cluster
to
see
what's
happening,
and
the
very
bottom
is
actually
potentially
the
most
important
one.
It's
related
to
storage
right,
so
we
have
different
modes
different
capabilities
here,
that's
actually
the
simplest
one,
so
you
can
specify
you
can
say,
use
all
the
nodes
and
use
all
the
drives.
C
C
That's
that's
easiest
in
our
I
would
say
we
can
be
more
granular
and
more
specific
here,
so
you
can
specify
nodes
with
a
particular
label
that
are
actually
it
may
be
labeled
like
storage
nodes,
and
then
we
will
go
through
these
nodes
and
again
scan
the
drives
and
detect
which
which
there
are
which
drives
are
good
candidates
to
become
actually
storage,
daemons
and
then
add
them
to
the
cluster.
You
can.
Even
you
can
go
even
deeper
here
by
specifying
which
node
and
which
drives
you
want
to
use
as
storage
storage,
back-end.
C
If
you,
if
you
have
different
types
of
workloads,
so
you
can
be
assigning,
let's
say
one
pool
for
object,
workloads
or
wonderful
for
block
workloads,
and
these
workers
are
obviously
really
different
and
I
mean
in
terms
of
IO.
They
are
more
or
less
more
intensive
than
others,
so
you
can
have
dedicated
hardware
for
this,
so
this
will
actually
be
creating
the
pool
again
with
abilities.
You
can
use
a
storage
class
and
then
once
the
user
wants
to
claim
a
PVC,
then
they
will
just
use.
It
same
goes
for
the
file
system.
C
Here
we
we
can
deploy
the
MDS,
so
the
for
the
portion
of
the
distributed
file
system
of
staff.
We
have
Stephan
FS
as
well,
which
actually
relies
on
Ganesha
under
the
hood
in
Ganesha
is
actually
using
the
staff
metadata
file
system
to
expose
NFS
shares
self
objects.
Torrents,
a
public
self
of
storing
self
object,
store
user
are
actually
pretty
simple
right.
They
just
boost
reproduce,
get
to
instances
so
that
you
can
consume
a
REST,
API
endpoint,
similar
to
not
similar
to
it's
just
like
an
s3
endpoint.
C
So
if
you're
looking
at
developing
one
as
three,
then
you
can
put
strap
those
and
then
you're
ready
to
go
okay.
So
that's
the
that's
the
embarrassed
embarrassing
moment
of
this
talk,
because
we
actually
had
something
working
yesterday
and
like
a
couple
of
hours
ago,
everything
started
crashing
so
I'm
just
gonna.
Let
Federico
explain
what
actually
happen.
Yeah.
B
C
B
We
have
a
bit
of
the
details
of
the
cluster
construction
in
the
slide,
so
at
least
we
can
tell
you
what
the
cluster
looks
like,
but
I
wanted
to
show
you
what
we
actually
had
running.
So
this
was
validated
last
night
around
11
hours
ago,
and
then
three
hours
ago,
when
we
were
doing
the
final
rehearsal,
the
master
stopped
answering
us.
B
What
is
going
on
is
actually
not
at
kubernetes
level.
It's
typical
embedded
development
problems,
so
we
have
to
get
you
booth
running
if
you're
familiar
with
that.
That's
the
typical
embedded
bootloader,
the
kernel,
obviously
and
user
land
and
then
get
kubernetes
on
top
the
higher
we
get
in
the
stack,
the
easier
it
is
so,
in
effect,
with
Rock,
we
didn't
encounter
any
significant
problems
and
kubernetes
was
very
straightforward
as
well.
B
So,
in
terms
of
when
things
are
running,
you
actually
get
to
do
get
pods
fairly
normally,
and
you
see
the
pods
that
would
come
from
a
little
cluster,
but
then
it
it
seems
to
have
a
significantly
high
probability
of
killing
itself
within
a
certain
number
of
hours.
Because
of
these
random
kernel
issues
and
the
way
they
are
happening,
they
corrupt
the
file
system.
That's
a
short
part.
The
plan
is
to
have
a
blog,
showing
how
we
are
doing
this
and
providing
the
entire
documentation.
But
obviously
we
have
some
more
some
more
details
to
work
out.
B
B
B
B
We
can
get
the
board's
for
about
forty
nine
dollars
and
the
SSDs
for
about
sixty
nine
dollars
and
so
we're
around
eight
hundred
dollars
for
an
eight
node
cluster
and
then
we're
doing
slightly
fancier
things
like
having
a
unified
power
supply
so
that
we
reduce
the
amount
of
cabling
and
just
try
to
make
it
look
nice.
But
those
are
thrilled,
though
the
cluster
proper
is
eight
hundred
dollars.
B
We
are
using
the
components
that
I
described
before
we're
using
kernel
419,
that's
kind
of
the
default
part
for
I,
don't
know
if
you
how
familiar
you
are
with
embedded
development.
If
you
are,
you
know
that
the
problem
with
these
embedded
boards
is
that
they're
very
far
from
data
center
hardware,
it's
a
completely
different
world.
So
for
getting
a
reliable
Linux
distribution
on
and
by
the
hardware,
you
have
to
have
all
the
drivers.
B
The
most
popular
distribution
for
doing
this
right
now
is
called
armed
Ian,
but
the
arm
beyond
images
for
specimen
v5
are
not
perfect
and
we
introduced
a
few
more
problems
ourselves
with
our
ability.
The
pesky
details
need
to
be
worked
out,
but
once
we
get
the
kubernetes
part
running,
the
rest
seems
to
be
seamless.
We
just
need
to
manage
to
keep
the
cluster
up
for
more
than
12
hours,
so
the
part
is
OS
reliability,
not
not
kubernetes
problems.
B
C
So,
if
you
have
an
existing
set
of
cluster,
then
you
can
actually
bootstrap
root
in
tribulations
and
then
tell
it
to
connect
to
this
existing
storage
cluster.
And
then
the
only
thing
we
will
be
doing
here
is
that
we
won't
be
managing
surfs
life
cycle,
but
we
will
only
bootstrap
resources
that
we
can
consume
by
RC
artists
such
as
the
persistent,
the
riders
gateway
and
the
NFS
file
system.
So
that's
the
idea
and
it
this
is
actually
one
of
the
first
step
to
actually
achieve
achieve
by
taking
a
takeover
of
an
existing
cluster.
C
So
the
ultimate
go
for
that
is
basically
to
be
able
to.
You
have
an
existing
cluster.
That
is
not
containerized
of
the
communities
Christo,
and
the
goal
here
is
just
to
take
over
all
of
these
resources
and
then
have
them
be
managed
by
by
work.
That's
in
the
future
still
we're
working
toward
that
goal.
C
C
That's
basically
again
announcing
the
user
experience.
Last
but
not
least,
Percodan
amin
provisioning
is
one
of
the
most
desired
feature
and
it's
something
we're
actually
pushing
into
the
community
the
community's
community.
It's
actually
really
equivalent
to
what
we
do
currently
with
volumes,
but
at
the
rest
object
layer.
So
not
only
you
can
because
at
the
moment
you
can
only
request
the
Gateway,
but
the
users
don't
care
about
the
Gateway,
but
what
they
care
about
is
basically
the
bracket
and
the
Pretender.
So
the
ultimate
goal
here
is
to
claim
a
bucket
and
then
simply
consume
it.
C
So
you
just
ask
for
a
bucket
you
get
when
you
get
you
credentials
that
you
can
start
working,
you
don't
have
to
do
anything.
You
don't
have
to
know
anything
about
relegate
weight
and
stuff.
The
only
thing
you
have
to
do
basically
is
just
ask
for
that
bucket.
So
that's
one
of
the
things
we're
working
on
there
is
a
PR
for
that,
not
sure
if
Jeff
or
Jenner
in
the
room
well
you're,
actually
working
on
this
and
yeah.
C
That's
a
super
super
nice
yeah,
one
more
thing:
we
need
15,
that's
but
at
least
again
I'm
super
super
happy
to
announce
that
we
actually
got
look
into
the
operator
hub.
The
operator
hardware
is
a
centralized
place
that
that
actually
provides
operators
from
the
community.
So
if
you're
familiar
with
OLM,
it's
an
operator
concept
that
is
basically
an
operator
lifecycle
manager
that
you
can
push
shopping
trip
unit
is
and
then
you
can
inject
CSS
to
it
and
then
you
will
be
able
to
it
will
be
able
to
bootstrap
your
operator
for
you.
So
yeah.