►
Description
Kubernetes is neither an NFVI (NFV Infrastructure) nor a VIM (Virtualized Infrastructure Manager). It is an orchestrator. Moreover, it’s an extensible orchestrator. It can be extended via plugins, such as CNI, or via custom controllers, which in Kubernetes are called (confusingly, as we’ll see) “operators”.
When developing CNFs (Cloud Native Network Functions) we can deliberately choose not to extend Kubernetes and instead use only its built-in controllers and their associated resources: Deployments, DaemonSets, Services, PersistentVolumeClaims, ConfigMaps, Secrets, etc. It is entirely possible to follow cloud native principles without deep integration into the platform itself. However, doing so leaves many of the advantages of Kubernetes on the table and can increase complexity by forcing us to add non-native layers of orchestration. In this guide, we'll focus on custom controllers (“operators”) and how they can and why to implement the operator design pattern for Telco Workloads.
A
A
Hello,
everybody
and
welcome
to
another
openshift
commons
briefing,
I'm
diane
mueller
and
I'm
really
thrilled
to
have
with
me
today,
tom
lehron,
from
the
telco
solutions
and
enablement
group
at
red
hat
part
of
the
the
office
of
the
cto
coming
to
talk
to
us
about
kubernetes
operators
for
telco
workloads,
which
basically
covers
every
buzzword,
bingo
of
every
conversation.
I've
had
probably
for
the
past
week
with
the
telco
teams
and
the
operators
team
and
everybody
does
love
a
bit
of
kubernetes.
A
So
I'm
really
thrilled
tal
has
really
thought
a
lot
about
the
pattern
of
kubernetes
operators
and
wrote
an
incredible
internal
white
paper,
which
he's
turned
into
an
external
one.
That
will
give
you
a
link
to
at
the
end
of
this,
and
I
think
he
really
kind
of
expressed
the
role
of
operators
and
and
really
explained
the
greatness
of
using
this
pattern.
So
I'm
going
to
be
quiet
and
let
tall
walk
through
his
his
thinking
around
this
topic,
and
you
can
ask
questions
in
the
chat.
A
We'll
leave
some
time
at
the
end,
wherever
you
are
on
whether
you're
watching
on
youtube
or
bluejeans
or
in
twitch,
you
can
ask
questions
and
we'll
relay
them
back
to
tall
at
the
end.
If
we
don't
get
to
all
of
your
questions,
we
will
post
it
in
the
blog
and
try
and
answer
them
in
a
blog
after
this.
So
with
that
intro
tall.
Thank
you
very
much
for
today
and
taking
the
time
tal
tell
us
about
yourself
and
take
it
away.
B
Everything
I'm
an
engineer
in
the
telco
solutions
and
enablement
team
at
red
hat,
we're
part
of
ecosystem
engineering.
I've
been
working
with
with
telco
workloads
on
kubernetes
for
a
very
long
time
and
and
operators
is
always
the
the
topic
that
comes
up
so
correct.
There's
a
lot
of
buzzwords
here,
but
hopefully,
after
this
presentation,
you'll
you'll
realize
that
there's
some
substance
behind
it.
So
I
will
dive
right
in
because
I
have
quite
a
lot
to
cover.
B
B
Why
are
telcos
using
kubernetes
operators,
and
why
are
they
even
interested
in
that?
Then
we're
going
to
move
to
talking
about
what
operators
really
are,
then
we're
going
to
go
back
to
telco
and
look
at
actual
use
cases
where
kubernetes
operators
can
can
really
help?
And
finally,
I'm
going
to
discuss
some
practical
considerations.
So
much
of
this
presentation
could
be
interesting
for
people
who
are
not
working
in
telco
and
are
just
generally
interested
in
what
operators
can
do
so
we'll
move
in
and
out
so
oops.
C
B
Why
are
we
talking
about
telco
workloads
and
kubernetes
operators
I'll
give
you
some
pre-history
in
this
graph
in
this
slide?
So
the
issue
is
network
equipment
and
from
from
the
very
start,
telcos
have
been
decoupling
things.
This
is
a
history
of
decoupling
to
an
extent
where
you
had
big
networks
run
by
big
telecommunications
company.
Think
about
a
transatlantic
telegraph
right
in
the
19th
century.
You
have
all
kinds
of
equipment
along
the
way,
but
it's
always
been
decoupled.
B
That
is,
we
had
network
equipment
providers
that
made
and
sold
the
comp
the
the
equipment
and
also
serviced
it,
and
then
you
had
the
operator
and
here
operator
in
the
telecommunications
sense.
So
we
have.
We
have
a
little
bit
of
confusion
here,
but
these
in
this
case,
when
I
say
operator,
I
do
not
mean
a
kubernetes
offering
so
the
the
network
operator,
the
telecommunications
company,
would
be
working
with
these
other
companies.
B
That
would
actually
provide
the
hardware
and
it
was
from
the
start
also
a
very
multi-vendor
situation
where
you
might
have
equipment
coming
from
different
different
providers,
different
vendors
and
they
would
need
to
work
together
right.
You
have
telegraph
on
both
sides
of
the
atlantic,
they
might
be
bought
by
two
different
companies,
but
they
have
to
work
with
each
other,
so
standards.
B
The
second
big
phase
that
again
we're
still
part
of
this.
This
history
is
virtualization
and
a
colleague
recently
corrected
me.
You
know
we.
We
sometimes
think
of
this
word
if
we're
engineers
in
terms
of
things
like
virtual
machines
or
virtualizing
network
interfaces
like
sriov,
so
we
have
virtualization
technologies
built
into
our
computers
these
days
that
we
and
operating
systems
that
we
we
definitely
use,
but
virtualization
here
is
meant
in
a
different
way.
B
The
idea
here
is
that
virtual
is
as
opposed
to
physical
or
actual,
the
idea
being
that
we're
decoupling
the
the
hardware
from
the
software,
so
suddenly
our
equipment.
We
can
talk
about
it
being
virtual,
meaning
that
it
is
no
longer
a
physical
piece
of
equipment,
but
it's
actually
a
piece
of
software
that
can
run
on
a
piece
of
hardware,
and
this
decoupling
basically
introduced
new
kinds
of
players
to
to
the
game
if
it
was
before.
We
talked
about
these
network
equipment
providers
right
in
the
pre-history
slide.
B
Now
we
can
talk
about
hardware,
vendors
versus
software,
vendors
and
we're
also
changing
our
terminology
a
bit,
so
we
can
still
talk
about
equipment,
but
now
we're
talking
about
network
functions
and
I'm
going
to
dive
into
that
a
bit
more
in
a
bit,
and
so
we
can
see
here
that
there
we
really
have
two
different
kinds
of
network
functions:
the
physical
ones
and
the
virtual
network
functions
bnfs
and
again
I'll
emphasize.
The
virtual
here
does
not
mean
it's
a
virtual
machine.
B
Necessarily
it
could
be
right,
but
all
anything,
that's
decoupled
from
hardware
would
be
a
virtual
network
function,
even
if
it's
running
in
a
container
right,
because
that's
the
idea
here
of
virtual
important
parts
of
this
story
are
that
we
started
to
evolve
to
use
off-the-shelf
products
right,
off-the-shelf,
hardware
and
off-the-shelf
software,
and
this
story
is
kind
of
hard.
This,
this
evolution,
because
again
some
of
this
equipment
can
be
very
specialized
and
require
very
specialized
hardware,
and
but
in
some
cases,
especially
in
software,
you
know
that
that's
easier
to
to
move
around.
B
So
the
first
phase
is
the
one
that
we're
really
involved
in
right
now.
That's
I'm
calling
it
cloudification
here
we're
complicating
the
picture
even
more
because
in
in
terms
of
software,
we're
talking
about
cloud
native
network
functions
that
has
network
functions
designed
to
run
on
on
clouds
on
cloud
platforms,
but
then
we're
also
introducing
the
platform
vendors
who
actually
create
these
these
cloud
platforms.
B
So
more
and
more
players
more
and
more
decoupling
and
again
there
are
some
trends
in
the
evolution
here,
so
we're
still
using
a
lot
of
virtual
machines
for
sure,
but
the
trend
is
also
to
move
to
containers
in
many
cases,
and
possibly
the
most
interesting
trend
for
us
here
in
this
presentation
is
the
move
from
what
I
call
management
orchestration,
moving
from
something
like
openstack
to
something
like
kubernetes,
and
here
here
it's
important.
The
move
from
openstack
to
kubernetes
is
not
just
to
move
from
virtual
machines
to
containers.
B
That's
actually
the
least
interesting
part
of
it.
The
real
interesting
part
is
that
we
are
no
longer
providing
infrastructure,
but
rather
orchestrating
our
workloads.
That's
what
kubernetes
really
brings
to
the
table,
and
I'm
I'm
very
I'm
oversimplifying
this
quite
a
lot
here.
There's
a
lot
going
on,
but
my
point
here
is
just
to
give
you
context
and
I'm
mentioning
here
we're
also
using
public
clouds
in
some
cases,
and
that
might
sound
strange
to
you.
Why
would
a
telecommunication
a
network
function
run
on
a
public
cloud?
B
Well,
so
much
of
it
ends
up
being
looking
a
lot
like
enterprise
software
or
it
software
even
think
of
a
cellular
network
where
you're
connecting
your
cell
phone
with
a
sim
and
somewhere.
There
is
a
record
of
that
salmon
who
would
be
longstudent
to
what
account
there's
a
database
where
all
this
is
stored
with
security
keys,
etc.
B
That's
kind
of
a
normal
it
workload
that
could
potentially
run
on
just
truly
generic
hardware
and
truly
generic
clouds,
and
they
could
even
be
public
clouds.
So
so
that's
that's
the
overall
context
of
where
we
are
and
it's
it
was
important
for
me
to
present
all
this
because
all
of
these
steps,
these
three
steps
are
still
with
us
and
that's
part
of
the
story
of
where
kubernetes
operators
become
interesting
and
useful.
B
B
The
operator
design
pattern
for
parallel
computation-
and
my
point
here
is
that
it's
really
used
in
a
computational
sense
and
we're
we're
not
talking
about
functions.
You
know,
even
when
we
talk
about
network
functions
and
before
when
we're
talking
about
something
that
does
continuous
work,
so
don't
think
of
it
in
terms
of
arithmetic
and
also
don't
think
of
it
in
terms
of
a
computer
program
calling
a
function
right.
That
function
is
triggered
by
one
thing
and
then
it
ends
right.
B
B
Our
context
here
is
really
when
we
apply
this
to
a
declarative,
intent-oriented
orchestration
context,
just
like
kubernetes,
the
operator
has
a
special
meaning.
It
consumes
intent
as
its
input
and
emits
intent
as
its
output
and
the
next
slide.
If
that's
not
very
clear,
the
next
slide
we'll
talk
about
it
a
bit
more,
but
again,
I'm
emphasizing
the
fact
that
this
is
continuous.
B
B
B
So
the
deployment
controller
will
take
that
deployment
and
create
a
replica
set.
So
the
intent
consumed
by
this
operator
would
be
the
deployment
and
the
intent
emitted
and
managed
by
the
operator
would
be
the
replica
set,
and
this
is
a
pure
operator
right.
The
deployment
can
you
know
if
we
delete
the
deployment
the
replica
set
will
be
deleted
as
well
and
they're
tied
together.
The
second
operator
in
the
chain
is
also
pure
operator
and
that's
the
replica
set.
B
B
B
I
encourage
everybody
to
open
their
imaginal
imagination
a
little
bit
regarding
the
operator
pattern,
because
it's
not
it's
not
just
abstraction
to
concreteness.
You
can
think
of
it
as
a
kind
of
translation
as
well
we're
translating
intent
between
different
kinds
of
domain,
and
this
again
would
become
very
important
in
telco
workloads
where
we
were
constantly
doing
work
of
translation
between
domains
and
standards,
etc.
So
it's
not
just
about
getting
more
concrete.
It's
it's
about
a
different
kind
of
intent,
almost
or
or
even
translating,
between
paradigms
really.
B
Okay,
this
this
next
slide,
I'm
not
gonna.
Do
the
sad
trombone
sound,
but
I
hope
you
hear
it
in
your
head.
I
wish
I
didn't
have
to
include
this
slide,
but
unfortunately,
the
kubernetes
official
terminology
is
very
confusing
if
we
look
at
the
actual
links
that
I
provided
here.
So
when
we
talk
about
controllers
and
kubernetes,
these
are
the
built-in
controllers
and
there's
really
discussion
of
the
controller
pattern,
and
this
is
all
okay.
B
What's
not
so,
okay
is
the
discussion
of
the
operator
pattern,
which
is
simply
wrong.
This
for
25
years
we've
been
using
the
the
operator
pattern
terminology,
but
kubernetes
has
decided
to
use
that
in
a
different
way.
So
in
the
kubernetes
terminology,
when
we
talk
about
controllers,
we're
talking
about
the
built-in
operators,
so
everything
you
saw
here
on
the
previous
slide.
All
of
these
are
controllers,
even
though
they
implement
the
operator
pattern.
B
But
when
we
say
operator
and
according
to
the
official
terminology,
we're
talking
about
a
custom
controller,
so
any
custom
controller
that
that
you
create
and
add
to
kubernetes
would
be
called
an
operator,
even
if
it
doesn't
implement
the
operator
pattern.
So
it's
it's
it's
confusing
and
we
we're
just
stuck
with
phrases
like
we
can
talk
about
a
kubernetes
controller
that
implements
the
operator
pattern
or
kubernetes
operator.
That
does
not
implement
the
operator
pattern.
B
Unfortunately,
but
you
know
that
terminology
terminological
soup
is
just
there,
but
it
doesn't
change
what
we're
actually
interested
in
here
and
and
and
that's
the
important
thought
we
care
about
kubernetes
operators,
even
if
they
don't
apply
the
operator
pattern,
whether
they're,
pure
operators
or
impure
operators
is,
could
be
important
to
the
architecture
we're
creating,
but
in
the
end
they're
useful,
even
if
they're
not
applying
the
operator
pattern.
So
if.
B
As
a
custom
controller,
that's
one
way
to
think
a
bit
about
it
and
the
question
of
whether
you
need
the
operator
pattern
has
to
do
with
your
entire
architecture
and
your
entire
strategy.
Are
you
trying
to
create
a
kind
of
unix
philosophy
tool
that
could
be
reused?
For
example,
if
you
go
back
here,
the
replica
set
controller
replica
sets
are
usually
usable,
even
without
deployment
controllers
right,
they
can
be
used
directly
and
they
could
be
connected,
maybe
from
to
other
operators
that
would
sit
before
them
and
generate
intent.
So
so
yeah.
B
Too
much
that
you
have
to
apply
the
operator
pattern
everywhere
in
order
to
make
good
use
of
kubernetes
operators,
but
it's
important
to
know
what
you're
doing,
which
I
hope
this.
This
presentation
is
clarifying
and
also
another
thing
you
should
not
do
is
reinvent
wheels.
There's
there's
a
whole
bunch
of
operators
out
there
already
of
diverse
quality
and
diverse
abilities,
but
look
at
them
first
and
of
course,
something
off
the
shelf
might
not
do
exactly
what
you
need
and
then
you
you
need
to
generate
your
own.
B
So
sometimes
you
do
need
to
retrofit
a
certain
wheel.
Maybe,
but
this
is
almost
all
these
are
open
source
and
that's
possible
to
do
if
you
need
to
do
it.
Okay,
we'll
switch,
we'll
swift,
sorry
shift
gears
back
to
talking
about
telco
and
look
at
how
everything
we
discussed
until
now
could
be
used
for
for
telco
workloads.
B
B
Maybe
some
change
to
the
containers
will
happen
too.
So
there
is
life
cycle
management
in
kubernetes,
but
it's
it's
it's
very,
very
specific.
Unfortunately
many
stateful
components
are
are
not
they
don't
work
that
way.
You
know,
if
you
have
a
say,
a
database
cluster.
Let's
say
it's
a
database
cluster
with
10
instances
running.
They
might
work
together
in
ways
that
you
you
do
not
want
to
simply
remove
delete
one
right.
You
you
want
to
maybe
or
or
add
one.
B
Maybe
it
needs
to
be
synchronized
when
it
when
it
is
added
in
there
are
all
these
issues
with
state
that
are
more
complicated
than
just
adding
and
deleting,
and
that's
when
your
operator
come
in
an
impure
operator
to
to
handle
all
that
specialized
work
that
you
would
need
to
do
and
again
remember.
The
operator
is
continuous,
so
it
would
be
constantly
monitoring,
say
your
database
cluster
and
making
sure
that
you
know
if
a
node
falls
off
that
you
we
can
create
another
node,
maybe
add
it
in
the
correct
way,
etc.
B
There
is
a
lot
of
state
and
network
functions
too.
So
here
I'm
emphasizing
that
you
might
think
that
networks
are
are
very
stateless
things
right.
A
component
along
the
way
is
like
a
router,
a
gateway
where's,
the
state
there.
Well,
there's
configuration
states,
but
beyond
that,
there's
state
that
has
to
do
with
the
we
with
the
actual
network.
There
are
a
lot
of
issues
with
maintaining
connections,
sessions,
load,
balancing
them.
B
There
are
aspects
that
are
cross
components.
You
might
have
multiple
network
functions
that
are
actually
part
of
the
same
overall
session.
So
again
same
thing
with
databases.
Think
of
it
that
way,
you
can't
just
add
a
node
or
delete
a
node
whenever
it
works
for
you,
you
have
to
ensure
that
you
can
scale
out
or
heal
out
or
scale
down
without
packet
loss
and
while
maintaining
these
sessions
because
yeah-
I
I
think,
that's
pretty
obvious
so
again.
Stateful
components
are
a
great
reason
or
a
great
use
case
for
for
creating
operators.
B
Configuration
management
is
another
very
big
one
network
functions
I
think
more
than
any
kind
of
workload.
You
might
be
familiar
with
very
heavy
configuration
so
much
so
that
a
whole
set
of
standards
and
and
stacks
of
software
have
been
created
just
to
to
configure
to
configure
network
devices
and
network
functions,
and
there's
there's
complex
enough
that
you
can
really
imagine
a
complete
workflow
of
configuration
so
think
of
it.
B
Think
of
configuring,
a
router
well,
first
of
all,
you
need
to
find
out.
Well
maybe
how
many
interfaces
the
router
has.
So
you
need
to
query
the
router
and
according
to
if
it's
this
kind
of
router
or
that
kind
of
router,
you
might
have
a
different
branching
workflow
here
and
do
other
kind
of
work.
So
there's
a
lot
of
work
with
configuration
and
again
this
is
a
great
use
for
an
operator.
An
operator
can
encode
a
lot
of
that
configuration
and
because
it's
running
on
the
cluster,
it
can
do
that
locally.
B
So,
for
a
long
time
in
configuration
management
for
for
networks,
we
had
some
sort
of
orchestrator
that
sits
maybe
far
away
from
the
the
site
that
manages
the
configuration
from
afar,
but
by
moving
it
to
locally
to
an
operate
on
kubernetes.
We
we
are
delegating
some
of
that
work.
I'm
going
to
talk
about
that
a
bit
more
in
in
other
sides,
but
it
makes
sense
to
to
keep
the
code
close
to
the
data.
B
In
this
case,
and
here
I'm
linking
to
an
operator
that
I've
been
working
on,
that's
been
doing
exactly
that
it
lets
you
write
these
workflows
using
python
and
have
them
run
in
your
cluster
third
use
desegregation
and
modularity,
and
this
might
seem
obvious.
But
but
again
it
has
a
very
specific
vent.
B
So
even
if
they
lose
connection
to
the
central
big
orchestrator,
they
can
still
operate
autonomously.
So
a
huge
advantage
for
reliability
too,
and
huge
advantage.
I
think
to
design
you
can
create
an
operator
that
does
one
thing
well,
rather
than
work
on
a
very
big
orchestrator,
which
could
be
an
enormous
project
and
yeah.
You
can
separate
it
off
so
so
it
fits
again
with
our
our
history
of
decoupling.
So,
let's
add
more
decoupling.
B
The
fourth
use
it
sounds
like
the
opposite
of
the
first
use
that
the
previous
use.
So
yes,
you
can
you
can
you
can
disaggregate,
but
you
can
also
use
operators
to
actually
integrate
and
we
see
a
lot
of
operators
like
this.
Actually,
operators
that
are
installers
so
think
of
creating
a
custom
resource
that
describes
your
whole
product.
We
see
that
a
lot
with
databases,
for
example,
so
you'd
like
to
install,
say
a
mariadb
cluster.
Well,
the
mariadb
operator
will
install
all
of
those
for
you.
B
Installer
is
not
the
best
word
for
those,
because
we're
also
talking
about
day
two
changes
that
is
after
it's
installed.
You
might
be
updating
it
and
you'll
want
to
see
those
changes
happen
as
well,
at
least
if
it's,
if
it's
a
good
operator,
there's
there's
some
history
to
this
that
predates
kubernetes
and
really
predates
clarification.
B
B
I'll
just
go
very
briefly.
For
a
long
time
we've
been
talking
about
specialized
via
virtualized
network
function,
managers
versus
generic
virtual
network
function
managers,
the
idea
being
that
an
equipment
provider
can
provides
the
equipment,
but
also
the
virtual
equipment
and
also
provides
a
management
suite
designed
for
that
equipment.
So
we
would,
if
we
want
to
manage
the
equipment,
we
would
work
with
that
management
suite.
But
a
lot
of
us
have
been
hoping
that
there
will
be
generic
versions
of
that.
B
So
we
wouldn't
have
to
reinvent
the
wheel
every
time,
we're
using
different
kinds
of
equipment
and
have
something
that
might
be
specialized
in
quirky,
so
so
that
kind
of
manager
would
be
able
to
be
configured
or
changed
or
programmed
in
some
way
to
handle
other
kinds
of
cns.
So
that's
again,
something
that
I'm
working
on
specifically
there's
a
project
called
turandot,
which
is
a
kubernetes
operator
that
uses
tosca
to
to
configure
what
kind
of
network
functions
you're
working
with
or
general
workloads,
so
so
yeah.
There
are
two
trends
here
right.
B
There's
one
trend
here
to
disaggregate
and
modularize
and
use
the
unix
philosophy,
but
there's
another
trend
where
we
can
encapsulate
all
the
work
inside
an
operator
and
the
question
is
well:
should
we
use
one
approach
or
the
other,
and
I
say
both.
I
think
both
of
them
make
sense
and
it
depends
on
what
you're
looking
at
the
the
rule
of
thumb
that
I
always
apply
is
keep
the
code
close
to
the
data.
B
B
B
Why
not
manage
them
from
kubernetes
too,
so
I've
I've
come
to
call
these
representational
operators
because
they're
operators
that
do
work
on
representations
that
exist
in
kubernetes
rather
than
the
resources
themselves
right,
they're,
not
running
in
the
cluster.
Now,
why
would
you
do
that?
Well,
you're,
you're,
basically
putting
all
your
work
in
one
place.
You
know
if
you're
already
committed
to
using
kubernetes
for
orchestrating
your
telco
workloads.
B
Why
not
use
kubernetes
to
also
orchestrate
your
pns,
and
it
makes
sense
too,
because
the
technologies
end
up
being
very
similar
too,
if
you
think
of
configuration
using
say
the
netconf
protocol.
Well,
the
physical
network
functions
are
using
it,
but
the
cloud
native
network
functions
might
be
using
that
as
well,
and
there
are
examples
of
the
industry
exactly
doing
that,
and
you
know
a
kubernetes
operator
or
called
a
controller.
B
B
And
what
are
the
pain
points,
and
this
is
from
doing
a
lot
of
work
with
operators
and
reporting
from
the
trenches
over
some
of
the
problems
you
might
encounter.
So
one
one
problem
is
the
custom
resource
definition,
implementation
and
kubernetes
sat
on
trombone
again
here,
because
custom
resources
are
namespaced
in
kubernetes,
but
custom
resource
definitions
are
not,
they
are
defined
cluster
wide,
it
might
sound
like
okay,
that's
not
a
problem.
B
You
can
just
well
you'll
just
install
them
in
the
cluster,
but
in
a
lot
of
management
scenarios
you
know
not
everybody
has
those
privileges.
The
question
then
becomes
is
the
operator
part
of
your
workload
or
part
of
the
platform
itself
and
there's
no
easy
way
around
this?
I
hope
that
kubernetes
ends
up
fixing
this
I'm.
It
would
require
a
major
change
to
into
how
it's
designed
and
how
custom
resource
definitions
are
designed,
but
I
think
that
would
go
a
very
long
way
to
make
operators
more
portable.
B
Custom
resources
are
are,
are
a
nice
tool
that
comes
with
kubernetes,
but
you
don't
have
to
use
it
if
it
doesn't
work
well,
for
you
and
putting
out
is
one
alternative.
You
can
use
config
maps.
There
are
also
general
purpose
resources
in
kubernetes
and
you
can
use
them
any
way.
You
want.
Why
not
use
them
to
store
your
your
custom
resources,
you
know,
use
them
as
descriptors
and
you
can
add
annotations
and
and
your
own
validation
and
schema,
etc.
B
You
can
build
that
yourself
on
config
maps.
Another
approach
is
to
use
annotations,
and
here
we
can
talk
about
fine-grained
operators,
that
is,
operators
that
don't
work
on
complete
intent
but
work
on
aspects
of
intent.
That's
still
an
application
of
the
operator
pattern
and
a
great
example
of
this
is
multis,
so
multis
would
look
for
annotations
on
existing
resources
and
those
become
part
of
the
intent
right,
they're,
not
part
of
the
custom
resource,
but
they're
actually
and
additions
to
existing
resources.
B
So
yeah,
that's
a
great
strategy
too.
You
don't
have
to
start
with
crds.
So
again,
I'm
going
back
to
that
page.
The
operator,
design,
parent
pattern
from
the
official
kubernetes
documentation
and
yeah
again
it
talks.
It
talks
a
lot
about
custom
resources,
but
no,
you
don't
have
to
use
custom
resources
if
that,
if
they're
hard
to
use
or
if
they
don't
do
what
you
need
to
do-
or
you
know
another
alternative
is
you
can
actually
store
your
intent
elsewhere
and
sometimes
it's
already
stored
elsewhere.
B
If
you
have
some
sort
of
orchestration
system
that
you're
working
with
or
a
database,
you
could
use
your
cut
the
custom
resource
or
config
map
or
an
annotation
to
just
point
to
that
so
work
with
ids,
instead
of
actually
storing
everything
there
and
that,
of
course
solves
the
problem
of
the
one
megabyte
limit
to
each
of
these.
So
if
you
store
it
in
your
own
database,
do
what
you
need
and
yeah
git
ops
right,
why
not
store
it
in
a
git
repository
too?
B
That's
another
approach
that
you
can
take,
so
your
custom
resources
will
be
the
live
pointers
to
intent,
stored
in
a
git
repository
and
your
operator
again.
Remember
it's
continuous.
It
would
have
to
monitor
it
and
look
at
changes
to
get.
You
can
do
that
through
triggers
for
commits
and
and
react
accordingly,
another
practical
consideration
and
it
a
little
bit
relates
to
the
previous
one.
B
Should
the
operator
be
installed
in
the
namespace
in
which
it's
going
to
work,
or
should
it
be
general
and
work
with
all
namespaces
and
for
different
kinds
of
operators,
it
could
make
sense
to
have
one
versus
the
other,
and
many
operators
actually
support
both
configurations.
So
if
you
want,
you
can
install
it
within
a
specific
namespace
or
separately
and
again,
this
goes
back
to
the
issue
of
well.
B
If
I'm
packaging
a
network
function
with
its
dependencies,
its
operators
or
do
I
say
well-
are
these
dependencies-
something
that
the
platform
has
to
provide
from
for
the
workload
to
run.
This
is
a
question
that
needs
to
be
solved
as
part
of
the
overall
deployment
strategy,
but
but
technically
it's
rather
simple.
It's
it's
usually
not
hard
to
create
an
operator
that
can
support
both
of
these.
The
the
kubernetes
api
server
always
requires
a
namespace
anyway,
when
it
works.
B
So
my
advice
is:
why
not
do
both
if
you
can
make
it
a
flag
when
the
operator
runs
to
say
whether
it's
running
in
namespace
or
clustered
mode?
The
difference,
of
course,
if
it
needs
to
be
work
in
cluster-wide
mode,
you
do
need
cluster-wide
permissions
to
install
it
and,
of
course,
if
it
uses
the
custom
resource
definition,
it
does
require
those
cluster-wide
privileges.
B
Another
practical
consideration
is
garbage
collection.
Kubernetes
has
a
garbage
collection.
It's
nice,
it's
very
useful
for
the
operator
pattern,
because
the
idea
is
that
you
can
make
the
emitted
intent
owned
by
the
consumed
intent.
So
if
you
delete
the
input,
the
output
will
be
deleted
too,
and
that
works
you
know.
If
you
delete
a
deployment
in
kubernetes,
the
replica
set
will
be
deleted.
B
The
when
the
replica
set
gets
deleted,
the
pods
get
deleted
and
eventually
the
containers
would
be
destroyed.
So
this
kind
of
built-in
garbage
collection
could
be
great.
It
could
really
add
robustness
right
if,
even
if
your
operator
crashes,
kubernetes
can
make
sure
to
delete
the
outputted
intent,
it's
not
good
for
stateful
components,
though,
for
the
all
the
reasons
we
stated
before,
where
you
know
there's
a
specialized
lifecycle
management,
so
we
don't
just
want,
say
the
database
to
be
deleted.
The
node,
maybe
it
needs
to.
We
need
to
update
a
connection
url.
B
Maybe
we
need
to
make
sure
that
the
data
in
the
node
has
been
synchronized
before
we
delete
it.
So
again,
an
operator
can
come
in
here
and
and
implement
its
own
garbage
collection.
So
just
something
to
be
aware
of
decide
for
yourself
whether
you
want
to
use
the
built-in
garbage
collection
or
not,
and
finally,
the
big
practical
consideration
really
is
developing
these
in
the
first
place,
I
it's
not
trivial.
Kubernetes
is
a
very
asynchronous
environment.
B
Kubernetes
is
written
in
go,
but
I
do
want
to
emphasize
that
you
do
not
have
to
write
your
operator
in
go
to,
even
though
it's
the
native
language
of
kubernetes,
but
whatever
you
do,
it's
not
going
to
be
trivial.
Jumping
into
developing
operators
requires
a
learning
curve
and
a
commitment
to
working
within
this.
This
complex
ecosystem
are
you
listed
here
a
whole
bunch
of
licks
that
people
can
take
a
look
at
to
see
either
get
inspiration
or
get
a
head
start.
B
B
Ansible
operators,
for
example,
are
very
easy
if
you
know
how
to
write
an
ansible
playbook,
you
can
package
it
inside
an
app
operator,
that's
a
great
way
to
start,
but
of
course,
if
you
have
to
work
with
network
devices
and
things
that
don't
have
the
ansible
support,
then
you
have
to
do
it
anyway,
and
then
you,
if
you
have
to
encode
the
the
lifecycle
management
logic
too
well.
Well,
there
you
go
so
so
it's.
B
That
I'm
kind
of
glad
that
I
managed
to
fit
it
in,
because
I
did
want
to
leave
room
for
questions.
I
will
notice
here.
Zayn
said
this
presentation
is
based
on
a
much
longer
document
that
I've
created
in
the
context
of
the
the
cncf
cnf
working
group.
So
we
have
a
working
group
there
that
we
are
trying
to
develop
best
practices
for
developing
cloud
native
network
functions
and
from
the
discussions
we
had
there.
B
A
As
as
always
tal,
it
is
a
tour
de
force
on
the
topic
and
it
makes
it
stretches
the
the
definitions
that
we're
used
to
and
may
be
comfortable
with
around
operators,
and
really
thank
you
for
this,
because
I
think
I'll
post
a
link
in
a
minute
into
the
chat
with
the
full
document
link.
But
I
I
really
appreciate
the
the
distinction
between
pure
and
impure
operators
at
the
very
beginning
of
this.
I
think
I,
before
I
read
your
paper
and
and
listen
to
this.
A
I
wasn't
really
thinking
of
them
in
in
those
terms
and
how
the
pre-history
of
all
the
operator
pattern
that
we
take
into
kubernetes
and
we
co-opt
for
our
own
purposes
and
how
important
naming
things
is
and
so
and
and
what
the
confusion
could
be.
So
I
think
what
you've
done
today
is
really
help
me,
and
hopefully
everybody
else,
who's
listening
to
to
really
start
thinking
about
this
in
another
way,
and
maybe
a
little
bit
clearer
thinking
about
it,
especially
in
terms
of
the
telco
stuff.
A
B
C
B
Love
to
have
a
write
once
run
everywhere,
universe,
where
you
can
write
the
code,
write
the
software
and
not
think
about
the
hardware
in
which
it
runs
and
we're
constantly
moving
more
and
more
towards
that,
and
but
it's
not
true
always-
and
it's
not
true,
for
it
really
depends
on
the
workload
too.
We
have
workloads
that
are
maybe
closer
to
the
physical
world
right
things
that
require
specialized
networking
equipment.
B
So
so
it's
not
true
that
you
could
just
deploy
something
to
a
cloud
and
expect
it
to
magically
work
in
many
cases.
So
we're
constantly
thinking
it's
not
a
simple
problem
to
solve
and
part
of
it
is
efforts
being
baked
into
existing
kubernetes
solutions.
So
red
hat's
openshift
does
a
lot
of
work
in
terms
of
managing
the
infrastructure
of
itself.
So
you
can
have
different
kinds
of
nodes.
B
We
can
write
once
run
everywhere,
even
if
it's
very
specific
plus
hardware
using
kubernetes
allows
us
to
orchestrate
at
scale
that
that's
the
big
sea
change
that
I
think
kubernetes
brings
to
the
table.
We
have
orchestrators
everywhere
right
now,
it's
different
from
openstack
in
which
you're
just
providing
virtual
machines,
but
then
something
would
need
to
install
software
on
those
virtual
machines.
Actually,
kubernetes
kubernetes
manages
our
software,
so
so.
A
Yeah,
I
really
like
the
phrase
you
used
about
kubernetes
being
an.
I
think
it
was
extensible
orchestrator
right
that
I
think,
was
a
key
thing
there.
He
has
a
follow-up
question
about
what
makes
a
cluster
telco
grade
he's
heard
about
cpu,
pinning
feature
etc.
But
what
is
it
that
makes
it
specialized.
B
B
There
are
timings
that
have
to
have
to
work
in
order
for
the
whole
chain
to
work.
So
it's
not
the
only
only
industry
with
standards.
Of
course,
right.
Lots
of
industries
have
reliability,
standards
and
regulations
from
medical
to
financial
and
security
standards,
etc.
So
so
I
would
say,
telco
grade
is
just
another
variation
of
those
kinds
of
standards.
B
B
So
the
question
isn't
so
much
as
telco
grade.
You
know
certified
in
a
platform
for
saying
this.
You
know
this
kubernetes
implementation
on
this
hardware
is
telco
grade,
but
rather
asking
the
question
over
what
workloads
are
you
going
to
run
on?
It
and
do
they
have
what
they
need
to
to
run
well,
and
those-
and
the
good
thing
about
that
story
is
that
it
becomes
not
a
telco
story
but
a
general
cloud
story.
B
You
know
if
we
solve
it
for
one
industry,
it
will
be
solved
for
others
as
well,
and
that's
an
advantage
that
telecommunication
companies
understand
too
moving
to
off-the-shelf
stuff
being
a
snowflake
is
not
a
good
position
to
be
in
being
in
a
multi-vendor
cloud
situation.
Gives
you
much
more
power
to
choose
different
vendors
to
switch
between
them,
to
negotiate
and
to
use
industry
knowledge
that
has
been
developing
for
years
and
apply
it
to
telecommunication
yeah.
A
A
Those
are
the
kinds
of
ideas,
and
I
say
ideas
because
I'm
from
new
england,
so
I
can't
say
it
the
other
way
that
that
help
all
of
the
other
industries
and
all
of
the
other
spaces,
which
is
why
it's,
I
think
recently,
we've
just
been
seeing
a
lot
of
the
things
that
telcos
have
had
to
do,
especially
around
edge
computing
and
keeping
the
code
close
to
the
data
and
working.
You
know
making
those
work,
decisions,
autonomous
and
local,
and
I
this
is
really
what
I
think.
A
What
as
that
is
at
the
crux
of
leveraging.
Kubernetes
operators
in
telco
is
really
just
another.
I
set
of
ideas
and
patterns
that
are
we're
going
to
see
applied
in
a
lot
of
places
and
and
move
through
lots
of
other
industries.
So
it's
really
been
yeah
the
think
telcos
telcos,
you
don't
think
of
them
as
really
personally
as
bleeding
edge
thinkers.
But
recently,
a
lot
of
the
work
that
we're
doing
here
at
red
hat
has
been
then
taking
telco's
use
cases
and
applying
them
elsewhere.
B
Yeah,
without
a
doubt
I'll
add
to
that
that
telco
use
cases
you
know
telco
grade
use
cases
are,
are
they
do
push
the
envelope
there
are
we?
We
can
definitely
say
that
in
terms
of
just
networking
and
by
networking
here
I
mean
just
low
level
networking
support
tcp,
for
example,
you
can
obviously
imagine
that
telco
workloads
cnfs,
would
require
much
more
sophistication
than
than
other
than
say
a
database,
an
enterprise
application.
So
I
think
approaching
the
telco
use
cases
and
trying
to
solve
them
benefits
everybody
in
the
cloud
space.
B
One
of
the
complaints
about
kubernetes
is
that
it
doesn't
it's
pretty
weak
with
networking.
It
doesn't
have
a
lot
of
opinion
about
networking.
You
know
you
bring
your
own
networking
to
kubernetes.
You
have
some
sort
of
sdn
software
defined
networking
solution
that
you
plug
in,
but
now,
but
kubernetes
itself
doesn't
care
too
much
about
that.
Well,
that's
nice!
B
Until
you
really
need
to
do
a
lot
of
low-level
networking
work
and
then
you're
asking
well
can
kubernetes
help
me
with
this
and
if
it
can't,
then
you
have
to
develop
all
these
systems
on
your
on
yourselves.
But
you
know
across
the
board.
I
think
the
telco
industry
is
pushing
kubernetes
to
to
be
better
with
networking
and
we
see
it
happening.
B
A
And
then,
and-
and
I
think
that
some
of
the
conversations
I've
been
having
is
with
paul
lancaster
who's
sitting
out
there
listening
in
I'm
sure
somewhere
and
around
the
certified
container
cns
and
the
big
push,
and
so
a
lot
of
the
vendors
that
have
cnfs
and
are
working
with
red
hat,
to
get
them
certified
to
work
on
kubernetes
and
with
openshift
and
and
doing
all
that
stuff
they
have
bisping.
You
know
once
those
things
are
there.
Paul
is
right
there,
yes,
paul
yeah.
C
C
You
know
what
we're
finding
is
as
these
as
the
isvs
or
the
business
units
that
create
software
within
the
network
equipment
providers
that
are
targeted,
they're
they're
in
customers
or
the
service
providers,
the
telecommunications
service
providers,
as
they
migrate
their
applications.
They
they
actually
take
advantage
of
a
lot
of
the
work
that
we've
done
from
an
operator
perspective.
So
there's
operators
that
we
actually
ship
with
opens
openshift
to
be
able
to
take
advantage
of
things
like
cpu,
pinning
or
sriov.
C
A
A
B
Kubernetes
is
an
upstream
project
itself
too.
So
within
that
space,
I
think
there
are
a
lot
of
different
kinds
of
opinions
and
needs.
I
can't
I
can't
expect,
for
example,
every
company
to
have
go
developers
right
or
to
be
able
to
invest
in
the
go
programming.
Language
telecommunication
companies
are
kind
of
interesting.
Some
of
them
do
a
lot
of
development
in-house
some
of
them
specifically
outsource
development
elsewhere.
B
There's
there's
a
lot
there's
a
variety
and
because
of
that
variety,
I
think
it's
important
to
also
provide
a
variety
of
solutions
and
approaches
to
doing
that.
So,
as
I
said,
you
can
use
ansible
operators.
That's
that's
a
great
start.
You
know
if
you're
already
invested
in
ansible
and
designing
playbooks,
if
you're
a
big
python
house,
there's
solutions
for
that
too.
B
The
main
thing
I
wanted
to
emphasize
is
that
kubernetes
is
itself.
The
api
server
in
kubernetes
is
language
agnostic
and
that's
a
huge
advantage
in
terms
of
of
plugging
into
it.
So
it
works
well
with
also,
you
know:
the
micro
services
approach
to
development,
agile,
where
you
have
teams,
maybe
working
on
on
a
very
specific
unit,
a
functional
unit,
a
network
function
or
an
operator,
and
they
might
be
using
different
technologies
from
other
teams
right
for
various
reasons.
Maybe
one
is
written
in
c
plus
and
working
directly
with
drivers.
B
The
other
is
written
in
ruby
because
it's
talking
to
some
middleware
database
system,
but
all
of
these
can
work
together
within
the
ecosystem
of
network
functions
and
up
kubernetes
operators.
So
so
it's
the
kind
of
work
that
I
do,
which
is
very
a
lot
of
evangelism
and
promoting.
I
I
I
emphasize
the
truth,
which
is
there,
there's
a
diversity
of
ways
to
approach
approaches.
A
That
is
one
of
the
things
I
love
about
your
talks
is
they
make
us
think
we
all
have
opinions
and
we're
all
opinionated,
but
I
think
sometimes
having
that
the
overview
of
the
landscape,
like
you
did
a
previous
talk
on
introducing
tosca
and
an
ama
on
that,
and
I
throw
the
link
on
that
too.
One
of
what-
and
these
are
the
kinds
of
talks
that
I
think
start
discussions
and
start
conversations
and
hopefully
drive
some
standardization
and
some
really
useful
in
innovations
into
the
into
the
space.
A
So
I
am
grateful
for
you
for
taking
the
time
to
first
of
all
write
that
wonderful
paper
which
I
will
send
out
over
the
the
internet
and
and
send
you
a
post
with
the
youtube
video
up
on
our
youtube
channel
and
on
the
openshift
commons,
and
we
are
definitely
going
to
have
you
back
and
every
time
it's
surprising
to
me.
So
I'm
looking
forward
to
whatever
it
is
you're
working
on
next
and
I
hope
you'll
come
back.
So
thank
you
very
much
for
today.
A
As
always,
yes,
we'll
definitely
give
you
the
podium
and-
and
do
this
again,
so
everyone
who's
out
there
listening
in
on
the
internets-
and
I
will
we'll
put
all
the
links
up
and
the
slides
shortly
and
I
think
the
raw
video
will
be
up
on
youtube
almost
immediately
because
we're
doing
live
streaming
to
youtube
so
rico.
Suave.
Thank
you
for
your
questions
and
everybody
else.
Paul
thanks
for
being
here,
we'll
have
you
back
again
soon
take.