►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Being a Good Citizen of the Multi-operator World - Devdatta Kulkarni, CloudARK
DevOps teams are increasingly using multiple Kubernetes Operators in their clusters. What does it take to develop an Operator that is a good citizen of the multi-Operator world? The talk focuses on this question. To develop an Operator that plays nicely with other Operators in a cluster, Operator developers need to consider various aspects like, what should go in Custom Resource Spec definition, in the CRD YAML and in the Operator's Helm chart? To help Operator developers build Operators that are good citizen of the multi-Operator world, we have developed a framework named as, Operator Maturity Model (https://github.com/cloud-ark/kubeplus/blob/master/Operator-Maturity-Model.md). Community Operators like DataStax Cassandra, PressLabs MySQL, Elastic ELK, Cert Manager have benefited from this model. The talk will cover this framework for the broader benefit.
https://sched.co/ekBe
A
A
These
days
apart
from
cloudera,
I
also
teach
courses
in
cloud
computing
and
modern
web
applications.
Development
in
the
computer
science
department
at
university
of
texas
at
austin
kubernetes
operator
pattern,
has
become
widely
popular
for
running
various
kinds
of
applications
on
kubernetes
operators
are
getting
built
for
stateful
applications,
complex
services,
internet
id
workflows,
etc.
A
A
kubernetes
operator
adds
custom
resources
to
a
cluster.
The
set
of
resources
on
any
cluster
depends
on
what
operators
have
been
installed
on
it.
For
instance,
here
we
have
a
mysql
operator
and
a
custom
application
operator
installed
on
the
cluster
and
they
add
mysql
cluster
and
app
cr
custom
resources
to
it.
The
ensemble
of
kubernetes,
built-in
and
custom
resources
that
are
deployed
together
to
serve
a
specific
workload
can
be
thought
of
as
an
application
stack.
A
Increasingly,
we
are
seeing
lot
of
devops
teams
use
multiple
kubernetes
operators
to
build
their
custom
passes.
This
makes
perfect
sense.
Actually,
as
the
operator
technology
provides,
the
right
building
blocks
to
deliver
multi-tenancy
that
is
required
for
any
pass
through
the
custom
resource
instances.
A
Here
are
two
examples
of
such
passes
that
we
have
encountered
in
our
work
with
different
customers.
On
the
left
hand,
side
there
is
a
moodle
pass
built
to
deliver
an
e-learning
solution
on
kubernetes.
This
uses
two
operators,
moodle
and
mysql
moodle
operator-
is
developed
in-house
to
deliver
specific
workload.
Workload
requirements
for
moodle
moodle
instances
depend
on
mysql
instances
which
are
delivered
by
the
mysql
community
operator.
This
way
it
serves
multiple
moodle
application
stacks
to
different
customers.
A
A
This
custom
passes,
built
on
kubernetes
are
multi-talent
and
multi-operator
environments,
given
that
an
operator
may
need
to
run
alongside
other
operators
in
a
cluster.
The
important
question
that
operator
developers
need
to
ask
today
is
whether
their
operator
is
ready
for
this
multi-tenant
and
multi-operator
world.
A
A
In
today's
talk,
I
am
going
to
focus
on
the
four
specific
problems
and
associated
guidelines
to
address
them.
These
problems
are
how
to
enable
atomic
deployments
of
application
stacks,
how
to
enable
co-location
of
stack
components,
how
to
make
application
stacks
robust
against
pod
restarts
and
how
to
perform
accurate
chargebacks
per
application
stack
check
out
our
comprehensive
list
of
guidelines
on
our
github
page
to
know
more
about
these
different
categories
and
the
guidelines
to
discuss
these
questions.
I
am
going
to
use
the
example
of
the
moodle
pass,
which
I
referred
to
earlier.
A
Moodle
is
an
e-learning
software
and
I
am
going
to
be
using
only
two
custom
resources
that
are
part
of
this
stack.
There
are
several,
but
for
the
sake
of
discussion,
I
am
going
to
focus
on
only
the
two
custom
resources,
moodle
and
mysql
cluster.
The
moodle
operator
handles
the
moodle
custom
resource
and
creates
a
pod
with
the
moodle
software
on
it.
The
mysql
operator
handles
the
mysql
custom
resource
and
creates
the
pod
for
mysql,
and
then
the
moodle
pod
uses
the
mysql
port
for
its
database
needs.
A
A
Well,
the
autonomousity
in
this
case
means
that
we
want
to
avoid
situations
where
only
some
parts
of
an
application
stack
are
deployed
and
not
others.
So,
for
example,
we
want
to
avoid
situations
where,
let's
say
the
moodle
part
has
gotten
deployed,
but
the
mysql
pod
has
failed
to
get
scheduled,
or
vice
versa,
where
the
mysql
pod
gets
scheduled,
but
the
moodle
pod
has
failed
to
get
scheduled
on
any
worker
node.
A
So
then
the
question
is:
how
can
such
atomicity
be
guaranteed,
especially
when
there
are
operators,
some
community
operators
in
your
in
your
multi-operator
environment,
some
that
you
might
have
developed
on
your
own?
So
how
do
you
really
guarantee
this
atomicity
and
then?
The
second
question
is:
what
can
operator
developers
do
to
ensure
that
the
pods
that
are
deployed
as
part
of
their
custom
resources
get
guaranteed
scheduling
from
the
kubernetes
scheduler?
A
Now,
when
we
are
working
with
custom
resources,
the
the
pod
is
not
visible
right
because
the
operator
is
going
to
handle
the
custom
resource
and,
as
part
of
creating
that
custom
resource,
it
is
going
to
create
some
parts.
Maybe
it's
not
even
actually
directly
creating
a
pod.
It
is
probably
creating
a
deployment
or
a
stateful
set.
So
the
question
is:
what
can
the
operator
developer
do
to
leverage
the
built-in
kubernetes
mechanisms
of
resource
request
and
resource
limits?
A
Because
if
you
look
at
the
kubernetes
scheduler,
if
all
the
containers
that
are
part
of
a
pod,
spec
are
provided
with
resource
requests
and
limits,
then
that
part
is
going
to
get
guaranteed.
Scheduling
behavior,
so
the
kubernetes
scheduler
will
will
provide
guarantees
that
that
board
will
get
scheduled
on
some
worker
node,
which
has
adequate
capacity.
A
Okay.
So
as
an
operator
developer,
our
guideline
is
that
you
can
provide
in
the
custom
resource
spec
a
way
or
the
spec
properties
through
which
these
resource
requests
and
resource
limits
can
be
specified
and
then
implement
your
custom
controller,
which
is
part
of
your
operator,
to
pass.
These
scheduling
hints
down
to
the
pod
that
the
custom
controller
is
going
to
create
when
instantiating
a
custom
resource
instance.
A
So
once
you
do
that,
if
you
are
operator
developer,
then
kubernetes
will
provide
that
guarantee
of
scheduling
that
port
on
some
worker
node,
which
which
has
the
capacity
now
it
may
so
happen
that
you
don't
have
access
to.
You
are
not
developing
an
operator.
You
are
using
some
operator
from
the
community
and
then
the
question
becomes
you
don't
know
the
source
code
of
that
operator
to
actually
modify
the
custom
controller.
A
Then,
in
that
situation,
what
can
you
do?
So?
Our
guideline
is
one
option
that
you
can
explore
in
that
situation
is
use
some
admission
controllers,
maybe
something
like
opa
to
to
intercept
such
parts
and
then
then,
at
that
point
you
will
be
able
to
change
the
spec
to
to
add
these
scheduling
hints
on
the
custom
resource
on
the
containers
that
are
defined
as
part
of
that
part.
A
A
For
example,
you
may
want
to
provide
some
differential
service
or
treatment
to
different
customers
as
part
of
your
past,
and
for
that
reason
you
may
want
to
just
use
some
worker
nodes
to
to
basically
deploy
a
complete
stack,
and
you
want
all
the
parts
of
that
stack
to
run
on
that
worker
node.
That
could
be
one
reason.
A
A
A
There
is
pod
to
node
affinity
and
there
is
also
the
affinity
between
parts,
so
part
affinity
and
node
affinity
and
the
way
it
works
is
if
there
is.
Let's
say
if
there
is
a
node
and
if
it
has
a
the
the
node
may
have
some
labels
and
then
in
your
pod
spec.
If
you
specify
a
node
selector
with
the
label
of
one
of
the
nodes
that
the
node
has,
then
the
kubernetes
scheduler
will
schedule
that
part
on
that
particular
node.
Okay,
so
that
that's
available
directly
in
kubernetes.
A
A
So
if
you
design
your
custom
resource
spec
to
allow
specifying
a
node
selector
and
then
if
you
implement
your
custom
controller
to
pass
that
node
selector
down
to
the
port
that
it
will
be
creating,
then
it
will
be
possible
to
co-locate
these
your
pod
with
some
other
pod,
which
may
be
coming
from
another
custom
resource
or
just
directly
created
deployment
doesn't
matter.
But
having
that
having
that
ability
to
specify
node,
selector
and
then
passing
that
label
down
to
the
pod
will
allow
co-location
of
your
pod
with
other
parts.
A
The
third
problem
that
we
are
going
to
look
at
is
pod
restarts
and
how
to
make
an
application
stack,
robust
against
pottery
stocks.
An
example
of
pottery
start
is
when
a
sidecar
container
is
injected
into
a
pot.
Spec
a
pod
gets
restarted.
I
mean
anytime,
you
modify
the
port
spec.
The
pod
is
going
to
get
restarted.
A
So
where
do
we
see
really
pod
these
sidecar
injections
happening?
So
there
are
certain
operators,
community
operators,
that
that
they
will
modify
a
part
spec.
An
example
of
that
is,
are
the
volume
backup
operators?
What
they
do
is
some
of
the
ones
that
we
have
looked
at.
They
will
essentially
in
order
to
take
a
backup
of
a
pods
mounted
volume,
they'll,
add
a
container
to
a
pod
spec
and
then
that
container
the
sidecar
container
has
access
to
that
volume
and
then
they
are
able
to
that
way.
A
Take
backup
of
that
volume
now
when,
in
order
to
do
this,
that
operator
will
once
a
pod
is
specified
to
that
operator.
That
operator
will
modify
that
pod,
spec
and
inject
this
side
curve.
So
in
this
example,
we
wanted
to
take
backup
of
the
moodle
of
moodle
custom
resource
and
the
pod
that
the
moodle
operator
has
created
for
that,
so
the
volume
backup
operator
once
it
injected
the
side
car
into
the
moodle
pod,
it
caused
the
moodle
pod
to
get
restarted.
A
Okay.
So
now,
what
is
that,
as
an
operator
developer,
you
need
to
be
cognizant
of
such
pod
restarts,
because
what
may
happen
is
after
a
pod
gets
restarted.
You
have
to
still
go
back
and
verify
whether,
whatever
the
application
level
in
variants
that
we're
supposed
to
be
valid,
whether
they
are
still
valid,
you
need
to
check
that
and
that
that
is
something
that
you
need
to
do,
so
they
are
still
maintained.
A
So,
for
example,
in
in
the
case
of
the
moodle
stack,
the
the
requirement
was
that
there
were
these
moodle
plugins
that
had
to
be
installed
on
on
a
particular
stack
and
once
the
moodle
pod
got
restarted,
those
plugins
had
to
be
reinstalled
because
otherwise
they,
after
a
restart
there,
were
situations
where
those
plugins
would
not
be
present.
So
the
the
operator
had
to
subscribe
to
these
pod
restart
events
and
then
ensure
that
the
plugins
got
installed
again
when
the
parts
restarted.
A
Finally,
as
a
fast
provider,
we
are
also
interested
in
accurate
chargebacks
at
different
levels.
We
may
want
to,
for
example,
find
out
what
is
the
cpu
memory,
consumption
of
a
particular
custom
resource,
say,
for
example,
the
mysql
customer
resource,
how
much
cpr
memory
that
one
is
consuming
or
we
may
want
to
find
out
cpu
memory
consumption
at
the
level
of
an
entire
application
stack
which
in
this
case
consists
of
a
moodle
custom
resource
and
a
mysql
custom
resource.
A
A
So
to
answer
these
questions,
what
is
needed
is
we,
we
need
to
be
able
to
discuss,
discover
all
the
resources
that
are
part
of
an
application
stack.
So,
for
example,
here
you
see
in
this
picture,
the
moodle
application
stack
actually
consists
of
several
built-in
resources
like
in-grace
and
stateful,
set
service,
config
map,
persistent
volumes,
persistent
volume
claims
and
there
are
custom
resources,
cluster
issuer
moodle
and
my
sql
cluster,
the
in
order
to
do
accurate
chargebacks.
A
We
need
this
entire
graph
of
custom
resources.
I
mean
resources
of
a
stack,
because,
without
that
entire
graph
and
without
the
ability
to
find
out
all
the,
for
example,
parts
that
are
part
of
a
stack,
we
won't
be
able
to
accurately
find
out.
What's
the
cpu
consumption
of
the
entire
stack,
because,
ultimately,
what
what
we
need
to
find
out
is
what
all
parts
are
part
of
a
application
stack
and
then
find
out
the
cpu
consumption
of
the
containers
that
are
part
of
those
parts
and
the
only
way
you
can
find
out.
A
A
A
Resources
are
related
to
one
another
through
these
various
different
relationships,
owners,
labels,
annotations
or
spec
properties,
and
in
order
to
be
able
to
discover
that
entire
resource
relationship
graph,
we
we
need
to
be
able
to
follow
these
relationships
within
a
stack
and
that's
how
we
will
be
able
to
discover
the
entire
graph
and
then
that's
how
we
will
be
able
to
find
out
all
the
all
the
parts.
All
the
persistent
volume
claims
that
that
are
part
of
that
particular
stack.
A
So
the
way
these
operator
developers
assumptions
need
to
be
captured.
Is
we
have
developed
a
simple
mechanism
to
capture
these
assumptions.
It
consists
of
five
different
annotations
that
can
be
put
or
that
need
to
be
put
on
the
custom
resource,
definitions
or
crds
that
are
packaged
as
part
of
an
operator.
A
These
annotations
offer
a
simple
declarative
way
to
capture.
For
example,
what
sub
resources
will
be
created
by
a
custom
resource
by
an
operator
for
handling
a
custom
resource
or
what
type
of
labels
annotations
or
spec
property
base
relationships
can
exist
between
a
custom
resource
and
other
resources
for
for
for
that
operator,
to
work
to
do
its
work
here.
I
have
three
examples
of
using
these
crd
annotations
with
the
custom
resources
we
have
seen
previously.
A
Well,
the
third
example
is
showing
that
the
cluster
issuer
custom
resource
depends
on
a
specific
annotation
on
the
ingress
resource
for
to
to
do
its
work
and
by
the
way
the
cluster
issuer
custom
resource
is
coming
from.
The
third
manager
operator,
and
it
it
helps
with
generating
ssl
certificates
from
authorities
like
let's
encrypt
and
the
way
it
works,
is
you
need
to
specify
a
particular
annotation
on
the
ingress
resource
and
that
basically
is
captured
through
this
crd
annotation?
A
We
are.
There
is
no
way
to
capture
such
operator
developers
assumptions
today
and
these
annotations
enable
defining
and
capturing
these
annotation
these
assumptions
as
part
of
your
crds.
So
our
appeal
to
my
appeal
to
all
the
operator
developers
is
to
look
at
these
annotations
and
add
them
to
your
operators.
A
A
So
once
these
resource
relationship
graphs
are
discovered
using
q
plus,
we
have
cube
cutting
plugins,
which
can
enable
you
to
do
application
stack
level
chargeback
so,
for
example,
the
cubectl
metrics
plugin
that
is
possible
to
use
that
to
track
cpu
memory,
storage
usage
at
custom,
resource
level
or
at
application
stack
level.
In
in
this
picture.