►
Description
Co-chairs: Diane Mueller and Rob Szumanski (Red Hat)
Topics:
Ansible Operator Update – Shawn Hurley
The Ansible Operator is:
- using the new Ansible k8s modules to manage kubernetes resources, allowing for easy templating
- using Ansible Runner to allow for structured output
- using helpers that wil help a user set metadata of a resource in kubernetes i.e. the owner reference
OpenShift Commons Operator Lifecycle Management with Evan Cordell Red Hat
Operator Lifecycle Management offers:
- app store-like experience for discovering and installing operators
- automated upgrades for operators
- framework for building rich, re-usable user interfaces
- package management and dependency resolution
Building Vitess Operator using the Operator SDK – Dan Kozlowski (PlanetScale)
- what is vitess?
- why build an operator?
- common crd patterns
- potential beyond provisioning
A
So
this
is
the
the
operator
framework,
big
meeting
for
August
17th
and
we're
just
waiting
for
everyone
to
join
us,
but
a
very
tight
agenda.
We're
only
going
to
start
about
two
minutes
after
the
hour,
I
guess:
I'll,
wait
and
Shawn
early.
Our
first
guest
speaker
is
here:
he's
gonna,
be
talking
about
the
ansible
operator
that
he's
working
on
or
maybe
has
ready
both
get
an
update
on
that
today
and
are
there
co-chair
Rob
Lansky's,
just
joined
Dan
Kozlowski
will
be
talking.
A
Bands
from
planet-scale,
so
he's
gonna
be
talking
about
the
Vitesse
operator
today
and
we
have
one
other
guest
I,
don't
see
his
join
us
chest.
Evan
Cordell
is
going
to
talk
about
the
operator.
Lifecycle
management,
but
people
who
are
just
joining
and
coming
in
the
room
so
we'll
see,
is
how
people
come
in
and
say
this
is.
A
If
you
haven't
previously
joined
the
Google
mailing
list.
Please
do
so
I
see
Evan
now
my
oven
and
we'll
just
hopefully
you
do
most
of
our
communication
through
the
Google
mailing
list
on
the
Google
Groups
pretty
active,
and
we
also
have
a
slack
channel
and
openness
of
Commons.
If
you
would
care
to
join
that
as
well
that
we
can
hook
you
up
with
so
Rob,
do
you
have
any
quick
words
and
wisdom
before
we
get
started
with
Sean's
talk
and
Sean?
Do
you
want
to
start
sharing
your
screen
and
let's
get
steady,
stirred
up
they're.
B
Just
gonna
say
thank
you
very
much
for
joining
and
don't
have
too
much
to
say
other
than
I
was
going
to
put
a
link
to
a
new
operator
from
Redis
labs
to
operate
Redis
clusters,
which
is
really
cool
and
I
believe
they've
at
least
did
earlier
this
week
or
late
last
week.
So
if
you
want
to
take
a
look
at
that,
the
docket
in
the
channel.
A
I'll
try
and
get
the
reddest
guys
to
come
to
one
of
the
hundreds
of
meetings
and
talk
about
it
too.
So
all
right!
So
if
you're
coming
in
late,
these
add
your
name
to
the
agenda.
Our
first
guest
is
going
to
be
talking
about
the
ansible
operator,
Sean
burly,
who's,
a
red,
Hatter,
I'm
gonna.
Let
you
just
get
started
Sean
and
we'll
roll
along
okay.
C
Hi
everybody
I'm
Sean
Hurley
I
work
at
Red
Hat,
with
a
team
of
guys
working
on
the
ansible
operator,
which
will
be
eventually
be
part
of
the
operator
SDK
so
really
quickly.
We
just
kind
of
want
to
talk
about
why
we
chose
ansible
as
a
for
the
operator.
One
of
the
really
powerful
things
about
ansible
is
that
we're
using
the
new
kubernetes
modules
that
will
manage
communities
resources
it
makes
it
really
easy
to
template,
contains
resources
just
using
ansible.
So
you
can
do
all
the
ansible
things
that
you
can
do
and
also
manage
various
resources.
C
How
you
have
come
to
expect
with
other
tools.
We
also
are
able
to
use
ansible
runner,
which
allows
us
to
run
a
playbook
or
a
standalone
role,
means
that
we
can
reuse
roles
and
just
run
them
right
from
ansible
Runner
as
well
as
a
playbook.
It
also
gives
us
structured
output,
which
will
help
us
help
an
operator
author
manage
their
status
of
their
resource,
and
then
we
also
have
some
helpers
that
will
allow
us
to
kind
of
hide
the
details
of
knowing
about
our
references
and
things
and
metadata
of
a
cover
news
resource.
C
So
I'll
just
quickly
go
through
the
design
of
what
we
got
started
with
so
at
the
top.
Here
we
have
a
user
going
ahead
and
creating
a
custom
resource
where
they
will
define
their
application
state
and
then
the
ansible
operator
will
be
watching
for
that
resource
yeah.
The
inspo
operator
is
based
on
the
operator
SDK,
and
so
it
has
all
of
the
the
go
watch
logic
based
on
the
operator
SDK,
and
then
it
has
a
config
file
which
Maps
the
a
version
kind
of
that
resource
to
the
ansible
playbook
or
roll
the
ansible
playbook.
C
A
roll
is
then
run
whenever
an
event
happens
on
that
custom
resource
and
that
playbook,
a
roll
will
actually
create
the
application
state
in
the
cluster.
So
that
will
create
all
your
resources
that
you
need
and
allows
you
to
manage
your
application
using
ansible
and,
like
I,
said
previously
we're
using
the
the
new
modules,
so
there's
really
helped
out
so
we're
gonna
quickly
talk
about
that
configuration
file
that
I
just
talked
about
we're
a
Maps,
the
group
version
kind
to
a
playbook
or
a
roll.
C
C
So
next
thing
we
can
talk
about
is
how
are
we
going
to
hand
parameters
to
ansible
based
on
the
spec
that
the
user
defines
for
their
resource,
and
so
what
we've
determined
is
that
the
spec
key
values
can
map
well
to
ansible
extra
bars,
and
these
will
allow
us
to
say
a
user
defines
their
spec,
and
then
we
tell
the
ansible
with
that.
What
those
user-defined
parameters
are.
The
ansible
operator
also
will
take
a
first
stab
at
managing
the
status
for
the
operator
author,
as
the
operators
are
supposed
to
own
statuses.
C
So
you
might
be
asking:
how
do
you
end
up
using
the
ansible
operator?
One
of
the
ways
that
we
we
envision
using
the
ansible
operator
is
using
a
base
image
and
then
you're
only
responsible
for
creating
that
config
file,
creating
a
custom
resource
definition
and
then
writing
the
ansible
code
that
you
need
to
do
the
logic
and
work
that
you
want
it
to
do.
C
C
C
C
So
what?
And
we
can
also
look
at
this
docker
file
here,
because
that's
the
other
thing
that
matters
we
can
see
that
we're
using
our
base,
we're
going
to
copy
in
the
busybox
role,
we're
gonna
copy
in
the
playbook
and
we'll
also
copy
in
the
ansible
config,
the
config
file
for
the
ansible
operator,
so
we're
gonna
go
ahead
and
just
deploy
this
operator.
I
already
have
ok,
sorry
about
that.
C
You
can
see
actually
got
our
first
event.
What
is
taking
so
long.
There
we
go
so
now.
Ansible
is
running.
You
can
actually
see
the
ansible
output
right
now
and
we
have
our
busybox
pod
running.
So
we
can
go
ahead
and
see
what
that
guy
is
doing.
As
you
can
see,
it's
just
printing
out
hello
world.
Now,
let's
say
we
wanted
to
change
that
message.
What
we
could
do
is
just
edit
my
CR.
A
C
You
can
see
it's
saying
new
hello
world,
and
so,
if
you
remember,
we
just
took
this
role
and
we're
starting
to
use
it
and
that
role
really
didn't
have
anything
to
do
with.
Let's
say
owner
references.
So
as
an
animal
operator,
author
who's,
just
writing
ansible.
You
might
not
know
that
you
need
to
set
an
owner
reference
to
be
able
to
do
garbage
collection
on
the
ansible
operator.
Actually
has
some
mechanism
to
help
you
with
that.
C
C
C
What
we're
thinking
is
that,
because
the
ants
will
operator
code
will
be
a
part
of
the
upper
address
UK,
you
can
actually
use
the
operator
SDK
ansible
operator
code.
The
same
way
you
would
the
operator
SDK
code.
This
means
that
you
can
actually
compose
pieces
of
ansible
that
you
might
want
to
run
from
the
nor
a
go
operator
that
you're
writing
to
give
you
more
flexibility,
so
you
can
do
things
that
you
would
like
to
do.
So
that's
one
other
way
that
we're
planning
on
allowing
people
to
use
this.
C
So
if
you
want
to
learn
more,
this
is
like
a
pre-alpha
application.
We're
still
very
much
in
the
early
stages
and
it
will
be
PR
will
be
submitted
in
about
one
to
two
months
to
the
operator
SDK
to
to
get
this
fully
baked
in
right.
Now,
it's
in
waterhole
ansible
operator,
there's
a
link
right
there
for
you
guys
to
go
to,
and
most
of
our
team
is
hanging
out
already
in
the
Caretti
slack
in
the
kubernetes
operators
channel.
C
A
B
We
would
say
one
thing
about
the
goal,
for
this
is
really
exciting,
which
is
to
the
technology
skills
that
you
know
an
optical
has
and
so
going.
You
know
a
great
language,
no
can
be
a
little
bit
of
oak's
might
not
have
skills,
or
you
know,
if
you're
in
focus
versus
depth
of
15,
you
shouldn't
have
to
learn
a
programming
language
to
make
an
operator,
and
so
this
is
really
gonna
bridge
those
gaps
where
you
can
start
with
instable.
B
If
you
want
to
start
calling
some
custom
cuts
go
code
and
then
you
can
kind
of
level
up
to
that,
then
maybe
you
know
you'll
fully
transition
to
go,
or
maybe
not
so
that's
really
exciting,
and
so
this
joins
our
helm
operator
kit
and
the
works
much
the
same
way
where
you
just
use
this
base
image
you
not
to
produce
anything.
You
just
add
a
few
files
in
your
chart
and
you're
good
to
go,
and
so
this
will
operate
much
the
same
way
and
which
is
really
exciting.
B
To
make
this
a
wider
audience
that
can
produce
operators
which
is
really
exciting
and
then
the
other
thing
that
ansible
bring
to
the
table,
which
is
really
really
cool,
is
that
there's
this
plethora
of
ansible
already
written
out
there?
And
so
you
can
connect.
You
know,
orchestrating
external
hardware,
load
balancers,
for
example,
all
the
things
you
can
do
in
the
ansible
ecosystem,
with
an
operator
into
your
kubernetes
cluster
natively,
which
is
super
super
exciting.
B
A
D
Thank
you,
I
am
Cornell,
I
am
an
engineer.
It
had
the
operator,
lifecycle,
manager,
team
and
so
I
wanted
to
go
over
some
of
the
philosophy
and
and
architecture
and
implementation,
and
then
do
demo,
oh,
and
because
we
do
get
a
lot
of
questions
about
it.
What
is
it
why
do
I
want
it?
How
does
it
work
and
I?
D
Don't
think
we
really
put
together
anything
for
public
consumption,
that
sort
of
lays
it
all
out
there
for
you
very
well
so
I'm
gonna
start
right
at
the
basics,
but
I'll
go
through
quickly,
so
I
know
we're
in
the
operator
SIG's.
So
everyone
has
an
idea
of
what
an
operator
is
already,
but
for
us
for
operator,
lifecycle
manager.
We
really
think
of
an
operator
as
a
community's
application,
a
community,
it's
native
application,
I
should
say,
and
by
that
I
just
mean
that
it's
not
for
an
application.
D
User
interface
is
the
kubernetes
api,
and
so
all
of
the
assumptions
around
OLM
are
all
around
this
idea
that
you
interaction
with
an
application
via
the
communities
API
and
and
orchestrating
on
via
the
communities.
Api
I
was
going
to
put
a
picture
of
Jack
Sparrow,
but
I
wasn't
so
sure
about
that
copyright
issues
there,
but
so
like.
D
We
all
have
this
idea
of
an
operator
instead
of
control
loop,
so
watch
T,
IDs
and
that's
all
well
and
good,
and
you
need
those
things
but
as
far
as
we're
concerned,
that's
all
implementation
detail
of
this
concept
of
extending
the
community's
API
and
if
your
green
ideas
comes
out
with
better
tools
for
doing
that
in
the
future.
Well,
don't
keep
up
with
those
same
exact
way.
So
we
know
what
an
operator
is.
D
What
is
operator
lifecycle
well
I'm,
going
to
propose
this
definition
that
the
operator
lifecycle
is
the
definition
or
specification,
the
installation,
the
resolution
by
which
I
mean
dependency
resolution,
the
upgrading
and
the
automated
upgrading
and
then
the
removal
of
operators.
So
these
are
all
the
responsibilities
of
the
operator
lifecycle
manager
are
defining
and
managing
these
different
aspects
of
an
operator
over
its
lifetime
in
a
cluster.
D
So
what
did
some
of
the
goals
of
the
operator
lifecycle
manager?
This
is
what
we're
trying
to
provide
with
the
piece
of
software
called
olan.
We
want
to
give
you
an
app
store
like
experience
for
discovering
and
installing
operators
so
what's
available
to
you
and
what
will
work
when
you
run
it
on
your
cluster
and
then
also
simplifying
the
installation
of
an
operator
down
to
just
a
single
coop
level
of
pliers
and
go
like
in
a
user
interface.
The
other
aspect
is
automated
upgrades.
D
This
is
some
philosophy
that
we've
carried
over
from
for
us
continued
Linux
techtronic.
We
wanted
that.
We
know
that
there
is
value
in
updating
things
automatically,
because
we
can
ship
out
fixes
to
any
problems
as
soon
as
we
know
about
them.
So
we
want
to
bring
that
philosophy
into
operators
as
well.
D
Ideally,
when
you're
writing
your
own
operator,
you
don't
need
to
bender
in
15
other
operators
just
to
get
the
orders
up
and
running.
We
have
lessons
around
how
a
dependency
resolution
can
work
from
other
ecosystems
and
we'll
take
those
lessons
and
apply
them
to
operators
as
well.
So
that's
some
of
the
philosophy.
Why
we're
doing
this?
What
we're
trying
to
get
out
of
it?
Let's
go
over
the
actual
architecture
at
a
high
level.
Right
now,
om
is
defined
by
two
separate
operators.
D
D
It
manages
these
objects
that
we
call
catalog
sources
which
have
virtual
objects
of
packages
and
then
subscriptions
and
install
plans
as
well
I'm
going
to
exactly
what
each
of
those
is,
but
only
to
emphasize
that
we
have
a
layered
architecture
where
one
thing
has
very
specific
responsibilities,
which
is
just
the
cluster
service
version,
and
then
another
operator
has
a
different
set
of
responsibilities
and
overlays
those
on
top
of
the
other.
So
what
is
a
cluster
service
version?
This
is
our
specification
file
or
what
an
operator
is
and
how
it
should
run.
D
It
is
100%
analogous
to
something
like
a
D
package
definition
or
an
RPM
if
you're
talking
about
operating
system,
dependency
management,
dependency
management,
it's
also
perfectly
now
I
guess
to
set
up
top
PI.
If
you
have
a
Python
project
or
a
brew
formula
using
homebrew
or
Mac,
it's
the
definition
of
MIT
metadata
about
how
to
install
and
run
in
the
orchestrate
piece
of
software
on
the
underlying
system.
D
Specifically,
we
have
a
set
of
like
textual
metadata
things
like
description
who
created
it
where
to
come
from
how
to
get
support,
that's
fairly
common
among
package
managers,
but
specifically
for
kubernetes.
What
what
do
we
need
to
define
an
operator
we
have
specifically
inspect
could
be
a
set
of
the
plan
inspector,
but
these
are
literally
defining
the
pods
and
deployments
that
will
be
running
the
operator
software
itself
and
then,
aside
from
that,
the
only
information
we
really
need
to
know
is
the
series,
as
the
Sierra
T's
defined,
that
interface
for
the
operator
in
kubernetes.
D
So
we
say
that
each
operator
can
declare
that
it
owns
a
set
of
CR
DS.
These
are
the
ones.
This
is
essentially
an
operated,
declaring
that
to
the
Oval
and
framework
that
it
will
be
responsible
for
the
life
cycle
of
particular
C
IDs
and
then
there's
a
set
of
required
CI
use-
and
this
is
a
list
of
series
not
that
the
cluster
service
version
will
own,
but
that
the
operator
will
interact
with
in
some
other
way.
D
The
rest
of
our
dependency
management
is
hung
on
Toby's,
doomed,
concepts
of
owned
and
required
series,
because
here
these
are
the
interfaces
I'm.
For
each
of
these
things,
we
have
what
we
call
descriptors,
which
are
essentially
indexes
into
the
C
IDs.
It's
somewhat
similar
to
some
of
the
newer
features
in
coop
like
the
weather,
Cup
and
the
additional
printer
columns
and
CR
DS.
Where
you
you
go
into
an
object,
and
you
give
some
additional
metadata
about
that.
So
this
is
how
we
drive
some
of
our
UI.
D
This
is
how
we
drive
some
of
the
DOI
interaction
as
well,
so
the
OLM
operator
is
responsible
for
within
every
namespace,
managing
the
set
of
cluster
service
operations
that
exists
so
for
the
most
part,
those
will
be
independent
operators,
but
I
wanted
to
call
out
what
happens
when
you
are
in
the
in
the
upgrading
state
or
an
operator
in
which
you
have
to
cluster
service
operations
that
define
the
same
operator
but
at
different
versions.
Some
of
the
metadata
that
we
collect
for
operators
is
which
one
it
replaces.
D
D
So
that's
essentially
the
only
responsibility
of
the
OLM
operator,
the
other
operator,
which
we
call
the
catalog
operator,
which
has
nothing
to
do
with
Service
Catalog.
It
is
responsible
for
these
other
resources.
The
primary
one
is
the
install
plan
and
the
install
plan,
similar
to
our
analogy
for
cluster
service
versions.
The
install
plan
is
analogous
to
using
yum
or
apps
for
orchestrating
packages
on
an
operating
system
or
pip
or
using
if
you're,
using
Python
with
the
setup
dot.
D
I
file
I
also
wanted
to
call
out
that
the
install
plan
is
also
analogous
to
various
log
files
that
you
find
from
various
ecosystems,
because
it
takes
a
high-level
desired
user
state
and
translates
that
into
a
ideally
immutable
set
of
references
that
we
can
apply
to
other
namespaces,
get
the
exact
same
set
of
things
I'm.
So
an
install
plan,
like
I
said,
has
an
input.
That's
essentially
just
the
desired
SIA
speeds.
You
can
have
multiple
install
plans
for
a
namespace
or
you
can
have
multiple
CSU's
defined
in
an.
A
D
Plan
this
trigger
is
the
operator
to
go
in,
find
whatever
CSV
is
this
here.
These
it
knows
about
whatever
definitions
that
have
so.
These
are
not
things
that
are
installed
in
the
cluster
yet,
and
then
it
writes
out
the
definitions
of
those
into
the
status
block
of
the
install
plan
and
then
another
control
loop
waits
for
install
plans
to
be
completely
resolved
and
then
applies
those
resources
into
the
cluster.
D
So
where
does
an
install
plan
to
find
the
CSV
sincere
DS?
We
have
a
set
of
objects,
called
catalog
sources
that
define
what
CFCs
in
series
are
known
and
then
organizes
those
CFCs
and
series
into
packages
and
channels
which
is
just
an
organization
mechanism
for
saying,
for
example,
these
50
DSPs
all
are
installing
the
NCD
operator
that
will
call
that
a
TD
package
and
then,
within
that
chipset
of
50
there's
a
chain
of
20
of
them
that
are
in
the
alpha
channel.
D
We
are
the
beta
Channel,
so
this
is
just
a
way
to
collect
and
organize
all
of
the
definitions,
things
that
are
available
to
the
user
to
install
and
then
once
we
have
those
two
things.
We
have
an
additional
object,
called
the
subscription
and
subscriptions
signal
to
the
operator
to
a
catalogue
source
frequently
for
updates
and
then,
whenever
an
update
occurs,
to
create
a
new
install
plan
or
that
object.
This
is
our
full
sort
of
graph
of
objects.
D
If
you're
subscribed
to
a
package,
for
example,
a
TV
on
you
specify
an
action,
a
channel
that
points
to
a
catalogue
source.
So
our
operator
goes
and
looks
at
the
catalogue
source
for
any
updates
whenever
it
finds
an
update.
It
creates
an
install
plan
with
the
new
versions
of
the
csps
defined.
We
have
another
controller
looking
for
new
install
plans.
It
sees
that
there's
a
new
install
plan
with
the
new
desired
CSV.
D
It
resolves
that
from
any
of
the
catalog
sources
that
it
needs
to
talk
to
and
write
those
back
into
the
status
of
the
install
plan.
Then
we
take
those
here
as
well
as
he
has
to
be
sincere
deeds
and
actually
apply
them
to
the
cluster,
and
once
that
has
done
everything
else
is
handled
by
the
OLM
operator.
So
there
will
be
a
giant
set
of
poster
service
durations
in
a
namespace,
and
some
of
those
might
be
new,
some
of
those
I
removed.
D
Some
of
them
might
be
upgrading
from
old
versions
and
the
OLM
operator
handles
all
those
migrations
for
you
and
also
creates
the
you
see
are
these
themselves.
So
you
end
up
with
a
state
in
your
namespace,
where
you
have
all
your
Seri
needs
to
find.
These
are
all
of
the
new
API
interfaces
you
can
use
via
group
cuddle
or
the
CI.
So
now,
I'm
going
to
flip
over
to
a
live
demo
of
this
I
have
a
namespace.
This
is
an
open
ship.
D
311
cluster
I
have
two
I,
have
one
catalog
source
right
now
and
it
has
to
do
packages
to
find
they
happen
to
have
just
one
panel.
At
the
moment,
when
I
create
a
subscription,
I
am
generating
a
subscription
for
the
sed
package
from
my
particular
catalog
source
and
then
telling
it
to
track
a
particular
channel
great
that
we
have
a
UI
for
it,
but
it
has
actually
created
an
install
plan
in
the
back.
The
install
plan
was
created
by
the
operator
because
of
my
subscription.
D
In
this
case
it
didn't
automatic
approval,
so
the
install
plan
is
immediately
applied.
We
also
have
manual
approval,
which
requires
a
user
with
right.
Access
to
go
in
and
manually
approve
an
install
plan
before
it
creates,
and
then
we
specified
a
specific
cluster
service
version
that
one
and
then
in
the
status
blog
it
actually
goes
and
puts
the
definitions
that
will
eventually
be
applied
in
the
UI.
D
You
can
go
and
see
that
in
this
case
this
particular
cluster
already
had
these
particular
@td
API
versions
installed
as
series
and
then
the
one
thing
that
wasn't
existing
when
the
namespace
was
the
actual
CSV,
and
so
that
was
created
for
us.
If
I
go
back,
the
operator
page
I
can
see
that
this
is
representing
the
sed
CSV.
Well,
I
can
go
look
at
the
yam.
Well
for
that,
and
so
the
sed
defines
a
set
of
own
custom
resource
definitions.
D
The
sed
one
does
not
require
any
other,
so
it's
it
doesn't
depend
on
anything
else,
but
it
does
describe
the
deities
that
it
uses.
So,
for
example,
here
is
a
spec
descriptor.
So
these
are
the
descriptors
I've
used
to
drive
the
UI.
This
says
that
if
you
go
into
and
that's
a
deep
luster
object
and
you
look
at
the
spec
that
science
app,
you
can
expect
it
to
be
of
this
particular
type
and
this
particular
type
is
for
us.
A
pot
count
object.
It
has
a
description
here,
but
we
can
go
into
I.
D
Can
go
and
create,
and
it's
a
neat
cluster
and
once
I
do
that
I
am
driving
the
UI.
This
is
generated
based
on
that
descriptor.
So
this
is
saying
that
the
size
is
a
pod
count
and
so
I
know.
Ui
has
the
switch
editor
I
can
edit
a
pod
count.
Server
cluster
slightest
I
can
increase
the
size
of
five
pods
and
that
gets
translated,
vacuumed
the
yellow
on
the
back
end.
D
D
But
here
we
go
a
fourth
benefit
as
it's
adjusting
to
my
poster
size
of
five,
but
we
know
that
these
are
the
objects
that
it's
creating
based
on
the
Itsuki
Buster,
and
if
I
go
and
I
delete
the
entity
cluster
myself,
it
will
eventually
GC
all
of
the
related
resources
for
me.
So
that's
what
the
when
Shawn
was
talking
about
that
around.
A
B
D
Ideal
scenario
is
that
there's
some
like
persona
split
between
an
administrator
and
a
user,
and
so
the
administrator
would
go
instead
of
namespaces
with
the
operators
installed
for
you
and
then,
as
a
user,
you
just
go
in
and
you'll
see.
You'll
have
that
are
back
access
to
create
an
entity
post.
You
just
go
in
and
create
one.
That's
that's
all
the
interaction
you
have
to
feel
it.
So
it's
up
to
the
administrator
to
set
up.
You
know
things
like
subscription
policy,
whether
or
not
a
particular
name
state
should
be
updating
constantly
or
not
effective.
D
D
A
So
next
up
we
have
den
cause
or
cousin
Belsky
I.
Let
you
pronounce
your
last
name
and
from
planet-scale
is
going
to
talk
about
his
attests
greater,
but
him
introduce
himself
and
what
he's
been
doing
and
and
I'm
just
gonna
tell
him
make
sure
your
font,
size
and
terminal
is
a
little
bigger.
As
high
America
fell,
it's
kind
of
small.
E
E
All
the
tests
first
thing
I
wanted
to
do
is
sort
of
go
over
a
little
bit
of
what
Vitesse
is
before
I
jump
into
our
operator
and
then
what
we
did
here
is
we
built
a
few
versions
of
kubernetes
operators
and
so
I
I
called
this
the
tale
of
two
operators,
but
there's
actually
like
four
of
them
and
I'll
get
into
why
we
did
what
we
did
as
well.
As
a
few
lessons
we
learned
and
a
few
challenges
that
we're
currently
facing,
but
first
things.
E
E
So
what
it
is
it
splits
a
single
database
into
a
number
of
shards.
Each
of
those
shards
has
a
number
of
replicas
and
it
gates
each
my
sequel
process
with
a
Vitesse
process
that
we
call
a
VT
tablet
and
then
it
provides
a
gateway
which
looks
like
a
normal,
my
sequel,
server
to
a
client.
So
at
the
end
of
the
day,
you
have
a
gateway
layer
that
is
horizontally
scalable,
you
have
a
sharding
layer
behind
it.
That
is,
can
have
as
many
shards
as
you
want,
and
you
have
a
control
plane.
E
E
So
that
is
that
is
with
has
some
interesting
things
about.
It
is
because
this
was
developed
at
Google.
It
was
developed
to
run
on
Borg,
which
is
the
inspiration
for
kubernetes.
So
this
has
been
running
for
years
in
something
kind
of
like
kubernetes.
So
when
we
took
a
look
really
making
it
run
well
in
kubernetes,
it
was
a
pretty
natural.
Thick
now
been
developed
for
almost
eight
years
to
to
work
in
that
environment.
E
E
Everything
has
been
designed
to
sort
of
fit
together
and
scale
independently,
and
there
are
a
lot
of
different
components
that
need
a
lot
of
different
configuration
options
based
on
how
you
do
it,
and
a
lot
of
that
can
be
abstracted
away
into
a
very
simple
cluster
architecture,
also
because
it
has
all
these
various
components
in
various
layers.
You
need
to
do
more
than
just
bring
up
containers,
so
you
can
get
a
mostly
working
Vitesse
cluster,
just
by
using
straight
kubernetes
manifests.
E
But
there
are
a
few
steps
that
have
to
be
taken
before
you
spend
up
certain
components
and
then,
after
you
spend
up
certain
components,
really
make
sure
that
everything
can
talk
to
each
other.
So,
in
that
case,
it's
a
prime
candidate
for
an
operator,
since
it's
really
close
to
really
close
to
just
straight
kubernetes,
deploy
with
just
a
little
bit
of
stuff
on
top,
and
in
fact
this
project
was
used
with
test
was
used
to
demo
a
few
other
I
guess
operator,
operator
technologies.
E
E
We
looked
around
at
what
we
wanted
to
do,
and
these
are
really
the
reasons
why
we
chose
to
go
with
make
yet
another
operator,
though,
when
we
are
talking
about
running
a
database
in
production,
we
wanted
to
be
more
than
just
provision
it.
We
also
wanted
to
handle
a
lifecycle
events.
So
if
you
update
your
version
of
my
sequel,
we
don't
just
want
to
bring
down
all
your
database
servers
and
then
turn
them
all
back
on.
E
There's
a
sequence:
that's
we
want
to
follow
and
know
a-rollin
deploy,
wouldn't
necessarily
do
things
in
the
right
order,
so
we
knew
we
needed
a
little
logic
there.
We
also
wanted
to
be
able
to
handle
what
I
have
called
the
deep
state,
and
that
is
things
like
your
schema
inside
your
database
to
a
lot
of
the
other
technologies.
E
There
is
no
visibility
into
actually
what's
going
on
inside
the
database,
so
when
it
comes
to
things
like
doing
DDL
transformations,
if
we
wanted
to
assist
in
that,
we
were
going
to
need
something
that
actually
had
very
fine-grained
control
and
was
very
extensible,
then
the
the
last
reason
why
we
decided
to
go
with
the
operator.
Sdk
was
for
code
reuse.
So
the
way
we
have
our
project
architected.
E
E
We
used
the
most
basic
state
reconciliation
that
there
is
in
that
we
just
used
the
name
collision
protection
that
qu
Rani's
offers
to
make
sure
that
we're
not
recreating
things.
Elements
in
our
operator
had
deterministic
names.
So
when
we
try
to
the
second
time,
we
just
got
a
already
exists
there
and
what
about
our
business?
And
because
of
that,
we
were
able
to
pretty
much
eliminate
all
the
internal
state,
and
that
was
one
of
our
first
design
goals.
Was
this
operator
should
be
completely
stateless?
It
should
come
up
given
a
CR
D.
E
It
should
just
provision
that
cluster
directly.
So
this
was
the
c
rd
that
we
came
up
with
and
it's
a
direct
transliteration
of
what
a
Vitesse
cluster
looks
like
so
in
the
test.
There's
this
concept
of
a
cell,
which
is
a
failure
domain.
If
we
were
running
in
AWS,
this
should
be
an
availability
zone
or
even
a
region,
and
then
key
spaces
are
actually
your
database
key
spaces
have
replicas
or
key
space
have
shards,
which
is
the
actual
splitting
of
your
entire
database
into
various
parts.
E
E
E
We
decided
on
a
few
initial
improvements,
so
the
biggest
change
in
the
biggest
sort
of
improvement
over
that
original
CRD
is
incorporating
the
selector
pattern
wherever
we
can-
and
this
is
something
that
I
think
you'll
see
a
lot
through
operators
that
people
have
made
so
the
place
where
I
I
think
the
best
example
of.
If
you
look
at
the
prometheus
operator,
the
Prometheus
operator
uses
what
I
will
call
the
selector
pattern,
we're
actually
defining
the
service
electors,
so
you
can
put
in
as
many
different
service.
E
E
That's
what
our
extremely
large
CR
de
and
turned
it
into
something
that
looks
like
this,
and
so
now
we
had
a
number
of
CR
DS
that
had
these
selectors
on
them,
and
there
was
much
rejoicing
right
because
that
solves
a
number
of
problems
you
can
have
a
you,
can
have
different
components
that
are
defined
sort
of
independent
of
each
other.
I
can
now
change
my
cells
without
read,
without
changing
my
cluster
and
I.
E
We
had
only
moved
the
problem
around
a
little
bit.
You
still
had
to
do
a
lot
of
definition
to
get
a
Vitesse
cluster
up
and
running
so
for
improvement.
Number
two.
We
decided
well
how
about?
We
also
add
defaults.
So,
instead
of
having
to
define
all
of
your
CR
DS,
you
can
give
some
defaults
and
let
the
operator
to
find
the
CR
DS
for
you
sort
of
do
parameter.
Expansion
inside
the
operator,
instead
of
explicitly
having
to
you,
know,
generate
all
of
the
all
the
shard
information
and
load
each
of
those
into
the
operator.
E
We
can
just
give
a
sane
set
of
defaults
and
say
how
many
of
those
things
we
wanted
and
then
let
the
operator
create
them
for
us.
So
this
would
be
if
you
decided
that
you
wanted
to
have
a
replica
on
my
sequel,
replica
running
a
specific
container
listening
on
a
specific
port
with
a
specific
TLS
secret,
you
could
just
define
that
container
profile
once
and
then
tell
the
operator
I
want
each
of
my
shards
to
be
running
ten
of
these
replicas
and
then
the
operator
would
spin
up
10
of
these
replicas.
E
This
was
a
really
good
exercise
for
us,
as
we
went
through
the
second
CRD,
because
it
made
us
be
very
explicit
about
how
we
were
going
to
configure
things
and
where
the
configuration
points
were
so
when
we
have
to
define
the
configuration
points,
then
we
can
go
through
and
make
sure
that
we
have
flexibility
in
actually
defining
the
cluster.
We
can
say
if
we
wanted
to
do
half
of
the
cluster
defined
explicitly
and
of
it
expanded
from
defaults.
E
We
wanted
to
build
that
in
so
that
made
us
then
go
through
and
think
about
how
we
were
gonna
be
able
to
do
that
at
the
end
of
that
whole
process.
We
ended
up
with
a
cluster
that
can
look
like
this,
so
that's
64
dark
cluster,
that
from
our
initial
CRT
it
actually
get
collapsed
down
into
this.
This
would
be
a
cluster
definition
with
64
shards,
two
replicas
per
shard
in
every
cell.
That
was
labeled
super
awesome
cluster
and
here
I
only
have
one
to
find.
E
E
The
third
improvement
is
when
we
actually
haven't
made
yet-
and
this
is
our
current
challenge-
that
we're
having
the
improvement
that
we
are
currently
targeted.
Working
on
is
using
the
amount
of
calls
we
have
to
the
kubernetes
api.
It
turns
out
that
when
we
get
these
extremely
large
clusters,
we
found
that
we
were
making
a
lot
of
calls
to
the
SDK
functions,
so
SDK
create,
as
well
as
a
lot
functions,
to
get
the
state
of
running
containers
to
make
sure
that
we
weren't
duplicating
things
and
then
internal
to
the
test.
E
So
we
wanted
to
make
sure
that
we
could
reduce
that
and
that's
currently
what
we're
in
the
works
of
doing
and
we're
trying
to
figure
out
how
we
can
make
sure
that
we're
reconciling
correctly
so
that
when
outage
occurs
or
something
happens,
to
the
resources
that
they're
getting
reaper
visioned.
In
short
order,
but
then
also
not
bringing
down
the
api
server
because
we're
making
so
many
API
calls,
so
that
is
sort
of
the
third
improvement
that
we
are
currently
working
on
and
I
know
on
the
mailing
list.
E
Other
there
has
been
some
chatter
about
this
specific
topic:
students.
In
addition
to
just
concerns
about
stability.
There
was
also
concerns
about
cost
and
other
things
to
consider
when
we
were
really
instrumenting
these.
These
operators
that
are
making
a
lot
of
automated
calls
back
to
the
API
masters.
So
those
are
the
improvements
that
we
made
version
2
and
then
looking
at
version
3
or
evaluating.
E
If
we
could
do
some
of
these
things
with
the
operator
SDK,
which
sort
of
go
beyond
what
you
could
do
with
just
straight
kubernetes
deployments,
so
online
schema
transformations
are
a
big
one.
For
us,
a
lot
of
people
may
be
familiar
with
the
tool
ghost
from
github.
That
works
really
well
with
the
test,
and
we
want
to
make
sure
with
an
operator,
it's
a
great
opportunity
to
just
make
that
even
more
seamless,
so
that
your
schema
transformations
can
be
completely
automated.
E
We
would
want
to
be
able
to
deploy
in
different
data
centers
around
the
world
and
make
that
as
easy
as
possible.
So
right
now
the
path
the
people
I'm
looking
at
is
write
multiple
kubernetes
clusters
and
then
just
orchestrating.
On
top
of
that,
and
since
we're
running
effectively
a
go
program,
we
have
options
on
how
to
coordinate
across
the
kubernetes
deployments.
E
Another
great
thing,
which
I
actually
think
I
just
saw
in
the
the
OLM
operator,
is
I.
Haven't
here's
chat
integration,
but
really
it's
human
intervention.
So
when
events
occur,
like
upgrades
or
life
cycle,
events,
don't
do
them
without
approval,
and
that
is
something
that,
especially
when
you
consider
databases
and
where
they
sit
in
your
application
stack.
The
idea
of
requiring
human
approval
is
something
that
a
lot
of
people
are
are
insistent
upon.
E
So
again,
because
we
do
have
a
lot
of
integration
points,
we
can
do
chat
integration
or
we
could
do
a
ticketing
system
integrations
that
you
can
actually
give
approval
before
events
are
taken
and
then
the
last
thing
sort
of
attest
specific
is
auto
very
sharding.
Since
it's
a
sharded
database,
you
can
expand
the
size
of
the
cluster
because
you
would
like
and
and
that's
something
that
we
can
also
do
the
operator.
It
that's
got
problems
with
other
style
of
operators,
because
it
is
a
multi-step,
sometimes
multiple
day
thing
that
happens.
E
But
it's
you
know
with
the
with
the
operator
SDK.
We
can
maintain
state
in
a
global,
lock,
server
or
in
some
external
entity.
We
can
do
those
big
multi-day
lifecycle
events,
so
that
is
the
story
of
our
two
operators
of
sort
of
our
path
through
our
path
through
creating
them.
The
tax
is
an
open-source
product.
So
if
you
want
to
take
a
look
at
it,
it's
Vitesse.
Do
we
hang
out
on
slack
that
second
link?
E
A
D
B
E
C
Hey
this
is
Shawn.
Just
a
quick
note.
I
think
that
the
controller,
runtime
design
story
and
operator
SDK
might
be
an
issue
worth
watching
for
you
I
think
this
should
help
with
that
problem,
because
I
believe
the
controller
runtime
will
start
to
use
an
informer
cache
to
do
get
and
list
requests
which
will
probably
significantly
help
in
this
regard.
But
that's
just
a
guess
from
a
guy,
so
I
don't
really
know.
If
that's
true
I.
E
We
use
the
NCD
operator
well,
it
depends
so
for
the
sed
cluster
that's
associated
with
each
cell,
which
is
the
failure
domain.
We
just
provision
that,
with
the
sed
operator,
we
ask
that
you
load
the
sed
operator
before
you
load
our
operator,
so
that
we
can
just
we
can
provision
that
the
sed
CRD,
the
global
one's
a
little
bit
different
since
Vitesse
ken's
vancouver
nineties
clusters.
The
sed
operator
doesn't
quite
work
there,
so
we
just
have
some
custom
code
to
spin
up
at
seedy
cross
region
at
seedy,
cluster
I.
E
So
the
Etsy
be
the
global,
lock
server.
The
global
lock
server
is
the
coordination
point
for
multi
cluster
or
communication.
So
Vitesse
knows
how
to
run
a
completely
isolated
data
centers
if
there
doesn't
need
to
be
any
synchronization
between
them,
but
that
global,
lock
server
does
need
to
be
accessible
by
each
cluster
and
does
need
to
maintain
stay
for
all
of
them.
That's
why
we
provision
that
outside
of
kubernetes,
we
could
also
early
demos
that
we
did.