►
From YouTube: Developing Operators with the Kubernetes Operator Pythonic Framework (kopf) - Sergey Vasilyev
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Developing Operators with the Kubernetes Operator Pythonic Framework (kopf) - Sergey Vasilyev, Zalando SE
The Kubernetes operator pattern has revolutionized the way applications are deployed and managed in Kubernetes, but much of the tooling around building operators has focused on easing development in Golang. The Kubernetes Operator Pythonic Framework (kopf) levels the playing field, bringing much of the tooling to the Python ecosystem, and expanding it in several areas. Kopf provides powerful, high-level abstractions that make it simple to write Kubernetes operators in Python, allowing you to focus on your application logic without needing to dive headfirst into Kubernetes internals. In this talk, you will learn how to make your own Kubernetes operators in a few lines of Python code, and how to bring your own domain entities directly to Kubernetes.
https://sched.co/UaXj
A
A
A
And
I
will
briefly
provide
you
some
information
about
this
company,
so
the
londo
is
one
of
the
biggest
online
retailers
and
the
fashion
platform
in
europe,
with
almost
half
a
million
of
articles
on
sales,
and
we
operate
in
17
countries
also
in
europe
and
technically
so
go
into
the
technical
side.
We
have
something
about
13,
millions
of
300
millions
of
site
widgets
on
the
shop
per
month
and
the
size
of
the
company
is
pretty
big.
A
A
A
It
wasn't
in
the
main
purpose
of
kubernetes.
The
magic
happens
there,
so
everything
just
works
out
of
the
box,
however
kubernetes
actually
is
under
the
hood,
is
an
API.
So
from
the
software
development
point
of
view,
it's
a
restful
api,
a
database
for
the
persistence,
it's
an
access,
control
system,
it's
a
common
line,
toolkit
the
whole
ecosystem
of
the
common
line
toolkits
and
the
UI
applications
of
a
CI
CD
applications.
So
everything
already
is
ready
out
of
the
box.
A
And
if
you
want
to
extend
since
custom
resources,
work
I,
did
it
open
that
as
one
or
two
years
ago,
all
these
api's
and
common
line
tools
and
databases
became
available
for
the
applications
for
the
application
developers,
so
they
can
be
used
to
orchestrate
their
arbitrary
resources
in
the
same
way
as
the
internet,
built-in
resources
like
pods,
you
just
define
a
gamma
file.
You
apply
it
and
voila
you
have
it
all,
also
out
of
the
box,
but
having
a
kubernetes
as
a
database
over
HTTP
just
means
know
what
0
value
to
us.
A
On
top
of
that
some
business
domain
logic
should
be
added,
and
here
come
the
controllers
and
operators
into
play,
and
this
pattern
of
controllers
and
operators
basically
implies
that
some
application
watches
for
the
changes
order
of
the
customer
resources,
notices
them
somehow
in
the
first
place,
and
second,
it
reacts
with
all
changes
by
applying
changes
to
some
other
resources
or
creating
some
other
resources,
or
maybe
the
same
resources
that
trigger
these
changes.
This
reaction-
and
this
is
called
an
operator-
and
basically
this
pattern.
A
This
pattern
of
reacting
and
watching
base
makes
governance
an
Orchestrator
of
everything,
not
just
only
of
containers.
The
common
scenario
for
this
is
to
basically
a
completed
application
deployment
so,
for
example,
so,
for
example,
on
this
elasticsearch
operator
also
made
by
the
Londo.
You
not
only
deploy
elasticsearch
with
a
single
Yama
file.
If
you
have
an
operator
engine,
of
course,
but
also
this
operator
looks
inside
of
this
application
and
measure
some
metrics
and
scales
it
up
and
down
when
needed.
A
But
when
you,
as
a
software
developer,
start
to
develop
your
own
operator,
you
quickly
find
yourself
in
a
in
a
hassle
of
an
infrastructure
holder.
So
you
have
to
talk
to
the
API.
You
have
to
authenticate
with
this
API.
You
have
two
parallel
eyes
and
multiplex
the
events
coming
from
that
API.
You
have
to
marshal
the
data
to
the
functions
and
the
results
back
to
the
resources
you
have
to.
A
You
have
to
do
the
persistence
in
case
the
operator
fails
and
restarts
you
have
to
do
the
login.
You
have
to
do
the
uro
hunting
and
so
on,
a
lot
of
infrastructure
logic,
which
is
not
the
business
domain
of
yours.
And
what
do
we
do
when
we
see
a
lot
of
infrastructure
logic
which
is
not
a
dominated?
We
make
a
framework,
and
there
is
one
already
in
for
the
kubernetes
operators.
So
it's
called
cob
kubernetes.
A
A
So
how
difficult
is
it
to
make?
A
single
operator
turns
out
it's
only
three
lines
of
code,
basically
importing
a
library
connecting
it
to
the
resource
and
the
occasion
when
you
want
to
react
and
writing
a
function
which
accepts
some
data
about
this
resource.
Maybe
some
system
objects
like
the
logo
or
four
lines.
If
you
count
the
Domenic
itself,
which
does
something.
A
How
difficult
is
it
to
run
an
operator?
Well,
you
just
do
on
your
note
on
your
notebook.
You
just
do
cop
run
some
file,
dot,
pi
and
trance,
and
you
can
start
manipulating
the
objects
immediately
and
the
operator
reacts
immediately
on
your
notebook
just
operating
the
resources
in
the
remote
in
the
remote
cluster
that
even
you
can
start
just
in
few
minutes
with
the
operators.
A
Well,
we
assume
that
they're
custom
resource
definitions
are
already
applied
and
now
let
me
quickly
run
you
through
the
features
of
this
framework,
just
to
see
them
which
hassle
doesn't
remove
from
you
and
how
does
it
look
like
in
general
when
you
write
the
operators
so
first
things?
First,
there
is
a
event
handler
the
low-level
event
handler,
which
is
a
closed
equivalent
of
cube
control
gap
watch.
It
basically
connects
with
the
API
and
marshals
all
the
events
to
the
function.
If
the
function
fails,
it
fails.
A
It's
never
admitted
it's
never
right
right
and
you
can
also
get
all
the
all
the
information
about
these
results
in
the
in
the
arguments
of
this
function.
But
cop
also
doesn't
stop
at
this
at
this
point.
At
this
level,
it
goes
beyond
that
it
remembers
and
keeps
track
of
the
objects
state
of
the
objects
last
handled
State
and
detects
the
essential
differences
in
the
in
these
events.
Essentials
means
that
those
that
are
not
technical,
like
not
estoppel,
some
of
the
metadata
may
be,
including
like
labels,
but
adapt
annotations,
but
not
the
rest
of
it.
A
As
I
said,
it
keeps
track
of
the
state
when
the
hundreds
succeed
finally
and
to
these
functions,
you
can
also
receive
not
the
current
state
of
the
object,
but
also
the
old
state
of
the
object
and
the
give
of
the
fields
that
have
changed,
which
is
quite
convenient
when
you
want
to
react
to
the
some
specific
fields
changed
like
step,
dot,
template
or
speck
dot
items,
or
whatever
else,
and
the
most
difficult
part
of
developing.
Basically,
any
software
is
the
Europe
handling,
and
here
cough
also
balls
and
dusted
for
you.
A
It
assumes
that
all
the
eros,
all
the
exceptions
in
Python,
are
just
temporary
and
if
we
try
multiple
times
and
try
try
again,
the
Eden's
will
just
disappear.
Maybe
self
resolved
like
a
connection
euros
and
they
will
be
tried
until
sometime
out
you
can
define
the
timeout
when
the
unload
will
be
considered
totally
permanently
failed
or
some
number
of
referees,
or
you
can
explicitly
say
that
this
handler
failed
completely
and
permanently
forever,
no
need
to
try
it.
A
A
But
what
happens
when
you
cannot
identify?
What's
the
reason
of
the
hero
and
the
important
part
of
the
ease-of-use,
which
is
the
main
value?
One
of
the
main
values
of
this
framework
and
the
human
friendliness
is
that
we
can
reduce
the
breakpoints
and
humans
like
breakpoints,
especially
the
breakpoints
in
their
IDs,
and
since
cops
can
run
locally,
not
only
in
the
cluster
but
locally
on
your
machine.
It
can
basically
use
your
IDE
and
put
the
breakpoints,
and
here
on
the
screen,
you
can
see
that
a
break
two
break
points
are
set.
A
And
there
are
few
other
things
which
are
worth
mentioning:
I
will
not
put
an
additional
slide
for
any
of
them,
but,
as
I
said,
there
is
Python
login,
which
is
directly
mapped
to
the
resources,
little
source
events
in
kubernetes,
but
not
the
whole
login,
of
course,
but
only
the
related
to
them.
To
this.
A
Sorry
resource
and
if
you
have,
for
example,
your
custom
authentication
methods,
your
custom,
very,
very
specific
clusters,
custom
authentication-
you
can
also
implement
that
the
special
handlers
like
unloading
handler
you
can
implement
the
startup
and
cleanup
activities.
If
you
have
the
global
state
managed-
and
you
can
make
this
operator
in
fact,
your
operator-
that
you
write
with
this
framework
embeddable
into
other
applications.
A
So,
for
example,
you
can
write
an
application
which
is
a
UI
for
kubernetes,
which
runs
this
operator
to
monitor
the
state
of
the
objects
and
keep
them
in
the
global
memory
and
just
show
them
somewhere
in
the
web
or
in
the
UI,
and
also
there
is
a
toolkit
for
testing
the
operators.
There
is
no
code
toolkit
for
managing
the
resource
hierarchies
and
more
and
more
and
more,
and
if
you
have
some
wishes.
So
this
is
a
young
framework.
It's
only
six
or
seven
months
old.
A
A
By
the
side
of
these
fancy
features,
let
me
show
you
maybe
three
patterns,
just
only
three,
but
there
can
be
more,
of
course,
but
three
patterns
which
can
be
easily
implemented
with
this
framework.
One
of
the
most
common
pattern
is
basically
creation
of
the
children
as
sources
when,
when
their
parent
resource
is
created
and
monitoring
their
statuses.
So
in
this
case,
which
we
use
in
our
team,
it's
the
job
creation,
also
persistent
vol-plane
creation,
and
then
we
are
not
interested
in
the
status
of
the
jobs.
A
We
are
interested
in
the
statuses
of
the
pods,
which
executed
and
basically
based
on
these
statuses.
We
make
decisions
on
what
to
do
with
the
parent
resource,
how
to
report
its
success
arm
or
failure
or
the
status,
and
the
exchange
in
this
case
is
implemented
via
the
status
field
of
the
objects.
So
here
is
the
code
how
it
can
look
like
the
code
fits
on
there
screaming
for
the
hole.
Okay
right,
okay,
I
remove
their
inverse,
but
generally
the
code
fits
on
the
screen.
You
can
see
that
we
just
create
the
job
resources.
A
A
So
this
is
one
pattern.
The
next
pattern
is
how
to
orcas
in
hope.
Can
they
wear
the
scale
orchestrate
not
only
the
resources
in
the
coop
in
this
class,
but
they
also
can
all
orchestrate
the
resources
outside
of
the
cluster
completely
irrelevant
to
the
cluster,
like
sage
maker,
like
spark
clusters
like
whatever
else
we
are
the
initial
customer
resources
in
the
cluster
and
the
implementation
for
this,
it's
also
simple,
smallest
I
had
to
reduce
their
font
size
to
fit
it
on
the
screen
slightly,
but
still
it
fits
on
the
screen
here.
A
I,
don't
know
who
imagined
to
the
details,
but
here
you
start
at
background
tasks
whenever
the
source
is
created
and
in
that
task
you
pull
the
remote
system
somehow,
but
it
are
every
10
seconds
and
you
have
the
status
and
put
it
back
on
the
resource.
And
this
what
you
see
on
the
screen
is
what
I
call
the
infrastructure
hassle.
This
shouldn't
be
here.
I
couldn't
implement
this
feature
too
fast,
but
this
is
how
it
should
look
like
and
will
look
like
in
maybe
2
1
2
weeks,
just
every
10
seconds.
A
You
can
use
kubernetes
as
this
remote
system,
but
it's
it
is
not
always
safe,
because
if
you,
for
example,
add
to
run
pods
to
an
operator
and
the
token
is
leaked
stolen
by
the
hackers,
the
hackers
can
execute
an
arbitrary
image
or
container
in
your
cluster
and
do
a
lot
of
bad
stuff.
Instead,
you
can
implement
the
operator
so
that
it
runs
in
another
cluster,
where
the
jobs
actually
decoded
and
listens
for
the
resources
for
their
changes
and
put
the
results
back
to
the
original
cast
where
these
resources
are
created.
A
It
is
that
simple
we
just
defined
only
on
login
handler,
where
you
define
the
remote
connection,
information
with
the
tokens,
no
matter
how
you'll
tip
leeteuk
these
tokens
and
that's
it-
it
works
and
we
use
this
well,
we
try
this
solution
as
a
proof
of
concept
in
our
team
tool.
However,
we
now
migrate
to
another
solution,
which
is
much
more
simple,
but
this
is
a
solution
proven
by
practice.
A
There
are
some
still
there
still
some
important
features
which
are
absent,
so
I
have
shown
you
how
to
orchestrate
the
cross
resource
events
across
the
solar
resource
relations,
but
you
have
to
basically
manage
all
these
patching
and
Status
reactions
between
them
yourself.
One
feature
which
is
going
to
be
implemented
little
actively
soon.
Is
that
one
decorator
basically
to
declare
the
relation
between
the
resources
like
the
parent
resource
attica?
Add
children
resources
like
my
application
run,
which
listens
to
the
pods
and
have
it
that
the
same
simple
as
it
was
for
the
timer
on
time
handlers.
A
Another
frequent
request
from
the
users
is
how
to
do
the
reconciliation.
For
example,
we
have
some
remote
system
or
even
kubernetes
object,
and
we
want
to
check
them
from
time
to
time
and
for
that
well,
there
is
already
some
solution.
Partial
solution
is
that
we
can
resume
that
the
cost
of
resource
when
the
operator
is
restarted,
even
if
nothing
has
changed
but
I'm,
showing
you
the
timer
solution
and
also
there
will
be
some
background.
Daemons
else
easily
implemented
with
the
same
decorator
approach.
A
So
another
group
of
topics
is
that
these
operators,
once
they
are
implemented
once
the
framework,
is
finished
with
the
features
the
operators
has
to
be
easily
integrated
into
the
existing
operator
development
kits
like
the
operator,
lifecycle,
management
and
etc,
etc.
So
some
code
generation
will
be
added.
A
It
can
be
used
to
write
the
ad
hoc
operators,
and
ad
hoc
in
this
case
means
that
you
can
write
an
operator
just
for
your
application.
It's
not
reusable
like
phosphorus
or
elastic
search
operators
or
whatever,
just
for
one
specific
use
case
for
one
specific
application,
which
is
probably
in-house,
and
you
don't
worry
that
it
will
be
difficult
to
maintain
it
because,
well
you
don't?
A
You
only
have
the
business
project
there
and
also
it
is
easy
to
write
the
operators
with
Python,
because
now,
as
far
as
I
know,
the
operators
are
usually
written
with
coal,
and
this
basically
excludes
Python
community,
which
is
huge
enough
from
this
field
of
operators,
which
can
benefit
from
Python
community.
They
are
actually
and
one
more
time,
is
that
I
would
like
to
highlight
it.
Ok,
kubernetes
is
the
orchestrator
of
everything
not
only
of
containers,
so
you
can
maybe
forget
about
the
containers
and
orchestrate
your
own
pipelines.
A
A
B
A
Yeah
I
tried,
I,
tried
few
libraries.
Initially
it
was
kubernetes
official
Python
client.
However,
it
is
developed
so
that
all
them
met
with
it's,
not
object-oriented,
so
we
have
create
memes
based
port
method,
a
creating
space
job,
and
you
have
to
keep
the
mapping
of
these
methods
to
the
object
that
you
want
to
manipulate.
It
was
very
difficult
and
I
didn't
want
to
create
this
making
and
then
I
switched
to
PI
cube
in
G,
which
is
more
object-oriented.
A
However,
both
kubernetes
client
and
Hydra
PG
verse
in
central
synchronous
and
they
used
synchronous,
socket
reads,
and
it
was
a
big
problem,
because
there,
the
framework
this
framework
is
a
synchronous
and
basically
scenes.
Let
me
say
two
or
three
days:
there
is
a
release
of
new
of
new
version,
which
is
fully
a
synchronous,
so
Evan
pipe
UPG
is
not
used
as
a
client.
For
that,
however,
what
kubernetes,
client
and
pi
q
kanji
are
used
for
piggybacking
there.
A
C
Have
you
found
any
particular
use
cases
where
you
don't
necessarily
need
the
custom
resource
definition
with
it
like
an
operator
that
does
things
on
resources
that
already
exist
in
the
cluster?
Or
do
you
find
that
really
the
operator
pattern?
There
really
only
works
best
when
you
pair
it
with
a
custom
resource
definition
that
is
keeping
an
eye
on.
A
It
depends
on
what
problems
do
you
want
to
solve?
So
if
you
want
to
orchestrate
something
of
your
own,
then
you
have
customers.
Then
you
use
custom
resources.
If
you
want
to
write
a
controller,
then
you
can
watch
for
the
pod
jobs
deployments
and
all
other
built-in
objects
about
the
built-in
resources
and
that
will
also
work.
So
this
is
not
a
problem.
All
the
resources
are
the
same
for
this
framework.
So
if
you
see
any
idea
of
how
it
can
be
used,
then
you're
welcome
to
write
it
in.
C
A
D
A
A
lot
so
admission
hooks,
for
example,
I
forgot
to
mention
the
admission
hooks
they're,
currently
absent
they're
in
the
backlog.
I
will
add
them
sometime
later.
The
difficulty
is
that
it
will
be.
They
have
to
run
also
when
the
operator
runs
locally
on
the
live
on
the
notebooks,
and
there
should
be
some
port
forwarding
magic
implemented.
A
D
Yeah
Tess
no
I
don't
have
a
particular
task
in
mind.
I
haven't
written
an
operator
before
so
it's
I'd
like
I,
know:
Python
should
I
learn,
go
if
I
want
to
do
an
operator
and
if
it's
just
a
matter
of
expanding
copy
itself,
to
make
it
available
as
opposed
to
for
some
reason
the
API
just
can't
do
it.
Then
that's
fine.
A
E
When
you
want
to
watch
resources
in
your
coordinators
cluster
in
Python,
though,
there
is
no
API
call
for
that,
it's
just
to
watch
for
the
Python
client,
how
you
guys,
implement
and
essentially
use
the
cash
to
make
like
a
good
implementation
of
this,
because
I'm,
assuming
that,
if
you
want
to
watch
bunch
of
resources
and
and
go,
has
these
capabilities
already.
But
it's
not
yet
in
Python,
okay,
yeah.
A
Maybe
I
can't
answer
this
question
because
I
don't
know
what
are
the
information
goal
in
this
framework,
it
is
implemented
as
a
simple
streaming.
Api
call
over
the
HTTP
with
the
watch
equals
true
and
some
additional
magic
with
the
resource
Russian.
So
it
continued
every
time
it
is
reconnected
and
basically
every
individual
resource.
It
has
only
one
connection
for
that.
So,
if
you
have
only
one
resource,
you
will
have
one
connection.
E
A
The
question
is:
what
are
the
most
common
issues
I
see
when
the
when
developing
these
operators
and
deploying
them
well
actually
deploying
them
is
the
most
is
the
biggest
problem
right
now,
because
I
have
to
write
a
lot
of
llamo
files
for
the
deployment.
The
intention
is
to
make
a
little
SDK
to
generate
those
channels
based
on
the
Python
code.
A
So
we
put
I
would
like
to
produce
data
from
the
code,
not
code
from
the
data,
and
other
issues
are
always
surprises
and
complexities
of
talking
with
the
API,
sometimes
disconnects,
sometimes
doesn't
behave
as
as
intended,
but
generally
they're
easily
solvable
just
on
the
go.
There
are
no
big
difficulties
with
this.
F
A
A
D
D
Have
you
ever
had
an
issue
with
the
watch
api
so
with
the
official
Python
SDK
I've
used
like
the
stream
watch
events,
but
sometimes
I
need
to
restart
my
pod,
because
for
some
reason
it
just
can't
see
events
anymore,
how's,
your
frame,
your
operate
operators
using
your
framework
ever
had
that
issue
or
years
I've
had
that
issue.
There.
A
No,
a
few
reports
that
these
watching
sometimes
freeze
my
assumption
currently
that
it
was
because
of
the
synchronous
sockets,
because
there
was
a
clear
issue
when
the
operator
was
terminated.
It
was
hanging
there
for
16
90
seconds
until
this
something
happens
in
kubernetes
in.
If
the
cluster
is
empty
enough,
nothing
happens.
The
socket
remains
in
the
blocking
read
state
and
it
never
ends
and
the
operator
hence
there.