►
Description
Fission: Serverless Functions for Kubernetes [B] - Soam Vasani, Platform9 Systems
"Serverless" functions allow users to easily create services from source code without dealing with packaging, deployment, scaling, etc.
Fission is a serverless function framework built on Kubernetes. Users write functions and map them to event sources, such as HTTP requests, timers, Kubernetes watches, and message queues. Users don't have to deal with container images, registries or even learn Kubernetes in much detail.
Functions consume CPU and memory resources only when running. Fission makes on-demand function loading very fast by keeping an idle pool of containers running, in effect creating a distributed "threadpool".
Fission is useful for:
* Creating web app backends or REST APIs
* Implementing webhooks
* Watching Kubernetes Resources for changes to implement simple custom
controllers
* Creating Kubernetes Initializers with minimal work
In this talk we'll give a few demos for these use cases, and cover:
* How fission works with function dependencies in a language-agnostic manner
* The development lifecycle of Functions: testing and incremental deployment
* The composition of functions into workflows using the Fission Workflow project
* Observability: log aggregation, metrics with Prometheus, tracing with Opentracing/Zipkin
* The interaction of functions with service meshes (Istio/Envoy)
About Soam Vasani
Soam Vasani created and works on the Fission framework at Platform9 Systems. He's also worked on Platform9's Kubernetes cluster deployment and management product. His past work includes distributed filesystems, a log analysis stack, and infrastructure management products; as well as the GNU debugger and toolchain. He's interested in distributed systems, devops tools and frameworks, and programming languages.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Welcome
to
our
talk
about
cgi-bin,
which
is
definitely
a
really
new
technology.
It
makes
developing
web
applications
really
simple.
All
you
have
to
do
is
drop
a
really
simple
pearls
clip
script
into
a
certain
directory
called
CGI
bin
on
your
web
server,
and
and
that's
it
it's
deployed,
and
you
can
just
run
it
from
whatever
path
it's
at
so,
unfortunately,
our
talk
is
not
exactly
about
CGI
bin.
It's
about
functions
as
a
service,
which
is
definitely
a
new
technology
for
kubernetes.
A
This
talk
is
about
phishing,
so
so,
on
a
more
serious
note,
it's
been
more
than
two
decades
in
cgi-bin
and
we
lost
that
simplicity
that
it
had.
So
today
you
have
this
long
series
of
steps
build
a
container
push
it
to
a
registry
figure
out.
Your
kubernetes
configuration
make
sure
that
luster
is
set
up
correctly
and
then
repeat
some
of
these
steps.
Every
time
your
your
app
changes
rebuild
the
container
push
it
figure
out
how
your
versions
run.
So
this
is
not
to
say
that
this
new
world
is
terrible.
We've
gained
a
lot.
A
A
Rarely
so
so,
once
you
have
enough
of
those
services,
your
cluster
capacity
has
to
account
for
all
of
the
services
that
you've
deployed
and
our
idle,
and
also
for
the
services
that
you
that
are
actually
loaded,
which
are
the
ones
that
you,
which
are
the
only
ones
that
you
really
would
like
to
pay
for.
So
ideally,
the
services
that
are
idle
should
be
free.
So
what
if
we
could
solve
both
these
problems?
A
What
if
we
could
have
dinner's
and
the
power
of
containers
in
kubernetes,
but
really
simple
their
workflows,
and
what,
if
we
could
have
our
cluster
capacities,
be
more
directly
proportional
to
our
to
our
actual
usage
and
the
answer
to
that?
The
answer
to
both
of
those
things
is
functions
as
a
service.
A
It's
one
of
the
really
good
answers
to
both
of
these
questions
and
to
focus
on
one
of
the
points
in
functions
as
a
service
is
that
if
you
want
services
to
be
free
when
they're
idle
and
if
you
also
want
them
to
have
good
performance,
especially
latency
when
when
they
do
get
a
bunch
of
requests,
then
you
need
to
make
sure
that
you
go
from
no
instances
to
enough
instances
really
quickly
when
your
traffic
does
come
in.
So
this
brings
us
to
fishing.
Fishing
is
a
functions
as
a
service
for
kubernetes,
the
user.
A
You
write
short
lists,
short-lived,
stateless
functions,
you
define
them
declaratively
and
at
the
source
level,
we'll
talk
much
more
about
that
and
they
are
free.
When
idle,
you
only
pay
for
the
storage
of
those
functions.
They
consume
CPU
and
memory
only
when
they're
running
and
they
started
quickly
on
demand.
So,
as
a
user,
you
have
only
three
concepts:
to
learn:
inflation
functions,
environments
and
triggers
functions
more
properly,
their
modules
that
the
entry
point
is
a
function.
A
The
function
runs
inside
an
environment,
so
an
environment
efficient
is
all
of
the
language
specific
stuff
that
that's
in
a
all
of
the
language.
Specific
stuff
efficient
is
in
environment,
so
you
have
no
Jersey
and
an
environment
is
a
container
which
loads
a
function.
Dynamically
and
a
trigger
is
a
mapping
of
an
event
to
a
function.
Call
so
HTTP
triggers
map.
Http
requests
to
function
calls
their
message,
q,
triggers
and
so
on.
A
So
let's
get
a
little
bit
into
how
fission
executes
functions,
and
this
is
one
of
the
ways
fission
execute
functions,
but
it's
the
more
interesting
way.
So
we'll
talk
a
little
more
about
this.
Fundamentally,
the
way
to
get
fast
cold
starts
is
by
having
a
pre
warmed,
pool
of
something
and
since
efficient
runs
on
kubernetes.
It
has
a
pre
one
pool
of
containers
running
in
parts,
and
so
there's
a
fishing
client
and
fish
and
resources
are
stored
as
kubernetes
custom
resources.
So
the
client.
A
Let's
lesson
you
know
it
functions
as
these
colorful
circles
and
the
client
uploads
these
functions
into
the
kubernetes
api
and
oversimplifying
a
little
bit
and
the
patient
pool
manager
notices
that
these
functions
have
been
created
on
the
kubernetes
api.
It
figures
out
what
environments
these
functions
are
running,
so
these
functions
need.
So
if
there
is
a
node.js
and
a
Python
function,
it
creates
pools
of
node
and
Python
containers
and
let's
focus
on
HTTP
requests.
So
if
there's
an
HTTP
request,
it
comes
into
the
router
and
let's
say
it's
a
blue
request
for
that
blue
function.
A
Then
the
let's
say
it's
the
first
time
this
function
has
drawn
so
now
that
request
is
waiting
and
we
need
to
create
an
instance,
a
running
instance
of
this
function,
while
that
request
is
waiting.
So
this
is
a
cold
start,
so
what
we
need
to
do
is
the
router
makes
a
request
to
the
pool
manager
which
draws
a
already
running
part
from
that
pool
it
loads
that
function
into
that
pod
and
it
hands
over
the
address
of
that
part
to
the
router,
which
then
proxies
that
request
into
the
pot.
A
Similarly,
as
more
requests
come
in
the
same
process
repeats,
this
cold
start
process
takes
on
the
order
of
100
or
so
milliseconds
give
or
take
those
pods
are
cached
as
for
a
while,
even
if
there
are
no
more
requests
for
a
few
minutes
so
that
subsequent
requests
can
hold
on
subsequent
requests
can
reuse
those
pods
and,
if
they're,
no
more
requests.
If
a
function
is
idle
for
several
minutes,
those
parts
are
killed
and
you
regain
that
CPU
and
memory
that
they
were
using.
So
the
function
is
free
again.
A
A
A
A
Okay,
we're
connected
to
the
cluster.
There
are
those
nodes.
You
can
just
see
the
fish
in
deployment.
It's
not
super
important.
What
those
pods
are
will
run
a
really
simple
hello
world
function
and
will
create
an
environment
for
that
function
in
on
this
time
across
red
was
already
created,
but
all
you
have
to
do
is
specify
that
node.js
environment,
that
fish
and
ships
with-
and
here
we're
just
looking
at
the
pool
that
got
created.
So
all
of
those
containers,
all
of
those
pods
are
idle.
A
You
can
tune
how
many
there
are,
but
in
this
case
there
are
three
and
we
create
the
fission
function.
This
this
function
is,
is
now
uploaded
to
fission.
It's
stored
as
a
kubernetes
custom
resource,
but
it's
not
executing
yet
there's
no
runtime
resources
allocated
to
it.
Just
yet,
then
we
set
up
an
HTTP
trigger
for
this
function
also
called
a
route
and
we
actually
hit
that
route
with
curl
and
let's
see
so
here,
you
can
see
it
took
so
we
ran
called
three
times
on
that
hello.
A
World
function
it
first
of
all,
it
actually
worked
and
we
got
hello
world.
Secondly,
you
can
see
that
the
first
one
was
a
little
bit
slower
than
the
subsequent
invocations,
the
first
one
to
290
milliseconds,
that's
90
milliseconds
and
that's
about
100,
so
there's
something
between
100
and
200
milliseconds
of
overhead
in
the
cold
start,
and
that
part
is
cached
and
you
can
find
it
with
your
usual
kubernetes
commands.
The
part
is
labeled
with
the
function.
So
if
you
have
any
monitoring
tools
that
that
use
labels,
you
can
keep
using
them.
Okay,
so
that's
so!
A
That's
hello,
world
I'm
gonna
switch
back
a
little
bit
to
our
slides
and
talk
about
specifying
an
application.
So
so
we
saw
that
this
this
demo
creates
create
your
function
using
the
command
line,
which
is
great
for
starting
out
and
experimenting.
But
how
do
you
deal
with?
How
do
you
do
that
in
production?
Where
do
you
save
that
command
line?
A
Do
you
write
a
script
and
check
it
in
that
sounds
kind
of
horrible,
so
ideally,
you'd
have
some
sort
of
declarative
specification
and
you
check
it
into
source
control
and
that's
great
for
doing
for
doing,
updates
for
having
idempotent
behavior
and
I
what
we
really
want
is
both.
We
need
we
want
command
lines
to
get
started
and
declarative
specifications
so
that
we
can
check
them
in
and
have
ongoing
maintenance,
and
we
can
do
both
of
these
things.
We're
doing
something
that
we
call
config
by
example.
A
A
Okay,
so
let's
take
a
simple
Python
function
and
it's
over
there
I'm
actually
gonna
fullscreen
that
and
now
what
we've
done
is
we've
done.
The
same
function
create
command
that
you
saw
earlier,
except
that
we've
saved
what
it
did
in
a
specification
right
and
that
specification
is
saved
in
a
specs
directory
at
the
root
of
your
application
and
there's
a
lot
more
in
that
llaman.
But
this
is
basically
a
Cuban
Aires
custom
resource.
A
You
don't
ever
have
to
write
the
sea
animal
from
scratch.
You
can
just
create
them
using
these
command
lines.
Okay
and
then
you
can
just
apply
this
spec.
You
can
check
that
ml
in
and
you
can
apply
it
to
the
same
placer
to
any
other
cluster,
and
so
that
function
is
now
created
and
we
can
run
that
test.
A
A
Especially
in
a
developer
workflow,
so
what
we
can
do
is,
since
we
have
a
declarative
specification,
we
can
get
fish
in
the
fish
inclined
to
watch
your
file
system
and
every
time
you
change
the
file
to
rebuild
the
function,
upload
it
and
deploy
it,
and
this
is
basically
to
make
your
dev
workflows,
really
easy
and
fast
right.
So,
let's
save
that
and
the
efficiency
Ally
says:
okay,
I
noticed
a
file
change,
I'm
gonna
apply
the
spec
again
and
now,
if
we
just
reload
this,
we
see
it's
deployed.
A
Ok,
so
you
have
a
feedback
loop
that
works
on
the
order
of
milliseconds.
Instead
of
a
really
long
time.
We
didn't
rebuild
any
containers
here.
Your
code
was
just
deployed
in
the
cluster.
Now
this
is
great
for
a
dev
workflow,
but
what
you
can
do
is
you
can
pull
out
the
the
file
you
can
once
you
qualify
your
file
and
you're
satisfied
with
the
testing.
You
can
then
save
that
file
and
check
in
your
specifications
and
use
them
in
production,
and
everyone
on
your
team
can
have
an
easy
development
workflow
using
this.
A
This
kind
of
thing,
so
so
this
is
Python
which
in
which
is
fairly
easy
to
do
these
things,
because
it's
an
interpreted
language.
So
so
we
don't
have
to
build
it.
But
what
about
things
like
go?
Alright,
so
we
go,
we
need.
We
need
to
build
the
function,
and
we
can
do
that
too
in
fission,
and
we
can
also
do
it
declaratively.
A
B
A
So
we
save
it
and
now
fission
needs
to
build
that,
so
it
just
builds
it.
The
the
the
pattern
is
again
decorated,
the
declarative,
so
the
source
code
is
uploaded
and
a
build
controller
watches
that
package
notices
that
their
source
code
runs
the
go
compiler.
Nothing
needs
to
be
installed
on
your
on
your
local
machine.
All
of
the
actual
compile
happens
on
the
cluster
itself
and
you
can
reload
that
and
that
works
too
alright.
So
you
can
keep
watching
this.
You
can
keep
making
changes.
A
A
Okay,
so
things
that
you
care
about
at
runtime,
two
of
the
things
that
you
care
about
are
observability
and
and
we'll
talk
about
auto
scaling
as
well.
If
you
are
at
at
ben
securement
keynote
yesterday,
you
saw
how
modern
outages
are
kind
of
like
murder,
mysteries,
because
there's
so
many
interacting
components
and
so
and
if
you
think
about
it,
functions
as
a
service
makes
those
murder
mysteries
even
worse,
because
now
you
have
10
times
as
many
interacting
components.
A
So
observability
becomes
really
important
when
you
have
a
lot
of
interacting
functions
so,
like
everyone
else,
we're
really
excited
about
sto
and
we've
integrated
fishin
with
sto,
so
that
your
function,
pods
have
the
sto
sidecar
have
envoi
and
they
report
into
it
reports,
data
into
Prometheus,
graph,
fauna
and
so
on.
Let
me
find
that.
A
A
A
Okay,
so
now,
let's
actually
generate
a
little
bit
of
load
and
see
that
see
how
that
shows
up
in
our
in
our
dashboards.
This
is
the
function
being
loaded.
Hey
is
a
simple
load
generator.
What
we're
doing
here
is
sending
it
two
thousand
requests
with
it.
Send
it
200
with
a
concurrency
of
20,
so
20
requests
simultaneously
going
on,
and
that
was
the
little
test
request
we
did
earlier.
So
we
should
ideally
see
this.
A
A
Let's
pick
an
arbitrary
trace
and
look
at
it,
and
we
can
see
here
that
this
trace
is
pretty
simple,
there's
just
two
spans.
This
is
the
ficient.
The
first
one
up
there,
you
probably
can't
read
it
is-
is
efficient,
router
and
the
lower
one
is
the
actual
fission
function
which
just
does
hello
world.
In
this
case,
the
router
took
already
something
milliseconds
and
actual
function
took
28
milliseconds,
so
this
might
have
been
the
cold
start.
This.
A
Is
a
slightly
more
representative
trace,
not
really
not
sure
why,
anyway,
this
trace
actually
shows
you
the
overhead
of
the
router,
which
in
this
case
is
of
single-digit
milliseconds,
in
this
case
it's
2
and
1/2
milliseconds,
which
is
on
the
high
side.
But
at
least
you
have
some
visibility
into
your
system
and
we
can
talk
a
little
bit
later
about
what
we're
trying
to
do
about
overheads
of
our
router
all
right.
So
that's
that's
observability,
using
st
or
service
mesh
and
I
want
to
talk
a
little
bit
about
having
more
traffic
on
functions.
A
So
the
execution
method
that
I
showed
you
earlier
is
great
for
having
really
low
latency
for
loading
a
function
into
a
part
and
being
able
to
send
it
a
request,
but
it's
not
great
for
having
for
systems
that
have
high
throughput
but
may
or
may
not
care
about
latency
for
those
systems.
You
need
something
like
the
kubernetes
horizontal
pod
or
a
scaler
and
I'm
going
to
demo
that
in
a
in
a
video,
because
the
actual
order
scalar
takes
a
few
minutes
to
actually
to
actually
notice
things.
So
this
is
a
speeded
up.
A
Video
of
the
autoscaler
running
so
sufficient
function,
the
efficient
environment
is
created
and
we
allow
you
to
specify
a
min
and
Max
CPU
and
memory.
These
are
defaults
for
functions
created
in
that
environment,
the
environment.
The
function
can
override
it,
so
we're
creating
a
function
and
we
define
a
min
and
Max
scale.
A
A
That's
again,
a
simple
hello
world
function:
we've
artificially
kept
that
we've
artificially
kept
the
the
maximum
CPU
really
low,
so
that
hello
world
will
actually
order
scale.
Otherwise,
you
would
you
would
be
fine
with
a
very
small
number
of
instances.
You
can
see
that
fission
created
a
horizontal
power
order,
scalar
for
that
function
automatically
and
again.
A
We
set
the
target
a
bit
low
for
this
demo,
but
you
can
set
the
target
to
something
more
realistic,
like
80
90
percent
and
it
set
a
min
and
Max
scale
that
you
had
provided
at
the
at
the
function
level
and
currently
there's
just
one
replica.
So
again
we
start
our
load
generator
with
a
concurrency
of
250
and
we
send
it
to
that
same
URL
and
what
we're
looking
for
is
that
CPU
to
go
up
and
the
part
or
a
scalar
to
catch
up
and
create
more
replicas,
so
it
does
so
pretty
quickly
again.
A
This
video
is
sped
up
about
2x,
yeah,
2x
I
think
so
so
you
quickly
reach
three
replicas.
The
CPU
usage
is
still
pretty
high
once
the
actual
load
generator
finishes,
the
CPU
usage
comes
way
back
down
and
the
part
autoscaler
notices
that
and
it
actually
has
a
in
a
scaled
down
time
of
several
minutes,
which
is
why
this
is
a
video
but
five
or
six
minutes
after
your
load
stops.
A
The
autoscaler
brings
your
instances
back
down
to
to
the
minimum
scale
that
you
had
set
up
with
the
function,
so
this
is
function,
auto
scaling
and
there's
work
going
on
to
combine
both
these
back
ends
so
that
you
have
low
latency
for
the
startup
as
well
as
high
throughput.
When
you
have
a
lot
of
load
all
right,
so
that's
auto-scaling
go
back
to
our
slides
for
a
bit
and.
A
I'll
talk
a
little
bit
about
larger
applications
of
that
contain
interacting
functions,
so
there's
a
lot
of
different
ways
that
you
can
have
your
functions
interact.
You
can
just
do
plain
old,
HTTP
requests
from
between
functions
and
because
of
this
do
integration.
You'll
have
some
measure
of
observability
into
those
you'll
have
some
insight
into
into
how
those
requests
are
going,
but
it
would
be
pretty
cool
if
that
entire
interaction
between
functions
were
abstracted
away
in
some
way.
A
So
we've
created
a
system
called
ficient
workflows
where
you
can
create
some
sort
of
flowchart
in
some
sense
of
functions
and
the
workflow
engine
will
will
coordinate
those
functions.
It
will
manage
both
data
and
controls,
flow
and
and
you'll
actually
be
able
to
to
have
functions,
talk
to
each
other
without
calling
each
other
explicitly
using
a
using
a
separate
workflow
that
that
operates
on
them.
So
I'm
going
to
switch
back
to
a
demo.
A
So
again,
I'm
going
to
show
a
really
trivial
workflow
here,
we're
going
to
run
the
the
UNIX
fortune
command
as
a
function
and
it
I
don't
know
it
it
outputs
some
sort
of
random
quote,
and
we
have
this
function
which
outputs,
an
ASCII
art
cartoon
whale
that
that
contains
a
speech
bubble
containing
whatever
it.
Whatever
it's
sent.
A
Okay,
so
now
we're
going
to
create
a
workflow
that
combines
these
two
functions
without
actually
having
the
first
function,
call
the
other,
and
we
do
that
by
defining
a
a
workflow
in
gamal
and
it's
got
tasks
and
each
task
is
efficient
function
call
and
the
first
one
is
the
generate
fortune
task
which
cause
fortune
and
the
second
one
is
the
whale
with
fortune.
So
it
calls
whale
say
with
the
input
of
that
first
task.
A
So
the
input
here,
that's
the
data
flow
and
there's
a
requires
which
makes
sure
that
the
generate
runs
before
this
task
and
that's
the
control
flow.
So
you
can
define
dependencies
this
way.
This
is
a
really
simple
workflow
with
two
tasks.
But
if
you
had
more
tasks
you
could
you
have
implicit
parallelism,
parallelism.
A
And
hopefully
we
should
see
a
whale
saying
something
silly
eventually,
all
right.
The
code
is
entirely
run.
Okay,
so
the
first
function
was
run.
The
workflow
engine
interpreted
the
Yama
and
and
allowed
you
to
allow
those
functions
to
just
have
the
data
sent
from
one
function
to
another.
We
don't
have
a
whole
lot
of
time
to
dive
into
how
the
workflow
engine
works,
but
essentially
it
uses
a
message
queue
to
be
to
have
persistent
events
tracked
as
the
workflow
executes.
A
So
as
each
task
finishes,
it's
tracked
in
the
message
queue
and
that
triggers
the
workflow
engine
again
to
invoke
the
next
function
and
that's
also
how
concurrency
work.
So
if
you
have
a
task
that
depends
on
a
bunch
of
different
tasks
and
they
don't
have
any
dependencies
between
them,
then
all
those
tasks
could
run
in
parallel
and
again,
you
would
do
this
without
explicitly
having
any
kind
of
explicit
parallelism
in
the
workflow.
Okay,
so
I've
run
through
my
demos
and.
A
Workflows
is
a
relatively
early
project.
It
should
have
a
beta
mid
to
late
next
year.
Yeah
and
security,
scalability
and
performance
are
the
focuses
of
the
project
for
the
next
few
months,
at
least
for
more
on
the
roadmap
and
everything
else
check
out,
fishin,
Dario
and
github,
and
talk
to
us
on
slack
or
Twitter
and
I.
Think
we
have
a
few
minutes
for
questions.
Anyone
hey
thanks
somebody
to
switch
the
song
yeah.
It's
all.
A
Hi
hi,
can
you
explain
how
imports
like
requirements
are
baked
into
a
function?
Yes,
so
I
can
open
up
an
example,
essentially
the
the
declarative,
bail
system
that
I
showed
you
that
the
Builder
is
the
one
that
can
do
imports
and
requirements
gathering
as
well.
So
you
can
provide
a
build
script
with
a
function
or
you
can
use
the
built-in
one
in
the
environment
and
when,
whenever
the
function
is
built,
it
uses
C.
Let
me
just
use
my
editor.
A
I'm
having
a
hard
time
finding
it,
but
let
me
tweet
out
that
link
later
to
actually
show
you
how
that
works.
But
essentially
you
can
have
a
spec
of
requirements
using
whatever
is
idiomatic
in
the
language
of
its
Python.
There's
a
requirement
start
txt.
If
it's
good,
there's
a
either
a
glide
file
or
the
various
other
dependency
tools,
and
you
can
write
a
script
that
that
runs
those
and
fish
and
then
packages
it
and
is
in
charge
of
transporting
that
package
to
the
function
when
it's
supposed
to
Thanks.
A
Do
not
instead,
we
do
not
deeply
integrate
with
any
message
queue.
The
the
picture
I
showed
you
earlier
of
triggers.
That's
how
we
integrate
with
any
message
queue.
So
there's
a
Nats
message:
queue
trigger,
there's
a
Kafka
trigger.
That's
the
super
early
right
now
and
folks
from
Microsoft
have
contributed
and
as
your
storage
queue
trigger
as
well.
So
that's
it
for
message:
queues
the
workflow
engine
uses
not
streaming,
which
is
a
durable
version
of
Nats.
Yes,
where.
A
A
B
B
A
A
You're
done
what
the
new
kubernetes
initializes
feature
right
today,
right
now,
you'll
have
to
create
a
the
the
best
way
to
do.
It
would
be
to
create
a
trigger
for
four
part
for
kubernetes,
initializers
and
and
then
declare
it.
You'd
need
a
new
type.
That
says
this
is
an
initializer
and
the
implemented
the
fission
implementation
of
that
would
watch.
The
initial
would
be
a
custom
controller
that
watches
that
and
executes
the
certain
function
when
that,
whenever
the
the
API
call
specified
in
the
initializer
actually
occurs,
so
this
isn't
implemented
yet
in
fission.
A
But
we
want
to
have
a
way
for
you
to
do
this
without
having
to
do
all
the
work
of
writing
the
controller.
We
do
have
kubernetes
watches,
so
you
can
watch
an
arbitrary
set
of
resources
after
they're
created,
so
it's
quite
different
from
the
initializer,
but
if
you're
trying
to
do
custom
behavior
on
top
of
kubernetes,
that
fits
some
of
the
use
cases
so.