►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
The Control Loop As An Application Development Framework - Nick Santos, Tilt
Kubernetes emerged as A Good Idea® in part because it gives you real-time, circular feedback: it's a control loop. Something watches, something reacts, equilibrium is maintained. In this talk we’ll discuss how this model is useful not only for orchestrating containers, but for many applications that handle real-time feedback loops. And thanks to Kubernetes, most of the scaffolding for it is already out there, ready to be used. As a use-case the speaker will talk about a development tool that reacts to source code and server status changes in real-time—a perfect match for a Kubernetes-style control loop. Adopting control loops led to a simpler, more modular app. It made the codebase easier to grasp for new developers, and the application as a whole more uniform and easy to extend. Plus, Kubernetes already has a rich ecosystem of tools for it. Lastly, the speaker will discuss other examples in which this model applies and whether this model makes sense for your own applications.
A
A
Let
me
tell
you
about
the
motivation
for
this
talk
and
why
I'm
giving
it
so
my
name
is
nick.
I
work
on
a
product
called
tilt
and
we
interoperate
a
lot
with
kubernetes
and,
as
we
interoperated
a
lot
with
kubernetes,
we
started
to
realize
how
much
the
design
decisions
that
kubernetes
makes
were
influencing
the
design
decisions
we
were
making
in
our
own
app
and
it
when
we
kind
of
we
kind
of
thought
about
why
this
was
happening,
and
we
said
well,
you
know
if
we
squint
really
hard.
A
Kubernetes
is
just
a
framework
for
solving
control
loop
problems
and,
if
you
squint
really
hard
till
also
needs
to
solve
a
lot
of
control
loop
problems
and
we
really
admired
the
way
that
kubernetes
broke
those
problems
down
into
pieces
and
solved
them,
and
so
we
said,
okay
kubernetes
actually
has
a
pretty
good
opinion
on
how
to
solve
these
kinds
of
problems.
Why
are
we
building
all
this
libraries
for
ourselves
to
build
this
app
to
build
control
loops?
A
So
let
me
just
start
by
talking
about
what
I
mean
by
the
control
loop
problem.
Control
loops
are
a
lot
of
places
in
the
kubernetes
documentation.
This
is
a
very
old
engineering
concept.
The
idea
of
a
control
loop
is
that
in
an
environment
where
things
can
change
at
runtime,
where
you
need
to
react
to
changes
where
you
need
to
steer
to
avoid
obstacles,
you
really
need
an
engineering
system
that
can
self-regulate
that
can
continuously
respond
to
changes
and
keep
the
system
in
a
consistent
state.
A
A
If
your
or
request
succeeded
software
control
loops
are
constantly
adjusting
their
behavior
in
response
to
some
environment
around
them,
whether
it
be
the
amount
of
bandwidth
you
have
available,
whether
it
be
the
amount
of
cpu,
you
have
available
that
you
can
schedule
a
lender
whether
it
be
you
know,
the
file
system
or
operating
system.
That's
changed
around
you.
A
A
lot
of
this
talk
is
really
inspired
by
this
book,
which
chris
nova
and
justin
garrison
wrote
a
few
years
ago,
and
the
one
reason
I
really
admire
this
book
is
that
it
tries
to
abstract
away
from
the
general
from
a
lot
of
the
implementation
choices
that
different
cloud
native
systems
make
and
really
tries
to
tackle
like
okay.
What
are
the
patterns?
What
are
the
abstract
problems?
What
are
the
problems
that
any
system
that
tackles
these
problems
are
going
to
have
to
handle?
A
A
A
I
wrote
this
blog
post
in
the
spring
that
got
shared
a
lot
and
it
was
a
kind
of
a
troll
post
and
I
was
surprised
it
got
shared.
So
much
is
called
kubernetes
is
so
simple.
You
can
explore
it
with
curl
and
what
I
wanted
to
kind
of
break
down
in
that
blog
post
was
the
way
that
when
we
talk
about
kubernetes
we're
actually
talking
about
two
separate
things
that
we're
talking
about
one
part
of
kubernetes:
that's
really
a
huge
ecosystem
of
tools
that
manage
cloud
infra,
and
that
is
cool.
B
A
A
I
also
want
a
big
shout
out
to
the
kubernetes
community,
who
has
done
a
lot
of
good
work,
really
breaking
this
up
into
small
repos
repos,
with
like
really
clear
division
of
responsibilities
that
really
layer
on
top
of
each
other,
such
that
each
repo
is
building
on
the
repo
below
it
we're
going
to
go
right
through
them.
If
you're
contributing
to
your
days,
it's
all
going
to
sound
very
familiar.
The
first
kind
of
the
the
basic
like
bones-
and
you
know,
potatoes
of
kubernetes-
are
the
go
structs.
A
A
lot
of
documentation
in
kubernetes
is
really
auto-generated
from
ghost
drugs,
but
I
really
just
enjoy
just
reading
the
ghost
trucks,
because
they're
very
simple,
they're
very
well
documented.
They
all
have
a
kind
of
a
consistent
structure.
They
all
have
a
spec
that
just
defines
the
desired
behavior
of
the
resource
and
a
status
that
just
defines
the
observed
status
of
the
resource
that
other
things
that
need
to
respond
to
that
status
can
watch.
A
Going
one
level
up
from
the
basic
ghost
trucks
we
have
api
machinery
once
we
have
a
system
with
objects,
we
now
need
a
way
to
refer
to
those
objects.
We
need
a
way
to
refer
to
those
objects
types.
We
need
a
way
to
name
those
objects
and
refer
to
them
by
name.
We
need
a
way
to
group
those
objects
together
and
so
amy
api
machinery
says:
okay,
we
have
a
bunch
of
objects.
A
The
next
level
up
once
you
have
you've,
reached
your
needs
of
how
to
refer
to
objects.
You
now
need
http
eyes
for
each
structure.
Now
it's
interesting
that,
like
I,
I
honestly
believe
that,
like
one
of
the
geniuses
that
kubernetes
does
not
make
you
define
your
own
http
ros,
it
just
says
you
know
what
we
know.
What
http
routes
word
you
need
we're
just
going
to
auto,
generate
them
for
every
object
that
you
need
to
manage.
A
We
know
how
to
refer
to
it,
by
name
and
by
label
we're
going
to
auto,
generate
http
apis
to
create
update,
get
delete
patch
list
and
watch.
We
will
talk.
This
is
a
little
bit
more
complicated
than
just
a
basic
crud
app,
and
we
will
talk
about
that.
But
for
now,
what's
really
nice
is
that
this
api
server
can
just
generate
at
runtime
new
routes
for
new
resources.
A
Once
you
have
http
apis,
you
need
prototyping
tools,
tools
that
let
you
from
a
cli
interoperate
and
interact
with
those
objects,
and
that's
why
we
now
have
its
own
repo,
kubernetes.I,
o,
slash,
cube,
cuddle
and
also
kubernetes
io
cli
dash
runtime
that
are
just
the
command
line.
Tools
that
for
working
with
this
construct
cube
code
will
get
keep
going
describe
edit
keep
going
applied,
apply.
Kubecoil
patch
are
good
examples
which
all
are
just
really
ways
to
kind
of
prototype
things
around
those
http
apis.
A
Once
you
have
a
prototyping
framework,
you
really
need
a
way
to
make
it
robust
and
to
really
build
your
control
loop
to
be
able
to
not
only
to
really
take
okay.
This
is
a
set
of
http
apis.
I
have
to
manage
resources.
How
do
I
continuously
respond
to
changes
in
those
resources
and
controller?
Runtime
is
the
framework
to
do
that.
It
turns
http
calls
into
a
fully
realized
control
loop.
It
listens
to
changes,
it
caches
things
appropriately.
It
manages
the
caches
for
you.
A
It
doesn't
really
expose
to
you
the
http
apis,
it's
making
under
the
hood.
It
just
really
just
exposes
to
you,
as
as,
if
they
were
events,
these
are
just
changes
you
need
to
respond
to,
and
the
important
bit
of
this
is
that
to
build
a
control
loop,
you
need
a
lot
more
than
the
classic
crud
web
app.
You
can't
just
have
create,
read,
update
and
delete,
you
need
list
and
watch,
and
those
are
the
http
tools
that
let
you
build
these
standard
control
loops
that
we
can
use
to
manage
any
resource.
A
Okay,
stop
there
for
a
minute.
This
is
the
basic
building
blocks
of
a
control
loop
that
kubernetes
breaks
down.
Why
does
this
apply
to
tilt?
As
I
said,
if
you
squint
hard
enough
tilt
is
also
a
control
loop
tilt.
Is
we're
not
going
to
talk
much
about
tilt,
but
just
so
you
have
the
context.
Tilt
is
watching
your
source
code,
it's
auto,
deploying
and
restarting
services
when
the
source
code
changes
and
it's
raising
alerts
when
services
are
inhaled.
So
it's
really
just
kind
of
responding
to
a
lot
of
environment.
A
When
we
first
wrote
tilt,
we
ended
up
building
our
own
control
loop
infrastructure,
we
just
kind
of
hand
rolled
it's
like
this,
isn't
too
complicated.
We
can
just
build
our
own
control
loop,
it's
just
a
loop
right.
Let's
just
have
a
little
loop
that
reacts
to
changes
to
a
you
know
a
data
structure
and
it
will
be
simple,
and
the
problem
is
once
we
built
that
very
simple
control
loop.
We
actually
realized
oh
yeah
as
an
infra
tool.
We
need
a
lot
of
tools
around
it.
A
A
They
wanted
to
kind
of
write
new
dashboards,
new
kind
of
tools
that
were
responding
to
the
changes
and
able
to
kind
of
display
updates,
and
we
kind
of
would
end
up
writing
a
lot
of
those
things
ourselves
for
our
own
internal
tooling,
but
it
was
kind
of
a
pain.
It
didn't
work
very
well.
We
kind
of
were
very
lazy
about
it,
and
we
said
you
know
what
kubernetes
has
already
solved
these
problems.
It
solved
them
really.
Well,
we
don't
need
to
build
an
api
server.
We
don't
need
to
build
our
own
cli
framework.
A
A
A
The
first
object,
we're
going
to
try
is
called,
is
called
tilts
file,
watcher,
there's,
actually
a
long
history
of
tools
that
have
implemented
file
watching
as
a
control
loop
enter
is
a
big
one.
Nodemon
is
a
big
man.
Facebook's
watchman
is
a
big
one.
They
all
are
demons
that
run
on
your
machine.
They
set
up
the
file
watches
for
whatever
api
your
os
supports
for
doing
file
watches
whether
the
api
is.
A
This
turns
out
to
be
a
really
great
fit
for
kind
of
how
kubernetes
thinks
about
declarative
data
models
that
you
can
declare
hey.
I
want
to
watch
all
the
files
watching
go.
I
want
to
declare,
I
want
to
watch
all
the
files
that
match
star.go
and
I
want
to
continuously
react
to
those
changes,
and
I
really
only
care
about
the
most
recent,
the
kind
of
the
current
state
of
things
the
most
recent
stuff.
A
This
is
what
file
watch.
Spec
looks
like
the
very
first
in
the
kind
of
desired
state.
We
just
said:
okay.
This
is
a
pretty
well
understood
problem:
how
to
watch
files
or
lots
of
good
libraries
for
watching
files,
we're
just
going
to
create
a
control
loop
kind
of
a
api
server
wrapper
around
it,
and
so
we're
going
to
define
some
structs.
A
This
struct
has
something
called
watched
paths,
which
is
just
the
recursive
directory
of
things
that
we're
going
to
watch,
and
it
also
has
a
list
of
ignores
that
files.
We're
going
to
ignore
by
our
files
that
we're
going
to
have
our
us
ignore
and
we're
also
going
to
have
a
wi-fi
watch
status,
which
is
both
going
to
give
us
low-level
operating
system
errors
like
hey.
You
ran
out
of
inotify
nodes
and
also
file
events
that
hey
here
are
the
most
recent
files
that
change
that
you
should
be
responding
to
before
we
built
the
system.
A
A
lot
of
our
support
was
around
helping
people
diagnose
file,
watching
problems,
since
tilt
is
really
just
a
file
watcher
and
a
lot
of
tools
like
really
need
to
be
good
at
watching
files.
But
the
problem
is
with
people
would
have
all
sorts
of
problems
either
they
had.
You
know
written
the
pattern
wrong
or
misunderstood
the
pattern
or
misunderstood
relative
versus
absolute
directories
or
if
they
had
screwed
up
the
syntax
or
some
all
sorts
of
weird
errors,
and
the
problem
was
that,
like,
if
diagnosing
bugs,
was
really
difficult
where.
B
A
A
Once
we
move
this
to
a
control
loop,
what
we
realize
that
the
control
loop
kind
of
pattern
of
the
world
really
lets
you
expose
the
state
to
people
and
lets
them
run
their
own
interactive
experiments.
We
would
often
just
say:
hey
this
works
just
like
kubernetes,
you
can
say
tilt
describe
file,
watches
and
they
would
say.
Oh,
this
is
perfect.
This
is
exactly
what
I
need.
A
I'm
going
to
change
the
spec,
I'm
going
to
touch
a
file
and
I'm
going
to
watch
the
status,
and
I
can
run
my
own
interact
experiments
to
see
if
I
got
this
right
and
just
made
it
way
easier
to
help
people
diagnose
their
own
problems.
Just
incredibly,
just
the
the
number
of
file
watch
issues
we
got
dropped
off.
The
other
thing
that
we
do
a
lot
of
is
the
very
early
migrations
we
did
was
the
port
forward,
so
port
forward
ina.
A
It
has
to
continually
steer
to
keep
that
tunnel
up
and
we
used
to
have
all
sorts
of
problems
where
people
would
say
poor
foreigners
and
doesn't
work
and
we'd,
ask
them
okay,
what's
going
on,
they
say
I
don't
know
and
we
just
have
mega
bugs
they're.
Just
like
you
know
a
dozen
comments
saying
I
just
portforward
isn't
working
and
we
kind
of
would
say:
okay,
let's
copy
and
paste
these
error
messages
into
google
and
then
we'd
find
a
kubernetes
issue
thread
and
it
would
also
be
the
same
thing:
it'd
be
40.
A
People
on
an
issue
thread
and
kubernetes,
saying
up
portfolio
isn't
working,
please
help
me,
and
so
we
said
okay,
this
is
frustrating.
Could
we
could
we
do
better?
Could
we
write
adapt
this
to
a
reconciler?
Could
we
have
it
really
publish
its
spec
and
status
and
help
people
understand
what
parts
of
their
portfolio
are
failing?
So
we
kind
of
forked,
the
port
forward
and
client
go
and
adapted
to
this
pattern
once
we
did
this
it
actually
exactly
the
same
thing
happened.
It's
just
certainly.
A
The
joy
number
one
that
we
discovered,
which
is
probably
not
surprising,
is
that
cube
cuddle
is
really
great
and
has
really
figured
out
a
lot
of
the
right
verbs
for
how
to
diagnose
problems
and
really
don't
underestimate
the
social
conventions
of
the
tools
that
people
could
say.
Oh
yeah,
this
works,
like
cube,
cuddle,
I'm
used
to
cube
cuddle
if
every
tool
worked
like
this,
that
would
just
make
my
life
a
lot
easier.
A
A
I
wanted
to
write
a
kind
of
a
blog
post
that,
just
like,
writes
kubernetes
controllers
in
bash,
but
instead
I
just
wrote
some
fun
like
tilt
extensions
in
bash,
so
tilt
just
ends
up
being
a
wrap
around
cube
cuddle.
So
my
my
bash
extension
just
says:
tilt
get
command
dash
dash
watch
streams
that
list
of
commands
that
are
changing
to
a
script
that
reconciles
them
that
script
every
time
a
command
changes.
It
creates
a
button
for
that
command
and
ends
up
just
creating
a
bunch
of
buttons
in
the
tilt.
A
Ui
dynamically,
based
on
what
you're
running
right
now-
and
this
has
allowed
us
to
like
really
just
prototype-
you
know-
bash
is
vash-
is
such
an
underrated
technology.
Prototyping
things
in
bash
is
just
always
gives
me
some
joy.
I
want
to
talk
about
a
couple
of
changes
and
problems.
We
ran
into
the
first
big
overall
change.
I
know
it's
always
popular.
To
complain
about
go
dependency.
Management
jason
gave
a
talk
at
kubecon
eu
this
year.
I
didn't
actually
know
he
was
going
to
be
the
track
host.
A
So
I'm
really
glad
he's
he's
here
that
I
can.
I
can
praise
him
about
this
problem
of
pulling
and
kubernetes
the
library.
There
are
certain
dependencies
that
are
really
problematic,
mostly
they're,
not
kubernetes
dependencies
they're,
mostly
upstream
kubernetes
dependencies
I'll.
Let
I'll
let
the
slide
speak
for
itself.
I
I
don't
have
a
good
solution
to
this,
but
but
pulling
and
kubernetes
library
definitely
does
pull
in
some
problematic
projects.
A
And
and
lastly,
if
you
want
to
learn
more
about
this
problem,
this
is
not.
We
are
not
the
only
team
that
is
doing
something
like
this.
There
were
two
two
talks
at
kubecon
eu
that
both
talked
about
this,
the
kcp
talk,
which
I
think
is
kubernetes
control
plane
which
really
talks
about
how
to
better
expose
an
api
server
as
a
library
that
other
people
can
use
for
non-kubernetes
management
and
then
bad
idea,
which
is
jason's
talk
about
doing
a
kind
of
a
similar
thing.
A
Thank
you
very
much.
That
is
the
end
of
the
talk.
I
want
to
give
a
shout
out
to
ellen
korbs,
who
was
this
originally
meant
to
be
a
joint
talk
that
I
was
supposed
to
give
with
them,
and
instead
they
decided
to
give
a
kubernetes
action
movie
on
friday,
which
you
should
go
see.
It's
called
beyond
kubernetes
security,
but
and
also
the
tiltdev
team,
who
gave
a
lot
of
notes
on
this
talk
and
and
made
it
a
lot
better.
So
thank
you.
B
I
should
just
go
look
at
the
other.
Bad
idea
talk,
but
so
I
mean
pulling
in
client
goes
are,
is
huge,
maybe
not
necessarily
problematic,
in
the
way
that
I'm
getting
the
sense
that
grpc
is
like
as
far
as
just
bad,
you
know
pulling
a
bunch
of
things.
You
don't
need
or
not
playing
nice,
but
I
mean
client
go
is
massive.
B
It's
gotten
better,
but
I
remember
like
back
in
the
glide
days
before,
go
mod
and
like
people
passing
around
like
giant
lists
of,
replaces
and
like
dynamically
building
big
lists
of
very
specific
version
opinion,
because
I
mean
it's
great,
that
the
go
packages
are
built
in
to
like
all
these
different
repos.
But
getting
a
cohesive
set
of
those
to
build
client
go
was
a
pain
for
a
while
so
yeah.
I
guess
the
question
is
like
what
what
did
what
is
what
is
short
of
me
going
to
watch
the
whole
talk?
B
A
Okay,
I
will
I'm
not
sure
if
there
are
any
grpc
people
in
the
room
that
I
don't
want
to
criticize
them
too
hard.
I
will
say:
clientgo
has
actually
been
gotten
used
to
be
really
awful
to
pull
in
as
a
pensie
has
gotten
really
good,
I'm
not
sure
who
is
responsible
for
that,
but
I
owe
them
a
drink
grpc.
A
Let's
just
say,
I've
had
some
conversations
with
grpc
people
where
they've
turned
things
into
errors,
but
not
really
provided
any
mitigation
strategy
for
what
happens
if
one
of
your
dependencies
upgrades
grpc
and
how
you're
supposed
to
handle
that
new
error,
that's
generally
the
problem
their.
I
think
their
official
deprecation
policy
is
that
they
do
not
follow
semver,
that
they
are
allowed
to
break
any
anything
that
is
marked
experimental,
but
there's
no
way
to
know
unless
you
read
the
documentation
whether
an
api
is
experimental
or
some
downstream
project
uses
experimental
api.
A
What's
the
density
open
source
tilt
in
the
product?
Yes,
oh
yeah,
tilde
is
an
open
source
product.
That's
that's!
Basically
it
yeah
you
can
you.
Can
you
literally
can
just
check
out
tilt
tomorrow
and
just
see
the
parts
of
kubernetes
that
we
use?
There's
a
separate
repo
called
tilt
api
server,
which
just
shows
which
is
kind
of
a
standalone
api
server,
not
unlike
kcp
or
bad
idea.
That
just
kind
of
shows
you
how
to
set
one
up.