►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Apps SIG - Janet Kuo, Google & Kenneth Owens, Brex
Kubernetes SIG Apps covers developing, deploying, and operating applications on Kubernetes with a focus on the application developer and application operator experience. In this deep dive, we will look at the general experience for application developers and operators along with specifics of the Workloads API (Deployment, StatefulSet, DaemonSet, Jobs, etc.) and Application CRD. We will also do Workloads controller code walk-through. https://github.com/kubernetes/community/tree/master/sig-apps
https://sched.co/Uah4
A
A
We're
two
of
the
co-chairs
of
sig
apps
and
co-chairs
are
really
just
in
SIG's
kind
of
an
administrative
burden
like
we
organize
meetings.
Basically
in
that
function,
we're
also
two
of
the
tech
leads
and
they
think
we're
the
set
of
people
that
are
both
chairs
and
tech
leads.
So
that
means
we're
code
owners
for
the
workloads,
api's
there's
another
one
right
there
he's
from
Red
Hat.
B
This
is
our
agenda,
so
I'll
start
with
on
walking
through
the
controller
what
globe
controller
internals.
So
we
take
three
of
the
workload
controllers
to
just
give
you
do
a
code
walkthrough
to
help
new
contributors
or
existing
contributors
to
understand
the
controller
code
structure.
So
I'll
start
with
on
the
basic
reconciliation
loop
of
a
controller.
B
So
for
a
given
controller
in
kubernetes
we
usually
have
the
desired
state
and
then
the
current
state
and
based
on
that,
we
make
some
changes
and
we
keep
doing
that
in
the
inner
loop
and
this
change
is
level
triggered,
which
means
that
we're
not
looking
at
the
diff
between
the
div
in
the
state
we're
just
looking
at
the
current
state
and
what's
the
expected
state
in
the
resource
pack.
So
let's
take
on
deployment
controller
as
an
example.
B
So,
to
give
you
a
quick
recap
on
what
deployment
does
is
that
it
creates
replica
sets
and
replicas
that
then
create
pods
and
deployment
supports
on
a
feature
called
rolling
update.
So
you
can
rolling
update
your
containers
by
on
changing
this
back
of
a
deployment
and
then
what
it
does
internally
is
that
it
will
scale
up
and
scale
down
replica
sets
and
let's
take
a
look
at
the
source
code
of
that.
B
B
Good
enough,
okay,
so
let's
start
with
this
piece.
So
all
the
controllers
are
just
part
of
the
kubernetes
controller
manager.
We
ship
it
as
a
binary
and
deployment
controller
is
one
of
them
and
the
other
controller
code
lives
in
the
package
in
the
kubernetes
kubernetes
repository.
You
can
find
it
under
package,
slash
controller
and
under
there
there
will.
One
of
it
is
deployment,
and
you
can
see
other
controllers
here.
So
if
you
go
inside
the
deployment
controller
code,
you
can
see
the
new
deployment
controller.
B
First,
let's
talk
about
something
called
shared
informer,
so
you
can
see
that
we
passed
some
some
parameters
into
this
deployment
controller
and
it's
it's
using
a
framework
in
kubernetes.
That's
called
share
informer.
So
what
it
does
is
that
the
controllers
can
use
that
to
receive
notifications
of
the
changes
to
the
resource.
For
example,
if
I
have
the
deployment
informer,
I
can
get
the
events,
the
notification
of
the
creation,
the
update
or
a
deletion
of
the
deployment
and
from
there
I
can
access
a
shared
cache.
So
this
is
not
required
for
all
controllers.
B
B
And
then
you
only
read
our
list
from
the
cache
in
the
controller
and
it's
called
shared
Informer,
because
this
cache
can
be
shared
among
different
controllers
and
then
one
of
the
cavea
is
that
when
you
write
code
in
controller,
you
should
never
ever
mutate
objects
in
the
cache,
because
when
you
do
that
it
will
affect
other
controllers.
That's
also
reading
from
this
cache.
B
It's
just
the
the
real
business
logic
in
this
deployment
controller.
So
after
a
receive
some
events
from
all
those
resources
that
it
watches
it
will
try
to
sync
the
deployment
and
the
sink
is
just
the
loop
that
I
have
mentioned
here:
ok
and
and
then
we
have
something
called
work
you
so
you
can
see
here.
There's
a
function
called
in
queue
at
the
point
and
and
there's
in
queue.
So
what
does
the
in
queue
mean?
It
means
that
we
are
storing
the
deployment
resource
in
a
queue.
B
We
call
it
work,
you
it's
another
framework
in
kubernetes,
so
the
benefit
of
using
that
is
that
you
can
queue
all
the
changes
that
you
need
to
make
who
to
you.
You
can
include
the
changes
to
a
resource,
and
then
you
can
work
on
that
that
later
in
the
reconcile
in
the
in
this
case,
it's
the
same
deployment.
B
So
you
can
have
multiple
workers
that
takes
task
from
takes
resource
from
the
work
you
and
then
you
can
reconcile
it
by
running
sync
deployment
later
and
this
work,
you
will
make
sure
that
no
two
deployments
will
be
operated
by
the
same
worker
at
a
time
and
then
so
in
here
you
can
see
that
we
run
this
controller.
It
will
run
start
some
workers
and
you
can
specify
how
many
number
of
workers
you
want
for
this
controller,
and
then
you
can
see
it's
running
this
thing
called
worker
and
then,
if
you
scroll
down.
B
The
key
is
just
a
index
in
the
work
queue
for
you
to
fight
this
resource
and
the
controller
and
the
deployment
controller
just
used
that
namespace
and
then
as
key
to
fight
this
deployment.
This
works
because
we
only
store
deployment
in
this
work.
You
we
don't
store
other
kinds
of
resource.
So
if
you
have
multiple
kinds
of
resource,
then
you'll
need
a
different,
different
function.
To
define
here
for
this
work
you
and
then,
and
we
can
scroll
down
a
little
bit.
B
You
can
see
that
I'm
getting
and
this
I'm
finding
this,
given
this
key
I'm
finding
this
deployment
from
the
key
I'm
splitting
the
namespace
and
name
from
the
key
and
then
get
the
deployment
and
notice
that
this
is
listing
from
the
cache.
So
we're
not
there
right
it
directly
talking
to
API
server,
and
then
we
got
the
deployment
from
the
cache
and
we
don't
want
to
mutate
it
by
accident.
So
we
call
this
deep
copy
here
and
then,
let's
see,
there's
anything
interesting
so.
B
Here
is
the
deployment
deletion.
Timestamp
is
not
nil.
That
means
that
this
department
will
be
deleted.
So
we
don't
need
to
reconcile
it
anymore.
We
don't
need
to
like
create
or
scale
up
scale
down
replica
shots.
We
just
need
to
think
the
status
status
of
it,
so
we
can
just
return
sync
status
only
and
then
and
then
scroll
scroll
down.
B
We
still
keep
it
here
for
backward
compatibility
and
then
finally,
here,
if
it's
a
scaling
event,
is
it
even
if
on
the
deployment
is
not
paused,
we
run
the
same
function
to
scale
where
proportionally,
and
the
last
piece
is
that,
finally,
we
can
do
the
deployment
rollout.
So
deployment
supports
two
different
types
of
rollout
strategy.
The
first
one
is
the
recreate.
That
means
that
I
will
delete
all
your
old
parts
completely
and
after
all,
those
old
has
are
deleted.
B
You
can
create
new
ones
and
then
for
rolling
update
is
just
rolling
and
scaling
up
and
down
old
ones
and
new
ones.
So
that's
it
and
because
there's
a
lot
of
details
here,
but
after
this
walkthrough
it
should
be
pretty
easy
for
you
to
dive
deeper
into
deployment.
And
then,
let's
go
back
to
here.
The
next
controller
that
I
want
to
talk
about
is
replica
set
because
it's
related
to
deployment
and
what
replica
set
does
is
that
it
just
creates
on
a
number
of
identical
pods
from
a
part
template
and
then
the
source
code
lives.
B
Here,
it's
under
your
package
control
area
roughly
at
and
then
and
interesting.
So
the
I
can
show
you
like
the
basic
of
this
replica
set
controller,
but
you
can
see
that
a
lot
of
code
is
very
similar.
For
example,
it
also
has
the
informer
framework,
and
it
also
has
the
sync
Handler-
and
it's
listing
from
the
cache,
and
one
thing,
that's
particularly
interesting
in
a
replica
set
controller
is
that
it
has
something
called
expectations
and,
if
I
scroll
down
to
the
sink,
let's
find
a
sink
handler
thing.
B
B
Expectations
and
I
find
it.
Oh
yeah,
you
can
see
that's
doing
something
like
deleting
expectations.
Creating
expectations,
so
expectation
is,
is
a
pair
of
atomic
counters
for
you
to
track
on
the
the
creation
and
deletion
of
resource
and
the
in
this
case
and
replica
set
controller
is
using
this
to
track
the
creation
and
deletion
of
parts,
and
you
know
you
can
also
find
it
in
other
controllers,
like
jobs
and
daemon
set,
and
why
do
we
need
this?
Why
don't
we
need
it
and
deployment
controller,
for
example?
B
This
is
because
that,
if
you
still
remember,
we
are
reading
from
cache
and
the
cache
could
be
outdated
when
you
read
it.
So
imagine
that
when
the
replica
set
controller
creates
it,
for
example,
if
told
to
create
three
parts
and
then
in
the
next
sink
loop,
it's
looking
at
number
the
number
of
current
parts
in
the
system
and
doesn't
find
those
three
parts,
because
the
cash
hasn't
been
updated.
B
B
So
with
this
expectations
you
can
say
in
this
loop
I
expect,
for
example,
three
more
parts
and
then
the
controller
will
not
be
synced
on
until
the
expectations
are
met.
So
you
can
set
a
counter
to
creation
equal
to
three
and
then,
when
the
controller
observes
that
the
part
is
being
created
from
the
informer
from
the
event,
then
every
time
it
will
check
the
expectations
like
I
saw
this
expectation.
B
So
in
the
rebel
set
controller,
there's
also
something
called
look,
filtered
active
parts,
and
we
have
this
because
we
don't
want,
for
example,
if
there's
a
node
duck
unit
and
the
no
diet
no
pot
can
be
scheduled,
then,
or
maybe
there
is
there's
a
race
condition.
Then
the
cubelet
may
reject
the
part
to
be
scheduled
on
that
node.
So
the
part
will
be
failed,
but
then,
if
Rebecca
said
doesn't
ignore
that
high,
which
is
felled,
then
it
will
have
on
last
number
of
active
pods
forever
so
to
to
compensate
that
problem.
B
We
ignore
the
inactive
pods
and
finally,
the
real
logic
lives
in
the
managed
replica
set
manage
replicas
and
in
in
this
function
it
just
scale
create
and
delete
parts
based
on
the
expectations.
So,
for
example,
it
calculates
how
many
parts
it
needs
to
create.
Then
it
will
set
that
I
expect
creations
and
then
till
it
which
replica
set
expect
this
creation
and
the
number
of
creation
it
expects.
A
It's
basically
the
same
as
deployment
in
replica
set.
You
can
think
of
staple
set
as
the
controller,
with
increased
guarantees
and
lowered
expectations,
as
Janet
was
saying,
with
replica
said,
we're
trying
to
get
a
heuristic
number
of
pods
running,
but
you
can
go
past
that
scaling
point
right
and
you
can
be
under
at
any
point
I'm
the
guarantees
of
stateful
set.
Are
we
going
to
give
you
no
more
than
this
many
pods
running?
So
if
you
have
a
stateful
set
with
four
replicas,
we're
never
going
to
give
you
five
right.
A
A
A
Yeah,
so
at
the
top,
these
things
controller
revisions.
This
was
an
interesting
problem,
so
replica
set
came
first
right.
This
is
just
a
way
to
give
you
a
fungible
number
of
replicas
of
the
same
pod
made
pod
useful.
What
was
deployment
really
about
deployment
was
about
having
low
disruption
updates
for
your
replica
sets
for.
A
Right
so
inherently
in
deployment
when
I'm
scaling
this
replica
setup
and
scaling
this
replica
set
down
in
order
to
do
a
rolling
update
if
I
need
to
do
a
rollback,
a
low
disruption,
rollback,
all
I'm
doing
is
scaling
the
other
direction.
So
it
became
a
problem
of
looking
at
the
template.
Looking
at
the
template
of
a
replica
set
and
saying
oh
well,
these
be
this
replica
set
actually
implements
the
Declaration
of
the
current
intent
of
the
deployment.
A
So,
instead
of
doing
a
rollover
and
creating
a
new
replica
set
and
just
wiping
everything
which
is
usually
what
you
don't
want,
when
you're
doing
a
rollback
at
two
o'clock
in
the
morning
will
actually
say:
oh
well,
this
replica
sets
already
halfway
up
and
it
already
matches
the
spec,
so
we're
gonna,
we're
gonna
scale
this
one
up
and
then
scale
this
failing
one
down
now,
stateful
SEC
doesn't
have
an
intermediate
representation
like
replica
set.
It
works
directly
on
pods,
so
we
needed
a
way
to
achieve
the
same
thing.
A
Hence
controller
revision
now
I
think
the
demon
set
and
staples
that
are
still
the
only
actual
like
entry
controllers
that
use
it.
But
if
you
are
right
in
your
own
controller
and
you
need
a
way
to
solve
that
problem-
to
store
off
the
Deathlord
of
intent
that
you
can
detect
later
in
order
to
do
graceful
roll
backs,
you
can
use
this
object
and
its
entry.
So
what
we're
doing
at
the
top
right
here
is
we're
actually
gests.
A
So
here
we're
just
getting
the
current
revision
of
the
staple
set,
the
update
version,
the
staple
set
and
the
collision
count.
So
the
current
revision,
basically
is
here's
the
controller
revision
that
realizes
the
declaratory
user
at
this
point
in
time.
The
update
revision
is
this
is
the
what
the
users
are
requesting.
Actually,
let
me
rephrase
that
differently.
The
update
revision
is
saying:
here's
what
the
user
wants.
Now
the
current
revision
is
saying:
here's
what
I'm
currently
trying
to
reconcile.
So
the
current
revision
is
what
the
staple
set
controller
thinks.
A
A
So
drunk
history,
truncate
history
is
basically
doing
the
same
thing
that
say,
crime,
job
controller
does
with
its
jobs
or
what
deployment
does
with
replica
sets.
We
can't
store
an
unbounded
number
of
controller
revisions.
We
can't
store
an
unbounded
number
of
replica
sets.
We
can't
store
them
bounded
number
of
jobs,
though
we
used
to
in
all
cases
at
any
rate.
A
So
this
is
the
most
interesting
function.
So
what
word
can
you
guys
hear
me
by
the
way?
Is
this
lamb
enough?
Okay,
alright,
then
the
volume
seems
to
go
up
down
of
the
ante
rate,
so
we
update
staple
set.
What
we're
doing
is
we're
gonna.
Look
at
the
current
revision,
we're
going
to
look
at
the
update
revision.
A
We
set
the
we
set,
the
generation
update
the
number
of
replicas,
and
now
we
check
the
pods.
So
this
is
where
things
get
hairy
right
so
for
staple
set.
We
have
to
actually
sort
the
pods
in
the
ordinal
order
right
and
they're.
Two
different
rollout
strategies
that
we
support.
So
the
first
strategy,
which
was
the
original
one
was
ordered,
ready,
ordered
ready,
is
what
you
would
want
to
do
like.
Let's
say
you're
running
something
like
my
sequel
right
and
let's
use
Postgres
people.
That's
that's
popular
these
days,
everyone
hates
my
sequel.
A
Everyone
loves
Postgres,
so
let's
say
you're
rolling
out
Postgres
and
you
actually
want
to
set
up
a
replication
topology.
If
you
were
using
staples
set
to
do
this
one
way
you
can
do
it
using
a
little
bit
of
duct
tape
and
spit
you
roll
out
pod,
0
that'll,
be
your
master.
When
you
roll
out
pod
1,
you
pass
the
ordinal
into
that
pod.
A
You
have
a
configuration
script
that
says:
ok,
I'm,
pod,
1
I'll
go
look
at
pod,
0,
that's
going
to
be
my
master
by
default
and
I'm
gonna
put
myself
into
the
replic
and
chain
there,
and
then
you
set
up
the
replication
topology
thatwe,
ordered
ready
was
meant
to
support
that
type
of
rollout.
It's
still
used
in
like
Vitesse,
for
instance,
which
is
based
primarily
on
MySQL.
They
use
staple
sets
and
I'm
pretty
sure
they
still
use,
ordered
ready
policy
to
roll
up
their
replicas.
A
A
Allow
burst
would
be
true
right,
so
when
allow
burst
is
true,
all
of
that
kind
of
goes
out.
The
window.
You
don't
get
ordered
ready
anymore,
instead
of
doing
ordered,
ready,
it's
just
going
to
go
ahead
and
throw
them
all
out
there
at
once.
Simultaneously
zookeeper
doesn't
care
like
there's
a
well-known
theorem
that,
basically,
you
can't
do
leader
election
in
an
anonymous
ring
anyway,
so
they
all
have
to
be
configured
with
an
ID
and
they
have
to
be
configured
with
network
coordinates
of
their
peers
right.
A
So
you
throw
them
all
out
there
and
they
will
pretty
much
just
figure
it
out.
I
think
the
default
kafka
staple
set
from
confluent
actually
still
does
ordered,
ready.
I
think
they
still
do
ordered
ready
for
zookeeper,
but
one
of
the
things
I've
pushed
on
them
and
what
I
do
personally
in
production
is
turn
it
off
and
just
do
burstable
because
generally
Kafka
figures
it
out
and
generally,
as
you
keeper
figures
it
out.
A
The
bigger
problems
with
Kafka
and
staple
sets
are
more
along
the
lines
of
when
I
elect
my
broker
and
then
I
do
the
other
thing
that
staple
set
does,
which
is
roll
and
update.
I
can
have
many
many
many
leader
elections
and
switch
that
broker
leader
over
again,
that's
kind
of
one
of
the
big
problems
with
staple
set.
Today
we
don't
have
a
way
of
saying,
kill
this
specific
pod.
We
always
say
we
go
in
reverse
order
of
the
pods
that
we
rolled
out
anything
else.
Interesting.
A
But
the
other
thing
about
people
say
is
that
if
you
get
a
terminating
pot
in
the
middle
of
a
rollout,
so
let's
say
I
roll
up.
I
0
I
roll
up
hot
one.
I
roll
up
high
I'm,
not
gonna,
move
to
pot
4
until
a
pod
3
is
up
and
ready
right
now.
Burstable
allows
me
to
bypass
that
for
the
purposes
of
turning
them
up
and
tearing
them
down,
but
what
I'm
doing
a
roll
in
update
I'm,
never
gonna
bypass
that
this
is
the
thinking
behind.
A
That
is
like
okay,
sure,
if
you're
gonna
go
and
turn
them
all
up
and
you're
saying
just
do
it
all
at
the
same
time
we're
gonna!
Let
you
know
that
it
makes
sense,
updates
need
to
be
more
control
and
that's
basically
just
old-school.
Always
when
you're
doing
a
rollout
do
it
gradually,
don't
just
go
gung-ho.
A
C
A
A
Still
being
contributed
to,
we
were
going
to
use
that
as
an
example
of
how
you
can
use
coupie
builder
to
build
your
own
controllers
out
of
tree
one
of
the
takeaways
like
the
two
takeaways.
We
want
you
to
have
heard
here.
Our
controllers,
here's
how
they
work-
and
here
are
the
things
you
should
think
about.
If
you're
going
to
contribute
to
the
injury
controllers,
we
also
wanted
to
get
su.
A
Here's
how
we
do
out
of
treatment
rollers,
because
the
way
this
things
are
working
right
now
we're
more
than
willing
to
take
in
Shepard
contributions
that
live
out
of
tree
and
the
general
direction
from
Sagarika
texture.
Is
that
we'd
like
to
see
you?
They
can
really
do
everything
out
of
tree
if
possible,
but
we'd
love
to
get
deployment,
we'd,
love
to
get
daemon
set
and
those
core
controllers
and
how
they
live
is
sub
contributions
that
we
just
suck
into
the
binary
for
a
release.
A
B
For
deployment
I
rolled
back
definition,
it's
not
about
the
feature:
it's
not
working,
it's
mostly
because
the
controller
is
doing
something
that's
sort
of
anti-pattern.
So
how
the
rollback
is
done
is
that
you
set
us
and
reveal
a
row
back
to
which
revision
in
the
controller's
back
I
mean
in
the
deployments
back
and
then
the
controller
will
then
change
the
part
template
in
this
back
to
the
older
version.
B
So
the
controller
itself
is
mutating
on
the
deployments
back,
which
is
not
what
we
recommend
people
to
do,
because
the
spec
should
be
change
by
users,
so
we're
thinking
how
we
can
implement
this
better.
So
we
deprecate
this
feature
just
for
that
reason,
and
I
want
to
say
something
that
I
forgot
to
mention
earlier
about
the
expectation
so
on.
B
Why
we
have
that
in
Rebecca,
say
instead
of
having
it
but
not
having
it
and
deployment
is
because
when
deployment
creates,
Rebecca
sets-
and
it
creates
Rebecca
said-
with
an
M
generating
a
hash
from
the
pot
template.
So
the
name
is
always
deterministic,
but
when
Rebecca
said
generates
part,
the
department
is
just
a
random
stream.
So
that's
why
we
need
the
expectation.
A
A
B
A
B
B
C
Answer
your
question:
there's
there's
this
project
within
six
scheduling,
called
D
scheduler
and
as
one
of
the
the
reasons
for
the
scheduler
is
to
be
able
to
Wan
rebalance
your
cluster
evenly
if
you're
calling
scaling
up
or
down
or
I'm,
not
sure
if
that's
fixed,
but
there
is
a
plan
for
having
a
fixed
that
if
your
pod
is
in
a
crash.
Looking
for
whatever
reason
on
the
particular
node,
it
will
get
affected
and
it
will
go
through
again
a
a
scheduling
phase
which
might
fix
the
problem.
C
A
Can
add
to
that
concisely?
We
will
probably
never
make
the
core
workloads
controllers
that
run
services
anything
other
than
run.
Always
the
the
run
always
is
actually
capturing
the
users
declared
up
in
ten
and
don't
get
me
wrong.
We
could
be
moved
from
this,
but,
like
you'd,
have
to
have
a
real,
clear
story
about
what
it
means
to
have
a
service
that
runs
to
completion
like
batch
workloads
run
to
completion,
stateless,
serving
workloads
or
stateful
storage
servers
don't
run
to
completion
and
to
add
on
to
more
route
to
it.
A
He
said
like
we,
there
are
still
troubles
in
terms
of
like
mediating,
interaction
between
the
scheduler
and
the
controllers.
That's
something
that
we
definitely
want
to
focus
on
going
forward.
It's
not
clear
right
now,
like
so,
for
instance,
what
is
it?
Security
policies
right,
so
I
can
declare
security
policies
inside
of
a
replica
set
that
will
get
scheduled
onto
a
node
that
don't
support
the
security
policy
that
I've
configured
right,
like
that's
a
bad
thing
right
now,
our
answer
is:
don't
do
that
we
need
to.
A
We
need
a
better
answer
there
right
so
figuring
out
how
to
deal
with.
That
is
currently
a
topic,
that's
in
flight,
so
no,
we
probably
won't
do
anything
but
restart
always.
But
yes,
we
are
continuing
to
address
and
be
concerned
with
the
general
interaction
between
the
components,
the
core
components
of
the
cluster
and
trying
to
improve
that
experience.
B
A
Mother
Monday
now
9:00
a.m.
Pt
yeah.
We
have
a
slack
channel.
We
have
a
community
page.
If
there's
any
other
questions,
you
want
to
ask
feel
free
to
come
to
the
meeting
and
ask
them.
If
there's
any
contributions,
you
want
to
make
or
get
started
on,
it's
a
great
place
to
kind
of
come
and
check
out
if
ya,
just
come
and
say
hi.
If
you
like.
Thank.