►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
The Life of a Kubernetes Watch Event - Wenjia Zhang & Haowei Cai, Google
The watch event is essential to the kubernetes architecture. It’s the key to maintaining high availability in the kubernetes control plane. Have you ever wondered how a watch event is propagated? In this presentation, we will cover how kubernetes delivers a watch event through control plane storage, API server, and finally to clients, and what happened to a watch event across server replicas. Attendees will leave with a full understanding of the life of a kubernetes watch event, which could help you make better decisions to implement your controllers in a much more scalable and performant way.
To learn more: https://sched.co/GrUX
A
Thank
you,
everybody
for
coming
to
our
session
today.
We're
gonna
talk
about
the
life
of
a
coronaries
watch
event.
So
a
lot
of
introduction
of
ourselves.
My
name
is
wenja
I
work,
I'm,
a
software
engineer
working
at
Google
I
worked
on
kubernetes
what
API
machinery
and
sed
and
joining
me
is
highway.
He
is
also
a
software
engineering
engineer
working
on
Google.
He
worked
on
API
machinery
and
client
libraries,
so
this
is
agenda
of
our
talk
today
we
will
start
with
talking
about
what
is
a
community's
watch
event
and
then
we'll
go
over.
A
Why
is
it
important
for
kubernetes?
Why
do
we
care
and
then,
after
that,
we're
gonna
going
to
the
most
important
part
of
a
talk?
It's
gonna
be
the
longest
session
about
how
is
the
life
of
communities
watch
event.
We're
gonna
go
down
into
each
component
that
this
watch
event
is
going
to
go
to
and
see.
What's
going
on
with
the
watch
event
in
the
particular
components
by
the
end
of
the
session,
hopefully
everybody
could
get
something
useful.
A
Maybe
you
could
use
watch
more
efficiently
to
write
your
controller
or
you
have
a
better
idea
how
to
contribute
to
this
part
of
kubernetes
woad
all
right
now,
the
first
part.
What
is
the
kubernetes
watch
event?
Let's
break
it
down
to
smaller
parts.
First,
what
is
a
watch?
We
know
that
kubernetes
api
is
resource
based
programmable
interface,
that
provided
via
HTTP,
so
it
support
all
the
standard,
HTTP
verbs,
post,
pert
patch
to
lead
and
get
now
on
top
of
all
that
watch
is
another
verb
that
makes
kubernetes
api
so
special.
A
It
is
this
incremental
change
notification
that
is
showing
the
change
to
the
resources,
and
it
also
consists
that
that
allow
all
the
other
components
to
cache
and
synchronize
the
state
of
all
the
resources
so
watch
establish
a
single
connection
that
allow
the
users
to
have
low
in
latency
feed
of
the
changes
well
to
explain
it
more
easier.
I
put
the
poll
here:
you'll
always
need
a
bad
guy.
I
mean
I
talk
about
superheroes
when
you
say
establish
polling.
A
You
have
to
establish
multiple
connections
which
add
on
more
the
actual
load,
actual
agency,
which
could
be
avoided
when
you're
doing
watch
I'm
not
denying
that
there
are
a
lot
of
cases
where
polling
is
better
than
watching,
but
not
in
the
content
of
today's
talk.
That's
a
picture
I
took
from
yesterday's
party.
The
coop
come
party
is
great.
It's
awesome.
A
Now
we
talked
about
the
watch,
then
what
is
the
event
we
got
out
of
the
watch
event,
a
single
change
to
a
watch,
the
resource
it
contain
the
resource
object
itself,
as
well
as
the
change
type
or
event
type.
There
are
some
examples:
adding
a
pot
could
be
event
or
modifying
replica
set
or
deleting
a
note
right
now
we
have
the
definition.
Let's
talk
about.
Why
do
we
worry?
Is
it
important
for
communities?
A
Let's
see
one
of
the
most
important
criminales
core
design
concept,
level
triggering
and
soft
reconciliation,
so
in
kubernetes
everything
is
a
declarative
configuration
object
that
represents
the
desire
state
of
the
resource
in
this
kubernetes
job,
to
watch
all
the
changes
and
make
sure
the
actual
state
of
the
world
would
eventually
get
to
the
desired
state
right.
It's
very
forgiving
of
what's
happening
in
between
in
that's
why
communities
is
self-healing
and
works
so
well.
I
actually
start
to
apply
this
concept
to
my
kids
I'm.
A
Why
is
watch
event
important
in
this
extra
state
to
desire
state
transition
say
if
we
have
a
person,
we
want
this
person
to
be
at
state
of
happy
and
strong
confident.
How
do
we
get
that
person
to
that
state?
I,
say:
status,
accumulation
of
events
from
the
first
day
that
person
was
born,
probably
one
of
the
most
important
event
that
happens
to
a
person.
A
It's
not
the
state.
Yet
so
something
happened
to
that
person.
Maybe
it
played
volleyball
and
win
a
championship.
The
state
of
person
is
modified
later
on,
I
got
the
automa,
maybe
some
other
events
happen
to
that
person.
You
eventually
get
to
that
state
in
the
same
concept
apply
to
a
pot.
If
you
have
a
pot
with
a
desired
state,
you
could
have
a
lot
of
events
happen
to
that
pot.
A
When
the
replica
controller
created
that
pod,
it's
like
the
first
day
that
person
is
born
and
to
later
on
you
add
some
labels
like
when
the
person
won
the
championship
and
an
image
change
image
version
change
to
that
pod
and
a
lot
of
other
events,
eventually
the
pot
get
to
the
state.
So
if
you
watch
the
change
carefully,
coronaries
could
have
everything
under
control
efficiently.
A
Now
we
know
what
and
why?
Let's
talk
about,
how
is
the
life
of
communities
watch
event
one
of
the
most
important
concept?
Yet
that
could
help
you
understand.
The
life
of
corn
at
ease
watch
event
is
resource
version.
If
you
don't
already
know
everything,
every
crew
Nettie's
object
have
resource
version
field
that
represent
the
version
of
the
object,
let's
say
them
in
another
way.
So
it's
like
a
fingerprints
that
differentiate
this
object.
In
a
specific
version,
so
there
are
two
dimensions:
object
and
version.
This
is
a
diagram
showing
the
relationship
of
resource
version
and
object.
A
So
in
the
kubernetes
system
you
could
have
a
lot
of
objects.
Pods
replica
sets
nodes
and
so
on
and
for
each
particular
object.
There
could
be
a
lots
of
events
happening
to
it.
Every
time
event
happened
to
the
resource.
Every
time
the
resource
ridden
there
is
resource,
new
briefer
source
version
created
and
the
parrot
to
that
pot
that
object
now,
let's
just
focus
on
this
pot
and
ignore
all
the
other
stuff.
Hey,
that's
the
happy
and
strong,
confident
pub
we
just
talked
about
so
now
in
this
graph.
A
That's
putting
the
resource
version
at
first,
the
reefer
source
version
say
it's
five,
one
nine
and
then
later
on,
when
the
pot
is
modified,
change
to
six
zero
three
and
they
keep
changing.
It's
like
you're,
not
the
same
you
as
your
last
year
last
month,
I'm
not
the
same
as
yesterday
be
excited.
Give
this
talk.
This
event
changed
me
right.
My
talk
is
getting
deep.
A
This
is
an
event
that
we
want
to
take
a
look
and
then
who
are
the
parties
who
care
about
this
event
who
want
to
watch
this
event?
A
couple
of
them
come
to
my
mind.
Schedulers
job
is
to
schedule
the
part
of
note
so
I
bet
they
want
to
watch
the
event
of
pot.
Then
controller
manager
is
one
of
those
who
is
interested
watching
the
event
and
there
other
than
the
system
controllers.
There
are
maybe
some
a
lot,
some
other
customer
controller
who
have
different
business
logic,
but
eventually
they
want
to
watch
this
event.
A
So
this
is
the
destination
of
this
event
and
let's
go
back
to
the
very
beginning
where
it
started.
So
when
the
pot
is
created,
your
first
store
at
the
sed,
which
is
the
data
store
for
kubernetes
control
plane,
and
then
it
goes
from
a
CD.
It
goes
to
API
server
and
then
goes
to
client.
Go
eventually
from
clang.
Go
could
go
to
all
the
clients
who
is
interested
in
this
watch
event.
Alright,
with
that
overview,
let's
go
into
each
component
and
see
more
details
watch
event
in
a
CD.
A
Lcd
is
distributed,
key-value
store
for
kubernetes
it
for
coronaries
control
plan,
and
then
it
implements.
This
watch
feature
that
which
provided
the
event
based
interface
for
asynchronously
monitoring,
change
to
keys
so
everytime
when
there
is
some
change
happen
to
a
key,
the
Watchers
get
notified,
which
is
exactly
what
the
communities
need
and
in
a
CD
there
is
this
concept
revision,
which
maps
to
the
resource
version
we
just
talked
about
in
API
server.
A
So
this
is
the
diagram
showing
the
wasp
watch
stream
between
a
city
and
its
client.
It's
a
binder
that
bi-directional
stream,
so
the
client
could
right
to
the
stream
to
establish,
watch
and
on
the
other
direction
it
get
the
watch
event
and
in
the
kubernetes
control
plan.
The
client
of
a
CBS
API
server,
which
leads
to
the
next
component
API
server.
A
Now
this
is
a
tongue
twister
slide,
I've,
already
simplified
it
down
to
three
components.
If
you
read
the
source
code,
it's
like
all
the
possible
combination
of
cache
and
watch
the
two
words.
A
lot
of
fun,
read
it
so,
the
first
component
as
any
watcher
from
the
name.
You
can
get
the
idea
it
get
the
event
from
a
CD
and
bang
API
server
and
have
some
extra
process
going
on
including
flow
control
and
transformation
of
it
at
CDs.
Three
events:
two
API
server
events.
A
After
that,
it
goes
in
to
watch
cache,
which
is
probably
the
one
of
the
most
important
components:
api
server
that
is
responsible
for
watch
in
cash
I'm.
Sorry
watch
in
lists,
so
it
contains
cash
and
store
cash,
is
basically
a
circular
buffer
and
store
is
just
an
interface
that
connect
to
the
underneath
bailiffs
or
so
what's
going
on
is
when
you
have
an
event
coming,
it's
written
in
the
cache,
and
the
circular
cache,
of
course,
have
a
limited
capacity.
A
When
that
capacity
is
reached,
it
will
like
the
new
events,
will
be
written
on
the
top
of
the
oldest
one,
and
then,
when
you
do,
the
watch
you
watch
on
was
starting
from
a
specific
resource
version.
If
the
resource
version
is
doing
the
cache,
it
start
watching.
Everything
happened
after
that.
If
the
resource
version
is
too
old,
not
in
the
cache
anymore,
it
do
the
list
and
get
the
status
from
the
store
and
then
start
watching
from
that
point
on
after
that
component
is
a
bunch
of
Watchers
for
each
different
clients.
A
B
B
Franco
is
a
credit
package
that
contains
multiple
goal
and
client
libraries.
That's
used
for
talking
to
a
kubernetes
cluster,
subsequently
talking
to
you,
Korea's
API
server
to
those
who
are
familiar
with
Clank.
Oh,
let's
take
a
quick
look
at
some
basic
functionality.
It
provides.
We
have
the
client
set
which,
which
contains
typed
clients
for
talking
to
a
specific
current
list,
API
kind
you
can
do
it
with
type
safety
guarantee
and
portable
of
sterilization
McClinton
for
efficiency.
B
We
have
the
dynamic
client,
which
is
a
flexible
interface
for
talking
to
multiple
currency
via
kind
within
a
single
client.
We
also
have
the
rest
client,
which
is
the
underlying
interface
for
both
client-side
and
dynamic
client
clinco
was
initially
developed
for
kubernetes
internal
use.
It's
exported
as
a
single
package
for
developers
to
extend
create
api
for
their
own
purpose.
That's
a
quick
overview
of
some
basic
capability
of
client
go,
but
that's
not
our
focus
today.
B
B
The
main
components
of
a
Informer
includes
a
reflector
which
watched
and
lists
upstream
API.
Multiple
callback
functions
that
dispatched
notation
to
downstream
work
queue
and
two
layers
of
cache
or
sometimes
called
stores
in
between
the
two
layers
of
cache
serves
different
purpose
and
has
some
interesting
attributes.
We
will
come
back
to
them
later
in
detail.
B
Okay,
so,
let's
take
a
look
at
general
crew
net
controller,
workflow
level
triggering
and
software
constellation
is
a
big
concept
in
kubernetes,
a
controller
speaks
with
cube.
Api
server,
constantly
reads
the
current
state
of
the
cluster
and
drives
the
current
state
to
its
desired
state
by
sending
requests.
If
we
take
a
closer
look
to
some
of
the
building
controllers
in
Carreras,
you
will
find
that
we
can
logically
split
a
controller
into
two
parts.
B
Clinical
component
works
as
a
template
and
controller
specific
business
logic
that
facing
the
template.
Inside
of
a
controller
in
former
constantly
watched
changed
for
the
for
a
specific
API
client
within
a
controller,
a
reflector
does
a
initial
list
of
the
API
kind
and
constantly
watch
following
change.
For
example,
we
we
had
the
part
that
which
I
mentioned
before
that
continuous
journey
from
a
PI
server
to
kind.
The
first
step
is
to
get
through
the
reflector.
B
After
that,
every
watch
event
gets
piped
to
a
Delta
Pi
for
queue,
which
then
trigger
callback
function.
A
added
event
triggers
around
that
function.
The
updates
event
triggers
an
update
function.
The
callback
functions
are
registered
by
controller
authors
with
their
controller
specific
business
logic.
So
basically,
the
life
cycle
of
a
watch
event
ends
here.
The
changing
at
C
D
is
represented
by
a
watch
event
and
gets
delivered
across
multiple
components
from
API
server
and
client
go
and
get
to
a
controller
trigger
some
controller
specific
business
logic.
B
But
let
me
just
give
you
a
little
bit
more
background
about
what
code
was
going
on
afterwards
to
tell
a
more
complete
story
when
controller
would
say
event.
Usually
it
wants
to
react
to
it.
Clinical
provides
a
useful
package
called
work,
use
that
helps
controller
to
do
the
reaction
more
efficient
and
more
safe.
B
The
walk
you
basically
inside
of
the
callback
function,
the
controller
can
add
items
to
the
work
queue
for
it
to
get
process
process
later.
The
work
here
has
a
few
interesting
attributes.
First
allows
the
controllers
to
bring
up
multiple
workers
that
process
at
him
concurrently,
so
it's
more
efficient
and
it
guarantees
that
a
single
item
only
get
processed
once.
B
B
B
B
For
example,
say
we
previously
watched
the
create
event
for
part
one
and
we
store
it.
Instead
of
the
local
cache
resource
portion
cuts
through
one
nine
one,
five,
one,
nine
we
have
its
back,
and
now
we
have
watched
just
another,
updated
event
for
the
same
part,
instead
of
keep
to
copy
off
part
instead,
local
cache
Lokesh
reflects
the
change,
saves
this
new
resource
version
and
post
back
in
its
cache
every
facts:
the
current
state
of
the
cluster,
this
outs.
B
B
After
processing
the
changing,
subclass
or
a
consumer
wants
to
send
request,
fukube
a
customer
to
drive
the
current
state
to
its
desired
state.
The
controller
can
use
clients,
libraries
that
client
go
offers
that
we've
just
mentioned
before.
To
do
that.
Okay,
as
a
recap,
the
watch
event
lifecycle
basically
ends
in
the
black
box.
So
why
is
sim
former
trustworthy
in
delivery
of
watchbands
I
mean?
Can
we
miss
the
watchband
somehow?
B
How
does
the
individual
work
event
accumulates
to
reflect
the
current
state
in
South
Korea?
Ladies,
let's
take
a
closer
look
in
the
list
and
watch
function
instead
of
reflect,
go
reflect
herself.
So
this
is
a
pseudo
code
of
lists
and
watch
function
in
self-worth
reflector
when
a
reflector
gets
created,
it
takes
a
list
watch
her,
which
is
an
interface
that
can
do
lists
and
watch
requests
for
a
specific
API
climb.
B
The
roof
actors
does
a
list
request
to
a
PI
server
and
gets
a
list
of
the
object
objects.
The
list
object,
gets
piped
to
the
theatre,
Q
Delta,
Pi
pool
cue
and
triggers
long
add
on
updated
until
it
public
functions.
Okay,
of
course,
if
you
just
create
a
Informer
with
empty
cache,
the
other
objects
listed
are
new
to
the
Informer,
so
they
were
all
trigger.
Uncle,
add
functions,
but
we
will
see
some
different
case
later
here.
We
just
extract
the
actual
resource
version
we
just
listed
with
the
announced
terminating
full
loop.
B
The
reflector
tries
to
open
a
watch
connection
with
the
latest
resource
version.
If
we
hit
her
anger
and
narrow,
we
validate
the
arrow
and
see
if
we
need
to
restore
from
a
list
if
the
API
server
is
not
responsive
and
returns,
a
connection
refused
the
error
most
likely
it
doesn't
worse,
restart
from
list
list
because
most
likely
will
end
up
restart
watch
from
with
what
we
just
watched.
B
The
two
layers
of
cache
has
this
interesting
attributes
that,
when
the
reflector
triggers
are
released,
it
compares
listed
objects
with,
what's
inside,
of
its
local
cache
and
triggers
an
ad
function
for
new
listed
objects
that
didn't
exist
in
the
local
cache.
An
update
function
for
those
objects
already
exists,
but
changed
until
it
event
until
it
Quebec
functions
for
those
who
used
to
be
inside
of
local
cache
but
didn't
exist
in
the
list.
Objects
in
this
way,
Informer
recovers
from
broken
watch
and
the
controller
specific
business
logic
doesn't
even
notice
that
and
can
continue
without
crashing.
B
B
So
it's
in
it's
important
to
understand
how
kubernetes
use
the
resource
version
to
keep
API
server
and
clients
up-to-date
during
communication,
as
which
I
mentioned
before
a
recent
version,
is
like
a
fingerprint
of
a
query
object.
So
let's
have
a
recap
of
how
resource
person
takes
place
in
which
bets.
Basically,
everything
has
a
resource
version.
B
Individual
API
object,
as
a
resource
version
will
get
one.
For
example,
pod
will
get
a
poly,
you
get
a
resource
version.
It's
metadata
for
this
topic
have
objects,
there's
a
top-level
resource
version
for
the
entire
list,
and
each
API
object
in
its
items
has
a
resource
version
for
itself.
The
resource
version
of
the
top-level
list
is
what
should
be
used
when
you
start
watch
to
observe
events
occurring
after
that
list,
you
can
specify
resolution
in
less
than
watch
requests.
B
Both
lists
and
watch
are
handled
by
the
same
endpoint
in
API
servers,
so
you
use
this
option
for
post
case
to
specify
the
recent
version
for
a
list
request.
If
you
don't
specify
the
resolution,
it
has
server,
will
query
at
CD
to
get
the
object
list.
If
you
specify
resource
person
greater
than
zero
API
server
will
actually
wait
for
its
cash
to
get
synced
with
EDD
to
make
sure
the
result
is
at
least
as
fresh
as
given
resource
version.
B
B
When
you
pass
in
a
scenario.
Sometimes
the
cache
of
some
API
server
may
can
be
more
up-to-date
than
others.
So
when
inspired
by
resourceful
equal
to
zero,
it
indicates
that
you
are
willing
to
get
data
from
a
arbitrary
style
cache
so
make
sure
that
your
component
can
tolerant
to
use
that
and
for
which
request.
If
you
don't
specify
resource
version,
the
result
will
be
which
events
starting
from
random
time
point.
If
the
resource
version
zero,
the
result
will
be
an
event
for
every
existing
objects
in
the
cache
and
change
occur
after
that
which
was
established.
B
Remember
that
a
controller
can
be
composed
by
two
parts:
the
client
go
as
a
template
to
deliver
the
event
from
Exedy
to
your
controller
and
the
red
part.
Is
your
controllers
specific
business
logic.
So
let's
do
the
name
from
Korea's
culture
player
and
build
a
mini
schedule.
Guess
what
it
does
is
schedule
your
pot
on
three
nodes.
B
B
Callback
function
to
get
notified
of
the
rip
watch
events.
So
let's
do
this,
we
add
a
event
handlers
at
future
resource
event
and
if
the
pot
event,
which
meant
returned
from
poly
informer,
is
a
how's
that
schedule
it's
great,
we
just
keep
it
in
cash
and
for
the
left
bottom
part.
You
cannot
see
the
last
part,
but
it's
okay,
I'm
zoomed
in
for
it's
either
to
render
here,
but
nothing
surprising,
going
out
here
for
a
unscheduled
pod.
B
We
added
it
to
a
schedule,
cue
that
we
want
to
process
later
and
same
thing
with
the
four
node
we
have
a
node
cache.
Basically,
that's
what
you
need.
You
can
evaluate
a
pod
compare
with
or
the
nodes
you
you
have
and
see.
If
there's
any
candidate
that
has
enough
resources
for
the
pod
to
be
scheduled
down
the
node
and
Romeo
workload.
B
So
we
are
down
great
right
guess
what
the
real
group
schedule
is
way
more
complicated
and
the
mini
scheduler
that
we
just
build.
If
you
take
a
look
at
the
source
code
of
the
scheduler,
it
has
ten
informer
for
ten
different
API
kinds
in
kubernetes.
That's
all
it
needs
to
have
the
information
to
make
sure
that
a
pod
can
fit
on
the
node
and
run
your
workload
successfully.
B
Can
you
mentally
how
scheduler
can
do
this
kind
of
the
information
processing
without
watch
events?
Alternatively,
we
can
send
list
request,
fukui,
API
server
and
pulling
periodically,
but
like
every
one
second,
five
second,
and
do
the
serialization
deserialization
encoding
decoding
and
depending
on
the
size
of
the
API
object
in
the
cluster.
It
just
won't
scale
right.
B
But
with
watch
events,
we
can't
with
this
efficient,
okay,
key
takeaways.
So
in
summary,
a
koreas
watch
event
is
a
efficient
resource
change
notification,
which
meant
is
the
key
to
commence
the
label
triggering
and
software
conciliation,
which
went
is
trust,
water
and
fish,
and
if
you
are
trying
to
write
a
controller,
you
can
use
the
informal
component
it's
built
for
it.
It's
developed
for
build
event,
oriented
controllers
and
helps
you
to
not
make
methods
watch
thanks
for
joining
us
today.
We
hope
that
you've
enjoyed
cube,
come
and
enjoy
Seattle
any
questions.