►
From YouTube: Building a Fault-Tolerant Resources Controller on Kubernetes - Morgan Bauer & Srinivas Brahmaroutu
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Building a Fault-Tolerant Custom Resources Controller on Kubernetes - Morgan Bauer & Srinivas Brahmaroutu, IBM Corp. (Intermediate Skill Level)
CRD (custom resource definition) is widely used to extend the behavior of Kubernetes. As all other kubernetes resource have controllers, so do CRDs. It is important that the custom resources are managed reliably during any failures of the custom controller specific to any CRD, such that a consistent state is maintained across failures. This talk will share the experiences we've gained: how leader election is used to ensure liveness, design principle of namespaced CRDs vs. clustered CRDs, how to handle unexpected events from APIServers, and maintaining the running system.
About Srinivas
Srinivas Brahmaroutu works as a Software Engineer at IBM Corp. He has many years of experience around IBM cloud offerings. He has worked on many strategic open source projects including Cloud Foundry, Docker and Mesos. Currently he works on Kubernetes contributing to test-infra and conformance.
About Morgan
Morgan Bauer works on Open Source at IBM in San Jose as part of the Cloud Division. Morgan is a maintainer on the core Docker Engine and also a founding contributor of the Kubernetes Service-Catalog.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hi
everyone
thanks
for
joining
our
session.
This
is
Srinu
s,
Rama,
Rao
and
Morgan
Bar
way
honk.
He
could
not
make
it
to
this
conference,
he's
the
one
who
brought
us
this
year's
case
he's
been
working
on
this
for
about
a
year
through
lots
of
changes
that
happened.
Finally,
we
landed
with
custom
resources.
At
this
point.
We
talked
about
that
and
have
Morgan
kick
off.
B
Testing,
ok,
good
yeah,
like
he
said
this
is
a
use
case
brought
to
us
by
a
co-worker
who
works
locally
with
us
at
IBM
in
svl,
which
is
in
San
Jose
and
fault-tolerant
customer
resource
controllers.
So
let's
begin
we're
going
to
go
over
what
our
CR
D
is,
what
why
they
have
a
how
they're,
basically
like
standard
API
objects,
and
why
you
need
a
controller
to
perform
actions.
B
An
important
part
of
the
fault
tolerant
bit
is
the
leader
election
process.
Why
you
want
your?
You
know
stuff
to
be
fault,
tolerant
in
general,
we
have
these
different
use
cases.
These
different
patterns
that
we've
or
that
way
has
has
noticed
that
exists
when
when
the
controllers
have
been
implemented
internally,
and
so
we're
gonna
go
over
the
two
that
we
have
here,
that
are
called
controller
and
control,
II
I'm
sure
this
is
a
pattern
that
already
exists
and
has
official
name.
B
But
you
know
we
wrote
the
slides
when
we
wrote
the
slides
and
then
we
have
some
basically
best
practices
on
you
know,
name
spacing
and
implementation
details
etc.
So,
let's
begin,
what
do
we
have?
We
have
custom
resources
and
controllers,
so
we
have
this
new.
Well,
it's
relatively
new
in
the
last
year
or
so
see
our
DS
custom
resource
definitions
have
become
a.
B
B
The
other
side
of,
of
course,
of
the
API
server
is
the
controller
model,
and
so
you
have
to
have
a
controller
that
either
sees
the
C
RDS
or
creates
the
CR
DS,
and
this
implements
the
standard
sort
of
spec
and
status
declarative
model.
That
kubernetes
has
other
implementations
of
this.
Have
you
seen
you
know,
obviously,
during
the
conference
of
big
one
is
the
operator
pattern
and
then
at
least
I've
seen
during
the
conference
that
the
the
controller
is
sort
of
SDK
cube.
B
So
how
did
this
come
about?
This
came
about
because
Wei
has
been
implementing
an
actual
CRD
controller
and
Ciardi
use
case
in
the
backend
started
out,
with
extending
the
actual
kubernetes
code
base
with
a
bunch
of
private
patch
sets
that
were
maintained
on
top
of
the
kubernetes
source
code
and
they
carried
forward
from
the
11.4
1.3
1.5
time
frame,
and
then
this
is
obviously
not
good.
B
So
they
took
it
from
one
steep
er's
became
a
thing.
They
took
these
custom
definitions
from
entries
or
static
resource
and
definitions.
Tp
are
sort
of
never
got
off
the
ground
in
a
great
way.
It
was
marked
as
beta,
but
was
never
beta
in
the
way
that
beta
is
typically
used
inside
of
the
kubernetes
codebase.
It
had
lots
of
things
that
were
broken
from
the
beginning,
that
some
of
them
got
fixed
and
basically
CR
DS
is
third-party
resources
version
2
and
of
around
the
1.7.
B
Okay,
next
next
part
is
the
leader
election.
So
you
want
your
controller
to
be
able
to
be
fault,
tolerant
and
one
of
the
ways
to
do
that.
In
terms
of
you
know,
a
node
goes
down
or
your
leader,
crashes
or
whatever
is
to
have
a
quicker
ability
to
know
that
it's
dead
and
either
failover
or
restart
it
leader
election
is
one
of
the
modes
provided
by
the
client
go
toolkit
and
so
that
they've
taken
advantage
of
this
sort
of
provided
functionality.
B
So
if
we
think
through
some
of
these
use
cases
on
well,
why
can't
I
just
say
give
me
five
copies
of
the
controller,
and
you
know
if
one
goes
down.
The
other
four
can
keep
going
well,
at
least
in
this
object
world.
You
don't
want
to
have
five
different
controllers,
getting
the
same
resource
watch,
update
stream
and
then
all
five
try
to
do
something
to
create
that
state
reconcile
the
declarative
state
and
then
they
all
do
the
thing,
and
you
get
five
updates
when
you
really,
you
only
needed
one.
B
What
if
we
just
say,
ok,
there's
one
replica
and
we
let
Cube
reconcile
that.
Ok,
if
it
goes
down
it
comes
back
up.
I
guess
I
did
this
in
the
opposite
order,
but
the
problem
with
that
is
that
it
takes
an
awful
long
time
to
notice
that
the
the
pods
go
down
and
at
least
in
the
default
configurations,
and
while
you
can
create.
B
B
It's
the
actual
backing
thing
as
I
understand,
it
is
provided
by
Etsy
D,
but
it
is
exposed
in
a
cube
manner
in
that
they're
there.
But
there
is
a
client,
Glo
library
extension
and
it
creates
either
an
endpoint
or
a
config
map
that
has
basically
the
they
all
the
controllers
that
are
trying
to
do
this
and
try
to
basically
take
the
lock
and
then
try
to
keep
the
lock
and
one
goes
down.
B
There's
a
formula
in
the
code
that
you
should
go
read
that
says
how
you
should
try
to
set
these
three
different
periods
and
then,
once
you
do
this,
you
basically
specify
a
deployment
where
replica
sees
is
greater
than
one.
When
this
actually
occurs,
one
replicas
will
be
the
leader
and
the
rest
will
be
passive
and
you
need
to
make
sure
that
you
do
enable
pod
anti
infinity,
because
it
doesn't
really
do
you
much
good
to
have
multiple
copies
of
something
all.
B
A
A
A
A
We
work
in
names
face
demo,
we
create
a
namespace
and
then
we
create
our
back
rules
to
run
within
that
new
face.
I
can
show
you
our
back
if
I
have
time
in
the
end,
or
we
also
discuss
about
how
to
control
the
custom
resource
using
our
back
rules,
but
in
general.
At
this
point
we
are
installing
a
custom
controller.
A
A
A
A
A
A
A
Okay,
much
better
so
somehow
my
are
back
did
not
run
so
we
got
an
end
point
now.
We
know
that
one
of
the
two
parts
that
we
have
is
the
leader.
So
it's
listed
out
there
as
a
with
the
holder
identity.
So
when
you
do
Q
control
get
parts
on
that
demo,
we
got
two
parts.
The
first
part
is
the
leader.
So
if
I
do
keep
control
logs.
A
A
Describe
I
better,
not
do
cut
and
paste
describe,
for
example,
flu,
that's
much
better,
so
this
is
the
example
fool
and
essentially
what
we
did
here
is
we
have
a
custom
controller
that
is,
that
is
looking
at
this,
who
are
custom,
racers
and
created
a
deployment
for
us
and
running
nginx.
What
we
do
at
this
point
is
we
we
see
if
we
can
generate
some
traffic
on
this
by.
A
A
A
So
on
back
to
the
previous
window,
what
we
can
now
do
is
see
there
are
passive
and
active
controllers
running,
so
we
want
to
shut
down
the
active
controller
Nord,
creating
artificial
node
failure
or
hardware
failure.
So
if
we
need
to
know
where
that
particular
part
is,
if
we
run
this,
we
see
that
f1
is
our
primary
active
controller
that
is
running
on
key
of
node
one.
The
trick
I
do
since
I
am
running
in
in
darker
I.
A
A
A
B
B
So
we
saw
that
we
saw
that
it
came
back
up.
Eventually
we
had
a
little
difficulty
with
some
of
it,
but
we're
here
so
I'm
gonna
describe
the
two
different
sort
of
patterns
that
we've.
That
way
is
seen
in
his
implementations
of
multiples
of
these
things
behind
the
scenes,
so
one
of
the
what
he
likes
to
call
it
as
controller
versus
control,
II
and
the
controller
pattern
is
where
we
say
the
CR
is
the
thing
that
you
interact
with
directly.
You
create
custom
resources,
you
poke
them,
you
interact
with
them.
B
They
are
the
the
essence
of
whatever
your
your
program
is
designed
to
do
so.
You
will
use
cube
control
to
create
them
where
you
use
client
code
and
then
you'll
have
your
own
custom
controller
that
watches
whatever
the
CR
is
and
perform
some
action.
So
if
you
go
to
the
the
actual
example
controller,
is
this
type
of
pattern?
B
This
concept
is
I
would
say
more
evident
in
things
like
the
operators,
so
at
City
Operator
you
create
a
custom
thing,
you
create
you,
a
install
the
controller
and
it
looks
at
the
created,
see
our
object
and
does
a
bunch
of
stuff
to
create
some
kind
of
@cd
incidents
or
a
Prometheus
instance.
So
the
other.
The
other
pattern
that
we've
seen
is
the
controller
pattern.
Where
you
create
normal
objects
in
cube,
you
have
a
first
controller
that
sees
changes
to
those
objects
and
creates
a
custom
resource
object
that
you
don't
normally
interact
with.
B
So
the
the
use
case
that
we
have
for
this
was
custom
load,
balancer
types
and
the
the
internal
cloud
that
they
have.
So
you
you
create
a
service
and
the
service
is
some
kind
of
special
load
balancer,
and
so
there
is
a
CR
that
represents
the
fact
that
this
load
balancer
gets
created
and
then
that
CR
is
what
is
used
as
sort
of
the
end
point
for
management
of
the
actual
load.
Balancing
technology
behind
the
scenes.
B
Ok
describe
that,
so
some
sort
of
you
know
best
practices
or
useful
ways
to
look
at
name
spacing
versus
cluster
availability
is.
If
your
object
is
the
controller,
you
probably
want
to
have
it
in
a
namespace
such
that
people
can
have
one
in
their
namespace
and
then
people
not
in
that
namespace
can't
touch
it.
B
But
if
it's
a
controller
pattern,
you
want
to
put
your
CR
in
these
sort
of
the
global
namespace
or
in
a
special
namespace
that
nobody
can
normally
touch,
except
for
your
controller
so
that
it's
isolated
and
people
can't
touch
it
because
they
shouldn't
be.
It's
really
an
you
know,
a
computer
consumer
only
and
you
can
use
our
back
to
enforce
all
of
these
things.
A
A
A
A
A
A
A
We
are
not
going
to
create
the
bar
object
directly,
so
we
need
to
change
something
in
the
cluster
that
would
trigger.
In
this
case,
we
do
a
deployment
and
the
deployment
is
triggers
our
controller
to
create
the
bar
object,
the
custom
resource,
and
we
do
not
control
the
the
bar
object
by
ourselves
and
in
turn
the
bar
object
will
create
a
service
object
for
the
deployment
we
have.
We
are
creating.
A
A
A
B
Already
the
slide,
ok
last
bit
here,
make
sure
when
you're
writing
your
your
client
code,
they've
noticed
a
lot
of
things
being
hard-coded,
so
you
want
to
be
relying
on.
You
know
things
like
interfaces
over
direct
struck
references
right
now,
at
least
an
hour,
a
particular
example
here,
there's
no
particular
validation
being
done
on
CR
days.
I've
heard
that's
a
big
thing,
that's
coming
in
in
111!
B
So
right
now,
both
of
our
CR
objects
are
wildly
different.
Once
you
actually
look
at
the
the
code,
but
when
we
actually
defined
them
they're
the
same
actual
definition
with
something
that
has
no
bearing
on
the
actual
type
definition
in
types
go,
so
you
have
to
make
sure
that
you
do
validation
in
your
in
your
client
code
as
it
is
right
now
versus
it
being
done
automatically
as
per
standard,
Kubb,
API
server
and
the
next
that
that
leads
into
sort
of
these.
B
B
You
know
harkens
back
to
that
that
earlier
statement,
I
made
where
you
don't
want
to
have
multiple
controllers
acting
on
the
same
resource
and
in
the
last
bit
not
on
here
is
that
you
need
to
be
very
vigilant
in
your
in
your
code
for
validating
that.
The
object
that
you
are
getting
over
the
watch
stream
a
still
exists
and
be
is
not
a
a
duplicate
of
some
other
event.
B
B
I
mean
that's
it.
That's
a
general
cube
pattern
in
terms
of
its
basically
establishing
the
state
machine
and-
and
you
should
create
you
know
you
should.
In
my
opinion,
you
should
diagram
out
the
state
transitions
that
you
have
and
make
sure
that
you
enforce
that.
You
know
the
state
transitions
happen
and
that,
in
terms
of
spec
to
status
that
you
almost
sort
of
write,
what
you're
about
to
do
that
becomes
your
status.
You
go.
B
A
B
The
standard
timeouts
on
cube
or
something
like
five
minutes,
which
in
the
best
case
you
know
it,
is
a
couple
seconds
over
five
minutes
of
all
the
different
timeouts
happen
like
one
after
the
other.
If
they
happen
in
sort
of
a
bad
state,
it's
like
five
minutes,
plus
5,
more
minutes,
plus
5
more
minutes
a
couple
of
different
times
so
I've
seen
intent.