►
From YouTube: Planes, Raft, and Pods: A Tour of Distributed Systems Within Kubernetes [B] - Bo Ingram, Craftsy
Description
Planes, Raft, and Pods: A Tour of Distributed Systems Within Kubernetes [B] - Bo Ingram, Craftsy
Kubernetes does some pretty neat things for you — autoscaling your app, rolling deploys, and more! In this talk, we’ll take a look at how Kubernetes leverages distributed systems to make its magic happen. We’ll do an overview of all components, but we’ll be concentrating on etcd, the controllers, and the scheduler. We’ll examine etcd and take a dive into the Raft algorithm to show how Kubernetes handles distributed state. We’ll take a look at some of the controllers to show how they reconcile the cluster’s state. We’ll also be shining a spotlight on the scheduler and show how we go from unscheduled to happy and running. Lastly, we’ll take the things we’ve learned and show how they work together to deploy an app by tracing an actual deployment through a cluster.
About Bo Ingram
Bo Ingram is a Java platform engineer at Craftsy who spends his time flipping back and forth between backend feature development and infrastructure work. He has a problem where he buys more books than he can ever hope to read.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hey
everybody,
my
phone
says
it's
to
my
computer.
Just
said
it
was,
and
the
clock
in
the
hall
assessed
is
so
we've
achieved
distributive
consensus.
I
think
it's
time
to
start
this.
My
name
is
beau
Ingram
I'm
here
today
to
talk
to
you
on
a
talk
entitled
Plains
raft
and
pods.
It's
gonna
be
a
tour
of
distributed
systems
within
kubernetes.
So
a
little
bit
about
me,
like
any
millennial
I've
got
a
list.
My
Twitter
handle
first
I
am
voluptuous
on
Twitter
beau
Ingram
was
taken,
so
I
had
to
get
fancy.
A
That's
my
dog
Ernie
on
the
right.
He
is
universally
recognized,
as
quote
the
man
and
I,
consider
taking
out
all
my
slides
and
just
showing
pictures
of
him,
but
I
figured
you
all
would
leave
and
they
throw
me
out
and
it
probably
wouldn't
be
good.
So
I
am
a
platform
engineer
at
Kraft,
see
we're
based
in
Denver
Colorado
and
we're
an
online
education
and
e-commerce
company
for
makers
and
crafters.
We
sell
classes
in
topics
like
quilting,
sewing,
cooking
photography
and
we've
got
supplies.
A
We've
got
patterns,
we've
got
kids
to
help,
you
make
projects
and
all
sorts
of
fun
stuff.
So
I
spend
most
of
my
time
working
on
backend
features
there,
but
also
do
some
of
the
operations
he
tasks,
sometimes
as
well
as
some
of
your
traditional
site.
Reliability
is
tasks
I'm,
probably
somebody's
definition
of
DevOps,
not
mine,
but
somebody
is
so
you
know
I'm
big
on
definitions.
So,
let's
start
off
with
what's
a
distributed
system,
so
in
distributed
systems
for
fun
and
profit
maketo
Takada
says
distributed.
A
Programming
is
the
art
of
solving
the
same
problem
that
you
can
solve
on
a
single
computer
using
multiple
computers.
Now
that's
a
little
bit
of
a
sarcastic
kind
of
tongue-in-cheek
definition,
but
a
distributed
systems
problem
is
a
problem
that
you
have
on
one
computer.
It
needs
to
go
to
more
computers,
usually
because
it's
gotten
too
big.
We
can
say
that
a
distributed
system
is
several
nodes
components
pieces,
whatever
working
together
to
accomplish
the
same
goal.
A
So
it's
cube
con,
but
what's
kubernetes,
you
know.
We
heard
a
great
talk
this
morning
about
like
what
are
all
the
things
kubernetes
can
be
well
for
me,
I
like
to
go.
Look
at
the
documentation.
The
docs
say
that
it's
an
open
source
platform
for
automating
deployment,
scaling
and
operations
of
application
containers
around
clusters
of
hosts,
providing
container
centric
and
structure.
A
What
I
read
that
four
times?
And
you
know
every
time,
I
practice
this
talk
I
had
to
run
through
that
slide
twice
just
to
get
the
cadence
of
it
right.
It's
a
mouthful
but
I
like
to
think
kind
of
back
to
what
the
number
one
definition
was
this
morning,
which
is
that
kubernetes
is
a
flexible
platform
for
running
containerized
apps,
not
word-for-word
what
it
was
but
close
enough,
but
kubernetes
you
know,
provides
capabilities
greater
than
hi
I'm
a
container.
Please
run
me:
it's
got.
A
This
really
awesome
feature
set
right,
you
know,
can
handle
rolling,
deploys,
do
load
balancing
you
got
health
checking,
all
sorts
of
stuff
and
they're,
great
and
I.
Think
a
lot
of
talks
talk
about
how
kubernetes
does
these
things,
but
I
want
to
do
something.
A
little
different
tonight.
I
want
to
peek
behind
the
curtain.
I
want
to
see
how
the
components
of
kubernetes
work
together
to
achieve
these
things.
My
question
for
us
tonight
is
today
is
how
does
kubernetes
leverage
distributed
systems
so
start
off
with
kubernetes
has
containerized
apps
well?
A
What's
a
container
container
could
be
said
as
the
usage
of
Linux
see
groups
and
namespaces
to
provide
process,
isolation
for
an
abstracted,
filesystem,
see
groups
specify
limits
for
how
much
memory
and
CPU
a
process
can
use
and
a
namespace
can
stop
process
from
interfering
with
other
processes.
But,
like
all
definitions,
that's
great
and
somebody
has
set
it
better.
Undoubtedly,
a
beach
in
yesterday
she
said
a
container
is
just
a
baby
computer
inside
another
computer
way
simpler
than
anything.
I
ever
said.
A
A
containerized
app,
though
something
we
met,
running
kubernetes
could
be
something
like
an
engine,
X
server
or
a
little
micro
service
or,
if
you're
feeling
stateful,
perhaps
a
Redis
instance.
So
when
you
think
of
kubernetes
right,
you
come
to
kubernetes
for
the
first
time
you
hear
containers
containers,
you
know
I'm
going
to
join
the
container
team,
but
in
kubernetes
the
base
unit
of
scheduling
and
the
way
kubernetes
thinks
about
things
is
with
pods.
A
So
iPod
is
one
or
more
containers
that
share
a
unique
IP
address
in
the
cluster
you're
guaranteed
to
have
all
containers
running
on
the
same
host.
A
pod
can
share
disk
volumes.
This
is
pretty
helpful
in
many
situations.
For
example,
if
you
get
an
app
that
logs
to
a
file,
you
could
have
a
sidecar
container
sitting
alongside
it
in
the
same
pod
that
handles
fording
your
logs
off
to
a
log
aggregation
service
like
log
stash
or
spunk.
A
So
the
way
we
typically
manage
these
pods
is
through
it.
A
deployment
is
a
declarative
way
of
managing
pods.
We
can
specify
what
containers
we
want
to
run
and
how
many
instances
of
those
pods
we
want
to
have
in
our
cluster
under-the-hood
deployments.
Use
a
thing
called
replica
sets
to
ensure
that
the
requested
number
of
pods
is
equal
to
the
number
of
pods
running
in
the
cluster
for
that
deployment.
A
So
when
we
create
a
deployment
which
contains
the
container
images,
we
want
to
run
for
our
pods
and
how
many
pods
we
want
to
have
kubernetes
is
going
to
handle
creating
a
replica
set.
This
replica
set
one
sure
we
have
a
desired
number
of
pods.
It's
good
replica
set
will
create
our
pods
and
then
eventually,
I
created
pods
will
be
scheduled
by
another
component
and
then
run
by
yet
another
component.
A
So
going
through
this
a
little
fast
right
now,
I'm
gonna,
start
kind
of
at
the
top
level,
we're
gonna
dive,
deeper
and
then
dive
deeper.
So
we
talked
kubernetes
is
made
of
distributed
components
when
I
first
got
into
kubernetes.
What
I
imagined
is
I
would
go
to
my
computer
right
and
I'd,
be
like
okay,
yum,
install
kubernetes
and
there'd
be
a
binary
and
then
I
try
to
run
it
and
it
wouldn't
be
on
my
path.
I'd
put
it
on
my
path.
I
would
point
it
at
other
nodes,
sit
back
and
relax.
A
My
monitor
would
glow
angels
would
sing
and
I'd
bask
in
the
glow
of
magical
container
orchestration,
except
that's
not
the
way
it
works.
Kubernetes
uses
distributed
systems
to
provide
the
ephra
mention
magical
container
orchestration,
it's
made
of
several
components,
and
each
of
them
is
worth
their
own
deep-dive.
Each
of
them
does
their
own
pretty
neat
stuff
and
each
plays
a
crucial
role
in
running
and
operating
applications.
A
So
let's
take
a
look
at
some
of
them
now
I'm
talking
more
detail
about
each
of
them
in
a
moment,
but
I
want
to
introduce
them
to
you
before
we
get
too
far
into
things.
So
the
four
components
on
the
slide
here
at
CD,
the
API
server,
the
controllers
and
the
scheduler-
make
up
the
control
plane.
These
run
on
the
master
nodes
of
the
cluster,
so
starting
off
at
CD
at
CD
is
a
distributed.
A
Key-Value
store
that
serves
as
the
clusters
database
now
having
access
to
EDD,
CD,
read
and
write
access
is
equivalent
to
having
root
access
in
your
kubernetes
cluster.
So
to
control
this
we
stick
in
front
of
it.
A
component
called
the
API
server
other
components
in
kubernetes.
Talk
to
ed
CD
through
this
API
server
via
good
old
rest
calls
next
up
the
controllers,
so
the
controllers
handle
routine
cluster
tasks.
They
operate
on
what
is
known
a
reconciliation
loop
pattern.
A
The
controllers
will
check
your
desired
state
check
your
current
state
diff
the
two
and
then
modify
your
current
state
so
that
it
matches
the
desired
state.
There's
a
variety
of
different
controllers
and
we're
gonna
explore
a
couple
later
on.
So
last
up
is
the
scheduler,
as
you
might
surmise
from
something
that
has
things
scheduled
on
it.
There's
a
scheduler.
The
scheduler
is
in
charge
of
scheduling,
unscheduled
pods
on
the
nodes.
Couplet
is
the
star
of
the
show,
couplet
watches
for
notes
that
have
been
assigned
to
its
pod
and
then
runs
them.
A
Couplet
is
constantly
checking
Etsy
D,
by
extension,
the
API
server
and
as
well
as
its
local
config
for
pods
to
run
it's
constantly
going
hey,
let's
see
D
in
the
API,
so
every
have
any
pods
me
to
run
for
me
any
pods
and
he
pods
great
any
pods
cube
proxy
on
the
nodes
handles
any
networking
things
needed
it's
in
charge
of
maintaining
port
forwarding
and
managing
networking
rules
and
since
we're
running
containers,
we
need
some
sort
of
container
runtime,
whether
it's
docker
rocket
cryo
what-have-you.
So
these
are
the
components
of
kubernetes
Etsy
D.
A
A
So
whenever
you're
talking
about
the
distributed
components
of
kubernetes,
you've
got
to
start
with
Etsy
D
Etsy
D
explains
so
much
about
kubernetes
behavior.
So,
as
I
mentioned
earlier,
Etsy
D
is
a
distributed.
Key-Value
store
that
serves
as
the
database
for
our
cluster
if
you're
familiar
with
console
or
zookeeper,
it's
pretty
similar
to
those
as
well
so
distributed,
implies
that
there's
multiple
nodes
running
all
working
together
to
store
data.
So
in
order
to
operate
majority
of
the
nodes
and
our
Etsy
D
cluster
need
to
be
up
running
and
healthy.
A
A
Alright,
the
cap
theorem.
It
says
it's
impossible
for
a
web
service
to
provide
all
three
of
consistency,
availability
and
partition
tolerance,
so
we're
gonna
define
consistency
always
reading
the
most
recently
written
value
for
a
given
key
will
define
availability
as
every
request
to
a
non
failing.
Node
receives
a
response.
The
partition
tolerance
will
be
defined
as
a
system.
Our
system
can
handle
the
dropping
of
messages
between
two
nodes.
So
in
the
schema
provided
by
the
cap,
theorem
EDD
is
a
consistent
partition,
tolerant
system.
So
imagine
a
5
node
cluster.
A
If
three
of
our
nodes
take
a
dive,
the
other
two
are
going
to
stop
responding
to
requests
until
the
quorum
has
been
restored
and
we
have
more
than
the
minimum
number
of
healthy
nodes.
So
why
does
kubernetes
use
EDD?
Well
if
your
goal
is
to
have
a
distributed
system
for
running
containerized
applications,
you
need
something
that
plays
nice
with
clustered
systems.
So
Etsy
DS
Doc's
say
that
once
again,
big
quote:
Etsy
D
is
designed
for
large-scale
distributed
systems
that
never
tolerate
split-brain
behavior
and
are
willing
to
sacrifice
availability
to
achieve
it.
A
Split
brain
behavior
is
when
you
have
multiple
masters.
Each
one
thinks
they're
in
charge
and
they're
each
setting
their
own
state,
so
you
have
the
you:
don't
have
distributed
consensus
and
you're
disagreeing
on
what
the
state
is
not
great
for
kubernetes.
So
you
can
see
the
cap
theorem
references
in
this
quote,
though
NC
d
is
going
to
sacrifice,
availability
to
preserve
consistency
now,
I
think
there's
an
important
clarification
to
make
here,
looking
further
ahead
right.
A
So
if
EDD
CD
is
running
as
the
kubernetes
data
store
and
drops
below
the
minimum
number
of
healthy
nodes
and
is
forced
to
sacrifice
availability,
that
doesn't
mean
that
all
the
applications
running
on
your
cluster,
your
micro
services
or
nginx
service
servers,
your
rightist
instances,
they're
not
all
suddenly
unavailable.
What
it
means
is
that
kubernetes
the
platform
is
unavailable.
A
Your
ship
is
dead
in
the
water
and
it
you
know
not
a
great
state
you're
in,
but
your
ships
not
sinking
you're
still
taking
traffic,
but
you
can't
schedule
new
pods,
add
new
deployments
or
do
anything
that's
going
to
require
a
consensus,
a
net
CD,
your
existing
applications
will
still
be
running
and
taking
traffic,
so
Etsy
D
does
the
typical
crud
operations
and
a
couple
other
things
on
top
of
that
via
rest
calls
or
command
line
tool.
But
this
simple
interface
hides
the
answers
to
some
complex
problems.
A
How
do
the
nodes
agree
on
what
the
value
is
for
a
given
key
we've
got
like
three
nodes
out
there.
I
alluded
as
a
distributed
consensus
earlier
saying
that
my
you
know
all
the
clocks
said
it
was
two
o'clock,
but
how
does
that
happen?
If
not
done
smartly,
things
can
go
awry
pretty
quickly.
So
how
does
Etsy
D
agree
on
the
value
for
a
given
key
while
still
upholding
its
consistency,
guarantee
it's
the
raft
algorithm?
That's
a
really
great
logo
by
the
way
I
like
that.
A
So
the
raft
algorithm
is
a
method
of
achieving
distributed
consensus
having
multiple
distributed
servers,
agree
and
what
the
value
is
for
a
given
key
in
a
fault-tolerant
manner.
Now,
there's
a
really
trivial
way
to
solve
this.
You
can
just
say
you
know
what
the
answer
is.
Always
three
everyone
agrees
on
it.
You
know
it
doesn't
matter.
The
time
doesn't
matter.
The
date
doesn't
matter
where
you
are
most
formal
papers,
though,
for
distributed
systems
say
you
can't
do
that.
A
I
really
think
that
it
started
with
like
some
exasperated
person
somewhere
arguing
with
somebody,
but
let's
take
a
look
at
a
not
wrapped
system
to
illustrate
the
difficulties
in
distributed
consensus.
This
is
not
raft.
It's
not
EDD
CD.
It's
been
specifically
designed
to
show
how
distributed
consensus
can
be
difficult,
don't
go
away
and
implement
this
and
run
it
in
production,
bad
idea.
So
in
our
example
system
we're
gonna
have
three
nodes:
a
B
and
C.
Each
one
holds
a
single
value.
Our
nodes
will
receive
requests
from
a
client.
A
It
will
write
to
disk
and
then
broadcast
the
new
value
out
to
all
other
nodes.
So
let's
examine
some,
not
great
scenarios.
What
happens
if
there's
multiple
updates
to
the
value
at
the
same
time?
So
here
we
go,
we've
got
our
three
nodes:
a
B
and
C.
Everyone
agrees.
The
value
is
X
everyone's
in
agreement.
Things
are
happy,
things
are
great
and
then
oh,
no,
two
clients
come
in
client,
one
tells
node
a
hey.
The
new
value
is
y
client,
tutos,
node,
B,
hey
the
new
value
is
going
to
be
Z.
A
So
a
and
B
each
write,
the
new
value
to
disk,
writes
Y
to
disc
B,
write
Z
to
disk.
Meanwhile,
you
know
she
doesn't
know
anything
yet
season,
4
trouble
so
a
broadcast.
The
new
value
Y
and
B
broadcast
the
new
value
Z.
So
we're
going
to
assume
that
a
is
messages
arrive
before
bees,
just
via
some
arbitrary
ordering.
A
You
can
see
here.
We've
got
problems
a
and
C
think
the
value
is
Z,
whereas
B
thinks
the
value
is
y.
We
have
not
achieved,
distribute
consensus,
not
great.
So
let's
take
a
look
at
another
scenario.
In
this
new
scenario,
same
baseline,
a
B
and
C
are
all
in
agreement
that
the
value
is
X.
Things
are
going
great,
then
Oh
No,
the
cluster,
undergoes
a
network
partition.
Si
is
unable
to
talk
to
a
or
b
and
a
or
b
can't
talk
to
see
all
the
messages
to
and
from
C
are
being
dropped.
A
Meanwhile,
a
client
comes
along,
not
aware
that
the
cluster
maybe
isn't
in
great
shape
and
says,
hey,
guess
what
the
new
value
is
y.
So
a
broadcast
this
out
and
says:
hey
everybody.
The
new
value
is
y
B
gets
it
successfully.
It's
like
okay
gotcha.
Meanwhile,
the
message
is
to
C
are
being
dropped,
so
C
doesn't
find
out.
Meanwhile,
time
goes
by
and
C
recovers.
A
Let
C
hasn't
gotten
the
messages,
because
the
messages
were
lost
forever
and
so
a
and
B
think
the
value
is
y,
whereas
C
thinks
the
value
is
X
sad,
so
seemingly
simple
systems
can
fail
in
all
sorts
of
fun
ways
when
exposed
to
concurrent
operations
in
order
for
consensus
to
be
achieved,
we're
going
to
require
greater
coordination
between
our
nodes.
We
can't
just
fire
and
forget,
like
our
terrible
example
system.
This
is
where
Raph
comes
into
play.
So
raft
is
a
consensus
algorithm
for
managing
a
replicated
blog.
A
A
replicated
log
is
a
series
of
commands
executed
in
order
by
a
state
machine.
So
we
want
each
log
to
have
the
same
commands
so
that
the
end
of
the
day,
our
log
has
the
same
commands
in
the
same
order,
and
each
machine
will
have
the
same
state.
So
to
do
this
raft,
a
LexA
leader,
the
leader,
is
put
in
charge
of
managing
the
log
accepts
entries
from
clients,
replicates
them
to
the
other
nodes
and
then
tells
the
nodes
when
it's
safe
to
actually
commit
them.
A
If
a
leader
fails,
a
new
one
is
elected
in
its
place,
so
my
mom
always
said
to
me:
Bo,
you
can
be
a
leader
or
you
can
be
a
follower
generally.
I
was
in
trouble
that
often
included
the
statement.
If
all
your
friends
jumped
off
a
bridge,
would
you
as
well,
but
in
raft
you
have
other
options,
you
don't
have
to
be
just
a
leader
or
a
follower.
You
can
be
a
candidate
as
well,
so
nodes
can
be
in
three
states.
Followers
just
hang
out
and
kind
of
respond.
A
Two
requests
from
leaders:
leaders
like
hey,
replicate
this
followers
like
alright
gotcha,
so
leaders
handle
all
the
rights
and
handle
all
the
fun
coordination
between
the
followers
and
our
third
state
candidate
candidate.
Is
the
state
used
to
elect
a
new
leader
when
a
leader
goes
away
or
disappears
or
whatever
candidate
is
the
state
used
to
elect
a
new
leader?
A
node
will
declare
itself
as
a
candidate
and
say
hey
vote
for
me.
So
we
talk
about
leaders
well
part
of
dealing
with
order.
A
Time
stamps
is
dealing
with
time
raft
uses
an
incrementing,
integer
timestamp
called
a
term
which
is
tied
to
an
election
terms.
Last
until
there's
a
new
leader
and
there's
one
leader
per
term
terms
serve
as
a
logical
clock
and
aid
in
detecting
obsolete
info,
like
stale
leaders.
If
the
leader
finds
out
that
there's
a
node
with
a
high
another
leader
out
there
with
a
higher
term
that
leader
is
the
leader
of
the
lower
term,
will
stand
down:
okay,
not
in
charge
anymore.
A
So
the
leader
sends
heartbeat
messages
out
to
the
follower
nodes,
saying
hey
still
alive.
It
happens
every
fairly
frequently,
okay,
still
alive,
hey
I'm
still
alive.
So
what
happens
if
a
follower
doesn't
get
a
heartbeat?
Well,
it
immediately
assumes
the
leaders
dead
and
that's
a
dark
but
effective
assumption
our
follow
rule
increment
its
term
and
declare
itself
as
a
candidate.
That
means
it's
election
time.
Our
new
candidate
is
going
to
send
an
RPC
request
out
for
votes
to
all
of
the
other
servers.
A
So
we
heard
yesterday
morning
about
how
HBO
uses
kubernetes
with
Game
of
Thrones.
Well,
the
singing
game
of
Thrones
was
you
know
you
win
or
you
die
in
the
game
of
ref
leadership
elections.
The
stakes
are
lower.
You
win
or
you
lose
servers
vote
for
the
first
candidate
who
asks
for
their
votes,
assuming
the
candidate
is
at
least
as
up-to-date
as
the
voter.
This
means,
if
the
voter
has
entries
that
the
candidate,
doesn't
the
voter
isn't
going
to
vote
for
that
candidate.
Now,
if
a
candidate
gets
a
majority
of
the
votes,
it
wins.
A
No
one
gets
a
majority
there's
another
round
of
elections
and
Raph
uses
randomize
timeouts
to
make
sure
that
eventually
somebody
will
win,
assume
your
clusters
in
an
okay
state.
So
the
after
the
leader
gets
a
majority
of
its
votes.
It
sends
out
a
new
set
of
heartbeat
messages
to
notify
others
of
its
victory
and
life
in
the
cluster
goes
on
suanne
draft
how
to
rights,
work!
Well,
rights
I'll,
go
to
the
leader.
The
leader
gets
every
right.
A
The
leader,
then
appends
the
command
to
its
log
locally
then
tells
other
servers
via
RPC
to
append
it
to
their
logs
as
well.
Now
it's
the
followers
behind
it's
gonna,
say
nope
I'm
behind
can't
handle
this
right
now
and
it's
like
okay.
So
if
way
too
many
nodes
in
your
cluster
behind
spoiler
alert,
nothing's
gonna
happen.
So
once
the
majority
have
appended
things
to
their
nodes
locally,
so
the
leader
hasn't
appended
and
a
majority
of
the
followers
have
it
appended.
The
leader
commits
it
to
its
own
log
locally.
A
Now,
in
subsequent
messages,
two
nodes
leader
is
gonna.
The
leader
is
going
to
inform
the
other
nodes.
The
followers
of
the
index
of
the
last
committed
entry,
so
nodes
are
gonna
notice.
This
me,
like
okay,
I,
you've,
sent
me
everything
up
to
twelve
and
you
say
twelve
was
committed
and
I
haven't
committed.
Nine
ten,
eleven
and
twelve
well
I'm
gonna
commit
nine
ten
and
eleven
and
twelve.
So
it's
going
to
commit
everything
up
to
the
last
committed
index.
A
Now
this
solves
problems
in
our
bad
system
from
before
our
simultaneous
right
scenario
is
solved
by
the
requirement
that
all
rights
go
to
the
leader.
The
leader
is
going
to
receive
each
right
and
each
right
is
a
new
entry
with
a
new
index.
So
if
two
nodes
manage
to
somehow
try
to
update
a
value
simultaneously,
each
one
is
going
to
be
a
new
right
with
a
new
index.
One
will
occur
before
the
other.
A
So
we
talked
about
the
cap
theorem
a
little
while
ago,
in
how
raft
and
sed
is
a
consistent
partition
tolerance
system.
Well,
this
is
achieved
through
requiring
a
majority
of
the
nodes
to
act.
Elections
require
a
majority
of
the
nodes
to
agree
to
elect
a
leader.
All
rights
require
a
majority
of
the
nodes
to
replicate
the
transaction.
In
order
for
the
transaction
to
be
successful,
we
lose
more
than
half
the
nodes
they
weren't
able
to
do
anything.
So
that
was
a
brief
overview.
Have
raft.
A
I
do
semi
deeply
into
it,
but
I
didn't
cover
all
the
safety
guarantees
and
some
of
the
more
formal
things
I
think
the
raft
paper.
The
formal
paper
is
actually
really
great.
Read
I'd
highly
recommend
checking
it
out
also
for
the
visual
minded
of
you
and
really
for
everybody,
because
it's
this
is
a
really
great
visualization.
There's
a
visualization
out
there
called
the
secret
lives
of
data,
which
shows
the
way
raft
clusters
interact
in
various
scenarios.
It's
definitely
worth
checking
out.
A
A
The
replica
set
controller
is
in
charge
of
making
sure
that
for
any
given
pod,
spec
or
any
kind
of
pod
image
thing,
there's
a
given
number
of
pods
running
at
any
time
now,
kubernetes
actually
suggest
you
don't
interact
with
replica
sets
directly
so
we'll
instead
use
the
deployment
in
the
deployment
controller
to
manage
them.
Now
the
replica
set
controller
knows
which
pods
to
manage
through
labels,
so
we'll
define
labels
on
the
pods
we
create
and
tell
the
replicas
set
which
which
labels
it
should
look
for.
So
that
knows
which
pods
to
manage.
A
Well,
what's
a
label
labels
are
just
key
value
pairs
and
the
replica
set
has
a
selector
another
sort
of
key
value
pair,
which
tells
it
which
labels
to
select
and
manage
now
the
deployment
controller.
The
deployment
controller
manages
the
whole
deployment
process
of
your
app,
so
the
deployment
controller
provides
a
declarative
way
of
managing
pods
and
then
uses
that
to
roll
out
your
desired
changes
in
a
friendly
way.
So
deployments
will
handle
like
rolling
deploys.
You
can
roll
back
your
application
if
it's
not
going
well.
A
So
we
talked
about
the
reconciliation
leap
a
little
while
ago
how
the
deployment
process
is
managed.
Reconciliation
loop
is
going
to
cut.
You
know,
check
the
current
state
check
the
desired
state,
diff
the
two
and
then
modify
the
current
state
so
that
it
matches
the
desired
state.
So
if
we're
rolling
out
say
a
new
version
of
the
deployment,
the
deployment
controller
is
reconciliation,
loop
will
change
the
desired
state,
ie
the
counts
of
the
old
and
new
replicas
sets.
A
So
the
scheduler,
the
scheduler
watches
for
unscheduled
NIT
pods
watches
for
unscheduled
pods
and
assigns
them
to
a
given
node
what
it's
looking
for,
specifically
our
pods
without
a
node
name,
constantly
squaring
the
API
server
and,
by
extension,
EDD.
Looking
for
these
unscheduled
pods.
One
thing
to
note
is
that
the
scheduler
doesn't
isn't
actually
in
charge
of
running
pods.
It's
only
in
charge
of
assigning
pods
to
a
given
node.
So
he
does
this
by
a
scheduling
algorithm.
First
off
it
filters
out
nodes
that
aren't
desired
or
just
aren't
a
great
fit.
A
It
then
ranks
the
remaining
nodes
and
picks
the
top-ranked
node
after
ranking
is
complete
so
step,
one
we're
filtering
its
predicates,
we're
filtering
out
nodes
that
aren't
desired
or
not
a
great
fit.
I
think
the
simplest
scheduling
strategy
is
to
try
to
assign
a
pod
directly
to
a
node.
So
you
can
do
this
via
a
node
name
specified
in
the
pod
spec.
The
host
name.
Predicate
is
in
charge
of
checking
this.
It's
going
to
compare
the
nodes
name
to
the
node
name
specified
in
the
pods
back.
A
If
any,
so,
it's
going
to
exclude
every
node
that
doesn't
match,
and
it
will
let
us
schedule
directly
to
a
node
assuming
it
passes.
The
other
predacons
there's
some
resource
aware
predicate
Slater
that
we'll
check
that
will
I
could
prevent
it
from
scheduling
so
match.
Node
selector
is
kind
of
a
variance
of
hostname
kubernetes
lets
you
provide
label
selectors
to
associate
resources
using
labels.
This
predicate
checks
whether
the
node
selector
supplied
in
the
pod
spec,
is
valid
for
a
given
node.
For
example.
Let's
imagine
that
you
only
want
to
schedule
on
high
bandwidth
nodes.
A
You
want
the
network
power.
You
could
specify
a
high
bandwidth
label,
put
it
on
your
node
and
then
add
a
selector
to
your
pod,
spec
to
only
scheduled
on
nodes
with
this
label.
So
there's
also
the
concept
of
affinity,
which
is
a
more
expressive
and
flexible
version
of
the
node
selector
concept.
You're,
not
just
limited
to
an
exact
match.
Can
you
set
matching
and
all
sorts
of
fun
things?
Also,
you
can
match
not
only
on
nodes
but
pods
as
well
using
affinity
accordingly.
A
There's
a
predicate
that
checks
whether
or
not
the
pod
plays
nice
with
the
affinity
settings
of
the
pods
on
that
node.
The
inner
pod
affinity
matches
predicate,
so
taints
are
kind
of
the
inverse
of
affinity.
We
can
specify
a
taint
that
tells
pods
to
avoid
certain
nodes.
Pods
won't
schedule
two
nodes
with
taints
unless
the
pod
explicitly
tolerates
them.
So
if
you
didn't
want
to
schedule
to
a
node,
for
example,
you
could
add
a
no
schedule
taint
and
nothing
will
schedule
to
it
unless
they
specifically
add
a
no
schedule.
A
Toleration
the
pod
tolerates
no
taints
predicate
checks,
whether
or
not
the
pod
has
opted
in
to
tolerating
a
given
nodes
taints.
So
the
scheduler,
like
I,
mentioned,
also
checks
the
resources
of
each
node.
The
pod
fits
host
port
predicate
checks,
whether
or
not
the
host
port,
a
hard-coded
port
specified
in
the
pod
spec
is
available
for
that
node.
If
it's
not
available
we're
going
to
filter
out
that
host
from
being
scheduled
to
for
this
given
pod,
so
pod
fits
resources.
Is
another
resource
aware
check.
Pod
suits
request
a
given
amount
of
CPU
in
memory.
A
This
predicate
checks
whether
the
node
is
capable
of
satisfying
that
request,
so
check,
node
memory
pressure
and
check
node
disk
pressure,
or
at
some
other
predicates.
They
won't
schedule
on
to
nodes
whose
memory
usage
or
disk
usage
is
too
high
check.
Node
condition
is
a
more
extreme
version
of
this
check.
Node
condition
prevents
pods
from
being
scheduled
to
nodes
who
are
unavailable
or
network
unavailable
who
aren't
ready
according
to
kubernetes
or
just
plain
out
of
disk
space.
A
Those
are
conditions
that
would
you
know,
make
a
nodes
suboptimal
for
scheduling
too
I
didn't
list
them
here,
but
there's
also
some
volume
checks.
So
if
you're
on
a
cloud
provider,
there's
often
no
limit
to
how
many
volumes
you
can
have
attached
and
make
sure
that
we're
not
using
more
than
the
allowed
number
of
volumes,
or
maybe
that
there's
no
conflicts
on
volume
claims.
So
after
we
filtered
out
our
nodes,
we
have
a
set
of
nodes
that
are
valid
ish
for
scheduling
some
are
probably
better
than
others.
A
Now
you
could
also
be
left
with
no
nodes,
in
which
case
your
pot
is
unschedulable.
In
that
scenario,
you're
going
to
need
to
go
back
and
modify
the
pods
back,
or
maybe
wait
for
some
other
nodes
to
come
up
or
take
some
sort
of
action,
possibly
in
order
to
make
your
pod
schedule
pool
again,
but
if
you've
got
a
set
of
nodes
that
you
can
possibly
schedule,
it's
time
to
rank
ranking
applies
a
series
of
priority
functions
that
return
a
score.
A
higher
score
is
more
desirable,
so
functions
are
run
against
each
node
they're.
A
All
added
up
and
the
node
with
the
highest
score
is
the
winner.
Ties
are
broken
randomly
by
picking
a
winner,
so
here's
some
of
the
ranking
functions,
and
so,
when
you
think
of
scheduling
right,
you
want
to
put
your
paadam,
maybe
the
least
used
node.
So
the
least
requested
priority
helps
out
with
this.
It
calculates
how
much
CPU
and
memory
equally
weighted
and
added
together
would
be
left
after
scheduling
our
pod
to
that
node.
So
this
kind
of
helps
with
balancing
resource
usage
across
the
cluster.
A
Now
balanced
resource
allocation
is
a
more
focused
version
of
this.
It's
attempting
to
prevent
nodes
from
being
largely
weighted
towards
CPU
usage
or
memory
usage.
It's
specifically
trying
to
avoid
news
with
like
95%
CPU,
but
only
5%
memory,
so
it
checks
to
see
how
the
pod
would
affect
the
balance
of
resource
usage
on
that
given
node,
this
function
is
going
to
favor
nodes,
who
would
have
CPU
utilization
closer
to
memory
utilization
after
scheduling,
so
next
up
is
selector
spread
priority.
We
lose
a
great
chunk
of
the
benefits
of
having
multiple
copies
of
an
application.
A
They're
all
sitting
out
there
on
the
same
node
that
node
goes
down.
You'll
lose
every
copy
of
your
app
that's
running
on
the
node.
There's
a
metaphor
about
eggs
and
baskets
I
think
that's
really
helpful
here.
This
function,
minimizes
the
amount
of
pods
from
the
same
service
or
replicas,
set
on
the
same
node,
causing
our
currently
being
scheduled,
pods
to
favor
nodes
without
it's
managed
siblings.
A
So
the
image
locality
priority
function,
while
containers
have
images
and
oftentimes
we
have
to
download
them.
This
function
prioritizes
nodes
that
have
already
downloaded
more
of
the
pods
images.
If
you've
got
them
all
you're
gonna
get
the
highest
score.
If
you've
got
none
of
them,
you're
gonna
get
the
lowest
score.
It's
a
node
affinity.
Priority
and
tank
Toleration
priority
notice
any
priority
favors
nodes
that
have
that
match
the
pods
nodes
affinity,
preferences,
taint
toleration,
priority
prioritizes
nodes
that
have
taints
that
are
specifically
tolerated
by
the
pod.
A
If
you've
gone
out
of
your
way
to
request
some
affinity
or
explicitly
tolerate
something,
kubernetes
is
gonna
notice.
That
and
say
you
know
what
you're
gonna
get
a
little
boost
to
schedule
into
this
node.
So
what
happens
when
we
submit
a
deployment
to
kubernetes?
We've
talked
about
all
these
components,
but
let's
see
how
they
work
together.
So
I've
got
this
deployment
here.
I
toss
together,
a
really
quick
hello
world
go
app.
A
The
it's
gonna
be
listening
on
port
8080,
I've
labeled
it
with
the
TAT
the
tag
app
hello,
world
and
I
say:
I
want
three
of
you
and
I've
also
specified
a
service.
A
service
will
select
pods
and
provide
access
to
them.
I've
see
the
spec
selector
app
field
I've,
given
it
the
tag,
app
hello,
cube,
Khan,
and
so
the
service
is
going
to
select
that
pods.
With
that
tag,
we've
also
said:
hey
I,
want
you
to
Ford
port
80
over
to
port
8080
and
I.
A
Want
you
create
a
load
balancer
for
me,
so
how
are
we
gonna
actually
submit
this
good
old
cube,
cuddle
I'm
on
Team
Q
cuddle
by
the
way
for
pronunciation?
There's
differing
opinions,
I
believe
cube
cuddles
the
best
one,
so
we
use
Q
cuttle
to
get
our
app
running
in
kubernetes.
It's
how
we
interact
with
the
cluster
in
view
resources
as
well
as
modify
them,
gives
us
a
great
view
of.
What's
going
on.
You
can
check
the
status
of
our
pods
our
deployments,
but
really
importantly,
for
this
section
use
it
to
construct
a
timeline
of
events.
A
So
what
do
we
expect
to
happen
here?
Well,
we're
gonna
create
a
planet,
a
deployment.
You
know,
cube,
cube,
cuddle,
create
a
chef
deployment.
You
animal,
our
deployment
is
gonna,
create
a
replica
set.
A
replica
set
will
create
three
pods.
Our
scheduler
will
schedule
those
three
pods
and
then
Kubel
it,
which
is
listening.
Looking
for
pods
to
run
on
agent,
its
node
is
going
to
run
the
schedule
of
pods.
A
So
what
actually
happens
now
before
we
go
on
I
just
want
to
warn
you.
These
are
real
events.
They
have
been
formatted
to
fit.
My
screen
and
viewer
discretion
is
advised
so
starting
off
with
we
can
see
here.
The
involved
object
kind.
So
our
deployment
is
the
involved
object,
the
one
that
I
just
just
created,
assuming
I,
just
gone,
cue,
cuddle,
create
a
chef,
and
the
message
is
that
we've
scaled
up
the
replica
set:
okay,
well,
I,
guess
that
means
we
created
a
replica
set
as
well.
A
So
we
created
a
replica
set
for
our
deployment
and
set
it
at
three.
You
can
see
also
the
source
of
it.
The
deployment
controllers
done
this
its
noticed
via
API,
the
API
server
net
CD,
hey
a
new
deployments
been
created.
I
should
check
that
out
and
do
things
accordingly.
The
reconciliation
loop
here.
What's
the
current
state?
What's
the
desired
state?
Okay,
the
current
state
is
there's
not
a
replicas
set
for
this.
The
desired
state
is
there
should
be
a
replica
set
for
this.
Let's
create
it
and
then
scale
it
up.
Alright.
A
Now
we
can
see
the
source
component.
The
replica
set
controllers
kicked
in
it's
created
a
pod.
We
can
see
the
involved
object
at
the
top,
as
well
as
the
replica
set.
So
a
replica
set
has
created
a
pod
for
us
now.
We've
got
two
more
to
create
before
we
do
that
the
schedule
is
kicked
in.
So
we
can
see.
The
source
component
is
the
default
scheduler
and
the
involved
object
is
a
pod.
The
default
scheduler
is
doing
something
to
the
pod.
A
It
is
successfully
assigned
this
pod
nwc
7k,
looking
at
the
end
of
it
to
a
given
node
in
gke.
So
that's
pretty
great
and
then
hey.
Look.
The
replica
set
controllers
back.
It's
created
a
pod,
ooh
three
hfh!
You
see
the
involved
object.
Is
the
replica
set
its
created
a
pod
for
us?
Is
it
gonna
be
scheduled?
Next
it
is
yay,
so
the
scheduler
has
scheduled
our
second
pod
and
it's
assigned
it
to
33
jg.
What's
the
last
mo
signs
here,
33
jg!
A
So
that's
interesting,
we
talked
about
you
know
the
various
functions
in
ranking
things.
One
of
them
was
select
or
spread
priority.
It's
gonna
favor
spreading
things
out
to
other
nodes.
Well,
because
they're,
all
added
together
and
they're,
all
weighted
selector
spreader
already
actually
lost
out
here.
I
had
some
other
things
running
in
my
cluster
and
said,
even
though
we
desire
to
have
things
in
multiple
places,
this
node
is
such
a
great
fit
that
we
want
everything
to
go
here.
A
So
this
is
something
different
couplets
kicked
in
couplet
has
been
like:
okay,
I've
got
a
pot
of
gonna
deal
with
well.
First
I'm
gonna
pull
my
image,
so
it's
pulling
Boni
Ingraham,
slash
hello,
HTTP
latest
from
docker
hub,
and
it's
downloading
that
oh,
the
second
pod
is
being
having
its
images
download.
It's
downloading
that
single
image.
A
It's
downloading
the
third
image.
Why
is
it
downloading?
At
the
same
time?
That's
a
great
question:
I
assume
it's
because
I
probably
shouldn't
freestyle
on
this,
but
I
believe
it's
because
it
hasn't
completed
the
download
on
any
of
them
yet
or
it
could
be
my
image
settings
on
the
deployment.
So
we
successfully
pulled
the
image
yay
it's
downloaded.
A
We
can
see
that
we
downloaded
the
pod
like
the
involved.
Object
is
the
pod.
Couplet
is
the
source
component.
That's
handling
this.
We
could
also
see
there's
a
third
field
and
involved
object
as
well
that
we
haven't
seen
before
filled
path.
So
it's
saying
okay,
I'm
trying
to
find
the
containers
in
this
hello,
coop
con
deployment
and
that's
how
it
got
the
image
thing
that
it's
downloading.
So
after
it's
successfully
pulled
the
image
it
creates
a
container
and
then
it
starts
a
container.
So
cubelet
has
downloaded
a
pulled.
A
An
image
successfully
downloaded
it
created
a
container
and
then
started
a
container.
So
I
expect
to
see
this
cycle
two
more
times,
because
we
have
three
pods.
We
need
to
start
up
each
with
a
single
container,
so
we
successfully
pulled
the
image
we've
created
the
container
for
their
next
pod
and
we've
started
it
as
well.
The
third
pod
yep
we
successfully
pulled
the
image,
we've
created
the
container
and
we
started
the
container.
So
we
did
it
yay.
A
If
you
can
navigate
it
to
your
browser,
you
would
see
Hello
coupe
con
Kelsey
did
almost
the
exact
same
thing
yesterday
morning.
I
just
want
to
say:
I
am
I
github
out
there
or
my
docker
hub
container
out
there
three
days
ago,
but
either
way
great
minds.
Think
alike.
I
am
honored
to
have
the
same
thought.
So
what
have
we
done
today?
So
we've
looked
at
the
various
components
that
make
up
kubernetes
we've
shown
how
kubernetes
handles
distributed
state.