►
From YouTube: We're Marketers. If we can Learn Distributed Systems + K8s, so can you! - Betty Junod & Paul Burt
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
We're Marketers. If we can Learn Distributed Systems + K8s, so can you! - Betty Junod, VMware & Paul Burt, HashiCorp
Why is Kubernetes designed the way it is? Distributed systems are hard. That's the short and unsatisfying answer. The long answer is that like quantum mechanics, it's something that tends to make most of us uncomfortable. This talk is an introduction, and a way to build intuition about what makes a system like Kubernetes "correct." We'll do so by contrasting Kubernetes's design choices against other modern systems. We'll look at the implementation details of Docker Swarm, HashiCorp Nomad, K3s, and a "batteries included" Kubernetes distro like VMware Tanzu. In doing so, we'll discuss a number of distSys concepts. We'll learn about CAP theorem, Gossip protocols, High Availability (HA), and the RAFT consensus algorithm. Finally, we'll look at real world cases. Why do so many tools rely on Etcd's RAFT? What caused Target's 2019 cascading failure? Attendees will walk away with a better idea of the problems confronting distSys, and an intuition of what "correct" looks like.
A
A
Awesome
so
we
are
super
excited
to
talk
to
you
about
something,
that's
near
and
dear
to
our
hearts.
First
off
the
slides
and
the
reading
list,
we've
got
an
extensive
reading
list
at
the
end
of
this
can
be
found
at
this
link
and
we'll
make
these
slides
available
from
to
cncf,
so
you
all
can
download
it.
A
A
Everyone
else
does
it's:
there's
fomo
everyone's
using
it,
but
we
really
wanted
to
start
with
the
question
of
why.
Why
do
we
want
kubernetes?
We
really
are
thinking
why
it's
it's
that?
Why,
behind
why
we
want
distributed
systems
and
really
distributed
systems,
are
all
about
failure
so,
fundamentally
accepting
that
systems
will
fail.
Your
applications
will
fail.
Your
server
will
fail.
Your
network
will
fail.
Anything
can
fail
at
any
given
time,
and
it's
not
that
it
there's
it's
someone's
at
fault
for
that,
but
it's
a
specific
mindset
that
applies
to
your
design.
A
So
if
you
accept
that
failure
is
inevitable
in
any
layer
of
the
stack,
then
what
you
do
is
you
will
design
your
systems
to
fundamentally
handle
that
failure
and
handling.
That
means
two
things
not
only
building
your
system
so
that
it
is
more
resistant
to
failure
so
that
you
can
prevent
it,
but
also
knowing
that
planning.
A
A
So
failure
is
inevitable
and
we
know
this
because
we
used
to
live
in
a
world
where
we
deployed
thing
a
single
app
to
a
single
server
and
we're
like
this
is
great
now
it's
running,
but
we
all
knew
that
if
that
thing
went
down
the
whole
thing
failed
right.
You
can't
get
to
the
app
anymore.
So
what
do
we
did?
What
do
we
do?
We
said
we
need
more
servers.
A
So
great
we
went
from
one
to
three
to
five
or
whatever,
and
now
we
have
a
cluster
of
servers,
and
so
we
have
some
fault
tolerance
built
in
so
that
if
one
of
them
fails,
we
can
route
the
traffic
to
another
one.
We
can
write
some
rules
throughout
the
traffic
to
another
machine
and
this
is
not
a
new
problem.
It's
not
something
that
started!
A
You
know
just
recently
with
you
know
in
the
recent
years,
because
of
kubernetes,
it's
actually
a
problem
that
we've
been
trying
to
solve
for
a
very
long
time
started
way
back
in
1969
with
arpanet.
This
was
a
military
project,
military
and
academic
project.
Thinking
about
can
we
build
a
decentralized
system
so
that
if
a
headquarters
location
was
suffered,
a
catastrophic
failure
for
whatever
reason
can
the
systems
and
command
center
still
be
online
because
we've
distributed
that
information
out.
A
A
And
what
happens
is
when
you
start
to
distribute
systems
when
you
start
to
scale
out
even
just
in
the
cluster,
and
then
you
start
to
think
about
distributing
those
systems
across
different
geographic
locations.
You
start
asking
you
start
having
to
answer
a
whole
lot
of
new
questions.
Right
as
you
distribute
systems
you
need
to
understand.
How
will
these
systems
know
what
is
the
correct
plan?
What
was
the
idea
intent
for
that
system?
A
How
should
it
behave,
what
should
be
running
and
then
how
will
those
systems
talk
to
each
other
so
that
they
know
what's
going
on
you
know?
Is
everything
behaving
the
way
it
should
and
when
it
finds
out
that
something
isn't
what
is
it?
What
are
the
instructions?
What
are
its?
What
is
it
supposed
to
do
to
get
it
back
to
the
desired
state?
A
A
simple
example
here
is
for
those
of
you,
who've
ever
played
the
game
of
telephone.
You
know,
as
a
child,
you've
got
like
you
and
your
ten
friends
lined
up
in
a
row.
The
first
person
says
whispers
a
little
message
into
the
second
person's
ear,
and
you
know
by
the
time
it
gets.
The
other
end,
it
is
nothing
like
what
you
first
said
so
distributed
systems
in
many
ways
trying
to
solve
that
problem.
A
B
B
B
Suffice
to
say
most
modern
systems
and
what
we'll
really
be
looking
at
in
detail
with
raft
use
a
leader
and
follower
topology,
and
then
they
have
to
find
consensus
or
figure
out
availability
in
the
face
of
a
failure
based
on
that
topology.
They
also
have
mechanisms
baked
in
to
achieve
quorum
in
case.
Maybe
a
leader
fails
if
you
can
still
communicate
with
the
majority
of
the
cluster.
B
In
this
case,
two
out
of
three
nodes,
you
can
still
have
consensus
and
return
responses,
so
we'll
look
at
elections
and
how
all
that
stuff
works
in
a
bit
too.
So
first
up
is
availability.
Availability
means
we
always
return
a
response,
even
if
there's
a
failure-
and
in
this
case
that
means
that
the
data
might
be
slightly
out
of
date
or
it
might
be
out
of
sync-
that
server
that
got
disconnected
can
still
return
responses
if
it
needs
to
just
means.
B
Our
data
is
a
little
bit
stale
and
this
isn't
a
big
deal
for
things
like
likes
or
anything
along
those
lines.
So
it's
a
great
way
to
go.
If
you
need
an
ap
system,
think
about
that
for
social
media,
that
sort
of
stuff
on
the
flip
side,
if
we're
dealing
with
money
or
shopping
parts,
we
kind
of
have
to
have
the
correct
data.
So
what
we
really
want
is
a
cp
system
in
that
case
and
kubernetes
as
we'll
learn
is
based
on
raft,
which
generally
gives
you
a
cp
system.
B
Let's
talk
about
what
consistency
is
specifically
because
database
consistency
is
a
bit
different
than
distributed
systems,
consistency
so
with
database
consistency.
What
you're
really
looking
forward
to
is
all
the
rules
of
the
database
are
followed,
so
the
schema,
the
constraints
that
you've
placed
on
stuff.
All
of
that
gets
followed
when
a
transaction
gets
committed,
whereas
distributed
systems
consistency.
B
What
you're
looking
for
is
a
majority
of
the
nodes
in
your
cluster
are
in
agreement
when
it
comes
to
returning
a
response,
so
you're
checking
that
there's
agreement
on
the
data
that
you're
returning
to
make
sure
that
it's
correct
another
issue
is
availability.
High
availability
is
slightly
different
than
some
of
the
other
availability.
B
B
So
one
thing
to
note
about
cap
theorem
is
it's
sort
of
a
boundary
it
sort
of
makes
us
take
the
problem
seriously,
that
we
can't
just
do
everything
which
was
an
issue
when
we
first
started
designing
distributed
systems,
but
it
doesn't
necessarily
help
us
build
distributed
systems.
Thankfully
there
are
things
that
help
us
build
distributed
systems
so
yeah.
B
B
B
B
A
good
question
we
should
cover
that
so
paxos
is
a
formal
system
that
was
built
to
help
design
distributed
systems.
So
it's
been
proven
with
a
language
like
tla,
plus
that
it's
a
correct
system
and
the
issue
with
paxos.
Is
it's
really
hard
to
understand
so
in
getting
his
paper
reviewed,
diego
angara?
The
creator
of
raft,
you
know,
got
comments
about
how
just
ridiculously
hard
paxos
was.
So,
in
addition
to
correctness
and
performance,
one
of
the
goals
of
raft
is
to
be
a
subset
of
axos.
That
is
also
understandable.
B
So
we'll
cover
that
in
a
second
here.
One
thing
to
note
with
all
this
is
that,
as
our
size
of
our
cluster
grows,
when
we're
building
a
cluster,
that's
based
on
raft,
it
gets
a
little
bit
slower
because
the
networking
gets
more
complicated.
We
have
to
talk
to
each
of
the
nodes
and
try
and
get
a
majority
rule
on
things
before
we
deliver
a
response
from
the
leader
that
happens
for
reads:
rights,
election
results,
everything
is
sort
of
a
majority
rule.
B
And
as
an
example,
let's
talk
about
how
rights
happen,
kind
of
work
out?
How
reads
happen
as
a
result
of
this,
but
the
leader
will
get
a
write
request.
It
will
stage
the
write
in
the
replicated
log.
The
replicated
log
is
sort
of
the
shared
data
state,
that's
linearizable,
so
we
can
replicate
it
deterministically.
B
The
leader
will
then
send
a
message
out
saying:
hey
majority
of
the
systems,
I'd
like
you
to
stage
this
on
your
log
and
they'll,
give
the
leader
response.
Once
it
hears
back
from
a
majority
of
the
systems,
it
will
then
say:
okay,
great
everybody
go
ahead
and
stage
that
commit
or
commit
that,
and
then
that
is
committed
to
the
log,
and
that
is
the
state
of
truth.
B
At
this
point,
when
a
read
request
comes
in
after
this,
the
leader
just
has
to
check
with
the
majority
to
make
sure
that
it
is
indeed
agreeing
with
that
majority
and
as
part
of
this,
the
heartbeat
message
goes
out
to
each
of
the
followers
when
it's
taking
some
of
these
actions
like
an
append
entry
or
it'll,
just
go
out,
naturally
as
a
heartbeat
message
itself
and
the
heartbeat
message.
Basically
tells
the
system
or
the
followers
in
the
system
that
they
are
still
connected
to
the
leader
and
everything
is
good.
B
Other
followers
may
also
promote
themselves
candidacy
and
vote
for
themselves,
but
each
member
of
the
system
is
only
able
to
vote
once
so
eventually,
one
of
these
systems
will
be
validated
as
getting
a
majority
of
the
votes
and
be
elected
to
a
leader
and
each
time
this
occurs.
A
term
number
is
incremented
and
that
term
sort
of
helps
the
system
recover
from
a
failure
or
know
which
log
is
the
most
up-to-date.
In
case
the
cluster
is
restored
or
networking
sort
of
comes
back
online.
B
Sure
yeah
and
to
your
point
earlier
about
how
do
we
communicate
or
how
do
we
know
what's
correct
a
lot
of
it
is
just
talking
to
the
other
people
in
the
chain
and
making
sure
that
they
agree
with
you,
rather
than
just
passing
the
message
along
without
doing
that
sort
of
checking.
So
this
gets
very
complicated
relatively
quickly,
and
the
creator
raft
has
some
great
talks
out
there.
There
are
a
lot
of
other
talks
about
raft,
but
we
have
links
to
this
with
more
detail
on
the
actual
specification
that
check
out.
A
B
A
Failure
right
and
with-
and
this
is
important
for
bigger
and
bigger
systems
right,
because
you
can't
manually
touch
everything
you
know-
you'd
probably
be
getting
as
an
operator
like
a
gazillion
or
pings
a
day,
and
so
a
lot
of
this
is
around
having
the
system
do
that,
checking
autonomously
and
adjusting
for
these
things
at
a
time,
but
sometimes
with
all
that
kind
of
automation
built
in
it
can
be
problematic.
If
something
is
off,
so
can
you?
I
know.
B
Let's
take
a
look
so
target,
thankfully
shared
an
example.
Thank
you
to
all
of
the
companies
that
share
their
failure
stories.
That's
just
how
we
learn
this
place
is
so
complex.
They
had
a
upgrade
to
their
kafka
cluster,
which
did
messaging
and
collected
logs
for
all
the
various
systems
they
had
sort
of
a
heterogeneous
network
and
that
caused
intermittent
network
issues
where
the
kafka
system
sort
of
slowed
down
during
that
upgrade.
B
B
Yeah,
exactly
that's
that's
the
gist
of
a
thundering
herd,
so
it's
an
event
comes
in
everything
wakes
up
at
once,
and
because
it's
all
automated
it's
all
pushing
forward
at
once.
It's
trying
to
do
good
things,
but
because
it's
all
synchronized
at
the
exact
same
time,
it
really
hits
the
system
like
a
tsunami
wave
in
a
way
everything
just
starts
toppling
down
like
don
domino's.
B
So
in
this
case
I
think
target
said
over
41
000
nodes
which
spun
up
really
quickly
and
added
to
their
service
discovery
before
everything
sort
of
calms
down.
So
this
can
get
really
big
and
really
nasty
very
quickly.
It's
just
one
of
the
fun
things
you
sort
of
get
to
deal
with
when
you're
working
with
distributed
systems.
A
So
this
is
an
interesting
area
to
dive
into
so
you
know
back,
let's
go
back
to
our
timeline
a
little
bit.
We
had
arpanet
we're
talking,
1969
way
back
when
the
next
kind
of
sets
of
innovations
that
impact,
what
we're
doing
with
distributed
systems
are
in
the
80s
and
90s.
First
is
the
x86
cpu
standards,
and
that
actually
brings
forward
this
era
of
more
commodity-based
hardware,
which
then
leads
to
mass
virtualization.
A
You
know
we
have
like
mass
adoption
of
the
internet
since
the
late
80s
with
that
came
the
birth
of
web
scale
companies,
so
companies
that
fundamentally
delivered
experience
over
the
internet
right.
So
web
apps,
you
see
like
the
the
likes
of
google,
facebook,
twitter
and
netflix
were
all
kind
of
founded
in
the
late
90s
to
the
2000s,
and
so
that's
only
possible
with
the
internet
and
broad-based
connectivity
for
everyone.
A
So
you
had
that
infrastructure
that
global
infrastructure
plumbing
there,
the
availability
of
commodity
hardware
and
concepts
like
virtualization,
so
people
were
able
to
buy
lots
of
lots
of
servers,
running
big
data
centers
and
from
that
they
needed
to
solve
problems
of.
How
do
I
make?
How
do
I,
you
know,
make
better
use
of
my
individual
hardware
resources
cloud
computing.
A
Once
people
started
to
move
to
like
using
things
in
the
cloud
that
then
further
extracted
this
concept
and
then
the
last
bits
are
containers
and
open
source.
There
is
a
clear
line
in
orchestration
systems
in
a
pre
and
post
container
era,
and
because
what
containers
did
is
actually
kind
of
change
it
from
being
just
a
hardware
level
thing
that
you
needed
to
fix
and
a
data
center,
a
data
center
kind
of
pooling
to
now,
like
I'm
doing
some
stuff
in
cloud,
I'm
doing
some
stuff
on
prem
and
also
now
I've
blown
up
my
construct.
A
A
So,
let's
first
start
with
mesos
meso
started
out
in
2009
out
of
berkeley
and
really
what
they
were
looking
at
is
how
do
we
turn
a
bunch
of
data
centers
data
center
orchestration?
So
how
do
I
cluster
a
bunch
of
systems
in
the
data
center
to
make
it
look
like
one
giant
effectively?
One
giant
server
like
I
could
have
like
a
10
000
servers
underneath,
but
it
looks
like
one
server.
So
then
I
can't
schedule
a
workload
on
top
of
that
and
be
able
to
pull.
A
A
A
A
B
A
And
how
do
the
apps
communicate
what's
interesting
here
with
nasa's?
Is
they
drew
a
clear
line
and
what
that,
while
data
center,
while
networking
is
very
important
in
the
data
center,
a
large
part
of
that
is
out
of
the
scope
of
what
mesos
will
actually
orchestrate,
and
so
instead,
what
they've
done
is
made
the
networking
pluggable
and
integrate
two
existing
network
solutions.
A
So,
as
an
example,
meso
supports
two
container
runtimes,
the
mesos
container
and
docker
containers,
and
so
with
that
they
actually
support
the
cni
spec,
as
well
as
a
docker
container,
networking
spec,
and
so
with
that
they
allow
to
create.
You
know
an
enable
ip
for
containers.
You
can
create
a
network
and
then
attach
the
containers
to
them.
A
A
And
so
like,
what
is
what
is
swarm?
It's
the
way
to
cluster
docker
engines
and
swarm
actually
had
two
lights.
First,
we
now
the
original
swarm
is
now
called
classic
swarm
and
then
around
2016
there
was
something
introduced
called
swarm
mode
in
the
first
generation
of
swarm.
It
was
yeah,
simple
clustering
networking
had
some
ability
to
like
link
containers
to
each
other.
It
was
directly
done
in
compose
files,
which
is
the
which
is
how
you
write.
B
A
A
Even
as
the
migration
happened
to
the
next
generation
of
swarm
mode,
it
was
like
the
idea
that
you
didn't
need
all
of
these
other
bits
in
order
to
make
a
clustered
environment.
So
swarm
mode
has
a
lot
of
similar
kind
of
constructs,
as
does
kubernetes
constructs,
but
what
they
did
is
actually
built
that
all
into
the
docker
engine.
So
as
you
instantiate
new
nodes,
turning
one
of
those
into
a
manager
node
or
a
worker
node,
a
single
command
things
like
joining
and
leaving
clusters
single
commands,
also
with
a
lot
of
built-in
defaults.
A
So
upon
instantiating
it
as
a
clustered
node
things
like
pki
other
security,
certs
and
tokens
were
all
kind
of
automatically
handled
by
the
swarm
itself.
So
really
focusing
on
that.
You
know
the
ease
of
the
experience
and
how
are
applications
represented
well
in
the
docker
construct.
There's
something
called
a
compose
file.
It's
a
yaml
and
there's
also
a
compose
version.
A
One
and
compose
version
two,
because
as
applications
and
the
use
of
containers
got
more
interesting
and
more
complicated
as
people
started
using
more
of
those
technologies,
compose
v2
also
took
those
things
into
account
and
so
that
it
defines
the
the
the
containers
how
they
should
be
instantiated.
What
you
know
what
images
are
going
to
be
built
from
services
and
the
environment,
how
they
should
be
scheduled
and
how
they
should
be
networked,
they're
all
part
of
that
or
potterback
file.
B
A
Yeah,
it's
been
super
popular.
I
just
I
just
saw
on
twitter
today
that
one
of
my
someone
I
know
they're
now
actually
receiving
applications
from
their
software
vendors
as
compose
files.
So
it's
great
it's
a
very
simple
way
to
define
things
and
you
can
kind
of
define
all
of
these
things
in
an
order,
and
with
that
you
know
specifically
like
networking
right
once
you
start
having
multi-container
applications.
They
fundamentally
need
to
talk
to
each
other
right
so
in
in
the
docker
construct
of
swarm
mode.
A
There's,
you
know
swarm
classics,
form
and
score
mode
in
classic
swarm.
You
did
things
like
links
and
you
would
actually
define
in
the
file
like
I'm,
going
to
link
container
a
to
container
b
with
swarm
mode,
actually
there's
a
number
of
default
network
drivers
that
were
provided
so
things
like
you
can
do
bridge
networking
host.
You
know
host
networking
where
it's
kind
of
you
don't
need
any
isolation.
A
Everything
on
the
host
can
talk
to
each
other
over
the
networks,
so
that
containers
across
a
number
of
different
hosts
can
talk
to
each
other
as
well
as
a
mac
vlan,
which
is
something
that's
also
popular
in
kubernetes.
So
they
shipped
with
those
network
drivers
allowing
the
operator
to
kind
of
set
up
whatever
they
wanted.
B
Yeah,
so
hashcorp
runs
nomad
or
builds
nomad,
primarily
and
nomad
has
a
lot
of
strength
similar
to
some
of
the
other
stuff.
We've
talked
about
one
big
strength
that
I
think
differentiates
it
is
it's
really
easy
to
plug
into
an
existing
paths.
Platform
on?
That's,
because
it's
very
simple
and
flexible
has
a
lot
fewer
moving
parts
than
most
of
these
other
systems.
B
It's
also
a
bit
more
flexible
on
what
it
manages
or
what
it
schedules.
So
it
can
run
a
bunch
of
different
types
of
processes,
in
addition
to
containers
and
multi-cluster
and
federation
are
already
a
feature.
So
it's
fun
to
play
around
with.
If
you
like,
experimenting
with
that
stuff,
similar
to
what
you
mentioned
with
a
docker
betty,
a
job
file
kind
of
contains
everything
that
you
would
expect
to
see
in
a
deployment
and
a
service,
it's
sort
of
a
monolithic
file.
B
So
you
can
run
it
with
console,
doing
service
discovery
and
service
mesh
type
stuff
for
you
or
you
can
just
run
it
there
by
itself,
in
which
case
it
shares
the
host
network
for
any
of
the
applications
that
are
running
on
top
of
it
and
then
you're
free
to
kind
of
customize
it.
However,
you
like,
should
we
cover
k3s
next
yeah
cool,
so
this
is
sort
of
like
kubernetes
little
brother
k3s
is
a
smaller
lighter
and
it's
a
single
binary.
B
It's
sort
of
a
batteries
included
solution,
so
it's
similar
to
other
distributions
of
kubernetes,
where
all
the
tools
that
you
kind
of
need
to
make
it
work.
Are
there
packaged
with
it
as
it's
installed,
so
it
contains
helm,
for
instance,
and
it's
unique
in
that
most
of
these
systems
that
we've
talked
about
operate
on
raft
or
some
raft-like
system.
B
So
far,
this
runs
on
top
of
a
traditional
relational
database
instead
of
xcd.
It
can
also
run
on
ncd,
but
you
know
that
comes
with
trade-offs
of
in
the
case
of
a
failure.
Things
may
not
quite
be
as
correct
as
they
would
be
in
a
system
designed
with
raft
and
xcd,
but
it's
a
nice
trade-off.
If
you
want
to
do
something
like
have
a
managed
postgres
service
from
the
cloud
manage
your
your
kind
of
state
management
layer,
which
is
what
xe
generally
does
for
kubernetes.
B
So
k3s
is
great
because
you
get
the
same
kubernetes
ecosystem,
because
it's
just
a
slimmed
down
version
of
kubernetes
and
same
deal.
You
also
kind
of
network
things
together
in
a
very
similar
way.
I
think
they
include
their
own
load
balancer
as
an
add-on,
which
is
nice,
but
for
the
most
part,
it's
basically
kubernetes
under
the
cover.
B
It's
a
set
of
nodes
that
xcd
gets
deployed
on
and
the
control
plane
may
include
other
components
like
poor
dns,
the
api
server
controller
manager,
a
proxy
scheduler
which
helps
place
workloads
that
come
in
this
is
all
just
sort
of
what
runs
the
stuff
that
your
cube
cuddle
connects
to
when
you're
trying
to
deploy
something
it
makes
the
decisions,
it
does
reconciliation
for
you.
This
is
kind
of
the
brains
of
kubernetes
and
what
all
these
things
do.
B
That's
yeah
exactly
that's
you
nailed
it
and
then,
when
we
look
at
one
of
the
worker
nodes,
that's
connected
to
kubernetes,
where
the
actual
apps
get
deployed
to
you
know:
they're
things
like
cube
proxy
and
cubelet,
which
receive
communication
from
the
control
plane
and
send
communication
back.
They
actually
execute
the
commands
that
they
receive,
and
then
we
need
something
to
run
containers.
So
container
d
is
running
there
as
well,
and
you
can
probably
explain
container
d
better
than
I
can
so
what's
container.
A
D,
betty
yeah,
so
container
d.
When
you
look
at
docker,
it's
it's
and
you
have
the
docker
engine
container
d
was
a
component
of
that
as
the
core
container
run
time
and
what
we
had
done
years
ago
at
docker
is
take
that
part
and
then
actually
donate
it
into
the
cncf
and
make
it
part
of
the
community
so
that
everyone
in
the
ecosystem
can
leverage
that,
for
the
core
runtime
aspects
of
containers.
B
A
Yes,
and
so
this
is
an
interesting
one,
and
so
a
great
way
to
great
kind
of
a
human
example
of
this
is
you're
out
with
your
with
a
group
of
friends,
one
night
and
one
of
your
friends
trips
and
falls
and
they're.
Obviously
bleeding
and
you
say:
hey:
are
you
hey
buddy?
Are
you
all
right
and
they're
like
I'm
totally
fine?
A
You
can
obviously
see
that
there's
a
problem,
but
that
person
has
decided
to
tell
you
that
they're
fine
so
and
then
the
rest
of
your
friends
you're,
trying
to
figure
out
what
to
do
so
in
the
example
of
distributed
systems.
Is
that
you
know
you're
sending
traffic
and
you're
sending
data
and
the
system
is
not
returning
response
or
behaving
like
it
should,
but
all
the
health
checks
turn
out
fine.
So
it's
kind
of
like
this
mismatch
in
the
desire.
B
Yeah
and
that's
a
problem
for
raft
raft
saves
us
from
a
lot
of
problems,
but
most
protocols
can't
save
us
from
all
problems.
So
there's
always
going
to
be
stuff
like
this,
that
kind
of
lurks
in
the
shadows
and
will
bite
you
when
you're
working
on
distributed
systems
kind
of
closing
things
off.
I
really
like
this
example
from
kent
c
dodds.
B
He's
got
a
master's
degree
in
management
information
systems
and
when
you're
evaluating
any
of
this
type
of
technology,
it's
incredibly
complex
and
incredibly
difficult
and
sort
of
the
beauty
of
working
in
the
open
and
doing
things
collaboratively
through
open
source
and
with
the
community
is
we
all
get
to
kind
of
share
this
burden
together
and
learn
from
each
other?
So
you
know
we
learned
from
the
target
example
earlier,
because
they
were
great
gracious
enough
to
share
with
us.
There's
a
lot
of
other
great
examples
you
can
learn
from.
A
Yeah
we've
got
this
really
long
reading
list
for
you,
with
all
with
links
to
everything
that
we've
mentioned
in
the
talk
as
well
as
other
things.
There's
also,
you
know,
there's
vendor
options
where
they
handle
some
of
these
other
managing
some
of
those
other
bits
for
you,
so
that
you
can
get
some
of
the
experience
but
then
not
have
to
handle
all
of
the
lifecycle
stuff
with
it
and.