►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Health Checks: A Boon or a Curse? - Venil Noronha & John Murray, Stripe
Health checks are an essential part of distributed systems of today that allow one to operate services in a reliable manner. Without health checks, operating a large latency-sensitive system becomes impossible. However, as the composition of environments differ, it can become a real burden to support health checks. Additionally, as the system scales, users tend to face the dreaded N-square problem, and then hit a tipping point, and traditional solutions don't seem to work. In this talk, we will discuss the benefits of health checks in Envoy, some problems we have encountered at scale at Stripe, and options to mitigate such issues.
A
B
A
A
Let's
now
begin
with
the
conceptual
overview
of
health
x,
what
is
health
checking
to
understand
that?
Let's
take
an
example
of
two
services
service,
a
and
service
b
and
service.
A
here
simply
makes
a
request
to
service
b
to
get
the
cat's
gif,
and
if
service
b
is
down,
it's
going
to
result
in
an
error,
this
error
would
go
unnoticed
until
there
are
several
such
requests
that
fail,
and
someone
then
realizes
that
there
is
a
spike
in
errors
on
a
dashboard
somewhere
or
someone
gets
paged
due
to
these
high
number
of
errors.
A
We
could
introduce
a
new
component
called
the
health
checker,
and
the
health
sticker
could
basically
do
two
things.
It
could
first
query
for
the
status
of
the
server
that
is
service
b
in
this
case
and
then
report
the
same
to
the
client,
which
is
service
a
in
this
case
service.
A
can
then
make
a
decision
on
whether
to
actually
make
a
request
to
service
b.
That
is,
if
it
is
healthy
or
simply
hold
back
or
act
differently
if
service
weighs
down.
A
This
pattern
is
very
useful.
If
you
have
redundancy
in
the
system,
let's
see
how,
in
this
example,
we
have
three
instances
of
service
b,
two
of
which
are
down
and
only
one
is
healthy,
and
because
now
we
have
the
health
checker
in
the
system.
It
clearly
knows,
which
instance
is
healthy
and
which
ones
are
not.
A
On
the
right
side,
you
see
example
configurations
the
first
one
says
that
onward
should
consider
a
host
unhealthy
if
it
returns
20
consecutive
5xx
responses
onward
would
then
eject
that
host
from
the
healthy
pool
of
hosts.
For
two
minutes,
the
second
configuration
is
used
for
detecting
timeout
or
connection
issues
that
is
going
to
consider
a
link
to
a
service
unhealthy
if
there
are
consecutive
five
locally
originating
errors.
A
The
primary
difference
between
active
health
checks
and
passive
health
checks
is
that
active
health
checks
is
where
you
need
to
configure
onward
to
make
extra
calls
along
with
regular
data
plane
traffic,
but
in
passive
health
checks.
Envoy,
doesn't
use
extra
network
bandwidths
to
figure
out
if
app
stream
is
healthy
or
not.
A
Let's
now
look
at
the
final
option,
which
is
externally
sourced
health
in
this
case
onward,
doesn't
really
use
any
traffic
between
itself
and
the
service
to
figure
out.
The
status
of
these
upstream
services
onward
instead
relies
on
the
xts
server
to
inform
it
about
the
status
of
each
of
these
upstream
services.
A
There
is
an
external
health
checker
component,
which
knows
the
exact
status
of
each
of
these
upstream
services,
and
it
feeds
this
information
to
the
xts
server
and
then
envoy,
using
the
eds
mechanism,
that
is,
the
endpoint
discovery
service
mechanism
figures
out.
If
a
given
endpoint
is
healthy
or
not.
A
B
B
This
means
that
we
have
a
cluster
configured
on
the
server
side
envoy,
that
health
checks,
the
local
application
on
a
fairly
frequent
interval,
and
then
we
have
a
standardized
health
check-in
point
that
envoy
exposes
where
we
return
the
health
size
of
that
cluster,
meaning
that
we
never
actually
pass
through
any
traffic
to
our
backends
for
health
checking.
It's
all
handled
within
envoy.
B
Our
health
checks
also
aggregate
host
health
with
service
health,
so
that
local
cluster
reforms,
health
checks
to
the
local
back
end.
But
then
we
also
have
kind
of
a
different
set
of
side,
cars
and
processes
that
determine
if
a
server
is
healthy,
and
so
our
health
checks
will
also
look
at
this
value
and
decide
whether
we
should
be
healthy
or
not.
B
B
Historically,
this
has
worked
really
well
for
strength.
We
have
some
low
volume,
latency
sensitive
connections,
so
you
can
think
of
like
some
cross
region.
You
know
maybe
one
data
center
in
us
another
data
center
in
europe
or
asia,
and
for
regulatory
reasons
we
might
have
to
locate
some
of
our
data
processing
within
those
regions,
but
then
it
may
periodically
request
something
back
from
a
us
data
center.
B
These
connections
are
very
lazy
sensitive
because
the
cost
is
very
high
and
relatively
low
volume.
So
having
active
health
checking
means
that
we
don't
have
to
rely
a
ton
on
retries.
We
generally
can
land
on
that
on
a
healthy
host
because
of
back
of
health
checks,
and
also
we
have
traffic
shifting
controls.
We
think
blue
green
deployments,
or
things
like
things
of
that
nature
hosts
go
from
none
or
very
little
traffic
to
high
amounts
of
traffic.
B
So
that's
all
I'm
going
to
say
about
stripes
health
checking,
but
I
do
want
to
talk
about
quite
a
bit
about
problems
that
we've
kind
of
seen
at
scale
and
some
some
solutions.
You
might
consider
so
an
overview
of
the
problems
slow
times,
detection
right.
This
is
problematic
for
high
volume,
high
volume
connections,
so
connections
where
there's
a
lot
of
traffic
flowing
if
your
health
check
interval
is,
is
fairly
wide,
then
there's
a
decent
amount
of
time.
B
Where
failures
can
happen
before
you
react
high
health
check
volume
right,
so
you
have
these
like
shared
foundational
services
that
have
a
ton
of
downstream
they're.
All
doing
all
this
health
checks
upstream
and
that
can
that
can
cause
a
lot
of
problems
and
the
active
pros
themselves
because
of
this
volume
can
kind
of
add
significant
overhead
and,
lastly,
network
costs,
right
health
checks,
kind
of
traverse,
all
the
paths,
regardless
of
all
the
effort
and
work
we've
put
in
to
optimizing
our
our
network
efficiency
and
costs.
B
Okay.
So
let's
talk
about
slow
time
detection.
So,
let's
start
with
an
example:
we're
going
to
have
active
health
check
set
to
an
interval
of
five
seconds,
meaning
that
our
downstream
is
pulling
the
upstream
for
itself.
Every
five
seconds
network
link
processes,
twenty
five
thousand
qps.
So
by
network
link
I
mean
a
single
downstream
upstream
pair
right,
sending
25
000
requests
a
second,
so
the
potential
impact
of
that
server
becoming
unhealthy
is
essentially
the
duration
number
of
seconds
times.
B
B
The
drawback
to
this
is
that
there's
additional
steady
state
costs
right.
Even
when
there's
not
traffic
flowing
you're,
sending
active
health
checks
that
new
interval
that
may
cause
you
know
more
more
load
on
the
server
and
there's
only
so
far.
We
can
drive
this
down
to
to
try
to
mitigate
that
potential
impact
amount.
B
A
better
strategy
is
to
implement
passive
health
change
right
or
an
envoy
outlier
detection
same
thing,
so
this
is
faster
than
active
health
checking
right
because,
as
we
saw
earlier,
we
can
look
at
you,
know
consecutive
5x
and
maybe
we'll
fail
after
50
requests
right.
That
can
happen
very
very
fast.
B
It
has
lower
overhead
compared
to
acapella
checking
right,
meaning
that
we
only
pay
the
cost
when
there's
actual
traffic
flowing
from
a
downstream
to
an
upstream,
and
if
we
want
a
holistic
setup,
we
can
still
configure
this
with
active
health
checking
right
if
we
still
want
to
satisfy
that
traffic.
Shifting
scenario
that
we
talked
about
earlier
or
those
low
volume,
latency
sensitive
links,
we
can
still
have
acapella
checking
on
top
of
it
to
provide
more
of
a
holistic
solution.
B
B
In
that
case,
it
likely
has
a
lot
of
downstream
it's
kind
of
all,
performing
active
health
checking
on
that
single
upstream
host.
So
what
are
some
issues
we've
encountered?
Well,
first,
is
like
high
back
end
load
right.
I
mean
think
about
your
your
back,
an
application
actually
serving
all
these
health
check
requests
and
this
this
load
can
impact
normal
traffic
as
well.
B
If
your
backend
application
becomes
overloaded
high
connection
load
right,
we
we
have
to
have
a
connection
in
order
to
make
an
active
health
check,
and
if
we
have
a
lot
of
down
streams
like
in
that
diagram
checking
a
single
upstream
host.
You're,
just
gonna
have
a
lot
of
connections,
dedicated
health
checking.
In
fact,
these
health
check
load
could
easily
exceed
your
actual
traffic
load
in
terms
of
the
connections
created
to
service
that
high
cpu
load
right.
B
So
both
the
application
and
envoy
still
have
to
do
work
every
time
they
make
a
health
check
request.
So
anecdotally,
what
are
some
things
we've
seen?
Well,
we've
seen
a
host
burning,
two
plus
cpu
cores
like
exclusively
to
serving
health
check
traffic
or
one
of
these
kind
of
like
central
services
that
had
a
lot
of
downstream
health
checking.
It
we've
had
to
deploy
larger
instance
types
just
to
account
for
the
overhead
of
health
check
volume
and
we've.
Also
in
the
past,
observed
a
general
slogan
of
envoy
right,
even
with
implemented.
B
Caching
and
just
hitting
those
cached
endpoints
onboard
may
take
30
plus
seconds
to
respond
simply
because
it
was
so
overwhelmed
with
the
amount
of
health
check.
Traffic
we've
also
seen
hosts
kind
of
reaching
their
file
descriptor
limits
because
of
the
number
of
connections
created
for
health
checking.
I
mean
this
has
a
real
impact
on
live
traffic
as
well
a
bunch
of
other
functions
of
the
host,
all
right
so
mitigations.
What
can
we
do
to
make
this
better
one?
We
can.
We
can
cache
the
back
end.
B
Health,
there's
two
types
of
caching
here:
one
is
the
traditional
type
of
caching,
where
the
response
from
the
the
back
end
health
check
service
is
cached
for
some
duration
of
time
and
then
once
it
expires,
active
health
checks
are
passed
through
to
the
application
again
and
we
catch
that
value
again.
This
reduces
the
load
on
the
backend
applications,
but
we
still
may
see
spikes
of
activity
as
the
cache
expires
and
then
there's
the
kind
of
internal
cluster
caching,
which
kind
of
what
I
described
earlier.
B
One
thing
we
can
do
is
we
can
reduce
the
health
check
interval
right,
so
maybe,
instead
of
five
seconds,
we
make
it
make
it
ten
seconds
right
that
that
reduces
our
load,
because
you
know
just
reduce
the
number
of
requests
that
we
send.
The
caveats
here
are
that
this
this
may
violate
your
business
constraints
right.
B
Maybe
your
traffic
shifting
controls
or
your
deployment
systems
are
relying
on
that
interval
value
for
for
how
they
operate,
and
you
know,
as
we
saw
in
the
previous
example,
maybe
it
competes
with
faster
time
to
detection,
reduce
reachability
note,
there's
no
asterisk
here
there
is
no
there's
no
drawback
to
this.
This
is
essentially
meaning
that
you
should
be
looking
at
your
surface
graph
and
the
active
traffic
that
is
flowing
across
it
and
for
links
that
that
are
maybe
defined
in
the
reachability
graph,
but
aren't
being
used.
B
You
should
prune
those
there's,
there's
really
no
drawback
to
making
sure
your
reachability
graph
is
properly
print
control,
plane,
subsets.
So
this
is
the
idea
that
the
control
plane
delivers
a
subsetted
view
of
your
your
service
graph
to
each
envoy,
artificially
shrinking
the
size
of
your
fleet
to
reduce
the
health
check
volume,
that's
being
created
right.
Each
envoy
node
will
then
help
check
less
upstreams
because
they
simply
don't
know
about
them.
B
This,
however,
is
very
difficult
to
correct
for
errors
and
having
the
full
set
of
nodes
in
the
reachability
graph
and
envoy
is
really
important
for
reliability
and
error
recovery
and,
lastly,
we
could
move
to
a
centralized
health
check
system
right.
This
is
most
interesting
because
it's
the
most
flexible
we
can
account
for
for
volume.
We
can
account
for
the
interval
right.
B
Big
asterisks.
Here,
though,
because
this
is,
you
know,
a
lot
of
work
if
you're
gonna
go
this
route
and
there
are
likely
a
ton
of
hidden
landmines
with
this
approach.
Okay,
next
is
nowhere
costs.
Here
you
can
see
we
have
three
different
services
running
in
three
different
regions
or
availability
zones,
and
that
thick
bar
kind
of
represents
a
cost
boundary
right
and
we
can
see
they're
doing
health
checks
across
these
crop
across
these
cost
boundaries,
and
you
know.
B
B
You
know
first
right,
this
doesn't
work
because
we
still
need
those
active
health
checks
for
those
traffic
shifting
primitives
and
for
those
kind
of
low
volume,
latency
sensitive
links.
Right
again,
we
can
reduce
the
reachability.
You
should
do
this.
This
is
a
definite
win
and
very
few
drawbacks.
Here
again,
we
can
do
control,
plane,
subsets
right.
We
can
reduce
the
number
of
edges
in
the
reachability
graph
and
then,
therefore
we
can
reduce
costs
again.
B
This
is
still
hard
to
correct
for
errors
and
again
we
can
move
to
centralized
health
checks
and
the
interesting
part
of
this
design
is
that
using
a
system
like
this,
we
could
optimize
for
locality
and
putting
those
those
health
check,
nodes,
local
in
the
region
and
data
center
for
the
items
of
health
checking,
you
can
do
smart
things
like
sharing
discs
across
those
cost
boundaries
to
kind
of
reduce
the
the
impact,
and
so
what
are?
What
are
the
conclusions
we
can
draw
from
this?