►
From YouTube: Logs Told Us It Was DNS, It Felt Like DNS, It Had To Be DNS, I... Laurent Bernaille & Elijah Andrews
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Logs Told Us It Was DNS, It Felt Like DNS, It Had To Be DNS, It Wasn’t DNS - Laurent Bernaille & Elijah Andrews, Datadog
It all started with a team reaching out because they had DNS issues during rolling updates. Business as usual when you host hundreds of applications on dozens of Kubernetes clusters… Four weeks later: We are reading kernel code to understand the corner cases of dropping Martian packets. Could this be the connection between gRPC client reconnect algorithms and the overflowing conntrack table we can feel but not see? In time, we solved the issue. And for once… it wasn't DNS! In this talk, we will focus on one of the most complex incidents we have faced in our Kubernetes environment. We will go through the debugging steps in detail, dive deep into the mysterious behaviors we discovered and explain how we finally addressed the incident by simply removing three lines of code.
A
A
So
today
we're
going
to
talk
about
the
time
that
it
wasn't
dns
everything
seemed
to
tell
us
that
it
was
dns
the
logs
it
felt
like
dns,
it
had
to
be
dns
and
it
wasn't
so
first
off
we're
here
representing
datadog.
This
is
a
talk
about
datadog's
infrastructure,
not
about
the
product
that
we
have,
but
just
in
case
you're
not
familiar
we're
an
observability
platform.
A
We
monitor
all
the
things
we
have
a
sweet
booth
in
the
exhibition
hall
and
you
should
go
check
it
out,
we're
doing
a
raffle
for
a
nintendo
switch
this
afternoon.
So
you
should
go,
get
your
badge
scanned
and
see
what
we
have
to
offer
anyway.
The
numbers
that
I
want
to
focus
on
for
this
talk
are
the
way
that
we
run
kubernetes
and
we're
really
big
on
that
technology.
We
run
it
at
a
big
scale.
A
Just
to
give
you
an
idea.
We
have
some
figures
on
the
right-hand
column
here,
so
we
run
thousands
of
tens
of
thousands
of
nodes
and
hundreds
of
thousands
of
pods.
We
have
dozens
of
kubernetes
clusters
with
anywhere
from
hundreds
to
thousands
of
nodes.
So
we're
running
this
at
a
very
large
scale.
We
run
on
all
the
major
cloud
providers
and
we're
growing
very
quickly,
but
before
we
dive
too
deep
into
the
talk
I'll,
let
you
know
who
we
are
so
I'm
elijah
andrews,
I'm
a
software
engineer
at
datadog.
A
A
So
you
can
see
there
are
two
lines
here
on
that
top
graph.
The
one
of
the
lines,
the
the
purple
one
corresponds
to
the
server,
and
you
can
see
here
that
during
a
rolling
restart,
which
is
what
was
happening
here,
there's
a
huge
spike
in
errors,
the
red
line
there
is
showing
the
this
from
the
client's
perspective,
the
client
actually
retried,
so
it
didn't
show
up
as
an
error
rate
in
the
client.
However,
our
latency
went
up
by
a
lot
because
the
clients
had
to
retry,
and
this
was
degrading
performance
in
our
application.
A
So,
as
we
always
do
when
we
have
problems
with
our
infrastructure,
we
check
the
logs
and
when
we
trace
the
requests
that
were
failing,
we
saw
that
they
were
telling
us
that
it
was
dns.
That
was
having
a
problem
and
it
really
looked
like
a
dns
problem.
It
had
to
be
dns,
it's
always
dns
right,
okay.
So
let's
talk
about
dns
now
so
before
we
go
too
far
into
details,
I'm
just
going
to
give
you
like
a
high-level
view
of
the
applications
involved
here.
So
at
the
center
we
have
the
metric
service.
A
This
is
a
service
that
powers
our
time
series
data
system
at
datadog.
So
what
it
does
is
it
provides
like
when
you
go
to
open
up
a
dashboard
in
our
product,
and
it
shows
you
time
series
data,
that's
all
collected
by
the
metric
service.
It
also
powers
our
alerting
engine.
So
in
datadog
you
can
set
thresholds
and
monitors
on
time
series
data
and
when
those
conditions
are
met,
it
can
alert
you
and
that
the
alerting
engine
pulls
the
metric
service.
A
In
order
to
evaluate
the
your
your
monitors
behind
the
scenes,
the
metric
service
has
a
few
dependencies.
It
has
to
talk
to
an
index
for
our
time
series
data
and
our
storage
layer.
So
what
it's
really
just
doing
is
stitching
together
all
the
time
series
data
from
the
different
sources
that
it
knows
about.
A
So
here's
what
was
happening.
We
are
seeing
the
clients
of
the
metric
service
see
query
errors
when
a
rolling
restart
was
happening
and
the
dns
errors
that
we
were
seeing
were
corresponding
to
the
service
discovery,
to
discover
its
dependencies,
the
index
and
the
store
we
use
dns
for
service
discovery,
and
so
here's
what
our
dns
setup
looks
like.
So
this
is
a
somewhat
conventional
kubernetes
dns
setup.
We
run
the
metric
service
in
a
pod
in
kubernetes
and
over
udp,
a
dns
request
flows
to
no
local
dns.
A
So
no
local
dns
is
a
caching
and
forwarding
dns
resolver,
it's
a
stripped-down
core
dns
configuration
and
then
when
no
local
dns,
if
node
local
dns
receives
a
query
and
it
doesn't
have
it
in
its
cache.
The
query
then
flows
to
cluster
dns,
which
is
a
cluster-wide
core
dns
configuration
that
also
does
caching
and
it
can
result.
It
is
authoritative
for
resources
that
exist
within
the
cluster.
However,
you
saw
in
our
first
slide.
We
run
dozens
of
kubernetes
clusters
and
they
all
talk
to
each
other.
A
So
sometimes
resources
exist
outside
of
the
cluster
need
to
be
discovered.
In
order
to
do
that,
we
use
an
external
dns
provider.
In
this
case
we
are
running
our
stuff
on
ec2
in
aws,
so
our
external
dns
provider
was
route
53,
which
is
their
dns
solution,
and
so
any
resource
that
metric
service
needs
to
discover
that
doesn't
exist
in
the
same
cluster
is
resolved
by
rev53,
and
so
here
we
are
seeing
the
dns
errors
emanating
from
the
metric
service.
A
And,
interestingly,
we
saw
that
the
failure
was
happening
in
no
local
dns
itself,
which
is
kind
of
weird,
because
that's
just
happening
over
the
node
interface
like
it's
not
going
over
a
network,
and
it's
not
really
where
we
expected
to
see
the
problem,
so
we
dug
in
and
we
actually
saw
that
no
local
dns
was
running
out
of
memory,
and
this
was
really
surprising
to
us
because
it
should
never
happen.
The
reason
it
should
never
happen
is
because
we
set
a
concurrent
request
limit
in
no
local
dns.
A
We
say
if
you,
if
you
receive
over
a
thousand
requests
at
a
time
you
should
reject
them
and
put
back
pressure
on
the
client
so
that
you
don't
run
out
of
memory.
So
this
just
implied
to
us
that
the
sizing
that
we
had
for
no
local
dns
was
wrong,
like
maybe
we
weren't
giving
it
enough
memory
to
serve
a
thousand
concurrent
requests.
So
the
first
thing
that
we
did
here
was
we
increased
the
amount
of
memory
that
we
were
giving
it.
A
A
A
So
we
looked
more
at
all
of
our
all
of
our
graphs
and
tried
to
figure
out
what
was
happening
and
this
one
stood
out
to
us
so
node
local
dns
has
a
layer
called
a
forwarding
layer
and
that's
the
layer
that
it's
a
plug-in.
A
core
dns
plug-in
that
is
responsible
for
forwarding
the
request
to
cluster
dns,
and
we
noticed
that
the
forward
plug-in
was
telling
us
that
no
local
dns
was
having
health
check
failures
when
trying
to
talk
to
the
cluster
level
resolver.
A
And
this
explained
why
we
were
hitting
max
concurrent.
Here
we
looked
at
the
timeout
for
creating
a
connection
from
the
local
dns
to
cluster
dns
and
we
saw
that
it
was
five
seconds
and
so,
if
an
incoming
query
comes
in
and
there's
no
connection
to
the
to
cluster
dns,
that
request
will
block
for
five
seconds
as
we
timeout
trying
to
create
the
connection,
and
this
means
that
we'll
hit
max
and
current
of
a
thousand
with
only
200
requests
per
second.
So
it's
starting
to
make
a
bit
of
sense,
but
it's
still
really
weird.
A
At
this
point
we
thought
maybe
we
were
having
a
networking
issue
like
perhaps
we
were
saturating.
The
network
on
this
instance,
and
so
we
looked
at
what
we
should
be
able
to
achieve
so
we
run
this
on
m5
4x
large,
which
is
an
ec2
aws
instance
type,
and
that
instance
is
allowed
to
peak
at
10
gigs
per
second
on
network
throughput,
and
it
should
be
able
to
sustain
5
gigabits
per
second,
and
you
can
see
our
graph
of
the
throughput
here,
we're
nowhere
near
even
the
sustained
guarantee
so
that
all
looks
okay.
A
But
we're
still
seeing
things
that
suggest
that
our
network
is
saturated
right,
we're
seeing
tcp
retransmits
we're
dropping
packets
and
maybe
we're
seeing
microbursts.
Instead
right,
there
could
be
really
spiky
traffic
that's
coming
in,
but
the
graphs
that
we're
looking
at
here
are
aggregated
for
every
15
seconds.
So
maybe
like
that
data
is
kind
of
being
eaten
by
our
reporting.
A
And
when
you
see
this
type
of
thing
in
networks,
what
you
normally
do
is
look
at
the
counters
in
the
network
driver
which
count
like
events
that
happen
and
aren't
subject
to
this
sort
of
like
time
aggregation
like
observability
issue
that
we
see
a
lot
in
networking
and
luckily
about
two
weeks
before
we
hit
this
incident,
we
added
support
for
the
elastic
network,
adapter
metrics.
So
if
you're
not
familiar
with
the
elastic
network
adapter,
it's
it's
a
networking
interface
that
runs
in
aws's
hypervisor.
B
So
once
we
had
instrumented
the
instances
to
look
at
the
low-level
ena
metrics,
we
saw
this
promising
graph
on
the
bottom
left
side
of
the
slide,
which
showed
that
we're
actually
actually
exceeding
the
limit.
However,
if
you
look
at
all
the
autograph,
it's
not
correlated
with
the
deployment
we're
always
going
over
the
limit,
sometimes,
but
it's
maybe
just
normal
tcp
behavior
right
tcp
is
self-regulating
and
it's
not
correlated
with
the
graph.
So
it's
not
another
problem.
B
The
interesting
thing
is,
as
we
were
saying
before,
we're
running
tens
of
thousands
of
aws
hosts
and
we
have
never.
We
had
never
encountered
anything
related
to
contracting
at
on
the
edubles
level
before
so
that
was
very
surprising
to
us.
I
mean
it
made
sense
that
the
limit
existed,
but
we
had
never
seen
it
before,
which
was
a
surprise.
B
So
the
first
thing
we
did
is
well.
We
tried
other
instance
types,
so
the
first
instance
type
we
tried
was
networked
optimized
instances
because
they
have
highest
throughput.
So
it's
very
good,
as
you
can
see
here,
the
two
lines
in
the
middle
show
ingress
and
ingress
drops
for
the
network
optimized
instances.
So
it's
much
better,
so
it
was
very
promising.
B
B
So
we
tried
bigger
instances.
Instead,
we
we
took
an
instance
that
was
twice
bigger
and,
as
you
can
see
on
this
graph
here,
it
solved
the
issue
completely
right.
You
can
compare
the
light
blue
graph
and
the
purple
one,
and
you
can
see
the
purple.
One
is
much
better,
because
everything
is
basically
zero.
No
contract
exceeded
errors,
no
errors
so
extremely
promising
for
us,
except
well,
it's
addressing
our
issue,
but
it
would
mean
that
our
metric
service
infrastructure
would
get
twice
as
expensive,
which
was
a
bit
of
a
hard
sell.
B
B
B
B
At
that
point,
we
had
no
idea
what
to
do,
because
things
made
no
sense,
and
so
we
went
even
lower
level
and
we
tried
we
started
looking
at
the
vpc
for
logs,
which
are
basically
like
if
you're
familiar
with
cisco.
It's
like
netflow
type
of
data,
where
you
have
information
about
tcp
connections,
you
have
one
floor
in
each
direction
because
they
are
not
stitched
together
and
you
get
very
detailed
information
on
what's
up
what's
happening
on
the
network.
B
We
looked
at
egress
flows
and
we
grouped
flows
by
source
ip
and,
as
you
can
see
on
this
graph
here,
we
have
flows
from
the
old
ip
during
our
wall
out
and
then
flows
from
a
new
ip,
which
makes
sense
right.
We
we
roll
out,
we
replace
the
pods,
the
ip
change,
so
we
see
flows
created
by
the
old
pod
and
then
focused,
but
no
one.
What's
pretty
weird,
there
is
the
very
big
spike
you're,
seeing
where
we're
spiking
up
to
50
000
flows.
B
So
at
that
point
we're
we
were,
I
mean,
as
I
was
mentioning
before
this
information
is
we.
You
have
two
connections.
We
have
two
flows
for
its
connection,
one
for
egress,
one
for
egress
and
and
so
we
were
trying
to
see
what
was
happening
for
ingress
flow,
so
flow
actually
getting
into
the
instance,
and
the
graph
is
exactly
similar,
which
makes
sense
right
because
tcp
when
you
have
a
connection
with
tcp,
you
have
flow
in
both
directions
except
there's,
a
second
spike
that
is
not
aligned
at
all.
With
with
what
we
see
egress.
B
B
B
B
It
waits
for
this
connection
to
finish
and
tell
the
clients
to
terminate
so
during
these
10
seconds,
it
makes
complete
sense
to
get
resets,
because
this
is
what
grpc
is
going
to
do.
However,
after
this
10
seconds,
the
body
is
deleted
and
the
ip
is
not
there
anymore
right,
it's
deleted.
So
there's
nothing
to
answer
and
that's
why
we're
not
saying
an
answer
to
this
in
packets?
B
Another
thing
we
knew
at
that
time
is
well
given
we
have
all
the
flows.
We
could
identify
the
application
connecting
to
our
metric
service
and
we
identified
that
the
alerting
engine
was
actually
making
the
connections
and
well,
we
looked
at
the
contract
on
the
host
of
much
highlighting
engine
nodes
and,
as
you
can
see
here,
they
are
spiking
very
high.
So
it
seems
to
confirm
that
this
is
what's
happening.
B
So
at
that
point
what
we
know
is
well.
We
have
dns
errors
because
no
local
dns
can
connect
to
upstream.
We
know
we're
actually
saturating
the
breast
contract,
because
we've
we've
seen
hundreds
of
thousands
of
sins
right
and
we
know
which
application
is
seen.
Flooding
the
metric
service,
the,
but
it
doesn't
explain
why
what
we're
seeing
in
the
instance
is
so
different
from
what
edwards
is
seeing
in
its
at
the
apparel
level.
B
So
on
our
nodes,
we
use,
we
use
celium
to
provide
cni
and
the
way
this
works
is
we
have
the
serium
operator
allocate
ip
additional
ip
to
nodes
that
are
used
for
pods
and
we
allocate
ips
on
an
additional
interface.
So
not
the
main
interface
of
the
host,
an
interface
dedicated
to
pods
and
once
the
so,
the
operator
is
responsible
for
maintaining
an
ipo
for
pods,
and
when
you
create
a
pod,
the
serial
agent
is
going
to
grab
an
f3
ip,
allocate
it
to
the
pod.
B
B
We
also
need
to
route
traffic
outside
of
the
pod
and
and
and
we
need
to
use
the
right
interface
so
to
do
that.
We
use
source
routing
and
we
achieve
this
with
an
ip
rule.
So
selem
is
creating
the
ip
rule
and
say
traffic
from
this
ip
should
be
using
this
route
table
which
will
use
the
additional
interface
to
get
traffic
out.
B
So
in
a
stable
state
where
an
alerting
node
is
sending
in
trying
to
connect
to
a
metric
service
pod,
it
sends
a
scene
and
you
can
see
the
contract
is
consistent
everywhere,
where
the
connection
is
starting
to
open.
Then
we
get
the
synag
and
the
connection
transition
to
establish
state
and
it's
aligned
everywhere
on
the
node
in
the
hypervisor
in
on
the
alerting
node.
B
So
when
we
delete
the
pod
and
we
still
get
traffic
to
the
old
ip
because
it
takes
some
time
for
service
cover,
information
to
propagate
so
client
will
try
to
reconnect
to
the
old
ip
and
the
ip
is
still
held
by
the
by
the
interface.
So
the
vbc
fabric
will
send
traffic
to
the
to
the
node,
and
the
thing
is
we
have
this
scene
and
this
ip
is
not
known
anymore
right,
because
the
all
the
routing
information
has
been
garbage
collected.
B
So
we
were
wondering
what
what's
happening
to
this
packet
all
right.
So
we
have
this
in
packet
incoming
and
we
don't
know
where
it's
going.
So
what
we
did
is
well.
We
tried
to
to
to
do
a
connection
ourselves
and
and
simulate
it.
So
we
connected
from
another
node
and
we
capture
traffic
with
tcp
dump,
and
so
we
see
the
syn
packet
coming,
but
no
answer
whatsoever.
Nothing
at
all.
B
B
And
this
is
where
things
start
to
get
a
bit
more
interesting
and
more
fun,
which
is
this
error
message
here,
which
shows
that
we're
hitting
reverse
pass
filtering,
like
the
candle,
refuses
to
do
something
with
this
packet,
because
it's
doing
something
wrong
according
to
the
kernel
and
reverse
pass
filtering?
Is
a
security
feature
we're
going
to
talk
about
just
after,
but
this
was
confirmed
in
the
kernel
logs,
where
we
see
this
warning
there,
which
is,
was
something
about
we're,
seeing
a
machine
packet
so
packet.
We
should
never
see
on
this
interface.
B
For
those
of
you
who
are
not
familiar
with
reverse
pass
filtering,
it's
a
secret
feature
from
the
kernel
to
prevent
ip
spoofing,
so
you
can
send
a
packet
with
a
source
ip,
that's
not
supposed
to
be
there
and
you
have
different
modes.
The
standard
mode
is
if
the
return
pass
would
use
a
different
interface
drop
the
packet,
so
that's
tricks
mode
and
and
load
the
seven
as
much
in
packets
and
there's
lose
mode,
which
is
only
drop
packet
if
there's
no
return
route.
B
What
does
this
mean
in
terms
of
connection
tracking?
So
what's
interesting
here
is:
if
applications
attempt
to
connect,
they
will
follow
their
own
contract,
they
will
feel
the
hypervisor
contract,
but
then
the
same
packet
hits
reverse
pass
filtering
in
the
kernel
and
it's
just
dropped.
So
it's
not
added
to
the
note
contract
which,
which
explains
why
the
contract
is
so
different
in
terms
of
size
compared
to
the
others.
B
So
everything
made
sense,
except
this,
which
was
very
confusing
to
us.
You
remember
before
I
was
saying
that
reverse
pass
filtering
can
be
both
in
it
can
either
be
in
strict
mode
or
lose
mode,
and
we
set
it
to
lose
mode.
We
see
the
default
in
most
distributions
and
the
thing
is
when
it's
set
to
lose
mode.
B
It
means
that
you
only
drop
a
packet
if
there's
no
possible
egress
routes
for
the
source
ip,
but
of
course
we
have
a
default
route
on
the
nodes
right,
the
the
main
port,
the
main
interface
on
the
node,
and
so
we
should
be
able
to
route
traffic
through
this.
So
the
traffic
should
be
incoming
on
es6,
the
pod
interface
and
egressing
on
ens-5
and
then
just
be
dropped
by
aws,
but
it's
not
what's
happening
so
that
was
very
confusing
to
us.
B
B
B
Okay,
all
this
makes
sense
now,
and
this
is
starting
to
get
interesting.
The
only
way
to
get
to
this
error
was
actually,
if
this
variable
there
no
idea
was
set
to
true,
and
this
viable
is
set
very
early
in
the
function
and
it's
set
to
true.
If
the
interface
has
no
ip,
so
it
turns
out
our
pod
interface
doesn't
have
an
ip
address
assigned
because
they
don't
need
one
right.
B
We
just
transit
traffic
through
them
and
we
don't
set
an
ip
on
it,
and
so
at
this
point
like
maybe
this
is
the
problem
of
heating,
so
we
simulate
it.
So
the
first
step
is
the
one
we
did
before
and
then
we're
like.
Well,
let's
add
a
random
ip
whatever
on
the
additional
interface
and
see
what's
happening
and,
as
you
can
see
here
as
soon
as
we've
added
a
random
ip
to
the
additional
interface,
it's
actually
working
now
right,
we
have
a.
We
have
a
route
through
ens
5..
B
So
here
is
the
status
now.
So
you
remember,
we
have
dns
issues
because
the
code
dns
can
connect
to
upstream
we're
actually
seen
flooding
the
aws
contract
and
we're
still
flooding
the
contract,
but
we're
not
impacting
the
contract
on
the
host
because
we're
dropping
packets,
because
we're
hitting
reverse
pass
filtering
because
of
a
weird
edge
case
in
the
kernel
where
we
don't
have
an
ip
on
the
interface.
A
So,
in
order
to
understand
why
we
are
sending
so
many
sins,
we
have
to
look
at
the
way
that
our
rpc
system
is
set
up.
So
we
had
two
main
questions
here.
Why
were
we
sending
sin
requests
for
so
long
like
there
was
a
long
period
where
we
were
sending
a
bunch
of
these
requests
and
also
there
were
a
bunch
of
like
really
big
spikes
in
there,
and
we
were
wondering
why
those
were
happening,
and
so
just
a
reminder
about
what
our
rpc
setup
is
here.
A
So
we
use
dns
for
service
discovery
that
goes
out
and
hits
the
external
dns
provider
in
this
case
rep53,
and
we
also
use
grpc
as
our
rpc
mechanism
and
one
important
thing
it's
going
to
be
important
in
a
second
is
that
we
do
client-side
load
balancing.
So
this
means
that
the
every
alerting
engine
talks
to
a
bunch
of
metric
services
directly
like
they,
they
have
a
bunch
of
ips
and
they
talk
to
them,
there's
no
like
load
balancer
or
any
connection
pooling
in
the
middle
okay.
A
So,
first,
why
were
we
sending
sin
requests
for
so
long?
So,
in
order
to
understand
why
why
that
period
was
so
long
as
it
was,
we
had
to
look.
We
looked
at
the
way
that
our
external
dns
was
configured
so,
if
you're
not
familiar
with
external
dns.
This
is
a
controller
that
you
can
run
in
your
kubernetes
clusters
and
what
it
does
is.
A
It
looks
at
pod
events
like
pods
coming
online
and
pods
going
offline,
and
then
it
takes
the
ips
that
it
returns
and
it
puts
them
into
an
external
like
cloud
provider,
dns
provider,
and
so
what
was
happening
here
was
like
when
a
metric
service
pod
was
being
deleted.
External
dns
would
capture
that
event
and
then
go
and
update
route
53.,
and
so
what
we
noticed
was
we
actually
found
a
really
interesting
piece
of
behavior
in
the
version
of
external
dns,
where
we
are
running
so
this
first
step
here.
A
The
metric
service
pod
is
deleted
and
then
it
the
pod,
receives
sig
term
and
we
give
it
a
10
second
timeout,
and
at
this
point
we
call
grpc
graceful,
stop.
It
sends
go
aways
to
its
existing
connections
and
resets
to
new
connections
coming
in.
But
one
thing
we
found
is
that
in
our
version
of
external
dns,
it
was
not
when
sig
term
was
sent
to
the
pod
that
it
was
deregistered
from
dns.
A
A
There's
a
fun
balancing
act
here
between
like
how
frequently
you
want
updates
and
how
big
you
want
your
batches
to
be
because
you
can
hit
problems
with
rate
limits
in
your
cloud
provider.
If
you
don't,
if
you,
if
your
batches
are
too
small.
So
here
we
set
our
sync
loop
to
15
seconds,
so
that
meant
that
it
took
up
to
15
seconds
for
that
loop
to
run
and
be
updated
in
rep
53
and
then
again,
there's
a
in
dns.
A
So
it
turns
out
that
this
is
actually
bound
by
a
grpc
setting
called
min
time
between
resolutions
and
we
use
the
default
value
which
is
set
to
30
seconds.
So
the
interesting
thing
about
this
is
that
actually,
the
updates
that
the
alerting
engine
would
see
were
actually
quantized
in,
like
30
second
increments.
So
it
would
see
an
update
either
after
30
seconds
or
more
likely
after
60
seconds.
A
If
you
look
at
like
the
average
for
all
the
other
delays
and
sometimes
even
90
seconds,
but
nothing
in
between
and
that
quantization
was
actually
really
confusing
to
us.
When
we
were
looking
at
all
the
all
the
graphs
and
seeing
that
everything
was
like
a
minute
or
90
seconds,
and
we
were
that
solved
that
mystery,
and
so
this
lines
up
with
the
propagation
time
that
we
are
seeing
in
the
graphs
lahore
was
showing
earlier.
So
the
deletion
starts.
A
We
see
a
huge
spike
in
sins
and
then
the
clients
progressively
start
using
the
new
ips
and
eventually
after,
like
60
or
90
seconds,
no
clients
are
using
the
old
ips.
So
that
makes
sense
if
this
is
really
a
balancing
act
like
if
we
wanted
to
make
this
shorter,
we
would
have
to
put
more
load
on
our
dns
infrastructure
and
on
the
cloud
provider
and
so
like.
We
were
okay
living
with
this
to
understand.
A
Why
we're
seeing
those
huge
spikes,
this
one's
a
bit
more
complicated
to
explain,
and
it
requires
you
to
understand
the
way
that
we
use
grpc
at
datadog.
So
we
started
using
grpc
at
datadog
many
years
ago
and
when
we
started
using
it,
we
had
the
way
that
we
would
chart.
Our
applications
was
by
writing
really
thick
clients.
So
the
clients
were
completely
aware
of
how
their
servers
were
sharded
like
they
would
download
partitioning
tables
and
stuff,
and
we
would
actually
resolve
the
dns
entries
into
ips
and
just
pass
the
ips
to
grpc.
A
But
the
way
that
grpc
is
really
intended
to
be
used
is
actually
you
give
it
a
host
name,
and
it
resolves
that
ip,
like
in
the
background
and
your
application
doesn't
have
to
worry
about
that
at
all,
and
we
actually
started
having
a
bunch
of
incidents
where
we
were
using
grpc
in
an
in
conventional
way
that
people
weren't
used
to
because
we
were
passing
it
ips.
We
were
using
some
weird
settings
to
support
that.
A
So
at
one
point
we
switched
to
using
like
the
normal
way
of
using
grpc,
where
you
just
pass
it
a
hostname,
and
when
you
do
that
change,
one
thing
you
have
to
change
is
the
grpc
load
balancing
policy,
so
the
default
load.
Balancing
policy
in
grpc
is
called
pick
first
and
what
it
will
do
is
it'll
just
pick
one
ip
and
use
that
form
a
connection
to
it
and
use
it,
and
when
that
connection
fails,
it'll
pick
another
one.
A
Oh
okay,
it's
obvious,
like
we
were
using
pick
first
and
we
switched
to
round
robin
so
like
we
all
the
alerting
engines
were
talking
to
one
metric
service,
and
then
they
started
talking
to
all
of
them.
But
that
actually
wasn't
the
case.
It
wasn't
that
simple
remember.
We
are
still
doing
cloud
inside
load,
balancing
in
either
case
in
either
case
we
still
had
one
grpc
channel
with
one
ip.
The
only
thing
that
we
actually
ended
up
changing
was
the
layer
in
which
the
dns
resolution
is
taking
place,
which
was
super
weird.
A
So
we
dug
a
bit
more.
We
read
the
grpc
code
and
we
realized
that
pick
first
and
round
robin
actually
have
a
very
subtle
but
important
difference
in
the
way
that
they
handle
connection
failures.
So
they
manage
connections
in
the
background
for
you,
so
you
don't
have
to
have
that
in
your
application
code.
But
in
pick
first,
when
that
disconnection,
when
a
when
a
connection
is
severed,
it
actually
doesn't
try
to
reconnect
until
you
try
to
use
it.
A
It
does
on-demand
reconnection,
and
this
means
that
like
when
you
make
a
request,
it
will
block
the
request,
as
it
tries
to
make
the
reconnect
in
pick
first
or
in
round
robin.
Rather
they
tried
to
be
a
bit
smarter
here.
They
said:
wouldn't
it
be
cool
if,
in
a
background
thread
it
automatically
tried
to
reconnect
so
that
by
the
time
your
application
went
to
go
use
that
connection
it
was
ready
and
one
thing
that
we
did
when
we
were
using
pick
first
load
balancing
is
like
many
years
ago.
A
This
was
just
happening
in
background
threads
everywhere
for
every
single
connection
that
it
had
and
we
did
some
math,
and
so
we
had
thousands
of
alerting
engines
that
were
all
trying
to
reconnect
to
each
metric
service
pod
every
300
milliseconds,
which
meant
that
we
were
sending
tens
of
thousands
of
sins
per.
Second,
we
were
just
sin
flooding
ourselves,
and
this
explains
those
huge
spikes
that
we
are
seeing
in
those
graphs,
and
so
the
fix
here
was
actually
just
deleting
a
few
lines
of
configuration.
A
Surprisingly,
so
we
just
use
the
default
reconnection
settings
in
grpc
where
it
tries
to
reconnect
like
on
the
order
of
seconds,
not
milliseconds,
and
we
just
sort
of
like
let
it
do
its
job
as
it
was
intended,
and
it
turns
out
that
was
actually
the
core
problem
here
that
stopped
our
syn
floods.
So
here
you
can
see
we
did
a
rollout
here.
A
B
B
More
seriously,
I
mean
we,
we
use
very
powerful
abstractions.
We
use
cloud
networking
we
use
kubernetes
networkings.
These
abstractions
are
very
powerful
and
magical
in
a
sense
where,
when
they
work
and
honestly
most
of
them,
they
work
perfectly,
but
when
they
don't
and
they
leak
the
underlying
complexity
to
you,
you
have
to
dive
deep
into
it
and
it's
sometimes
pretty
pretty
difficult.
B
Well,
grpc
setup
can
be
complex
and,
as
and
and
we've
seen
that
making
changes
can
be
dangerous.
We
noticed
that
very
low
level
instruments,
very
low
level,
metrics
and
and
and
logs
can
be
extremely
interesting.
So
here
the
ena
metrics
and
the
vpc
flow
logs
actually
helped
us
make
sense
of
what
we're
seeing
as
elijah
was
just
saying.
This
required
a
very
complex
team
efforts.
It
took
us
weeks
to
to
fix,
and
so
we
also
we're
only
two
on
stage
today,
but
wendell
madden
knife
also
helped
quite
a
lot.
B
So,
as
you
can
imagine,
debugging
this
incident
was
was
long
and
painful,
but
we
really
learned
a
lot
and
that's
also
why
we're
here
sharing
it,
because
we
believe
that
many
of
you
could
be
could
be
interested
in
by
this
and
we're
just
over
time.
So
we
won't
be
able
to
take
questions,
but
we're
going
to
stick
around
for
sometimes,
if
you
want,
if
you're
interested
in
that
kind
of
fun,
debugging
issues
we're,
definitely
hiring,
and
we
have
a
lot
of
other
subtle
problems
like
this
one.