►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Kubernetes DNS Horror Stories (And How to Avoid Them) - Laurent Bernaille, Datadog
DNS is one of the Kubernetes core systems and can quickly become a source of issues when you’re running clusters at scale. For over a year at Datadog, we’ve run Kubernetes clusters with thousands of nodes that host workloads generating tens of thousands of DNS queries per second. It wasn’t easy to build an architecture able to handle this load, and we’ve had our share of problems along the way. This talk starts with a presentation of how Kubernetes DNS works. It then dives into the challenges we’ve faced, which span a variety of topics related to load, connection tracking, upstream servers, rolling updates, resolver implementations, and performance. We then show how our DNS architecture evolved over time to address or mitigate these problems. Finally, we share our solutions for detecting these problems before they happen—and identifying misbehaving clients.
https://sched.co/Zepr
A
We're
currently
serving
200
000
dna
squares
per
second,
which
is
which
is
quite
loud
and
challenging
to
do
well
here,
is
a
quick
outline
of
what
we're
going
to
discuss
today.
First,
we're
going
to
discuss
about
how
dns
works
in
currencies
in
general
then
about
global
challenges.
People
face
usually
when
they
run
dns
and
kubernetes,
then
I'll
share
some
fun
stories,
depending
on
your
definition
of
fun,
about
dns
and
finally,
I'd
end
with
what
we
do
now,
so
the
genetic
in
kubernetes.
A
So
here
is
how
it
works.
So
when
you
start
a
bird,
the
cubette
is
going
to
inject
the
dns
configuration
inside
the
containers
and
the
way
the
bit
will
do
that
will
do.
This
is
based
on
its
configuration,
where
you
give
it
the
gns
service
to
to
use
to
to
to
give
containers,
dns
access,
and
this
will
translate
into
a
resolve.com
file
inside
containers,
with
a
list
of
search
domain
which
makes
discovering
services
in
the
cluster
and
outside
easy
and,
of
course,
the
name
server
to
use.
A
When
the
container
wants
to
access
dns,
it's
going
to
send
a
query
to
the
dns
service
in
the
cluster
and
the
proxy
you
use
whether
it's
iptables
on
ipvs
is
going
to
load
balance.
These
queries
to
dns
pods,
which
can
either
run
cube,
dns
or
called
dns,
and
these
dns
servers
are
going
to
do
two
things.
The
first
one
is
they're
going
to
be
authority
for
the
clusters
or
for
the
cluster
domain
and
also
they're,
going
to
forward
queries
for
to
a
nutrition
provider
for
any
domain.
They
don't
know
about.
A
A
So
the
first
answer
from
the
dns
code
is
going
to
be
nx
domain
because
it
doesn't
exist.
The
resolver
is
going
to
continue
and
this
time
queries
for
points
that
metrics
that
serve.
Is
that
personal
local
by
using
the
second
search
domain
and
get
an
answer
so
less
efficient.
This
time,
because
we
did
two
queries,
but
so
it's
pretty
straightforward.
A
This
leads
to
several
problems.
The
first
one
is
latency,
of
course,
because
if
you're
acquiring
this
you're
going
to
make
12
queries
which
is
going
to
take
longer
and
also
if
any
of
this
packet
is
lost,
your
resolver
is
going
to
retry.
But
it's
going
to
wait
for
5
seconds,
because
the
default
dns
timeout
is
5
seconds
in
index.
A
A
A
A
So
we
had
this
very
nice
idea
at
the
time,
which
is
well,
let's
disable
ipv6
right,
so
we
disable
ip6
in
the
channel,
and
we
just
did
a
very
simple
resolution
for
google.com
and
you
can
see
that
in
in
this
example,
we
do
five
queries
and
we're
not
doing
any
ipv6.
So
that's
pretty
good
right.
We
wanted
to
celebrate
at
that
time,
but
look
after
every
interactive
v6
has
been
disabled
on
every
host.
A
We're
still
getting
a
lot
of
fpg6
queries,
so
we're
a
bit
sad
and
we're
trying
to
understand
what
was
happening
so
what
triggers
igd6.
So,
as
I
was
saying
before,
according
to
posix,
you
should
be
getting
ipv4
native
six
queries.
However,
the
gdp
implementation
in
linux
is
using
hints
that
are
notes
that
it's
not
supposed
to
use
in
the
posix
in
the
project,
but
it's
doing
it
because
it's
helpful
and
one
of
the
hints
used
by
tbc
is,
is
ai,
addr
info,
which
is
only
making
ipv6
queries
if
it
finds
an
ipv6
address.
A
A
So
we
we
continued
digging
and-
and
the
thing
we
found
is
that
well
we're
not
only
running
ubuntu
and
gbc,
we're
also
running
some
alpine
containers
that
are
using
bezel
as
your
lipstick
provider
and
well
look
at
this
when
you
do
this
very
simple
query
on
an
ubuntu
based
image
and
an
alpine
based
image.
In
one
case
you
get
on
the
ipv4
and
in
the
second
case
you
get
ipd
for
an
ipv6
and
the
reason
is
well.
Mother
actually
implements
the
project
specification
exactly
and
not
the
helpful
hint
that
gbt
provides.
A
A
A
So,
as
you
may
know,
go
implements
two
two
ways
to
do:
tns
you
can
have
either
navy
go
or
you
see
go
so
so
we
figured
well.
Let's
use
c
go
so
in,
in
which
case
we
will
false
go
to
use
gbmc
and
then
it's
native
implementation
and,
as
you
can
see
here,
even
when
we
use
seago
we're
also
getting
an
ipv6
query.
A
But
if
you
look
at
the
at
the
very
small
details
and
these
two
lines
in
these
two
examples-
you'll
see
that
it's
actually
different
and
that
the
results
are
behaving
differently
because
one
is
using
a
different
source
port
for
the
ipv4
and
ipv6
query.
So
it's
actually
doing
something,
but
it's
not
behaving
as
we
expect,
because
we'd
expect
gdbc
based
queries
not
to
do
ipv6.
A
We
found
that
when
you
use
c
go,
go
is
actually
specifying
hints
and
the
hints
it
specifies
are
different
from
the
default
tlipc
ones
and
what
is
very
important
here
is
addrconfig,
which
will
not
do
at
the
b6
resolution.
If
there's
no
physics
address
it's
not
in
these
hints,
so
it's
very
sad
because
in
go
it
means
you
can't
easily
remove
hp6
query
and
the
only
way
I've
found
to
do
that
is
actually
to
false,
go
to
only
query
ipv4
by
choosing
tcp
4
in
the
example
at
the
bottom.
A
A
A
So
that's
very
clever
right,
because
you're
only
doing
two
queries
now,
instead
of
10..
But
sadly
it's
it's.
It's
not
magical,
because
to
do
that,
the
codiness
autobus
implementations
needs
to
know
in
which
the
namespace,
the
bodies
just
to
be
able
to
remove
the
first
full
search
domain,
and
to
do
that.
A
Another
option
that
people
are
starting
to
to
look
into
seriously
and
that's
available
upstream,
is
to
use
not
local
dns.
So
it's
actually
a
very
simple
idea
where
you
run
a
local
dns
cache
and
on
every
node,
as
a
demon
set
and
have
all
your
containers
queries
this
local
dns
and
the
good
thing
is
well.
You
have
a
local
cache
and
queries
are
then
forwarded
upstream
to
the
dns
pods.
A
So
it's
it's
very
efficient
because
you're
not
doing
any
udp
queries-
and
you
remember
from
before
that.
Udp
queries
were
triggering
this
contract
issue
that
wasn't
great
and
also
what
you
can
do
is
you
can
bypass
the
dns
service
for
any
non-communities
domain,
which
means
you
reduce
the
load
very
a
lot
on
your
online
assembly,
which
is
which
is
good
news.
A
A
So
when
the
police,
the
container,
is
going
to
make
this
query
is
going
to
create
the
service
ip
vp
in
my
example
and
ipvs
is
going
to
translate
this
ip
to
a
pod,
ip
and,
of
course,
to
make
sure
that
next
the
following
packets
are
going
to
use
the
same
connections.
It's
going
to
create
an
ipvs
contract
entry
which
I
displayed
at
the
bottom
there.
A
So
imagine
now
that
part
a
is
deleted
because
you're
starting
a
running
update,
for
instance
or
you're,
just
reducing
the
size
of
your
deployments.
Well,
any
new
query
is
now
going
to
be
routed
to
a
new
board
right.
So
this
is
my
second
example
here,
where
the
second
query
is
sent
to
part
b.
So
everything
looks
good
right.
A
However,
what
happens
if
a
new
query
is
using
the
same
source
port?
Well,
what's
going
to
happen?
Is
the
entry
still
exists
in
the
ipgs
contract,
and
so
it
could
be
routed
to
the
backend
that
has
been
deleted
and
since
the
kernel
knows
that
this
backend
doesn't
exist,
it's
going
to
drop
it
silently
and
it's
not
a
problem
for
most
applications,
except
when
you
do
a
lot
of
queries
which
would
happen
sometimes.
A
A
A
So
what
you
can
also
do
is
reduce
the
udp
timeout
for
the
ipvs
contract
by
default.
It's
set
to
five
minutes,
which
is
a
very
long
time,
and
this
is,
you
can
now
configure
this
in
keep
starting
with
your
proxy
118
and
plus,
and
in
our
case
we
now
set
it
to
30
seconds,
which
is
which
is
better
because
your
entry
is
going
to
expire
much
faster,
and
this
means
that
the
likelihood
of
getting
a
port
collision
is
going
to
be
much
lower.
A
So,
of
course,
we
still
get
some
errors
that
are
less
and
there's
also
very
good
news
in
the
way
andrew
who
is
another
proxy
maintainer
actually
did
a
kernel
patch
to
expire
entry
in
the
contract
when
the
backend
is
deleted
directly
to
avoid
waiting
for
the
next
packet
and
avoid
getting
an
error.
So
that's
very
good
news,
thanks
andrew
so
I
know
you
came
here
for
all
the
fun
stories
and
I
mean
honestly,
it
depends
on
your
definition
of
fun,
but
now
that
I
can
talk
about
them,
it's
it's
it's
better!
A
So,
let's,
let's
talk
about
that
so
sometimes
well,
we
have
dns
issue
because
well
your
geneticism,
stable
and
a
typical
example
is
godliness
but
getting
getting
killed
because
they're
consuming
too
much
memory
and
of
course
it's
not
great
for
applications
because
well,
your
dns
spots
are
getting
killed
and
you
can
see
the
pattern
on
the
graph.
It's
it's
it's
kind
of
weird
right
because
things
were
stable
and
it
suddenly
like
went
up
pretty
fast.
A
So
what
actually
happened
here
is
an
api
server
was
restarted
and
all
the
coordinate
spots
had
to
reconnect
and
they
had
to
build
a
view,
a
full
view
of
the
cluster
in
their
indexing,
the
clients
and
to
do
that
they
had
to
process
all
the
codes
and
all
the
services,
because
at
that
time
we
were
using
photopass
and
at
startup
it
turned
out
to
require
a
lot
more
memory
to
do
this,
because
you
have
to
process
everything
in
a
few
seconds,
and
so
that
was
all
right.
That
was,
that
was
hard.
A
So
we
use
proportional
autoscaler
to
to
decide
on
the
number
of
coding
exports
we're
running,
and
so
we,
the
number
of
bud,
is
actually
proportional
to
the
size
of
the
cluster,
and
so
it
worked
completely
fine.
While
the
cluster
was
only
growing
in
size,
okay,
but
at
one
point
well,
some
applications
were
tr
was
starting
to
do
autoscale
well
and
we're
also
scaling
down,
and
so
nodes
were
disappearing
and
so-called
in
the
spots
were
removed,
and
that
was
new,
and
this
triggered
an
interesting
behavior
which
is
well.
A
So
this
is
when
we
started
to
enable
autopilot
from
the
first
time
and
what
we
did
is
we
moved
from
the
configuration
on
the
left
to
the
configuration
on
the
right,
so
configuration
on
the
left
has
two
zones,
one
for
which
coordinates
is
authoritative,
which
is
close
to
zero
and
the
second
one,
which
is
the
default
zone
where
you're
forwarding
queries
to
the
cloud
resolver
and
when
you
enable
autopass.
What
you
do
is
everything
is
in
the
default
zone
and
you
first
try
the
community
zone
and
you
use
other
paths
to
do
all
the
magic.
A
So
this
seems
like
simple
and
straightforward
right,
but
this
completely
broke
a
staging
cluster
and
well
can
you
spot
what
broke
it?
It?
It's
actually
a
very
small
challenge
in
the
in
the
before
we
did
the
challenge.
We
were
caching
queries
sent
to
the
action
provider
and
we
didn't
do
it
for
the
community
zone
because,
of
course
everything
was
already
in
memory.
So
it
doesn't.
It
didn't
really
matter,
however,
when
we
moved
to
other
paths.
A
By
doing
this
configuration
we
actually
removed.
Caching
for
the
upstream
cloud
provider
dns,
which
means
all
the
queries
were
now
sent
to
the
cloud
dns,
resolver
and
well.
It
turns
out
if
you've
been
doing
dns
on
adwords.
You
probably
know
that,
but
there's
actually
a
strict
limit
in
the
number
of
dns
queries.
You
can
do
for
any
given
network
interface
and
the
straight
limit
is
1000
packets
per
second.
If
you
go
above
this,
aws
will
just
drop
your
query.
A
Another
thing
we
tried
to
do
that
sounded
like
a
great
idea
at
the
time
was
well.
We
had
issues
with
udp,
and
so
we
decided
to
move
to
tcp.
So
in
this
very
simple
example,
we'll
say
well
when
we
do
it
when
we
send
and
forward
any
query
upstream
with
that
provider,
let's
use
tcp
it's
more
reliable.
It's
going
to
be
better
right!
Well,
except
remember
the
strict
limit
of
packets.
A
If
you
look
at
the
graph,
I
just
I
just
showed
about
contract
entries,
you
can
see
that
there's
actually
two
different
group
of
nodes
and
all
these
nodes
were
running,
coordinates
pods.
So
it's
kind
of
weird
that
these
nodes
are
so
different
because
dns
queries
were
perfectly
load
balanced
across
all
the
cards,
and
so
we
would
expect
to
see
this
be
very
balanced.
A
Some
were
running
a
kind
of
415
and
some
were
running
kind
of
5.0
and
there's
actually
a
few
candle
patches
in
5.0
that
improve
the
contract
behavior
with
udp,
exactly
for
dns
queries
actually
and
what
it
changes
is
that
country
contract
entries
for
dns
only
get
a
service
against
gta
by
default
instead
of
three
minutes,
which
means
they
expire
much
faster,
and
so
the
number
of
entries
in
the
contract
is
much
lower.
And
if
you
want
details
about
the
patches,
I
gave
the
references
on
time
and
what's
nice
with
dns?
Is
that
sometimes?
Well?
A
So
we
did
what
happened
that
time
is.
We
did
an
update
of
core
dns
on
the
cluster
and
everything
was
working
completely
fine,
except
for
a
single
application,
and
this
application
was
speedy
bouncer,
which
is
a
postgrad
proxy
and
pt
bouncer,
wasn't
able
to
connect
to
postgres,
and
everything
else
was
working
perfectly
fine,
so
well.
What
we
had
to
do
is
captured
in
a
strategy
to
try
and
understand
what
was
happening,
and
here
is
an
extract
of
the
capture
with
important
flags
passed
and
explaining
on
the
slide.
A
So
a
few
things
I
wanted,
I
want
you
to
notice
is
well.
First
of
all,
all
the
queries
are
using
the
same
source
board,
which
is
a
very
different
behavior
than
the
ones
we
were
seeing
with
gbc
or
go.
For
instance,
that's,
weird
and
also
what's
even
weirder,
is
about
on
the
right
hand,
side
where
you
can
see
this
dns
query
names
with
random
case.
A
A
So
in
that
case
the
query
was
giving
more
than
the
size
of
the
udp
packet
and,
of
course,
codiness
was
saying.
Well,
it's
truncated
and
you
should
go
to
tcp
to
get
the
full
answer.
It
turned
out.
Our
pg
bouncers
in
this
cluster
were
compiled
with
a
resolver
that
isn't
supporting
tcp
upgrade
and
was
just
ignoring
packets
when
truncated
bit
was
said
and
not
doing
anything
about
it
and
what
so
wait
it
was
working
before
we
did
the
continuous
update
right.
Well,
it
turned
out.
A
A
A
So
we
looked
at
network
monitoring
and,
of
course
you
can't
measure
udp
packet
drops
by
using
by
e
easily.
However,
what
you
can
do
is
look
at
the
tcp
connections
on
your
network
and
if
you
see
retransmits,
it
means
it
means
packets
are
getting
lost,
and
in
this
graph
you
can
see
that
there
are
a
lot
of
retransmits
happening
exactly
at
the
time
of
the
error.
A
So
it
was
really
in
the
dns
that
time
but
well.
This
was
the
first
impact
and
what
users.
So
so,
let's
see
what
we
run
now
to
to
be
safe
at
the
level.
So
what
what
we
do
is
we
run
the
local
dns,
which
I
was
talking
about
before
and
either
ipvs
for
older
cluster
or
celia
for
more
recent
clusters
and
we
use
tcp
for
any
connection
to
the
dns
service,
because
it's
a
lot
better.
However,
we
use
udp
to
connect
to
the
cloud
resolver
because
of
the
limitation.
I
was
mentioning
before.
A
And
if
you
look
at
the
configuration
of
the
containers
themselves,
well,
it's
exactly
the
same
as
before,
except
the
nameserver
we
use
is
the
ipg
bound
by
the
local
resolver
and
of
course
we
did
something
else
which
is
we
decrease
the
timeout,
because
the
default
is
five
seconds,
and
so
we
decreased
it
to
one
second.
So
if
ever
we
do
the
packet,
then
we
are
going
to
get
a
restrive
to
one
second
step:
five,
which
is
not
great
but
better.
A
A
A
A
I
mean,
even
if
you
do
a
very
good
job
with
dns
and
your
infra
you're
gonna
have
issues
sometimes,
and
you
want
your
application
to
to
react
as
well
as
possible
to
these
issues,
so
try
and
standardize
on
a
few
resolvers
only
because,
as
as
you've
seen
in
my
examples,
we
talked
about
four
different
resolvers
and
there
are
a
lot
more
out
there.
It's
very
difficult
to
understand
their
exact
behavior
and
to
optimize
for
all
of
them.
So
try
to
limit
the
number
of
results
that
you
use.
A
You
will
have,
I
mean
it
will
make
debugging
much
simpler,
also
try
and
try
and
avoid
doing
resolution
for
each
encoding
request
and
reconnecting
to
back
ends
by
either
using
long-leaf
connection
to
your
backhands
or
to
asking
dns
resolution
to
avoid,
depending
on
dns
resolution,
working
great
to
serve
synchronous,
queries
and
finally-
and
I
think
this
is
the
most
important
thing
is-
include-
tns
failure
tests
in
your
application
tests.
So
if
you
do
cause
testing
include
tns,
because
you
want
to
see
how
your
application
is
going
to
behave
when
you
lose
a
few
packets.