►
From YouTube: Switching the Engine (DNS) in Kubernetes: Benchmarks ... - Michael Grosser & Jake Sanders
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Switching the Engine (DNS) in Kubernetes: Benchmarks and Possibilities - Michael Grosser, Okkur Labs & Jake Sanders, Google
DNS is one of the core components making Kubernetes run. It’s essential for most services and service discovery. It’s critical, underappreciated and overlooked at the same time. With the recent switch from Kube-DNS to CoreDNS as the main engine for DNS we wanted to take a retrospective look at the improvements and a future look at new possibilities. In this talk we’ll discuss common DNS issues such as UDP packet loss and resulting tail latency issues, compare the performance and reliability of Kube-DNS and CoreDNS, plus spec out and test possible iterative setups with advanced CoreDNS features such as local Daemonset caching, TCP support, prefetching, DNS over GRPC and GRPC based watches.
To learn more: https://sched.co/GrWu
A
Welcome
to
switching
the
engine,
getting
some
insights
into
what
makes
Cuban
is
run
or
lease
from
the
DNS
perspective,
or
else
I
mean
you
all
want
to
go
to
the
after
party.
So
we
won't
dive
into
everything.
So,
who
are
we
I'm
Michael
I'm,
a
maintainer
at
Cardenas,
I'm,
running
an
open
source
research,
lab
called
collapse
and
also
support
companies
via
wrecker
solutions,
focusing
on
kubernetes
who
cloud
Cortinas
and
with
things
such
as
speed
based
DNS
and.
A
So,
let's
dive
into
our
agender,
we
first
wanted
to
give
a
bit
of
background.
I
know
there.
A
lot
of
are
a
few
talk,
sorry
around
Cardenas,
but
we
want
to
set
the
stage
on
what
we
are
actually
talking
about.
Then
we
will
dive
into
a
new
cluster
add-on,
which
is
called
node
local,
which
will
cache
a
bit
more
inside
the
class
err.
Then
we
have
a
few
benchmarks.
A
We
go
step
by
step,
so
we
will
look
into
some
local
benchmarks
and
then
into
cluster
benchmarks,
and
then
we
have
one
example
of
occasion
which
is
anecdotal
on
what
might
happen
with
a
switch
and
in
general,
what
you
can
do
with
an
actual
application
in
a
real
world
scenario.
The
next
one
is
basically
the
outlook.
So
what
will
happen
with
cárdenas
and
basically
Cardenas
and
Cuban
Eddy,
as
in
the
future?
A
So
basically
the
background.
The
primary
use
for
DNS
with
Cuban
Eddie's
is
services
careers.
You
all
know
it's
basically
gluing
everything
together.
So
in
there's
the
internal
use
and
the
external
use,
we
will
focus
a
bit
on
the
external
use
case,
especially
on
our
end-to-end
tests,
but
everybody
knows
that
DNS
is
the
address
book
of
the
internet.
So
it's
connected
to
database
requests.
Remote
file,
send
email
and
sell
it
bad.
So,
let's
dive
a
bit
deeper
into
the
history,
so
we
had
cube
DNS
before
and
just
to
prevent
the
confusion.
A
The
service,
the
DNS
service
inside
of
communities,
will
still
be
called
cube
Dinah's,
but
he
will
be
switched
out
with
cárdenas.
So
what
is
cube
DNS
or
what
was
the
old
way
of
doing
things?
It
was
a
bundle
made
up
of
DNS
mask
which
is
used
as
a
resolver
locally
on
a
lot
of
Linux
distributions
and
as
a
cache
Tipitina's
was
using
it
as
a
cache
and
bundle.
This
together
was
Skye
DNS.
It
is
a
go
written,
DNS
server
and
was
written
by
meek
and
a
few
other
people.
A
So
it's
a
great
foundation
from
a
design
perspective.
It
builds
on
a
good
ecosystem.
It's
building
up
on
caddy,
which
is
a
go
written
web
server.
It
has
a
lot
of
extensibility
due
to
being
based
on
a
middleware
and
based
system,
so
the
cache,
for
example,
is
one
plug
in
we
have
metrics
Prometheus
metrics
built
in
and
also
the
Cuban
Eddie's.
Api
is
just
one
plug-in
of
many,
so
why
was
it
built
and
build
again
by
meek,
meek?
Even
so
anecdotally?
A
He
wanted
to
run
wanted
to
have
more
metrics
inside
of
his
DNS
server,
which
is
which
was
at
that
time
a
bind
based
one.
So
adding
a
new
feature
to
bind
seemed
to
be
really
really
difficult.
So
instead
he
started
a
new
dns
server
from
scratch.
I
mean
that
what
do
you
do
when
you
want
to
have
nice-looking
stats
right?
We've.
A
So,
let's
dive
into
the
different
protocols
that
are
supported
so
Cardenas
doesn't
just
support
the
base.
Dns
protocols,
one
is
basically
UDP,
was
what
the
first
iteration
it's
lossy
and
that
basically
gives
a
bit
of
problems
down
the
road.
The
next
iteration
was
TCP.
It
has
connection
it's
a
much
better
protocol
for
reliability
reasons
and
we
have
two
modern
ones
that
are
actually
specific
specified
already.
So
they
have
a
specification
and
they're
used
in
production.
A
B
In
addition
to
becoming
the
default
cluster
name
server
in
113,
another
feature
related
to
core
DNS
was
also
introduced.
This
time
it
is
actually
no
local
caching
which
will
go
on
to
solve
many
problems
for
you
guys.
Hopefully.
So
if
you
leave
here
feeling
like
feeling
like
your
days
made
please.thank
Pahlavi
through
ramesh,
she
did
the
job
of
going
through
the
whole
kept
process
iterating
on
the
design,
merging
the
pull
requests
getting
this
to
you
guys
it's
all
about
community,
and
she
is
one
of
them.
So
just
some
background
info
on
the
problem
itself.
B
Dns
is
really
a
child
of
the
80s
and
70s
and
80s,
and
there
are
lots
of
things
that
date
it
as
such
I'd
like
to
take
you
on
a
journey
through
time,
back
back
to
the
time
when
the
internet
was
just
a
bunch
of
telephones,
kind
of
yelling
at
each
other,
the
phones
kind
of
sucked,
most
of
the
things
that
you
talk
about
weird
stuff,
like
TPS
reports
and
nuclear
codes.
You
know
nothing
too
interesting
and
there
weren't
really
any
sort
of
high
in
high
performance.
B
Speaking
of
the
70s
and
80s
UNIX
and
Lib
C
followed
the
recommendations
of
the
specification
very
faithfully
and
the
rest
is
kind
of
history.
You
know
many
clients
will
still
directly
or
indirectly
depend
on
Lib
C
or
its
behavior
it'll
depend
on
the
specification
or
defaults
of
the
resolver
configuration,
so
these
decisions
have
really
propagated
to
the
present
day,
but
even
when
DNS
clients
have
the
ability
to
force
DNS
over
TCP,
that
configuration
may
not
be
easily
configurable
at
runtime,
so
practically
speaking,
many
applications
are
just
going
to
be
susceptible
to
these.
B
You
know
timeouts
on
failed
lookups
and
especially
unlucky
victims
might
have
to
retry
over
and
over
and
over
and
over
and
just
you
know
very
bad
day
all
agencies.
These
matters
are
severely
aggravated
by
the
way
that
kubernetes
services
are
implemented,
including
a
cluster
DNS
itself.
When
cluster
IPS
are
resolved
to
a
specific
pod
IP,
the
net
filter
Suites
contract
table
becomes
a
critical
bottleneck
and
single
point
of
failure
for
lookups
directed
to
the
cluster
DNS.
You
know
those
that
you
know
are
clustered
local
addresses.
B
This
is
partly
due
to
the
fact
that
Linux
installations
normally
limit
the
maximum
number
of
entries
in
this
contract
table
to
something
that
you
know
might
be
classically
reasonable.
Back
when
I
make
a
gigabyte.
Was
you
know
unthinkable?
Usually
it
you
know
to
the
sixteen
and
then
once
that
limit
is
reach
extra
packets
or
just
simply
dropped
on
the
floor.
B
So
if
you
ever
like,
if
you
ever
run
into
this
situation,
you're
definitely
gonna
want
to
look
in
your
kernel
logs
and
see.
If
you
have
one
of
these
net
filter
table
fools
packet
loss.
That
kind
of
thing
also
unlike
TCP
UDP,
is
connectionless
and
doesn't
explicitly
close
a
connection
which
means
that
contract
entries
for
you
to
be
connections
get
assisted
for
persisted
for
way
longer
than
they're
normally
useful.
B
This
causes
contention
if
there
are
simultaneous
requests
from
the
same
IP
import,
resulting
in
one
or
more
packets
being
dropped
that
way
as
well.
Even
you
know,
independent
of
the
table
itself
being
full.
Unfortunately,
many
clients
will
also
paralyze
the
DNS
requests
for
separate
record
types
such
as
those
for
ipv4
and
ipv6
and
then,
finally,
if
you
have
a
bunch
of
entries
configured
in
your
search
path,
as
is
often
the
case
in
our
pods
failed
lookups
will
keep
looking
up
down
every
entry
in
the
search
path.
B
What
this
all
means
practically
is
that
you,
as
a
service
owner,
are
left
holding
the
bag
for
OPIC
failures
that
often
completely
defined
it
bugging
drop
packets
in
DNS
timeouts,
usually
happen.
Wait
at
the
bottom
of
the
stack.
You
know
in
the
kernel
itself
and
are
often
obscured
by
higher
level
programming
abstractions.
B
If
you
don't
really
know
about
these
default
behaviors
and
implementation
details
you're
liable
to
end
up
on
a
bunch
of
wild-goose
chases,
and
you
know,
setting
off
pagers,
and
you
know
just
not
having
a
good
time,
but
I'd
also
like
to
know
that
this
isn't
a
problem,
that's
specific
to
Cooper
Nettie's.
It's
actually
existed
in
the
wild
for
quite
a
while.
You
know
very
often
for
other
previous
implementations
of
dns,
for
exactly
the
same
reasons
and
I
guess
there's
you
know
now
is
the
time,
but
I
have
good
news,
guys
and
gals
and
non-binary
pals.
B
B
So
here's
just
like
a
diagram
of
how
the
how
the
request
flow
works
with
a
node
liquid
cash
request
from
a
client
pod
will
go
into
the
cache
and
either
cache
hit
will
happen
or
cache
miss
and
on
the
event
of
a
cache
miss
the
request
will
either
be
routed
to
the
cluster
DNS
or
to
your
favorite
public
name
server
that
is
specified
in
the
resolver
configuration
you
know
in
whatever
way
you
have
done
so,
but
just
as
a
sanity
check
just
to
prove
that
this
actually
works.
Let's
go
through
a
few
benchmarks.
B
Just
for
this.
First
one
I
created
this
low
generator
that
basically
looked
up
normal
a
records
for
like
the
10,000
most
popular
website.
It's
kind
of
a
toy
application,
but
I
also
used
the
TC
utility
in
order
to
drop
packets
at
a
certain
rate
and
just
like,
observe
the
effect
that
it
had
on
on
low
cups
just
between
localhost
and
the
public
name
server.
B
As
you
can
see,
UDP
is
just
like
slightly
more
performant
than
TCP
in
ideal
conditions,
but
as
soon
as
a
network
at
Network
shop
starts
happening,
UDP
rapidly
falls
behind
and
you
rapidly
start
seeing
you
know
longer:
average
latencies
and
huge
tail
latency
and
if
you're
a
service
owner
that
cares
about
p99,
you
know
you
to
have
those
nines.
This
is
gonna
ruin
your
day
because
you
know
you're
just
not
going
to
get
them.
You
know,
as
you
can
see,
if
you're,
if
you
only
need
99
percent
availability,
then
you're.
B
Okay,
until
you
know,
one
percent
of
packets
just
go
missing,
I
guess.
But
additionally,
even
if
even
if
you
aren't
having
a
ton
of
packet
loss,
you
can
see,
the
TCP
connections
will
generally
have
more
consistently
lower
latency
x'
than
UDP,
which
is
just
sort
of
a
bit
more
unpredictable
on
to
a
cluster
DNS
benchmarking-
and
this
is
another
huge
thanks.
Christow
have
her
from
infobox.
Who
actually
did
this
work
when
core
DNS
was
made
available
and
111
in
order
to
verify
that
it
was
a
good
solution
for
the
cluster
DNS
itself?
B
What
they
did
what
he
did
was
he
ran
the
numerous
kubernetes
scaling
benchmarks
that
happen
when
you
integrate
a
PR.
You
know
with
flat,
with
several
thousand
cholesterol
esters
in
air,
pods
and
services,
and
then
also
added
the
memory
Delta
that
was
observed
when
hitting
each
both
core
DNS
and
cube
DNS
at
their
highest
QPS.
B
So,
in
this
regard,
it's
just
like
a
strictly
better
alternative,
and
not
only
is
not
only
does
it
grow
at
a
lower
rate,
it
only
requires
a
single
pod
rather
than
the
three
or
more
pods
that
cube
DNS
has
are
requires
currently,
additionally,
what
the
amount
of
requests
that
core
DNS
can
actually
serve
is
roughly
comparable
to
cube
d
n
s
4
cluster
internal
lookups.
It's
just
like
10%,
less
QPS,
you
know
available
for
cluster
internal,
but
similar
Layton
sees,
however,
for
cluster
external
lookups.
A
Okay,
so
let's
jump
into
the
application,
so
local
benchmarks
and
cluster
benchmarks
in
terms
are
really
interesting,
but
I
mean
totally
all
care
about
is
the
application
itself.
So
how
would
that
affect
a
specific
application
to
basically
set
the
stage
for
real-world
benchmark,
which
needs
a
lot
of
traffic,
so
we
needed
to
find
a
nice
project
or
application
that
actually
needs
a
lot
of
requests.
A
So,
let's
look
into
what
we
use,
so
our
application
is
text
direct
which
is
dns-based
redirect
engine
which
stores,
basically,
all
the
data
for
each
redirect
in
a
text
record
in
your
name
server
or
in
your
domain,
registrar
and
basically
I
mean
we'll
see
you
to
keep
the
control
of
your
entry
point
and
your
data.
What
that
basically
makes
it
makes
happen
is
that
the
instance
that
does
the
redirect
is
completely
stateless.
On
the
other
hand,
for
each
request,
there
will
also
be
a
request
to
DNS
to
fetch
the
data.
A
This
can
be
cached,
so
it's
not
bad
problematic,
but
it
gives
us
the
ability
to
basically
get
a
lot
of
DNS
requests
from
one
single
application,
which
also
sort
of
is
more
in
the
real-world
application
space.
So
text
direct
is
open-source
based
it's
based
on
caddy,
so
it's
just
a
simple,
simple
plugin
and
we're
trying
to
enable
you
to
run
your
own
redirect
or
vanity
well
Shana.
So
let's
look
into
how
the
request
flow
for
texture.
It
works
so
we're
a
Cuban,
Aires
fans
humanities.
A
We
look
at
Cuban
and
said
open
source
software
drugs
because
we
know
that's
true,
so
we
enter
that
in
the
browser.
As
always,
the
browser
would
look
up,
okay,
which
server
do
we
actually
need
to
contact
for
that?
So
it
will
look
up
an
A
or
cname
record
for
combination
or
open
source
software
or
drugs,
and
then
the
request
comes
into
the
instance,
the
techs
erect
instance,
which
is
by
default,
as
we
said
completely
stateless.
So
where
does
the
data
come
from?
It
actually
uses
the
model
of
the
underscore
which
is
used
for
services.
A
Usually
so
it
doesn't
collide
whether
I
can
redirect
sub-domain
or
something.
So
this
day
the
instance
looks
up
a
text
record
on
underscore
underscore
redirect
that
opens
the
cuban
aerosol
open-source
software
or
drugs,
and
we
see
an
example
of
how
such
a
record
would
look,
so
we
basically
have
a
version
to
make
it
backwards,
comparable,
the
type
which
is
host.
So
that's
just
the
easiest
type
that
might
happen.
A
It's
usually
used
to
redirect
the
root
of
such
as
examples
that
come
to
some
septum,
a
which
can
then
be
linked
to
your
cname,
so
that
some
providers
solve
the
differently
which
isn't
conforming
with
a
standard.
It's
sometimes
called
like
an
a
named
record
where
they
do
that
on
the
fly,
but
we're
conformed
with
the
text,
basically
with
a
DNS
standard.
So
now
that
everything
happened,
it
will
be
a
transparent
redirect
to
Cuba
meracydia.
A
So,
let's
set
that,
let's
see
what
the
different
setups
are,
so
we
want
to
test
different
things,
so
we
were
the
first
one
to
compare:
what's
the
performance
or
real-world
performance
for
kube
DNS,
so
that
we
have
the
old
data,
basically
so
cooped
in
s.
What
we
do
is
basically
Texas
erect
is
we're
just
running
in
the
cluster,
so
we
used
all
the
different
cluster
services
so
we're
using
a
service
IP.
We
have
the
overhead
of
a
part.
We
have
all
the
over
hats.
A
You
would
have
running
an
application
inside
of
Cuba
Nerys,
so
that
might
make
it
much
more
usable
for
you
as
like
an
anecdotal
benchmark.
So
what
happens
before
or
what
happened
before
with
Cuba
DNS,
the
application
would
send
a
UDP
request
over
the
wire,
so
it
might
run
into
packet,
drops
tier
cube
DNS,
which
then
does
a
request
to
the
upstream
DNS
server.
In
our
case,
we
just
use
the
Google
public
one,
but
usually
it
depends
on
your
cloud
provider.
A
It
could
be
at
CD
resolve
that
gets
us
or
your
company
dns
resolver,
whatever
fits
so
the
new
way
of
doing
things
as
we've
learned,
is
using
cárdenas
inside
your
cluster,
so
we're
still
sending
UDP
over
the
wire.
So
we
still
have
a
lossy
connection.
It
upgrades
and
it
has
a
cache.
So
that's
similar,
but
we
learn
it
has
better
performance
and
we
will
see
that
in
the
bench
fronts
later.
Well,
it
gets
interesting.
Is
the
cluster
read
on
node
local?
So
we
want
to
test
a
few
things
here.
A
We
are
basically
sending
UDP,
so
your
application
still
doesn't
need
to
know
about
TCP
base,
DNS
or
any
upgrade
of
protocols.
It
can
basically
be
a
legacy
application
only
supporting
DNS
on
UDP,
but
the
difference
here
is
it's
running
as
a
daemon
set
on
your
specific
host.
So
you
P
only
goes
over
local
host,
which
is
usually
not
as
lossy
as
your
cloud
provider.
A
So
the
there
are
two
different
modes
to
that:
node
local
can
cache
or
it
can't
or
it
doesn't
cache.
So
in
this
case
we
test
both,
but
what
it
always
does.
Is
it
upgrades
the
connection
to
TCP
and
the
good
thing
about
having
or
controlling
both
the
client
and
the
server
which
in
both
cases
is
cárdenas.
The
node
look
was
just
a
bit
more
minimalistic
is
that
we
can
you
reuse
connections.
So,
instead
of
opening
one
connection
per
request,
we
will
just
open
a
few
and
reuse
these,
so
the
TCP
overhead
is
almost
minimal.
A
So
what
else
can
we
do
so?
What
happens
in
big
clusters?
Is
you
have
your
usual
5,000
server
cluster
as
you
are
running?
Hopefully?
So
what
happens
is
that
you
have
two
or
three
nodes
that
actually
host
the
core
Dinah's
instance
and
all
the
external
requests
go
to
these
these
cluster
servers
and
then
spam,
your
upstream
DNS
server
from
one
or
two
single,
a
piece
which
could
result
which
actually
happened.
In
our
case,
we
spammed
Google
Public
DNS
as
much
that
they
basically
blocked
us
so
that
results
really
in
a
lot
of
drop
packets.
A
A
So
we
had
a
really
reliable
network,
so
we
ran
on
gke
just
as
a
side
note,
which
we
tried
to
induce
a
lot
like
with
running
through
multiple
services
or
something,
but
there
were
just
not
enough
UDP
to
pack
droplets
drop
packets,
so
we
had
a
small
gremlin
that
might
actually
happen
that
it's
in
your
network
already
or
in
your
cloud
provider
and
it's
dropping
packets.
Happily
without
actually
you
having
to
deploy
something
extra.
A
So
now
we
know
what
the
sample
setup
was,
but
what's
the
results
like
we
want
to
know
what
were
the
numbers.
So,
as
you
can
see
here,
we
have
six
different
test.
Setups.
The
first
one
in
blue
is
the
UDP
over
the
wire
with
cube
DNS.
So
as
decent
performance
and
not
a
lot
of
Taylor,
we
will
have
a
tell
latency
graph
later,
which
basically
gives
you
a
bit
better
understanding
of
what
talents
you
might
make
or
might
happen
to
your
application.
A
The
second
one
in
red
is
basically
the
new
set
up
for
you,
which
is
using
Cardenas
as
the
central
DNS
instance,
and
we
see
a
lot
better
performance.
So
we
are
already
upgrading
your
DNS
by
default
to
be
more
performant
and
more
reliable,
and
now
we
enter
the
node
local
set
ups.
So
what
is
actually
deployed
when
you
use
the
cluster?
Note
note
local
cluster
ad
on
right
now
with
1.13
it
will
cache
automatically.
A
It
will
send
all
internal
traffic
to
coordinates
and
then
will
spread
up
the
load
of
all
external
class
cluster
requests
right
on
your
notes,
so
it
will
use
the
configuration
on
at
cVTC.
Slash
hosts
results.
Sorry
slash
resolve
to
basically
figure
out
which
upstream
provider
to
use
in
the
different
cloud
providers.
That's
usually
like
an
internal
cluster
DNS.
So
that's
pretty
fast,
so
we
can
see
we're
adding
with
node
local
we're,
adding
an
different
new
application.
A
A
So
with
the
cache
that
improves
dramatically
and
if
we
now
look
at
the
orange
one,
which
is
adding
a
cache
and
spreading
out
the
load.
We
are
much
much
faster
overall,
interesting
enough
because
we're
looking
at
nothing
all
right
so
because
we're
looking
at
the
the
performance
UDP
drop,
drop
packets
won't
yet
show
up.
So
our
graph
for
the
chaos
doesn't
really
show
anything
interesting.
But
if
we
look
at
the
tag
latency,
we
can
see
the
difference
so
the
teal
color
is
basically
our
carriers
drop
packets.
A
So
we
can
see
that
a
lot
of
tail
agencies
at
it
and
we
can
completely
remove
that
tail
latency
by
using
a
local
cash,
either
node,
local
or
just
upgrading
the
connection.
So
if
you
have
problems
with
that,
that
might
be
the
solution
for
you
and
we
can
see
and
talk
bad.
What
happens
with
that
out
of
the
talk
and
let's
dive
into
what
that
basically
means
not
just
from
a
numbers
perspective
but
as
what
we
learned
from
basically
this
whole
exercise.
A
A
So
a
nice
thing
that
also
came
up
in
the
benchmarks
was
that
the
search
path,
as
we
said,
amplifies
a
lot
of
bad
behavior.
So
our
application
connects
to
a
DNS
server
and
it
was
speaking
out
to
the
DNS
server
of
a
local
host
name.
So
we
didn't
use
the
local
IP
address.
We
used
local
host.
What
happens
was
with
go
and
Alpine
by
default.
It
doesn't
look
at
at
CDE
hosts
and
therefore
doesn't
resolve
the
local
host
on
within
the
pot.
What
happens
is
on
each
request?
A
A
So
what
happened
was
that
locos
wasn't
resolved
in
the
pot
so
require
basically
tried
the
whole
search
path.
So
it
wasn't
just
one
request
that
was
failing.
It
was
six
different
ones
that
were
failing
and
in
our
case
we
were
still
using
cube
DNS
as
the
central
cluster
server,
which
doesn't
cache
negative
lookups.
So
basically
we
were
hitting
uncashed
six
million
queries
on
the
central
one.
With
that
resolved
we
basically
learned
don't
use
localhost
as
a
name
or
if
you
do
and
use
generally
DNS
lookups
always
add
a
dot
at
the
end.
A
If
you
know
it's
an
absolute
one,
because
you
will
never
go
into
the
search
path,
so
basically
the
negative
caching
is
fixed
by
corniness,
because
we
now
actually
cache
negative
responses.
So
that
won't
happen
that
extremely
so,
what's
the
outlook.
So
we
talked
about
the
history.
We
talked
about
the
really
nitty-gritty
of
results
and
benchmarks,
but
what
might
be
next
so
G
RPC,
which
is
hopefully
native.
A
You
had
running
cárdenas
or
cube
DNS,
and
things
like
that
want
to
try
out
at
you
might
be
happy
to
try
out
such
as
outer
path
and
stuff
like
that.
So,
thank
you
and
the
special
stands
to
my
team
from
collapse,
which
helped
me
with
a
lot
of
the
performance
benchmarks
and
the
special
shout
out
to
Aaron.
Thank
you.
A
So
relying
on
something
being
in
the
kernel
when
you
have
virtualized
stuff
on
top
is
problematic,
especially
if
you
consider
not
just
having
containers
which
share
the
kernel
but
having
the
MS
running,
for
example,
which
don't
share
the
kernel,
then
you're,
for
example.
Out
of
luck.
The
other
thing
is
that
and
a
switch,
for
example,
doesn't
work
with
Alpine
because
they
don't
care
about
the
file.
So
that
is
another
problem
that
might
be
happening
and
we
have
a
lot
more
control
of
our
iterating
on
protocols,
doing
cache
and
adding
a
lot
more
features.
A
A
C
B
Mean
we've
also:
we've
published
the
code
to
one
of
my
github
repos.
So
if
you
guys
want
to
go
and
try
and
reproduce
you
yourselves,
I
mean
obviously
you're
gonna
have
to
want
to
spend
the
credits
on
gke.
You
know
by
all
means,
but
like
essentially,
what
we're
doing
is
we're
just
we're
trying
to
replicate
a
master,
we're
having
the
like
we're
splitting
up
a
fake
master
that
has
our
DNS
of
choice
in
there.
So
hopefully,
the
actual,
like
cluster
version,
wouldn't
actually
affect
the
results
of
the
test.
Yeah
centrally
I.
A
We
have
to
differentiate
so
crystal
hair,
for
example,
he
already
did
use
the
CI
system,
so
the
cluster
scaling
test,
you
get
the
resource
usage
and
basically
be
the
effect
already
running.
We
could
do
that
all
the
time.
Shh
I
think
that
might
actually
be
plan.
I
have
no
complete
idea
on
that.
Our
anecdotal
end-to-end
test
or
application
benchmark.
A
Isn't
that
helpful
running
on
a
persistent
basis,
because
it's
really
opinionated
on
a
specific
set
up,
so
we
ran,
for
example,
ten
thousand
different
domains
and
therefore
didn't
have
a
lot
of
caching
involved
even
was
with
a
million
queries
being
run.
So,
if
you're,
just
having
like
a
hot
path
of
like
ten
domains,
that
need
be
cached
and
that's
always
in
your
cache,
you
will
see
completely
different
performance,
so
that
I
would
feel
that
wouldn't
make
sense
in
that
regard.
Any
other
questions:
let's
do
the
mic.
First,
yeah.
D
I
think
one
of
the
things
you
mentioned
during
the
talk
was
you
had
a
hard
time:
detecting
drop
packets
in
the
cloud,
and
my
feeling
is
that
that's
just
gonna
be
the
norm.
I
mean
UDP
packet
loss
in
in
a
LAN.
Isn't
going
to
be
that
high?
So
do
you
think
you
guys
are
trying
to
solve
a
problem
that
doesn't
exist
or
is
minimal,
at
least
in
the
kubernetes
cluster
I.
B
Mean
there's
I
could
also
just
like
send
out
a
bunch
of
links
to
you
know
innumerable
like
stack
overflow,
server,
overflow,
github
issues
and
kubernetes
and
and
other
DNS
things
like
this
is
a.
This
is
a
relatively
common
issue
that
you
know
does
get
attention
and
there
is.
There
is
interested
in
the
community
from
like
for
this
specific
kind
of
solution
for
kubernetes
yeah.
A
It's
totally
second
bad,
so
we
have
their
production
classes
that
run
into
that
problem
multiple
times
and
all
over.
So
it
doesn't
matter
on
cluster
size.
It
doesn't
matter
on
cloud.
There
are
some
cloud
providers
that
drop
less
packets
and
basically
a
completely
reliable,
but
there
are
also
on
Prem
settings
or,
if
you
don't
configure
any
TCP
upgrades
and
you're
actually
sending
the
DNS
requests
over
the
Internet.
The
average
uep
drop
rate
is
up
from
like
1%
to
like
up
to
15%,
depending
where
the
upstream
dns
servers.
B
B
I
I
was
made
aware
that
there
at
least
a
couple
patches
have
gone
into
or
either
gone
into
the
kernel
or
are
pending
merging
into
the
kernel
that
solve
at
least
a
couple
of
these
data
races
that
would
result
and
drop
back,
but
that
has
not
launched
yet
and
I
literally
found
out
about
it
like
two
days
ago.
So
I,
don't
really
I,
didn't
have
really
enough
time
to
you
know
research
in-depth
at
all
but
yeah.
So
it's
not
just
like
the
actual
network
itself.
B
E
Just
before
I
have
my
question
about
the
nation
or
the
local
network:
there
is
no
heat,
eep-eep
lots
of
pockets,
I'm
published
book.
Oh,
we
run
bare
metal
in
around
50,000
systems
across
many
data.
Centers,
who
have
like
huge
number
of
switches
and
I
can
tell
you
from
our
experience.
There
is
a
UDP
packet
loss
on
a
local
network
and
you,
the
netbook,
as
you
said,
is
not
very
numerous
reality
and
my
question
I
mean
listen
to
know
what
was
the
code,
their
major
cause.
You
notice
a
latency
up
to
five
seconds.
E
B
So,
basically,
just
in
the
sort
of
toy
benchmarks
that
I
had
from
local
host
to
the
public
resolver,
this
mean
latency
was
entirely
tail
latency.
If
you
I,
actually
I
sort
of
debated,
whether
or
not
to
include
the
median
latency
in
here,
and
you
would
see
that
the
median
latency
is
like
right
around
38
and
37
for
each
one.
You
know
with
UDP
being
like
a
fraction
of
a
millisecond
faster,
so
it
didn't
really
change
that
yeah
it.
All
of
this
mean
latency
is
the
size
of
that
five
millisecond
bucket
yeah.