►
From YouTube: Rethinking the K8s DNS for the Modern Enterprise - Deepa Kalani & Venil Noronha, VMware
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Rethinking the K8s DNS for the Modern Enterprise - Deepa Kalani & Venil Noronha, VMware
The Domain Name System (DNS) is the component that provides the most vital piece of information for one to locate and communicate with services running in a Kubernetes cluster. This technology provides a set of features for name resolution, service discovery, metrics collection, query tracing, etc. However, this is only sufficient to satisfy the requirements of traditional workloads, and modern enterprises demand more. In this talk, we will discuss the state-of-the-art in the modern enterprise in the context of the Kubernetes DNS. We will present use-cases like extensive aliasing, multi-tenancy, security, etc. that stretch the capabilities of currently available DNS solutions like CoreDNS, Kube-DNS, etc. We will then examine possible approaches to solve these challenges and see where these technologies fall short and how they could be improved.
https://sched.co/UacZ
A
Essentially,
if
you
look
at
the
source
mesh
capabilities
today,
we
generally
talk
about
identity.
We
talk
about
service
discovery,
routing
as
well
as
policy,
but
oftentimes.
What's
forgotten
about
is
that
there
is
something
that
that
also
exists,
which
is
DNS,
which
is
an
integral
part
of
service
discovery,
and
it
kind
of
opens
up
another
set
of
use
cases
that
people
kind
of
need
to
think
about,
and
that's
essentially,
the
premise
of
our
presentation
today.
A
I'll
talk
about
use
cases,
while
Vanille
here
we'll
go
more
into
how
we
are
looking
at
it
from
kubernetes
and
envoy
and
hopefully
SEO
as
well,
so
going
a
little
bit
more
into
how
we
look
at
it
so
naming
in
general
for
applications.
Is
it's
very
complicated
in
the
sense
that,
as
you
go
through
your
development
cycle
and
applications,
progress
to
staging
rather
death
staging
and
then
towards
production?
A
A
Caching,
as
well
as
you
don't
want
services
across
different
different
namespaces
to
be
able
to
discover
each
other.
How
do
you
solve
that
today?
Within
the
bounds
of
kubernetes?
There
is
no
way
to
do
that.
So
that's
one
of
the
things
that
we
as
a
team
have
been
thinking
about
how
to
solve,
and
what
we
were
thinking
was
that
you
could
potentially
have
a
group
of
namespaces
bound
together
and
then
combine
that
and
have
a
DNS
kind
of
servicing
that
set
of
namespaces
and
in
general
by
doing
that.
A
A
Well,
typically,
we
look
at
the
elementary
data,
but
what
we
look
at
is
is
HDPE
statistics
or
the
RPC
stats,
but
no
one
kind
of
thinks
about
cases
where
DNS
is
actually
not
working.
And
how
do
you
kind
of
build
some
tooling
around
trying
to
see
whether
your
application
is
sluggish
because
of
DNS?
Very
simple
example
is
that
sometimes
v6
might
trample
before
in
terms
of
queries
and
slowly
or
slower,
your
application
down?
How
do
you
solve
that?
A
So
this
one
of
the
topics
that
we
we
kind
of
have
in
well,
then,
now
that
we've
talked
about
security,
sometimes
also
DevOps
engineer,
one
of
the
things
that
you
want
to
look
into
is
that,
as
services
are
accessing
other
services
within
a
cluster,
you
might
not
want
your
so
is
to
access
anything
which
is
outside
your
organization.
And
how
can
you
achieve
that?
So?
A
simplistic
example
is
that
I
want
to
deny
my
front-end
service
from
any
accessing
any
of
the
dot-com
sites
as
a
security
measure.
A
Another
example
would
be
going
back
to
the
tenant
isolation
case.
If
you
also
want
to
restrict
it
in
terms
of
policy,
you
could
potentially
say
that
tenant
day
is
not
allowed
to
access
and
be
services
from
from
a
service
discovery
point
of
view,
rather
than
once
the
request
is
actually
sent
out
so
yeah.
So
so,
within
the
bounds
of
this
one
of
the
things
that
we
have
been
looking
at
is
that?
A
A
This
is
a
little
more
evolutionary
where
the
the
ITF
and
the
community
have
been
starting
to
talk
about
the
toh
kind
of
adding
some
some
some
kind
of
color
to
it
essentially,
or
we
were
thinking
us
now
that
we
have
a
proxy,
can
you
can
you
kind
of
make
your
space
also
private,
in
the
sense
that
a
tenant
when
you
think
in
terms
of
tenant
isolation?
You
also
want
to
think
about
privacy,
where,
within
the
bounds
of
tenancy,
you
should
let's
say
for
some
reason:
you
are
able
to
to
look
at
the
NS
queries.
A
Can
you
encrypt
it
and
maintain
privacy
across
what
you're
actually
accessing?
So
this
is
one
of
the
use
cases
that
we've
been
kind
of
thinking.
That
would
be
useful
within
the
bounds
of
the
topic
around
DNS
and
yeah.
So
here
again,
essentially,
the
idea
was
that
we
were
during
the
discovery
process.
B
Okay,
before
I
dive
into
the
solutions,
I
want
to
ask
a
couple
of
questions.
Who
here
knows
what
a
service
mesh
is,
there's
a
lot
of
hands?
Who
here
really
understands
what
a
service
mesh
is
exactly
so,
but
the
good
thing
is,
there
are
a
lot
of
learnings
that
you
can
take
from
service
mesh
and
one
of
the
key
learnings
that
I
love
about
Suri
smashes
the
sidecar
proxy
pattern
and
I
will
be
using
that
pattern
to
showcase
the
solutions
that
we
came
up
with.
B
First,
let's
look
at
the
kubernetes
DNS
as
it
stands.
Today
we
have
a
couple
of
services
here,
a
and
B-
and
let's
say
we
want
service
yet
to
discover
service
B,
and
in
order
to
do
that,
it
will
send
a
request
to
the
cube,
DNS
or
Co
DNS
pod
in
the
cube
system
namespace
and
it
will
have
a
IP
or
a
cluster
IP,
and
the
request
is
typically
UDP
or
TCP,
and
once
the
response
is
received
from
the
DNS
server
service,
EA
can
connect
to
service
B
using
its
IP.
B
B
Here
is
an
open
source,
DNS
filter
that
we
used
in
order
to
get
here
and
the
payment
gateway
when
it
sends
a
UDP
request
or
the
DNS
query.
It
gets
trapped
into
the
DNS
filter
in
the
sidecar
proxy
and
then
the
DNS
filter
forwards.
The
UDP
DNS
query
into
the
DNS
server,
and
once
it
gets
a
response,
it
can
connect
with
the
users
API.
B
The
second
use
case
is
a
DNS
observability
here.
What
I
have
is
the
core
OS
Prometheus
operator,
and
there
is
a
CRD
named
service
monitor
which
allows
Prometheus
to
scrape
metrics
from
different
endpoints.
In
onward.
There
is
important
names,
stats,
/,
Prometheus,
and
here
we
run
it
on
port
8000,
and
since
we
have
the
DNS
filter
accessing
all
the
DNS
queries,
we
can
monitor
and
we
can
measure
every
query
and
we
can
also
log
queries
that
are
going
to
external
services
or
internal
services,
and
all
of
that
could
be
exposed
into
Prometheus.
Using
this
operator.
B
We
need
to
specify
the
application
we
want
to
track,
and
here
is
the
payment
gateway
application.
We
want
to
track
running
in
the
payments
namespace
and
in
real.
It
looks
like
this,
so
we
have
the
payment
gateway
and
the
proxy
in
front
of
it.
The
DNS
server
for
that
tenant
and
the
DNS
filter,
the
payment
gateway,
sends
a
DNS
query
to
the
DNS
filter
which
forwards
it
to
the
tenant
as
before.
B
The
next
next
use
case
is
the
DNS
filtering
use
case
here
we
want
to
whitelist
and
blacklist
certain
domains,
and
we
do
that
by
specifying
the
policy
here
and
in
the
whitelist.
We
select
all
the
services
running
in
the
payments
namespace
and
we
applied
this
policy
on
the
payments
workload
and
we
want
to
route
all
the
DNS
queries
to
tenant
one.
So
specifically,
if
there
is
a
query
that
wants
to
access
something
outside
of
the
payments
namespace,
it
would
be
matched
here
and
probably
it
would
be
denied.
B
There
is
also
a
blacklist
where
we,
where
you
can
specify
a
set
of
domains,
that
you
don't
want
your
services
to
discover.
In
this
case
it's
a
started
for
calm,
and
if
the
domain
matches
this
URL,
it
would
fail
to
even
identify
the
IP
address
for
it,
and
the
last
configuration
here
is
the
default
action.
So
let's
say
we
haven't
listed
any
domain
which
the
service
is
trying
to
access,
and
in
that
case
we
just
want
the
proxy
to
log
with
an
error
message.
We
could
also
specify
whether
the
domain
should
be
allowed
or
denied.
B
Even
so,
in
reality
it
would
look
something
like
this:
there
is
the
payment
gateway
which
wants
to
access
the
payment
info
service.
There
is
the
proxy
and
the
dns
server
for
the
tenant,
and
here
is
the
dns
policy
filter.
So
whenever
there
is
a
UDP
dns
query,
it
would
be
trapped
by
the
dns
policy
filter
and
it
would
consult
the
policies
database
and
in
the
previous
spec
that
I
showed
we.
B
We
write
the
policies
down
in
the
ml
file
itself,
but
this
could
also
be
an
external
database
and
once
your
decides
to
accept
that
DNS
query,
it
can
then
forward
the
DNS
query
to
the
tenant,
specific
DNS
server
and
once
the
payment
gateway
receives
the
response
back
from
the
DNS
server.
It
can
then
connect
with
the
payment
info
service.
B
But
let's
say
if
the
payment
gateways
trying
to
access
malicious
website,
for
example,
API,
dot
foo.com,
and
in
that
case,
that
dns
filter
can
quickly
terminate
such
DNS
requests
and
the
connection
will
not
even
happen,
but
there
is
a
catch
here.
Let's
say
the
payment
gateway
is
kind
of
smart
and
it
already
knows
the
IP
address
for
the
external
service,
which
is
API
dot
foo.com
here.
In
that
case,
we
can't
do
anything
at
the
DNS
level,
but
we
need
to
apply
some
sort
of
network
policies
to
get
rid
of
such
requests.
B
Here
is
an
extensible
model
of
the
DNS
filtering
spec.
In
this
case,
we
route
all
the
DNS
policy
queries
to
an
external
policy
server
which
understands
the
G
RPC
protocol
and
whenever
a
DNS
query
enters
into
the
proxy,
we
consult
this
external
service
and
by
default
we
can
deny
any
domains
that
the
policy
server
doesn't
understand,
and
one
other
thing
we
can
do
here
in
the
spec
is
we
can
specify
the
specific
application.
We
want
to
apply
this
policy
over
and
in
this
case
it's
payment
gateway
in
the
payments
workload.
B
B
So
again
we
select
the
workload
payment
using
the
namespace
selector
and
this
time
we're
out
such
DNS
queries
to
the
encryption
supporting
DNS
server
here,
and
we
can
upgrade
the
protocol
by
specifying
the
protocol
field
here
and
we
can
specify
what
kind
of
protocol
we
want
to
use
or
we
want
to
upgrade
to
so.
In
this
case
it
is
DNS
over
HTTPS
or
it
could
also
be
DNS
over
TLS,
for
example.
B
So
in
reality
it
would
look
something
like
this.
There
is
a
payment
gateway
and
the
payment
info
service.
We
have
the
proxy
like
before,
and
the
DNS
tenant,
and
in
this
case
the
tenant
supports
encrypted
queries
and
whenever
the
payment
gateway
makes
UDP
requests,
it
gets
trapped
by
the
DNS
filter
and
this
time
the
DNS
filter
originates
encrypted
connection
to
the
DNS
service,
st
dns.
B
B
That's
it
and
for
the
demo,
I
have
a
simple
application:
running
called
the
HTTP
bin
and
a
sleep
pod
running.
We
are
which
I'm
going
to
make
queries
for
the
HTTP
service.
I
have
the
same
application
running
as
a
clone
in
a
different
namespace,
so
the
first
namespace
is
acne
calm
and
the
second
one
is
foo.com.
B
We
have
the
tenant
specific
DNS
services
here
and
both
have
their
own
unique
IPS.
We
also
have
the
own
web
proxy
running
in
front
of
the
sleep
service.
We
don't
really
need
the
on
web
proxy
in
front
of
the
HTTP
service.
For
now
and
whenever
the
sleep
service
in
the
acne,
a
namespace
wants
to
discover
the
HTTP
bin
service
in
the
acne
namespace.
It
would
make
a
DNS
query
to
the
Envoy
proxy.
B
B
We
want
to
allow
such
request
as
well,
but
if
the
steep
part
in
the
acne
namespace
makes
a
query
for
the
HTTP
bin
service
in
the
foo
namespace,
we
don't
want
to
allow
such
requests,
and
the
same
is
true
for
the
other
way
round.
So
we
don't
want
the
HTTP
server
running
in
the
acne
namespace,
even
to
be
discovered
by
the
sleep
service
in
the
full
namespace.
B
Let's
see
that
for
real,
so
I
have
this
deployment
on
kubernetes
and
have
the
three
pods
corresponding
to
each
service
that
I
showed
earlier
and
I
am
running
the
same
set
of
pods
in
both
namespaces.
So
if
you
notice
core
DNS
has
just
one
container.
The
HTTP
bin
service
also
has
one
container,
but
the
sleep
pods
in
both
namespaces
have
two
containers.
One
container
corresponds
to
the
slip
container
and
the
other
one
is
the
proxy.
B
Now,
when
I
make
queries
from
the
sleep
services,
you
will
notice
the
difference
so
here
I'm,
making
a
query
which
is
a
nslookup
query
for
HTTP
bin,
not
acme
comm
from
the
sleep
container
within
the
Acme
namespace.
So
this
should
work
and,
as
you
see,
we
received
a
an
IP
address
corresponding
to
that
service.
In
the
second
case,
we
are
doing
a
similar
query,
but
in
the
Foo
namespace
so
from
the
Foo
namespace.
B
In
the
sleep
container,
we
make
up
query
for
the
HTTP
bin
service
in
the
same
namespace
and
we
received
an
IP
for
that.
But
in
the
third
case
it
fails
because
the
sleep
service
in
Acme
cannot
access
the
HDD
we've
been
in
foo
and
the
other
way
around
as
well.
So
sleep
service
in
foo
cannot
access,
it
should
have
been
in
Acme,
and
this
is
all
done
by
using
the
DNS
filter
that
I
spoke
of
earlier,
and
we
can
do
all
sort
of
creative
things
with
the
filter.
B
I
wanted
to
show
you
the
code
that
I
use
it
in
order
to
get
this
working.
So
we
have
the
DNS
policy
configuration
here
for
the
sleep
service
and
this
time
we
are
looking
at
the
sleep
service
for
the
foo
namespace
and
for
the
name
server.
We
drop
the
cube,
DNS
IP
address
and
instead
use
a
local
host
IP
address
and
we
don't
have
any
search
domains
and
we
have
some
other
configuration
here
and,
in
the
same
part,
we
are
running
the
Envoy
container
and
this
is
the
Envoy
configuration.
B
B
B
With
that,
I
want
to
summarize
by
saying
that
DNS
plays
a
very
important
role
for
service
discovery
and
application
migration,
and
then
current
Enterprise
Systems
multi-tenancy
is
extremely
important
and
it's
also
very
critical
at
the
DNS
layer
and
as
I
showed
you
in
the
demo.
The
on
web
proxy
can
solve
some
of
these
interesting
challenges
with
DNS
using
a
filter
like
the
DNS
filter
and
the
best
part
about
envoy
is
that
you
can
configure
it
at
runtime
by
using
its
XDS
api's.
B
So
you
don't
need
to
restart
the
proxy
nor
any
of
your
services,
and
everything
will
work.
So
if
you
have
new
policies
that
you
want
to
apply
at
runtime
on,
Y
will
take
care
of
it
pretty
easily.
You
can
also
integrate
these
DNS
filters,
along
with
third-party
systems,
to
provide
richer,
observability
security
and
filtering.
So
I
spoke
about
Prometheus
and
graph
on
ax.
B
We
could
also
consider
other
options
here
and
finally,
I
want
to
mention
that
we
want
to
contribute
this
work
to
the
open
source
and
existing
projects
like
sto
and
Envoy,
and
if
you
have
a
use
case
for
this
work,
then
please
get
in
touch
with
us
and
we
would
love
to
collaborate
with
you.
And
if
you
have
any
questions,
we
can
speak
now
or
also
be
available
at
the
booth
right
after
the
stock.
Thank
you.
C
A
Allow
you
to
go
per
say
if
you
want,
if
you
want
to
kind
of
look
at
the
DNS
queries
and
responses
and
also
build
some
telemetry
around
it.
That's
when
you,
you
know
you
look
at
this
versus
just
play
network
policies,
but
having
said
that,
it's
useful
to
even
have
the
Netta
guardrails
at
the
neck
network.
Layer
and
I
had
this
on
top
God.
E
In
regards
to
making
multiple
DNS
servers
everywhere,
as
they
need
lots
of
these
proxies,
like
what
type
of
impact
would
you
be
looking
at,
obviously
you're
putting
down
lots
more
core
DNS
pods
you're?
Putting
all
of
these
filters
down
how
much
of
an
impact
you
talking
about
if
you've
got
a
really
large
cluster
you're
gonna
have
a
lot
of
containers
that
are
running
so.
A
The
boundary
around
that
is
a
group
of
namespaces,
so
it
just
depends
upon
how
many
names
namespaces
you
wanna
have
the
DNS
serving.
So
the
dimension
is
not
at
the
level
of
the
services
or
the
pods,
but
it's
more
at
a
higher
level.
So
it
just
depends
upon
how
many
namespaces
you
will
have
on
that
cluster
or
across
clusters
as
well.
Yeah.
E
F
So
my
question
also
was
similar
to
the
network
policy,
so
that
is
one
second
thing
is
like.
Eesti
also
provides
our
back
right,
so
that
can
also
do
a
denial
of
service
for
the
service
levels,
so
you
can
restrict
like
which
service
can
call
which
service
we
can
put
various
filters
there
itself
right,
yeah.
A
But
you
can't
do
it
at
a
DNS
layer
today,
and
this
is
what
we
want
to
bring
in
interest
here
as
well.
You
can
do
it.
You
can
put
l7
policies
on
HTTP
and
G
RPC
requests,
but
not
at
the
service
discovery
layer
in
terms
of
DNS,
so
this
would
be
an
add-on
and
yeah
be
lightly
leveraged,
our
back,
but
but
yeah
no.
B
B
G
So
kind
of
adding
to
what
you
were
saying
so
with
sto
when
you
have
like
service
entries
or
you
have
the
registry
only
mode,
you
kind
of
can
do
this
kind
of
DNS
filtering,
because
you're
limited
to
the
namespace
or
whatever
you
are
trying
to
expose
to
your
Envoy
proxy.
So
why
is
this
sort
I
mean
this
would
be
including
like
a
new
CID
or
something
like
that.
But
why
isn't
it
something
that
we
can
expand
using
the
registry
only
mode,
for
example,.
B
G
It's
it
kind
of
boils
down
to
DNS
in
a
sense,
because
what
you
can
do
with
Sto-
and
you
have
the
registry
only
mode-
is
that
all
the
outgoing
requests
are
limited
to
the
whatever
you've
actually
white
listed
for
your
particular
workload.
So
what
I
was
visible
within
the
namespace
or
visible
to
it
within
like
another
namespace
for
a
particular
application,
so
it's
already
kind
of
restricted
and
so
anything
that
you're
trying
to
resolve.
If
envoy
doesn't
already
know
about
it,
it's
not
going
to
be
able
to
route
the
request
anyway.
Yeah.
B
So
this
solution
is
not
specific
to
sto
I
mean
you
can
take
this
and
run
it
on
like
just
on
Moi's
or
yeah.
You
don't
even
need
a
service
mash
per
say.
This
is
like
a
generic
solution,
but
you
can
adopt
this
in
different
ways.
I
mean
these
problems
can
also
be
solved
in
kubernetes
or
in
core
DNS.
So
this
is
one
way
that
you.
A
A
Ok,
if
you
had
to
rethink
it
and
see
if
there's
a
different
way
where
essentially,
okay,
dns
isolation
is
a
piece
they
can
even
cross
cluster,
but
then
with
this,
you
can
also
think
about
more
use
cases
right
like
we
brought
up
telemetry
and
observability,
which
is
kind
of
not
something
you
can
do
it.
The
current
is
your
code
base
right
now,
so
I.
H
Think,
thank
you
for
the
presentation.
I
was
thinking
of
Grundy.
We
there
is
the
node
local
DNS
cache
that
is
currently
in
beta
right
and
I
was
thinking
exactly
what
you're
doing
is
kind
of
these
like
decentralization
of
DNS,
but
not
at
non
level,
and
rather
at
like
namespace
level,
but
in
not
local
DNS.
There
is
CAF
8
that
you
lose
every
BT
if
your
DNS
is
not
ready,
if
your
DNS
server
is
not
ready.
H
B
I
think
that's
like
a
bootstrap
problem
which,
which
is
a
general
I,
mean
it's
a
general
problem,
I
think
with
any
of
the
DNS
solutions.
We
have
right
now,
one
way
I
mean
with
the
filter.
You
can
do
something
like
you
can
keep
worrying
for
the
uptime
of
the
DNS
server
and
then
you
can
probably
start
responding
so
and
that
could
be
a
part
of
the
health
check
of
the
proxy
itself.
So
you
can
play
around
like
that.
Yeah.