►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Cruise’s Self-Driving Networking Journey - Bernard Van De Walle & Jon Yucel, Cruise
Through its exponential growth, the Platform team at Cruise experienced a networking self-driving journey. We scaled our network across numerous clusters, multiple tenants, and multiple thousands of new pods instances a day. We will take you on a tour of our architecture and you will get a better understanding of how we choose to configure our network and security in order to support Kubernetes loads across multiple regions and multiple environments. We will specifically showcase how we do this on a public cloud (GCP) even though similar results could be achieved on-prem. You will come out of this session with concrete examples on what it takes to build your network and security needs for internal tenants at scale while keeping internal stakeholders happy (Platform, security and networking).
https://sched.co/UaXO
A
A
We
both
walk
into
the
platform
as
a
service
team
and
more
specifically,
into
the
traffic
team,
and
so
this
means
we
handle
things
like
service
mesh
in
the
talking
issues,
ingress
and
so
on,
and
this
is
what
we
like
to
do
when
we
don't
need
to
debug
ingresses
okay,
so
we
both
fought
for
Cruz
Cruz
is
a
self-driving
car
company,
headquartered
in
San
Francisco.
We
are
backed
by
a
couple
of
the
giants
in
the
automotive
industry,
such
as
General,
Motors
and
Honda.
A
We
are
aiming
to
build
the
world's
most
advanced
self-driving
vehicles
to
safely
connect
people
of
the
places,
things
and
experiences
they
care
about,
and
this
means,
if
you
went
over
the
last
couple
of
weeks
or
month
into
the
city
to
San
Francisco,
you
probably
saw
some
of
those
girls
driving
around.
They
are
still
learning,
but
eventually
we
want
to
make
these
public
service.
What
you
can
see
on
this
slide
is
an
animation
of
webley's
web.
This
is
one
of
our
open
source
tools
that
we
use
a
lot
internally
as
a
front-end
way
for
replaying.
B
They
say:
ok,
I,
hope
that
you
are
hearing
me.
Ok,
yes,
louder,
hey,
let's
fry
ok,
my
name
is
John
and
I
am
the
technical
lead
of
the
path
traffic
team
at
Cruz,
and
our
basic
focus
here
is
network
reliability.
So
our
focus
overlaps
with
our
journey
here,
which
covers
like
these
multiple
topics
like
we.
Our
journey
will
be
spanning
multiple
cloud
providers
and
we
have
these
multi
tenant
clusters
that
spans
through
multiple
environments
and
multiple
regions.
B
Our
DNS
story
covers
route
53
core
DNS
cloud
DNS
and
it
is
not
listed
here,
but
unfortunately,
still
cubed,
DNS
and
like
we
do
have
these
private
and
public
l7m
al
for
load,
balancer
and
I
had
the
opportunities
for
being
the
one
of
the
rare
first
pass.
Team
members,
so
I
was
able
to
just
experience
this
growth,
where
we
have
just
spun
up
our
very
first
Jiki
cluster,
where
we
were
running
a
few
pots
on
them
and
as
of
this
week,
we
are
running
somewhere
around
7k
pods
in
average,
among
all
our
clusters.
B
Okay.
This
is
the
first
attempt
step.
Let's
try
to
connect
from
this
part
a
to
a
service
running
an
AWS.
So
initial
item
only
included
VPN
tunnels,
because
why
VPN
tunnels?
It
was
easy
to
set
up
things
and
then
the
traffic
was
going
through
like
public
internet
through
IPSec
encryption
and
also
we
didn't
need
to
disturb
our
neurons
team.
So
we
were
able
to
just
set
up
things
by
ourselves
and
the
first
problem.
So
what
went
wrong?
B
We
made
this
first
request:
oh
okay,
maybe
I
should
just
put
this
I
think
it
is
better
now
anyway,
so
the
first
request
from
this
part
a
to
service
B
and
the
request
timeout.
So
what
was
wrong
here
like
we
were
pretty
sure
that
we
have
defined
our
return
routes,
because
we
were
able
to
access
from
this
GCE
instance
to
this
service
on
AWS
and
everything
is
looking
fine,
but
it
didn't
work
for
the
GK
cluster.
B
So
the
left-hand
side
had
the
default
IP
mask
configurations
where
it
thinks
that
the
whole
internal
network
is
a
flat
network.
So
we
are
able
to
reach
everything
else
around
there,
but
it
wasn't
true
on
our
case,
because
we
didn't,
we
specifically
didn't
advertise
these
class
B's
and
class
C's,
because
we
were
planning
to
reuse
them
across
the
different
environments,
and
that
was
why,
and
whenever
you
make
a
request
to
something
with
Class
A
address,
IP
mask,
wasn't
masquerading
or
source
netting
the
request,
and
that
was
causing
trouble.
B
So
for
this
particular
case,
we
have
picked.
These
side
range
that
the
cluster
was
running
on.
So
there's
this
small
detail
here,
so
the
cluster
was
running
on
this
side
of
range
and
like
it
matches
with
this,
and
this
is
just
a
highlight
of
overview
of
what
we
have.
This
is
a
common
slide
that
we
use
internally.
So
we
do
have
these
three
regions
they're,
staging
and
proud.
They
are
all
connected
to
each
other
and
then
on
all
these
environments.
We
are
running
multiple
clusters
for
multiple
purposes
or
on
multiple
regions.
B
So
this
is
the
topology
that
we
have
okay.
So
you
probably
you
already
have
the
question
like
how
are
these
guys
in
scale
with
VPN
tunnels?
No,
we
can't
so
the
idea
or
the
goal
here
or
the
problem
here
was
we
like
it's
pretty
easy
or
straightforward
to
hear
into
this
N
squared
tunnel
problem,
because
whenever
you
introduce
a
new
network,
you
need
to
just
establish
VPN
connections
with
all
other
networks
that
you
have.
B
You
need
to
manually
or
statically,
just
update
all
your
route
tables,
which
is
gonna,
be
tedious
and
like
chaotic,
pretty
soon
and
also
like
IPSec
calls
or
the
whole.
This
public
Internet
with
IPSec
like
it's
gonna,
be
a
reduced
performance
problem
and
the
network
engineering
team
solution
here
was
leveraging
interconnects.
So
with
that,
we
were
able
to
connect
all
every
new
network
to
these
interconnects
which
provided
us
like
a
couple
of
things.
We
had
these
bgp
routers.
They
were
responsible
for
dynamically
advertising,
all
these
rels
internally,
so
we
don't
need
to
manually
manage
them.
B
That
was
one
thing,
and
we
also
had
physical,
dedicated
connections
between
interconnects
and
other
networks
and
which
provided
us
like
to
like
get
rid
of
IPSec.
That
was
one
thing
and
the
other
thing
was.
We
had
like
improved
bandwidth,
which
was
around
100
gigabit
per
second,
which
is
also
a
good
game
and
yep
that's
pretty
much
it
so
normally.
This
is
very
highlighted
overview
of
how
things
are
set
up
on
a
networking
level.
A
Thanks
John
and
again
so
we
started
to
scale
always
until
your
crystals
at
some
point.
So
when
we
reach
the
critical
mass
for
a
single
cluster,
we
decided
to
create
more
clusters
for
indictment.
So
this
means
more
cluster
in
dev
was
called
more
cluster
and
staging
and
input,
and
the
way
we
decided
to
do
this
was
to
create
one
subnet
on
the
BBC
rock
crystal,
and
the
first
challenge
was
for
each
cluster.
A
A
It
would
be
different
if
you're
in
the
dev
VPC
or
staging
the
PC,
which
region
I
want
your
cluster
to
be
in
as
well
as
what's
the
sizing
of
the
cluster,
the
sizing
of
the
cluster
is
kind
of
tricky
because
you
need
to
know
in
advance
how
big
your
cluster
was
eventually
going
to
be,
because
once
you
create
a
custom
in
a
subnet,
it's
extremely
difficult
to
resize
it
to
bigger
IP
range,
and
so
we
came
with
a
set
of
constituents.
We
wanted
on
node
IPS
to
be
globally
unique.
A
We
wanted
to
do
this
so
that
from
anywhere
inside
Cruz
it
would
be
easy
to
reach
any
of
the
node
IPS
from
the
cluster.
We
also
want
to
the
pod,
eyepiece
and
Service
IPS
to
be
only
locally
unique
per
VPC,
so
those
could
be
used
across
V
pcs,
so
you
could
reuse
the
same
pod
IPS
in
dev
and
in
staging,
for
example.
So
the
challenge
here
was
that
it
was
extremely
tedious
to
choose
manually.
All
of
those
IP
ranges
based
on
those
constraints.
A
So
what
we
did
a
couple
of
months
ago
is
to
automate
this
by
using
Jerry,
say
hi
to
Jerry
Jerry's,
small
tool
that
is
automatically
connecting
to
open
IP
management
tool
in
the
backend.
We
use
networks
at
cruise
and
it's
so
time
a
quickly
going
to
use
those
constituents
as
parameters
and
you
will
get
assigned
a
set
of
IP
ranges
for
your
clusters.
A
So
this
is
how
we
automatically
create
clusters
based
on
those
IP
ranges
and
by
the
way
we
learned
something
on
the
way
is
that
it's
quite
valuable
to
do
the
meta
view
of
all
your
crystals
in
a
single
place.
For
us,
it's
netbooks,
because
it
said
it
will
help
you
to
debug
some
crazy
corner
cases
when
something
goes
wrong
and
you
need
to
reach
one
janilla
based
on
the
source
IP,
which
bud
which
node
IP
it
comes
from
from
each
cluster.
A
Ok,
next
thing
I
would
like
to
talk
about
is
service
ingestion
and
as
the
traffic
team.
This
is
one
of
the
main
things
we
deal
with
day
today
and
we
talk
to
a
lot
of
internal
tenants
and
they
all
want
to
somehow
make
their
service
available.
That
makes
sense
right,
so
we
made
this
chat
flow
it
basically
it
it
will
make
it
easy
for
customers
to
decide
on
internal
talent
which
type
of
service
they
need
to
use
in
order
to
make
the
micro
service
or
pods
available.
A
So
let's
go
to
this,
the
first
one
that
makes
the
most
sense
is
what,
if
you
are
inside
the
same
cluster,
in
which
case
very
simply,
you
just
need
to
create
a
community
service.
This
works
out
of
the
box.
We
will
use
it.
It's
fine,
now,
ok,
next
one,
what
if
you're
inside
Cruz
but
not
in
the
same
cluster,
and
you
TCP
or
UDP
circuit,
to
something
like
layer,
3
layer
4.
A
A
Ok,
the
next
one
I
want
to
spend
a
bit
more
time
on
it,
because
it's
what
we
use
the
most
at
cause,
it's
a
layer
of
seven
ingress
so
for
private
traffic.
This
probably
represents
about
95%
of
traffic.
At
cruise,
it's
basically
layer,
7,
internal
traffic,
coming
from
other
clusters,
other
end
points
or
other
VMs
or
something,
and
we
decided
to
use
the
standard
ingress
resource
to
deal
with
this.
We
also
decided
to
use
upstream
and
genex
ingress,
and
we
are
pretty
happy
with
it.
A
The
only
tweak
we
made
was
to
use
a
dedicated
node
pool
to
make
sure
that
we
don't
get
into
some
type
of
resource
congestion
or
like
noise,
enable
so
that
each
and
genex
ingress
controller
could
use
the
full
node
for
himself,
and
we
also
enabled
one
of
those
parameters
called
excel
traffic
policy,
which
allows
the
load
balancer
in
front
to
help
check
each
of
the
nodes
in
a
way
that
the
node
will
only
be
available
if
an
ingress
Ingenix
instances
is
present
on
that
node.
So
why
do
we
want
to
do
this?
A
Because
you
don't
want
to
send
traffic
to
a
node
that
doesn't
even
nginx
ingress
instance,
because
if
you
do
that
queue,
pox
is
going
to
redirect
it
to
another
node
with
an
actual
instance
available.
So,
by
doing
this,
you
basically
avoid
in
the
worst
case
one
extra
hop,
so
it's
better
for
performances.
A
So
if
we
had
the
first
big
outage
on
the
4th
of
July
by
using
this,
we
called
it
the
firewalk
outage.
What
happened?
Is
we
found
out
that
the
internal
load
balancers
from
Google
can
only
health
check,
maximum
total
50
nodes?
And
so,
when
you
have
a
cluster
of
way
beyond
on
tune
or
50
nodes,
what
happens
is
250
nodes
will
be
selected
randomly
to
be
health
checked.
We
are
the
crystal
so
big
that
all
the
nginx
ingress
instances
randomly
girl,
toads
or
the
health
check,
so
this
black
hole
the
traffic
and
yeah.
A
A
A
A
Ok,
let's
talk
a
bit
about
public
traffic,
so
we
don't
have
today
a
lot
of
public
traffic,
but
we
still
support
it
in
the
same
way
that
we
support
private
traffic.
So
that
means,
if
you
need
TCP
or
UDP
sockets.
What
we
do
is,
but
we
highly
discourage
this.
But
if
you
need
it,
we
will
create
a
public
network,
balanced
load
balancer
for
you.
If
a
public
IP
and
it's
up
to
your
workload,
200m
TLS
and
firewall
it
heavily
I
suggest
we
push
people
to
use
layer
7,
so
we
use
also
ingress
for
public
traffic.
A
We
initially
use
LDC
we,
but
we
decided
to
standardize
on
nginx
ingress
because
we
wanted
to
the
same
set
of
metrics
both
for
private
and
for
public
the
same
set
of
logs
and
same
set
of
metrics.
It
makes
it
way
easier
to
to
manage
so
what
we
did
is
simply
we
duplicated
the
private
set
up
for
ingress
and
made
it
public
and
we
differentiate
both
of
those
increases
with
annotation
for
different
annotation,
and
so
this
is
what
swimming
today
on
our
crystals
so
which
cluster
we
got.
A
Ok-
and
this
brings
me
to
one
lesson
that
we
learn,
is
you
want
to
support
a
small
subset
of
options
but
support
them?
Well,
what
we
do
as
part
of
the
traffic
team
is
push
people
to
use
layer,
seven
ingress,
and
that's
why
we
supported
the
best.
What
you
can
see
on
this
slide
is
the
two
type
of
annotations
that
we
document
internally
for
using
the
private
or
public
ingress.
B
So
let's
talk
about
a
little
bit
about
our
DNS
story
here
so
again,
I
will
start
with
the
most
naive
approach
that
we
fell
out
here,
which
is
okay.
We
do
have
all
our
records
stored
on
as
route
53
records,
and
then
we
have
these
DNS
mask
agents
running
on
AWS,
which
were
pulling
these
like
DNS
records.
So
with
this
initial
attempt,
what
we
have
done
here
is
we
have
updated,
cube,
DNS
top
domains.
B
So
whenever
there's
some
internal
request,
it
was
forwarding
these
requests
to
the
DNS
proxy
that
were
running
in
another
network
and
like
problem
solved
at
least
for
a
month
or
so
it
was
solved
and
then,
like
we
like,
started
running
into
some
latency
issues.
We
because
we
were
just
having
these
additional
network
hops,
which
were
not
ideal
at
all.
The
second
attempt-
okay,
we
noticed
that
we
can
use
coordinates
because
it
already
has
this
route
2
to
3
plug
in,
and
it's
gonna
just
solve
the
problem
for
us
was
it
like?
B
He
contributed
to
the
plugin,
and
then
I
made
things
product
ready
and
what
happened
in
the
end
was
we
were
able
to
just
fetch
all
the
records,
with
the
exception
of
alias
records,
which
was
kind
of
fine
for
the
time,
and
then
we
did
the
exact
same
configuration
here
so
cube,
DNS
was
configured
through
stop
domain,
so
we
were
running
these
coordinates
instances
by
the
way.
Externally,
these
were
not
running
on
the
cluster.
These
were
external
and
cubed.
Dns
was
for
being
conditional
forwarding
these
coordinates
instances
and
core
DNS
was
periodically
syncing
AWS
records.
B
B
So,
by
the
way
like
that,
I
also
noticed
something
here:
I'm
just
going
a
little
bit
sideways
here
yesterday,
I
attended
one
of
the
core
maintainer
of
Cordy
NS
Yoong
and
then
his
whole
talk
was
based
on
hybrid
DNS
structure,
where
he
used
or
showcase
AWS
and
cloud
DNS,
backends
and
I
felt
like
that.
Okay,
at
least,
we
had
some
sort
of
impact
on
the
open
source
community
with
this
contribution,
because
we
otherwise,
like
probably
it
wouldn't
be
included
in
the
demo.
B
So,
okay,
aside
from
these
DNS
servers,
we
also
have
some
common
DNS
structure
at
cruise,
which
is
we
do
have
these
a
records
which
are
pointing
to
our
load
balancers,
and
these
are
multi
tenant
clusters.
So
whenever
they
like
define
a
new
DNS
records,
instead
of
you
directly
using
the
load,
balance
or
IP,
they
are
using
these
C
names
that
we
like
create
it.
B
So
the
advantage
of
having
this
sort
of
approach
is,
whenever
you
make
a
change
on
your
load,
balancer
level
such
as
like
a
static,
IP
change
or
you
introduce
a
brand
new,
a
load
balancer.
You
don't
need
to
just
update
all
your
records
around.
Instead,
you
just
update
your
a
records
and
problem
solved.
A
A
B
Okay,
so
there
is
another
kind
of
fun
story,
so
okay,
ingress
monitoring,
one
of
the
or
we
face
this
particular
incident
a
couple
of
times.
So
we
are
just
sleeping
well
on
our
beds
and
then
everything
is
looking
fine,
we
didn't
receive,
we
don't
receive
any
alerts
or
anything
at
all,
but
the
problem
is
as
soon
as
we
wake
up.
We
are
just
with
us.
We
start
receiving
some
clients
like
calls
which
is
okay,
the
or
load
balancer
is
down.
B
Usually
they
are
not
saying
that
they
are
just
saying
that
DNS
is
isn't
working,
so
there's
a
problem
there
and
okay,
like
the
problem
here
is
mean
time
to
response-
is
pretty
high,
I.
Think
about
that,
like
we
are
responding
within
a
couple
of
hours
when
an
incident
happens.
So
how
can
we
mitigate
this
problem?
B
So,
okay,
we
need
a
bunch
of
progress
around
and
we
need
to
just
pick
one
of
the
existing
SAS
products,
and
this
has
product
needs
to
work
for
both
of
our
private
and
public
endpoints,
because
whenever
you
have
this
choice,
like
you're
gonna,
just
eliminate
most
of
the
options
there
and
eventually
we
were
planning
to
help
our
tenants
to
identify
problems
early
on.
So
we
pick
round
scope
for
this
purposes
and
we
also
developed
a
round
scope
controller
thing:
why
run
scope?
Because
it's
supported
both
public
and
private
endpoints?
B
We
were
able
to
just
run
some
data,
log,
8,
sorry,
ruskov
agents
internally,
and
they
were
allowed
to
just
ping,
our
internal
endpoints,
and
then
we
built
this
run.
Scope
controller
thing
around
that
and
the
idea
here
was
just
monitoring,
ingress
resources
and
then
picking
these
hostname
and
service
back-end
pairs
and
then
create
creating
round
scope
tests
as
a
result
and
like
it
worked
fine.
So
these
are
the
some
of
the
annotation
that
it
had
I'm
not
going
to
get
into
the
details
regarding
round
scope
stuff.
B
So
basically,
what
we
can
focus
here
are
so
you're
gonna
just
say
that
I
am
enabling
API
tests
for
my
ingress.
This
is
the
health
checking
path
and
we're
going
to
run
this
with
a
scale
and
internal,
and
this
is
the
prefix
where
we
can
prioritize
your
stuff,
which
is
an
optional
and
internal
thing
for
us.
B
Okay,
now
the
lesson
learned
part.
So
the
first
lesson
learned
was
like:
we
noticed
that
it
is
not
an
easy
job
to
create
controllers
like
I
was
under
estimating
the
work
about
that
we
are
gonna.
Do
a
couple
of
crowd
requests:
how
hard
can
it
be,
but
well
it
was
hard.
But,
aside
from
that,
the
second
kubernetes
related
problem
here
was
okay.
B
We
had
this
internal
proper
running
on
our
cluster,
which
was
pinging
the
sir
given
service,
IP
and
thought
afterwards
and
again,
a
customer
just
reported
and
all
this
here
and
there
again
said
that
internal
load,
balancer,
isn't
working
or
DNS,
isn't
working
whatever
and
okay,
but
Roberts
saying
the
opposite
here.
What
is
wrong
and
the
problem
here
was
when
you
have
low
balancer
type
services
depending
on
the
IP
tables
rules.
B
Kubernetes
doesn't
do
hair
pinning
so
instead
of
going
outside
of
the
cluster
and
the
request
coming
back
inside
it
just
it
is
just
directly
hitting
these
Service
IPS,
so
we
were
unable
to
identify
these
load
balancer
problems
by
running
agents
only
internally.
So
as
a
solution,
we
started
just
having
these
cross
cluster
robbers
where
we
started
using
all
over
the
place.
A
Okay,
thank
you,
John.
Let's
talk
a
bit
about
security.
How
do
we
secure
our
clusters
by
the
way?
I
want
to
start
this
by
saying
that
at
cruise
we
push
every
workload
to
use,
open
and
Ozzie
that's
kind
of
out
of
scope
for
this
slide,
but
from
a
pure
networking
perspective
familiar
through
layoffs,
ooh,
we
have
the
following
model
so
from
inside
the
VTC
by
default,
anything
inside
of
a
PC
is
able
to
reach
the
cluster,
the
pod
IPS
and
the
service
IPS
from
outside
the
V
PC.
A
By
default,
it's
a
whitelist
only
so
you
need
to
add
an
entry
to
be
able
to
access
a
cluster
or
Pacific
IP,
and
so
for
English.
We
explicitly
allowed
of
ingress
controllers
on
the
whitelist
and
so
yeah.
The
the
other
thing
we
don't
do
today
is
we
don't
have
network
policies
enabled
on
our
clusters,
but
we
are
actually
considering
enabling
maybe
a
default
namespace
isolation
on
that
level.
A
Okay,
something
else
that
you
might
have
heard
about
before
we
released
Kol
it's
on
github
that
it
was
produced
by
our
sister
team,
the
security
team
and
QA
listed
on
mission,
webhook
controller.
The
way
it
works
is
whenever
you
create
a
new
deployment
or
new
set
of
pods,
it
will
validate
it
and
make
sure
that
there
is
no
forbidden
or
insecure
parameters.
So,
for
example,
you
don't
want
to
add
a
pod
with
host
network
enabled
you
don't
want
to
have
extra
network
capabilities,
etc,
etc.
A
The
cool
one
is,
you
also
don't
want
to
enable
by
default
any
public
ingress
to
be
created.
You
want
those
to
be
white
listed
so
that
by
diff,
so
that
someone
doesn't
create
like
a
quick
endpoint
by
mistake,
for
example,
and
so
something
we
learned
is
if
you
are
going
to
block
some
of
the
user
requests,
you
need
to
give
a
pretty
user
friendly
outputs,
so
Kol
does
this.
A
Okay.
So,
let's
close
on
this
with
some
of
the
current
challenges
that
we
are
facing
today,
the
first
one
and
maybe
the
most
pressing
one-
is
multi
cluster,
specifically
multi
cluster
ingress.
We
have
that
point
in
which
we
are
load
balancing
a
lot
of
workloads
across
multiple
clusters,
but
you
don't
have
a
clear
way
today
to
accept
traffic
in
a
single
point
and
we
directed
smartly
to
different
clusters,
at
least
not
internal
traffic.
So
we
are
looking
at
this.
It
still
might
be
an
answer.
B
So
another
problem
that
we
have
faced
during
of
one
of
our
other
outages
was
like
the
lack
of
visibility.
With
that,
what
we
mean
is
we
do
have
these
multi-tenant
clusters,
where
every
node
is
running
at
this
50
pots
in
average,
and
then,
whenever
a
couple
of
those
starts
saturating
your
egress
traffic,
which
might
be
happening
through
your
net
instances,
we
are
unable
to
say
or
identify
which
one
of
those
work
those
are
causing
this,
because
we
were
able
to
just
address
the
particular
node,
but
we
don't
have
the
granularity
on
the
pot
level.
B
Isolation
is
fine,
but
there
are
still
two
other
levels
of
isolations
are
missing:
one
is
disk
I/o,
and
luckily
it
is
not
a
problem
of
past
traffic
team
and
the
second
one
is
network
like
Q
quality
of
service
stuff,
because,
like
we
would
like
to
have
some
sort
of
egress
and
ingress
traffic
limitation
set
up
and
the
other
thing
is,
we
still
need
to
do
a
couple
of
DNS
enhancements
with
this
said
what
we
mean
is
okay.
We
are
just
facing
the
exact
same
problems
with
everyone
else
regarding
cube
DNS,
and
we
liked
it.
B
We
don't
want
to
just
get
details
of
that
because
they
are
already
a
couple
of
good
talks
around
those
and
our
problems
are
or
the
solutions
here
will
be
replacing
cube,
DNS
with
coordinates
and
later
on,
enabling
not
local
DNS
cache
and
as
a
third
piece.
We
also
have
this
like
delegation
ownership
problem,
because,
as
we
have
shown,
all
the
records
are-
or
at
least
most
of
the
records
are
stored
on
AWS.
B
So
we
would
like
to
just
create
subdomains
and
delegate
the
ownership
to
some
sub
teams,
and
still
the
DNS
resolution
should,
and
last
but
not
least
so,
we
would
like
to
just
come
up
with
our
law
testing
framework,
because
whenever
years
start
talking
about
Easter
and
service
Mash,
there
is
this
additional
latency
cost,
and
we
would
like
to
make
sure
about
that.
It
is
not
gonna,
be
a
problem
for
some
of
our
customers,
so
we
are
planning
to
just
come
up
with
numbers
and
tell
them
that
okay,
this
is
the
additional
latency
cost.
If
you're.