►
From YouTube: Case Study: Integrating Azure IPv6 PrivateLink with Kubernetes - Meixing Le & Michael Wiederhold
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Case Study: Integrating Azure IPv6 PrivateLink with Kubernetes - Meixing Le & Michael Wiederhold, Databricks
Databricks offers a multi-cloud SaaS platform to enable data teams to solve the world’s toughest problems, using best in class technology and providing an open ecosystem. We run an infrastructure footprint consisting of 2M+ VMs across 40+ Cloud environments/regions, processing exabytes of data per day and generating billions of events. This talk shares Databricks’ journey of supporting IPv6 traffic with our existing Kubernetes infrastructures. At Databricks, we run 2000+ Kubernetes clusters (self-managed) across multiple cloud providers and regions with IPv4. To support the populate PrivateLink feature on Azure with IPv6, we explored various architectural options to integrate IPv6 with our existing Kubernetes clusters. After back and forth discussions and experiments, we ended up with a solution which is lightweight and fits our deployment and monitoring systems naturally.
https://sched.co/ekCW
A
After
that
mei
shing
will
discuss
some
of
the
challenges
we
faced
when
integrating
private
link,
how
we
eventually
did
our
integration
and
then
show
a
demo
of
how
customers
can
access
data
bricks
over
privately
before
we
dive
into
the
databricks
architecture.
I
first
want
to
talk
a
little
bit
about
databricks
and
the
platform
we
offer
to
customers.
A
Apache
spark
is
a
general
purpose,
analytics
engine
that
was
created
by
the
founders
of
databricks
and
it's
used
for
all
kinds
of
large-scale
analytics
jobs,
ranging
from
bi
workloads
to
machine
learning,
while
apache
spark
by
itself
is
a
great
product,
yet,
like
all
other
distributed
analytics
engines
can
be
time
consuming
to
manage.
When
you
have
large
clusters,
there
are
also
a
lot
of
different
types
of
software.
You
need
to
manage
outside
your
analytics
engine
like
notebooking
software
experiment,
tracking
software,
if
you're
doing,
machine
learning
and
so
on.
A
Databricks
realized
this
early
on
and
built
a
software
as
a
service
platform
that
runs
in
the
cloud
to
manage
all
this
complexity.
For
you,
this
means
that
your
data
scientists
can
focus
on
what
they're
good
at,
which
is
writing
analytics
jobs
and
not
on
maintaining
infrastructure,
which
is
not
in
their
area
of
expertise.
A
Since
databricks
was
founded
in
2013,
the
platform
has
grown
exponentially
and
is
used
by
many
fortune.
500
companies
for
mission,
critical
applications
as
use
of
this
platform
has
grown
so
is
databricks.
Today
we
have
over
6
000
customers,
more
than
1500
employees,
over
300
of
which
are
engineers,
and
we
have
350
million
in
annual
recurring
revenue.
A
So
why
is
databricks
so
popular?
As
I
just
mentioned,
we
provide
unified
analytics
platform
that
allows
customers
to
be
able
to
easily
write
analytics
jobs
to
get
value
out
of
their
data.
While
the
company
originally
started
out
as
a
company
built
around
apache
spark,
we
have
since
started
incorporating
other
types
of
analytics
engines
into
our
product
like
tensorflow,
so
you
have
a
choice
of
analytics
engines
to
run
jobs
on
our
platform
is
also
multi-cloud
and
runs
in
aws
and
azure.
A
We
provide
built-in
notebooking
and
reporting
and
allow
customers
to
easily
spin
up
and
spin
down
spark
clusters
in
a
cost
effective
way.
This
allows
data
scientists,
data
engineers
and
business
users
to
focus
entirely
on
the
things
they're
good
at
and
not
have
to
worry
about
the
details
of
the
underlying
platform.
A
A
The
databricks
the
data
plane
is
where
the
spark
clusters
run,
and
you
may
have
noticed
in
the
diagram
that
there
is
one
control
plane
and
a
lot
of
data
points.
There's
a
reason
for
this,
and
it's
because
databricks
uses,
what's
known
as
the
ndpc
model.
This
means
that
the
control
plane
runs
in
the
databricks
cloud
account,
but
the
data
plane
runs
in
customers
cloud
account.
A
A
A
A
These
control
planes
manage
hundreds
of
thousands
of
spark
clusters
every
day
and
launch
millions
of
vms
to
execute
customer
submitted,
spark
jobs.
These
spark
jobs,
process,
exabytes
of
data
in
order
to
produce
reports
and
other
business
insights
that
are
used
by
customers
all
right.
So,
let's
talk
a
little
bit
about
private
link.
What
is
private
link,
and
why
did
we
integrate
with
it?
A
Well,
it
turns
out
that,
even
though
customer
data
never
leaves
the
customer's
cloud
account
due
to
our
nvpc
model,
some
security,
sensitive
customers
are
still
concerned
about
the
results
of
their
jobs
or
other
potentially
sensitive
information
stored
in
a
platform
being
sent
over
the
public
internet.
Even
if
all
of
it
is
encrypted
during
transit.
A
They
want
to
be
able
to
guarantee
that
communication
to
databricks
never
occurs
over
the
public
internet
and
also
want
to
limit
access,
so
their
databricks
accounts
are
only
accessible
through
specific
endpoints
in
their
cloud
network.
In
order
to
provide
this
level
of
security,
all
major
cloud
providers
offer
a
service
called
private
link
at
a
high
level.
Private
link
has
two
components
to
it:
a
private
link
service
and
a
private
link.
Endpoint
private
link
service
is
configured
as
part
of
the
load
balancer
that
allows
traffic
into
your
vnet
and,
ultimately,
to
your
application.
A
As
part
of
the
private
link
endpoint
setup,
you
specify
a
private
link
service
that
the
endpoint
sends
traffic
to
traffic
between
the
private
link,
endpoint
and
private
link
service
always
goes
over
cloud
providers.
Private
internet,
giving
customers
a
more
secure
way
to
access
their
cloud
resource
without
any
communication
ever
leaving
their
cloud
accounts.
A
Alright,
so
that's
a
brief
overview
of
private
link
and
I'll
hand.
The
presentation
off
now
to
mei
shing
he'll
talk
about
the
challenges
we
faced
at
databricks,
integrating
private
link
and
how
we
solved
those
challenges.
We'll
conclude
the
talk
with
the
demo,
showing
a
databricks
workspace
being
accessed
over
privately.
B
B
One
private
endpoint
within
your
virtual
network
then
connect
the
private
endpoint
to
the
private
link
service
inside
the
service
provider's
virtual
network,
in
this
case
our
databricks
virtual
network,
and
then
you
can
send
your
traffic
to
this
private
endpoint
inside
of
inet,
which
is
a
local
kind
of
ipv4
address
and
azure
private
link
will
take
care
of
the
traffic
routing
using
azure
networking
instead
of
the
public
internet
and
send
the
traffic
to
the
private
link
service
or
inside
the
databricks
net.
That's
how
the
traffic
is
more
secure
and
more
private.
B
So
at
databricks
we
have
several
use
cases
to
integrate,
with
azure
private
link
to
benefit
our
customers.
The
first
use
case,
which
will
focus
mostly
in
this
talk,
is
the
user
to
web
application
traffic.
So
user
can
set
up
a
private
link
endpoint
inside
their
virtual
network
and
through
private
link.
They
can
talk
to
the
databricks
control
plan.
Web
application
yeah
they
can
use
the
notebook
there
to
launch
clusters.
Do
all
the
data
science
work
now.
There
are
other
use
cases
like
databricks
control
plan
to
databricks,
like
a
data
plan.
B
Communication
can
also
be
secured
through
the
private
link
feature.
So
what
is
the
challenge
on
the
infrastructure
side
to
integrate
with
private
link?
So
first
of
all,
databricks
is
a
first
party
service
on
azure.
What
does
that
mean?
So
actually
it's
called
azure
databricks,
so
it's
appeared
as
a
native
servicing,
azure
and
so
creating
a
databricks.
Workspace
is
as
easy
as
creating
other
resources,
for
example
virtual
machines
and
a
database
being
azure.
You
just
need
to
go
through
several
clicks.
B
Then
you
can
create
a
database
bricks
workspace
to
work
on
so
from
azure
site.
They
provided
two
types
of
private
link:
support
models.
First,
there
is
the
third
party
offering
which
is
available
to
all
the
azure
customers
and
it's
purely
on
ipv4,
so
the
second,
the
second
type
of
support
model,
is
called
the
past
version
of
private
link.
It
definitely
provides
deeper
integration
with
other
azure
services,
so
all
the
other,
like
first
third-party
service
on
azure.
B
We
have
to
use
the
path
version
of
private
link,
so
the
challenge
for
us
is
basically
there's
a
requirement.
You
have
to
accept
ipv6
traffic
on
the
control
plan
on
to
make
the
private
link
traffic
work
so
from
azure
site.
They
do
have
a
lot
of
ipv6
support
on
most
of
their
resources
in
terms
of
v-net
virtual
network
or
subnet
load,
balancer
vmss,
which
is
virtual
machine
skill
set
these
resource
all
support.
Dual
stack
like
you
can
assign
both
ipv4
and
the
ipv6
to
this
resource
at
the
same
time.
B
B
There
are
two
high-level
options
to
solve
this
problem.
The
first
is
the
proxy
solution.
We
can
convert
the
ipv6
to
ipv4
traffic
outside
of
kubernetes
and
then
just
talk
ipv4
to
the
kubernetes
services,
and
the
other
option
is
just
a
support:
ipv6
natively
in
kubernetes,
so
the
ipv6
private
link
traffic
can
directly
hit
a
service
running
on
kubernetes
yeah.
B
So
a
little
bit
about
the
background
of
running
kubernetes
at
databricks,
so
at
databricks
we
run
all
the
control
plane
surveys
on
kubernetes,
but
we
are
not
using
the
manager
kubernetes
service,
for
example,
aks
on
mainly
because
databricks
is
multi-cloud
yeah.
We
want
to
be
consistent
across
our
different
cloud
providers,
so
we
make
our
own
like
virtual
machine
images,
and
we
make
sure
these
vms
can
bootstrap
into
kubernetes
clusters,
and
we
have
a
lot
more
control
over
that.
B
We
can
make
sure
it
has
the
same
like
a
kernel,
os
version,
kubernetes
version
and
it's
easier
to
support
our
own
services
in
terms
of
the
configuration
we
totally
disable
the
ipv6
from
the
kernel
level,
because
we
don't
need
that
before
supporting
private
link
and
the
cli
plugin
we're
using
flanno,
the
container
runtime
is
darker,
the
version
we're
running
is
kubernetes
version.
One
run
is
a
v1016.
B
B
So
we
first
explored
the
first
option
like
a
proxy
solution
outside
kubernetes
cluster.
This
solution
is
also
used
by
some
other,
like
azure
internal
service.
So
the
basic
idea
is
a
simple.
Within
the
same
v-net,
we
can
provision
a
dedicated
load
balancer,
which
accepted
the
ipv6
private
link
traffic.
B
This
loaded
balancer
can
send
traffic
to
a
back-end
like
a
vmss
virtual
machine,
a
skill
set.
Then
we
can
run
the
private
link
proxy
on
this
vmss,
which
will
terminate
the
ipv6
traffic
proxy
to
ipv4,
then
again
talking
to
the
kubernetes
cluster,
so
the
kubernetes
cluster
load
balance
also
serving
the
public
traffic
which
is
ipv4
from
the
rest
of
the
users.
B
So
this
seems
to
be
a
straightforward
solution,
but
there
are
a
lot
of
challenges.
Actually,
the
first
question
is:
how
can
we
deploy
this
proxy?
Do
we
deliver
it
as
a
virtual
machine
image,
because
it's
wrong
on
top
of
a
virtual
machine
skill
set
yeah?
Do
we
deliver
as
a
virtual
machine
image
or
container
image?
It
is
completely
outside
kubernetes
right.
So
how
do
we
actually
deploy
it?
We
cannot
use
like
a
kubocity
or
commands.
B
Maybe
we
can
run
some
docker
commands
if
we
run
as
a
container
or
maybe
we
just
deployed
a
virtual
machine
using
the
virtual
machine
image,
but
we
don't
have
this
support
model
right.
It
will
be
a
completely
new
kind
of
service.
We
support,
and
the
other
question
is
how
how
we
monitor
this
service
right,
the
metrics
logins
and
these
kind
of
stuff.
We
only
have
native
support
from
communities.
B
We
don't
have
such
a
use
case
like
a
virtual
machine
skills
outside
the
communities,
then
provide
metrics
and
loggings.
So
these
are
the
problems
to
use
this
proxy
solution
and
then
because
of
that,
we
also
explore
the
second
option,
which
is
to
support
kubernetes
natively
with
ipv6,
so
because
we
still
need
to
serve
ipv4
traffic
for
the
public
like
access
which
are
not
through
private
link.
B
So
we
have
to
use
the
dual
stack
feature
in
kubernetes
if
we
choose
to
have
native
ipv6
support,
if
we
use
that
feature
dual
stack
on
the
load
balance
level,
we
can
accept
that
both
ipv6
and
v4,
the
overall
architecture,
looks
like
a
simpler,
which
is
good.
However,
it
this
option
has
its
own
challenges
the
first.
The
first
item
is
like
the
stability
concern
on
the
version
we're
running
on
1.16
kubernetes.
B
Maybe
several
services
and
like
ports
will
require
ipv6
support,
but
most
of
our
kubernetes
workload
will
not
require
ipv6,
and
at
last
it
could
be
a
huge
engineering
effort
right
about
prototyping
and
testing
to
make
sure
everything
works.
If
we
enable
the
dual
stack
feature
in
kubernetes
yeah,
but
we
do,
we
did
some
investigation
on
like
surrounding
the
dual
stack
on
communities
yeah.
First
of
all,
don't
confuse
with
like
ipv6
single
stack
feature
in
kubernetes,
which
already
entered
the
rfr
in
1.9
and
move
the
two
beta
in
1.18
yeah.
B
B
Actually,
that
will
not
affect
our
use
case
for
the
dual
stack
yeah
and
once
you
enable
the
dual
stack
it
will
assign
both
ipv4
and
the
v6
ip2,
literally
every
parts
running
kubernetes,
but
for
the
service
level
you
need
to
have
separate
service
for
like
a
one
for
ipv4
and
the
one
dedicated
service
for
ipv6
yeah
to
run
the
dual
stack.
There's
also
some
networking
prerequisites.
B
B
Second,
because
every
port
will
have
both
ipv4
and
ipv6
address.
So
the
container
networking
interface
you
choose,
the
cmi
must
support
the
dual
stack
as
well
as
we
are
using
flannel.
Actually,
flannel
will
not
support
a
dual
stack
feature
and
the
like
known,
cli
plugins,
can
better
support
the
dual
stack
feature:
kubernetes
and
calico,
but
least
case
it's
case
by
case
on
different
cloud
providers.
B
B
Yeah.
Can
we
combine
the
above
two
options
together,
whether
it's
possible
to
move
the
proxy
into
the
kubernetes?
Then,
if
we
do
that,
we
will
definitely
get
the
deployment
and
the
monitoring
tunnel
for
free
yeah.
We
know
how
to
deploy
kubernetes
workload
and
we
have
native,
like
metrics
logging
in
the
kubernetes
yeah.
So
on
the
load
balancer
level,
it
does
support
dual
stack.
B
B
So
is
it
possible
to
deploy
the
proxy
as
like
a
ports
on
kubernetes
but
give
the
virtual
machine
level
networking
to
the
parts
yeah?
There
are
definitely
some
features
called
use
host
network
where
to
work
for
the
dual
stack
yeah.
That's
on
the
third
option
we're
trying
to
explore.
We
don't
know
whether
it
works
or
not.
Then
we
prototype
it
and-
and
luckily
it
works
pretty
well-
and
so
here
comes
to
our
solution,
yeah,
basically,
and
to
make
it
what
came
to
end
right.
B
B
B
So
once
the
traffic
hit
heated
ipv6
ip
on
the
load
balancer,
it's
only
go
to
this
vmss
and
inside
the
kubernetes.
We
also
have
a
dedicated
order
pool
for
these
vmss
so
that
only
the
proxy
workload
will
come
will
running
on
this
node
pool,
not
interfering
with
other
kubernetes
workloads
yeah.
So
then
we
deployed
the
private
link
proxy,
which
is
a
v6
to
v4
proxy.
B
So
if
you
use
host
networking
and
enable
the
pod
security
policy
yeah
just
make
sure
you
specify
the
policy
to
allow
the
parts
to
use
host
network
then
make
sure
the
load.
Balancer
rules
are
correctly
configured
to
send
the
traffic
to
this
proxy
port
and
also
make
sure
you
do
the
proper
white
listing
on
the
private
link
on
traffic
so
that
before
a
v6,
trafficker
can
come
into
your
kubernetes
and
then
yeah
the
proxy
works.
So
we
got
a
lot
of
benefit
from
this
solution.
B
Then
it's
the
department
is
managed
by
kubernetes.
It's
very
easy!
If
we
want
to
update
the
proxy
and
it's
it's
deployed
as
a
stateless
deployment,
so
as
the
traffic
load
increase,
it's
very
easy
to
scale
up
just
increase
the
upcoming
replica.
Okay,
no
more
traffic
and
next
item
is
like
we
definitely
get
the
kubernetes
native
monitoring
analogy
yeah
at
last.
This
solution
will
also
work
for
other
use
case.
For
example,
our
data
plan
to
contribute
on
traffic
using
parameter
links
yeah.
B
This
is
some
screenshot
as
a
demo,
so
we
provisioned
this
private
link,
node
pool
inside
the
kubernetes
cluster,
which
is
a
special
vmss,
except
like
a
both
v4
and
v6
traffic.
So
when
you
deploy
the
proxy
port,
yeah
just
make
sure
have
the
host
networking
and
make
sure
it's
scheduled
on
this
private
link,
another
pool.
B
B
If
you
look
on
the
host
vm
so
on
the
azure
loader
balancer
side,
if
we
look
at
the
ip
yeah
we're
providing
a
ipv6
for
this
purpose
and
then
it
works
when
we
set
up
the
private
link,
if
you
you
are
using
your
user
like
a
v-net,
you
look
for
this
url.
It
will
return
you
a
local
ip
address.
You
just
need
to
connect
it
to
this
local
ip
address.
B
Actually
it
will
show
the
azure
data
breaks,
workspace
and
yeah,
since
your
traffic
is
routed
through
the
private
link,
yeah
to
recap:
yeah,
not
necessarily
to
integrate
with
azure
private
link.
But
if
you
need
need
any
like
ipv6
traffic
support
in
your
like
ipv4
kubernetes
yeah,
then
you
can.
You
can
use
this
approach.
Basically,
you
enable
the
dual
stack,
always
ipv6
everywhere
on
the
cloud
provider
infra
level
right,
then
you
can
set
up
with
a
v62
v4
proxy.
You
can
just
deploy
it
as
regular
kubernetes
deployments
yeah.
B
Just
that
you
use
the
host
networking.
Then
you
can
receive
the
ipv6
traffic.
Now
you
can
proxy
to
v4
and
then
the
router
will
work
all
the
way
to
your
kubernetes
yeah.
This
is
a,
I
think,
a
good
way
to
support
ipv6
traffic
with
ipv4
on
kubernetes.
If
you
don't
want
to
use
the
dual
stack
feature
yet
so
yeah
we
delivered
this
azure
databricks
private
link
is
in
private
preview
now
and
it's
also
available
in
azure.com.