►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Building Multi-Cloud Service Meshes at Snowflake - Charles Xu, Snowflake
Snowflake (NYSE: SNOW) products are multi-cloud. So is its infrastructure. Multi-cloud is hard because of cloud-specific primitives and cross-cloud feature disparity, but at Snowflake, we do it with hundreds of Kubernetes (k8s) clusters and millions of non-k8s VMs. This talk discusses the challenges we faced and the lessons learned. Some topics include:
How service mesh is critical in cloud-agnostic architectures
Our use cases of HTTP, mTLS, and TCP ingress, and the surprises with persistent TCP connections
Autoscaling ingress gateways while preserving source IP address at Layer 3
Blue-green upgrade the control plane and gateways: why traffic shifting by DNS updates is bad, and how we do it without DNS and in cloud-agnostic way
Open questions with multi-cloud that service mesh could not solve
A
Hey
everyone!
Thank
you
so
much
for
being
here.
My
name
is
Charles
and
I
work
at
Snowflake,
and
today,
I
would
love
to
tell
you
more
about
our
service
match
adoption
stories
at
snowflake,
specifically
because
snowflake
Prada
is
multi-cloud.
It
means
our
infrastructure
also
has
to
be
multi-cloud,
so
a
sizable
amount
of
challenges
that
we
Face
actually
stem
from
this
multi-cloud
or
Cloud
agnostic
requirements
going
too
fast
before
we
move
in
just
a
bit
more
about
me.
A
I
work
on
the
container
platform
team
at
snowflake
joint
snowflake
like
more
than
two
years
ago
and
before
snowflake
I,
was
at
Cruz,
also
doing
container
stuff
cruises
self-driving
car
startup
and
before
cruise
I
was
actually
at
Google
building
the
sdo
open
source
project
so
being
in
the
in
this
domain.
For
a
little
bit.
A
A
There
are
four
properties
that
I
think
stands
out
for
snowflake
in
general.
First
is
multi-cloud.
As
of
today
we
support
Azure,
gcp
and
AWS,
and
maybe
adding
more
in
the
future.
It's
multi-tenant
snowflake
has
more
than
one
customers,
and
sometimes
more
than
one
more
than
one
customers
would
share
the
same
compute
resources
and
it's
multi-region.
We
want
our
compute
resources
to
be
co-located
with
our
customers,
workloads
and,
lastly,
is
multi-environment
we're
supporting
commercial
deployments
as
well
as
Government
Cloud.
A
Specifically
about
container
platform,
as
of
today,
we
run
170
clusters.
Those
are
kubernetes
clusters
and
we
expect
to
build
a
lot
more
in
the
future.
We
intentionally
tried
to
limit
the
size
of
the
cluster
trying
to
limit
the
blast
radius,
so
we're
orchestrating
hundreds
of
clusters
as
heard
instead
of
pads.
That
means
we
also
have
a
strong
requirements
of
building
scalable
resilient
automations
to
do
really
simple
things
like
cluster
upgrades
or
upgrading,
all
the
open
source
components
that
we
deployed
on
the
Clusters,
like
external
DNS
or
a
certain
manager.
A
We
are
using
the
cloud
provider
managed
kubernetes,
so
they
take
care
of
the
control
planes
and
we
manage
the
worker
nodes
and
all
of
the
workloads
and
open
source
components
we
deploy
onto
them.
So
that's
one
last
thing
we
have
to
worry
about
is
multi-tenant
just
like
snowflake
product
in
terms
of
namespace
and
also
applications
running
in
the
cluster.
Sometimes
the
same
nodes
could
run
20,
30,
different
containers
and
they're
from
different
namespace
different
applications.
A
A
Api
calls
and
authentication
is
managed
differently
and
all
those
little
pieces,
they're
they're,
creating
frictions
in
terms
of
building
automations
like
essentially
you're
writing
the
same
code
three
times.
If
you
want
to
support
three
Cloud
providers
so
having
some
Cloud
agnostic,
abstractions
really
reduce
the
complexity
for
both
development
and
operation
and
service
mesh
is
a
key
part
of
it,
and
part
of
the
reason
is
that
we
can
push
this
policy
up
into
this
Cloud
agnostic
layer
for
istio.
For
example.
A
Four
layer,
Network
policies
and
that's
specifically
for
applications
running
outside
of
the
mesh
Opa
gatekeeper
is
a
very
flexible
and
Powerful
tools
for
us
to
Define
policies
like
we
only
want
this
list
of
container
images
from
this
list
of
hosts
to
be
running
on
this
cluster,
for
example,
instead
of
having
three
different
Celli
tools
to
access,
VM
or
kubernetes
will
use
teleport
to
manage
the
the
access
and
also
integrate
it
with
OCTA
for
identity
and
group
membership.
We're
using
group
membership
mostly
for
our
back
and
access
policy
definition.
A
And
lastly,
we
use
snowflake
internally
to
query
the
logs
and
ingesting
the
metrics,
as
well
as
defining
alerts.
Well
speaking
of
Doc,
putting
your
own
product,
okay,
I,
want
to
dive
into
the
service,
match.
Adoption
story
right
now
and
I
want
to
begin
with
all
the
scaling,
the
Ingress
Gateway.
Now,
obviously,
you
wanted
all
the
scalar
Ingress
Gateway,
because
your
traffic
is
pretty
much
unpredictable
and
also
during
the
night
and
weekends,
the
traffic
are
relatively
small.
A
You
want
to
save
money
by
all
the
scaling
and
I
want
to
tell
you
why
it
is
more
than
just
slap
and
horizontal
part,
all
scaling
setting
to
your
Ingress
Gateway
and
then
call
you
a
day,
and
the
reason
is
because
we
want
to
preserve
the
source
IP
address
at
layer,
3.,
preserving
The,
Source
IP
is
important
for
rate
limiting
and
also
allow
list.
Policy
enforcement
and
the
export
for
HTTP
header
is
susceptible
for
spoofing
and
we
also
have
TCP
services
that
doesn't
speak
HTTP.
A
It
has
this
interesting
Behavior
where
traffic
is
routed
to
the
nose,
essentially
at
no
port,
and
then
it's
IP
tabled
by
the
qproxy
to
the
pods.
So
there's
like
one
extra
hop
on
the
nodes
and
then
to
the
pods,
and
this
Behavior
comes
important
when
it
comes
to
IP,
address
preservation
and
the
traditional
L4
low
balancer
setup.
A
Has
this
periodic
health
check
to
the
nose
to
make
sure
that
there
are
actually
pods
on
the
Node
serve
in
the
traffic
and
on
AWS
it
happens?
Maybe
every
20
seconds.
This
is
configurable,
but
this
is
a
non-zero
delay,
and
that
means
life
cycle
management
for
Gateway
pause
is
very
challenging
because
when
the
Pod
is
terminated
on
node
n,
the
load
balance
would
keep
sending
traffic
to
node
and
until
node
n
is
smart
as
unhealthy,
and
during
that
short
period
your
clients
could
receive
503s
or
just
I
o
timeout,
because
there's
no
response.
A
And
this
is
shameful
to
say,
but
for
a
while
we
don't
have
Auto
scaling
setups,
we
just
over
provisioned
the
istio
Ingress
gateways
and
in
order
to
preserve
the
source,
IP,
here's
our
setup.
We
run
the
istio
gateways
as
demon
sets
and
we
have
a
dedicated
notebook
for
istio
and
within
that
polls.
Every
node
had
an
Ingress
Gateway
pod
running
on
it
to
handle
the
traffic,
and
we
sat
the
external
traffic
policy
to
local.
A
A
A
The
way
to
do
it
safely
without
dropping
traffic
is
that
we
first
need
to
deregister
the
hosting
node
from
the
cloud
load
balancer
and
making
sure
that
the
registration
completes,
and
then
we
issue
the
Pod
delete,
request
and
there's
a
way
to
do
so
in
a
cloud
agnostic
Way.
By
applying
this
note
label
and
then
according
the
node.
But
of
course
you
have
to
develop
some
custom
tooling
to
basically
pull
the
cloud
provider
API
to
make
sure
that
your
registration
completes
successfully
because
everything
happens.
Asynchronously.
A
So,
but
we
really
want
Auto
scaling
because
of
the
cost,
saving
and
also
just
to
be
safe.
When
there's
a
traffic
Spike
hitting
us.
Our
first
design
is
almost
a
perfect
straw
man
and
feels
very
intuitive
and
easy
to
implement,
which
is
to
use
an
admission
web
hook
to
intercept
all
the
plot,
delete
requests
for
the
Gateway,
Ingress
pod
and
then
just
hold
that
request
and
at
the
same
time,
to
register
the
the
node.
That's
hosting
the
Pod,
from
the
load
balancer
and
within
the
registration
completes
release
the
partly
request.
A
So
where
do
we
go
from
here?
There
are
three
possible
solutions.
The
first
is
to
explore
app
data
plane
to
replace
the
iptable
based
Q
proxy,
which
has
the
really
nice
property
of
still
retaining
The
Source
IP
address,
despite
setting
the
external
traffic
policy
to
Cluster,
so
that
you
can
place
your
pod.
A
A
The
second
option
is
to
use
a
custom,
no
pull
auto
skater
that
pretty
much
just
worked
around
the
30
second
restriction,
that
I
talked
about
for
the
admission
web
hook,
but
because
this
is
on
a
no
pull
auto
scaling
layer,
there's
really
not
a
timeout
around
node
deregistration
and,
lastly,
there's
a
kubernetes
enhancement
proposal
trying
to
address
this
exact
issue
that
I
just
described.
The
issue
is
that
it's
going
to
be
a
long
wait.
A
Upgrading
the
170
clusters
manually
would
be
insane
because
it
would
be
a
lot
of
work
and
very
error
prone,
so
it
requires
some
custom,
toolings
and
automations.
Specifically
internally.
We
don't.
We
Define
this
process
by
following
this
blue
green
upgrade
process.
The
Brooklyn
upgrade
process
has
two
nice
properties
we
can
validate.
A
The
new
is
the
release
before
shifting
the
workloads
over
and
we
can
roll
back
quickly
if
the
new
version
has
some
bugs
or
it
doesn't
interoperate
with
the
old
istio
version
that
we're
currently
running
so
I
want
to
show
you
a
really
quick
animations
of
how
the
blue
green
upgrade
Works.
So
we
were
on
the
same
page.
This
process
is
inspired
by
the
canary
release
that
the
istio
community
proposed,
but
let's
assume
that
every
all
these
new
components
is
on
Blue
right
now.
That
includes
the
control
planes
and
the
sidecar
proxy.
A
The
first
thing
we
do
is
to
deploy
a
new
version
of
istio
control,
plane
colleague
ring,
and
then
we
will
select
a
subset
of
namespaces
change.
The
you
know
the
is
the
revision
namespace
label
to
Green,
so
that
a
new
version
of
sidecar
proxy
is
injected
now
notice
that
we
we're
running
blue
and
green
meshes
at
the
same
time
in
the
same
cluster.
A
This
is
important
because
by
separating
a
pair
of
cluster
and
client
into
the
two
meshes,
we
can
test
the
interoperability
of
the
two
versions
of
his
Dios
and
the
test
is
basically
up
to
your
applications,
like
the
sets
of
features
that
you
use.
Mtla
is
really
limiting
traffic
shifting
Etc
and
lastly,
we
update
the
rest
of
the
namespace
to
the
green
mesh
as
well
and
then
eventually
retire.
The
blue
control
plane.
So
that's
very
intuitive
one
caveat
is
that
before
we
we
retire
the
old
control
plane.
A
We
need
to
make
sure
that
the
client
traffic,
all
the
client
traffic
has
been
drained
to
the
new
istio
gateways
and
because
we
have
essentially
two
separate
deployments
of
istio
service
meshes.
The
gateways
are
fronted
by
different
sets
of
load
balancers
and
that's
an
issue
because
to
drain
the
the
client
traffic
over
it
just
means
that
we,
the
first
implementation,
is
to
do
a
DNS
cutover,
so
that
we
update
the
DNS
8
record
for
the
cluster
Ingress
gateway
to
point
to
the
new
IP
address
of
the
new
load
balancer.
A
This
is
a
a
diagram
of
showing
how
the
DNS
update
works.
So,
on
the
upper
hand,
side
is
me
updating
a
DNS
worker
through
our
Authority
and
that
server
and
whichever
DNS
resolver,
eventually
picks
up
that
update
and
then
I
will
return.
The
new
response
to
the
client
and
the
response
can
be
pointed
to
either
blue
or
green
load
balancer.
A
Now,
there's
a
lot
of
issue
with
dnas,
primarily
because
DNS
we
have
no
control
over
client
caching
for
DNS,
which
is
the
second
point
right
here.
Where
shift
traffic
shifting
using
DNS
is
very
ineffective
in
practice,
we
noticed
that
after
we
updated
DNS
records
after
two
weeks,
there's
still
traffic,
hidden,
the
old
load
balancer
and
there's
nothing
we
can
do
about
it.
A
We
basically
just
set
a
deadline,
and
then,
after
that,
we
gave
up
and
just
retire
the
load
balancer,
but
going
back
to
the
top
where,
for
some
cloud
provider,
external
DNS
updates
are
not
Atomic,
it's
done
by
two
API
calls
delete
and
then
update
it's
possible
for
delete
to
succeed
and
then
update
to
fail.
So
then,
essentially,
all
the
DNS
resolution
would
return
NX
domain
and
that's
not
great
and
what's
worse,
is
that
your
client
can
cache
the
NX
domain
response
indefinitely
and
there's
nothing.
A
A
A
The
way
to
achieve
this
common,
sensible
balancer,
is
to
define
a
service
object
without
label
selection
and
and
configure
an
endpoints
object
manually
now.
This
is
important
because
if
you
use
a
label
selector
on
the
surface
object,
you
can
only
select
pause
from
the
same
namespace
where
the
service
resides,
but
because
we
have
is
the
blue
and
it's
still
green
and
they're
from
different
name
spaces.
We
do
want
to
support
pause
from
multiple
namespaces.
A
So
that's
why
we're
using
service
without
a
label
selector
and
we're
defining
this
endpoints
object
to
be
a
list
of
IP
addresses
of
the
selected
pause
and
which
could
be
a
mix
of
blue
and
green
Gateway
pods,
and
we
build
a
really
simple
controller
that
manage
the
endpoints
object
for
us.
So
we
don't
have
to
do
this
manually
because
pods
are
by
definition
in
ephemeral.
A
So
here's
a
really
quick
animation
of
this
common
lb
setup.
Initially,
the
common
load
balancer
in
Orange
points
to
the
the
blue
istio
mesh,
and
then
we
deploy
a
new
version
of
istio
and
notice
that
it
still
comes
with
its
own
load.
Balancer
and
those
are
ephemeral.
They
are
there
just
for
testing,
because
we
want
to
make
sure
that
traffic
can
actually
hit
the
is
still
green
gateways
and
in
order
to
test
the
new
gateways,
they
have
to
be
behind
some
load
balancer.
A
But
if
we
add
it
as
a
backend
for
the
the
common
load
balancer
in
yellow,
they
receive
production
traffic
instantaneously
and
that's
not
what
we
want
it.
So
we
have
to
have
a
separate
set
of
load
balancer
anyway,
just
for
testing,
and
then
we
update
the
endpoints
object
within
the
kubernetes
cluster
to
select
the
different
sets
of
Gateway
parts.
A
A
Okay
well
looks
like
I
got
some
time,
so
I
want
to
cover
this
kind
of
appendix
that
I
prepared
some
open
questions
with
multi-cloud,
istio
and
kubernetes
would
help
with
your
multi-cloud
adoptions,
but
it's
not
Panacea.
Specifically,
there
are
issues
that
has
been
addressed,
for
example,
the
application.
That's
that's
used
in
the
cloud
provider.
Services
still
have
to
use
cloud,
specific
libraries,
and
that
means
the
same
logic
need
to
be
implemented
three
times.
A
It's
the
part
identity
which,
on
gke
is
called
vocal
identity,
which
translates
a
kubernetes
service.
Account
to
a
cloud
service
account
or
role
depends
on
your
cloud
provider.
I
mean
that
only
solves
authentication
authorizations
to
the
cloud
resources
and
doesn't
solve
the
actual
API
Clause
of
interacting
with
the
cloud
and
custom
resource
definitions
can
solve
your
resource
provisioning,
but
it
doesn't
again,
it
doesn't
solve.
Interactions
of
your
application
is
actually
using
the
message
queue
or
the
bucket.