►
From YouTube: Kubernetes Everywhere: Lessons Learned From Going Multi-Cloud - Niko Smeds, Grafana Labs
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Kubernetes Everywhere: Lessons Learned From Going Multi-Cloud - Niko Smeds, Grafana Labs
Many companies are interested in deploying their products across multiple cloud providers, but few actually see it through. While benefits like avoiding provider lock-in and increased uptime during provider outages are attractive, several factors are important to consider. Grafana Labs successfully deployed across AWS, Azure, DigitalOcean, GCP, and Linode in more than 30 regions; that includes inter-cloud network connections. This talk will explore some of the large and subtle differences in networking and managed Kubernetes services between said providers. We’ll discuss the approaches we took while scaling our infrastructure across multiple environments, the challenges we faced, and what worked in the end.
A
Hello,
everybody
thank
you
for
joining.
I
want
to
start
by
asking
how
many
of
you
run
kubernetes
on
at
least
one
cloud
provider
pretty
much
everyone
how
many
of
you
run
kubernetes
on
two
or
more
cloud
providers,
probably
like
half
the
room,
so
we're
here
to
talk
about
going.
Multi-Cloud
you've
got
one
and
you
want
another
for
those
of
you
expanding
to
multiple
providers.
A
I'm
going
to
share
some
of
our
experiences
on
a
recent
project
and
the
goal
is
to
give
you
an
idea
of
what
was
required
and
what
to
watch
out
for
for
those
of
you
already
on
multiple
clouds.
I'm
curious,
if
you
can
relate
to
some
of
these
experiences,
if
you
have
different
issues,
run
into
or
suggestions,
I'd
be
interested
to
chat
after
the
talk,
my
name
is
nico.
A
A
A
A
At
grafana
we
have
a
synthetic
monitoring
product
which
allows
customers
to
deploy
like
health
probes
at
different
locations
around
the
world.
Think
things
like
icmp
or
http
health
checks
and
to
support
specific
locations.
We
have
to
deploy
this
product
across
multiple
providers,
there's
also
vendor
lock-in.
A
A
A
A
A
We
had
to
create
the
aws
organizations
and
accounts
the
networking
resources,
so
things
like
vpcs
vpns,
the
im
users
and
policies
so
that
engineering
staff
had
access
to
the
cloud
services,
their
products
relied
on
the
actual
kubernetes
clusters
and
a
number
of
essential
workloads.
So
I'm
going
to
touch
on
a
few
of
these
items
regarding
vpns,
we
connect
all
the
clusters
of
the
same
status
to
each
other
think
this,
like
dev
clusters
and
prod
clusters-
and
this
is
completed
using
each
provider's,
managed
vpn
service.
A
So
gcp
azure
aws
and
there's
a
few
reasons
we
do
this,
like
many
of
you
have.
We
have
prometheus
running
in
the
clusters,
scraping
all
the
cluster
metrics,
but
we
also
have
like
a
global
meta-modern
prometheus
server
servers
which
scrape
all
those
prometheus
servers
to
make
sure
they're
actually
healthy
and
alive.
A
A
I
was
reading
last
year's
cncf
survey
and
it
turns
out
more
than
three
quarters
of
companies.
Use
managed
kubernetes
platforms,
so
I
I
suspect
this
is
something
that
many
of
you
can
rely
on
relate
to
regarding
the
essential
workloads.
We
want
prometheus
installed,
scraping
system
and
service
metrics
internally
and
sending
alerts
to
that
global
alert
manager.
A
A
We
want
flux
which
we
use
for
continuous
deployment.
It
basically
syncs
objects
for
from
a
git
repository
into
our
kubernetes
clusters.
We
want
an
external
secrets
operator
which
calls
back
to
our
central
vault
cluster
and
sync
secrets
from
vault
into
the
kubernetes
name
namespaces,
and
it's
not
a
complete
list,
but
it
covers
most
of
the
essentials.
A
A
All
right
so
you've
been
tasked
to
build
cloud
infrastructure
on
a
new
provider.
Where
do
you
start
before
we
begin?
We
need
clear
requirements.
It
sounds
pretty
obvious.
I
thought
I'd
mention
it
anyway.
We
have
to
ask
ourselves
questions
like
what
workloads
are
we
going
to
be
creating
and
what
won't
we?
A
A
A
A
We
run
the
majority
of
our
workloads
on
kubernetes.
So
we
start
there
start
reading
about
amazon,
ecos,
amazon,
eks,
amazon's,
managed
kubernetes
service
begin
by
reading
documentation
watching
conference
videos
reading
blog
posts
trying
to
learn.
You
know
what's
new,
what's
different,
maybe
open
up
the
aws
console
spin
up
a
cluster
and
just
get
familiar
and
time
passes
and
we
start
developing
a
plan.
A
A
A
When
we
look
at
vpcs
comparing
gcp
and
aws,
it
turns
out
that
in
gcp
vpcs
are
global
resources
a
lot
of
acronyms.
What
do
I
mean
by
this?
I
mean
that
you
can
have
multiple
subnets
from
multiple
regions
in
a
single
vpc
and
it
turns
out.
We
actually
do
this.
We
have
some
use
cases
where
we
have
a
shared
vpc
with
many
clusters
across
many
regions.
A
A
A
We
create
subnets
from
the
10.0
ranges
for
things
like
nodes,
pods
and
services,
and
we
create
a
subnet
from
the
17216
for
the
gke
managed
control
plane.
When
we
look
over
aws
now
you
can
have
multiple
subnets,
but
they
must
all
be
from
the
same
private
range.
So
we're
just
going
to
use
10.0
for
everything.
A
Again,
it's
a
small
difference,
but
it
means
that
we're
creating
a
bit
more
drift
again
between
our
providers
and
finally,
in
the
networking
side
regarding
subnets
gcp
supports
up
to
a
slash
eight
in
the
largest
subnet
size,
whereas
aws
the
largest
is
slash
16.,
it
shouldn't
be
a
problem.
You
know
that's
over
65
000
addresses,
but
we've
been
deploying
our
pods
in
slash
14s
on
gcp.
A
All
these
small
differences
meant
that
we
had
to
refactor
our
ip
reservation
plan.
Basically,
you
know
I
mentioned.
We
connect
our
clusters
by
vpn,
so
we
need
to
ensure
that
there's
no
private
range
overlap,
and
we
really
didn't
consider
this.
I
think
when
we
first
created
that
plan
when
we
ran
solely
on
gcp
there's
a
few
other
things
we
can
compare.
A
A
If
we
look
at
volumes
so
network
attached
volumes,
persistent
volumes,
the
prices
and
performance
will
differ
slightly
between
provider
and
then
finally
object
storage.
At
grafana
cloud.
We
have
three
products:
mimir,
loki
and
tempo,
that
use
object,
storage
as
their
primary
data
store.
It
means
that
we
read
and
write
a
lot
to
the
gcs
and
s3
buckets,
and
it
turns
out
that
these
cloud
providers
actually
rate
limit
access
to
object,
storage,
but
based
on
different
conditions.
A
A
A
A
Initially,
it's
slower
progress
right,
you're,
going
to
have
to
actually
figure
out
the
terraform
resources
and
then
write
the
code
to
create
it.
But
it's
worth
the
trade
off
you'll
get
benefits
like
peer
review
and
suggestions
not
only
from
the
team
working
on
the
project,
but
others
in
the
company
who
are
familiar
with
these
tools
and
services.
A
A
A
A
So
the
second
lesson
is,
if
that
you
find
yourself
in,
if
you
find
yourself
stuck
in
documentation,
now
is
just
get
started.
Like
often,
the
best
way
to
learn
is
by
doing,
and
if
you
use
some
infrastructure
as
code
tool,
it's
really
easy
to
tear
down
iterate
and
rebuild,
and
the
earlier
you
hit
these
limitations.
The
earlier
you
can
plan
around
them.
A
A
We
basically
generate
credentials
for
the
service,
account,
save
them
into
vault.
That
external
secrets
operator
reads
a
secret
and
creates
a
kubernetes
secret
that
the
pods
can
use,
and
this
works.
You
know
in
summary,
we're
generating
saving
and
rotating
credentials,
but
it
needs
it
means
they
need
to
be
managed
and
it
means
they
can
be
leaked.
A
You
know
with
hundreds
or
thousands
of
credentials,
you
can
see
how
this
becomes
cumbersome
before
I
move
on,
I
should
mention.
Gcp
does
offer
an
alternative
to
this,
which
is
managed
identities.
However,
when
we
first
created
these
gke
clusters,
there
was
a
limitation
to
them,
so
we
opted
to
use
the
surface
account
credentials
directly.
A
A
So
the
third
lesson
is:
don't
blindly
do
what
was
done
before
it's
easy
and
beneficial
to
want
to
keep
things
as
similar
as
possible.
You
know
the
less
drift
between
providers,
the
easier
it
is
for
product
teams
to
onboard
and
start
deploying
their
applications
and
the
easier
it
is
for
the
infrastructure
team
to
manage.
A
A
We
have
that
first
eks
cluster
up
and
running
we've
installed
a
number
of
add-ons
things
like
the
cluster,
auto
scaler
load,
balancer
controller
and
custom,
networking
which
weren't
provided
by
default
and
all
those
essential
workloads
I
mentioned
earlier,
are
deployed
and
running
we're
really
at
an
exciting
stage
in
the
project.
It's
time
to
invite
internal
product
teams
to
deploy
their
products.
Aws.
A
A
A
So
we
found
our
apology
failing
to
start
due
to
docker
docker
hub
rate
limits.
Why
didn't
we
see
this
before?
Well,
it
turns
out
gcp
was
providing
us
an
image
cache
by
default,
and
we
just
weren't
aware
of
it
again.
It
was
fairly
simple
to
fix.
You
know
we
installed
a
docker
registry
mirror
internally
and
acts
as
a
pull
through
cache,
and
we
update
all
the
node
pools
to
utilize
that
internal
mirror.
A
So
the
fourth
lesson
is
load
test.
Your
applications
grafana
acquired
k6
around
one
year
ago
and
we've
begun
adopting
it
internally,
but
there
are
other
options
in
this
area.
It's
really
important
to
stretch
your
applications
to
the
limit
and
discover
issues
before
your
customers
do.
Have
we
not
run
these
load
tests?
It's
very
possible
we'd,
be
running
applications
with
degraded
performance
and
we'd
face
scaling
issues
when
it
mattered
most
right.
We
had
the
skill
actually
due
to
customer
demand.
A
You
get
the
added
benefit
that
you
get
a
benchmark:
the
performance
between
different
providers.
In
our
example,
the
new
provider
didn't
perform
as
well
initially,
but
perhaps
it
actually
performed
better
well.
That
would
have
given
us
a
reason
to
go
back
and
examine
how
we're
configuring
things
on
the
previous
provider.
A
A
We
need
to
talk
about
quotas
when
you're
building
on
a
new
cloud
provider
like
you're,
creating
new
accounts,
you
start
with
very
small
default
limits,
and
it's
important
to
remember
that
not
all
regions
are
the
same
right.
These
regions
are
supported
by
actual
physical
data
centers
somewhere
in
the
world
and
as
consumers
of
public
cloud
services,
we're
basically
blind
for
the
capacity
of
these
regions.
I
have
no
idea
if
region
a
or
region
b
has
a
larger
capacity
behind
the
scenes.
A
A
So
the
fifth
lesson
is:
pay
attention
to
quotas,
know
which
quotas
you'll
need
to
increase
and
submit
those
increase.
Requests
early
understand
that
you're
at
the
mercy
of
a
physical
data,
center's
capacity
and
monitor
and
alert
on
quotas.
You'd
want
to
find
out
that
nodes
are
failing
to
start
because
we've
hit
a
quota
and
nobody's
opened
a
support
ticket.
Yet.
A
So,
to
recap,
the
five
lessons
provider
services
are
similar
but
they're
not
the
same
understand
the
additional
complexity,
you're
you're
kind
of
opting
to
go
opting
to
undertake
when
you
go
for
multi-cloud
just
get
started,
don't
get
stuck
in
documentation
hell,
you
know
the
earlier
you
get
started
the
earlier.
You
discover
problems
and
limitations.
A
Don't
blindly
do
what
was
done
before
look
for
possible
improvements
throughout
the
project,
not
only
for
your
new
environments
but
the
old
environments
as
well
load
test.
Your
applications
discover
limits
and
issues
before
your
customers
do
and
know
your
quotas
save
yourself.
Some
headache,
pay
attention
to
quotas
and
understand
that
the
cloud
is
not
infinite.
A
A
It
just
worked
once
we
had
all
the
clusters
up
and
running,
we
were
able
to
use
our
standard
tooling
to
deploy
the
majority
of
workloads
without
modifications,
and
I
guess
that's
exactly
what
we're
aiming
for.
You
know
this
is
the
perk
of
using
a
standard
interface
for
deploying
and
managing
your
applications.
A
The
majority
of
issues
we
face
that
I've
discussed
were
related
to
like
the
underlying
cloud
infrastructure,
not
kubernetes
itself.
There
were
a
few
exceptions
to
this.
You
know
I
mentioned
managed
load
balancers,
so
we
look
at
things
like
ingresses
or
services
which
actually
rely
on
these
load
balancers.
We
need
to
make
adjustments
for
them
and
any
configuration
which
refer
to
object,
storage
again
we're
going
from
like
gcs
to
s3.