►
From YouTube: Migrating GitLab.com to Kubernetes: How Multiple Clusters Saved Cost - John Skarbek, Gitlab
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Migrating GitLab.com to Kubernetes: How Multiple Clusters Saved Cost - John Skarbek, Gitlab
As we grew past 10 million projects hosted on GitLab.com, it was clear that we needed to move from our fleet of chef-managed virtual machines to Kubernetes. Using GKE, migration started with stateless services like the GitLab Container Registry, Background processing, and Git requests. With over 100 Terabyte of daily Git data, regional GKE clusters provide the convenience of spanning multiple availability zones for redundancy, but cross availability zone egress was a concern. Splitting the regional GKE cluster into multiple zonal clusters for services that use a lot of bandwidth gave much more control over cross availability zone network traffic. In this talk, you will learn more about our journey and efforts with how we are shifting traffic to the new clusters.
A
A
So,
firstly,
what
is
gitlab.com.
We
are
the
get
lab
product
itself
at
sas
scale,
so
you
know
we're
serving
roughly
2
million
plus
customers.
At
this
moment
it
took
us
a
while
to
get
started
in
kubernetes
land.
We've
got
a
large
infrastructure.
We've
had
to
scale
out
quite
vastly
over
the
course
of
time,
and
in
doing
so,
our
infrastructure
team
has
had
to
break
up
parts
of
our
infrastructure
into
smaller
chunks
in
order
to
be
make
it
a
little
easier
for
us
to
fully
manage
that
large
infrastructure
at
the
scale
that
we
operate.
A
So,
with
those
two
building
blocks
in
place,
we've
been
able
to
get
our
feet
wet
within
kubernetes,
and
at
this
point
we
now
have
today
we
have
a
hybrid
architecture,
both
virtual
machines
that
are
still
managed
via
terraform
and
chef,
and
we
now
have
about
90
of
our
front
end.
Workloads
are
inside
of
kubernetes
at
this
point
would
have
done
a
few
more
services,
but
I
decided
to
stress
myself
out
and
do
a
talk,
so
here
I
am
so
when
it
came
to
getting
started
with
kubernetes.
A
Our
infrastructure
team
needed
a
way
to
simply
or
create
a
cluster
very
quickly
without
having
to
work
terribly
hard.
We
did
not
want
to
go
the
route
of
using
our
existing
infrastructure
management
practices
to
build
clusters,
manage
the
upgrades
and
figure
out
how
to
troubleshoot
them
and
such
we
wanted
to
do
something
very
quickly.
A
This
allowed
us
to
focus
our
efforts
where
we
deemed
things
more
important
to
us
that
included
items
such
as
making
sure
we
know
how
to
manage
and
deploy
our
actual
configurations
for
those
clusters
integrating
those
clusters
with
our
existing
infrastructure,
making
sure
the
applications
interoperate
very
well
between
our
virtual
machines
and
our
clusters
that
we
deployed
in
our
observability
stack.
We
want
to
make
sure
that,
as
we
migrate
things
between
virtual
machines
and
kubernetes
that
we
could
still
view
the
metrics
of
those
applications,
as
as
we
focus
on
its
migrations.
A
A
A
When
you
operate
inside
of
a
single
cluster
and
you're,
you've
got
a
web
service
deployed
you're
technically
deploying
a
few
objects.
One
is
the
web
service
itself,
along
with
all
the
web
service
pods
and
the
services
associated
with
those
and
you're
also
deploying
the
nginx
ingress
controller
in
our
case,
or
you
might
have
a
configuration
for
the
existing
controller
because
we're
using
a
google
cloud
as
our
provider,
we
get
an
a
low
bouncer
of
some
type.
In
this
case,
we
utilize
an
internal
load
bouncer.
A
At
this
point,
we've
got
two
potential
paths
in
which
network
traffic
can
take
one
of
them
being
a
very
nice
low,
latency
low
cost
path,
the
one
being
kind
of
a
heavyweight
path,
because
a
regional
cluster
is
set
up
in
a
way
in
which
you
have
web
service
pods
the
knowledge
of
those
web
service
pods
across
the
entire
cluster.
The
nginx
ingress
controllers
know
about
all
those
pods
because
of
this,
when
a
request
comes
into
an
ingress
control
located,
say
in
zone
b.
A
If
that
request,
lands
on
a
web
service
pod
located
in
zone
c,
two
potentially
bad
things
can
happen.
One
you
have
additional
network
latency,
because
you've
egressed
traffic
from
one
zone
to
another,
it
may
be
minor,
but
it
might
be
something
to
keep
in
mind
on
another.
Is
the
cost
of
the
network
egress
from
one
zone
to
another?
A
A
You
know
how
much
a
kubernetes
cluster
is
going
to
cost
because
they
tell
you
it's
going
to
cost
x
amount
per
hour.
We
know
the
nodes
that
back
or
creminates
clusters.
You
can
kind
of
predict
how
much
the
instances
are
going
to
cost
you
over
the
course
a
month.
Gitlab.Com
is
a
very
network,
data-driven
workload,
we're
subject
to
what
our
customers
send
to
us
and
the
data
that
they're
requesting
from
us.
A
A
However,
the
nginx
ingers
controllers
they're
not
really
doing
a
lot
of
work
here,
they're
simply
proxying
the
data
they
might
be
doing
a
little
buffering,
but
the
important
part
is
that
they're
taking
the
traffic
from
the
nginx
ingers
controllers
and
kind
of
proxying
it
across
all
these
web
service
pods.
Realistically,
they
don't
need
to
know
about
all
of
those
web
service
pods,
but
they
just
need
to
know
how
to
route
the
traffic
to
ones
that
are
closest
to
them.
A
A
So,
given
a
region
that
has
three
zones,
we
simply
created
a
cluster
per
zone.
So
now,
when
traffic
comes
in
through
rha
proxy
front
end-
and
you
know,
they're
spread
across
all
of
our
zones
as
well,
if
your
client
request
went
through
zone
b
in
aj
proxy
you're,
going
to
get
favored
to
a
cluster
that
is
also
located
in
zone
b.
A
A
We've
you've
got
the
potential.
We
have
a
situation
for
cluster
upgrades.
You
could
spread
them
out
over
the
course
of
time,
so,
instead
of
having
a
single
cluster
that
gets
upgraded,
you
have
multiple
clusters
and
you
know
should
an
upgrade
fail.
You
got
the
ability
to
spread
or
remove
a
cluster
from
rotation
to
fix
it.
H.A
prox
in
our
case,
has
knowledge
of
all
of
our
clusters.
A
So
what
other
options
are
available?
If
a
multi-cluster
configuration
may
not
be
available
for
everyone?
One
is
a
service
mesh
today,
gitlab.com
does
not
use
a
service
mesh.
This
gitlab
product
does
not
use
a
service
mesh,
and
this
would
have
been
a
lot
of
work
for
us
to
try
to
figure
out
how
to
implement.
It
would
have
been
a
combination
of
figuring
out
which
service
mesh
would
have
worked
for
us
learning
how
to
use
it,
configure
it
build
up
the
observability,
tooling
and
all
that
stuff.
A
Another
option
would
have
been
modifying
our
regional
cluster
configuration
there's
many
ways
to
configure
a
managed
cluster
inside
of
google,
but
we
saw
the
benefits
of
having
more
clusters
greater
than
a
single
cluster,
so
we
decided
to
just
go
down
that
route
and
then
today,
there's
more
features
available
than
there
were
in
prior
when
we
first
started
looking
into
this,
so
kubernetes
kubernetes
has
a
feature
called
topology
aware,
hints
something
that
was
not
available
back.
Then
I'm
no
clue
what
this
does,
but
it
might
be
worth
checking
out.
A
Here,
I
think
the
most
strategic
part
that
we
had
at
gitlab
was
determining
how
we
identify
our
clusters.
Would
that
be
a
name
or
adding
some
sort
of
location
tag
to
figure
out
how
we
want
to
observe
and
monitor
our
clusters
appropriately
and
then,
of
course,
simplifies
the
cloud
bill.
You
know
by
lowering
the
costs,
so
some
bonus
things
I
want
to
touch
on
that.
We
got
out
of
having
this
configuration.
A
One
is
a
fun
way
to
mitigate
incidents,
so
this
chart
is
showing
our
nat
port
usage
over
the
course
of
time.
When
we
first
created
our
additional
clusters,
we
were
asking
all
of
our
clusters
to
deploy
new
code.
At
the
same
time,
this
had
the
down
effect
of
after
a
period
of
time
where
we
had
grown
into
kubernetes
and
started
pushing
more
stuff
into
kubernetes.
A
A
All
we
had
to
do
was
simply
modify
our
ci
pipelines
to
change
how
many
clusters
we
deploy
at
a
time
so,
instead
of
deploying
in
parallel,
it
was
deployed
to
one
cluster
at
a
time
to
give
our
infrastructure
engineers
time
to
fix
that
problem
and
then
move
forward.
We
could
revert
that
change
and
deploy
to
as
many
clusters
as
we
want
to
another
one
that
I
want
to
highlight-
and
this
is
something
that
we
use
very
heavily
right
now
is
the
ability
to
test
cluster
configurations,
so
this
is
a
chart.
A
Showcasing
pod
counts
over
the
course
of
a
few
days.
You
can
see
right
in
the
middle
of
the
chart
that
we
made
some
sort
of
change
where
we're
using
far
fewer
pods
than
than
two
of
our
two
of
our
other
clusters.
So
we've
effectively
created
a
control
and
a
test
set
of
clusters
where
the
controls
are
the
two
clusters
that
we
did
not
touch
at
all.
A
A
So
in
this
particular
case,
we
are
trying
to
gain
more
efficiency
by
lowering
how
many
pods
that
we
were
running
for
this
particular
workload,
and
the
last
thing
I
want
to
highlight
is
maintenance
procedures.
So
we
have
the
ability
to
remove
traffic
going
into
entire
clusters
as
a
whole.
So
this
is
a
showcase
of
web
requests
coming
in
through
all
three
of
these
clusters
in
this
one
region.
A
So
we
had
found
an
interesting
bug
on
our
helm
chart
and
if
we
were
to
deploy
a
fix
for
the
home
chart,
it
would
have
been
incident
inducing.
So
in
order
to
do
that,
we
just
created
a
maintenance
procedure
where
we
pulled
traffic
from
that
cluster
entirely
deployed
the
fix
validated.
The
fix
was
in
place
and
and
put
traffic
back
into
it
and
repeated
that
process
for
all
of
our
clusters.
A
So
to
wrap
up,
I
think
one
thing
that
would
be
interesting
to
keep
in
mind
is
determining
whether
or
not
this
is
a
problem
for
you.
This
was
going
to
be
a
problem
for
us
because
we
did
start
off
using
original
cluster
and
not
knowing
that
network
egress
was
going
to
be
a
problem
between
zones
within
a
cluster
was
something
that
was
not
immediately
prevalent
to
us.
A
A
It
was
easier
for
us
to
deploy
more
clusters
than
it
was
to
look
at
other
solutions
and
also
conserve.
This
is
a
problem
with
other
workloads
outside
of
kubernetes.
Like
I
mentioned,
we've
got
this
problem
between
our
web
service
layer
and
talking
to
our
file
servers
and
such
so.
This
could
be
problems
elsewhere
within
your
stack,
so
I
spoke
really
fast,
because
this
is
my
first
time
so
my
apologies,
so
I've
got
plenty
of
times
for
question.