►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Tinder's Move to Kubernetes - Chris O'Brien & Chris Thomas, Tinder
Almost 2 years ago, Tinder decided to move its platform to Kubernetes. Kubernetes afforded us an opportunity to drive Tinder Engineering toward containerization and low-touch operation through immutable deployment. Application build, deployment, and infrastructure would be defined as code. We were also looking to address challenges of scale and stability. When scaling became critical, we often suffered through several minutes of waiting for new EC2 instances to come online. The idea of containers scheduling and serving traffic within seconds as opposed to minutes was appealing to us. During our migration in early 2019, we reached critical mass within our Kubernetes cluster and began encountering various challenges due to traffic volume, cluster size, and DNS. We solved interesting challenges to migrate 200 services and run a Kubernetes cluster at scale.
https://sched.co/UaVb
A
Thank
you.
My
name
is
Chris
O'brien
I'm,
a
senior
engineering
manager
with
tinder,
and
this
is
Chris
Thomas,
also
another
engineering
manager
with
tinder,
and
we
want
to
talk
you
through
the
decision
to
move
to
kubernetes
and
also
what
the
process
was
like.
I'm
sure.
Many
of
you
know
what
tinder
is,
but
in
case
you
don't
or
you
need
a
recap.
But
tinder
is
the
world's
most
popular
app
for
meeting
new
people
and
our
size
and
scale
means
greater
choice
and
access
to
a
diverse
set
of
matches
and
Chris
and
I
are
gonna.
A
So
ever
two
years
ago,
as
was
previously
mentioned,
tinder
decided
to
move
its
platform
to
kubernetes.
Kubernetes
afford
this
as
an
opportunity
to
drive
to
our
engineering
towards
containerization
and
lo
touch
operation
through
a
moodle
immutable
deployment
and,
as
a
result,
application
build
deployment
and
infrastructure
would
be
defined.
In
code
we
were
looking
to
address
the
challenges
of
scale
and
stability
when
scaling
would
become
critical.
A
So
starting
January
2018,
we
worked
our
way
through
various
stages
of
the
migration
effort.
We
started
by
container
izing
all
of
our
services
and
deploying
them
to
a
series
of
kubernetes
hosted
staging
environments
and
beginning
in
October
of
last
year.
We
began
methodically
cutting
all
of
the
legacy
services
over
to
their
counterpart
on
kubernetes
and
by
March
of
the
following
year.
We
finalized
our
migration
in
the
tinder
platform,
now
runs
exclusively
on
kerbin
natives.
A
So
the
legacy
architecture,
as
I
previously
mentioned,
were
all
hosts
on
Amazon
and
auto-scaling
groups.
They
were
fronted
by
a
load,
balancer
per
service
and
scaling
based
on
cpu
usage.
Now
we
had
minimal
tooling
to
automate
the
provisioning
of
these
auto
scaling
groups,
and
but
we
did
use
puppet
for
bootstrapping
node
configuration
and
also
setting
up
prometheus
nodes
to
monitor
each
of
the
service
services
code
deployments
were
pushed
basically
pushed
a
new
version
to
an
NFS
mount
and
in
a
fairly
archaic
way.
You
know,
would
would
trigger
a
service
restart.
A
So,
as
I
mentioned,
you
know,
the
first
step
of
the
migration
was
actually
building
these
container
images.
Luckily
tinder
had
had
already
started
a
push
towards
microservices
long
before
we
started
the
migration,
but
as
a
result,
we
had
more
than
30
source
repositories
with
a
variety
of
different
languages.
A
A
It
was
kind
of
the
most
relevant,
at
least
to
the
kubernetes
migration,
but,
as
you
can
see,
the
the
without
our
CI
CD
pipeline,
the
moved
acrimony,
aiders,
would
not
be
possible
and,
and
the
complexity
on
this
diagram
represents
the
the
years
of
exceptional
engineering
to
be
able
to
get
to
this
point.
So
at
the
very
top
you
have
our
CI
city
repository,
and
this
is
basically
defining
what
services
you
can
build
and
where
they
should
be
deployed.
And
then,
of
course,
you
have
your
code
repositories.
A
For
example,
your
your
go
or
your
go
repository
or
your
Java
repository
and
in
them
is
the
make
file,
and
basically
the
CI
CD
platform
will
schedule
a
builder
container
to
build
a
specific
image
for
whatever,
whatever
whatever
application
is
specified
as
part
of
the
manifest
it
is.
So
it's
not
just
Jenkins.
You
know
we
use
Jenkins
for
UI
and
for
scheduling,
but
under
the
hood
is
a
robust,
CI
CD
framework
that
we
were
using
to
do
all
sorts
of
things
obviously
deployed
a
kubernetes
provision,
infrastructure
using
terraform
and
in
the
future.
A
Also,
you,
you
know
also
build
into
start
to
leverage
it
to
execute
some
of
our
puppet
manifests
at
the
end
of
the
day.
What
ends
up
happening
is
that
the
image
that's
built
as
part
of
the
Builder
container
is
push
push
to
amazon's
container
registry
and
then
obviously
the
deployment
object
is
updated
within
kubernetes
through
the
CIC
platform.
To
point
to
that
new
image.
A
So
the
original
architecture,
we
used
a
cube
AWS
for
provisioning
as
we
are
an
Amazon
shop,
and
that
was,
you
know,
kind
of
readily
available
to
us
at
that
time.
Initially,
we
had
one
node
pool
one
big
node
pool,
but
we
quickly
separated
into
different
titles,
different
sizes
and
types,
and
what
we
found
is
that
running
fewer
heavily
threaded
jaw,
heavily
threaded
pods,
for
example,
Java
together
yield
a
better
performance
than
having
them
co-locate
with
single
single
thread
at
workloads.
A
Eventually,
we
settled
on
a
combination
of
c-5,
4x,
l
and
c
5
to
excel
instances
for
our
various
various
components,
so
control
plane
masters
at
CD,
our
single
thread
at
worker
pool
in
our
multi-threaded
worker,
blue
pool
in
Java
and
go,
and
we
also
settled
on
a
more
memory,
so
an
m5
instance
site
for
our
memory.
Intense
applications,
for
example,
are
monitoring
stack.
A
So
the
actual
migration,
basically,
what
we
did
is
we
changed
existing
service
to
service
calls
within
the
tinder
platform
to
to
new
load
balancers.
We
basically
peered
the
existing
vp
c
to
the
new
kubernetes
vp
c
and
amazon,
and
this
allowed
us
to
granularly
migrate
modules
with
no
regard
to
order
or
dependency
chain.
Endpoints
use
wait
at
dns
records
with
c
name
to
the
new
ELB,
and
this
afforded
us
an
opportunity
to
to
control
the
volume
of
traffic
and
also
facilitate
this
the
opportunity
to
roll
back.
If
we
had
any
issues
for
the
migration.
A
Obviously,
since
we
were
using
DNS
TTL
was
lowered
and
weight
was
adjusted
to
to
be
able
to
very
grandly,
adjust
and
on-the-fly
adjust
the
the
traffic
volumes
for
these
these
modules.
What
we
found
is
that
our
Java
repositories
honored
the
low
TTL
but
node,
but
our
nodejs
repository
did
not,
and
so
what
we
had
to
do
is
spend
some
engineering
effort
to
rework
our
connection
pool
so
that
it
would
refresh
those
connections
every
60
seconds
and
get
the
latest
latest
resolution
for
that
particular
host
name.
A
So
learnings
you
have
to
bear
with
me.
We're
gonna.
Go
a
little
bit
into
the
to
the
technical
weeds
for
a
couple
of
slides.
We
had
a
series
of
fairly
painful
outages
in
January
and
also
earlier
yeah.
You
know
early
in
earlier
this
year
and
it
was
kind
of
the
results
of
kind
of
two
key
things
that
unfolded.
A
So
the
first
thing
was
our
ARP
table
entries,
so
basically
January
January,
8th
2019.
We
were
down
for
several
hours
and
it
was
basically
it
was
basically
the
result
of
an
unrelated
scale
up
earlier
in
the
day
that
left
the
cluster
at
a
larger
size
than
ever
before.
As
a
result,
the
you
know
there,
the
the
ARP
table,
ran
out
of
available
entries
right.
A
So,
as
a
result,
you
know
we
had
to
raise
some
of
the
values
by
it's:
a
CTL
on
the
nodes
themselves,
right
to
expand
the
size
of
that
ARP
table
or
ARP
cache,
and
we
restarted
flannel
all
the
nodes
on
all
the
nodes
and
then
finally,
the
other
one
of
the
other
challenges.
We
worked
our
DNS
timeouts,
so
our
engineering
teams
were
constantly
complaining
about
error
rates,
at
least
for
some
of
our
bigger
modules
on
it.
A
That
you
know
is
the
direct
result
of
source
net
and
destination
that
and
the
issue
rate
issues
were
amplified
by
many
by
the
amount
of
lookups
that
we
were
doing
on
our
for
DNS
within
our
cluster.
So
we
actually
I
mean
it
the
first
case.
We
just
thought
it
was
a
matter
of
scale.
You
know
we
peaked
at
something
like
250,000,
DNS
requests
a
second
and
128
which
resulted
in
120
cores
of
usage
over
a
thousand
core
DNS
pods.
A
But
it
became
clear
to
us
that,
no
matter
how
many
pods
DNS
pods,
we
threw
at
this
particular
problem.
It
just
wasn't
getting
any
better
and
that's
kind
of
when
we
stumbled
upon
the
article
and
some
of
the
various
articles,
that
of
other
other
industry,
people
that
were
that
were
suffering
through
the
same
same
issues
that
we
were,
and
so
what
we
did
is
what
we
did
is
we
kind
of
took
the
the
race
condition
or
the
issue
with
with
s
net
and
D,
not
added
the
equation.
A
You
know
we
redeployed
accordion
s
as
a
daemon
set
and
injected
the
the
note
IP
into
resolve
Kampf,
so
that
that
a
pod
or
a
containers
first
lookup
would
be
on
the
the
node
itself.
It
would
not
be
on.
It
would
not
fall
back
to
some
service
that
resulted
in
estimate
and
D
net
translation
and
with
that
I'll
go
ahead
and
turn
it
over
to
to
Chris.
Thank
you
so.
B
B
Initially,
we
had
unbalanced
load
across
the
pods
due
to
ELB
connections,
sticking
to
the
first
set
of
ready
pause
that
would
come
out
of
each
deployment,
and
this
was
mostly
due
to
keep
alive
and
the
way
EEO
bees
keep
connections
open
to
nodes.
So
we
we
attempted
multiple
mitigations.
One
was
very
brute
kind
of
crude,
but
max
surged
to
a
hundred
percent.
So
you'd
hope
that
you
know
most
of
your
pods
will
come
up.
B
B
So
one
of
the
ways
we
got
this
to
work
very
smoothly
was
to
put
a
pre
stop
hook
on
the
sidecar,
which
calls
the
Envoy
health
check
failed.
Admin
endpoint,
as
well
as
a
small
I,
think
10
10
second
sleep
to
allow
inflate
connections
to
complete
and
drain,
and
that
proves
very
stable,
so
on
poi
itself,
I'm
sure
many
of
you
have
seen
it
or
used
it.
B
So
this
is
just
an
example
of
how
you
could
see
the
CPU
convergence
on
one
of
our
services
from
the
time
that
which
we
we
cut
it
over
from
running
on
the
old.
You
know
service
endpoint
in
kubernetes
to
one
of
these
Envoy
fronted
ones.
So
you
can
see,
some
of
them
are
as
little
as
25%
of
their
requested.
Cpu
all
the
way
up
to
you
know
nearly
200
percent
and
as
we
cut
it
over
slowly,
they
converged
into
a
very
very
close
point.
B
So
our
current
cluster
architecture
right
now
we're
around
2000
nodes,
18,000
cores,
or
so
we
have
6
control,
plane,
masters
25
to
30,000
pods
on
any
given
moments,
depending
on
how
the
auto
scaling
is
working
115
to
130,000
containers,
as
we
run
many
sidecars
on
each
pod,
many
meaning
anywhere
from
2
to
6,
our
Prometheus
stack
ingests
around
750,000
samples
per
second.
We
have
a
roughly
5
terabytes
per
day
of
log
ingestion
into
our
elk
stack
and
we're.
Currently
in
the
middle
of
an
envoy
service
mesh
migration.
B
B
So
you
can
see
we
have
our
services
separated
in
two
distinct
namespaces
for
each
one
and
they
each
ran
their
own
Prometheus,
pod
or
actually
two,
which
would
then
write
to
a
long-term
Prometheus
storage,
which
is
what
the
graph
fauna
server
actually
hits.
So
each
of
these
you
know
orange
and
green
boxes.
B
Basically,
you'd
have
to
vertically
scale
them
and
it
didn't
work
very
well
after
a
certain
point,
so
the
new
monitoring
stack,
we've
got
is
actually
based
with
an
Thanos
and
the
pods
actually
run
a
side
car
that
drop
all
of
the
excess
metrics
that
we
don't
need.
So
we
have
our
developers
specifically
put
in
rules
for
roll-ups
and
things
in
their
module,
manifests
that
will
tell
the
side
car
exactly
what
to
collect
and
everything
else
is
dropped.
B
And
then
we
have
two
Prometheus
pods
per
namespace
just
for
redundancy,
and
then
those
have
a
Thanos
side,
car
running
on
them
and
those
right
to
s3,
as
well
as
the
the
fan
of
stores,
are
then
capable
of
reading
the
the
long
term
data
out
of
s3.
So
then,
the
the
query
layer
is
what
Gravano
talks
to
and
it
effectively
scales
much
more
horizontally
than
before,
and
we
also
run
a
Prometheus
vertical
autoscaler
for
the
times
where
an
individual
service
needs
to
be
scaled
up
a
little
bit
to
handle.
B
B
That
will
allow
us
to
swap
these
clusters
in
and
out
upgrade
them
and
so
forth,
without
having
to
worry
about
losing
metrics
or
having
the
metrics
tied
into
the
cluster
directly
and
of
course
all
of
that,
then
you
know,
writes
and
reads
from
various
Amazon
services
and
that's
pretty
much
it.
Thank
you.
Yeah.