►
From YouTube: Scaling Kubernetes to Thousands of Nodes Across Multiple Clusters, Calmly - Ben Hughes, Airbnb
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Scaling Kubernetes to Thousands of Nodes Across Multiple Clusters, Calmly - Ben Hughes, Airbnb
In under a year, Airbnb went from 600 Kubernetes nodes across a couple handcrafted clusters to over 5000 nodes on tens of clusters. Successful adoption of Kubernetes by services led to more and faster adoption leading to challenges of scale. Facing this, Airbnb switched to a multiple production cluster architecture to get around single cluster scalability limits and ensure ample capacity for services.
This process increased the consistency of the cluster configurations while reducing manual operations. This talk will discuss the problems that were faced during scaling, the shape of the solutions, specific approaches that worked well (and didn’t), and how this was accomplished without a drastic shift away from existing pre-Kubernetes infrastructure tooling. A key result was reducing the time to create a new, production-ready cluster from over a week to under an hour.
https://sched.co/Uacr
A
Good
afternoon,
everyone
and
welcome
to
the
the
last
talk
in
the
aberrations
tract
today.
It's
scaling
kubernetes
to
thousands
of
nodes
across
multiple
clusters,
calmly,
I,
hope,
let's
see,
and
please
make
sure
to
move
into
rows.
So
everybody
that's
coming
in
right
now
can
can
get
a
seat
as
well
and
also
remember
to
rate
the
session
after
the
talk
now
to
the
introduction
of
our
next
speaker,
Ben
is
a
software
engineer
at
a
B&B.
He
has
worked
with
data
base,
scaling,
Ruby
and
node.js
performance,
an
incident
response
and,
of
course,
kubernetes.
B
Yeah,
thank
you.
Thank
you,
like
you,
said,
I'm
Ben,
Hughes
and
today
I'm,
going
to
talk
about
how
we
at
Airbnb
contend
it
with
massive
internal
adoption
of
kubernetes
that
led
us
to
quickly
scale
up
past
single
cluster
limits,
I've
been
at
Airbnb
for
over
seven
years,
working
on
various
areas
of
infrastructure
and
scalability.
So
much
of
what
we
consider
our
current
legacy
infrastructure
I,
often
remember
when
its
predecessor
or
predecessors
predecessor,
was
shiny
and
new.
So
it's
kind
of
an
interesting
up
bit
of
a
context
to
have
when
designing
new
things.
B
So,
let's
first
cover
the
basics.
For
those
of
you
who
aren't
familiar
with
Airbnb
Airbnb
is
one
of
the
world's
largest
marketplaces
for
unique,
authentic
places
to
stay
and
things
to
do
offering
over
7
million
accommodations
and
40,000
handcrafted
activities.
All
powered
by
local
hosts
and
today
I'll
talk
about
how
we
scaled
or
kubernetes
usage
I'll
start
with
how
our
architecture
evolved
and
got
us
using
kubernetes.
B
Of
course,
simply
creating
a
bunch
of
clusters
isn't
really
the
hard
part
like
with
most
things,
it's
maintenance
and
ensuring
consistency
that
end
up
being
the
greater
burden
we've
invested
in
tooling
and
strategies
to
keep
this
under
control,
while
also
integrating
with
our
existing
infrastructure.
So,
finally,
I'll
talk
about
how
we
try
to
enable
change
and
evolution
within
the
system.
Any
new
thing
can
hopefully
account
for
the
use
case
at
the
point
in
time
that
it
is
written
but
use
cases
change.
B
This
is
especially
important
because
the
kubernetes
ecosystem
is
complex
and
fast-moving,
so,
whatever
structures
you
have
around,
it
have
to
be
able
to
account
for
change,
plus
you're,
pretty
much
guaranteed
to
screw
up
some
seemingly
insignificant
config
that
comes
back
to
haunt
you
later.
So
how
did
we
get
here?
I
think
the
story
of
how
our
architecture
evolved
is
a
pretty
common
one
for
a
tech
company.
Isn't
this
decade
we
started
with
a
single
monolithic
code
base
that
did
absolutely
everything
and
it
was
a
rails
app,
so
he
called
it
monorail.
B
It
turns
out
that
everyone
who
has
a
large
rails
app
at
the
center
of
their
business
calls
it
monorail
I
was
crestfallen
when
I
discovered
that
we
weren't
so
creative-
and
this
was
this-
was
fine
for
a
while,
but
it
became
rather
unwieldy.
As
the
number
of
engineers
grew,
it
became
more
and
more
of
a
check
choke
point.
The
single
large
code
base
that
could
do
absolutely
everything
also
made
it
hard
to
reason
about
boundaries.
Monorail
was
honestly
just
a
huge
slog
to
work
with.
B
B
But
then
it
was
an
absolute
explosion
of
services.
We
really
took
a
micro
to
heart
and
over
the
course
of
a
couple
of
years
we
added
hundreds
of
services.
This
had
advantages
over
the
monolith
and
it
allowed
development
to
be
unblocked,
but
it
wasn't
an
unqualified
win
on
the
developer
experience
side.
Engineers
were
now
required
to
touch
many
many
things
to
roll
out
certain
application
changes
things
like
the
actual
application
code,
chef
configuration
of
their
systems-
and
this
was
made
way
worse
because
we
were
managing
many
secret
via
chef
data
bags.
B
You
know
things
like
terraformer,
alerting
rules.
This
just
was
a
generally
a
pile
of
a
pile
of
pull
requests.
Many
of
them
even
had
to
be
deployed
in
a
specific
and
often
non-obvious
order.
So
this
is
the
initiative
that
so
this
led
to
an
initiative
initiative
to
reduce
the
many
touches
to
one
touch,
and
this
is
where
kubernetes
comes
in,
but
also
a
lot
of
other
tooling
to
wrap
around
around
kubernetes.
B
We
didn't
just
give
people
cube,
cuttle
applied
a
chef
and
tell
them
to
have
fun
with
it,
and
kubernetes
provides
the
integration
point
so
that
all
of
an
app's
config
can
live
with
the
app.
This
capability
continues
to
grow
as
we
use
more
and
more
custom
resources
and
custom
controllers
to
manage
things
beyond
just
the
distribution
of
containers.
B
Our
goal
is
to
enable
all
app
and
config
changes
for
a
service
to
be
a
single
PR,
a
single
code
review
and
a
single
deploy,
but
hundreds
of
services
is
probably
still
too
many
like
keep
it
under
control
and,
of
course,
so
we
started.
We
had
stood
up
a
system
to
handle
all
of
these
services.
We
still
needed
to
get
all
those
services
on
the
system,
so
you
began
migrating
existing
services.
This
was
initially
an
unproven
thing,
so
initial
migration
was
very
slow
teams
just
started
to
dip
their
toes
in.
We
were
more.
B
We
were
also
more
likely
to
get
smaller
deployments
of
small,
smaller,
somewhat
less
complicated
services,
but
as
as
this
built
up
momentum
and
as
you
know,
teams
got
more
comfortable
with
this.
The
migration
continued
more
rapidly
and
we
started
to
see
larger
services
coming
over
and
around
this
time
we
started
to
think
about
questions
like
what
are
our
scaling
limits
and
at
some
point
the
calculus
shifted
from
how
can
we
get
stuff
to
use
our
thing?
B
I
had
a
number
of
conversations
about
what
sort
of
projects
to
work
on
and
someone
you
know
mentioned
that
we
should
start
thinking
about
multi
cluster,
so
you
know
kept
it
in
the
back
of
my
mind,
while
working
on
other
stuff,
but
by
December,
we
had
doubled
to
900
nodes.
So
how
big
can
we
make
these
clusters
again?
So
he
turned
to
the
reference
documentation.
There's
a
hard
limit
at
five
thousand
nodes.
B
After
about
the
2500
to
3,000,
node
range,
of
course,
more
recently,
Alibaba
did
some
really
great
work
and
published
a
blog
post
about
how
they
were
able
to
get
a
cluster
up
to
you
about
ten
thousand
nodes.
A
lot
of
those
a
lot
of
those
changes
are
actually
like
seem
to
be
pretty
generally
applicable,
so
I'm
really
excited
for
like
how
this
will
push
forward
kubernetes
scaling
going
forward,
but
at
the
same
time
that
we
were
pondering
what
our
theoretical
you
know,
future
scaling
limits
might
be.
B
We
also
had
some
sudden
more
pressing
questions
like
would
it
be
bad
if
or
bad
if
our
EDD
were
suddenly
getting
out
of
memory
killed
and
it
turns
out
this
question
was
not
hypothetical.
So,
as
we
closed
out
last
year,
we
were
having
a
lot
of
issues
with
Etsy
D,
getting
backed
up
and
falling
over.
All
it
took
were
small
shocks
like
a
lower
cash
hit
rate
on
the
API
server
or
a
flurry
of
events
to
cause
that
CD
to
get
overwhelmed.
B
So
we
effectively
got
a
taste
for
the
problems
that
that
came
with
scale,
but
we
were
still
rapidly
scaling
up
by
not
by
March.
We
had
doubled
again
to
1,800
nodes
and
the
team
was
working
hard
to
ensure
capacity
and
we
needed
a
plan
to
cross
2500
or
even
you
know,
five
thousand
nodes.
So
we
went
with
the
obvious
solution
of
standing
up
multiple
clusters.
When
you
can't
make
a
single
cluster
any
bigger,
we
must.
B
We
must
make
more
clusters,
so
in
April
we
had
capped
out
at
about
2,300
nodes
in
our
production
cluster
and
we
would
not
not
grow
the
prod
cluster.
Beyond
this
point,
we
found
some
substantial
improvements
in
place
where
we
were
able
to
free
up
some
capacity,
and
we
started
diverting
workloads
to
newly
created
clusters
in
May.
B
So
multiple
cluster
was
the
solution
that
we
had
been
considering
since
I
had
started
on
the
problem
and
it's
the
solution
that
we
went
with
and
it's
been
very
effective
for
us
and
at
the
same
time,
multi
cluster
can
be
a
bit
of
a
loaded
concept.
People
mean
a
lot
of
different
things
when
they
say
multi,
cluster
and
other
decisions
or
considerations
in
your
organization
can
make
it
easier
or
harder.
So,
when
you're
splitting
up
things
up
into
a
whole
bunch
of
clusters,
you
need
to
think
about
the
specifics
of
your
problem
space.
B
Are
there
placement
restrictions
on
your
workloads?
That
is,
do
certain
services
need
to
be
in
the
same
cluster
or
kept
very
far
apart,
and
this
may
be
the
case
if
you're
using
cluster
DNS
or
kubernetes
service
discovery
or
doing
a
lot
of
communication
over
an
overlay
network?
And,
moreover,
what
problems
are
you
trying
to
solve
with
multi
cluster?
B
In
our
case,
we
were
primarily
targeting
pure
node
count
and
cluster
size,
but
if
you're
trying
to
do
things
like
isolate
workloads
or
take
advantage
of
multiple
clouds-
or
you
know,
cloud
bursts
or
anything
like
that,
you'll
need
to
make
different
decisions.
And
there
are
you
know
there
are
efforts
in
the
kubernetes
community
that
are
underway,
like
cube
fat
or
Federation
v2,
that
target
kind
of
they
seem
to
target
kind
of
a
superset
of
a
lot
of
these
concerns
a
lot
of
like
these
sorts
of
related
areas.
B
B
So
overall,
with
these
considerations,
we
got
lucky
and
we're
able
to
pursue
a
fairly
simple
version
of
Multi
cluster
and
a
big
thing
like
probably
the
biggest
thing
that
we
got
lucky
on,
was
that
the
particulars
of
our
service
mesh
smart
stack-
and
you
know
our
migration
case-
caused
us
to
need.
First-Class
communication
between
our
legacy,
ec2
and
freh
and
and
our
new
kubernetes
services
and
this
avoided
collocation
requirements.
So,
like
you
know,
as
services,
we're
getting
migrated,
a
request
might
might
come
in
to
ec2
and
then
get
routed
to
another
service
in
kubernetes.
B
As
you
know,
we
see
fit,
and
so,
while
this
is-
and
we
do
have
one
restriction
just
to
kind
of
simplify
things
and
that's
that
our
services
run
on
a
single
cluster
and
we
select
that
cluster
at
the
time
that
the
service
is
created.
So
at
the
time
like
an
engineer,
is
you
know
setting
up
the
the
code
structure
for
their
brand-new
service?
B
So
here's
a
diagram
to
try
and
illustrate
what
what
we're
doing
and
we
run
our
pod
Network
on
an
overlay
network,
and
this
is
either
you
know,
flannel
or
canal,
depending
on
on
what
which
one
which
cluster
we're
on
and
the
overlay
network
is
restricted
to
the
cluster
itself.
So
this
would
prevent
so.
You
can't
communicate
with
pods
on
other
clusters
over
over
this
overlay
network
or
you
can
you
can't
go
from
ec2
instances
to
pods
on
any
cluster
over
this
overlay
network
directly
so
to
get
around
this?
B
We-
and
this
was
completely
just
for
the
way
that
we
were
doing
our
service
mash
and
service
discovery.
We
use
a
lot
of
node
port
services
running
in
local
mode,
so
your
application
would
be
listening
to
like
port
port
80
on
like
your
pod
IP
and
then
the
node
port
service
would
configure
a
high
number
port
on
the
node
IP
and
through
the
magic
of
IP
tables.
B
This
involves
this
will
involves
far
less
IP
tables
and
it's
a
it's
very
appealing.
So
we're
strongly
considering
using
one
of
these
going
forward,
and
so
that's
so
that's
what
we
have.
We
don't
really
care
where
services
are
run,
there's
not
much
to
guide
their
placement
and
for
simplicity
sake.
We
assign
a
cluster
at
creation
time
and
we
then
allow
those
clusters
to
grow
to
a
certain
size.
B
B
So
in
terms
of
creating
in
to
some
extent
managing
clusters
there
are,
there
are
existing
projects
like
cops
and
cue
Batman.
These
are
these
are
great
tools
and
if
they
work
for
you,
that's
fantastic
cops,
especially
is
pretty
upfront
about
being
opinionated,
and
some
of
these
opinions
ended
up
not
working
particularly
well.
For
us.
B
We
specifically
need
it
to
integrate
our
kubernetes
new
nodes
with
our
existing
tooling
around
ec2
instances,
and
we
didn't
want
to
hard
fork
our
underlying
infrastructure
and
have
to
reinvent
or
relearn
everything
about
managing
just
you
know,
ec2
instances
and
the
instances
should
it
be
completely
different,
just
because
they
happen
to
be
running
kubernetes
rather
than
some
other
service.
We
also
wanted
to
make
sure
that
we
solve
the
full
problem
as
we
saw
it.
B
So
what
even
is
a
cluster,
it
certainly
knows
and
all
those
nodes
need
to
be
configured
packages
installed,
SSH
keys,
you
know
the
works,
a
cluster,
isn't
that
much
without
EDD
to
store
its
state,
and
you
need
to
configure
all
of
the
options
that
go
to
the
various
components
of
the
control
plane.
You
also
need
to
setup
things
with
your
cloud
provider
like
I
am
roles
DNS
configuration
aSG's,
you
know
whatever
all
of
that
stuff
needs
to
be
configured
somehow
it's
can
its
kubernetes,
so
you
need
certificates.
B
The
genius
of
kubernetes,
of
course,
is
that
through
it
all,
problems
can
be
turned
into
either
public
key
infrastructure
or
networking
problems
which,
as
we
all
know,
are
the
easiest
problems
to
debug
and
so
for
this
type
of
config.
We
generate
chef
and
terraform
code
into
the
appropriate
existing
places,
but
this
isn't
all
that
makes
a
cluster
if
I
were
to
hand
you
a
cluster
with
all
of
these
set
up,
it
still
wouldn't
work.
B
We
don't
want
clusters
to
get
configured
using
randomly
applied
gamal
from
github.
This
is
kind
of
like
when
sis
admitting
involved
a
lot
of
ad
hoc
apt
getting
onto
individual
pet
servers.
Cluster
services
are
also
very
much
like
any
other
workloads
we
run,
and
ideally,
we
would
deploy
them
using
the
same
deployment
tools.
We
use
for
everything
else,
and
the
development
process
should
be
familiar
to
engineers
working
on
any
other
service.
We
also
want
to
make
sure
that
a
cluster
is
never
missing
a
component
and
that
components
stay
up
to
date
on
all
clusters.
B
B
We
also
deploy
to
entire
tiers
of
clusters
in
the
same
action,
and
this
prevents
individual
clusters
from
falling
behind
simply
because
we
forgot
to
deploy
to
them.
We
don't
yeah,
we
don't
want
prod
one
to
get
all
of
the
all
of
the
latest
latest
goodness,
while
prod
3,
just
kind
of
sits
there.
You
know
month
out
of
date
just
because
no
one,
no
one
triggered
the
deploy
there
and
so
deciding
what
goes
into
this
unit
is
pretty
straightforward.
B
If
this,
if
the
service
needs
to
be
local
to
the
cluster,
you
know
things
like
daemon
sets
deployments
like
Cordy
and
s-cube
dashboard,
others
other
things
along
those
lines.
Then
it
can
go
here.
Otherwise
it's
just
a
regular
service
and
it
can
go
up
to
whatever
single
cluster
it's
assigned
to,
and
so
we
refer
to
this
collection
of
cube
system
after
the
namespace,
where
a
lot
of
these
things
happen
to
live.
So
the
actual
deployment
process
is
a
mix
of
things,
but
it
doesn't.
You
know
it
does,
leave
heavily
on
our
cuber
Nettie's.
B
Much
of
the
core
config
is
structured
as
helm.
Charts,
though
we
mostly
use
helm
as
just
the
template
engine,
and
we
pull
all
the
dependent
the
dependency
helm,
charts
together,
using
an
umbrella
chart
and
template
into
a
single
manifest
that
we
then
apply.
Then
there's
a
few
stages
where
we,
you
know
synchronize
iam
roles,
and
then
we
and
then
we
have
a
number
of
apps
that
use
a
use,
cube
gin,
which
is
our
internal
kubernetes
sort
of
framework.
B
This
is
this
tool
is
primarily
around
generating
manifests,
but
it
does
also
have
some
imperative
deploy
logic
and
things.
That
kind
of
you
know
guide
the
deploy
process,
so
we
have
to
handle
that
separately
and
since
this
is
a
collection
of
many
different
things,
partial
failure
is
always
a
possibility.
So
we
try
to
make
this
predictable,
and
so
we
try
and
we
try
to
deploy
as
fully
as
possible.
B
B
Okay,
so
we
have
so
we
have
clusters
and
we
have
you
know,
configuration
for
them
and
we
have
the
services
that
run
on
clusters.
We
need
a
bit
more
structure.
On
top
of
this,
we
want
our
prod
prod
style
clusters
to
be
alike.
We
also
want
our
you
know,
test
or
dev
or,
like
anything,
that
we
have
multiple
clusters,
for
we
want
those
to
be,
you
know,
kept
similar
and
to
share
as
much
config
as
possible,
so
we
introduce
the
fairly.
B
We
introduced
a
layer
of
cluster
types
on
top
of
this,
so
you
know
prod
one
and
prod
to
share
the
cluster
type
of
prod,
so
cluster
types
are
like
classes.
Wall
clusters
are
like
instances
of
those
classes
since
we're
using
clusters
slowly
for
horizont
scaling
clusters
of
the
cluster
type
should
be
nearly
the
same.
Like
ideally,
you
know,
as
a
user
who's
deploying
their
service
to
one
of
our
clusters,
you
shouldn't
really
be
able
to
tell
what
cluster
you're
on
and
we
really
want
to
avoid
people
trying
to
land
their
services
on
the
good
cluster.
B
So
by
trying
to
make
all
of
our
you
know,
clusters
the
good
one
and
all
of
this
together
makes
the
cluster
launch
process
a
matter
of
using
scripts
to
generate
chef,
terraform
and
cube
system
code.
We
then
get
that
code
reviewed
and
merged
and
then
calls
the
at
CD
in
kubernetes
nodes
to
be
to
be
launched.
And
finally,
after
you
know,
after
this,
we
deploy
the
the
cube
system
artifact,
which
you
know
for
this
cluster.
B
It
will
be
the
initial
deploy
of
that
and
cause
everything
to
get
set
up,
and
at
the
end
of
that
we
should
have
a
fully
working
cluster.
This
entire
process
takes
under
an
hour
and
a
fair
amount
of
that
time
is
actually
just
getting
a
sign
off
on
the
changes
to
the
IAM
roles
and
we've
stopped.
We
stopped
trying
to
optimize
beyond
that,
because
it's
not
we're
not
doing
this
so
commonly
that
further
optimization
would
be
particularly
beneficial.
B
So
you
know
when
we
made
when
we
made
the
change
easy,
we
saw
that
we
got
more
change.
We
reduce
barriers,
and
you
know
things
started
happening
much
more
rapidly,
so
we
quickly
we
pretty
quickly
learned.
Some
of
our
initial
assumptions
were
wrong,
and
originally
our
chef,
terraform
and
cube
system
code
generation
was
one-shot.
You
generated
the
set
of
output
configuration
once
at
the
time
of
cluster
creation
and
made
further
modifications
in
place
into
the
files
that
you
generate
that
you
generated
with
the
generation
script.
B
So
this
pointed
to
a
continual
regeneration
process.
So
what
we
did
was
we
separated
out
the
config
into
a
bunch
of
high-level
config
files
that
serve
as
inputs,
and
then
these
feed
into
the
generators
and
the
generators
fully
regenerate
all
outputs
on
every
run,
and
so
this
prevents
drift
makes
it
easy
to
add
features
and
separates
out
the
interesting
variable
configuration
from
all
of
the
boilerplate.
B
So
some,
of
course
now
that
we
make
it
easy
to
make
chain
apply
change.
You
know
to
existing
clusters.
Some
change
is
relatively
easy
to
apply
if
you
yeah,
if
you
update
a
kubernetes
deployment,
kubernetes
will
figure
out
how
to
make
those
changes
and
apply
them
so
you're
good
to
go
there.
But
you
know
even
stuff,
like
upgrading
kubernetes
is
fairly
straightforward.
There's
an
ordering
that
you
have
to
do.
You
know
you
have
to
you,
upgrade
the
masters
rotate.
Those
out
upgrade
the
minion
minion
nodes
rotate
those
out,
and
but
it's
a
change.
B
That's
meant
to
be
applied,
it's
probably
something
you
actually
want
propagated.
It's
a
you
can
get
from
state
a
to
state
B
fairly
fairly
easily.
But
not
everything
is
like.
This
networking,
in
particular,
is
is
very
sticky.
It's
very
hard
to
change
things
like
ciders
or
CNI
plugins
in
place
on
a
running
cluster
without
disrupting
workloads
like
it's
possible
that
you
can
do
it
but,
like
you
know,
one
one
wrong
step
and
you
know
everything
kind
of
falls
apart.
B
It's
much
easier
just
to
move
workloads
between
clusters,
so
it
would
be
bad
if
you
know
our
new
total
generation
scripts
caused
caused
us
to
switch
existing
clusters
to
a
new
cider
that
we
want
it
to
use.
So
one
concept
that
we're
experimenting
with
in
our
configuration
is
having
the
ability
to
fix
values
for
things
that
are
to
change.
B
So
we
fix
these
values
in
the
configuration
at
the
time
that
the
cluster
is
created,
and
we
can
then
change
the
way
that
the
value
is
derived
for
new
clusters
without
having
to
change
it
for
old
clusters.
So
you
know
we
could
so
you
know
if
we
want
it
to
fix
the
node
cider
mask
size
at
24,
in
plus
in
clusters,
when
they're
generated.
We
could
do
that
and
then
you
know
change
that
value
going
forward,
and
eventually
the
older
of
the
older
clusters
can
be
rotated
out
and
replaced.
B
The
way
of
you
know
what
our
users
can
do,
so
we
want
to
let
let
them
drop
down
and
modify
the
code
that
would
otherwise
be
generated
and
overwritten.
So
every
comment
that
we
we
generate
or
every
file
that
we
generate,
has
a
comment
at
the
top
of
it.
That
can
be
marked
to
prevent
overwrites
from
happening,
and
this
is
proven
very
useful,
as
people
are
doing
frankly,
a
lot
of
very
weird
stuff
in
kubernetes
lands
so
definitely
allows
them
to
have
their
fun
without
us.
B
So
this
is
a
this
is
unlocked
our
continued
scaling
and
where
we're
currently
at
is
that
we
have
22
cluster
types
at
22,
different
sort
of
like
use
cases
for
that.
A
lot
of
these
are
fairly
mental,
and
you
know
we're
doing
a
lot
of
work
to
sort
of
prove
things
out,
and
you
know
we'll
see
where
that
leads,
and
these
represent.
These
represent
36
clusters,
many
of
which
are
many
of
which
are
part
of
a
the
prod
cluster
group,
which
is
expanding
to
support
our
you
know
our
actual
service
workload
and
alt
all
together.
B
This
represents
over
7,000
kubernetes
nodes,
and
the
exciting
thing
is
that,
from
our
current
strategy,
we
see
no
upper
bound
on
the
number
of
nodes
that
we
would
be
able
to
support.
We
can
just
continue
or
creating
more
clusters
to
support
to
support
the
increased
capacity
needs
so
yeah.
Thank
you
all
for
your
time
and
you
can
learn
more
about
our
engineering
work
on
our
blog
at
medium.com,
slash,
Airbnb
engineering,
and
we
are
also
hiring
so
head
over
to
careers,
airbnb.com
and
and
come
work
with
me.
Thank
you
very
much.