►
From YouTube: Only Slightly Bent: Uber’s Kubernetes Migration Journey for Microservices - Yunpeng Liu, Uber
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Only Slightly Bent: Uber’s Kubernetes Migration Journey for Microservices - Yunpeng Liu, Uber
Uber started using docker containers at scale in 2015, and has gone through a few generations of cluster management and service discovery technologies. In early 2019, we started working on migration from Mesos to Kubernetes to support secure service mesh and machine learning workloads. This is a complex problem - there are thousands of services, tens of millions of containers to be launched daily while maintaining high machine resource utilization. To that end, a lot of customizations are built into our Kubernetes stack including elastic resource sharing, oversubscription, fast rollback and deploy, changes to service discovery and attestation etc. This talk will cover: - Overview of Uber Compute Infra - API server benchmark and tweaks - Custom controller and scheduler logic - CRI: resource, health check, logging, isolation - SPIRE and service discovery setup at Uber
https://sched.co/Uabh
A
Okay,
hello:
everyone
thanks
for
joining
me
in
this
session.
My
name
is
Jim
Payne
software
engineer
at
Wooper,
I'm,
working
on
building
and
maintaining
compute
infrastructure
at
Lubert.
So
into
this
talk,
I'm
going
to
share
the
lessons
and
the
takeaways.
My
team
has
learned
from
evaluating
and
exploring
kubernetes
migration
for
uber.
A
Okay.
This
is
the
agenda
of
today's
talk,
we'll
all
first
take
an
overview
of
uber
and
the
uber
z--
compute
infrastructure
and
explain
the
team's
motivation
of
moving
to
kubernetes,
then
we'll
all
dive
into
the
evaluation
and
exploration
experience
of
communities,
migration
at
uber
at
the
end,
we'll
all
summarize
our
learnings
into
a
few
takeaways
and
to
share
with
the
audience
today.
A
Okay,
before
we
kicking
off
any
technical
discussion,
let's
first
take
a
look
at
uber
business
skill,
so
uber
has
completed
more
than
15
billion
community
cumulative
trips
over
15
million
uber
chips
happen
on
the
platform.
Every
day
now,
uber
is
available
in
more
than
700
cities
across
65
countries
on
six
continents,
uber
has
surpassed
hundred
million
monthly
active
users
in
June
2019,
and
over
3.9
million
active
drivers
globally
in
q4
2018,
as
a
company
uber
now
has
more
than
27,000
employees
worldwide
and
more
than
3,000
engineers
would
like
to
build
in
the
operate.
A
Such
a
large
commute
platform.
There
are
thousands
of
software
systems,
and
the
service
is
constantly
running
behind
the
scene
and
uber
has
built
its
own
computer
infrastructure
to
a
skill
to
support
these
systems
and
services.
Let's
start
with
a
few
statistics
to
review
the
scale
of
ubers
computer
infrastructure.
Hooper
currently
has
thousands
of
micro
services
running
on
development
activities.
Uber
engineers
viewed
more
than
5,000
of
container
images
every
day
deploy
more
than
thousand
of
micro
services
and
more
than
ten
thousands
of
container
instances
every
day.
A
So
the
spitless
service
is
actually
the
main
resource
consumer
on
the
massive
compute
platform
uber
has
built.
Uber
malos
is
stateless
workload
as
micro
services,
as
we
mentioned
previously,
there
are
thousands
of
micro
service
raining
on
the
uber
platform.
They
are
mostly
containerized
and
right
now
running
on
the
missus
custard.
To
get
to
the
current
state.
Roopa
has
gone
through
a
few
major
migrations.
A
At
the
beginning,
everything
was
running
on
bare
metal,
no
containerization
at
all
in
early
2015,
Uber's
started
to
continue
rise
services
and
it
deployed
its
own
container
ox
attrition
system
to
support
a
static
containers,
placement
in
2016
uber
adopted
apache
missiles
and
apache
aurora
as
its
new
resource
management
and
container
oxidation
system
later
on,
who
were
also
deployed
and
developed
its
own
missile
scheduler,
as
known
as
peloton
to
replace
Aurora
so
thoroughly
speaking.
If
you're,
looking
at
the
history
of
uber
compute
infrastructure
is
a
long
journey
full
of
migrations.
A
So,
besides
the
massive
amount
of
stylist
workload,
there
are
also
other
use
cases
on
the
uber
computer
infrastructure.
There
are
a
large
volume
of
batch
workload
running
on
compute
clusters
and
where
our
co-locating
makes
the
workload,
including
both
billets
in
the
batch
on
the
shared
clusters
for
workload
that
requires
Hardware
customization
computer
infrastructure
actually
offers
centralized
support
to
random
on
dedicated
hardware's.
A
A
So,
given
the
use
cases
we
have
uber,
compute
team
has
developed
its
own
customized
resource
scheduler.
As
known
as
peloton
as
I
mentioned,
it's
mainly
fulfills
the
following
needs.
It's
a
unified
resource.
Scheduler
for
co-locating
makes
the
workload
on
compute
clusters
it
integrates
with
different
applications.
Obstetrician
systems
such
as
SPARC
tensorflow
and
the
you
deploy
it
can
be
run
both
on
promise
or
in
the
cloud
and,
more
importantly,
it
supports
elastic
resource
sharing
among
teams
and
organizations.
A
A
So
so
far
we
have
goes
through
the
overview
of
uber
compute
infrastructure
and
I.
Haven't
mentioned
any
single
word
related
to
cloud
native
and
kubernetes.
So
people
may
ask
it
seems:
obrah
has
already
developed
an
entire
computer
ecosystem
based
on
attached
missiles,
and
it's
working
fine
right
now
why
we
are
talking
about
kubernetes.
A
And
let
me
share
with
you
guys
here
is
the
why
so
the
actual
motivation
to
migrate
to
kubernetes
derived
from
the
real
business
requirements.
So
a
big
challenge,
uber
engineering
team
currently
faces,
is
how
to
ensure
authentication
and
authorization
for
inter
service
communication
among
thousands
of
micro
services.
A
common
solution
would
be
just
building
a
secure
service
match.
It
will
help
address
the
security
and
compliance
concerns
without
having
to
change
the
code
of
thousands
of
micro
services
so
to
build
a
secure
service
match.
A
Uber
compute
infrastructure
needs
to
support
part
inside
cars,
but
unfortunately,
the
existing
missiles
based
architecture
doesn't
really
support
pod
and
it's
not
easily
extensible,
since
it
requires
using
task
groups
and
the
nested
containers
feature
with
unified
container
eliezer
in
missiles,
which
is
only
supported
by
missiles.
In
this
case,
if
we
use
it,
we
will
actually
lose
the
container
runtime
portability.
A
Hence
we
started
to
actually
explore
other
alternatives.
It's
not
hard
to
realize
that
kubernetes
has
native
support
for
pod,
sidecar
security,
plugins
and
init
containers,
so
these
are
all
necessary
or
useful
features
to
build
secure
service
smash.
Additionally,
we're
also
impressed
by
the
thriving
community
supported
kubernetes
has,
and
if
we
build
a
new
solution
on
top
of
kubernetes,
it
will
be
natively
supported
by
new
cloud
native
applications
as
well.
A
But
what
this
really
means,
so
basically
this
means
a
major
iteration
on
the
core
technologies
that
currently
powers
uber
computer
infrastructure.
We
must
first
evaluate
this
is
indeed
a
viable
solution
for
existing
business
needs
as
well.
So
there
are
two
major
concerns
wrists
against
the
kubernetes
internally
over
the
first
one
is
scalability,
so
we
often
heard
people
questioning
about
kubernetes
cluster
skating
ability
beyond
5k
notes
and
currently
uber
already
have
clusters
larger
than
five
keynotes
uber
computer
infrastructure
also
runs
large
volume
of
batch
workload.
A
We
have
to
ensure
coronaries
can
handle
the
existing
batch
workload
footprint
within
over.
The
second
concern
we
have
is
around
portability.
So,
as
we
mentioned
in
the
overview,
rubra
has
existing
customized
service
orchestration
systems
for
different
use
cases,
and
this
support
unique
service
deployment
characteristics
such
as
instance,
count
based,
sharding
and
in
place
upgrade.
We
need
to
ensure
the
technology
iteration
under
needs
is
easily
portable
to
those
upstream
services
and
the
systems
as
well.
A
So
after
gathering
feedback
and
summarizing
the
concerns
as
engineers
who
thrived
before
technologies,
we
decided
to
go
ahead
and
evaluate
and
address
those
concerns.
So
we
focused
on
scalability.
First,
the
basic
methodology
to
evaluate
a
software
systems.
Scalability
is
benchmarking,
so
we
started
with
benchmarking
SCD,
since
it
is
the
persistent
storage
solution
kubernetes
uses
under
needs,
so
we
evaluated
as
CD
writes
throughput
against
the
key
size
in
the
value
size
on
its
public
announced
the
documents
a
CD
claimed
it
can
support
up
to
50
K
bytes
per
second
and
Henry
50k
reads
per
second.
A
This
far
exceeds
our
needs,
but
we
wanted
to
run
additional
benchmarking
against
a
larger
value
size.
The
sample
benchmarking
result
on
the
right
side.
Demos
trace
the
throughput
demonstrate
that,
though
the
value
says
actually
effects
as
CDs
through
good
performance,
but
it
can
still
provides
good
reads
through
write,
throughput
with
large
value
size,
given
how
cornetist
actually
use
a
CD.
We
also
wanted
to
rent
additional
benchmarking
against
a
number
of
watches
on
that
CD.
A
A
A
So
as
an
example,
benchmark
results
showed
launching
for
TK
pass
on.
8K
notes
took
a
few
minutes.
This
is
not
really
a
convincing
results
for
kubernetes
migration,
but
we
didn't
stop
there,
because
one
interesting
thing
we
noticed
in
the
mint
benchmark
result
is
the
number
of
pod
events
stored
in
sed
actually
10
times
of
it
certain
times
of
the
number
of
pots,
as
we
showed
off
in
one
example
graph
on
the
right
side,
so
to
find
out
why
we
went
back
and
dig
deeper
into
the
party
events.
A
So
after
some
code
reading
and
researching,
we
had
some
interesting
findings.
It
turns
out
that
the
large
volume
of
holding
events
consumed
networking
storage
resources
for
both
kubernetes
api
server
and
SED,
but
they
are
not
really
persisted
states
that
coober.
Let's
leverage
to
operate
pots,
there
are
more
as
locks
in
fact,
having
TTL
config
on
them
also
indicates
the
same
conclusion.
At
the
same
time,
we
are
also
engaging
with
folks
from
other
companies.
A
They
share
the
same
thoughts.
It
helped
us
confirm
that
this
is
actually
the
right
direction
to
go
with.
Based
on
the
new
learnings,
we
ran
the
benchmarking
against
sed
and
coop
API
server
again,
but
this
time
with
our
job
in
all
the
parliaments,
the
throughput
had
significant
improvements.
It
only
took
30
seconds
to
launch
40k
pass
on
8k
notes,
which
is
a
pretty
decent
benchmark
results
to
demonstrate
the
scalability
of
a
CD
and
cube
API
server.
A
Okay,
so
after
a
few
rounds
of
scalability
evaluation,
we
agreed
on
the
following
statements.
So
SCD
performance
is
heavily
impacted
by
a
number
of
watches,
which
is
more
or
less
linear
to
the
number
of
nodes.
But
overall
it's
a
good
enough
for
uber
footprint,
a
city
plus
cube
API
server
performance
on
part
creation
and
launch.
It's
also
good
enough
for
uber
footprint.
After
topping
the
parliament's.
A
Next,
we
moved
on
to
address
the
portability
concerns
with
the
scalability
evaluation
conclusion
in
mind.
As
we
mission
above
uber
already
has
peloton
as
a
unified
resource
scheduler.
It
provides
unique
values
to
the
uber
ecosystem.
We
needs
to
carefully
consider
how
the
integration
would
work.
The
first
alternative
we
came
up
with
is
to
actually
modify
peloton
to
become
a
kubernetes
scheduler
plugin.
This
diagram
on
the
right
side
briefly
demonstrate
how
the
overall
solution
would
looks
like
basically,
a
CD,
pupa,
PI,
server
and
cubelet
was
adopted.
A
Directly
peloton
will
be
modified
as
a
kubernetes
scheduler
plugin,
and
we
need
to
develop
custom
controllers
to
bridge
existing
service
education
system
with
kubernetes
api,
so
they
are
obviously
pros
in
the
columns
for
this
alternative
for
the
pros.
This
is
basically
a
idea
Matic
for
criminalized
eco
system,
because
we
are
leveraging
the
suggested
plug-in
model
to
implement
custom
schedulers
on
top
of
kubernetes,
and
also
in
this
we
can
actually
reuse
a
lot
of
kubernetes
functionalities
and
team
overall
agree.
Eventually,
this
is
how
we
should
get,
but
there's
also
counts.
A
The
first
one
is,
the
change
is
at
a
very
large
scale,
so,
especially
considering.
If
this
is
the
first
step
of
a
migration,
it
will
be
very
risky.
Second,
the
migration
is
kind
of
costly,
because
essentially
we
need
to
swap
the
underneath
AP
is
the
upper
stream
systems
are
using
from
pallet
on
to
kubernetes
api
and
no
need
to
say,
because
we
can
tell
from
the
graph
we
needs
to
build
actually
many
translation
layers
to
bridge
between
the
systems
with
kubernetes
api
server.
A
Okay,
the
other
alternatives
were
also
considered,
is
to
actually
have
peloton,
integrating
with
both
kubernetes
and
missiles,
mostly
as
it
is
right
now.
So.
Looking
at
architecture
diagram
on
the
right
side,
peloton
host
manager
will
be
responsible
for
southbound
integration,
with
missiles,
master
and
communities
API
server.
A
There
should
be
no
significant
changes
for
the
northbound
API
integration,
with
a
variety
of
upstream
systems
in
this
alternative
kubernetes
and
the
missiles
can
actually
share
the
same
peloton
control
plane
and
the
peloton
host
manager
will
hide
away
details
of
kubernetes
in
the
missiles
from
upstream
systems
and
thus
impact
on
jobs.
Back
in
this
case
can
be
launched
on
either
missiles
or
kubernetes.
Casters.
A
Okay,
if
we
zoom
into
the
integration
point
for
the
second
alternative
further,
this
alternative
essentially
requires
to
build
an
abstracted
interface
to
operate
an
internal
host
cash
within
peloton
host
manager.
It
requires
a
plug-in
implementation
for
both
kinetic
missiles.
The
missiles
plug-in
will
be
responsible
for
communicating
with
missus
master,
and
the
KS
plug-in
will
be
responsible
for
communicating
with
cube
API
server.
A
The
pros
of
the
second
alternative
is
it
will
allow
missiles
in
the
kubernetes
to
share
the
same
peloton
scheduler.
The
scope
of
this
change
is
much
smaller
compared
with
the
first
alternative
and,
most
importantly,
it
makes
the
swap
invisible
to
clients
and
helps
to
achieve
it
as
parent
immigration
as
an
opposite
to
the
other
alternative.
This
solution
is
not
automatic,
it
doesn't
really
leverage
the
criminality,
scheduler
plug-in
model,
and
there
are
fundamental
differences
between
missiles
and
the
kubernetes,
which
actually
raised
a
bunch
of
consistency.
Challenges
when
designing
the
host
cache
inside
Python
host
manager.
A
So
now,
let's
go
back
to
the
portability
concerns
we
presented
previously.
Apparently,
the
second
alternative
stands
out
since
it
doesn't
really
swap
the
northbound
API
from
pallet
on
to
kubernetes
and
the
integration
challenge.
The
integration
changes
required
from
different
existing
systems
on
top
of
pallet
on
our
well
scoped
and
limited.
It
keeps
pallet
on
as
the
schedules
on
kubernetes.
It
provides
more
flexibility
to
ensure
support
for
those
existing
unique
service
deployment,
characters
characteristics.
A
A
So
after
evaluating
the
scalability
and
the
portability
challenges
we
are
convinced,
migration
to
kubernetes
is
viable
within
the
compute
infrastructure
for
uber,
but
in
the
entire
oeuvre
ecosystem.
There
are
broader
integration
challenges
we
need
to
solve
so
I'm,
at
least
at
least
dated
a
few
of
examples.
The
first
one
is
logging,
so
uber,
currently
Buse
in
the
maintains
custom
logging
infrastructure
based
on
Kafka
and
the
elastic
search
as
part
of
the
criminales
migration.
Compute
team
needs
to
come
up
with
a
solution
to
integrate
uber
logging
infrastructure
with
kubernetes.
A
The
next
one
is
security.
For
example,
ubers
security
engineers
are
looking
into
using
spire
for
service
eternity.
Then
compute
team
needs
to
work
with
the
security
engineers
to
provide
proper
integration
with
spire
in
kubernetes
and
other
big
challenges.
Uber
builds
its
own
service
discovery
and
routing
solution
back
in
2016.
To
solve
specific
problems,
uber
was
facing
to
figure
out
a
solution
to
support
existing
service
discovery
in
the
routing
without
making
changes
in
the
thousands
of
micro
services.
A
A
A
A
So,
basically,
for
the
kubernetes
migration,
the
first
half
of
the
solution
would
provided
is
based
on
kubernetes
container
runtime
interface,
so
kubernetes,
container
runtime
interface
is
a
plug-in
interface,
which
enables
cubular
to
use
a
wide
variety
of
container
runtime
technologies
without
the
need
of
recompile.
The
CRI
mainly
consists
of
a
program
through
a
protocol
buffers
Java,
C
API,
and
the
libraries
works
as
a
middleware
between
cubelet
and
actual
container
runtime,
and
provides
the
ability
to
manipulate
the
request
and
pod
sandbox
cycle.
A
So
this
flow
chart
shows
a
few
key.
Our
pcs
in
the
CRI
I
will
walk
through
it
to
describe
the
customization
we
introduced
here.
So
first
before
running
hot
boxes,
odd,
the
uber
part
UID
annotations,
will
be
injected
into
the
request
annotation
and
it
will
contained
the
part
of
UID
associated
with
the
sandbox.
This
annotation
will
be
needed
later
in
the
remove
sandbox
call
in
order
to
remove
the
host
lock
directories
associated
with
the
sandbox.
A
After
stop
pause,
sandbox
is
caught
and
succeeds
the
time
stamp
that
the
sandbox
was
stopped
at
will
be
recorded
and
before
removed.
Palestine
box
is
called
CRI
will
actually
check
the
timestamp
of
when
the
sandbox
has
been
stopped.
Crio
war
failed
a
sandbox
removal
request
with
an
aborted
States.
If
it
is
not
enough,
time
has
passed
since
the
removes
in
a
box
in
the
future
by
default.
A
Kubernetes
will
try
again
to
remove
the
sandbox
CRI
will
allow
the
request
to
continue
only
if
there
has
been
enough
time
passed
since
the
sandbox
was
actually
stopped,
so
this
customization
in
the
CRI
actually
allows
us
to
specify
locations
of
local
log
files
associated
with
the
pod
ID
and
set
up
host
Direction
a
host
directory
mount.
It
also
allows
us
to
persist
local
logs
for
a
configurable
short
time
window,
even
after
killing
the
policy.
A
So
the
second
half
of
the
solution
is
a
custom
darker
logging
driver.
It
is
responsible
for
logging
to
stand
her
out
in
a
standard
and
supports
streaming
them
in
two
separate
files.
It
also
supports
custom
log
file,
location,
which
is
actually
not
supported
in
the
doctor
default
JSON
format.
Logging
driver
additionally,
it
also
handles
lossless
log
rotation
and
optional
log
compression,
so
combining
the
CRI
customizes
and
the
custom
doctor
logging
driver
together.
We
actually
provided
a
log
in
solution,
meets
all
the
requirements
to
integrate
it
with
uber
log
in
infrastructure.
A
Ok,
let's
move
on
the
second
case
study
we're
going
to
talk
about.
Basically,
we
were
looking
into
clustering
networking,
so
you
guys
probably
can
recall
app
mentioned
several
times
in
the
presentation.
The
uber
has
its
own,
basically
a
different
custard
networking
setup
compared
with
kubernetes
permitted,
and
it
is
actually
a
big
challenge
for
kubernetes
migration
at
uber.
In
kubernetes,
every
pods
gets
its
own
IP.
A
The
Yoopers
misses
classes
right
now,
all
the
containers
actually
shared
a
host
network,
namespace
uber
software
networking
system
relies
on
host,
IP
and
dynamic
ports,
a
location
for
service
discovery
and
robin.
As
a
result
of
this,
there
is
actually
no
IP
tables
management
in
ubers
computer
infrastructure.
A
A
A
And
the
solution
we
come
up
with
is
actually
using
CNI,
plugin
and
again,
together
with
the
CRI
customization,
so
the
diagram
on
the
right
side.
Briefly,
demos
is
how
the
network
traffic
is
forwarded.
For
example,
the
egress
traffic
from
the
service
container
will
be
first
forwarded
to
the
routing
sidecar
container
part.
First,
then,
the
traffic
coming
out
of
the
rotting
sidecar
well
goes
through
scene,
ID,
port
mapping
plug-in
and
before
it
it
to
the
host
Network
namespace
and
eventually
leaves
the
host.
A
It
allows
us
to
actually
adopt
kubernetes
IP
per
pod
primitives
and
still
using
the
existing
host
IP
and
a
dynamic
port,
a
location-based
service
discovery
and
rotting
solution
to
solve
the
service
dependency
on
static
port
on
host
network
namespace.
We
have
to
ensure
that
services
can
talk
to
a
list
of
whitelisted
ports
over
loopback
IP
when
a
container
is
started.
The
customization
ECRI
modifies
the
IP
table
the
pot
networking
namespace
to
redirect
packets
from
selected
ports
on
the
pots
localhost
to
the
same
set
of
ports
on
the
host
logos.
A
This
customization
in
CRI
will
also
clean
up
the
IP
table
rules
before
stopping
the
sandbox,
the
pod
sandbox.
So,
combining
this
two
piece
of
work
together,
we
are
able
to
migrate
the
gap
to
mitigate
the
gap
in
clustered
networking
setup
without
actually
modifying
the
source
code
of
thousands
of
services.
A
Ok,
so
we
finally
reached
the
end
of
all
the
technical
discussions
we
would
like
to
share
in
this
talk.
But,
more
importantly,
we
summarize
the
learnings
uber
computer
team,
god
from
evaluating
and
exploring
criminales
migrations
for
uber
in
the
past
few
months
into
three
key
takeaways.
We
want
to
actually
share
them
with
the
audience
today,
so
the
first
one
is
active.
A
Community
engagement
is
very
important,
for
example,
when
we're
making
the
decision
of
chopping,
poly
events
engaging
with
the
community
action
and
talking
with
engineers
from
other
companies
actually
allow
us
to
learn
from
the
work
other
people
already
are.
They
did
and
helped
us
confirm
right
technical
decision.
We
are
going
to
make
second
hand
song
evaluation,
actually
helped
in
decision
making
a
lot.
A
A
Last
portability
is
actually
the
key
of
massive
migration
at
companies
like
uber
any
infrastructure,
migration
and
infrastructure.
Migration
is
like
it's
very
likely
to
fill
if
it
actually
requires
changes
in
upstream
systems
and
the
thousands
of
services
when
solving
variety
of
integration
challenges,
the
team
always
take
the
portability
into
consideration
and
make
the
migration
as
trying
to
earn
as
we
can.
B
A
So,
fairly
speaking,
all
the
work
we
have
presented
are
only
for
exploring
and
evaluation.
So
we
as
an
engineer
team,
we're
providing
a
proper
technical
solution
to
the
management
to
actually
demonstrate
how
the
migration
would
looks
like
and
how
both
scope
to
work.
So
it's
actually
up
up
to
the
management
to
decide
when
and
how
much
resource
is
proper
for
the
company
to
actually
invest
in
this
migration.
So
fairly
speaking,
none
of
the
solution
is
actually
working
in
production.
Right
now,.
E
A
A
A
A
Good
question
so
choosing
the
second
alternative
in
terms
of
pallet
town
and
the
kubernetes
integration
is
mainly
focusing
on
the
portability
concerns
it's
not
really
for
actually
trying
to
keep
both
missiles
in
the
kubernetes.
It's
mostly
trying
to
make
the
internees
swap
from
missiles
agents,
for
example,
to
cuba
less
transparent
to
the
appetites,
so
the
long
term
go
definitely
more
so
based
on
the
evaluation
we
have.
The
long
term
goal
should
eventually
migrate
off
mesas
ecosystem
and
fully
up
to
the
community
system.
A
C
So
generally,
when
you
run
like
a
system
at
scale
and
it's
been
identified
by
some
of
the
people
and
you're
reconfiguring
the
cycler
policies
at
a
frequent
time,
those
V
configurations
can
cause
like
a
lot
of
odd
events
which
can
cause
like
basically
the
loop
not
to
scale
so.
But
in
your
case
you
just
mentioned
you
could
drop
if
more
demands.
But
how
do
you
plan
to
scale
if
there's
like
a
lot
of
reconfiguration
going
on.
C
So
how
like
you
might
have
to
reconfigure
the
pod,
the
psych-up
problem
proxies
or
the
pods,
and
it's
being
shown
that
earlier
that
when
there's
a
lot
of
sidecars
or
these
proxies
and
if
you
reconfiguring
them,
which
this
can
cause
like
a
scalability
issue,
how
do
you
handle
this?
How
do
you
plan
to
handle
this?
That's.
A
A
pretty
good
question
because,
as
soon
as
we
type
dip
into
the
apart
+
psych
our
bottles,
we
realize
the
sidecar
is
actually
pretty
expensive.
To
be
honest,
it's
something
that
we
should
probably
be
careful
worse
when
you're
introducing
new
sidecars,
so
I
can
give
a
brief
estimation.
For
example,
right
now
on
the
uber
computing
infrastructure,
we
have
more
than
10
sort
of
inference
or
say
host
agents
running
if
we
think
about
cost
concern
rising
every
single
of
them
into
a
converting
into
a
sidecar.
A
It's
it's
basically
causing
a
massive
amount
of
resource
waste,
potentially
as
well
as
some
turns
in
the
cluster
performance
as
well.
So
that's
a
very
good
question.
I!
Don't
really
have
a
good
answer
for
you,
but
one
thing,
I
can
say
is
one
sort
of
principle
the
team
is
taking.
Is
one
we're
thinking
of
a
need
of
a
sidecar?
We
are
very
careful
and
thoughtful.
We
only
introduce
a
car
only
if
it
is
necessary.
Okay,.