►
From YouTube: Panel: Improving and Managing Kubernetes at Scale - Xiang Li, Alibaba; Corin Dwyer, Netflix
Description
Speakers: Amit Bose, June Liu, Xiang Li, Harry Zhang, Corin Dwyer
Companies like Alibaba, Uber, and Pinterest are managing a huge fleet of machines with demanding and complicated workload. To evolve our infrastructure and adopt Kubernetes, we faced many challenges around scalability, reliability, flexibility and operationality. And today, after overcame those difficulties, we are running some of the largest Kubernetes clusters in the world.
In this panel, we would like to share our real world experience on improving and managing Kubernetes with harsh requirements. We believe the stories are interesting themselves, and many of the lessons we learned also apply to small-mid size cluster operators and users.
A
Hello,
Q
Khan.
There
we
go
how's
everybody
doing
this
afternoon:
energy,
energy
passion
all
right,
so
I'm,
Alan,
Stewart,
I'm,
a
principal
software
architect
at
Microsoft
and
Azure
engineering,
a
little
small
company
doing
a
few
things
with
cloud
and
I'm
here
to
introduce
our
steam
panel
talking
about
improving
and
managing
kubernetes
at
scale.
I
know:
that's
not
a
topic!
Anybody
is
concerned
with
here
right,
cuz,
kubernetes,
just
magically
scales
itself,
no
okay.
A
So
when
we
get
started
so
first,
we
have
Amit
who's,
a
senior
software
architect
at
uber,
okay,
Jun
who's,
a
staff
software
architect
at
Pinterest,
Jiang
who's,
a
senior
software
engineer
at
Alibaba,
Harry
who's,
our
esteemed
moderator,
for
the
session
who's,
a
software
engineer
at
Pinterest,
so
I'll
get
out
of
the
way
and
turn
you
over
to
our
colleagues.
Give
them
a
warm
welcome.
Oh
sorry,
I'm
this
somebody
and
we
have
Koren
who's,
a
senior
software
architect
at
the
netflix
and
chill'
company.
Thank
you.
B
All
right
welcome
to
the
panel
what
a
conference
has
been
to
almost
two
days
and
it's
been
great,
a
lot
of
talks
and
chats
and
panels
as
well.
So
welcome
to
today's
panel
to
talk
about
kubernetes
in
production
in
different
companies,
and
today
we
are
very
honored
to
have
invited
four
people
from
four
different
companies
and
all
of
them
are
serving
their
users,
their
customers
from
all
over
the
world.
There
are
Alibaba,
Netflix
and
Pinterest
and
uber.
B
So
before
we
actually
start
our
panel
I'd
like
to
do
a
small
poll.
The
first
point
is
about
kubernetes,
serving
how
many
of
you
guys
run
kubernetes
on
public
cloud
and
manage
them
yourself.
Please
raise
your
hand
wow
a
lot
of
them
on
public
cloud,
but
using
management
services
like
eks
or
in
KS
or
UK
eat,
plan
E,
okay,
how
many
of
you
guys
were
in
kubernetes
on
on
Prem
data
center
more
than
that?
B
B
Okay,
so
it
sounds
like
most
people
go
to
the
second
category.
They
like
how
many
of
you
guys
have
been
going
through
or
is
going
for
well
be
going
through
service.
My
gracious
from
existing
infrastructure
to
kubernetes
awesome
so
hopefully
like
for
the
companies
here
we
are
all
sharing
a
lot
of
stories
with
was
people
and
we're
gonna
take
questions,
and
hopefully
this
session
is
going
to
be
informative
and
helpful.
B
C
Sure
this
is
a
great
crowd,
more
than
I
expected
so
I'm
Koren,
Dwyer
I'm,
a
senior
software
engineer
at
Netflix
and
I
work
on
the
computer
form
team,
I,
primarily
work
on
the
control,
plane
and
scheduler
for
tightest
Netflix's.
Open-Source
multi-tenant,
container
management
platform
and
Titus
enables
a
variety
of
workloads
at
our
company,
ranging
from
machine
learning
jobs
to
stream
processing
to
the
video
encoders,
as
well
as
critical
web
services
that
power
the
Netflix
product
and
these
workloads
they
run
on
thousands
of
machines
or
hundreds
of
thousands
of
CPUs
across
three
Amazon
regions.
C
So
historically,
Titus
did
not
use
kubernetes.
It
was
built
on
meso
s--
as
a
uber
framework,
and
it
used
meso
as
a
way
to
manage
machines,
basically
handled
the
machine
registration,
as
well
as
a
mechanism
to
send
requests
to
create
and
terminate
containers.
And
earlier
this
year
we
decided
what,
if
we
were
able
to
replace
that
maysa
resource
provider
with
something
like
kubernetes.
So
we
hook
this
up
to
kubernetes
api
server
using
pods
and
nodes
and
some
c
RDS,
and
we
were
able
to
get
similar
functionality
and
we're
currently
in
the
process
of
migrating.
D
Hi
I'm
John
from
Alibaba
Technica
Alibaba
cloud,
so
our
team
is
a
call
native
team
at
ababa,
so
our
team
does
two
things.
So
why
is
to
manage
the
internal
infrastructure
for
the
Alibaba
ecommerce?
So
the
e-commerce
is
a
very
large
business
in
China
and
we
have
a
very
unique
use
cases
for
the
closer
management
system
and
our
team
also
serve
the
kubernetes
surveys.
His
colleague,
it's
very
similar
to
a
eyesore
gke.
The
services
are
Alibaba
cloud,
so
our
team
does
two
things
and
we
adopt
kubernetes,
I,
think
late
earlier
in
last
year.
D
Before
that,
historically,
we
had
our
in
hospital
management
system
and
we
moved
to
Cuban
areas
for
a
few
reasons.
So
first
we
believe
in
open
source,
especially
open
source
in
the
infrastructure
space.
Second
kubernetes
and
cloud
native
has
a
very
strong
community
and
we
want
to
absorb
all
the
innovations
into
Alibaba
infrastructure.
D
Third,
we
want
to
serve
our
customers
with
the
same
technology
that
we
use
inside
Alibaba
Group,
and
that's
why
we
started
our
journey
with
kubernetes
and
right
now
we
are
running
kubernetes
clusters
for
our
critical
worth
note
inside
Alibaba
Group
and
we
run
very
large
clusters,
several
thousand
nodes.
When
we
have
several
these
clusters-
and
we
runs
tens
of
thousands
of
party
in
those
clusters-
I
think
probably
we
and
the
unfine
angel'
are
running
the
biggest
companies
closer
in
the
world.
E
My
name
is
Jun
and
I
work
for
the
cloud
platform
team
at
Pinterest.
Our
team
runs
the
cloud
platform
at
Pinterest,
which
includes
beyond
word
and
the
kinetise
the
motivation
for
us
to
adopt
illuminators.
First,
we
want
to
relieve
our
engineers
from
the
pain
of
knowing
all
the
infrastructure
details
have
to
manage
their
own
runtime
and
we
just
couldn't
afford
the
luxury
of
every
team
needs
to
have
an
infrastructure
engineer
and
a
sari
to
do
that.
E
So
it's
a
long
time,
long
time
paying
for
our
engineers
in
thanks
for
kinetise,
which
both
its
extensibility
and
all
kinds
of
features
it
has
now.
Engineers
are
free
from
all
these
burdens,
and
we
also
inspired
by
the
operational,
is
earnest.
Where's
grenadiers
seems
like
a
cluster
of
operation.
Cluster
like
a
cluster
OS
upgrading,
has
always
been
a
pain
for
our
I
sorry's,
but
after
we
migrated
to
kubernetes,
we
have
done
several
countless
rounds
of
quantitative
upgrade.
E
Oh
s
upgrade
result
without
any
interference
with
our
applications,
or
our
users
has
no
knowledge
about
all
this
operation
at
all
and
our
inference.
Infra
governance
team
really
loves
how
easy
to
track
the
usage
on
the
communities,
computable,
her
phone
and
third
but
not
least
important,
is
we
are
really
loves.
A
great
community
support
with
kubernetes
was
all
result.
Is
the
CRD
expense
with
all
this
a
two
level
of
autoscaler?
E
Without
all
this
all
kinds
of
different
features,
our
cloud
platform
won't
be
so
so
successful
and
right
now
well
protocol
ization
cluster
and
we
it's
a
heterogeneous
cluster.
We
run
all
kinds
of
workloads,
including
machine
like
machine
learning
or
Big
Data.
We
have
services
virus
of
different
sizes
from
very
small
macro
services
to
the
huge
plates,
and
we
also
even
have
stateful
services
running
our
cluster
and
from
all
different
kind
of
teams.
F
Hello,
my
name
is
Amit
Bo's
I
work
on
the
uber
computer
infrastructure
team,
so
we
provide
cloud
infrastructure
for
running
uber
micro
services,
as
well
as
bad
jobs
like
for
machine
learning,
offer
big
data
so
as
of
today,
uber
has
a
very
mature
infrastructure
for
supporting
both
micro
services
and
bad
jobs.
That
is
based
out
of
Miso's,
and
it
is
it's
a
it's.
A
large-scale
system
like
to
give
you
some
idea.
We
have
over
35
clusters
and
our
biggest
clusters
have
more
than
6,000
notes
for
our
and
us
on
the
microservices.
F
We
have
like
each
cluster
running
like
more
than
hundred
K
containers,
with
10
K
updates
happening
per
day
and
on
the
bad
side
we
have
like
a
million
containers
being
supported
within
a
day,
so
we
operate
at
a
really
large
scale
and
we've
built
up
all
our
infrastructure
around
mesos.
But
over
time
we've
seen
that
the
community,
the
industry
is
kind
of
converging
around
kubernetes
and
that's
what
brings
us
to
communities
in
addition
to
our
driving
community
where
applications
are
now
built
on
kubernetes.
F
B
Thank
you.
Thank
you
guys.
We
are
at
different
stages
and
we
learn
our
lessons
and
different
stages
as
well.
So
for
today's
panel
we've
prepared
a
couple
of
questions
that
people
would
be
commonly
interested
in
and
we're
gonna
start
these
questions
first
and
have
our
panelists
share
their
ideas,
their
stories
and,
at
the
end,
we're
going
to
be
handling
over
the
microphones
to
our
audiences
and
take
the
questions
as
well.
So,
let's
start
with
something
at
the
very
high
level.
So
for
other
companies
are
we
starting
adopting
kubernetes
in
different
stages?
B
D
So
for
the
hosting
model,
we
provide
two
models.
So
first
we
have
shared
pool,
is
a
public
pool
so
that
criminals
cluster
is
managed
by
our
demons.
The
node
of
the
cluster
is
also
managed
by
our.
Our
team
is
like
a
service
model,
so
the
user
of
the
class
are
only
used
cube.
His
API
is,
they
can
submit,
drops
to
kubernetes,
submit
deployments
to
kubernetes,
that's
one
model
and
we
have
another
model,
some
of
the
business
unions.
D
They
won't
have
their
dedicated
kubernetes
clusters
and
we
do
program
providing
dedicated
kubernetes
to
those
customers
and
they
use
like
they
have
their
own
node
management
systems
and
they
connect
to
the
API
server
that
we
manage.
So
we
really
have
two
kinds
of
customers,
and
some
of
our
customers
also
want
to
run
our
leak
cloud
and
our
cloud
is
to
use
the
tactical
like
UK
as
type
of
thing.
So.
D
E
Sure
so,
just
as
I
mentioned,
we
wrongs
real
multi-tenancy,
Hector
Jenner's
cluster
for
the
whole
company.
So
we
all
of
our
teams
and
projects
they
share
the
same
cluster
and
they'll
even
share
the
note,
and
we
have
a
gateway
service
in
front
of
kubernetes.
So
the
users
can
only
program
all
talk
to
that
gateway
service
which
will
do
all
kinds
of
validation,
trip
away
all
kinds
of
stuff.
E
Well
after
they
pass
that
we
we
all,
they
can
submit
jobs
for
that
gateway
services,
and
we
we
our
side
a
way
to
have
a
some
small
way
to
have
some
separate
clusters
due
to
compliance
reason.
A
reason
we
take
security
as
very
with
tech
security
has
very
high
Ferraris
and
one
way
around
kubernetes.
We
found
it's
very
important
that
we
need
to
patents.
We
need
to
integrate
with
existing
infrastructures
we
have
before
the
kubernetes
world.
We
already
invest
a
lot
in
our
security
traffic
server
smash
all
kinds
of
B
infrastructures.
E
They
are
battered,
there's
no
way
for
us
to
abandon
them
and
the
a
battle-tested
for
our
scale,
and
we
found
we
we
average
CRT,
is
a
lot
to
integrate
kubernetes
with
and
that
they
both
that's,
enable
us
to
integrate
with
them,
and
our
users
even
doesn't
need
to
know
about
all
this
components,
while
they're
existing
and
migrating
from
the
wheel
or
the
to
the
kubernetes
world
is
much
easier.
Due
to
this
structure,
I
see.
B
F
C
B
Let's
move
forward
to
the
second
topic,
so
I
know
a
lot
of
people
are
interested
in
the
scale
how
fast
or
how
big
or
how
how
how
efficient
that
kubernetes
can
be.
So
the
second
question
is
about
like
what
is
the
scalability
and
how
efficient
kubernetes
is,
and
what
does
it
mean
scalability
and
efficiency?
What
does
it
mean
to
you
and
what
does
it
look
like
how
to
deal?
How
do
you
deal
with
this
problem?
F
So-
and
this
is
this-
is
like
something
that
we
need
at
least
because
we
we
want
to
support
our
big
data
workloads
which
are
currently
running
on
something
like
yin
and
Yan,
supports
much
higher
throughput
rates
towards
this
and
to
really
understand
whether
we
could
satisfy
the
scale
with
kubernetes.
We
did
some
experiments
on
aside.
For
example,
we
looked
at
etcd,
which
we
thought
might
be
a
bottleneck
and
simple
benchmarking,
simply
against
ABCD.
We
found
that
the
read
and
write
throughput
was
sufficient
for
our
scale.
Then
we
moved
on
to
testing
against
API
server
right.
F
So
you
have
a
simple
kubernetes
system
where
you
directly
launch
pods
against
the
API
server
without
involving
the
deployment
controller
or
the
scheduler,
so
that
you,
you
essentially
bind
our
pods
upfront
and
what
we
found
was
that
with
something
like
8k
node
cluster,
and
it
took
us
about
five
minutes
to
launch
40k
pods,
and
this.
This
was
something
that
was
not
acceptable
to
us.
So
then
we
went
ahead
and
investigated
what
was
wrong?
F
So
we
decided,
let's
just
drop
the
body
event
and
see
who
or
how
it
goes,
and
by
doing
so,
we
were
actually
able
to
reduce
the
time
it
took
to
large
launch
these
four
teacake
pods
to
about
30
seconds.
So
these
are
the
kind
of
optimizations
that
we
are
looking
at
to
find
out.
You
know
how
best
we
can
take
the
system
that
is
there
in
in
kubernetes
and
make
it
fit
to
our
needs.
B
D
Our
scale
is
pretty
large
is
similar
to
a
Google
bog.
So
initially
we
feel
that
scuba
D
may
be
the
hardest
part
for
kubernetes,
for
example,
for
the
API
server
or
ICD.
But
after
some
investment
we
feel
that
actually
API
server
and
a
city
is
flying
for
this
kind
of
skill,
and
that's
over
people
mentioned
that.
Actually,
the
storage
and
networking
is
the
hardest
part,
because
you
need
to
provide
in
the
C&I.
For
each
part,
you
need
to
provision,
say
a
precision
volume
for
each
container
that
actually
take
a
large
amount
of
time.
D
So
we
are
right
now
opening
writing
that
that
part
a
lot
and
for
for
efficiency
I,
think
we
not
only
care
about
the
efficiency
of
kubernetes
itself.
We
also
care
about
the
efficiency
for
the
node,
for
example,
how
much
CPU
utilization
you
can
heat,
and
for
that
we
do
collocations
with
co-locate
the
big
data
world,
always
our
online
serving
workload
and
cool.
It
is
not
designed
for
time,
so
we
made
some
modifications
to
that.
So
we
add
CPU
sets
with
the
current
civil
share
model
of
qu
of
coolant,
and
we
also
have
two
level
scheduling.
D
Sometimes
we
run
the
big
data
framework
on
top
of
kubernetes
and
dupont,
collocation,
which
means
that
we
allocate
a
very
large
part
for
big
data
framework
and
then,
whether
big
data
framework
to
launch
smaller
container
engine
or
smaller
process
in
that
container,
and
that
another
thing
that
we
are
doing
right
now
is
we
use
the
color
container
to
actually
write
some
of
the
big
data
work.
Not
because
we
found
that
some
of
the
big
data
work
of
actually
affecting
the
kernel
performance
and
that
affect
the
low.
C
D
Requirements
of
the
online
service
worker
so
I
think
efficiency
of
the
computing
application
is
also
important
for
the
alibaba
use
case,
and
we
also
care
about
the
developer
efficiency,
which
means
that
how
fast
the
developer
can
actually
deploy
their
application
to
the
kubernetes
platform,
and
for
that
we
are
improving
our
path
platform
to
make
it
more
kubernetes
native,
so
that
people
can
use
it.
So
previously,
the
path
platform
is
not
super.
My
company's
native,
we
migrated
from
the
old
system
to
the
new
system
right
now.
D
B
You
and
I
think
like
for
the
rest
of
the
two
companies
and
Netflix
and
Pinterest,
and
we're
currently
undergoing
massive
migrations
from
one
system
to
another.
So
let's
spend
some
time
in
talking
about
the
migrations.
What
is
the
biggest
or
kubernetes
adoptionism,
so
the
next
questions
is
going
to
be.
What
is
the
biggest
pain
point
you
guys
face
when
adopting
kubernetes
and
what
is
the
status
of
each
of
the
companies
dealing
with
that
Pinterest
Jun?
Do
you
want
to
start
with.
E
So
actually
was
doing
the
active
in
the
active
transition
stage
from
the
VM
war
to
the
vanities
world.
So
we
learned
a
lot
of
hard
lessons
actually
hard
lessons
during
the
migration
world.
So
one
one
thing
is
that
you,
you
can
never
just
it's
not
achievable
that
you
build
a
perfect
cluster,
perfect
computer
class
platform
for
oil,
for
all
your
customers.
You
have
to
do.
A
lot
of
patches
differ
in
the
future
design.
Different
features
different,
also
implement
all
different
the
tools,
while
you
are
onboarding
each
customers.
E
So,
as
example
like
we,
we
build
our
visibility,
dashboard
for
the
whole
class
to
a
while
onboarding,
our
largest
service
fleet,
and
we
take
their
the
the
team,
the
web
apps
team,
their
feedback
Mauricio
and
try
to
operate
over
a
visibility
dashboard
as
soon
as
possible.
We
get
their
feedback
and
we
also
discover
a
lot
of
issues
with
like
how
we
manage
the
tens
of
low
tensile
operators.
E
Otherwise,
if
you
make
the
promise
and
cannot
deliver
that,
that's
really
bad
and
on
the
other
side,
we
found
that
we
have
a
lot
of
overhead
on
supporting
the
customers.
Think
about
its
active
transition,
States
and
our
engineers.
They
are
very
comfortable
and
very
used
to
the
old
VM
world.
They
know
of
the
concept
of
the
VMS
VPC,
all
of
this
old
stuff
they
can
access
which
into
the
holster
ways
of
security,
supported
to
all
kinds
of
debugging
here.
But
it's
not
that's
easy
to
support
all
this
kind
of
work.
E
A
work
just
work
the
way
of
their
working
on
communities
so
once
they
jump
to
the
kubernetes
shape
they
found.
Oh,
why
can't
I
as
such
to
my
products.
The
connection
is
now
so
stable.
Why
can
I
not
do
this
and
many
times
I
need
to
talk
to
my
customers?
See
we
build
is
a
login
part
of
pipeline
into
our
clusters.
You
need
to
read
more
logs,
so
this
is
man-size
change.
We
have
to
support
them
and
do
a
lot
of
education,
so
we
found
that
educational
sessions
or
code
labs.
C
So
I
think
I
haven't
actually
mentioned
this
yet,
but
so
Titus
doesn't
really
use.
I
didn't
mention
this
explicitly.
Titus
is
not
using
all
the
kubernetes
pieces
at
this
point,
we're
only
using
api
server
at
CD
and
virtual
couplet,
which,
if
you
haven't,
heard
virtual
couplet,
yet
you
can
think
of
it
more
as
a
high
level
shim
that
understands
the
the
couplet
protocol,
but
has
extension
points
to
where
you
can
plug
in
your
own
logic.
C
So
because
it's
not
only
the
containerized
workloads
or
applications,
but
also
how
do
we
do
things
like
monitor
our
kubernetes
clusters?
So,
for
instance,
we
don't
use
Prometheus
at
nav.
We
have
our
own
system
called
Atlas,
that's
open
source,
so
we've
had
to
do
a
lot
of
work
on
learning
how
to
convert
from
one
concept
to
another,
as
well
as
just
gaining
experience,
and
this
newer
technology
for
us.
Thank.
B
G
So
one
of
the
things
I've
seen
here
across
UConn
is
that
it's
not
just
running
kubernetes
cluster
you've
got
all
kinds
of
add-ins
got
it
in
got
custom
resources
got
operators.
How
do
you
guys
manage
the
upgrades
for
those?
Do
you
do
you
plan
upgrades
an
existing
cluster,
or
do
you
build
a
new
cluster
with
the
new
stuff
and
expected
migrate
workloads
over
shall
I.
D
Think
that's
a
very
interesting
question
because
abhava
we
have
boards
well
more
than
starting
controllers
and
we
have
a
lot
of
sub
party
operators
and
how
to
make
sure
that
those
controllers
are
doing
the
right
thing
and
how
to
make
sure
that
those
operators
are
safe
is
a
big
challenge
for
us
right
now.
So
we
are
trying
to.
We
are
trying
to
things
so
one
thing
we
are
trying
to
do:
we
try
to
build
a
set
car
controller.
D
So
basically
we
want
to
make
sure
that
our
controller
will
not
talk
to
ApS
over
directly,
but
they
talk
to
a
PS
over
through
a
sidecar
and
we
started
manage
those
set
cars
to
make
sure
that
they
are
not
doing
crazy
stuff,
and
the
second
thing
that
we
are
doing
is
to
limiting
the
access
of
all
these
controllers
by
injecting
another
API
server
between
the
master,
API
server
and
the
controller.
Basically
the
controller
that
we
don't
trust
only
access
to
one
small
API
server
and
that's
is
we
call
it
virtual
cluster.
D
E
E
We
have
a
very
grand
test
for
our
you
know
for
our
own
developers,
we
have
staging
clusters
and
we
have
a
production
cluster,
so
users
need
to
all
the
deployment
ghosts
starting
from
the
playground
clusters
after
they
pass
all
the
and
end-to-end
tests
they
promote
to
the
staging
cluster
and
then
to
the
protocol,
sir.
So
we
have
all
this
graduation
and
for
some
critical
operator
like
as
a
kubernetes
version
upgrades,
we
even
we
even
requires
a
baking
time
on
the
station
cluster
for
at
least
one
week,
and
we
have
money.
E
Of
course
we
have
monitoring
on
the
engine
tests
and
all
the
other
monitoring
we
have
and
the
other
the
last
the
last
layer
actually
is.
We
require
like
very
critical
users
to
critical
applications.
The
master
have
a
presence
on
the
staging
cluster
and
have
monitoring
on
their
workload
on
the
staging
cluster.
So
whenever
a
new
version
of
cluster
components
two
rows
to
the
staging
cluster,
if
these
receive
any
alerts
of
their
own
publications,
we
need
to
look
into
that
post.
You
need
to
look
into
that
and
see
that
I
will
say.
E
D
G
C
I
guess
the
compute
platform
team
at
Netflix
is
11
people
and
we
design
build,
operate
and
support
the
entire
company
and
I
mean
I'm
sure
everyone
here
would
love
more
resources,
but
I
think
I.
Think
it's
just
a
you
have
to
do
a
good
job
of
balancing
and
I
think
you
got
to
just
keep
monitoring
all
these
aspects,
and
you
have
to
figure
out
and
make
a
case
when
it's
no
longer
feasible
for
you
to
continue
on
with
your
current
responsibilities.
F
Uber
has
about
25
engineers
in
the
computing,
and
one
of
the
things
that
we
find
is
works
for
us
with
a
small
beavers
to
automate
as
much
as
possible.
Right.
If
you
build
automation
that
does
the
mundane
tasks,
move
that
out
of
the
way,
then
you
can
actually
concentrate
on
doing
something
more
value-added.
So
that's
what
we're
trying
to
do!
Pinterest.
B
E
So
who
computer
platform
team
has
less
than
one
less
than
ten
people
and
half
of
it
is
doing
service?
Mesh
and
Heather
has
us
doing
kubernetes
and
other
computer
form
stuff,
and
we
need
to
support
the
whole
company.
So
that's
I
said
we
need
to
budget
our
engineering
resource
work
very
carefully.
Otherwise
we
don't
the
first
thing.
Is
you
never
want
to
promise
some
feature
that
you
are
not
going
to
deliver,
and
so
this
is
a
very
carefully
and
we
also
in
his
way,
but
I
say
we
we
need
to
we.
E
D
You
we
have
about
ten
people
working
on
coke,
Bernie's
and
other
ten
people
working
on
the
I.
Sorry
said
all
the
things,
but
we
also
have
other
people
working
on,
say,
capacity,
planning,
application,
support
and
like
past
platform
and
our
other
areas,
so
the
overall
team
is
pretty
large,
but
about
twenty
people
working
on
kubernetes
alone.
Thank.