►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Running eBay’s High-Performance Workloads with Kubernetes - Xin Ma, eBay
In the past two years we’ve been expanding our k8s deployments significantly fast by moving more and more production workloads into kubernetes. We’re now running multiple thousand-node k8s clusters from our data centers across the world, hosting various production workloads, including software load balancers on the edge, search engine, production databases, hadoop clusters, and GPU / machine learning applications, etc. Most of these workloads are critical to eBay’s business requiring higher performance and reliability. In this presentation, I’ll share our best practices as well as lessons learned when migrating high performance production workloads into containers running with kubernetes. You will hear stories to containerize applications without code change, leverage k8s capabilities to turn them into good k8s citizens, and more importantly, how we tune and optimize their performance.
https://sched.co/MPct
A
Morning,
everyone
welcome
to
my
session
I'm
Shane
from
eBay
and
today,
I'm
gonna
talk
about
how
we're
on
high
performance
wore
clothes
at
scale
with
kubernetes
and
you're.
Gonna,
hear
lots
of
best
practices
and
lessons
learned
for
our
performance
and
scalability
here's
agenda.
First
I'm
gonna
give
a
very
quick
intro
on
ebay,
SKU,
Banaras
deployments
after
that
I'm
gonna
walk
through
the
way
we
build
and
manage
kubernetes
clusters
with
a
fleet
management
system
based
on
Q&A
series
in
operators
and
after
building
these
clusters,
I'm
gonna
talk
about
control
and
performance.
A
How
do
we
turn
the
control
plan
to
make
them?
You
know
be
able
to
run
thousands
of
nodes
with
heavy
workloads
and
and
eventually
I'm
gonna
touch
base
on
the
data
and
data
plan
and
user
space,
basically,
how
we
build
containers,
how
we
write
the
prospects
and
how
we
do
performance
tuning
at
different
layers
to
around
those
workloads.
A
First
things:
first
eBay
started
Cuban
area.
Since
2015
we
run
an
internal
distribution
named
tested,
IO
and
was
in
the
past
few
years
we
moved
lots
of
workloads
from
BAM,
arrows
and
VMs
into
containers,
and
as
of
now,
we
have
50
plus
clusters
and
we
run
multiple
V
pcs
for
different
environments.
We
have
those
flat
flat,
netbook
and
overlay
network
based
on
ovn,
and
we
have
multiple
2000
plus
size.
Clusters
was
heavy
workloads
running
on
them
and
in
total,
and
we
are
growing
like
for
daily
basis.
A
But
I
just
took
that
like
two
weeks
ago
that
we
have
160,000
pause
running
on
about
like
24,000
hosts.
Most
of
them
are
bare
metals.
We
only
run
BAM
bottles
for
production,
and
there
are
so
many
different
production
will
close
Ronnie
on
there.
They're
like
web
front-ends
databases,
search
engines
and
even
Hadoop,
so
we
are
also
running
only
edge.
We
pretty
much
runs
on
so
many
edge
locations
so
that
we
can
get
closer
to
the
eBay
end
users.
A
We
are
doing
some
sort
of
edge
proxy
software
load
balancer
on
the
edges
and
we
pretty
much
handle
SSO
termination
from
the
edge,
and
we
do
some
caching-
and
this
is
a
significant
boost
for
our
user
experience
from
Europe
and
APEC,
and
we
are
rapidly
growing
our
footprint
on
the
edge
as
well.
So
our
goal
is
to
unify
a
base
fleet
on
kubernetes
and
we
are
trying
our
best
to
get
there
as
early
as
possible.
A
So
now,
after
getting
some
idea
on
ebay,
skew
Banaras
footprints,
you
might
wonder
how
did
we
get
there
and
the
short
answer
is
we
use
cube
annuities
itself
to
make
it
all
happen.
So,
as
most
of
you
know
that
Cuban
Aries
is
a
declarative,
API,
centric
and
portable
system,
it
can
be
used
to
do
so.
Many
different
things
in
different
ways.
We
use
it
to
build
our
private
cloud
that
runs
Cuban
Aries
and
that's
our
fleet
management
system,
like
any
other
kubernetes
services.
A
It
runs
with
an
iPad
server
it
we
run
it
city
on
Twitter
to
do
some
sort
of
its
city,
cluster
management,
and
once
you
have
the
API
server,
you
pretty
much
will
model
your
data
center.
That's
the
most
fun
part,
so
we
pretty
much
model
every
bits
from
the
data
center
from
racks
to
switches
from
subnets
and
even
routers,
and
we
pretty
much
models
everything
from
the
infrastructure
layer
and
we
have
to
pretty
much
model
the
software
bits.
We
do
our
model
for
operating
system.
A
We
do
os
fees
so
that
it's
easily
modeled
and
the
whole
operating
systems
would
bring
this
track
by
committee,
my
ID
and
then
we
have
a
compute
nodes.
That's
a
provision
hosts
and
we
use
salt
to
deploy
kubernetes.
So
we
model
salt
master.
We
automate
the
part
to
deploy
a
salt
master
to
deploy
kubernetes,
and
once
you
have
all
those
models,
you
have
to
write
a
lot
of
controllers
to
do
things
for
you.
So
first
is
supervision,
so
you
pretty
much
create
compute
nodes
with
kubernetes,
so
we
support
multiple
providers.
A
We
do
OpenStack
at
the
very
early
stages,
because
eBay
used
to
be
a
super
big
OpenStack
shop
and
we
support
DCP
as
well,
but
in
order
to
scale
and
write
high-performance
workloads,
we
want
to
do
bare
models
and
we
don't
want
to
have
all
those
OpenStack
components.
We
used
to
do
ironic,
but
we
see
many
issues
with
expanding,
especially
on
RabbitMQ,
so
we
decided,
let's
use
our
own
primero.
A
Provisioning
was
easy
stuff,
like
kickstart,
we
use
kickstart
and
Foreman
and
we
write
our
own
bare
metal
provisioning
system
and
that's
completely
based
on
cuban
ADC
RTS
and
it's
gonna
call
for
me.
An
API
to
do
provisioning
is
completely
declarative
and
it's
also
acting
as
a
provider
for
our
fleet
management
system
and
and
then
you
need
to
install
a
salt
master.
We
pretty
much
can
install
salt
master
and
pull
a
salt
fee
from
get
based
on
the
commit
ID
and
then
build
a
kubernetes
cluster
yeah.
A
Of
course
you
pretty
much
have
to
do
salt
minions
as
well,
because
you
are
running
South
High
States
to
deploy
kubernetes,
so
we
actually
implemented
so
many
similar
as
cuban
aires
implementations,
like
salt
deployments
and
node
pools,
those
are
just
like
deployments
is
managing
replica
status
and
pots
and
we
actually
support
transactions
schedulers
and
roaring
updates.
We
do
rolling
updates
for
OS
upgrade
for
cuban
areas
releases,
and
you
know
some
specific
workflows
can
even
plug
their
workflows,
specific
logic
into
our
rolling
update
strategy,
and
we
do
that
for
our
search
engine
use
cases
too.
A
So
after
introducing
those
concepts,
let's
take
an
architectural
view.
This
is
a
single
aziz
view
and
we
normally
have
one
fleet
management
system
managing
one
ebay
availability
zone
to
make
itself
content.
So
this
is
our
AZ
control
plan.
Basically,
so,
let's
see
how
it
works.
So
first
you
have
api
server.
You
have
controllers.
Basically,
we
pack
all
those
controllers,
we
call
them
test
master
and
we
have
to
pre
create
a
few
objects,
so
OS
image-
I'm
gonna
mention
that
later,
but
we
use
OS
tree
to
build
our
own
version
of
operating
system.
A
We
leverage
our
pmos
tree
and
project
atomic
and
this
operating
system
can
be
built
from
spec
and
this
operating
system
object
in
our
api
server
contains
the
OS
tree
URL
the
camera
ID
for
this
operating
system
footprint
and
we
create
something
called
compute
flavors
and
it
will
pretty
much
cover
the
the
way
you
want
to
provision
your
bare
metal.
So,
for
example,
you
have
three
disks.
A
You
want
to
pretty
much
do
some
LVM,
some
software
raid
MDM
and
you
can
all
put
them
into
specs
and
we
are
going
to
automatically
generate
the
kickstart
file
based
on
that.
So
now
you
have
predefined
those
models.
Let's
say
we
have
a
new
rax
coming
into
our
data
center.
So
after
you
rock
n
roll,
you
have
all
the
cabling
done
and
you
power
them
all
right.
A
We
call
it
rack
controller,
we
populates
everything
based
on
the
unit
of
rack
and
it
pretty
much
create
the
switches,
the
subnets,
the
racks
and
even
the
routes
for
the
whole
infrastructure
layer,
and
once
you
have
those
assets
sitting
there
ready
to
be
consumed
for
our
homegrown
provisioning
system.
We
are
going
to
create
compute
nodes,
so
compute
nodes
can
be
provisioned
by
different
providers,
just
like
any
other
orchestration
systems,
so
I'm
just
giving
an
example
of
how
do
we
do
that?
It
was
our
homegrown
provisioning
system.
A
So
once
you
create
compute
node
our
scheduler,
what
we
can
asset
and
it's
gonna-
create
the
noting
foreman
and
we
are
gonna
automatically
generate
the
partitioning
templates
for
the
specific
node
based
on
the
partitioning
table
you
want
to
do
and
then
you
know
it's
gonna
interact
with
Foreman
and
we
do
lots
of
controllers.
Like
IP
location,
we
do
provisioning,
we
do
DNS
and
you
know
CMDB
registration,
all
kinds
of
stuff
and
they
are
working
with
orders
and
they
are
subscribed
to
either
notations
or
finalizes
to
complete
the
complex
flow
of
provisioning.
A
Once
you
have
the
computers
provision,
you're
gonna,
have
you
can
deploy
your
communities
clusters,
so
we
use
salt
to
the
preliminaries.
You
can
just
go
to
get.
We
have
our
salt
state
report.
You
manage
kubernetes,
you
take
a
committee,
you
put
that
into
the
salt
appointment
object
which
will
pretty
much
create
the
salt
master
for
you,
and
we
are
going
to
run
salt
minion
on
those
compute
nodes
based
on
their
different
roles.
Some
of
them
are
masters.
A
Some
of
them
are
notes
and
that's
how
we
build
a
cuban
aires
cluster
and
there's
also
those
other
controllers
that
we
do
to
do
scheduling
transactions
and
rolling
updates.
So
I
won't
go
further
to
the
implementation
details,
but
I'm
gonna
have
a
separate
session
to
deep
dive
into
this
in
the
upcoming
Shanghai
cucum.
But
to
summarize
this
is
an
important
piece
of
running
our
high-performance
fleet,
because,
with
our
fleet
management
system,
we
model
a
base
data
center.
We
have
lots
of
controllers,
doing
different
things
and
we
pretty
much
have
every
bit
in
get.
A
What
is
this
mean?
Is
that
so,
if
you
from
from
the
ground
up,
so
you
have
operating
system,
we
built
it
with
OS
3
and
we
have
it
modeled
in
a
PS
server
with
CRTs
and
we
use
salt
to
manage
host
config
as
well
as
during
kubernetes.
So
the
salt
isn't
it
and
you
can
get
a
commit,
ID
imbued
a
cluster
and
and
moving
forward.
A
You
have
like
fleet
so
which
does
provisioning
setting
up
cluster
lifecycle
management
remediation
and
those
are
all
CR
DS
that
you
check
in
to
get
so,
which
makes
us
think
of
like
get
ops,
which
is
like
a
natural
choice
for
us
and
and
then
you're
gonna
have
some
sort
of
PR
driven
operations
and
everything
to
be
decorative.
This
is
the
beauty
of
this
whole
system
and
that's
what
empower
us
to
viewed
consistent,
kubernetes
clusters
across
award
and
having
the
capability
to
really
scale
our
clusters
for
high
performance
workload.
A
Ok,
now
we
have
lots
of
clusters
built
out
there,
but
how
do
they
perform?
So,
let's
take
a
quick
look
at
the
eBay's
cuban
aries
control
plan.
Config
we
run
five
masters
active
active
and
they
are
making
one
each
CD
cluster
and
each
node
is
running
API
server
talking
to
its
local
it
city.
We
of
course
controller
manager,
scheduler
running
on
the
same
note
and
we're
running
a
Dom
manager
and
all
of
those
control
plan.
A
Bits
are
managed
by
salt
because
they
are
static
pots
and
for
the
notes
we're
running
Cuba
in
queue
proxy,
with
a
lot
of
demons
as
managing
like
storage,
networking
and
monitoring,
and
this
all
worked
fine.
But
what,
if
you
are
running
like
thousands
of
nodes
of
a
hundred
thousand
pots,
you
see
issues
right.
So
the
typical
issues
you
are
seeing
is
heavy
lessening
and
watching,
and
it
maps
to
different
layers
and
most
critical
part
is,
is
city
and
API
server.
So
you're
gonna
see
a
lot
of
high
memory
usage
from
the
HDD.
A
You
have
very
frequent
leader
election
changes
and
you
are
gonna,
have
very
slow
scheduling
as
well
as
delayed
controller
sync
and
of
course,
API
server
also
is
gonna
suffer
and
it's
going
to
have
like
frequent
restarts
and
crashes.
So
how
do
we
do
performance?
Turning
for
this?
Most
of
the
work
here
is
done
by
one
of
our
team
members.
Her
name
is
England
and
the
idea
is
to
use
cube
mark
to
get
to
know
the
clusters
limit.
A
First
is
a
server
side,
so
API
server
and
HCD
are
the
most
critical
part.
So
first
things
that
we
found
is
that
you
need
to
evenly
distribute
the
load
to
the
five
master
notes
that
I've
been
talking
about
right.
So
one
issue
we
found
specifically
interesting
is
that
some
of
our
heavy
controllers,
for
example,
provisioning
controllers
from
test
master.
A
It
creates
one
clanked
and
it
has
a
session
stickiness
to
one
api
server
and
this
kind
of
controller,
if
it
always
like
DDoS
the
api
server
you're,
pretty
much
lost
of
one
api
server
and
you
have
a
flat
peon
control
plan,
so
you
can
solve
that
from
two
different
layers.
First,
is
that
you
do
some
sort
of
load
balancing
from
the
API
server,
but
this
wouldn't
solve
our
issues
that
that
I
just
mentioned.
A
You
also
need
to
enhance
your
clients
so
that
they
can
do
some
sort
of
auto
balancing
form
itself
so
that
it
can
do
like
multiple
clients
and
then
it
can
be
smart
about
talking
to
API
servers.
I'm
gonna
mention
more
about
it
later,
but
next,
its
its
city,
so
our
practices
that
always
have
dedicated
SSDs
for
HDD
so
that
you
have
guaranteed
I
ops,
I
think
this
is
rather
obvious,
but
the
other
one
is.
Sometimes
you
want
to
split
the
city.
Some
of
the
folks
yesterday
also
mention
like.
A
If
ingress
can
service
controller
is
underperforming,
you
can
split
that
specific
piece
for
in
service
controller,
but
for
us
we
we
sometimes
we
need
to
split
the
events.
Outsides
of
the
man,
its
city-
and
we
sometimes
do
like
a
separate
disk
for
its
city
snapshots,
because
snapshot
is
also
heavy.
We
wants
to
do
backup.
Our
HDD
backup
happens,
I
think
every
half
an
hour.
So
we
always
are,
you
know
very
concerned
about
losing
data,
and
the
snapshotting
also
has
some
impact,
so
you
always
want
to
be
careful
about
a
city
and
the
next
one.
A
A
So
you
pretty
much
don't
know
like
some
controllers
is
doing
some
super
nasty
stuff
and
some
people
are
doing
like
tens
of
thousands
of
deployments
and
and
you
you
pretty
much-
don't
want
them
to
DDoS
your
API
server
and
that's
pretty
much
the
server
sites
attorney
and
from
the
client
side.
It's
always
about
writing.
Good
controllers.
First
example
is
about
the
list,
options
and
resource
versions.
So
so
many
of
you
might
know
that
if
you
put
the
resource
version
equals
to
zero,
it
gets
the
API
servers
cache.
A
But
if
you
put
that
into
Neal
it
goes
into
the
HDD,
so
we
always
want
to
pretty
much
avoid
directly
hitting
it
city
for
some.
Some
of
the
cases
really
need
to
do
that,
like,
for
example,
controller
restarts.
You
want
to
reconcile
and
do
stuff
like
that,
you
don't
want
to
be
behind,
but
when
you
are
actively
running
always
avoid
that
and
use
as
many
informers
as
possible.
This
is
also
like
a
pretty
common.
You
know
performance
concerned
about
writing
controllers.
A
You
are
going
against
the
API
server
and,
of
course,
you
have
namespaces
and
the
type
of
API
calls
you
are
getting
like
get
listening,
so
you
get
an
idea
of
how
your
controller
is
doing
and
how
they
are
performing.
So
that's
very
important
for
you
to
measure
the
performance
and
enhance
your
performance,
so
the
last
part,
I
am
just
quickly
goes
through
because
that's
already
fixed
in
the
upstream.
A
So
sometimes
you
have
some
sort
of
leftover
parts,
so
they
are
doing
they're
in
terminating
some
of
the
hardware
failed
and
in
the
earlier
versions
of
Cuban
Ares
scheduler
is
using
mutex
instead
of
RW
mutex.
So
it's
one
part
can
slow
down
your
whole
cluster,
but
that's
fixed
later,
as
I
mentioned.
So
basically
we
you
always
need
to
manage
your
lifecycle
of
the
cluster,
so
our
fleet
management
also
does
that
it
does
remediation
interacting
with
our
brick
fixed
system.
A
We
can
take
down
the
bad
nose,
send
them
into
break
fix,
cling
up
from
the
kubernetes
clusters
and
then
remember.
We
have
this
node
poo
and
deployment
concept
and
we
are
going
to
replace
with
a
spare
asset
and
match
it
as
it
is
and
add
it
back
into
the
cluster
to
complete
the
story
of
remediation.
So
now
we
are
moving
to
the
data
plan
because
we
are
mostly
done
with
a
control
plan,
so
we're
gonna
talk
about
how
we
run
those
high-performance
will
close
some
of
our
best
practices.
A
So
first
thing
is
you're
gonna
build
containers
for
them,
so
some
obvious
principle
is
that
you
always
try
to
do
the
containers
as
a
native
container,
but
sometimes
somewhat
close,
doesn't
like
it.
So
we
have
a
lot
of
heavy
workloads.
They
still
want
to
run
some
init
system.
This
is
like
a
classic
PID
discussion
that
has
been
going
on
for
a
while.
So
can
you
run
system
D?
The
answer
is
yes,
but
it's
a
little
bit
too
much
in
order
to
run
system
D
inside
a
container
for
your
pit
one.
A
You
need
to
do
some
cleanup
from
your
container.
You
need
to
amount,
I,
think
a
C
group
FS
as
read-only,
and
then
you
need
to
make
some
temp
FS.
We
do
that
with
empty
door
and
okay
now
you
have
it
running,
but
is
it
that's
it?
No!
It's
actually.
We
see
lots
of
other
issues
like
somebody
actually
captures
some
darker
image
from
a
Hadoop
server
and
they
think
they
can
just
write
like.
But
later
we
found
out,
because
this
is
a
heavy,
high-priority
project.
A
We
give
them
all
the
permissions
we
can
get
and
then
you
know
they
pretty
much
limit
the
system.
Configs
and
I'm
gonna
talk
about
that
later.
But
for
this
specific
case,
CentOS
7
has
a
system
D
that
has
hard
core
at
65
K
you
limit,
so
any
like
Hadoop
jobs
as
containers
spin
off.
They
cannot
reach
the
100
K
limit
they
want.
So
we
have
to
pretty
much
patch
it
and
compile
a
new
system
D
and
put
on
to
that
all
they
have
to
upgrade
that
container
image.
A
The
actually
alternative
is
to
use
some
smaller
in
a
system
like
supervisor
D
or
you
are
your
own
init
system,
but
with
the
new
features
coming
up,
there
is
a
share
pit
namespace
and
which
allows
you
to
manage
those
processes,
and
you
have
your
post
container
acting
as
a
pit
one.
So
maybe
you
don't
even
need
init
system
later
the
next
one
is.
You
have
to
be
super
careful
about
the
capabilities
you
give.
So
it's
a
security
thing,
but
it's
also
performance
thing
because
the
same
container
I
just
mentioned
it.
A
Imagine
that
it
took
from
a
bare
marrow
host.
It
turn
D
and
it
has
all
those
capabilities
that
you
give
and
it
completely
mess
up
your
hosts
and
it's
so
hard
to
troubleshoot
because
from
the
host
you
don't
actually
see
anything
and
then
the
last
one
I
want
to
mention
is
the
containers
resource
limit
awareness.
So
for
some
of
the
legacy
systems
they
want
to
do
a
lot
of
things
from
the
containers.
They
do
opt
on
top
check
the
memory
and
see
things
around
to
run
some
jobs
some
scripts.
A
But
most
of
you
know
that
from
containers
you
see
most
of
the
things
from
the
host.
Well,
of
course,
you
can
pass
a
lot
of
information
to
the
containers
by
download
AP
is
by
EMV
s,
but
there's
some
solution
that
Kinetico
has
they
use
it
to
manage
our
X
DS.
It's
called
Erik
CFS.
It
actually
do
a
custom
proc
file
system
that
overrides
a
few
things
like
app
time,
memory
and
CPU,
but
only
does
CPU
set
well.
It's
quite
obvious.
A
So
you
can
manage
the
over
commitment
ratio
by
yourself
by
declare
your
birth
small
pots,
like
you
put
on
like
a
request
and
limit
based
on
a
ratio,
and
then
you
can
tolerate
the
hosts
over
commitment
by
yourself.
You
make
the
call
and
then
there's
a
lot.
You
run
a
lot
of
props,
so
written
is
prop
and
liveness
prop
is
a
super
powerful
tool.
For
example,
rhiness
prop
the
typical
use
cases
to
enable/disable
web
traffic
and
liveness
prop.
A
You
can
use
on
trick
to
do
some
self
triggered
restarts
and
I
just
want
to
quickly
go
through
them,
but
I
want
I
do
want
to
mention
some
performance
bits
of
the
pass
back.
One
thing
is
for
stateless
applications,
many
of
our
city's
applications.
They
want
some
sort
of
a
thermo
drives
and
it
seems
that's.
The
only
option
right
now
is
to
use
MTR
backed
by
hard
drives.
Well,
it's
not
a
perfect
one,
because
you're
gonna
see
lots
of
issues
like
applications
just
dump
their
data
into
MDR.
A
They
don't
really
care,
and
then
you
ended
up
adding
size
limits
into
the
empty
doors,
but
so
as
to
evict
them
once
they
are
reaching
their
limits
right,
but
the
implementation
is
cubelet
is
gonna.
Do
some
sort
of
D
you
against
that
folder
like
every
20
seconds,
and
when
you
have
lots
of
them,
you
can
I
have
some
performance
overhead.
So
for
that
we
are
actively
looking
at
the
upstream
CSI
inline
volume
support,
not
just
for
the
kind
of
a
formal
support,
but
also
we
want
to
unify
our
flavors
to
run.
A
You
know
the
local
storage.
Specifically
we.
If
you
look
at
and
compare
stateless
applications
versus
versus
stateful
applications,
we
actually
have
stateful
says
running
with
high
performance,
local
volumes
if
both
of
them
can
be
backed
by
the
same.
You
know,
I/o
vm
BG,
it's
much
easier
for
us
to
manage.
Okay.
Now
we
have
built
up
ads.
We
need
to
run
them
at
the
hosts,
runtime,
so
host
runtime
performance,
most
of
the
bits
are
from
the
kernel.
We
can
spend
another
day
talking
about
the
kernel,
but
basically
a
few
basics.
A
We
want
to
make
sure
that
our
we
share.
So,
first
is
that
you
want
to
unify
the
platform
and
manage
as
less
kernels
as
possible.
Why
is
that?
You
have
lots
of
drivers
to
certify
you
have
you
know
if
you're
managing
so
many
different,
kernels
and
I,
for
example,
eBay
is
managing
ODM
custom
hardware's?
We
have
lots
of,
like
GPU
use
case
many
adding
kernels,
adding
a
lot
of
maintenance
cost.
And
now,
let's
say
you
decide
on
running
one
kernel,
you're
trying
to
run
the
latest,
so
the
upcoming
4:19
kernel
seems
to
be
very
interesting.
A
It
brought
in
a
patch
for
a
network
packet
patching
and
that
should
improve
network
through
by
like
over
30%,
so
we
are
also
looking
at
it.
So
the
next
one
for
host
on
time.
Turning
is
about
CPU
and
power.
So
we
always
want
to
push
your
server
to
the
limit
when
it
runs
high
performance
will
close.
So
we
want
to
make
sure
that
it
running
on
the
performance
mode
instead
of
power
saving
mode.
A
So
you
pretty
much
flipped
like
P
State,
and
you
always
want
to
make
sure
that
there's
many
odd,
not
any
other
services
of
the
system
like
canonical
has
like
own.
The
amount-
that's
reverting
your
changes
and
we
found
it
interesting
that
there's
a
maxi
state.
We
always
wanted
to
set
it
as
zero.
But
it's
not
absolutely
true
that
zero
gets
you
more
performance,
because
you
always
think
about
some
use
cases,
that's
super
heavy
and
then
they
need
to
use
a
lot
of
turbo
from
the
CPU.
A
So
basically,
what
we
found
is
that
setting
it
to
zero
is
actually
way
less
comfy
performing
as
sitting
into
the
default
nine
for
maxi
state
when
it
comes
to
the
turbo
use
case,
and
then
the
memory
there's
a
like
very
common
ones,
about
THP,
like
many
of
the
database
folks,
will
warn
you
that
HP
is
gonna
buggy,
but
what
we
found
is
we
actually
get
a
lot
of
benefits
from
THP,
especially
for
our
search
engine
use
cases.
So
it's
pretty
much
like
walk
low,
specific
you
always
have
to
identify
by
ourself
and
I.
A
Do
want
to
mention
about
the
swap,
because
you
know
swap
is
one
feature
that
we
want
to
avoid
for
Cuban
Ares.
One
lesson
learn
is
that
we
have
a
small,
noisy
daemon
set
that
actually
had
runs
on
a
box
that
has
swept,
and
it
has
some
sort
of
memory
leak
and
once
it
reached
to
over
60%,
it
actually
wakes
up
the
swap
and
then
case
web
de
slows
down
the
whole
host.
So
what
we
ended
up
doing
is
so
you're
gonna.
We
ended
up.
A
You
know
overriding
memories,
wepner's
and
d
for
it
to
zero
for
everything,
unless
you
put
a
specific
connotation
so
try
to
avoid,
swap
because
it
kind
of
mess
up
with
c
group
and,
of
course,
we're
gonna
talk
about
like
io,
scheduler,
cfq
or
dead
lying
and
I
won't
go
further
into
that.
But
one
thing
we
found
is
that
we
default
to
cfq
because
we
found
some
issues
with
deadlines
when
we
are
doing
it:
city
backups,
but
it's
basically
your
choice.
A
I
do
want
to
talk
about
our
storage
classes,
because
we
have
many
layers
for
storage
class.
We
have
hot
tears
and
standard
tears.
We
pretty
much
runs
local
volumes
for
most
of
the
high-performing
ones
and
we
run
our
own
version
now,
but
we
are
trying
to
switch
to
the
upstream
dynamically
local
volume
solutions
from
CSI
and
we
run
stand
around
cinders
so
that
we
can
get
the
set
volume
money
to
bare
metal
without
going
through
the
hypervisors
and
viens
stuff.
A
Nova
from
you
know,
OpenStack,
and
there
are
lots
of
system
configs
I
just
want
to
highlight
two
one
is
unique:
you
want
to
limit
your
stuff
like
you,
there
are
lots
of
heavy
workloads.
You
want
to
limit
the
kind
of
resource
they
are
taking
so
that
the
hosts
sit
down,
don't
get
bumped
up,
and
then
you
want
to
increase
a
few
things.
For
example,
max
map
count
for
elastic
search,
so
you
always
have
to
find
a
balance
between
some
workloads.
A
That's
overusing
some
others
are
or
not,
but
some
of
the
things
we
are
trying
to
do
is
first,
is
the
CPU
set
and
Numa.
So
this
is
a
feature
we
want
to
do
and
then
block
IO
C
group.
Obviously
we
want
to
do
some
better
resource
management
and
isolations
and
OS
query
as
well
as
we
are
doing
some
software-defined
storage.
A
Ok
last
part
is
networking
performance
I'm,
not
too
much
of
a
networking
guy.
So
I
don't
have
too
much
details
here,
but
luckily
we
don't
have
too
much
time
left
either.
So
we
do
RPS
and
RFS.
We
set
that
into
the
vs
devices,
and
this
is
super
helpful,
which
gets
us
to
the
similar
as
BAM
arrow
through
boots.
We
do
IP
VLAN
for
high
performance
work,
clothes
for
standard
ones.
We
use
obvious
and
obvious,
but
some
search
engine
will
close.
A
A
If
you
can
fix
how
we
do
some
like
traffic
controls,
fq
cuddles
and
tcp
slow-start,
so
I
won't
go
further
into
that,
because
we
don't
have
enough
time
left,
but
we
do
want
to
do
IP,
VLAN
and
eb
PF,
and
we
are
actively
thinking
about
the
solutions
to
network
us.
The
last
part
is
sometimes
you
want
to
isolate
your
workloads.
You
see
so
many
different
configs.
There
is
mentioned,
you
profile
them
and
you
want
to
isolate
them.
So
how
do
you
do
that?
So
stopping
one
workload
to
step
into
each
other?
A
You
pretty
much
can
do
like
no
schedule.
You
do
is
tense
and
toleration,
but
there
is
no
schedule
and
no
execution.
So
what
we
found
is
that
no
execution
is
actually
too
aggressive.
It
actually
evict
some
of
the
no
selecting
parts
which
we
actively
use
for
like
volume
recycle,
so
you
don't
want
you
go
with
no
execute.
We
do
know
it
no
schedule
and
you
always
want
to
monitor
your
performance.
There's
a
lots
of
tools.
A
I
listed
here
and
there
you
need
to
do
a
loss
of
benchmarking
and
burning
tests
and,
more
importantly,
if
you
can
automate
that
part,
we
actually
are
actively
working
on
integrating
all
those
performance
bits
for
specific
use
cases
into
our
OS
image.
Ci
CD,
as
well
as
our
certification
flow
for
new
hardware.
So
that's
pretty
much
it
lots
of
contents.
I
might
have
run
a
little
bit
to
rush
for
some
of
the
stuff.
If
you
miss
anything,
you
can
find
the
slides
online
I'm,
not
sure
how
much
time
we
have
for
Q&A.