►
From YouTube: Performance and Scalability Tuning Kubernetes for OpenShift and Docker by Jeremy Eder, Red Hat
Description
Performance and Scalability Tuning Kubernetes for OpenShift and Docker - Jeremy Eder, Red Hat
"Learn tips and tricks on how to best configure and tune your container infrastructure for maximum performance and scale. The Performance Engineering Group at Red Hat is responsible for performance of the complete container portfolio, including Docker, RHEL Atomic, Kubernetes and OpenShift. We will share: - Latest Performance Features in OpenShift, Docker and RHEL Atomic, tips and tricks on how to best configure and tune your system for maximum performance and scale - Latest performance and scale test results, using RHEL Atomic, OpenvSwitch, Cockpit multi-server container management - DevOps, Agile approach to Performance Analysis of OpenShift, Kubernetes, Docker and RHEL Atomic - Test harness code and example scripts
Audience
The audience is anyone interested in deploying containers to run performance sensitive workloads, as well as architecting highly scalable distributed systems for hosting those workloads. This includes workloads that require NUMA awareness, direct hardware access and kernel-bypass I/O."
About Jeremy Eder
Container Performance Lead at Red Hat. Specializes in measurement and analysis of performance metrics, and using that analysis to guide performance-tuning of real-world infrastructure. Leads a team of engineers focused on performance, scalability and architecture of container-based infrastructures in the Atomic and OpenShift family of Red Hat products. Contributor to Kubernetes, Docker and the Linux kernel in a variety of areas where things need to go fast. Red Hat Summit 2011-2015 http://youtu.be/Vr0c8jEuxe0 http://youtu.be/qQosbcBvAj0?t=25m24s https://www.youtube.com/watch?v=ckarvGJE8Qc DevConf.cz: https://www.youtube.com/watch?v=XNrIYOk2Zog Red Hat Webinar: https://www.youtube.com/watch?v=KAbduQgMbT0 Whitepaper: 2016 - OpenShift v3 Scaling, Performance and Capacity Planning https://access.redhat.com/articles/2191731
Abstract"
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
A
A
We
do
upstream
development.
We
do
test
execution
and
design
of
test
plans,
and
we
work
very
closely
with
the
quality
engineering
team
at
Red
Hat
to
ensure
that
they
run
these
things
formally
in
regression
test
plans.
So
we
feel
like
we've,
got
decent
coverage
and
we
can
always
do
better
and
what
I'm
one
of
the
things
I'd
like
to
hear
from
you
guys
today
is
what
this
scale
mean
to
you
and
what
this
performance
means
you.
A
So
if
any
of
these
slides
are
you
know,
if
you
have
any
comments
about
these
slides,
please
just
just
shout:
oh
and
there's
a
link
I'll
put
these
slides
on
SlideShare
afterwards
or
on
Twitter,
but
there's
a
hyperlink
here
to
our
Trello
board,
which
is
public
all
of
RedHat
OpenShift
Engineering's
Trello
boards,
so
our
agile
workflow
backlogs
are
on
Trello
and
publicly
available.
So
you
can.
You
can
see
exactly
what
we're
doing.
A
A
A
What
kubernetes
can
do
right
now,
so
this
slide
goes
in
every
deck.
That
I
do
it's
a
food
group
analogy
slide
I
want
to
give
you
guys
a
sense
of
how
I
characterize
a
work
load
when
it
comes
to
me,
for
example,
if
a
new
application
is
under
development
at
Red
Hat
and
they
say
formance
engineering
guys,
you
need
to
take
a
look
at
this
and
tell
us
if
our
architecture
is
sound
or
not.
A
These
are
some
of
the
things
that
we
look
at.
These
are
four
food
groups
of
performance.
Every
application
can
be
decomposed
into
these
resources
and
then
you
further
optimize
into
each
resource.
I've
heard
some
really
interesting
stuff
in
the
CPU
space.
About
proactively
monitoring
in
like
cycles
per
instruction
on
on
Google's
cloud,
etc,
invisible
some
pretty
sexy
stuff,
but
these
are
things
we
do
internally
and
then
this
is
kind
of
how
these
things
need
to
be
in
a
healthy
balance.
If
any
one
of
them
is
contended,
your
application
is
not
going
to
perform
correctly.
A
So
here,
with
my
enterprise
and
Red
Hat
hat
on
there's
some
additional
concerns
that
you
might
have
outside
of
the
performance
realm
that
are
equal
or
even
more
important
than
the
actual
performance,
and
these
are
things
that
we
take
extremely
seriously
at
Red
Hat,
and
this
is
why
we
call
ourselves
enterprise
the
things
like
support
and
security.
We
take
extremely
seriously
part
of
our
major
contributions
to
kubernetes,
have
been
in
this
area
and
and
of
course,
stability,
reliability,
bug,
fixing
being
able
to
pick
up
the
phone
and
call
someone
etc.
A
So
take
the
Red
Hat
corporate
hat
off
for
a
minute
decomposing,
a
particular
workload.
Let's
say
we
have
a
build
form.
Red
Hat
has
decided
to
compose
all
of
our
containers
that
we
shipped
to
customers
using
open
shift,
so
we
use
openshift
as
a
build
farm
internally
to
compose
like
if
you
do
dr.,
pol,
rel,
seven
or
whatever
you
get
an
image
that
was
run
through
this
open
chef
based
build
farm,
and
this
is
publicly
available
to
be
available.
A
Of
course,
like
everything
we
do
so
we
did
some
work
on
optimizing
that
environment
we
found
issues
with
docker
not
being
parallel
safe
for
builds,
for
example,
running
into
into
IO
contention,
and
some
of
it
was
learning
how
much
IO
does
it
actually
take
to
do
a
build?
Is
there
more
optimal
ways?
Can
we
do
some
caching
to
to
shield
the
the
storage
subsystem
from
having
to
do
repetitive,
IO,
and
we
also
had
to
implement
some
queues
in
front
of
the
actual
docker
processes
so
that
we
wouldn't
flood
the
docker
API
I?
A
A
A
So
these
are
the
types
of
things
that
I
keep
in
the
back
of
my
head
and
at
least
from
my
point
of
view,
the
only
thing
that
matters
is
making
your
apps
run.
Well,
so
that's
kind
of
the
performance
side
of
it
does
that
does
that
make
sense?
Is
that
methodology
ring
true
at
all?
Who
can
give
me
one
example
of
an
application?
That's
not
a
web-based
application
that
they
want
to
run
and
coupon
cube.
A
A
The
value
of
devops
cannot
be
realized
without
portability
and
the
standards
standardization
of
at
least
the
at
least
the
container
image
format
we're
not
able
to
build
a
successful
ecosystem
of
ISVs
software
ifvs
around
kubernetes
in
the
container
platform,
without
some
sort
of
standardization
they're
not
going
to
run
and
do
we
know
these
Java
like
if
you're
selling
a
java
app
you
deploy
it
as
like.
You
sell
it
as
a
zip
file,
you
don't
pack
it
as
it
as
it
Deb
or
an
RPM.
A
So
the
idea
being
there
was
no
portability
at
the
time
when
we
were
doing
these
package
managers
for
the
last
20
years.
We
hope
that
the
container
image
format
can
solve
that,
and,
incidentally,
Red
Hat
plays
a
somewhat
of
a
role
in
both
of
these.
Both
of
these
consortiums,
or
at
least
I,
would
call
them
foundations.
So
what
do
we
do?
That's
different
than
what
the
or
in,
in
addition
to
what
the
kubernetes
and
community
do
for
scale:
testing
of
open
ship
of
kubernetes
back
to
the
word
Enterprise,
so
you
have
kubernetes.
A
A
Some
sort
of
storage
on
it
as
well,
so
these
are
things
that
Red
Hat
sells
additional
products
for,
and
these
are
types
of
things
that
we
are
going
to
add
on
top
of
the
normal
kubernetes
scale
tests,
so
we're
gonna,
do
you
know
build
performance.
I
mentioned
the
render
for
the
build
form
earlier.
We're
gonna
do.
Registry
scalability
logging
and
metrics
OpenShift
has
like
an
e
FK
stack
to
be,
as
fluent
d,
which
I
noticed
was
was
just
accepted
into
CNC
F.
A
So
onto
the
the
actual
lab
tests
and
I
guess
may
or
something
we
we
found
out
about
this
cluster
and
we
Alec
and
they
were
allocated
300
nodes
on
it,
because
we
had
a
goal
of
a
thousand
node
kubernetes
cluster
on
this
level.
You
might
have
seen
this
blog,
but
it's
on
the
CNCs
blog.
But
since
we
were
only
allocated
300
of
those
nodes,
we
had
to
do
some
sort
of
virtualization,
which
was
not
in
our
original
plan.
A
The
point
of
the
exercise
was
to
push
it
till
it
breaks,
which
is
every
performance.
Engine
favorite
thing
to
do
is
to
find
the
failure
scenarios,
and
we
also
wanted
to
come
up
with
a
reference
design
which
we've
done
around
a
scalable
kubernetes
environment,
environment.
That
is
also
highly
available.
A
A
A
This
is
donated
to
it
by
Intel
by
the
way
to
the
CNC
F
anyway,
each
node
has
an
nvme
device
in
it,
which
is
extremely
hard
to
come
by,
and
if
we
try
to
do
that
on
a
public
cloud
provider,
they
don't
offer
those
things
yet
and
even
if
they
did,
it
would
be
insanely
expensive.
It's
one
cool
thing
about
this
lab.
Is
you
get
really
high
end
stuff
at
zero
cost?
A
I
guess
if
I
had
a
want
for
this
thing,
they'd
have
more
ram
and
they'd
have
40
gig
Nick's,
but
other
than
that.
It's,
like
you,
know,
gift
horse
in
the
mouth
type
of
thing.
If
you
want
to,
if
you
want
to
schedule
yourself
on
this
lab,
there's
a
FIFO
queue,
that's
managed
in
github,
/c
n
CF,
slash
cluster.
If
you
want
to
apply
for
it,
and,
incidentally,
I
look
through
some
of
our
next
like
what's
next
in
a
minute,
but
we're
gonna
get
on
this
lab
again
at
some
point.
A
A
We
did
the
whole
thing
highly
available,
so
there's
there's
no
single
point
of
failure.
One
thing:
that's
actually
missing
from
this
dire
is
a
load
balancer
that
goes
in
front
of
the
master,
the
API
servers,
so
we've
got
three
API
servers
in
this
scenario:
one
load
balancer
VM
and
the
queue.
Let's
talk
to
the
load,
balancer
a
3-node
fcd
cluster
routers
registries
and
a
thousand
VMs
the
VM
inch,
if
you're
interested
the
VM
configs
down
here
below.
But
it's
not
really
that
important,
so
yeah.
A
So
that's
what
we
built
and
then
ended
up
being
fed
into
parts
of
it
we're
fed
into
a
recent
reference
architecture
document
that
RIT
has
put
out
how
to
run
OpenShift
on
top
of
OpenStack.
We've
also
learned
got
there's
four
of
them:
how
to
run
OpenShift
on
OpenStack,
on
VMware,
on
GCE
and
on
AWS.
They
just
came
out
this
week,
so
you'll
be
able
to
see
the
very
latest
and
greatest
from
exercises
like
this
and
our
counterpart
in
my
in
in
my
team
and
the
system,
design
and
engineering
group
that
builds
the
reference
architectures.
A
Ok,
so
we've
got
this
environment
built
out,
which
was
its
own
set
of
hilarious
jokes,
but
after
it
was
after
it
was
up
and
running
I
guys
that
America's
talk
yesterday
and
then
he
said
he
had
this
picture
of
a
nuke
and
it
it
really
is
that
we,
we
kind
of
call
ourselves
crawling
over
broken
glass
as
a
service.
This
is
all,
and
the
idea
is
that
is
what
we
want
to
do.
That's
what
we
get
paid
for.
A
The
idea
is
that,
after
we're
done
crawling
and-
and
you
know,
afterward
abused,
the
products
are
so
much
better.
We
have
you
know.
We
have
products
that
have
run
the
full
gamut
of
some
pretty
hardcore
tests.
I
mean,
like
I
said
we
can
always
do
better
and
we
will,
but
we
feel
like
we
have
a
decent
mix
and
here's
what
we
here's,
what
we
have
if
you
go
to
github
openshift
SBT
you'll
find
our
repository
of
tests
and
I'm
hopeful,
and
it
done
this
has
been
negotiated.
A
Ultimately,
we
would
want
to
gate
merges
on
scale
tests
if
we
had
infinite
budget.
So
I
don't
know
if
that
will
ever
happen,
but
I
think
we
can
do
something
with
like
mocking
large
environments
and
maybe
learn
a
lot
more
I
know
that's
what
there's
a
tool
called
cube
mark,
which
does
that
right
now,
anyway,
so
I
think
the
main
most
there's
a
there's.
A
this
repo
here
is
currently
has
a
couple
of
network
tests
in
it
from
from
the
from
the
Google
guys.
A
So
we've
also
got
things
like
reliability
and
longevity
testing,
where
we'll
run
for
a
couple
of
weeks
and
make
sure
we
don't
have
any
memory,
leaks,
etc.
These
are
things
to
kind
of
harden
the
environment
and
we
found
we
found
some
leaks
and
back
so
I
think
that's
all
pretty
cool
I
think
you
know
so.
Cholesterol
order
is
part.
One
and
I'll
go
through
it
in
a
minute
and
then
the
other
piece
is
after
the
cluster
is
up
and
running.
A
If
you're
scaling
to
a
thousand,
you
can't
have
each
VM
go
out
to
the
registry
and
pull
the
15
containers.
It
takes
to
actually
make
things
work
and
you
also
want
to
not.
It
would
also
have
want
to
have
updates
baked
in
etc.
So
we've
we
built
out
a
bunch
of
ansible
to
do
this
and
it
will
spit
out
a
cue
cow
which
we
used
on
OpenStack
and
continues
on
OpenStack
or
Rev,
and
it
will
also
spit
out
potentially
an
ami,
so
it
will
go
out
to
Amazon,
create
an
instance.
A
Do
the
whole
thing
and
then
the
image
will
be
available
for
us
to
use
I
think
the
main
steps
I
mention
earlier.
We
do
some
filesystem
juggling
like
partition,
resizing,
etc,
and
then
we
do,
you
know,
make
sure
we
have
all
the
updates
and
containers
pre
polled,
and
then
we
actually
load
the
rpms
as
well
onto
the
system.
So
everything
as
much
as
we
can
is
preloaded
and
those
are
big
wins,
because
otherwise
we
couldn't
do
the
full
scale,
it
would
take
forever.
A
A
So
the
cluster
loader
itself
takes
a
llamó
config
like
this,
and
these
templates
are
somewhat
controversial
right
now,
but
essentially
it
can
take
a
number
of
projects
or
namespaces
which
our
tenants
and
OpenShift
land
and
it
will
create,
however,
many
of
whatever
type
of
templates
that
you
have.
So
if
you
were
to
download
this
guy
today
and
you
have
templatized
and
moved
your
app
into
cube
already,
you
take
your
you.
Take
your
template.
You
feed
it
into
this,
you
Hamill
file
and
it
will
build
out
your
environment
exactly
as
you
with
your
application.
A
What
we've
done
is
take
the
exam
of
the
examples
from
kubernetes
and
and
make
them
in
you
know
and
put
them
into
this
repo.
So
I
mentioned
they're
controversial,
because
cube-like
doesn't
have
a
concept
of
templates
right
now.
Openshift
does
and
there's
other
technology
like
there's,
probably
a
dozen
other,
like
templating
technologies
out
there,
but
anyway,
so
we're
reporting
as
many
as
we
can.
We
like
to
build
up
a
pretty
big
corpus
of
other
applications
that
we
could
you
know
throw
at
these
environments.
These
configs
can
also
include
like
a
persistent
volume.
A
They
can
get
as
complicated
as
you
want.
It
could
be
as
simple
as
like
a
pause
pod,
if
you're
familiar
with
that
or
it
could
be
as
complex
as
a
multi-tiered
application.
That
includes
persistent
storage
and
in
fact
those
are
the
ones
that
we've
got
here
so
the
Redis
example
the
guestbook
example
with
persistent
back
in
storage
in
a
database,
and
it
will
also
set
up
all
of
the
routing
and
secrets
and
build
configs,
etc.
A
A
Okay,
so
in
this
scenario,
what
we
found
with
OpenShift
version
two
was
that
this
is
not
kubernetes
version
of
open
shipped
to
the
previous
version,
which
had
our
own
implementation
of
containers.
We
found
that
flooding
the
API
server
was
not
a
good
idea,
so
we've
got
this.
These
control
loops
and
back
offs
and
delays
that
are
built
into
it.
In
this
scenario,
I'm
not
doing
a
tremendous
amount
of
load
on
the
system,
so
I
don't
really
need
those
those
in
there,
but
that's
what
they're.
A
A
There's
not
really
much
to
it,
make
it
verbose,
use
the
cluster
loader
portion
of
the
ete
test,
so
the
IDI
test
says
like
400
and
plus
tests
in
it.
We
only
want
to
run
the
cluster
loader
portion
of
it
and
then
they
use
Viper
for
it.
This
is
written
by
a
co-worker
of
mine
named
Sebastian
uke,
and
so
it
uses
Viper,
and
then
you
pointed
at
a
config
file.
The
idea
is
this:
config
file
could
be,
could
represent
your
environment,
so.
A
A
Why
does
the
hardware
and
software
is
capable
of
doing
so,
the
single
applet,
so
the
single
test
harness
can
do
you
know
as
much
scale
as
as
you
might
want
so
pretty
simple
demo,
but
we're
hopeful
that
we
can
find
a
way
to
get
that
upstream
into
kubernetes
said
you
guys
can
all
right
now
it's
on
github
and
it's
in
Python,
but
we're
we're
gonna,
try
and
get
into
upstream
cube
proper
contribute
back
to
the
community.
So
any
questions
about
that
utility
does
anyone
think
they
could
make
use
of
it?
A
Yeah,
that's
what
it
is.
So
what
did
we
learn?
We
fired
this.
We
fired
the
the
cluster
loader
at
at
this
big
at
this
big
open
shift
environment.
What
was
some
of
the
takeaways
well,
this
is
our
configure.
We
ended
up
with
end
up
with
a
thousand
nodes,
13,000
tenants
or
projects.
Whatever
52,000
pods
and
all
these
other
objects
were
loaded
into
at
CD
in
this
hidden.
A
Incidentally,
we
plan
to
go
a
lot
higher,
but
we
did
hit
a
bug
in
the
API
server
that
had
very
recently
been
filed
upstream
at
the
time,
so
the
API
server
like
was
crashing,
and
at
that
time
our
time
was
up
on
the
CNC
F
lab.
So
we
didn't
get
a
chance
to
continue,
but
we'll
do
that.
You
know
a
bigger
scale.
The
next
time
we
get
on
that
gear.
A
So
a
couple
things
we
found
a
lot
of
our
teams.
Output
is
actually
capacity
planning
information
and
that
gets
into
our
product
documentation.
So
probably
the
initial
conversation
that
we
have
with
customers
is
how
much
gear
do
I
need
to
buy
or
how
much
cloud
resources
so
I
need
to
provision,
and
so
we
create
these
formulas
based
on
our
actual
tests
that
they
can
use
to
extrapolate
and
decide
how
much
gear
they
need.
A
In
the
scenario
on
the
Left,
we
found
that
loading
all
of
that
stuff
that
I
just
showed
you
into
ed
CD
takes
a
fair
amount
of
space
on
disk
more
than
we
had
originally
documented,
and
so
we've
gone
ahead
and
updated
that
documentation
and
used
this
as
justification
to
say,
look
you
need
to
give
40
gigs
to
at
CD
if
you're
gonna
do
anything
at
scale.
Fortunately,
F
CD
is
really
easy
to
move
like
it's
got.
You
know
just
that
one
database
files
you
just
like
my
sequel.
A
A
Incidentally,
this
is
at
CD
version
2
in
case
you
were
curious
and
on
the
right
hand,
side
we've
got
some
other
gharana
graphs.
I.
Think
the
only
interesting
part
about
this
is
that
you
notice
this
drop-off
in
CPU
and
that's
actually
when
the
API
server
fell
over
and
luckily
I
Chi
said
that
bug
had
been
fixed
so
and
this
is
an
alpha
version
of
openshift
3
3.
So
this
bug
never
shipped
in
the
product
question
yeah.
A
Open
ships
installer
is
based
on
is
written
in
ansible,
which
is
incredibly
flexible,
totally
paralyzed
and
not
really
sure
how
we
would
survive
without
it.
So
we
also
use
ansible
for
every
other
thing
in
the
world.
One
thing
we
noticed
is
that
when
you're
gathering
so
ansible
will
go
out
to
each
node
gather
what
see
what
it
calls
facts
about
that
node
and
then
suck
them
back
over
the
network
and
hash
them
in
memory.
A
What
happens
is
when
it
does
that
hashing
across
thousands
of
nodes,
you
get
a
tremendous
amount
of
CPU
utilization,
which
you
need
to
know
about,
because
if
you
run
out
of
CPU,
sometimes
those
those
installed
portions
could
fail,
and
so
we
worked
with
the
ansible
guys
to
optimize
the
crap
out
of
that
part,
and
we
can
actually
shove
those
results
into
a
database
at
this
point,
instead
of
having
them
in
a
flat
file,
it's
a
little
bit
more
optimal,
just
sequel
Lite.
It
actually
helps
us
perform
this
problem
significantly
and
again.
A
A
Yeah
right
so
we
ran
into
when
we
did
the
initial
OpenStack
deployment
we
used
Swift
and
which
we
regretted,
but
in
the
next
iteration
we're
gonna
use
we're
already
using
SEF,
moved
from
volumes
with
snapshots.
So
we
expect
that
prob.
We
had
a
problem
with
Nova
that
a
lot
like
it
just
took
forever
to
blow
out
these
these
1,000
VMs
and
so
we're
trying
to
optimize
that
and
so
for
our
next
strobe.
Our
next
runs
we'll
have
those
optimized.
A
A
That's
specifically
about
the
density
test,
that's
in
the
IDI,
so
that's
what
those
definitions
apply
to
OpenShift
as
well.
We
take
Qbert,
we
take
kubernetes,
as
is
we
had
a
couple
things
that
weren't
yet
upstream,
but
our
going
upstream
things
like
for
a
while
the
security
policies
weren't
upstream,
we
were
using
them.
But
my
point
is
that
these
those
SaaS
are
are
part
of
all
this
testing.
C
A
So
it's
not
in
this,
it's
not
in
this
debt,
but
we,
our
team,
has
built
a
tool
called
P
bench
which
is
a
distributed
system,
analysis,
tool
and
it'll
gather
all
of
the
CPU
profiles
like
go
P,
prof.
data,
and
so
we
have
that
for
every
run
we
do
across
all
of
the
math.
All
the
important
nodes
on
the
system
have
full
tracing
enabled
we
have
all
this
graphs
and
stuff
so
I
showed
you
a
graph
on
a
graph
of
CPU.
That's
one
example
of
that,
but
there's
also
p
prof
data
and
that's
the
one.
C
A
Yeah
yeah,
yeah,
so
I
think
I
think
the
open
tracing
guys
could
probably
play
a
role
here,
because
the
the
events
that
cube
throws
are
probably
not
as
granular
as
we
need
and
they're.
Also,
the
time
stamps
aren't
accurate
enough.
So
there's
improvement
to
be
done
there.
What
I
find
personally
and
they
feel
free
to
disagree-
is
that
the
only
thing
that
matters
is
the
is
the
user
experience
is
how
long.
Ultimately,
we
want
to
know
along
the
way
how
long
each
subsection
took.
A
We
want
to
make
sure
that
the
total
time
to
stand
up
a
pod
is
within
those
five-second
bounds,
as
well
with
pre
pulled
images
and
at
scale.
It
turns
out
that
you
end
up
violating
those
around
the
60,000
pod
range
or
something
like
that
so
and
a
lot
of
this
is
subjective.
It
depends
on
your
I/o
subsystem
performance,
so.
A
Absolutely,
and
so
we
every
time
we
do,
one
of
these
runs.
The
actual
activity
is
analyzing
all
of
the
data
that
gets
recorded
out
of
it
and
finding
out
where
we
cpu-bound
in
that
ansible
scenario,
I
actually
showed
you
the
good
graph.
The
bad
graph
was
a
disaster
and
we
fixed
all
those
bugs
and
ansible
along
the
way,
but
the
those
graphs
were
instrumental
in
telling
us
what
portion
of
the
ansible
playbooks
actually
had
those
kinds
of
issues.
So
in
the
ansible
scenario,
that's
one
thing
and
then
the.
C
A
Other
thing
is
in
the
API
server,
we
might
see
huge
CPU
spikes
and
then
we
can
go
in
and
look
at
the
p
prof
data
that
correlates
with
that
time
frame
and
get
back
traces.
Basically,
that
tells
us
how
much
CPU
time
was
spent
in
a
certain
function
or
whatever,
and
then
we,
you
know
it's
just
like
it's.
You
peeling
an
onion.
So
we
do.
We
do
have
like
extensive
debug
and
tracing
and
in
all
of
these
runs.
A
A
A
It
doesn't
have
ideal
tracing
facilities
right
now
and
there's
some
pretty
big
shows
I,
don't
want
to
say
showstoppers,
but
it
we
wish
it
would
be
see
and
that
we
could
debug
it
like
a
normal
language,
but
here
nor
there
we
have
everything
in
go
at
this
point,
I
was
going
to
run
through
a
couple
of
tuning
subsist
yeah.
It
wasn't
yep.
A
If
you're
talking
about
VX
LAN,
you
know
single
night
and
actually
visibility
into
the
networking
tier
is
something
we
specifically
asked
the
CNC
F
to
give
us,
so
they
were
managing
the
hardware
we
ran
into
like
firmware
issues
on
the
NEX.
We
ran
into
a
bunch
of
ugly
before
we
could
even
get
OpenStack
installed,
and
that
was
specific
to
that
lab.
So
when
they
bring
more
gear
on
we're,
gonna
have
to
probably
go
through
the
same
thing,
but
it
was
it
was,
and
we
we
know
that
wasn't
optimal.
We
did.
A
We
did
do
the
double
V,
X
LAN
encapsulation.
Here
the
next
time
we
go
through,
we
will
use
flannel
one
of
the
things
we
like
about
open
ship
is.
It
has
multi-tenant,
secure
networking
out
of
the
box.
We
need
OVS
to
do
that.
If
we
drop
back
to
flannel
the
trade-off,
is
you
don't
get
that
multi-tenant
security
anymore
depends
on
the
site
whether
or
not
that
matters
or
another
pluggable
network
implementation?
So
that's
another
thing.
We
work
with
all
these
partners
and
vendors
and
the
load
balancer
is
pluggable.
A
The
networking
subsystem
is
pluggable
as
well
totally
pluggable
and
it's
fully
supported.
So
thanks
for
bringing
that
up,
the
networking
performance
is
really
important
to
meet
you.
That's
where
I
kind
of
came
from
in
my
history
anyway,
so
for
CPU
bound
workloads
in
in
cube
one
three
one
of
the
main
features
that
went
in
I
guess
this
is
really
all
you
need
to
know
about
performance
management
on
cube.
Is
that
there's
a
quality
of
service
tiers
that
have
been
enabled,
and
so
there's
there's
a
what
is
it
batch
best
best
effort,
guaranteed
and
I?
A
Think
there's
a
third
one
batch
I
think
and
the
idea
there
is
that
if
there's
a
contended
resource,
the
guaranteed
pod
would
get
scheduled
and
run
on
the
CPUs
more
often
than
the
best
effort
pod.
The
idea
there
is
to
increase
utilization
and
also
have
some
relative
priorities
between
pods
that
are
running
on
the
same
system.
So
if
you
have
a
performance,
sensitive
workload,
you
would
want
to
give.
You
would
want
to
place
that
in
the
guaranteed
tier,
that's
really
the
only
takeaway.
A
You
also
want
to
make
sure
that
there's
a
limit
syndrome,
Quest's
capabilities
and
kubernetes
as
well.
You
want
to
make
sure
that
those
are
sized
to
fit
your
application
a
lot
of
times.
People
will
undersized
it
and
wonder
why
things
are
running
slow
and
the
reality
is
they're.
You
know
they're,
being
throttled
intent
like
by
the
design
of
the
system,
so
for
a
lot
of
these
scenarios
will
turn
off
those
those
or
make
everything
guaranteed
or
make
it
set.
No
limits
on
the
on
the
pods
back.
A
Other
applications
might
have
might
need
specific,
like
CPU
features
or
run
on
specific
cores
and
there's
there's
kind
of
weird
ways
that
you
can
manage
this
now
with
labels
on
nodes
and
upcoming
features
like
taints
and
toleration,
so
eventually
we'd
like
to
get
to
a
place
where
you
have
a
little
bit
more
fine-grained
control
over
where
things
run,
pneumatology,
PCI,
locality,
etc.
We
don't
know
if
that's
gonna
be
manageable
or
not,
but
it
is
something
we're
throwing
around
the
ideas
for
memory.
Latency
again
is
around
Numa
and
limits.
A
It
did
have
a
slide
here
about
networking.
So
a
lot
of
this
is
like
it's
totally
infrastructure
dependent.
If
you're
running
on
Google's
cloud,
you
have
their
Sdn.
If
you're
running
on
Amazon,
you
have
their
Sdn.
If
you're
running
on
OpenStack,
you
have
that
s
the
end,
etc.
Each
one
of
those
has
their
own
characteristics
in
terms
of
scale
throughput
and
packets
per
second
type
of
latency
numbers.
A
We've
found
that
running
on
public
clouds
is
we
basically
cannot
do
successful
regression
testing
on
public
clouds,
because
there's
too
much
variance-
and
this
is
something
we're
in
constant
struggles
with
because
they're
extremely
popular
the
public
clouds.
But
we
have
too
much
noisy
neighbor
problems
and
also
buying
the
amount
of
resources
it
takes
to
actually
get
huge
guests
that
have
all
the
provisioned
I
ops
that
you
might
want,
and
then
these
placement
groups
or
equivalent
technologies
on
different
clouds,
just
the
budget,
just
runs
right
through
the
roof.
A
So
we've
fallen
back
to
bare
metal
because
performance
analysis
needs
that
tight
control
and
if
you
don't
have
it,
then
you
end
up
chasing
red
herrings
all
over
the
place
and
it's
a
waste
of
time
for
storage,
rahat
and
I.
Think
Google
have
introduced
something
called
storage
classes,
which
is
a
way
of
tearing
your
storage.
A
Yeah
we,
the
build
farm,
is
the
short
short
lived
container
case
for
us.
Those
containers
usually
live
like
one
minute,
but
it's
not
like
functions
as
a
service
where
things
run
for
a
couple
hundred
milliseconds.
That's
not
the
type
of
thing
we're
looking
at
yet,
but
the
build
farm
analysis
was
very
high
churn
and
in
that
scenario,
I
think
the
use
case
is
going
to
be
kubernetes
scheduler
throughput
and
doing
something
better
than
increasing
the
parallelism.
That's
possible
with
docker
we've
thought
about
having
multiple
docker
daemons
to
kind
of
parallelized
things.
A
But
the
reality
is
it's
not
it's
not
highly
parallel.
I
would
call
it
mildly,
parallel
safe
at
the
moment,
but
you
can
run
into
situations
and
until
docker
1:12,
if
you
overwhelm
the
docker
daemon,
you
are
screwed
like
you
keep
your
all.
Your
apps
are
going
down.
Things
are
much
better
with
1:12,
where,
if
the
data
you
know
the
Damons,
not
the
parents
anymore.
So
if
you
kill
the
daemon,
you
don't
actually
lose
your
abs,
but
the
short-lived
containers
is
to
build
form
scenario.
I
shared
earlier.
B
A
Was
for
the
OpenStack
VM
storage,
so
the
mountain
unmount
times
are
there
in
there
I
mean
yeah.
We
had
it's
actually
not
that
big
of
a
deal.
It
actually
takes
more
time
to
allocate
a
veeth
than
then
a
block
device.
Quite
honestly,
I
mean
we're
using
device
mapper
for
our
storage
here.
But
overlay
is
one
scenario
we
investigated.
We
did
some
tricks
recently.
It's
like
overlay
would
get
us
faster
storage,
faster
graph
driver
because
it
has
memory
page
cache
sharing,
and
so
you
have
less
memory
usage.
A
So
there's
a
blog
on
that
on
developer,
blog
dot,
reddit
com.
So
what
we're
gonna
do
next?
The
gentleman
asked
about
the
normal
density
API
response
time.
Latency
is
we
need
to
work
on,
hopefully,
in
the
next
couple
releases
of
cube.
We
need
to
formalize
that
a
little
bit
more
and
extend
it.
Some
of
my
ideas
if
I
was
just
spitballing
is
we
need
to
know
like
CPI
kind
of
measurements,
around
all
of
the
subsystems
that
are
involved.
We
need
to
actually
test
real
applications
and
run
workloads
on
them.
A
So
if
you
grab
our
git
repository,
we
also
have
a
workload
generator
that
spins,
so
it
will
deploy
all
the
apps
with
cluster
loader.
Then
it
will
come
back
and
spin
up
a
dynamic
number
of
jmeter
pods
and
fire
off
the
tests
against
those
and
we'll
go
and
query
the
API
server
for
the
routes,
and
it
will
build
a
jmeter
test
plan
dynamically
and
then
fire
load
at
it
and
then
build
graphs.
A
I
think
that's
pretty
much
it
and
we
have
some
STD
testing.
Of
course
that's
always
ongoing.
Oh
and
another
thing
is
hyper-converged
storage,
so
Brad
sells
a
product
called
container
native
storage,
which
is
Gluster
and
it's
running
on
top
of
OpenShift
or
on
top
of
kubernetes
and
the
it
provides
the
dynamic
provisioning
of
persistent
volumes
directly
to
kubernetes
and
it
runs
within
kubernetes
as
pods
itself.
So
that's
something
we're
gonna.
Take
a
look
at
over
the
next
couple
weeks.
A
The
guys
are
extremely
busy
with
features
and
I
know.
Connor
is
here,
he's
doing
this
one
pod
level
cgroups
right
now,
I
mean
it's
not
about
the
individual
people,
but
the
idea
is
we're
tackling
many,
many
different
things:
huge
page
support
for
certain
applications.
We've
also
introduced
certain
sis
controls,
the
ones
that
are
namespace,
aware
we've
introduced
already
in
kubernetes
the
croisé
stuff
and
then
another
thing
we're
probably
going
to
need
is
a
way
to
feed
Hardware
features
into
the
scheduler.