►
From YouTube: Predictable Performance Through Prometheus and Topology Aware Schedu... Killian Muldoon & Tom Golway
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Predictable Performance Through Prometheus and Topology Aware Scheduling - Killian Muldoon, Intel & Tom Golway, Hewlett Packard Enterprise
The Kubernetes scheduler does an excellent job scheduling cloud native workloads but the same isn’t true for performance sensitive workloads. Scheduling based on Topology of hardware resources is currently impossible due to the scheduler’s lack of knowledge of node topology. Resulting performance is less predictable, and advanced scheduling strategies – such as trait based placement– are blocked. Topology Manager in kubelet aligns topology-based resource allocations from CPU and Device Manager but only after workloads are scheduled. With Observability tools becoming popular we solve this problem using Prometheus and custom scheduling enhancements. The focus of this talk is Topology Aware Scheduling and we discuss how exposing cluster level topology to the scheduler, and using Prometheus to track NUMA topology related metrics, enhances the default scheduler, empowering it to proactively use node level topology to make intelligent NUMA aware placement decisions leading to more effective cluster wide performance of workloads.
https://sched.co/ZeqI
A
B
Hi,
I'm
tom
goalway
from
hewlett-packard
enterprise,
and
I
work
in
the
advanced
r
d
organization,
where
I'm
focused
on
ways
to
provide
infrastructure
that
supports
disaggregated
application
architectures
today,
killian
and
I
will
be
discussing
the
need
for
a
greater
awareness
of
infrastructure
capabilities
in
the
scheduling
process
to
look
to
achieve
predictable
service
level
guarantees
for
applications,
digital
disruption
and
digital
transformation
have
been
viral
topics
for
quite
some
time.
How
many
times
have
you
heard
a
ceo
say?
B
B
The
shift
in
application
architecture
lee
is
leading
us
to
look
at
building
the
underlying
infrastructure
to
support
what
I
call
infrastructure
plasticity.
Now
that
means
delivering
the
infrastructure
model.
That's
presenting
itself
to
applications
with
the
capability
of
undergoing
organic
changes
on
the
fly
based
on
the
applications
class
of
service,
so
think
about
anything
from
system
level,
changes
to
large-scale
network
topological
remappings.
B
B
Applications
are
typically
architected,
either
monolithic
or
using
some
simple
client
server
paradigms.
If
you
look
at
data
volumes,
we
typically
see-
let's
say
a
typical
retail
database
with
all
historic
transactions
is
roughly
around
40
petabytes
system.
Engagement
is
about
highly
interactive
near
real-time
applications,
typically
handling
what
you
would
call
messy
data,
such
as
social
media,
for
example.
B
If
you
look
at
data
volumes,
a
typical
per
day,
posting
of
social
media
is
about
four
petabytes
system
of
action
supports
more
autonomy
within
the
systems,
either
augmenting
human
decisions
or
fully
controlling
actions
and
outcomes.
Processing
is
in
real
time
highly
sensitive
to
latency
and
gender
and
needs
high
accuracy
and
traceability.
B
B
B
B
With
cloud
native
application,
architectures
and
distributed
data
sources,
the
veracity
of
data
is
a
significant
challenge.
We
see
an
increased
need
for
trust
in
the
data,
with
a
focus
on
protection
of
data,
both
in
motion
and
at
rest,
but
we're
also
seeing
a
trend
towards
protection
of
data
in
use.
B
Application
architectures
are
shifting
towards
a
more
disaggregated
model,
offering
greater
agility,
supporting
elasticity
and
providing
greater
control
for
software
quality
assurance
by
leveraging
a
scale-out
disaggregated
model.
Applications
have
the
ability
to
be
deployed
using
a
composable
model
with
a
mix
of
core
code,
libraries,
open
source,
external
applications
and
microservices,
but
the
individual
components
of
the
application
will
be
executed
on
the
platform
that
best
fits
its
function.
A
B
B
Small
degradations
in
performance
may
seem
inconsequential
for
an
individual
microservice,
but
when
aggregated
to
the
application
level,
it'll
impact
business
process,
service
levels
and
not
all
applications
are
equal.
So
supporting
differential,
differentiated
service
levels
are
required
to
support
the
digitally
aware
enterprise.
A
Thanks
as
tom
was
saying
there
we're
starting
to
get
a
good
picture
of
what
application
the
applications
would
need,
we're
getting
new
ways
to
express
that,
but
we're
still
falling
really
short
in
the
hardware
side
kubernetes
from
the
orchestration
level.
It
knows
what
infrastructure
is
made
of,
but
it
doesn't
know
what
it
can
actually
do
right.
So
the
scheduler
can
take
requests
for
a
number
of
cpu
cores,
but
it
doesn't
actually
know
or
doesn't
care
about
how
much
processing
an
individual
core
can
do.
A
A
A
This
really
doesn't
scale
well,
it
leaves
us
with
kind
of
secret
recipe
configuration
files
in
where
directories
only
ever
get
understood
by
a
couple.
People
in
the
organization
kind
of
can
lead
to
gatekeeping
issues,
but
just
overall
just
organizational
issues
right,
communications
issues,
so
tom
laid
out
there
kind
of
a
way
to
express
what
applications
might
need
and
we
need
to
match
that
by
providing
a
way
to
describe
hardware
in
terms
of
in
terms,
applications
can
really
understand
right,
so
meet
them
on
applications
terms
rather
than
expressing
hardware
as
a
bunch
of
ones
and
zeros.
A
So
I
mostly
talking
about
packet
processing
workloads,
they
have
intense
performance
characteristics,
conflicts
get
really
complicated
as
a
result,
and
this
can
be
really
really
hard
to
manage.
So
here,
on
the
left
hand,
side
of
the
bottom
under
pod
requirements,
there's
just
a
kind
of
a
model
spec
for
resource
requests,
there's
a
load
of
different
resources
in
there.
Cpus
virtual
functions,
specific
speeds
in
there
specific
power,
headroom
needed
and
there's
new
alignment
needed
as
well.
Something
kubernetes
just
can't
do
right
now,
even
though
it
has
massive
performance
impact.
A
So
we
can
reduce
the
spec
if
we
build
in
kind
of
hardware
knowledge
into
a
traits
model
that
matches
the
application
model.
That
tom
was
talking
about
and
you
can
match
the
requirements
an
application
has
here.
You
know
fast
networking,
maybe
some
encryption
acceleration
power
needs
whatever.
It
is
and
express
that
as
a
single
trait.
A
So
when
we
describe
platforms
in
terms
of
capabilities,
you
get
to
push
a
lot
of
the
complexity
of
hardware
configurations
to
a
level
where
you
can
share
it
in
kind
of
a
consumable
way
across
organizations
publicly.
Even
it
allows
upgrades
to
be
made
so
the
same
application
can
be
deployed
multiple
times
at
multiple
class
of
service
expressions.
So
different
versions
of
the
application
actually
provide
different
service
level,
assurances,
maybe
different
speeds
in
terms
of
packet
processing.
A
It
could
be
so
you
can
actually
control
what
level
of
performance
is
is
being
given
been
offered
to
a
specific
application.
The
most
important
part
of
this,
though,
is
you
get
to
keep
application
developers
away
from
hardware?
They
don't
have
to
learn
about
different
cpus.
They
don't
have
to
learn
about
the
wattage
of
platforms,
cooling,
none
of
that
hardware
specific
needs
to
be
in
a
pod
spec
if,
if
you
get
to
this
capability
expression,
so
this
is
kind
of
a
proof
of
concept
of
this.
A
It's
not
going
to
be
scalable
across
all
sorts
of
different
hardware
to
be
doing
that
through
promptql
queries.
So
the
question
is:
how
do
we
actually
get
this
up
and
running
so
here
we're
looking
at
a
dashboard?
This
is
graffana
on
the
left-hand
side
and
terminal
windows
on
the
right-hand
side,
the
top
three
boxes.
Here
you
can
see
the
names
of
the
nodes
master,
one
node,
one
node,
two
they're
an
indicator
of
the
actual
trait
that
we're
working
with.
A
A
A
So
this
is
because
we
have
kind
of
power
hungry
workloads
across
the
cluster,
so,
on
the
right
hand,
side
in
our
terminal,
I'm
going
to
start
the
cloud
function
and
I'm
going
to
watch
its
progress
through
the
cluster
and
then
on.
The
bottom
right
are
the
logs
out
of
the
telemetryware
scheduling
system
so
down
there.
You
will
see
the
actual
decision
being
made
in
real
time
by
the
scheduler
based
on
these
traits.
A
So
a
bunch
of
logs
are
after
popping
up
there
at
the
bottom
right
you
can
see
for
each
node,
so
master
one
node
one
node
two
tell
em
to
our
scheduling
is
aware
that
the
node
is
actually
violating
a
rule
and
that
rule
is
the
name
of
our
trade.
It's
10
gigabit
networking
at
one
mega,
one
million
packets
per
second.
A
So
again
this
because
we
don't
have
enough
power
headroom
on
the
system
telling
to
our
scheduling
would
continually
as
the
the
normal
kubernetes
scheduler
the
default
scheduler
does
continually
retry
this
scheduling
with
the
back
off,
but
we're
not
going
to
wait
for
the
power
to
actually
come
back.
We're
not
going
to
wait
for
that
workload.
Finish
we're
just
going
to
manually
preempt
the
workload,
so
this
is
just
deleting
the
pod.
So
that's
a
power
hungry
application
is
going
down
and
we're
just
back
watching
on
our
cloud
network
function
again.
A
So
that
workload's
going
to
take
a
minute
to
spin
down.
So
while
we
do
that,
we
can
have
a
look
at
some
of
the
complexity,
that's
actually
being
buried
here.
So
in
this
text
box,
there's
no
need
to
read
all
of
this.
You
can
see
the
the
actual
make
up
of
a
single
query
that
satisfies
this
trait.
This
is
the
definition
of
that
trait.
So
we
have
just
setting
min
and
max
values.
So
it's
plus
minus
one.
A
We
check
what
free
cpus
are
there
so
which
ones
aren't
being
currently
used
by
pods
and
that
they're
new
my
line.
So
this
is
free,
free,
cpus
per
numa
node.
We
check
for
the
cpu
frequency
again
per
numa
node
huge
pages
power
headroom,
which
is
the
important
one.
Here
we
can
check
for
number
devices.
This
is
our
virtual
functions.
A
Our
network
devices
check
that
enough
bandwidth
is
actually
available
on
those
devices
and
then
at
the
bottom,
we're
just
adding
a
negative
score
for
the
missing
nodes,
because
this
is
a
filter
effectively,
but
we
want
to
make
sure
that
we
have
minus
ones
for
the
nodes
that
don't
meet
the
trade
as
well.
You
can
see
down
below
our
power
has
actually
come
back
so
after
descheduling
that
power
hungry
application,
our
indicator
is
now
green.
A
A
So
our
query,
there
prometheus
query,
is
doing
a
few
things
that
a
normal
scheduler
can't
do
at
all.
One
is
it's
looking
at
live
contention,
which
is
what
we
have
with
our
power,
and
the
second
thing
is
doing
numa
alignment.
Currently
there's
no
new
alignment,
that's
memory,
topology
in
kubernetes.
At
the
control
plane
level,
there
is
the
topology
manager
at
the
node
level,
so
our
workload
has
just
been
scheduled.
Judging
by
the
top
right,
so
it's
up
and
running.
We
can
see
the
bottom
right.
A
The
last
line
of
our
logs
there
is
that
the
filtered
nodes
for
our
hardware
trades
master
one
so
that
node
has
now
been
that
pod
has
now
been
scheduled
on
node
one,
so
just
to
show
it
up
and
running
we're
going
to
just
run
from
the
host
we're
just
going
to
run
a
packet
pro
processing
workload
on
it,
we'll
be
able
to
see
some
of
the
packets
coming
through.
This
is
a
user
space
dvdk
workload,
so
it's
actually
avoiding
the
kernel
in
order
to
get
really
high
high
packet
rates
in
and
out.
A
So
we
use
sri
vinix
and
and
all
that
so
we'll
see
the
workload
on
the
third
dial
here
at
the
bottom
begin
to
show
packet
throughput.
So
all
of
these
components
are
available,
they're,
all
open
source
and
there's
a
document
on
the
telemetry
scheduling
repo
with
the
collateral
for
this
demo
and
showing
you
how
to
set
it
up
and
implement
it,
and
we
can
see
there
as
the
packet
rate
increases.
The
trade
indicator
again
turns
to
-1
turns
to
red,
and
that
means
that
no
workloads
will
actually
be
scheduled
and
consume
that
bandwidth.
A
So
tom
put
forward
new
ways
to
think
about
application
requirements
in
terms
of
a
class
of
service
model,
and
we've
shown
here
that
it's
actually
possible
to
meet
that
the
other
side
by
generating
kind
of
hardware
traits
and
then
matching
them
off
of
each
other.
So
the
really
kind
of
complex
requirements
that
you
get
in
performance
performance
intensive
applications
right
now
you
get
loads
of
node
labels.
You
get
nodes
that
have
specific
resources
tainted.
You
get
big
long
lists
resources
all
this
complicated
cluster
management,
with
more
accelerators,
more
types
of
processors.
A
You
know
your
people
are
talking
about
the
the
year
of
the
xpu
right,
which
is
where
you're
going
to
have
multiple
different
processing
units
in
a
machine.
This
is
going
to
create
more
and
more
configuration
overhead
right.
You
got
more
room
for
error,
more
resources,
locked
up
and
just
kind
of
managing
these
recipes
to
to
get
your
workload,
the
performance
that
it
requires
just
have
this
kind
of
spaghetti,
yaml
building
up
over
time.
A
So,
even
though
it's
complicated,
this
particular
performance
is
going
to
be
incredibly
important
to
getting
certain
workloads
to
work
on
kubernetes
network
pack.
Packet
processing
is
one
of
those
also
high
performance
computing
workloads,
the
configs
that
you're
getting
now
they're,
causing
real
trouble
today
and
they're
only
going
to
get
more
complicated
over
time.
It's
going
to
be
more
taxing,
it's
going
to
be
more
work,
so
the
matching
that
we've
shown
here,
which
is
kind
of
application
class
service
to
live
platform
traits.
A
It's
one
way
to
harden
this
obstacle
right,
but
there's
a
huge
amount
of
work
left
to
be
done.
What
we've
shown
here
today
isn't
going
to
be
deployed
on
production
any
anytime
soon.
So
there's
a
few
things
really
missing
in
kubernetes
that
just
cause
pain,
so
memory,
topology
management,
major
roadblock,
seen
that
throughout
this
there
are
kind
of
efforts
going
on
through
the
signaled
resource
management
group
to
help
fix
these
issues.
A
We're
looking
forward
to
seeing
the
solutions
come
out
of
those
discussions.
We
hope
that
we'll
get
support
for
those
at
the
cubelet
and
at
the
scheduling
level,
maybe
elsewhere
in
the
control
plane
as
well.
If
it
needs
to
go
into
the
the
api
or
something
like
that,
but
the
bottom
line
is:
if
we
don't
want
developers
and
and
cluster
administrators
and
whoever
else
to
have
to
add
kind
of
hardware
engineering
skills
to
their
to
their
skill
set.