►
From YouTube: NHD - A Topology-Aware Scheduler for K8s for Low-Latency & HPC Applications - Cliff Burdick, ViaSat
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
NHD - A Topology-Aware Scheduler for K8s for Low-Latency & HPC Applications - Cliff Burdick, ViaSat
With an increasing number of HPC, NFV, and other low-latency applications moving to containers, the ability to schedule these workloads efficiently is important for increasing user adoption. The default scheduler in Kubernetes does an excellent job at scheduling cloud-native workloads, but is lacking the ability to schedule low-latency workloads properly. NHD attempts to bridge this gap by introducing a custom scheduler for Kubernetes that’s aware of hardware topology, CPU characteristics, and the application’s threading model. In this talk, we’ll go over the ways NHD integrates with Kubernetes, how it’s used, and the features it offers.
https://sched.co/UabY
A
Hi
everybody,
my
name,
is
cliff
verdict:
I
work
at
ViaSat,
we're
about
20
miles
up
the
street
if
any
of
you
guys
are
looking
for
somewhere
to
work
and
you're
interested
in
this
stuff.
Let
me
know
today
I'm
going
to
talk
about
a
topology
aware,
scheduler
that
we
built
for
kubernetes
and
we'll
go
over
some
of
the
reasons
behind
why
we
built
it
in
the
first
place,
because
that's
a
common
question
that
we
get
so
on
the
agenda
today.
A
So
before
we
get
there,
there's
some
talks
that
are
very
very
similar
to
this.
One
I
think
if
you're
coming
to
this
you're,
probably
interested
in
these
ones,
I
updated
the
room
numbers
as
well.
Two
of
these
are
directly
after
this
talk
at
the
exact
same
time,
so
you
have
to
pick
between
one
of
those,
but
then
there's
two
more
tomorrow
that
are
kind
of
along
the
same
lines
as
this
one.
A
So,
first
for
terminology,
HPC
stands
for
high
performance
computing
and
it's
usually
refers
to
a
dense
server,
a
dense
collection
of
servers
that
are
performing
really
high
speed
calculations,
usually
across
many
servers,
the
most
common
use
cases.
You
have
really
dense
GPU
servers
and
you're
fanning
out
a
whole
bunch
of
data
back
and
forth
between
them
to
work
on
a
problem.
The
network
function,
virtualization
is
where
you
implement
a
network
function
in
software
and
usually
it's
at
a
fairly
high
rate.
A
Multiple
gigabits
per
second
Numa
stands
for
non-uniform
memory
access,
and
this
refers
to
how
the
processors
access
the
memory
in
the
system.
Our
DMA
is
remote
DMA.
This
is
where
you
transfer
data
directly
from
the
NIC
to
a
GPU.
It's
not
referring
to
any
particular
protocol
here,
just
the
act
of
doing
that
and
these
applications
typically
need
very
high
in
network
throughput,
so
40
to
100
gigabit
per
second
interface
is
very
common.
Now,
with
HP
HP,
C&N,
EFI
applications
and
finna
bans,
another
type
of
interface,
that's
very
common
and
usually
there's
multiple
interfaces.
A
So,
instead
of
just
the
single
interface,
which
kubernetes
limits
you
to
usually
you'll
have
multiple
data
plane
interfaces.
In
addition
to
the
to
the
main
interface,
they
usually
require
low,
latency,
so
under
10
micro.
Second
turnaround.
The
time
for
a
packet
processing
is
pretty
common
and
to
get
those
numbers,
we
usually
use
smart
NIC,
so
they
have
hardware
offloads
like
checksum,
offloads
tunneling.
A
Some
of
them
have
encryption
and
compression
and
then
last
there's
the
accelerator
cards.
So
GPUs
are
probably
the
most
common
for
kubernetes
because
they
have
first-class
support
now,
but
you
could
have
FPGAs
CPUs
and
in
the
Google
compute
or
any
other
type
of
accelerator
card.
You
can
imagine
so
what's
needed
to
do
an
HPC
or
an
EFI
application,
so
user
space
networking,
specifically
using
DP
DK,
is
one
of
the
main
features
of
an
NFC
application
which
we'll
get
into
next
isolated
CPUs.
A
So
you
don't
want
anyone
else
running
on
the
same
CPU
as
you
Numa
awareness
for
both
memory
and
devices.
So,
if
you're
accessing
memory,
you
want
to
know
that
it's
local
memory,
not
remote
memory
and
similar
to
any
device.
That's
in
your
system,
you
want
to
make
sure
that
you're
taking
the
fastest
path
to
the
device.
A
So
really
briefly,
DP
DK,
for
those
that
don't
know,
is
a
user
packet
processing
framework.
One
of
the
main
features
of
in
the
reason
why
people
use
it
is
it
completely
bypasses
the
kernel
for
efficiency.
I
wrote
here
most
of
the
time,
because
there
are
PDK
drivers
that
don't
bypass
the
colonel,
but
it's
much
more
common
to
use
the
ones
that
bypass
the
colonel
and
you're,
giving
up
a
lot
by
not
going
through
the
colonel,
but
the
people
who
use
DB
DK
typically
don't
need
all
the
things
that
the
Colonel's
doing
to
the
packets.
A
One
of
the
main
differences
compared
to
normal
Linux
packet
processing
is
DP
decays
polling
based,
so
you
sit
there
and
you
have
a
core
that's
sitting
there
or
a
thread.
That's
sitting
there
on
a
core
pulling
the
NIC
as
fast
as
it
can
to
see
if
packets
are
coming
in,
as
opposed
to
being
interrupted
when
packets
come
in,
there's
also
dozens
of
pull
mode
drivers.
A
These
is
what
DP
DK
calls
the
drivers
that
are
used
with
it
and
the
pull
mode
drivers
are
nice
because
they
allow
you
to
write
an
application
using
the
DP
DK
API,
and
if
you
switch
to
a
different
type
of
hardware,
a
different
network
card,
you
should
only
have
to
switch
the
driver
underneath
it
and
not
change
any
of
your
code
and
usually
that
works.
But
if
you're
using
some
of
the
more
sophisticated
features
in
DP
DK,
you
might
not
get
one-to-one
parity
between
manufacturers,
but
from
a
high
level.
A
It
works
pretty
well
and
then
last
DVD
K
is
not
just
for
packet
processing.
It
has
a
really
high
performance,
Numa
ware
framework
for
things
like
inner
thread,
communications
or
ring
buffers
memory
pools.
The
memory
allocation
algorithm
is
extremely
efficient
in
crypto
and
compression
as
well.
They
can
perform
in
hardware,
and
even
if
you
haven't
heard
of
DP
DK,
it's
it's
usually
the
foundation
of
many
of
the
high
speed
packet
processing
libraries
out
there.
So
if
you
use
tier
X
by
Cisco,
there's
another
one
by
juniper,
they
all
use
DP
DK
under
the
hood.
A
So
what
does
it
mean
to
be
topology
aware
so
in
kubernetes?
From
a
simplistic
point
of
view,
this
is
what
it
looks
like
when
a
pod
gets
scheduled
comes
up
to
the
API
server.
The
scheduler
sees
that
a
pot
is
ready
to
be
scheduled,
find
some
resources
for
the
pod
and
then
schedules
it
to
a
node.
So
number
two
is
really
the
part
we
are
focusing
on,
because
that's
the
biggest
problem.
What
does
it
actually
mean
to
be
a
CPU
core?
What
does
it
mean
to
be
memory?
A
Is
it
all
the
same
amongst
all
your
nodes,
so
for
high-performance
computing?
I
mentioned
huge
pages
earlier.
So
that's
one
of
the
applications
where
memory
is
not
all
the
same.
Some
systems
may
not
have
huge
pages
turned
on
some
might
some
might
have
different
size,
huge
pages,
and
you
really
want
to
find.
What's
appropriate
for
your
application,
so
from
a
top-level
view,
when
you
look
at
a
single
socket
Zeon
with
hyper-threading
off
it's
fairly
simple,
you
have
in
this
case
20
cores.
A
Each
core
has
an
l1
and
l2
cache
that
it
uses
and
then
there's
a
shared
l3
cache
for
all
the
cores
combined
and
then
below.
You
have
a
DDR
controller
that
has
about
80
gigabytes
per
second
going
out
to
main
memory.
So
when
you're
scheduling
a
pod
on
a
processor
like
this,
it's
not
too
difficult,
you
can
treat
all
the
cores
is
mostly
the
same.
They
all
go
through
the
memory,
the
same
memory
controller
and
they
all
share
the
same
l3
cache
and
they
all
have
independent,
l1
and
l2.
A
Now
each
of
those
can
independently
go
out
to
DDR
at
about
80
gigabytes
per
second
again,
and
now
you
have
an
inner
processor
linked
in
Intel's
cases
called
the
UPI
or
qpi
that
runs
at
about
35
gigabytes
per
second.
So
this
is
now.
You
can
start
to
see
the
problems
where
there's
a
scheduling
issue
where,
if
you
put
an
application
on
core
0
and
then
another
application
on
core
20
on
the
processor
on
the
left,
you
could
have
a
lot
of
cache.
A
Thrashing
because
they're
not
aware
that
they're
on
the
same
they're
sharing
the
same
caches
there,
and
even
if
it
was
your
own
application
that
was
being
placed
on
0
and
20.
Your
own
application
might
have
a
situation
where
those
cords
don't
share
data,
so
you
wouldn't
want
them,
sharing
the
l1
and
l2
cache
and
then.
Similarly,
if,
if
you
have
a
thread,
that's
on
the
Left
processor,
you
don't
want
to
access
memory
on
the
right
processor
because
that
in
curves,
more
latency
and
more
throughput
penalties
going
through
the
UPI
bus.
A
If
you
zoom
out
a
little
bit
and
now
we're
adding
two
GPUs
and
two
Nix,
which
are
each
connected
directly
to
the
the
PCI
bridge
on
the
z
processor,
it
looks
very
similar
before.
But
now
you
have
the
PCI
Express
interface
above
that,
where
the
GPU
and
the
NIC
can
both
hit
the
processor
at
about
a
hundred
gigabytes
per
second
or
twelve
and
a
half
gigabytes
per
second.
So
this
isn't
too
difficult.
But
now,
if
you're
using
our
DMA,
you
need
to
make
sure
that
the
NIC
going
to
the
GPU
is
on
the
same
socket.
A
A
But
if
you
look
at
the
more
common
GPU
servers
now
you
have
four
GPUs
and
four
Nick's
which
two
are
on
each
two
are
bound
to
each
processor
through
a
PCI
uee
switch.
So
now,
in
addition
to
having
the
two
processors
there
and
the
link
back
to
the
processor,
you
have
each
of
the
CPUs
and
the
in
the
Jeep,
each
of
the
GPUs
and
the
Nick's
going
at
about
100
gigabits
per
second
back
to
a
PCIe
switch.
A
But
now
there's
only
a
single
hundred
gigabit
per
second
link
going
back
to
the
Xeon,
which
causes
a
bottleneck.
So
you
could
see
just
by
this.
If
you
needed
to
transfer
the
data
all
the
way
back
to
the
processor
and
then
back
up
to
the
GPU,
you
wouldn't
be
able
to
do
it
with
more
than
one
of
the
throughputs
of
one
of
those
nicks.
A
If
it's
100
gig
Nick,
so
you
really
have
to
be
aware
of
the
PCIe
topology
here
as
well
as
things
like
our
DMA
and
then,
if
you
want
to
get
even
more
complicated,
this
is
an
actual
server
from
Super
Micro,
where
there's
16,
GPUs
and
16
NICs.
So
you
have
a
total
of
32
PCIe,
316
Lane
links
between
the
PCIe
switches
and
the
devices,
and
not
just
that.
Now
you
have
a
tiered
PCIe
switch
topology
where,
from
the
top
level,
you
still
have
the
two
GPUs
and
tunics
plugged
into
each
PCIe
switch.
A
But
if
you
go
down
one
level,
you
can
still
go
back
up
to
the
other
GPUs
and
NICs
without
going
through
the
processor.
But
again
it's
only
one
link
so
you're
bound
by
that
12
and
a
half
gigabyte
per
second
link
and
again,
if
you're,
transferring
data
from
the
left
side
of
the
processor
to
the
right
side.
Now,
there's
a
huge
bottleneck
because
UPI
is
only
doing
35
gigabytes
per
second
roughly
and
you
have
32
times
12.5
gigabytes
per
second
from
the
top.
A
And
from
a
network
point
of
view,
if
you're,
assuming
that
you
have
pods
that
have
different
bandwidth
requirements,
some
pods
might
want,
for
example,
on
the
one
on
the
bottom
left.
It
needs
five
gigabits
per
second,
the
one
in
the
middle
might
have
two
separate
processing
paths
where
one
path
needs
about
25
gigabits
per
second
and
then
the
other
path
needs
30
gigabits
per
second,
and
then
you
have
yet
another
pod
with
40
gigabit
per
second
requirements.
A
Now
you
want
to
place
those
potentially
on
multiple
NICs,
but
you
also
want
to
place
them
in
a
smart
way
where
you
can
have
podshare
NIC,
and
in
this
case
this
node
has
three
data
plane,
NICs
that
these
three
pods
could
share.
But
you
obviously
can't
take
the
pod
on
the
right
that
needs
40
gigabits
per
second
and
tie
it
to
the
NIC
on
the
left,
which
needs,
which
is
only
a
10,
gigabit
NIC.
A
So
some
of
the
challenges
that
we
face
when
writing
the
scheduler
we're
isolating
CPU
cores,
CPU
selection
and
NIC
interface
sharing,
so
first
I'll
go
over
isolating
CPU
cores
and
how
we
solve
that
so
for
isolating
CPUs.
What
you're
doing
is
you're
trying
to
prevent
unwanted
processes
from
running
on
the
same
core
that
you
have
something
running
on,
and
the
reason
is
that
it
prevents
cache
thrashing.
A
If
you
have
two
things
running
on
the
same
core
and
they're
both
hitting
the
cache
a
lot,
then
the
cache
is
getting
evicted
quite
a
bit
and
it
also
reduces
the
time
slicing
so
and
you
can
actually
eliminate
it
too
in
Linux,
where,
if
your
process
is
the
only
thing
running
on
that
core,
then
you
don't
need
a
time
slice
between
any
of
the
other
processes.
So
one
of
the
problems,
though,
is
that
kernel
processors
are
little
trickier,
so
some
kernel
processes
can't
be
moved
at
all.
A
If
you
try
to
move
them
using
any
of
the
provisions
in
the
kernel
to
move
them,
it
won't
do
anything.
Things
like
interrupts
also
need
a
special
configuration.
If
you
try
to
move
the
interrupts
processes,
they
won't
do
anything,
but
you
can
do
that
on
boot,
so
there's
actively
two
ways
to
handle
isolating
CPUs,
so
the
first
is
to
use
isil
cpus,
which
is
something
that's
been
in
linux.
A
You
could
do
it
and
you
have
to
explicitly
set
it
because
if
you
don't
set
your
application
to
use
those
cores,
it'll
be
forced
back
onto
the
read
cores
in
this
picture.
The
second
way
is
CPU
sets.
So
this
is
a
more
recent
feature
that
was
added
in
the
kernel
in
about
2008.
So
these
are
nice
because
you
can
create
and
modify
them
after
boot
and
creating
a
CPU
set
doesn't
actually
do
anything
until
you
put
processes
into
the
CPU
set
to
isolate
them.
A
So
what
it
does
is
it
creates
a
group
of
CPUs
that
are
visible
to
any
of
the
processes
that
are
placed
into
that
group.
So
in
the
in
the
example
on
the
top,
the
group
name
is
slash
system
D,
which
is
a
common
one
if
you're
running
system
D
and
then
within
that
there's
five
cores
there
that
are
not
visible
because
you
set
them
to
not
be
visible
in
that
in
that
group.
A
So
when
a
process
is
moved
into
that
system,
D
group,
it
won't
be
able
to
see
those
cores
on
the
system
anymore,
I
can't
use
them
and
you
can
move
any
number
of
processes
that
you
want
into
that
group
and
the
other
nice
thing
is
it's
hierarchical.
So
in
the
bottom
example,
you
have
cube
pods
burstable,
which
is
one
that
you
get
with
kubernetes
and
then
cube
plods
burstable,
slash
pod.
A
One
so
it's
a
nested
group
inside
of
the
top-level
group
and
when
you
have
a
nested
group,
the
child
groups
have
to
inherit
all
the
rules
from
the
parent
group.
So
in
this
case,
there's
three
cores
that
were
not
visible
in
cube,
pods
and
then
one
level
down
and
the
child.
There's
six
cores
not
visible,
and
if
you
move
a
process
into
the
parent
they
will.
They
won't
see
those
three
cores.
If
you
move
them
into
the
child,
they
won't
see
all
six
cores
and,
as
I
mentioned
earlier,
kubernetes
uses
this.
A
So
what
we
did
was
we
solved
it
by
making
a
demon
called
ISIL
CPU
set,
and
this
is
essentially
a
daemon
that
runs
on
startup
and
it
makes
two
groups
on
boot.
One
is
called
the
isolated
CPU
set
and
one's
called
the
jailed
CPU
set,
and
the
first
thing
it
does
is
it
moves
all
the
tasks
or
processes
that
it
can
into
the
jailed
CPU
set.
A
So
anything
that
might
have
been
on
the
isolated
cores
at
that
point
will
now
be
moved
into
the
jailed
set
and
then,
if
your
pod
wants
to
be
moved
into
the
isolated
said,
it
communicates
with
this
demon
to
tell
it
that
it
wants
to
be
in
there
and
then
the
daemon
moves
it
to
a
CPU
set
that
is
allowed
to
go
on
the
isolated
cores.
So
what
this
does
is
it
gives
you
extra
guarantees.
You
have
ISIL
CPUs
still
in
there,
but
now
you
have
an
extra
set.
A
That
is
jailing
off
all
the
the
tasks
that
you
use.
Cpu
sets
and
you
can
move
any
of
your
processes
into
the
isolated
CPU
set
that
you
want
to
so.
The
next
thing
we
had
challenges
with
was
CPU
selection,
so
with
most
applications
you
don't
really
care
about
this.
If
you
launch
a
normal
pod
that
doesn't
it's
not
DP,
D
K
or
doesn't
require
a
special
CPU
core
handling.
Linux
will
take
care
of
that.
A
It'll
usually
leave
your
application
on
the
same
core
or
if
that
core
gets
overloaded
by
other
tasks,
it
might
move
it
to
a
different
one.
But
when
you're
working
with
Numa
and
PCI
aware
PCIe,
aware
topologies,
you
need
better
than
that,
because
you're
typically
looking
for
very
low
latency.
So
you
don't.
Not
only
do
you
not
want
your
process
is
moving
around,
but
you
want
to
pick
a
specific
core
that
you
can.
A
So
it
we'll
give
you
some
level
of
isolation,
but
it
won't
tell
you
what
the
isolated
cores
are
in
your
pod
and
in
addition
to
that,
it's
not
hyper-threading
aware.
So,
if
you
care
about
the
difference
between
hyper-threaded
cores
and
and
non
hyper
threaded
cores,
it's
not
going
to
know
that
it
just
treats
them
all
as
cores,
and
the
last
thing
we
had
to
face
was
sharing
a
NIC
or
sharing
an
interface.
A
So
again,
this
isn't
an
issue
without
D
P,
D
K,
because
if
you're
using
standard
CNI
plugins
like
calico
or
flannel,
they
all
go
through
the
Linux
kernel
and
they
set
up
things
like
IP
tables
or
IP
vs
to
route
packets
appropriately.
But
when
you're
working
with
DB
DK
you're,
bypassing
the
kernel
entirely
so
Linux
and
IP
tables
won't
even
see
the
packets
and
also
cube
proxy,
which
was
previously
there
to
steer
the
packets
for
services,
won't
see.
The
puck
won't
see
the
packets
either
and
also,
if
you're,
using
DP
DK.
A
You
want
to
share
the
interfaces
without
losing
efficiency,
otherwise
you
wouldn't
be
using
DP
DK.
So
we
looked
at
three
different
methods
for
doing
this.
The
first
was
a
virtual
switch,
so
with
a
virtual
switch,
all
the
pods
will
communicate
from
their
network
namespace
to
the
virtual
switch
over
some
interface,
which
then
hits
the
NIC
and
two
of
the
most
common
interfaces
are
a
V
host
user,
Verdejo
user
or
benef.
So
some
of
the
virtual
switches
out
there
if
you're,
not
familiar
OVS
DP
DK
OVS
VPP,
is
another
one,
there's
probably
more
I'm
missing.
A
So
the
nice
thing
about
this
is
you
have
an
external
controller
that
programs
a
switch,
so
you
don't
actually
have
have
to
have
a
switch
that
knows
about
kubernetes
to
begin
with,
but
you
can
have
a
controller
that
knows
about
the
kubernetes
network
policies
and
things
like
that
and
program
it
after
it
started
up
the
other
nice
thing
is
it's
fairly
flexible
because
it's
its
platform
agnostic?
You
can
use
a
lot
of
these
virtual
switches
between
VMs
and
containers
and
they
don't
care
which
one
they're
running
on,
but
the
problem
is
they
can
be
inefficient.
A
So
you
have
that
extra
interface
there
that
you're
going
through
and
depending
on
which
one
you
use.
It
can
be
extremely
inefficient,
like
the
host
user,
or
it
could
be
somewhat
inefficient,
like
mem,
if
they're
always
going
to
have
some
level
of
packet
copies
before
it
gets
to
the
NIC.
But
the
the
real
problem
for
us
was.
It
doesn't
have
our
DMA
support
and
we
really
needed
that.
So,
if
you're
bypassing
the
kernel
entirely,
then
the
V
switch
wouldn't
actually
the
packets.
A
So
the
next
one
we
looked
at
was
host
device
host
device
is
a
really
simple
scene,
I
plug
in
that's
in
the
upstream
scene,
IO
repo
that
basically
just
assigns
the
entire
device
into
a
pod,
and
so
once
the
pod
consumes
the
device.
No
other
pods
can
use
that
device
and
the
kernel
actually
won't
see
the
device
either
or
the
the
root.
Namespace
won't
see
the
device
either.
So
it's
nice
because
you
can
add
multiple
queues
to
a
device
and
you
can
have
your
pod
to
use
multiple
queues.
A
But
if
you
don't
have
a
lot
of
NICs
in
your
system
and
you're
trying
to
put
more
pods
than
NICs,
this
isn't
really
practical
and
the
last
one
we
looked
at
was
SR
iove.
So
with
us,
our
iove,
each
pod
gets
one
or
more
virtual
functions
on
the
NIC,
which
is
a
concept
in
the
hardware
of
the
NIC,
and
each
of
those
virtual
functions
has
a
unique
MAC
address
and
with
most
modern
NICs,
it
has
more
virtual
functions
than
you
could
fit
pods
on
a
node.
A
So
it's
not
uncommon
to
see
512
virtual
functions
or
a
thousand
24,
and
it's
also
very
little
overhead,
because
your
pod
is
sending
directly
to
the
NIC
another
nice
feature
on.
These
is
because
it's
in
the
NIC
itself,
you
can
actually
set
up
some
of
the
Nick's
to
do
kind
of
a
switching
interface
or
a
bridging
interface
on
the
NIC.
So
you
can
have
local
traffic
from
the
same
node
without
going
out
to
the
switch.
A
It
could
be
bridged
back
into
another
pod
and
SRO
V
has
mature
kubernetes
support,
so
there's
been
a
C
and
I
plugin
for
a
long
time,
but
there's
also
now
a
device
plugin
for
the
last
year
or
so.
But
the
only
downside
that
we
found
is
that
it
may
not
have
the
same
flow
programming.
So
if
you
need
to
use
some
of
the
sophisticated
flow
programming
features
than
the
Nix,
if
you're
using
SR
io
v,
some
of
the
virtual
functions
may
not
allow
you
to
use
certain
flow
features.
A
So
with
all
that
said
now
we're
going
to
talk
about
our
topology
aware,
scheduler
and
why
we
did
it
so
from
a
high
high
from
a
high
level.
The
NHC
scheduler
is
topology
aware,
and
that
means
PCIe
Numa
hyper-threading
GPUs
data,
plane,
NICs
and
I'm,
differentiating
that
from
control,
plane,
NICs
or
the
standard
cout
Nettie's
Knicks,
because
this
when
you're
using
multiple
Knicks,
the
CPU
and
GPU
architecture
as
well.
A
So
if
you
have
different
types
of
GPUs
or
different
types
of
CPUs
in
your
cluster,
it
would
know
about
those
and
then
helper
versus
data
path
course.
So
in
most
data
path
applications
you
have
some
cores
that
are
doing
things
like
management
or
they're,
listening
to
G
RPC
request,
and
you
don't
want
them
to
be
on
isolated
cores
necessarily,
so
you
want
to
treat
them
as
more
fungible
cores
it's
written
in
Python
3
and
it
has
a
G
RPC
interface
for
retrieving
cluster
statistics.
A
So
the
main
algorithm
for
the
scheduler
is
watches
for
pods.
To
show
up
the
that
are
looking
to
be
scheduled
by
this
particular
scheduler.
Translates
the
configuration
from
the
pod
into
a
common
format
that
it
uses
to
make
a
topology
structure
matches
the
pod
to
the
node,
translates
back
to
the
pods
configuration
structure
and
then
binds
that
to
the
node
and
the
reason
why
we
do
the
config
translation
is
because
the
scheduler
has
a
concept
of
a
generic
topology
structure
and
every
application
has
their
own
way
of
configuring
it.
A
So
the
matcher
step
is
the
most
important
one,
because
that's
the
one
that
actually
does
the
packing.
So
it
tries
to
find
the
best
possible
node
to
place
your
workload,
but
it
it
also
tries
to
pack
as
many
on
there
without
being
inefficient.
So
you
wouldn't
want
to
just
put
as
many
pods
on
a
node
as
you
can
and
then
overload
the
node
so
that
you're,
causing
jitter
and
latency
that
you
don't
want
and
also
it
can't
predict
the
future.
A
So
this
is
a
pretty
common
picture
of
a
data
plane
application
where
at
the
top,
you
have
two
hyper
threaded
helper
cores,
which
are
the
ones
I
was
talking
about
earlier,
that
are
for
kind
of
management
types
of,
and
then
in
this
case
your
pod
has
to
processing
groups.
So
these
are
two
independent
data
paths
that
might
have
no
relation
to
each
other,
but
you
want
them
in
the
same
pod.
A
So
on
the
top
case,
the
dark
blue
boxes
are
the
non
hyper
threaded
CPU
cores
that
you
require,
and
then
in
that
case
you
need
three
GPUs
as
well,
which
are
the
dark
red
circles
in
the
bottom.
Processing
group
in
this
case
is
using
hyper
threaded
cores
and
then
on.
The
left
side
is
the
receive
networking
requirements,
so
you
have
15
gigabits
per
second
and
80
gigabits
per
second
and
those
and
on
the
right
side,
is
the
transmit
networking
requirements.
A
So
if
we
look
at
how
we
place
this
on
to
a
node
now,
if
we
take
one
of
those
processing
groups
from
the
bottom
and
try
to
place
it
on
this
node
in
this
particular
case,
it's
asking
for
seven
non
hyper
threaded
cores,
which
is
the
dark
blue
down
there.
Three
GPUs
15
gigabits
per
second
on
the
receive
and
50
gigabits
per
second
on
the
transmit.
So
if
we
look
at
in
this
case,
there's
only
two
servers-
it's
a
very
simplified
view,
but
the
two
servers
are
broken
up
into
two
Numa
nodes.
A
So,
on
the
left
side,
the
left
green
box
is
Numa
node
zero
on
the
first
server
and
the
right
one
is
Numa
node,
one
on
the
first
server
and
similar
on
the
second
server.
So
the
first
thing
it'll
do
is
it'll
look
at
how
many
servers
or
how
many
Numa
nodes
can
suit
the
CPU
requirements.
So
in
this
case
it's
asking
for
seven
CPUs.
So
we're
looking
for
basically
seven
white
boxes
here
and
you
can
see
both
Numa
node
1
on
server
1
and
Numa
node
0
on
server
2
can
meet
that
requirement.
A
They
both
have
7
cores
that
they
can
assign,
but
then
we're
also
asking
for
3
GPU
cores
and
now
the
second
Numa
node
on
the
first
server
doesn't
have
that
many
GPUs
free.
So
it's
not
allowed
to
schedule
there.
So
it'll
put
it
on
the
second
server
in
Numa
node
zero,
because
also
the
root
the
networking
requirements
are
met
on
the
bottom.
There.
A
But
now,
when
you
turn
hyper-threading
on,
it
gets
a
little
more
difficult,
because
now
the
cores
are
broken
up
into
two
separate
groups.
There's
two
logical
groups
of
the
top
and
bottom
they're,
so
logical,
core
zero
would
be
paired
with
logical
course,
six,
one
with
seven
and
so
on,
and
so
now,
when
it's
when
it's
scheduling
it
has
to
be
aware
of
hyper-threading.
So
in
this
case
we're
acting
asking
for
five
hyper-threaded
cores
and
two
GPUs.
A
So
if
you
look
down
at
the
bottom
and
you
look
at
which
ones
have
two
GPUs
free,
both
Numa
nodes
on
the
first
server
have
that
free,
but
only
Numa,
node
zero
on
the
second
server
has
that
free
and
now
we
look
at
the
cores
that
it's
requesting
now
we're
asking
for
five
hyper
threaded
cores.
So
it
looks
like
Numa
node
one
on
server.
One
has
that
as
well
as
Numa
node
zero
on
server
two,
but
actually
it
won't
work
because
server,
one
on
Neumann
of
one
on
server.
A
One
can't
be
scheduled
to
in
this
case,
because
if
you
look
at
the
hyper
threaded
pairs,
which
are
two
eight
one-
seven
zero,
six
and
so
on
the
sibling
Corrin
in
each
of
those
cases
is
already
used,
and
you
you
don't
place
a
workload
on
somewhere.
That
already
has
a
pod
scheduled
on
a
sibling
core,
because
now
you
have
a
pod,
that's
not
yours!
That
would
be
sharing
that
sibling
core
and
you
can't
control
it.
A
So
in
this
case
the
only
place
that
could
put
it
is
on
Numa
node
zero
on
server
two,
because,
even
though
it
shows
four
or
eight
hyper
threaded
cores
unused,
these
would
still
take
up
the
two
siblings.
So
in
this
case
it
would
put
it
on
two,
eight
five,
eleven
and
maybe
one
at
the
top,
and
it
would
use
those
five
hyper
threaded
cores.
A
So
the
last
thing
it
does
is
it
binds
the
pod
to
the
node.
So
this
is
exactly
how
the
kubernetes
scheduler
does
it?
It
issues
a
bind
command
through
the
API
server
that
basically
sets
the
node
parameter
on
the
pod
to
the
node
that
we
want
it
bound
to,
and
then
it
makes
native
kubernetes
events
that
will
tell
it
that
it's
binding.
So
if
you
look
at
cube,
CTL
get
events,
you
will
see
something
like
at
the
bottom
there
showing
that
it's
scheduling
on
to
that
node.
A
So
one
of
the
things
that
comes
up
is
why
not
use
the
upstream
topology
manager.
So
this
was
introduced
in
1.16.
The
latest
version
of
kubernetes
as
an
alpha
feature,
so
the
topology
manager
is
a
little
bit
different
in
that
it
gets
topology
hints
from
hint
providers
to
map
devices
to
other
resources.
So
one
example
that
is,
you
might
have
an
NVIDIA
GPU
device
plug-in
that
gives
you
a
hint
about
which
Numa
node
it's
on
and
then
the
topology
manager
can
then
bind
that
particular
GPU
to
a
Numa
node.
A
So
but
the
biggest
difference
is
this
is
not
part
of
the
kubernetes
scheduler.
It's
part
of
cubelet
that
runs
on
each
node,
so
it's
possible
for
your
workload
to
be
scheduled
to
a
node
by
the
scheduler
and
then
kicked
off
by
a
cubelet,
because
the
resources
aren't
matching
the
numeral
requirements
that
you
wanted,
but
it
looks
like
they're
also
adding
PCIe
awareness
soon.
So
our
hope
is
that
the
upstream
topology
Manor
manager
can
add
the
same
features
that
we've
been
working
on,
so
that
we
don't
actually
have
to
use
our
scheduler
anymore.
A
A
So
if
you
do
go
get
the
code,
you
can
build
it,
but
it
it
will
not
push
anywhere
right
now,
pretty
soon
it
will
probably
be
on
docker
hub
or
maybe
Quay,
and
then
we're
also
looking
at
inter
intel's
resource
directory
technology,
which
is
effectively
a
whole
suite
of
things
that
you
could
do
on
a
cpu
to
manage
contention
and
manage
resources.
So
one
of
the
things
that's
most
interesting
for
our
scheduler
is
you
can
manage
the
cache
and
memory
bandwidth
allocations.
A
So
as
an
example,
if
you
had
a
pod
that
you
know
might
use
two
megabytes
of
l3
cache,
you
could
make
that
a
tree
source
going
into
this
to
the
pods
resource
spec,
so
that
when
the
scheduler
sees
it,
it
would
also
use
that
and
its
decision
to
place
it
on
a
node.
So
if
you
had
another
pod
that,
for
example,
is
going
to
use
the
entire
l3
cache,
then
you
would
only
place
that
pod
on
a
node
that
had
no
other
pods
running
on
it,
and
it
also
allows
you
to
segregate
the
cache.
A
We're
also
looking
at
some
speed
improvements
right
now.
It's
it's
a
little
slow.
It's
it's!
Definitely
not
as
fast
as
the
the
upstream
scheduler.
We
didn't
intend
it
to
be
that
fast,
but
it's
it's
definitely
slower
than
it
could
be,
but
right
now
we're
running
it
on
about
30
nodes,
and
it's
fast
enough
for
that,
but
at
a
higher
scale,
with
hundreds
or
thousands
of
nodes,
it
would
be
far
too
slow
to
be
used
there.
Another
thing
we've
looked
at
is
envy
link
topology.
A
So
if
you're,
if
you're
familiar
with
GPU
servers,
the
topology
for
the
energy
inter
GPU
connections
can
vary
from
server
to
server.
Some
of
them
have
full
mesh
some
of
them
don't
some
of
them
have
more
LANs
going
to
certain
GPUs
than
others.
So
if
we
had
that
awareness
as
well,
we
could
build
that
into
the
scheduler
and
then
last
was
to
delete
the
project.
So
we
we
don't
feel
like.
A
That
answered
a
lot
of
the
questions
I'm
getting
at
this
point,
my
wife
and
kids
and
Intel
and
Red
Hat,
mostly
because
they've
done
a
ton
of
work
in
this
area
that
we're
leveraging
so,
for
example,
the
node
feature
discovery
plug
in
the
Intel
I
think
originally
made
we're
using
that
heavily
to
make
decisions
on
how
to
schedule,
and
that's
it
any
questions.
Yep.
A
In
v8
sorry
nvidia,
a
multi-process
yeah
so
yeah
that
that
wouldn't
be
hard
to
do.
We
we
have
not
had
a
need
to
share
GPUs
across
pods
right
now,
because
if
you
go
back
to
that
that
one
server
that
has
either
eight
GPUs
or
sixteen
GPUs,
usually
our
pods-
we
wouldn't
need
to
share
those
across
many
more
pods
than
that,
because
the
type
of
CPU
and
and
networking
workloads
that
we
have
wouldn't
allow
for
that
many
pods
on
that
machine
anyways.
So
we
haven't
had
a
need
for
that.
A
But
I
know
that
people
do
that,
although
once
you
once
you
start
doing
multi
process
on
GPU
you're,
starting
to
ask
for
more
latency
and
jitter
than
than
you
would
without
that,
so
it
can
be
done.
But
if
you're
going
for
the
absolute
lowest
latency
and
jitter,
you
probably
don't
want
someone
else
running
on
the
GPU
that
you
don't
know
what
they're
doing
I
believe
the
latest
NVIDIA
GPUs
of
the
v100
Elise
has
more
segregation
as
to
more
fine-tuned
granularity
as
to
breaking
up
resources
on
the
GPU,
but
I
haven't
tried
it
yet.
B
A
A
Yeah,
so
so
we're
using
the
node
feature,
discovery
tool
from
I.
Think
it's
in
the
incubator
program.
Now
that's
what
reports
all
the
properties
of
the
node
and
I
forgot
to
mention
we
we
did
make
some
hooks
onto
the
NFTE.
The
node
feature
discovery
tool,
so
things
like
isil
cpus,
weren't
reported
by
that
originally,
and
it
allows
user
hooks.
Basically,
your
own
binary
to
report
more
things
about
the
node.
So,
in
addition
to
the
default
node
feature
discovery,
we
have
a
couple
more
hooks
in
there
that
are
used
by
this.
A
A
No,
no,
we
didn't
touch
any
kubernetes
code,
so
the
scheduler
since
curdled
kubernetes
allows
custom
schedulers
by
just
setting
the
scheduler
name
field
of
the
pod
properties.
The
default
scheduler
ignores
anything
that
doesn't
have
no,
that
has
any
field
other
than
the
default
scheduler
in
there
and
then
it's
up
to
your
own
scheduler
to
watch
for
those
pods.
So
we
didn't
have
to
modify
cubelet
or
any
of
kubernetes
at
all
for
this
to
work.
A
So
this
is
effectively
a
pod
that
sits
in
your
cluster,
just
not
like
the
scheduler,
but
it's
a
it'll
sit
there
running
as
a
normal
pod.
That's
a
replica
set
that
when
another
pod
comes
online-
and
it
has
this
particular
scheduler
name
in
the
scheduler
field,
then
that
pod
watches
for
it
and
it
assigns
it
to
a
node
instead
of
the
default
scheduler.
And
what
that
also
means
is.
You
can
have
both
the
default
scheduler
and
the
scheduler
running
at
the
same
time.
A
Yeah,
so
typically,
the
way
that
we
do
it
is
is
any
node
that
is
going
to
be
used
for
high
performance.
Stuff
will
have
will
be
tainted
for
this
particular
scheduler
and
the
default
scheduler
won't
use
it
and
then
all
the
other
nodes
that
are
not
used
for
that
type
of
stuff
will
not
have
the
taint
on
it
and
they'll
they'll
use
those
nodes.
B
A
So
I
don't
know
a
whole
lot
about
slurm.
So
I
can't
really
comment
on
that.
We,
the
reason
why
we
did
this
is
we
had
quite
a
bit
of
run
time
on
kubernetes
and
we
were
really
happy
with
things
like
the
deployment
time
and
the
stability
and
since
custom
schedulers
were
something
that
we
could
do
without
upstreaming
anything
it's.
It
was
just
something
that
we
we
happen
to
do
and
we're
we're
pretty
happy
with
it.
So
we
haven't
really
had
any
major
issues.
We
did
see
a
difference.
A
I
mean
before
you
do
things
like
isolating
CPUs
before
things
like
that,
daemon
that
moved
them
to
the
isolated
set.
We
were
seeing
stalls
of
like
one
millisecond,
which
doesn't
sound
like
a
lot
well,
when
you're
doing
real-time
applications,
it's
a
lot
and
those
go
away.
When
you
do
those
kind
of
things,
yep.
B
A
Yeah,
so
the
question
is:
are
we
using
ISIL,
CPUs
and
CPU
sets?
The
answer
is
yes
and
the
reason-
and
this
may
not
be
needed,
but
I
believe
that
the
kernel
itself
will
pay
attention
to
ISIL
CPUs
for
the
most
part.
So
if
you
said
ISIL
CPUs,
most
of
the
kernel
threads
that
can
be
moved
will
be
moved
right
away.
Cpu
sets
what
we
found
was
a
lot
of
things
wouldn't
be
moved.
A
They
would
be
in
the
root
CPU
set,
which
is
wide
open,
and
so
a
lot
of
the
kernel
threads
were
still
running
on
the
isolated
or
yeah.
The
isolated
CPU
said
so
we
had
to
force
them
off
with
CPU
sets,
so
it
was.
It
was
basically
combining
those
two
and
we
looked
in
the
kernel
documentation,
there's
not
a
lot
about
how
those
two
interact
together
it
it
seemed
like
ISIL
CPUs,
just
the
legacy
thing
and
and
if
you
can
avoid
using
it,
you
should,
but
using
CPU
sets
alone,
didn't
cut
it.
For
us.
A
There
was
some.
There
are
certain
things
we
couldn't
move
by
do
by
using
just
CPU
sets.
If
you
have
any
more
information
about
the
loved
to
talk
about
it,
okay
and
by
the
way,
I
have
heard
other
people
solving
that
problem
differently.
This
is
just
the
way
that
we
happen
to
do,
because
we
didn't
know
that
they
had
solved
it
differently
until
after
we
already
did
yep.
A
So
the
question
is:
how
do
you
specify
the
CPU
cores
and
the
GPUs
so
that
cubelet
will
recognize
it?
So
in
our
case,
cubelet
doesn't
care.
So
when
you
start
up
a
pod
from
this
scheduler,
the
the
pod
is
allowed
to
see
all
the
cores
and
it's,
but
it's
only
using
the
ones
that
were
mapped
into
its
configuration.
So
if
your
configuration
says
you're
assigned
four
five
six,
the
pot
is
going
to
find
itself
two
four
five
six
and
device,
one
for
GPU,
for
example.
A
So
it's
it's
not
the
best
way
to
do,
because
you
could.
You
could
have
a
rogue
pod
that
just
starts
using
other
ones
and
that's
not
ideal,
but
right
now
we
we
haven't
really
done
any
multi-tenancy
with
kind
of
bad
actors
where
we
had
faced
that
problem.
But
cubelet
doesn't
need
to
know
about
the
cores.
A
A
So
you
don't
have
to
use
GPUs,
for
example,
even
though
they
will
work
on
PowerPC
you,
the
hyper
threading
I
think
on
PowerPC
has
like
eight
threads
per
core
or
something
like
that.
So
that
should
work.
So
I
haven't
tried
it
I
don't
have
a
PowerPC
to
try
on,
but
if
it
doesn't
work
on
PowerPC
it
would
be
a
small
change.
I
think
any
other
questions.
Yep.
Oh.
A
That's
all
it's
doing,
yeah
the
and
I'm
I'm
trying
to
remember
why
we
did
that
I
think
the
reason
was
basically
the
pit
inside
of
the
container
is
different.
I
know
you
can
set
the
host
food
to
have
the
host
pet
outside
of
it,
but
then
it
becomes
more
of
a
security
issue,
because
now
you
have
to
expose
more
things
on
the
host
of
the
pod
to
move
itself.
I,
don't
remember
if
that
was
the
exact
reason
we
did
I'd
have
to
think
about
it
more,
but
it
I
yeah,
good
question.
B
A
Yeah,
so
so
the
problem
we
found
with
the
Nvidia
device
plug-in
is
it
doesn't
allow
you
to
select
a
device,
it
gives
you
a
random
device,
that's
on
the
machine
and
that's
not
good
enough
for
the
reason
I
explained.
So
the
the
pot
itself
has
the
physical
device
ID
and
you
no
longer
need
the
device
plug-in
anymore,
because
it's
not
using
the
consumable
resource
from
the
pod
spec
anymore.
It's
using
just
the
device,
ID
and
CUDA
visible
devices.
The
environment
variable
is
basically
wide
open.
A
B
A
So
in
our
particular
case,
we
we
already
had
an
application
that
did
that
we
had
an
application
that
had
a
config
file
with
physical
resources
in
it
that
we
would
hand
edit
and
then
start
at
the
pod,
and
that
was
cumbersome,
so
that
takes
the
place
of
this.
So
if
you
didn't
have
our
application,
you
were
coming
in
with
a
different
application.
A
A
Yeah
the
application
has
to
do
the
same
thing,
but
when
you're
using
DP
DK,
that's
the
the
first
thing
you
do
when
you
initialize
DP
DK
is
passed
a
list
of
physical
cores.
So
you
have
to
know
that
anyways,
you
have
to
know
your
list
of
physical
cores
so
yeah.
The
pod
has
to
do
that,
but
instead
of
maybe
having
a
static
list
of
physical
cores,
you'll
have
one
that's
passed
in
from
outside.