►
From YouTube: Sched.net: A Network-Aware Kubernetes Scheduler [I] - Akash Gangil & Salvatore Orlando, VMware
Description
Sched.net: A Network-Aware Kubernetes Scheduler [I] - Akash Gangil & Salvatore Orlando, VMware
Different workloads have different optimal scheduling requirements. For instance, a video streaming microservice would need higher network bandwidth than a microservice running the codec. Kubernetes exposes an interface to build a custom scheduler. In this talk, we will show how better scheduling decisions can be made with information about the network topology. The scheduler would make pod scheduling decisions as a function of network health, in addition to other resources like cpu and memory predicates used by the default scheduler.
The talk would highlight:
* How kubernetes empowers the users to build their own custom scheduler and challenges that come along with it.
* Why we needed sched.net over the current default scheduler?
* Demonstrate how better scheduling decisions can be made, if the scheduler is also aware of the current network state with a simple demo described below.
Demo setup would consist of kubernetes with OVN as a networking backend using the ovn-kubernetes plugin. It provides a well defined translation between Kubernetes and OVN logical network abstractions. Ex: namespace → logical switch, pod → logical switch port. Sched.net would be implemented by adding a predicate function which would determine “network health” from information gathered from OVN controller.
About Akash Gangil
Akash is currently a Sr. Software Engineer at Uhana. Previously, he was a software engineer at Networking and Security Business Unit at VMware. He worked at enabling VMware NSX as a networking fabric for Kubernetes. He recently graduated from Georgia Tech with a focus on Computer Systems and is passionate about middleware and distributed systems.
About Salvatore Orlando
Salvatore used to be very involved with Openstack Networking. He actually implemented the first version of the Neutron API, then called Quantum. It was so good it had to be re-done from scratch. At some point he decided he made enough damage and left. After spending some time picking colors for painting bike sheds - and shaving yaks - he decided to turn his attention to Kubernetes.
However, Salvatore is not only known for his wild commits. He is also famous for his pedant reviews; they're so boring and full of pointless comments that they are being used in a clinical trial for a treatment for insomnia. Salvatore is currently employed at VMware, who joined through Nicira.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Good
evening,
everyone-
and
thanks
for
coming
so
today
in
this
session,
we
are
mostly
gonna,
demonstrate
how
you
can
implement
your
own
communities,
shed
hero
and
show
a
proof
of
concept
network
of
a
Shatila
which
we
call
it
a
shed
net.
My
name
is
Akash
Ganga
and
I'm,
a
software
engineer
and
Ohana
and.
A
So
so
during
the
next
30
minutes.
This
is
how
we
have
structure
to
talk.
Suppose
we
are
going
to
talk
about
the
default
scheduler
in
communities
they're,
going
to
show
how
it
works
and
like
what
I
its
drawbacks
and
how
you
can
improve
upon
it.
The
after
we
would
discuss
a
network
of
a
shed
yellow,
which
we
called
shed
net,
and
we
talked
a
little
bit
about
ovn.
A
So,
let's
start
off
so
I
guess
most
of
you
would
be
already
familiar
with
the
scheduler
component
in
communities.
So
scheduler
is
a
component
which
makes
decisions
on
where
pods
in
your
communities
cluster
should
be
placed.
So
so,
by
default,
like
communities
provides
a
default
Sharia
law
and
to
to
actually
see
how
it
works.
Let's
just
take
a
scenario
example.
A
So
let's
say
we
have
five
container
host
or
minions
in
our
cluster
and
as
soon
as
the
user
schedules
creates
a
pot
spec
in
the
communities
cluster.
The
Sharia
law
keeps
on
checking
with
the
API,
so
if
there
is
a
pod,
that's
unscheduled
and
by
that
what
it
means
is
like
if
there
is
a
pot
spec
resource
in
the
API
server,
for
which
the
node
name
is
not
yet
specified
at
that
point
of
time
the
Shatila
kicks
in
and
it
it
works
on.
Finding
a
continue
goes
for
that
particular
pod.
A
So,
let's
assume,
like
the
the
user,
shed
you,
the
cluster
and
the
shillelagh,
kick
then
and
like
there
were
five
nodes
in
our
cluster,
so
the
process
so
is
like
it
can
be
considered
as
a
two
step
process.
So
in
the
force
process
for
step,
what
the
shed
Euler
does
it
passes
applies
certain
predicate
functions
to
all
the
nodes
in
the
cluster
so
and
these
predicate
functions
are
basically
used
as
a
filtering
mechanism
as
to
whether
that
node
is
fit
to
run
that
container
or
not.
A
So
we
run
that
list
of
predicate
functions
on
all
the
nodes
in
the
cluster
and
thereafter
we
are.
We
get
a
filtered
list
of
nodes
so
like
in
this
case,
for
example,
like
we
applied
the
predicate
functions
on
nodes
n,
1
to
n
5,
and
the
scheduler
found
that
only
n,
2
n
+
4
satisfied
whatever
the
pod
criterias
was
so
this.
This
is
the
for
step
in
in
the
scheduling
process,
and
the
next
step
is,
as
can
be,
guessed
is
deciding
which,
among
these
nodes
is
a
benefit.
So
for
that
the
share.
A
Euler
path
applies.
Certain
priority
functions
to
these
given
set
of
nodes
and
by
priority
functions.
It's
pretty
much
a
way
of
quantitative,
quantitatively,
saying
that
which
node
is
better
suited
for
that
particular
container.
So
they
are.
There
are
a
bunch
of
predicate
functions
and
what
each
of
the
bunch
of
priority
functions
and
what
each
of
them
returns
is
a
it's
an
integer
value
and
ultimately,
for
each
of
the
nodes.
A
We
take
a
weighted
weighted
value
for
by
running
it
over
all
the
priority
functions,
and
then
we
just
sort
the
weighted
score
of
all
the
nodes
and
we
take
the
topmost
node.
So,
in
this
case,
the
Shetler
applied
those
priority
functions
on
n,
2
and
n
4,
and
it
found
that
info
is
more
suitable
for
for
scheduling
that
particular
container.
So
I
think
it's
it's
pretty
simple
and
straightforward.
What
what
it's
trying
to
do
to
get
a
little
more
like
idea
of
what
the
default
or
what
the
predicate
and
priority
functions
look
like.
A
So
this
check
make
make
sure
that
if
say,
you
have
specified
host
port,
80
or
90
in
your
pot
spec
that
port
is
available
in
the
node
and
if
it's
not,
then
that
node
is
not
suitable
for
sharing
the
pot.
The
other
one
is
pod,
fed
resources,
so
by
default
in
terms
of
resource,
so
measurements
cubing,
like
the
scheduler,
is
only
taking
into
account
the
CPU
and
memory
usage
of
each
of
the
nodes
in
the
cluster,
and
this
is
reported
by
the
cubelet
back
to
the
api.
So
and
recently
there
is
some
support.
A
So
these
are
just
three
of
them.
So
I
guess
they
are
a
total
of
seven
or
eight
predicate
functions
or
which
are
specified
by
default
in
the
scheduler.
Next,
moving
on
to
like
what
the
priority
functions.
Look
like
so
these
are
the
ones
which
are
applied
in
the
second
stage
of
the
sharing
process,
so
like
the
first
one
could
be,
the
Selective
spread,
the
name
it
might
be
confusing
for
some.
So
what
it
essentially
means
is
when
we
are
trying
to
deploy
a
deployment
or
application
controller
or
service
in
our
cluster.
A
The
scheduler
wants
to
make
sure
that
all
the
pods
or
continuous
backing,
that
particular
abstraction,
like
the
service
or
the
deployment,
are
not
concentrated
to
a
single
node
in
your
cluster.
So
this
this
essentially
checks
to
spread
the
pod
among
all
the
nodes
in
your
cluster.
So
if
a
node
for
that
particular
deployment
is
already
on
a
node,
that
node
would
be
given
a
lower
priority
or
low
score
when
the
other
pods
for
that
service
are
to
be
scheduled.
Similarly,
we
have
image
locality.
A
This
is
pretty
self
that
understandable,
like
if
a
node
already
has
the
container
image,
then
obviously
it's
given
a
higher
priority
least
requested
nodes.
It's
so
again,
like
you
want
to
share
resources
pods
on
nodes
which
are
underutilized
to
have
a
more
uniform
distribution
and
similarly
node
labels.
So
node
labels
differs
from
node
selectors
in
the
way
that
selector
is
more
like
it's.
A
more
harder
condition
was
a
node
label.
A
A
A
Ultimately,
like
the
clusters
on
which
we
deploy
our
systems
are,
they
are
heterogeneous.
Not
all
nodes
are
similar,
like
some
might
have
better
hardware
than
the
others,
so
that
could
be
that
either
the
users
in
specifying
node
selectors,
but
that's
something,
that's
not
it
taking
into
account
by
scheduler
itself
and
the
third
one
is,
as
we
talked
about
the
weights
in
the
program
in
the
second
step
of
prioritizing
the
nodes
right
now,
the
default
implementation.
A
The
way
it
is
right
now
it
takes
all
the
priority
functions
equally
like
if
everything
is
weighted
as
one
so
like.
Those
are
some
of
the
things
that,
like
the
default
scheduler,
is
not
really
handling
and
can
be
improved
on,
but
the
good
part
about
is
like
community
sharia
law
provides
a
lot
of
folks
way.
You
can
plug
your
own
scheduler
or
like
make
changes
to
the
custom
said
you
know
so
I
mean
natural
inclination
like
for
anyone
is
like
if
they
are
running
your
cluster
and
how
they
can
do
a
better
job.
A
So
foresters,
like
then,
when
we
are
prioritizing
the
nodes,
we
think
of
we
think
of
prioritization
as
more
in
a
quality
of
service
sense
and
like
quality
of
service,
can
can
be
pretty
broadly
classified.
Like
you
could
have
networking,
storage
and
all
the
things
that
might
have
back
impact
the
like
the
performance
of
a
application.
A
So
if
we
take
that
into
account,
while
prioritizing
knows
that
could
be
a
good
metric,
a
similar
one
could
be
like
which
we
also
tackle
in
a
proof-of-concept
shed,
you
know,
is
we
try
to
balance
the
bandwidth
usage
of
all
the
nodes
in
your
cluster
so
like?
If
there
are
some
applications
which
are
very
network
intensive
and
some
of
those
are
not,
then
those
those
application
should
be
uniformly
distributed
in
your
cluster.
A
So
in
this
case
the
applications
which
are
running
in
the
pods,
and
then
you
can
also
like
try
to
play
around
with
the
rates
that
the
default
schedule
already
provides
us
so
so
now
to
just
show
like
how
we
can
improve
it.
We
walked
on
the
shady
local
shed
net
and
I
guess.
Salvatore
is
gonna
talk
about
some
of
the
details.
Well,.
B
Actually,
this
Mike
works,
so
you
can
keep
your
you
can
keep
your
position
there,
so
we
start
with
ovn
open,
which
is
an
acronym
which
stands
for
open
withdrawn
network,
which
is
the
network
backends
that
are
using
in
this
POC
that
are
demonstrating
today
year.
So
anyone
is
already
familiar
with
this
particular
network
backend.
B
Well,
apart
from
you
guys,
I
know
you
well
anyone
that
has
never
heard
of
these
like
this
is
the
first
time
in
my
life,
the
tired
of
this
ok,
so
I
can't
assume
that
you
word
of
it
and
then
you
discount
it
as
something
terrible
which
I
would
understand,
but
I
will
try
and
ask
you
to
reconsider
your
opinion,
because
it's
actually
quite
good
I
mean
I,
am
partial,
but
I
think
it
is
actually
quite
good.
It
is
entirely
open
source.
B
It
is
a
developed
by
the
open
with
which
community
and
if
you
want
to
look
at
the
list
of
contributors,
of
course,
there's
vmware,
but
we
also.
There
is
also
a
lot
of
contributors
from
reddit,
HP
Intel
and
a
lot
of
companies
that
I'm,
probably
forgetting
about
and
in
terms
of
more
substantial
things.
It
creates
management
of
allow
us
to
create
a
managed
virtual
network,
topologies
layer,
two
layer,
three
topologies,
and
it
also
provides
security
to
stateful
apples
and
it
has
a
buffer
layer
cell
layer
four.
B
That's
on
that
link
trying
out
the
integration
is
very
easy,
even
if,
unfortunately,
we
are
not
yet
able
to
provide
a
cube,
oedema
addon,
but
at
least
it
should
not
take
more
than
an
hour,
maybe
half
an
hour
to
set
up
a
simple
cluster.
We
do
be
an
integration
anyway.
Opn
is
not
really
the
focus
of
our
talk
today.
So
let's
look
at
our
experimental
test
bed.
So
in
this
test
bed
in
the
picture
that
you
see
there,
it
would
be
nice
to
have
one
of
those
pointy
things.
B
Otherwise,
I
have
designs
around
the
screen
like
this
and
go
on
this
screen.
We
have
our
topology,
which
is
made
of
three
kinds
of
components
in
green.
You
have
the
ovn
components
in
blue.
There
are
kubernetes
components
and
the
pink
components
are
the
cat
dotnet
elements
that
akaash
we'll
talk
about
briefly.
So,
let's
start
with
UV
n
bits
on
every
node,
where
minions
were
containers
are
running.
We
got
a
controller
component,
the
controller
implements
the
control
plane
and
manages
the
open
with
which
data
plane
instance
is
locally,
so
it
held.
B
It
only
hold
state
for
managing
virtual
network
interfaces
on
the
local
node.
So,
behind
this
controller
is,
of
course,
an
open
with
switch
Easton's
and
on,
though,
that
open
with
which
instance,
we
configure
the
node
local
interface,
and
then
these
controllers
are
interact
with
that
ouvir
and
north
digimon
that
you
can
see
here,
which
is
a
sort
of
a
northbound
driver.
It
exposes
indeed
a
northbound
API
for
management
planes
like
OpenStack
or
in
our
case
kubernetes,
and
that
component
yogi
nks
watcher
is
what
makes
the
integration
between
the
kubernetes
api
server
and
ovn
using
ovn.
B
We
build
this
logical
topology,
which
is
made
of
one
logical
switch
per
node,
which
are
interconnected:
biological
router,
of
course,
when
we
say
logical
routers,
this
is
just
a
an
abstract
concept:
there's
not
really
routing
occurring,
it's
a
distributed
router.
So
we
never
know.
If
you
have
two
containers
on
the
same
node,
you
never
go
to
this
thing,
so
it's
all
distributed
and
for
implementing
the
Kuban
service
abstraction.
B
It
doesn't
leverage
the
cube
proxy
component,
but
instead
it
leverages
ovn
layer
for
load,
balancing
capability
to
implement,
east-west
distributed
load,
balancer
and
therefore
to
load
balancer
instances,
one
for
TCP
services
and
one
for
UDP
services
are
associated
to
each
logical
switch.
So,
of
course,
this
is
a
very
simple
logical
topology,
but
it
enables
us
to
fulfill
all
all
the
kubernetes
use
cases,
at
least
in
this
phase,
where
we
don't
have
multi-tenancy
multiple
network
support,
when
these
will
be
adopted
by
the
kubernetes
Signet
and
incorporated
into
kubernetes.
B
Of
course,
this
topology
will
change
a
little
bit
so
and
I
think
that
this
is
all
for
the
ovn
side.
We
describe
the
components
and
topological
topology,
as
you
can
see
the
kubernetes
side.
We
have
the
usual
components:
the
API
server,
the
controller
manager.
As
we
said,
we
don't
have
the
cube
proxy
on
every
node
and
the
cubelet
is
running.
Do
VNC
ni
plug-in,
which,
unlike
other
CNI
plugins,
is
a
very
simple
component
that
just
configures
the
local
network
interfaces
and
plugs
it
into
the
OBS
Bridge
instance.
B
B
A
B
Node
anyway,
we
have
one
agent
that
collects
network
health
status,
which
we
Akash.
We
see
with
Akash
exactly
later,
what
this
natural
state
says,
but
it's
suffice
to
say
that
it
is
an
estimate
of
the
true
hood
currently
being
used
on
every
on
every
node
and
periodically.
This
nitrogen
agent
sends
Network
state
data
to
the
kubernetes
master.
Worse
cat
dotnet
is
running
and
now
scat
dotnet
is
not
really
a
replacement
for
the
kubernetes
scheduler.
B
It
is
more
correct
to
say
correct
me
if
I'm
wrong
and
announcement
of
the
kubernetes
scheduler,
so
it
is
still
the
same
kubernetes
scheduler
to
which
we
added
something
for
making
it
network
on
our
network.
Aware
so
I
think
this
concludes
the
discussion
of
the
testbed
yeah.
We
can
go
ahead
and
see
sure.
A
A
So
I
think
it's
pretty
simple
and
one
doesn't
need
to
understand.
So
how
would
I
actually
like
work
so
in
each
of
the
minions?
The
network
health
agent
is
is
collecting
this,
this
network
health
score
and
updating
the
node
annotation
every
one
second
to
the
API.
So
so
now,
whenever
a
part
is
scheduled
and
the
schedule
and
notices
the
need
for
it.
A
So
it
gets
to
know
that
the
schedule
in
the
scheduling
function
kicks
in
and
what
it
actually
does
is
it
reads
the
network's
code
for
all
the
nodes
in
the
cluster
and
based
on
that
particular
information.
It
makes
the
decision
on
which
node
should
the
pod
get
scheduled
on
so,
as
can
be
seen
by
three.
A
But
it's
not
something
that
so
that's
like
a
design
like
it's
just
based
on
preference
like
you
can
use
whatever
you
want,
and
secondly,
the
master
is
also
running
the
ovn
watcher
that
we
just
discussed
during
the
ovn
slides
and
the
minions
are
running
the
network
health
agent,
as
you
already
saw
her.
So
it's
posting
this
patch
to
the
api
server
and
is
alter
and
running
the
couplet
as
well.
So
that's
what
characterises
does
a
minion
so
in
the
demo.
A
A
So
one
of
the
pods,
which
is
the
engine
x5
quad,
it's
pinging
packet
sizes
of,
say,
55,
KB
to
a
destination
and
this
IP
address
is
actually
an
interface
on
the
master
node
Oh,
which
is
which
is
set
up
during
the
movie
in
topology.
And
similarly
we
have
other
nodes,
other
pods
in
our
setup,
and
they
are
there.
They
they
are
pinging
packets
of
less
sizes.
So
what
what
we
want
to
show
is
so
these
are
the
other
set
of
ports
which
are
actually
not
that
network
intensive.
A
So
we
have
engine
X
2,
then
engine
X,
3
and
similarly
for
so
engine
X,
3,
spinning,
packets
of
sizes,
3
KB
and
engine
X
for
spinning
packets
of
sizes,
30
KB.
So
now,
if
we
look
at
it,
since
we
have
the
agents
already
running
on
the
on
the
minions,
so
if
we
check
the
annotations,
the
node
annotations,
they
already
have
scored
so
I
think.
A
So,
like
the
node,
one
has
a
network,
health
score
of,
say,
0.74
and
the
node
two
has
a
network
health
score
of
one
point:
seven
two
and
the
reason
it's
not
zero
is
because
ovn
two
to
make
sure
that
tunnels
are.
It
has
some
gossip
traffic
which
actually
makes
sure
that
things
are
working
properly.
So
that's
why
right
now
you're
seeing
that
it's
it's
not
zero.
Although
there's
no
parts
in
our
cluster,
so
now
we
would
go
ahead
and
deploy
the
pods
in
the
setup.
A
So
one
thing
that
I
think
any
most
of
you,
since
you
have
played
with
communities
like
by
default
communities,
would
just
showed
you
the
pods
like
two
on
each
of
the
nodes,
but
in
this
case
hopefully
like
the
pod
with
the
high
network
traffic
is
scheduled
on
one
node
and
the
subsequent
pods
are
scheduled
on
another
node
to
make
the
load
more
uniform.
So
let's
just
go
ahead
with
that.
So.
A
A
So,
as
you
can
see
like
the
rest
of
the
pods,
were
scheduled
on
different
node
and
not
on
the
node,
where
engineers
fight.
What
should
you
and
the
reason
being
is,
as
these
pods
were
getting
shuttled,
the
network
agents
on
the
minions
were
reporting
the
metric
or
the
network
health
score
as
as
these
pods
were
getting
shoddy,
and
because
of
that,
the
scheduler
took
those
health
scores
into
it
scheduling
decision
and
it
plays
the
remaining
pods
on
other
nodes.
A
So
if
we
just
see
that,
then
it
work
health
scores
now,
so
we
tried
to
balance
so
so
126
health
score
is
given
to
the
poor,
with
the
note
on
which
Ingenix
fiber
scheduled
and
the
other
one
is
six
six
point:
zero.
Nine.
The
way
this
this
computation
is
done
is
just
a
1
by
logarithmic
throughput
and
it's
scaled
by
a
factor
of
100,
which
you
could
also
consider
as
a
as
a
weight
of
that
priority
function.
So
I
think
this
is
pretty
much
what
what
we,
what
we
wanted
to
show
that
you
can
implement.
A
You
can
customize
the
default
scheduler
such
that
to
able
to
better
meet
your
needs
and
which,
which
could
range
beyond
the
general
CPU
and
memory
resources
which
the
scheduler
considers.
So
what
we
essentially
saw
that
we
had
for
odd
specs,
which
we
wanted
to
do
in
case
of
default,
said
you
know
they
shed
you
more
evenly
like
to
pause
on
each
node
and
in
which
case
the
network
load
is
not
really
disputed,
while
in
case
you're
in
case
your
scheduler
takes
the
network
load
into
consideration,
then
it's
more
uniform.
A
So
now
there
could
be
multiple
approaches
in
which
you
can
do
this
I
guess
there
was
another
talk
on
data.
Where
should
you
reign
where
they
implemented
their
own
shed?
You
know,
so
you
can
also
go
ahead
with
implementing
your
own
scheduler
completely
outside
so
and
what
that
would
entail
is
you
are
not
touching
default
schedule.
You
deploy
a
moon
shell,
you
learn.
Whenever
you
are
deploying
a
part
in
the
pod
spec,
you
can
specify
the
scheduler
name.
A
So
that's
another
another
way
in
which
you
can
specify
a
custom
shade
below
or
you
can
use
the
extender
interface.
So
Kuban
d
schedule
a
by
default,
provides
us
another
plug
it.
Pluggable
mechanism
called
extended
interface
and,
and
under
this
it
allows
you
to
define
rest
endpoints
where
you
can
define
your
predicate
and
priority
functions,
but
I
guess.
The
downside
of
this
is
performance
in
some
of
the
cases
like
the
shading
time
is
much
greater
than
what
you
would
expect
Oh.
A
So
why
would
you
actually
even
care
about
all
this
stuff?
So,
like
you,
you
would
want
to
share
your
pods
based
on
more
data
and
more
context
in
new
topology,
so
I'm
not
sure
like
how
many
people
use
like
software-defined
infrastructure,
but
you
can.
You
can
use
in
terms
of
network
and
in
terms
of
storage,
your
infrastructure
could
entirely
be
in
software,
and
what
that
allows
use
is
is
the
programmable
nature
of
that
infrastructure
allows
you
to
leverage
the
add
information
to
a
ruling
of
the
nodes.
A
Then
again,
you
have
a
heterogeneous
set
of
applications,
as
I
mentioned
earlier,
like
not
all
applications
are
equal
and
there
is
no
one-size-fits-all
solution
for
everything
and
like
it's,
pretty
cool
like
cuny's
allows
you
with
so
much
ease
that
you
can
experiment
with
what
you
want
to
do.
So
this
also
want,
like
also
aimed
at
highlighting
that
oh
yeah
I,
guess
that's
mostly
it,
and
if
there
any
questions,
I'm
happy
to
take
them
yeah.