►
From YouTube: Efficient Scheduling Of High Performance Batch Computing For... Krzysztof Adamski & Tinco Boekestijn
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Efficient Scheduling Of High Performance Batch Computing For Analytics Workloads With Volcano - Krzysztof Adamski & Tinco Boekestijn, ING
Three years ago ING Wholesale Banking Advanced Analytics team set up an ambitious goal to gather in one place a curated portfolio of internal data sources together with a large scale compute platform. At its core the idea of allowing internal projects to get access to a rich toolset of open source and industry standards frameworks and preprocessed data to validate business ideas in the secure exploration environment. Extensive growth with over 300 internal projects so far and more than 2000 internal users proofs advanced analytics i.e. ML, AI, NLP capabilities should become easily consumable not only by specialized, dedicated teams, but make them close to subject matter experts. In this session we would like to shed more light on how a specialized cloud native Kubernetes scheduler (Volcano) enables us to deliver multi-tenant large scale processing capabilities. The optimal resource usage with stability of core services are key for our cloud native platform. To enable dynamic allocation and hdrf (hierarchical dominant resource fairness) we have created an extension to Apache Spark binaries. This allows users to use Volcano with Spark interactive mode in a Jupyter notebook. Additionally we have created interfaces to visualize all the scheduling metrics like the yarn ui.
A
Hello
welcome
to
our
talk.
My
name
is
Chris
and,
together
with
Tinker
we're
gonna.
Take
you
on
a
short
Journey,
how
we
adapted
Cloud
native
tools
to
build
our
data
analytics
platform
and
how
the
recent
addition
to
that
stack
with
those
Advanced
schedulers
help
us
to
you
know
achieve
a
bigger
scale
or
operate
in
that
multi-tenant
environment.
A
So
that's
going
to
be
our
story,
but
just
to
begin
with
we're
going
to
introduce
to
you
to
our
company.
So
ing
is
a
financial
institution
that
operates
globally
with
European
Europe
I
have
over
52
000
employees.
Working
for
us
yeah
with
our
mission
is
to
empower
people
to
stay
ahead
in
life
and
business,
and
also
the
truths
of
the
company
are
from
the
deep
economic
crisis
of
the
20s
of
the
20th
century.
A
Our
mission
right
now
is
also
to
help
to
transition
our
world
into
the
new
era,
with
to
shift
in
a
low
carbon
future
and
to
speed
up
the
innovation
in
finance
being
like
over
10
years.
In
that
banking
industry
is
also
a
fascinating
area
where
a
lot
of
regulations
are
there.
So
it's
adopting
a
new
technology.
A
The
mission
for
the
company,
but
the
mission
for
our
platform
is
to
become
a
data
driven.
What
we
want
to
emphasize
here
is
like
support
our
employees
with
the
self-service
platform,
so
we
are
in
a
platform
economy
and
what
what
we
built?
The
mission
is
to
empower
you
to
build
on
top
of
that
platform
and
to
solve
the
business
needs
the
growth
factor,
so
we
started
around
2018.
The
whole
platform
rules
are
in
2013
when
we
started
adopting
open
source
technology
to
bootstrap
the
product
initiatives
in
2018.
A
We
are
starting
to
look
in
the
new
version
of
the
platform
when,
finally,
we
see
that
the
transition
towards
the
cloud
native
tools
help
us
to
to
build
the
platform
with
the
new
foundation
on
the
infrastructure
layer.
The
numbers
may
be
not
impressive,
but
we
have
a
stable
growth
since
then,
and
the
adoption
rate
is
is
growing
and
over
400
projects
exist
on
our
data
index
platform,
so
that
self-service
Paradigm
really
helped
with
the
adoption
rate.
A
What
we,
what
we
built
is
essentially
a
product
of
the
product
driven
mindset
is
something
that
is
embedded
in
our
in
our
mindset
using
the
model
open
source
technology.
What
you
see
here
is
like
an
entry
point
to
our
platform
that
allows
users
to
use
the
significant
compute
power
with
the
some
security
and
compliance
configuration
embedded
inside,
supporting
not
only
Global
but
also
local
needs
of
our
users.
A
Three
Foundation
pillars
for
the
self-service
platform
is
scalability,
seamless
and
security,
and
we
wanted
to
emphasize
on
that
engineering
capabilities
of
our
platform
that
Engineers
can
start
their
data
Antics
Journey,
using
predefined
pipelines,
building
sharing
testing
the
deployment
and
create
create
new
insights
for
the
business
data
is
stored
in
a
secure
way.
We
what's
one
of
the
most
important
factor
of
the
platform
is
that
the
sharing
resources
sharing
data
Resources
with
the
products
is
based
on
it's
based
on
predefined
roles.
A
The
tool
set.
It's
is
then
also
delivered
for
you
know
the
latest
tools
that
it
is
then
available
for
the
users
we're
looking
for
the
ultimate
answer
and
in
my
perfect
way
the
ultimate
answer
would
be
like
a
search
box
when
the
users
would
like
to
type
whatever
I
want
to
do
next
and
everything
is
going
to
be
taken
care
for
the
users
and
we
will
guide
you
through
that
user
Journey
on
the
next
steps.
Director
is
a
little
bit
complex.
A
A
On
that
Journey
we
prepare
a
couple
of
interfaces
to
our
users
can
use
on
the
platform
what
we
saw
in
that
previous
picture,
the
landing
page.
So
the
latest
like
the
starting
point
for
the
users
when
they
can
choose
the
from
the
portfolio
of
the
platform
the
product
we
have
our
opinionated
data
science
toolbox.
That
is
obviously
with
the
Jupiter
notebook
Enterprise.
A
We
recently
switched
from
our
custom-made
built
environment
towards
build
packs
that
helps
us
to
build
to
build
this
environment
with
data
Discovery
reported.
The
cooperation
with
with
Lyft
folks
help
us
to
enable
the
data
Discovery
and
metadata
engine.
On
top
of
that
platform,
we
have
a
superset
for
our
bi
and
analytics
needs.
A
The
recent
transition
that
happened
for
us
was
that
that's
something
that
made
it
possible
is
to
transition
to
our
Cloud
native
tools.
So
we
swap
we
switched
from
the
Hadoop
ecosystems
towards
object,
storage
towards
kubernetes,
with
the
addition
of
the
caching
layer.
Something
that's
also
some
talks
were
here
yesterday,
helps
us
to
achieve
proper
performance
on
top
of
the
those
object,
storage,
implementation.
A
The
key
challenges
that
we
saw
so
far
with
the
of
the
cloud
native
tools,
as
as
you
can
see
the
job,
job,
management,
scheduling
and
multi
framework
support.
Still
Hadoop
was
there
for
us.
Despite
you
know,
leveraging
Cloud
native
Technologies
like
kubernetes
for
stateless
application,
it
was
even
possible
to
to
set
up
stateful
application,
but
still
for
data
analytics
workloads,
we're
missing
a
major
schedule.
Mature
scheduler,
like
yarn,
is
to
enable
those
jobs
running
concurrently
with
the
multi-tenant
environment.
A
What's
what's,
the
yarn
was
missing,
on
the
other
hand,
was
support
for
those
new
Frameworks
that
are
happening
like
tensorflow
pytorch.
So
that's
something
that
we're
mostly
also
looking
for.
What's
going
to
be
next
next
big
big
thing
for
our
platform,
with
that,
that's
a
short
transition
towards
volcano
and
we'll
hand
it
over
to
tinko.
To
give
you
a
little
bit
more
details
about
the
our
implementation.
B
Okay,
so
hello,
everyone
on
the
trail
to
Forest
solution,
so
yeah
Chris,
just
told
us
a
little
bit
about
it.
I
hope
you
can
take
this
image
in
mind.
For
us,
on-premise
means
that
we
have
a
fixed
cluster
available
and
on
our
previous
cluster,
we
had
this.
We
had
a
scheduling
which
was
split
between
Hadoop
and
kubernetes.
B
So
here,
if
you
look
at
from
a
resource
consumption,
you
could
see
that
if
we
have
a
certain
split
in
workloads,
kubernetes
and
Hadoop-
and
we
then
in
during
normal
office
hours,
you
might
see
that
when
the
kubernetes
were
actually
like
using
somewhat
but
not
everything
on
the
Hadoop
part,
we're
definitely
since
Park
is
wants
to
try
to
use
as
many
resources
possible.
We
try.
B
You
might
see
that
this
is
not
very
an
optimal
way
of
allocating,
and
what's
also
is
important
to
us,
is
that
we
have
a
distinction
between
batch
and
user,
so
interactive
sites
are
the
people
that
are
using.
The
platform
immediately
batch
are
like
the
big
processes
that
we
have
to
finish
on
time,
so
to
say
for
business
critical
applications.
B
There
are
different
many
approaches,
but
essentially
we
need
a
job
scheduler
for
kubernetes,
specifically
currently
for
spark,
but
in
the
future,
for
different
kind
of
Technologies,
and
what
volcano
offers
is
that
it
has
job
cues
with
weighted
priorities.
It
has
the
ability
that
you
can
essentially
cues
are
like
the
ways
how
you
divide
the
cluster
up.
So
if
there
are
four
users
running
on
the
cluster,
then
everybody
gets
one
fourth
of
the
cluster
to
use,
so
their
job
can
complete
as
fast
as
possible,
but
also
the
ability
has
has
to
commit
above
Q
limits.
B
So,
for
example,
if
two
users
aren't
using
anything,
then
other
users
may
use
their
resources.
So
it's
like
a
system
that
you
try
to
keep
try
to
claim
as
many
resources
as
possible,
and
it
also
has
the
main
ability
to
preempt
spots
when
more
ports
can
come
in
and
the.
Lastly,
it
also
has
configurable
strategies
to
deal
with
competing
workloads,
for
example
from
a
task
scheduling
perspective,
for
example.
Spark
it
is,
it
is
I
say
that
you
can
preempt
machines,
you
can
kill
machines
and
then
the
job
will
still
Computing
you
for
tensorflow.
B
B
B
So
we
as
we
see
it,
you
just
you,
have
server
scheduling
and
you
have
task
scheduling
and
all
different
kind
of
tasks
which
you
like
have
a
certain
predefined
moment
that
it
will
stop.
That's
like
tensorflow
by
torch,
spark
and
kubeflow,
and
you
can
think
of
any
other
application.
On
top
of
it,
and
essentially
the
the
configuration
is
I,
wouldn't
say
simple,
but
I
won't
say
easy,
but
it's
more
like
it
tries
to
be
as
as
simple
as
possible.
B
If
certain
job
bot
is
doesn't
work,
then
it
should
restart
like
five
times,
and
it
has
this
pluginable
architecture
in
which
you
can
select.
Okay,
I
want
to
use
different
kind
of
plugins
and
there
are
different
kind
of
actions
that
can
happen
within
volcano
itself
and
then
also
you
have
this
Q
part,
which
is
specifically
needed
for
our
spark
setup,
which
I
will
demonstrate
later.
B
So
if
you
would
look
at
the
volcano,
then
also
it
actually
encompasses
out
of
three
different
Services.
You
have
admission
service
which
will
check
if
everything
is
correct,
then
also
you
have
the
controller
manager,
and
then
you
have
the
sketchner
and
the
scheduler
is
the
main
one:
making
decisions
where
to
allocate
the
task
Etc.
B
A
B
But
this
is
not
the
thing
that
actually
with
for
like
high
performance
and
jobs
that
you
want
to
do
essentially
want
to
put
the
jobs
as
close
as
possible,
so
you
have
less
Network
traffic
and
so
that
it
can
complete
faster,
but
also
you
might
want
to
have
that.
For
example,
the
applications
of
all
different
users
are
spread
as
much
as
possible
so
that
you
don't
get
like
incompeting
workloads
on
one
nodes.
B
So
what
you
here
can
see
is
that
for
this
calculate,
while
in
kubernetes,
it's
mostly
like
by
spreading
all
the
balls,
is
that
4K
now
actually
has
many
different
plugins
attached
to
it.
So
you
can
have
like
drf,
which
is
the
dominant
resource
fairness
algorithm,
which
I
will
get
into
you
can
have
gang
scheduling
where
you
say,
Okay
I
want
to
only
deploy
all
the
four
pots
when
all
the
resources
like
are
available
for
all
four
parts.
So
that's
like
no
bolt
is
hanging.
B
I
want
to
add
priorities,
I
want
to
add
resource
quotas,
I
want
to
add
SLA.
So,
for
example,
I
want
to
check
whether
certain
pulse,
if
there
isn't
available
space,
that
it
doesn't
take
too
long.
So
there
are
many
different
kind
of
algorithms
you
could
think
of
when
you
approaches
from
a
task
scheduling-
and
this
might
mean
this
means
for
us
that
we
can
further
optimize
the
traffic
of
spark
in
the
kubernetes
cluster.
B
B
So
this
is
the
part
of
like
you
have
a
weighted
claim,
and
this
is
done
in
four
kind
of
use.
Excuse
and
you
can
over
commit
an
unused
resource
from
another
process
and
calculations,
the
dominant
resource
fairness,
so
the
the
calculations
are
done
on
the
most
dominant
research
which
is
being
used.
So
in
this
case
it's
CPU,
because
memory
is
used
way
less
and
there's
even
some
available
space.
B
B
So
if
we
would
go
to
a
stack
up,
then
there's
also
the
part
of
resource
starvation,
for
example,
if
you
have
two
notes
to
your
available,
then
you
want
to
have
a
long
running
surfaces,
for
example
with
us
that
is
like
Presto
tree
now,
or
we
have
a
cache
like
a
luxia
which
are
like
we
run
on
every
surface,
so
we
have
a
also
different
kind
of
compute
options,
and
also
we
have
a
caching
layer.
So
we
can
access
data
as
fast
as
possible.
B
That
means
that,
with
the
rest
of
the
space
is
essentially
available
for
tasks
but
yeah.
We
also
if
we
use
everything
for
tasks,
that
might
mean
that
we
cannot
do
any
deployments
anymore
or
we
cannot
make
any
changes.
So,
for
example,
we
added
some
available
available
parts
to
it
so
that
you
can
have
always
have
some
spare
capacity
in
the
cluster
so
that
we
can
do
deployments
and
without
any
issues.
B
B
Then
you
can
Define.
Okay,
I
want
to
use
root
user,
one
that
is
my
Q
name
and
then
I
want
to
use
this
group
name.
This
bot
group,
where
all
these
Bots
are
I,
say
that
I
said
that
this
abstraction
over
it
spark
itself
will
create
this
queues
and
both
both
groups
automatically
within
our
setup
and
then
the
startup
balls
are
automatically
assigned
to
the
volcano
Port
group
that
you
have
declared.
B
We
have
owner
references
and
drive
for
heartbeat
for
garbage
collection.
So
that
means
that
if
somebody's
spark
session
spark
driver
will
stop,
then
the
executors
will
go
down
itself.
This
is,
this
is
how
it's
done
in
the
spark,
and
then
we
have
Mark
spending
Bots
an
option
to
limit
the
amount
of
allocation.
B
So
spark
is
trying
to
ask
as
many
resources
from
the
full
kind
of
scheduler
as
possible,
and
if,
if
you
limited
the
bias,
then
it
will
ask
for
less
and
less
I
said
yeah,
but
the
main
part
is
is
that
we
have
Dynamic
allocation
being
used.
So
if
you're
running
for,
for
example,
now
for
the
user,
like
the
main
resource
requirements,
are
as
much
I
say
that
hidden
away
from
from
them
as
possible,
they
just
get
the
spark
both
in
which
they
can
run
the
query.
B
B
Then
it
comes
to
a
small
demo
which
I
wanted
to
show
here
comes.
The
part
which
I
will
say
is
that
we
work
in
a
highly
regulated
bank,
so
I
cannot
show
you
our
kubernetes
commands
Etc
so
because
there
might
be
some
sensitive
information
lying
around,
but
I
can
show
you,
for
example,
now
the
grafana
dashboards,
which
I
have
updated
a
little
bit
in
which
we
run
Spark
so
and
this
I
think,
will
immediately
make
it
clear
for
you
guys
how
it
works.
B
B
If
suddenly,
a
second
user
is
being
added.
That
means
that
automatically
that's
first
user
is
being
downscaled,
so
like
his
podcast
skills
or
killed
off
and
make
room
for
the
second
user
to
come
in
and
then
that
will
balance
itself
again
and
then
you
have
both
have
17
executors
to
your
to
their
high
side
to
the
leverage
and
the
their
job
is
like
still
running.
A
This
situation,
you
have
like
a
static
allocation
of
the
cluster,
but
then
you
manipulate
how
many
resources
particular
job
or
user
gets.
It
helps
you
to.
You
know,
have
the
cost
at
Bay
and
giving
as
much
power
as
possible
to
specific
jobs.
If
the
cluster
is
empty,
the
user
gets
full
capacity
of
the
cluster.
If
new
users
are
coming,
they
are
sharing
the
resources
and
therefore
something
really
missing
in
the
in
the
case
to
bring
that
data
analytics
computation
towards
computers.
B
B
B
So
back
then,
if
we
for
example,
then
all
Spark
processes
are
killed,
then
you
will
see
then
the
fingers
everything
is
going
down.
This
is
all
being
random,
using
spark
interactive
mode.
So
that's
quite
nice,
so
yeah.
This
is
just
with
these
commands
and
they
can
just
do
anything
on
top
of
it
and
we
provide
this
dashboard
to
the
users.
B
So
then,
into
a
little
bit
of
workload.
Monitoring
we
have
also
I
want
to
show
you
a
bit,
but
that's
still
not
done
is
that
we
also
can
have
a
drf
dashboard
in
which
we
want
to
show
a
little
bit
more
fine
grains
in
more
higher
Coral
situations
that
we
have
one
road
queue.
We
have
an
interactive
queue
which
you
know
interactive
user
and
we
have
a
project
queue
in
which
all
the
big
projects
are
that
we
run
and,
for
example,
there
we
can
give
the
project
queue
more
parity
over
the
rest.
B
A
B
And
then
also,
we
want
to
essentially
avoid
traffic
jams
and
then
also
we
were
thinking
about
adding
some
cluster
Rush
Hour
part
in
which
we
can
give
the
user
a
little
bit
more
context
about,
like
probably
when
the
cluster
is
more
more
used
at
the
moment.
So
that's
more
from
a
self-service
kind
of
ux
perspective,
because
we
essentially
want
to
make
this
hidden
away
as
much.
A
As
possible,
as
you
see,
it's
like,
we
are
elementing
eliminating
the
toy
for
data
Engineers
data
analysts,
so
they
start
the
session.
They
do
not
care
about
the
configuration
of
the
cluster
to
run
their
job
and
they
get
the
best
performance
possible.
What
we
also
want
to
make
it
visible
towards
them.
You
know
we're
going
to
be
the
best
time
to
during
the
day
that
you
can
run
your
job,
because
the
cluster
is
less
busy
exactly.
B
So
since
I'm,
not
a
volcano,
maintainer
I
want
to
just
give
all
the
love
to
all
the
kind
of
folks
that
essentially
made
this
terrific
scheduler.
That's
a
really
great
job.
I
personally,
also
think
that
it's
a
that
this
is
like
the
best
way
to
go,
because
they
have
a
nice
nice
abstraction
over
how
we
can
try
to
make
a
task
scheduling
more
formal
in
kubernetes
and
how
we
can
get
more
performance
out
of
it,
I'm,
not
very
great,
open
source
worker.
B
B
We
have
added
the
drf
dashboard,
we
added
Squarespace
automatic
queue,
management,
more
Prometheus,
metrics,
updates
to
graphene
dashboards,
Cube
State
metrics,
and
they
leverage
some
closer
white
permissions
and
I
have
reduced
that
a
bit,
but
essentially
the
what
they
have.
What
it's
all
done
is
working
and
I
think.
Definitely.
This
is
the
way
to
go
and
it
would
be
cool
in
the
future
to
also
support
like
tensorflow
by
torch,
like
different
kind
of
distributed
methods.
So
we
can
add
all
different
kind
of
tasks
on
top
of
kubernetes.