►
From YouTube: Introduction to the Kubernetes WG Batch - Aldo Culquicondor & Abdullah Gharaibeh, Alex Wang
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Introduction to the Kubernetes WG Batch - Aldo Culquicondor & Abdullah Gharaibeh, Google; Alex Wang, Alibaba
The Kubernetes Working Group Batch was newly formed in the beginning of 2022. The Working Group aims to be a forum to discuss and propose enhancements to support for Batch (eg. HPC, AI/ML, data analytics, CI) workloads in core Kubernetes. We want to unify the way users deploy batch workloads to improve portability and to simplify supportability for Kubernetes providers. In this session, you will learn about the WG goals and roadmap , as well as the early efforts performed by our contributors.
A
A
A
A
Well,
the
reason
is
kubernetes,
as
you
may
or
may
not
know
it
had
a
humble
beginning.
Kubernetes
was
built
for
serving
applications,
stateless
applications
and
over
time
we
we
have
been
increasing
support
for
stateful
stateful
applications
right
still
still,
of
course,
there
are
still
things
to
be
done,
but
there
was
never
a
focus
on
other
types
of
workloads
which
are
very
different
from
from
serving
applications
and,
of
course,
I'm
referring
to
to
batch
workloads.
A
So,
yes,
you
can
run
batch
workloads.
There
is
a
job
api
and,
of
course
you
can
create
your
own
crts
and
run
batch
applications,
but
there
are
feature
gaps
back
to
the
job
api
there
is.
There
is
very
little
policies
around
completion
and
failure
modes
in
the
job
api.
So
that's
something
that
limits
the
the
usage
of
of
the
job
api
itself
in
the
cubelet
or
well,
not
just
the
cubelet,
the
overall
kubernetes
system,
and
there
is
a
lack
of
some
support
for
specialized
devices.
A
Yes,
we
have
the
device
plugin
api,
but
it
doesn't
have.
If
I
want
to
say
how
much
memory
I
want
for
my
gpu,
you
cannot
do
that
or
or
if
you
have
pneuma
cells
in
or
numa
nodes
within
a
node.
You
cannot
well.
There
is
some
progress,
I'm
going
to
talk
about
it
during
the
presentation,
but
there
are
all
these
different
things
about
hardware
that
you
might
need
for
for
high
performance
which
are
lacking
all
or
nothing
but
scheduling
this.
A
A
And
yes,
this
is
kind
of
related.
So
once
you
have
multiple
parts
trying
to
you
have
multiple
jobs
and
each
job
is
a
part
of
the
job
with
multiple
parts.
Then
we
don't
have
but
all
or
nothing
scheduling.
So
we
we
also
don't
have
job
queueing,
so
you
either
create
a
job
or
or
you
don't
so
so
far
in
kubernetes.
The
status
quo
was
to
support
this
feature
through
crds
through
replacing
the
scheduler
all
sorts
of
ways
around
the
problem.
A
So,
yes,
the
working
group
started
actually
when
I
submitted
the
application
for
for
this
talk,
the
working
group
was
not
created.
I
could
do
it.
I
think,
because
I'm
a
tln6
scheduling,
so
I
I
was
allowed
to
to
submit
the
application,
but
we
had
already
asked
the
the
steering
committee
to
to
create
this
working
group.
It
looked
like
it
was
going
to
go.
Unfortunately,
did
in
february
the
working
group
was
formally
created,
but
the
work
didn't
start.
A
A
A
This
way
we
introduced
suspended
jobs
as
well.
This
it
might
be
hard
to
understand
on
itself,
but
basically
its
ability
to
create
jobs
but
defer
the
pod
creation
to
later
and
you'll,
see
why
this
is
important
in
the
next
slides.
So
we
also
introduced
a
ttl.
After
finish,
jobs
don't
unnecessarily
once
they
finish.
They
stay
there
in
the
api
server
consuming
well
etcd
resources.
A
A
Managing
pot
is
very
hard,
so
we
don't
want
you
as
developers
to
have
to
manage
spots.
Kubernetes
should
do
it
right,
so
we
hope
that
by
solving
this
problem
and
by
increasing
the
use
cases
of
the
job
api,
you
don't
have
to
think
about
this.
You
can.
You
can
create
your
crds,
of
course,
but
when
it
comes
to
managing
bots,
you
just
create
a
job,
and
you
don't
have
to
run
into
these
bugs
again
over
and
over.
A
We
also
introduce
this
small
feature
to
track
the
number
of
ready,
pods
in
deployments
or
sorry
replica
sets.
You
have
this
field
saying
active
bots
that
was
not
in
the
job
api.
We
included
it
and
crown
job
was
graduated
to
ga,
which
is
also
very
important.
There
were
several
optimizations
there
and
somewhat
related.
This
might
be
more
related
to
gaming
or
or
if
you
have
a,
I
don't
know,
I
don't
want
to
put
other
examples,
maybe
spark
jobs.
A
Actually,
if
you,
if
you
want
to
reduce
the
size
of
your
replica
set,
there
was
no
way
to
tell
which
bots
should
be
removed.
Pot
decision
cost
doesn't
provide
all
the
guarantees,
but
it
still
gives
some
flexibility
for
you
to
decide
when
to
to
which
spots
down
scale
quickly,
just
want
to
show
how
an
index
job
looks
like
very
simple.
A
We
have
our
different
tasks
that
we
have
to
accomplish
right,
the
five
at
the
top,
and
so
we
have
number
of
completions
five
and
then
we
have
number
of
parallelism
two.
This
means
that
at
every
each
point
of
time
we
only
have
two
parts
running,
but
they
will
basically,
the
first
part
will
take
the
first
work
item,
second
part
and
second
one
and
so
on
and
so
forth,
and
you
get
this
window
that
progresses
through
the
through
the
data.
A
A
B
Users
can
define
the
guaranteed
resources
and
mean
and
the
maximum
resources
and
max
by
using
this
object,
and
there
is
an
example
on
the
left
which
makes
it
possible
to
share
the
resources
dynamically
between
different
name
spaces
in
your
classroom
and
to
improve
the
resource
utilization
in
the
cluster
that
home
book
usage
without
the
right
will
names.
These
two
have
three
resources.
B
Things
with
white
can
borrow
resource
interactions
with
true
to
run
one
point,
and
when
things
with
true
is
resources,
the
borrowed
will
be
reclaimed
at
which
point
both
names
with
one
and
names.
We
too
can
use
their
guarantee
to
resources
and
the
last
one
is
being
had
each
one
learns
asked
to
scan
your
homes
on
the
notes
that
have
the
list
and
use
the
resources,
such
as
cpu
memory
and
the
other
extended
resources
like
gpm.
B
A
A
Now
the
problem
is
that
the
cubelet
takes
its
decisions,
but
it
doesn't
coordinate
those
with
the
cube
scheduler.
So
for
that
signal
was
working
on
this
crd.
We're
still
working
on
this
here,
the
for
exposing
the
the
decisions
that
the
the
topology
manager
manager
did
so
that
a
scheduling
plugin
could
use
that
information
to
schedule
based
based
on
that.
A
All
right,
so
this
all
happened
before
february.
On
february,
we
finally
got
approved
to
create
the
working
group
batch,
which
has
this
mission
discuss
and
enhance
the
support
of
batch
workloads
in
core
kubernetes.
The
goal
is
to
unify
the
way
users
deploy
batch
workloads
to
improve
portability
and
to
simplify
supportability
to
for
kubernetes
providers.
A
So
I'm
going
to
quickly
highlight
here
what
what
do
I
need
by
batch
workloads?
Usually
well,
we
could
be
talking
about
ml,
we
could
be
talking
about
hpc
and
we
can
be
talking
about,
maybe
something
less
obvious.
Like
cicd
when
you
have,
when
you
have
ci
a
ci,
you
have
a
job
just
to
run
your
tests,
for
example,
so
it
they
behave
similarly
to
these
other
types
of
workloads.
So
we
we
want
to
improve
the
support
for
for
all
of
these
things
incorporated
and
unify.
A
A
So,
of
course,
we
have
seagaps
as
the
particip
as
a
participant
stakeholder
seagaps
owns
the
the
all
the
batch
apis
think
auto
scaling.
They
are
listed
in
alphabetical
order,
so
cigar
to
scaling.
Of
course,
when
we
have
job
work,
when
we
have
multiple
jobs,
maybe
our
small
cluster
is
not
enough.
So
we
want
to
ask
how
to
scale
and
with
that
we
we
want
to
improve
cost
right
scale
up
scale
down,
so
we
we
wanted
to
make
a
co2
scale
in
part
of
the
initiating
initiative
as
well
signal.
A
We
already
saw
some
some
work
being
done.
There
are
multiple
caps
that
I
saw
open
around
this
area,
so
very
encouraging
to
to
see
and
seek
scheduling.
Of
course,
that's
where,
where
I
come
from
so
yeah,
those
are
the
stakeholders.
Once
we
created
a
working
group
batch,
we
decided
to
kind
of
define
what
what
things
we
want
to
work
on.
A
A
We
already
saw
static
partitioning,
but
there
is
still
work
to
be
done
for
supporting
mpi
ml
ai
again
we
we,
if
you,
if
you
are
using,
for
example,
keyflow
mpi
jaw-
that's
that's
great,
but
again
managing
pot
is
hard,
so
we
want
kubeflow
mpi
job
to
somehow
use
the
the
job
api
or
some
other
apis
that
that
we
have
for
for
groups
of
bots.
So
job
api
is
the
the
main
api
we're
working
on.
But,
of
course
all
the
apis
are
open
for
discussion.
A
A
A
A
We
actually
did
some
progress,
so
six
scheduling
started
sponsoring
an
a
new
project
for
job
queuing.
I
don't
have
a
presentation
of
q
today
because
we
already
had
one
on
tuesday
for
the
batch
day,
I
believe,
is
the
link
on
the
right.
This
is
a
full
presentation
on
on
the
on
the
system.
I'm
gonna
quickly
explain
what
it
is
and
the
other
link
is
for
ricardo's
presentation
the
keynote
in
the
morning.
A
So
yeah
go
ahead
and
watch
it.
It
was
pretty
pretty
cool,
so
cube.
It's
a
job
level
manager
so
again
we're
not
dealing
with
pots,
we're
talking
we're
dealing
with
jobs,
and
then
it
decides
when
the
job
should
start,
meaning
that
we
can
create
bots
or
if
there
is
no
capacity
and
then
I
have
a
high
priority
job.
I
need
to
reclaim
that
capacity.
So
I
can
stop
a
job
and
delete
all
those
spots
and
start
my
new
job
so
that
that's
the
that's
what
what
q
does
on
a
high
level.
A
So
for
that,
I
I
showed
you
much
earlier
there
we
added
the
suspend
field
in
the
job
api,
so
we
did
that
to
enable
queue
yeah,
it's
very
simple:
you
suspend
the
job,
no
pots,
you,
you
unsuspend
the
job
but
are
created.
So
this
way
we
prevent
the
scheduler
from
scheduling
something
that
might
not
fit,
and
we
take
the
decision
in
this
other
controller
at
a
much
which
has
a
view
of
all
the
jobs
that
are
running
now.
A
I've
been
talking
about
jobs,
but
in
queue
there
is
this
concept
of
workload,
which
is
a
smaller
it's
a
shadow
representation
of
a
job
such
that
we
can
support,
not
just
a
job,
but
we
need
to
support
crds
because
of
course,
today,
job
api
doesn't
doesn't
support
all
the
use
cases
and
maybe
we'll
there
is
always
a
chance
for
for
crds.
So
we
we
want
that
to
be
included.
A
The
main
design
principle
of
q
is
no
duplication,
so
q
coexists
with
everything
we
have
already
in
kubernetes
q
coexists
with
the
q
controller
manager,
which
knows
how
to
manage
manage
spots
for
a
job
which
knows
how
to
connect,
assign
a
pot
to
a
node
and
the
cluster
autoscaler,
which
knows
how
to
add
a
capacity
in
my
cluster.
So
q
coordinates
with
all
of
them
rather
than
replacing
the
all
of
those
components.
So
that's
that's
the
main
design
principle.
A
The
usage
I
mean
the
admins
define
cues
with
resource
flavors
quotas
and
borrowing
cohorts,
so
they
have
the
the
heavy
job,
let's
say
and
users
they
don't
change
their
behavior.
They
just
create
jobs,
they
just
say
the
queue
and
that's
it
so
we
released
a
v01
on
april,
we're
working
on
the
v02,
where
we
want
to
add
more
metrics
and
make
it
robust.
A
So
go
and
watch
the
two
presentations.
What
else
is
happening
in
what
is
happening
in
in
the
space?
We
have
a
few
a
few
of
these
debates.
We
are
talking
about
more
policies
for
jobs
we
want.
We
are
thinking
about
how
to
do
still
do
all
our
all-or-nothing
scheduling
in
in
the
keep
scatter.
For
the
cases
where
we
have
a
study
cluster
with
very
tight
resources,
it
might
still
be
needed
an
api
to
reserve
no
resources,
so
the
autoscaler
can,
like
can
preemptively,
add
more
capacity,
maybe
a
plugin
model.
A
There
was
a
proposal
for
a
plugin
model
for
the
cubelet,
so
we
can
more
easily
define
support
other
hardware.
We
are
welcoming
presentations
from
all
of
you.
If
you
have
a
batch
project,
a
crd
we
want
to.
We
want
to
understand
what
future
apps
you
have
and
we
want
to
to
keep
the
con
the
discussion
going
on.
A
We,
of
course,
how?
How
can
you
get
involved?
Well
come
present.
We
have
the
links
in
the
the
last
link,
the
working
group
batch
in
the
official
community
repository
there.
You
can
find
the
slack
you
can
find
the
mailing
list.
Well,
they
are
there
and
you
can
find
our
schedule.
Go
ahead.
Put
your
topic
in
the
agenda
and
our
lovely
organizers
who
are
sitting
right
here
will
will
well
moderate.
A
You
know
the
next
in
the
next
working
group
meeting
and
that's
all
oh
yes
before
I
forget,
google
is
running
live
panel
in
june
15th.
We
we
invited
the
relevant
leads
from
the
different
seeks
within
kubernetes
to
to
discuss
to
chat
more
about
what
is
missing
in
kubernetes
to
support
a
batch
so
yeah.
I
invite
you
to
to
go
there
and
register
so
questions
before.
So
I
completely
forgot
about
introducing
my
co-presenters.
Abdullah
is
right
here
for
questions.
A
C
D
Hi,
thank
you
for
the
talk
I
wanted
to
know.
How
does
it
integrate
with
some
existing,
for
example,
training
operator,
formerly
tf
job
operator,
because
I
remember
there
was
indexed
pods
already
at
the
time.
So
if
you
have
like
multiple
replicas,
for
example,
the
pods
would
be
indexed-
and
I
don't
know
now,
if
you,
if
these
now
features,
are
brought
by
the
batch
api,
spec
or
batch
api,
so
how
it
will
integrate.
A
Right
so
the
then
df
job
right
that
know
df
job
would
have
to
use
the
job
api.
So
the
con,
the
tf
job
controller,
would
have
to
create
job
objects
instead
of
creating
robots
and
then
would
benefit
from
the
the
index
job
we
we
engaged
with
the
keyflop
community.
We
wanted
to
to
get
this
done.
We
wanted
them
to
to
be
able
to
use
what
we
have
been
working
on,
but
it's
still
not
possible.
They
have
extra
policies
in
terms
of
termination
and
failures.
A
A
A
A
You
know
core
features
that
other
batch
schedulers
can
use,
so
kubernetes
can
focus
on
starting
starting
jobs
and
the
batch
schedulers
can
focus
on.
Well,
maybe
maybe
you
don't
like
use
fair
sharing
you,
you
can
implement
your
own
fair,
sharing,
algorithm
but
starting
the
jobs.
Maybe
some
basic
queueing
fifo
queueing
you,
you
just
still
use
that,
and
if
we
are
talking
about
multi-cluster
right,
the
the
kubernetes
core
should
support
some
of
those
operations.
A
Now
the
working
the
cncf
batch
working
group
is
looking
at
the
problem
at
a
let's
say,
at
a
outside
of
quantities
right
they
are
trying
to
integrate
or
they
are
trying
to
uniformize
multiple
different
batch
projects
which
may
or
may
not
be
kubernetes,
so
we
hope
to
engage
with
with
them
so
that
we
we
are
somehow
aligned.
But
that's
a
conversation
that
hasn't
started
yet.
A
F
D
F
A
So
I
suppose
you're
talking
specifically
about
queue,
oh
the
job,
api
yeah.
So
when
we
are
talking
about
scaling
job,
the
the
main
question
here
is
which,
when
we're
scaling
up
all
this
all
is
fine,
just
add
more
pots
when
we're
scaling
down.
That's
the
question
which
one
which
pots
do
we
remove
and
simply
the
discussion
is
just
to.
A
G
Yeah,
I
just
want
to
mention
that
in
the
job
api,
I
think
one
of
the
things
that
are
fixed
is
the
completions.
This
is
what
you're
mentioning
that,
like
you
define
that
my
job
is
complete
when
I
complete
10
instances
of
this
pod,
but,
like
you
want
to
scale
it
up
and
down.
This
is
the
same
thing
right
like
if
we
make
completions
a
mutable
field,
it
starts
with
one,
for
example,
and
then
you
increase
it.
F
F
Dynamically
generates
the
the
pots
that
it
needs.
So
you
submit
the
job
consists
of
just
the
driver
and
the
rest
is
code
within
the
driver
that
decides
on
what
it
does.
So
what
you're
asking
almost
for
is
the
driver
to
create
job
api
things
to
get
the
executors
up
and
running
and
scheduled
in
a
way.
So
that's
a
huge
request
that
you
put
on
the
applications
that
are
going
to
run
using
this.
You
you
want
to
do
it
without
changing
the
application
that
runs.
A
So
the
the
spark
jobs
and
mpi
jobs-
if
you
were
talking
about
there-
is
some
elastic
api
jobs.
So
when
we
are
talking
about
them
it
they
100,
don't
fit
the
job
api
today
right.
The
job
api
is
designed
for
completions
and
these
jobs
be
yes,
the
entire
thing
is
a
job,
but
the
individual
workers
are
not
tasks.
They
are
kind
of
like
small
servers,
so
they
are
more
similar
to
services,
so
more
similar
to
deployments
and
staple
sets.
So
the
answer
my
answer
today
is:
we
don't
have
an
api
for
that
today.
C
A
I
suppose
this
this
is
asking
about
the
job
controller,
so
it
comes
a
lot
to
what
well.
For
starters,
it
depends
a
lot
on
on.
What's
the
api
throttling
that
the
jaw
controller
has
so,
if
you
increase
that
it's
probably
going
to
behave
better,
but
at
the
same
time
that's
something
we
want
to
work
on.
We
recently
introduced
some
load
testing
for
the
job
controller,
just
the
past
release
and
well,
we
just
started.
We
put
the
load
test,
we
don't
have
a
concrete
answer
today.
We
want
to
improve
it.