►
From YouTube: Lightning Talk: Evolving the Job API to Become Defacto Standard for Batch Worklo...Aldo Culquicondor
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Lightning Talk: Evolving the Job API to Become Defacto Standard for Batch Workloads - Aldo Culquicondor, Google
Since Kubernetes 1.0 until recently, the Kubernetes Job API had a rather limited feature set. Meanwhile, multiple batch-oriented frameworks were developed, each re-implementing their own job API and controller to manage pods, with their own advantages and limitations, leading to fragmentation of the ecosystem. Starting in Kubernetes 1.21, contributors to SIG Apps decided to evolve the Job API, implement common patterns and increase its scalability, so that it can become the defacto standard for running batch workloads or for building specialized frameworks on top of. They introduced features such as indexed Jobs, suspended jobs, tracking with finalizers, failure policies, etc. In this talk, Aldo will walk you through all these efforts, the challenges that we faced when implementing them and the remaining opportunities we have identified to make the API more comprehensive and useful for a wider range of applications.
A
B
A
A
During
this
talk
during
this
batch
HPC,
they
that
there
were
a
number
of
people
projects
rewriting
the
job
controller,
and
this
of
course
leads
to
fragmentation,
which
makes
it
hard
for
providers
to
support.
You
know
we
were
hearing
earlier
that.
Well
now
we
have
all
of
these
different
apis.
We
have
to
support
which
ones
my
provider
supports.
So
that's
one
problem,
and
also
a
number
of
bugs
that
were
done
in
one
place-
are
also
replicated
in
all
these
places.
A
A
During
this
talk,
but
this
is
for
reference,
so
you
can
you
can
once
you
download
the
slides,
you
can
go
through
all
this,
the
documentation
for
all
of
these
ones,
I'm
also
going
to
talk
a
bit
about
what
are
we
thinking
for
the
future
and
what
are
the
moonshots?
What
do
we
want
to
bring
the
job
API
next,
so
the
the
first
one
is.
A
You
just
set
the
completion
mode
to
index,
and
just
you
set
the
number
of
completions
the
number
of
pots
you
want
to
run
and
then
in
the
command
line
or
well
in
your
in
your
pod.
In
your
workload,
you
just
access
the
the
environment
variable
and
we'll
execute
your
application
and
well.
This
is
what
this
is,
how
how
it
looks
in
the
this
is
how
you
can
think
about
index
jobs.
A
So
this
is
such
a
fundamental
feature
that
it
hasn't
existed
since
the
start
of
kubernetes
and
a
couple
of
years
back.
We
finally
added
it.
The
next
one
is
job
suspension
and
scale
and
mutable
scaling
directives.
So
this
provides
you
with
the
ability
of
creating
a
job,
but
not
start
the
the
the
pots
yet
think
of
it's.
This
is
this
has
already
been
mentioned,
but
you
could
have
a
a
case
where
you
don't
want
you.
A
You
have
way
more
workloads
running
or
you
desire
to
run
more
workloads
than
you
have
capacity
in
your
cluster,
and
you
don't
want
to
make
your
users
simply
well
try
to
run
the
application
and
run
into
scheduling
errors
or
or
simply
fail
the
the
Pod
creation,
the
job
creation.
You
would
like
to
queue
these
these
workloads,
so
we
added
these
Fields
into
the
into
really
just
this
one
field
in
the
job
API
to
say.
A
A
Then,
once
this
external
controller
considers
that
there
is
capacity
in
the
cluster
there
is
quota,
we've
dealt
with
a
fair
sharing
within
the
cluster
for
all
our
users.
We
can
start
a
job
and
maybe
even
place
it
in
a
particular
place.
In
this
case,
we
have
decided
that
this
job
is
going
to
run
in
the
in
this
Zone.
A
The
US
Central
1A
and
it's
going
to
run
on
spot
VMS,
because
this
is
what
we
have
capacity
now
of
spot
VMS,
and
this
is
exactly
what
one
of
the
projects
in
six
scheduling
is
doing.
It's
using
these
apis
to
to
implement
this,
these
semantics,
and
but
the
the
interesting
thing
here
is
that
well,
Q
is
one
project
that
can
use
this,
but
any
any
other
job
job
queuing
operator
can
can
use
the
same
apis
without
having
to
re-rewrite
the
job
controller
or
the
job
or
the
job
API.
A
This
one
is
a
feature
that
is
the
the
most
hurtful
for
me.
Basically
well
I
implemented
this,
but
it
was
quite
challenging,
and
this
is
why
I
I
was
mentioning
initially
about
bugs
how
this
this
bug
that
is
present
was
present
in
the
job
controller
was
present
in
a
lot
of
these
other
custom
job
apis
and
it
was
very
hard
to
fix.
A
So
what
was
the
problem?
Basically,
the
design
of
video
controller
was
meant.
It
was
originally
thought
of
low
scale
and
also
just
to
serve
services,
so
it
it
at
some
points
when
he
interacted
with
the
garbage
collector.
He
would
start
losing
track,
so
you
could
say
you
could
see
that
your
post
said
I
have
five
parts
completed
and
then
later
he
says,
I
only
completed
actually
three.
Why?
A
Because
the
pots
were
disappearing
and
the
job
controller
was
only
looking
on
the
at
the
pods
that
existed
in
the
cluster,
so
this
led,
of
course,
to
Port
Recreation,
and
then
people
ended
up
paying
more
to
run
their
jobs
and,
of
course,
this
meant
that
jobs
were
unusable
at
scale.
People
running
a
5000
parallel
job
wouldn't
be
able
to
use
the
job
API
or
maybe
one
yes,
but
not
not
more
than
three.
So
this
this
was
a
big
problem.
A
A
A
You
will
have,
of
course,
spots
with
a
finalizer
and
and
of
course,
you
will
see
that
your
job
is
finally
tracking
progress
correctly
and
not
losing
no
losing
not
decreasing
counters
of
completed
and
completed
jobs,
and
actually
we've
heard
from
some
of
our
customers
that
they
are
successfully
running
index
jobs
with
5000,
Bots
or
more,
which
of
course,
was
not
possible
before
this
is
another
feature
we
are
currently
working
on.
It's
it's
in
Alpha
today
and
we're
one
of
our
Engineers
is
working
on
the
on
the
Beta
release.
A
So
basically
in
kubernetes
you
have
a
number
of
problems
that
could
cause
your
pod
to
to
stop
running
right.
You
could
have,
of
course,
software
errors.
The
hardware
could
fail,
you
could
have
a
maintenance
event
like
upgrades
or
simply
maintenance,
or
you
could
have
the
eviction
API
or
maybe
a
high
priority.
Job
comes
in
a
cube.
Scheduler
simply
kill
support
the
support
garbage
collector
you
can
add
attained.
A
There
is
a
cluster
Auto
scaler
that
could
decide
that
suddenly
wants
to
preempt
some
pots
because
it
wants
to
optimize
utilization,
the
API
server,
and
you
could
maybe
write
your
own
controller
and
affect
a
running
pot.
So
all
of
these
components
can
can
really
kill
your
pod
and
you,
of
course
you
might
want
to
retry.
You
might
want
not
to
retry,
because
your
your
workload
doesn't
support
preemption
and
really
there
was
no
control
over
over
this
all
these
problems.
You
just
had
a
face
in
the
process.
A
This
has
failed
and
your
controller
could
only
say.
Oh
I
want
to
support
up
to
six
failures,
so
in
kubernetes
125
we
added
first
all
pot
condition
to
observe
what
what
happened
to
my
pod,
we
just
say:
well,
this
part
has
been
disrupted
and
it
has
been
disrupted
by
skill
scheduler
for
because
of
attained
Etc.
A
All
this
all
these
problems
and
in
126
we
are
working
on
the
support
from
Cuba
all
the
things
that
cubelet
might
do
to
kill
your
pod,
which
includes
exceeding
memory
limits,
for
example,
or
exceeding
the
usage
of
ephemeral,
storage,
and
maybe
we'll
we'll
think
about
some
other.
Some
other
conditions
in
the
future.
A
So
what
what
do
you
do
with
this?
This
is
just
the
information
being
passed
from
other
controllers
and
then
from
the
job
API.
You
can
use
these
conditions
to
decide
whether
or
whether
to
terminate
the
job
or
retry
it
or
how
many
times,
and
also
we
have
support
for
looking
at
the
exit
codes
of
the
of
the
pots.
A
Maybe
you
want
to
terminate?
Oh
sorry,
yes,
maybe
you
want
to
fail
your
job
because
of
Any
error
that
happens
in
the
in
the
in
the
binary,
but
you
want
to
retry
if
there
is
a
disruption
by
the
control
plane,
but
only
up
to
a
limit.
In
this
case,
the
back
of
limit
is
three,
so
I
want
to
count
this
failure
and
once
it
fails
three
times
I'm
done
with
it,
I
don't
want
to
retry
again.
A
So
this
is
all
the
features
we've
been
working
on
either
most
of
them
are
finished,
one
of
them
is
ongoing
and
then
in
the
near
future,
we
want
to
add
this
other
capability
to
index
jobs
where
you
control
the
back
of
per
index,
and
you
can
basically
guarantee
that
all
your
indexes
run
at
least
once
or
up
to
the
Limit,
and
you
can
say
even
okay
if
it
is
limit.
If
this
index
fails,
I
just
want
to
fill
this
index,
but
all
of
the
other
indexes
can
continue
running.
A
We
haven't
started
the
design
yet,
but
this
is
kind
of
like
the
potential
API
so
and
what?
What
comes
next
well,
we
are
thinking
of
some
some
moonshots,
but
but
really
what
we
want
we're
here
in
in
much
HPC.
They
basically
to
present
these
enhancements
and
also
to
ask
you
what
do
you
want
to
see
next
in
the
job
API?
A
What
how
do
you
think
a
job
V2
API
could
look
like,
for
example.
Maybe
why
do
we
need
not
index
jobs?
Maybe
all
indexes
all
jobs
could
be
indexed
would
be
a
huge
simplification
for
the
kubernetes
control
plane.
That
would
lead
to
even
better
performance,
or
maybe
you
want
to
see
multiple
pod
templates
pattern
with
a
startup
sequence,
which
is
something
we
haven't
looked
at,
or
maybe
you
want
recessable
jobs.
The
jobs
currently
are
fixed,
fixed
number
of
completions.
A
A
So
once
we
publish
our
API
design,
we
would
like
to
know
if
it
fits
your
use
case
and
well
to
do
that.
Of
course,
we
also
want
contributors
if
you
want
to
contribute
to
the
kubernetes
core
components,
but
this
is
all
the
places
we
communicate
in,
in
particular
the
working
group
batch
or
the
sync
UPS
in
slack
or
the
the
meetings
we
have
so
yes,
thank
you.
Thank
you
for
listening
to
me
and
please
keep
in
touch.
Thank.
C
A
Rather,
that's
well
from
the
perspective
of
the
job
controller.
That's
tangential!
The
job
controller.
Doesn't
care
about
scheduling,
it's
all
about
the
how
it
presents
to
the
developers
to
the
application
Developers.
If
you
want
my
take
as
a
sick
scheduling
lead.
That's
that's
a
different
question.
Is
that
what
we're
looking
for.
C
A
A
One
is
that
maybe
a
service
is
not
that
expensive
and
it
could
be
created
from
the
beginning,
and
it's
going
to
be
there
sitting
idle
because
the
job
the
pots
are
not
created,
so
the
services
don't
have
endpoints
and
whatnot
the
other
way
to
think
about
it
is
well
you
can
implement
it
in
the
controller
in
your
application
controller,
for
example.
Let's
say
we
have
interactive
interactive
job
controller.
A
This
interactive
job
controller
has
a
job,
a
kubernetes
job,
and
you
say:
well,
your
controller
only
will
create
a
service
once
the
job
is
unsuspended
the
kubernetes
jobs
and
suspended.
So
you
could
actually
just
listen
to
the
signal
from
the
job
API
and
control
the
the
rest
of
the
resources
based
on
that
those
are
the
two
ways
I've
so
far
thought
about
it.
But
of
course
it's
a
it's.
An
open
discussion
still.
A
F
A
E
A
I
can
point
here,
so
you
have
to
to
Fields
completions
and
parallelism,
so
completions
completions
is
how
many
are
in
total
and
parallelism
how
many
run
at
a
time.
So
this
is
actually
what
the
graph
is
showing
here.
So
it's
two
at
a
time
running
now
in
terms
of
steps.
We
believe
that
this
kind
of
things
can
be
done
in
the
application.
Level
I
mean
you
can
always
multiply
inside,
although
there
have
been
requests
about
basically
I
want
to.
A
You
know,
I
run
my
job
and
then
certain
indexes
fail
and
then
I
just
want
to
retry
the
indexes
that
failed.
So
that's
something
we
we
might
be
looking
into,
but
we
don't
have
support
for
running
all
the
indices
yet
so
that
would
be
the
first
step
and
then
we
can
implement
selected
indexes
to
run
but
yeah,
that's
something
we
are
definitely
looking
forward
in
the
in
the
next
releases.
A
G
A
Yes,
so
that's
that's
the
API
for
job
retries,
that's
one
of
the
disruptions
the
control
plane
can
do
and
then
you
can
express
in
the
API
whether
you
want
to
retry,
for
that
particular
kind
of
disruption
or
or
if
your
workload
doesn't
support
it
you
you
can
stop.
Does
that
answer
your
question
yeah.
C
H
A
D
Is
there
any
way
to
coordinate
that
at
this
time
so
that
you
know
the
job
doesn't
actually
run
until
both
of
these
things,
because
we'd
have
the
point
where
the
queuing
system
sets
it
unsuspends
it
and
the
other
system
then
tries
to
suspend
it
and
I
was
wondering
if
you
had
any
ideas,
how
to
prevent
it
from
starting.
In
that
brief
Gap.
A
So
I
can
tell
you
a
little
bit
of
what
we
are
doing
in
queue,
which
is
our
implementation
for
how
we
think
this
should
work
and
well
there's
a
few
things
first,
having
our
back
to
prevent
a
job
from
starting
is
one
possibility,
but
ultimately
the
controller.
Can
you
know
if
he
observes
a
job
try
to
unsuspend,
you
can
suspend
it
back
from
the
controller.
A
To
prevent
that
yeah
absolutely
well,
this
is
kind
of
we're
feeding
within
the
pattern
of
kubernetes,
which
is
eventual
consistency
right
in
kubernetes.
Nothing
is
like.
We
don't
issue
something
and
wait
for
it.
We,
or
rather
we
do.
We
do.
We
have
a
desired
State
and
the
control
plane
converges
towards
the
desired
State
State.
You
again,
you
can
control
some
things
with
our
back,
but
ultimately
the
controllers
should
remedy
those
situations.