►
From YouTube: Panel Discussion: Fragmentation of the Batch Ecosystem in Kubernetes, Challenges and Solutions -
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Panel Discussion: Fragmentation of the Batch Ecosystem in Kubernetes, Challenges and Solutions - Moderated by Abdullah Gharaibeh, Google; Diana J. Arroyo, IBM; Wilfred Spiegelenburg, Cloudera; Daniel Milroy, Lawrence Livermore National Laboratory; Albin S
Kubernetes historically focused on service-type workloads, support for load balancing, rolling-updates, spreading and autoscaling are few examples of features the community built for service workloads. While support for Batch workloads lagged in Kubernetes core, recent progress has been made to make Kubernetes a native home for batch workloads including major feature and scalability enhancements to the Job API and the establishments of the batch working group. This panel will discuss what is still missing in core k8s for batch support, what functionalities do we need to push upstream, and what should continue to be loosely defined so that we don't impose specific semantics on how batch jobs should run on k8s.
4:50pm – 5:00pm O
A
The
community
is
trying
to
convert
other
types
of
workloads
on
that
same
platform,
but
because
we're
having
focused
as
a
community
on
running
batch
workloads
on
kubernetes,
there
are
a
lot
of
gaps
that
we
know
that,
and
so
we
thought
like.
Okay,
maybe
we
should
discuss
this
and
have
a
look.
We
we
have
these
discussions
a
lot
right
like
during
the
day.
A
We
have
multiple
talks,
it's
giving
us
like,
basically
a
laundry
list
of
gaps,
but
people
are
still
running
batch,
workloads
and
kubernetes,
even
though
we
have
all
these
gaps,
and
so
we
thought
like
what
about
bring
four
different
platforms
that
were
built
on
top
of
kubernetes.
We
have
a
platform
from
G
research,
Armada
we've
got
unicorn,
we
have
mcad
from
IBM
and
we
have
qflux
or
flux
I,
guess
from
Los
Angeles
National
Labs.
B
B
So
how
basically,
can
we
merge
these
two
models
together
and
can
you
is
that
even
possible
to
begin
with
I
think:
that's,
that's
a
real
big
challenge!
I,
don't
know
if
there
that's
a
gap
per
se
in
in
batch
for
kubernetes.
It's
I
think
more
of
a
clash
of
models
that
hopefully
will
converge
on
one
another.
C
If
a
port
was
preempted,
as
it
turns
out
that
the
cube
schedule
will
preemptible
and
then
after
that,
create
an
event
saying
that
the
port
was
preempted,
but
there's
no
guarantees
that
this
event
will
actually
be
produced,
or
indeed
on
any
ordering
between
that
event
and
other
events,
and
so
looking
into
the
cluster.
It
can
be
difficult
to
determine
what
happens
to
your
pods
and
to
act
accordingly.
C
And
finally,
you
know
because
kubernetes
is
sort
of
grew
up
in
the
microservices
world.
There's
still
a
lot
of
tooling
missing.
For
example,
gang
scheduling
there's
a
lot
of
fragmentation
around
gang
scheduling,
and
this
is
something
that
we
really
want
to
support
in
our
model
and
it's
not
really
clear
how
we
will
do
this
as
many
options
just.
D
D
That's
all
there!
So
when
we
see
people
that
want
to
move
to
kubernetes,
the
first
question
is
what
they
ask
is.
So
are
you
going
to
give
me
all
the
functionality
that
I've
got
on
yarn?
So
do
you
do
you?
Do
you
do
queuing?
Do
you
do
quotas?
Do
you
do
all
that
stuff
on
the
Fly
and
do
you
implement
strict
security
on
Hadoop?
D
We
run
with
kerbros
enabled
every
single
thing
is
Kerberos
all
the
way
through
the
Pod
or
the
ga
VM
that
you
run
on
the
system,
so
that
principle
is
there
and
when
you
look
at
kubernetes
from
from
our
perspective,
was
also
like
we're
missing
that
user
information
going
through
the
scheduling
cycle,
so
accounting
for
resource
usage
and
I've
heard
a
number
of
people
say
that
that's
difficult.
So
how
do
you
get
that
through
there
preemption
from
from
batch
jobs?
D
It
often
matters
exactly
which
Bots
you
preempt,
if
you
preempt
the
driver
bot
from
spark
your
whole
spark
application
is
gone,
so
it
could
be
far
more
costly
to
PM
that
one
driver
Port
than
preempting
10,
executor
Bots
overall
so,
and
that
that
difference
you,
you
can't
make
the
difference.
The
scheduler
is
not
aware
of
that
information,
so
we're
trying
to
get
that
kind
of
information
into
the
system.
So
that's
missing
from
from
from
our
perspective,
so
on
top
of
a
number
of
the
other
things
that
are
already
mentioned,
it's
two
different
worlds:
almost.
A
So
before
going
to
the
to
the
audience,
the
questions
I
have
like
one
high
level
question
here,
which
is:
do
we
believe,
like
since
you've
been
working
with
kubernetes
for
a
while
now?
Do
it
there's
something
fundamentally
wrong
with
kubernetes
that
makes
it
wrong
or
like
limiting?
That
makes
it
hard
to
build
on
top
of
to
build
batch
Frameworks
on
top
of
it,
but
we
have
to
do
it
because
well,
it's
the
cool
thing
out
there
and
we
have
to
basically
unify
our
infrastructure
around
something.
D
I
think
I
can
give
a
quick
hook
into
Etc
the
when
you're
looking
at
batch
jobs
and
batch
things,
there's
users
that
that
will
submit
hundreds
of
jobs
at
one
go
which
could
be
thousands
of
bots.
So
overloading
Etc
D
is
a
real
problem
and
you
need
to
be
really
careful
and
that's
also
why
what
Apple
did
what
I
hear
we
had
a
meet
up
last
week,
Pinterest
also
they
all
stopped
scaling
at
800,
2000
notes.
They
say
we
don't
go
any
further.
We
do
not
want
to
get
into
problems
with
Etc
D.
E
C
The
other
thing
that
is
a
fundamental
problem
with
kubernetes
that
will
be
very
difficult
to
change,
is
the
events,
because
the
events
so
kubernetes
will
lose
something
and
then
afterwards
publish
an
event
that
it
did
it,
and
there
are
many
components
that
concurrently
publish
these
events.
Because
of
this
architecture,
you
can
never
guarantee
event
ordering
or
delivery,
and
so
they
will
never
be
reliable.
C
So
it's
also
something
that
you
try
to
leave
it
with
almada,
whereby
Armada
will
Ramada
is
itself
event
driven,
so
Ramada
does
guarantee
event,
delivery
and
ordering,
but
at
the
point
at
which
we
interact
with
kubernetes,
we
don't
have
these
guarantees
and
we
have
to
sort
of
try
to
do
the
best.
We
can.
E
B
Yes,
I'd
like
to
piggyback
off
of
what
several
of
the
other
presenters
or
panelists
have
said
so
far.
One
of
the
things
I
mentioned
toward
the
end
of
my
presentation
on
fluence
was
a
project
that
we're
working
on
called
the
flux
operator
and
the
idea
behind
this
is
to
instantiate
a
mini
cluster
inside
of
kubernetes
that
will
provide
full-featured
batch
scheduling.
B
So
this
is
the
scheduling
the
queuing,
basically
everything
inside
of
it,
and
there
are
some
difficulties
that
that
we've
run
into
so
far
and
I
said
before
that
there
aren't
really
limitations
that
that's
not
exactly
true.
So
one
of
the
things
that
that
we've
run
into
is
the
difficulty
in
figuring
out
when
some
of
the
worker
pods
are
ready
state,
so
the
the
way
that
it
flux,
the
way
that
flux
will
wire
up
is
that
the
workers
will
start
and
then
communicate
with
via
a
tree
base.
B
Overlay
Network
to
a
broker
rank
zero
and
they
basically
signal
that
they
are
ready
over
that
particular
Network.
So
if
there
were
an
internal
kind
of
a
intermediate
state
or
a
state
after
running,
that
was
more
specific
to
that
than
maybe
that
broker
could
query
that
state
from
within
this
group
of
PODS
and
then
transition
to
running
where
it
could
actually
execute
the
application
MPI
or
whatever
it
was
afterwards.
A
F
A
G
More
of
an
observation,
so
I
don't
have
an
HPC
batch
background.
I
have
been
working
with
kubernetes
since
2017
running
in
production,
in
a
big
Bank
and
as
an
architect
I
get
a
thought.
Should
you
want
to
do
a
basically
lift
and
shift
from
the
traditional
world
to
the
kubernetes
world,
or
should
you
put
some
of
the
changes
at
the
application
side
as
well
and
not
try
to
fix
all
the
problems.
D
But
if
you
look
at,
let's
say
a
spark
job,
when
you
process
data,
you
do
not
know
exactly
how
many
executors
you
will
use
or
will
will
have
to
use
until
you
know
what
your
data
is.
So
you
don't
want
to
take
that
Dynam
that
Dynamic
capability
away,
because
if
you
take
it
away,
then
you
either
extend
the
processing
time
or
us
resources,
because
you're
you're
only
using
half
of
the
executors
so
pulling
that
information
out
and
then
saying.
D
Oh,
if
you
put
that
in
a
Gateway
kind
of
thing
and
you
do
it
all
beforehand,
then
we
can
make
it
easier
for
kubernetes
to
work
with
yeah.
Your
mileage
may
vary,
so
you
will.
You
will
get
somewhere.
I
know
from
a
pechicon
that
there
are
a
couple
of
companies
that
are
working
on
on
pulling
that
information
all
into
an
application
outside,
but
they
also
say
that's
a
continual
process
and
they
analyze
every
single
spark
job
that
they've
run.
So
the
overhead
becomes
also
pretty
big
on
your
on
your
side.
A
C
So
I'll
just
comment
so
I
think
sort
of
a
modern
computer
is
one
of
the
sort
of
successful
design.
Principles
has
been
to
make
computers
so
that
it
appears
to
the
applications
running
on
them,
as
they
have
the
whole
computer
for
themselves.
Then
there's
no
contention
as
on
and
one
can
make
the
same
argument
for
the
modern
accumulators
and
distributed
world
right.
Let's
try
to
make
the
platform
such
that
to
the
from
the
point
of
view
of
the
applications
right.
C
H
Now
it's
working
so
I
I
think
one
of
you
mentioned
this
challenge
that
we
have
that
I
frequently.
We
we
here
mentioned
in
the
context
of
HPC
that
users
want
to
control
everything
and
they
want
to
like
Drive
everything,
but
then,
interestingly,
I
think
two
talks
before
we've
seen
that
funny
slide
from
from
that
check.
H
University
presentation
where,
like
we
were,
was
showing
like
how
bad
users
are
in
like
efficiency
of
their
workloads,
which
is
I
to
my
experience
at
least
I'm
going
to
do
the
presentation
how
bad
most
users
are
in
in
controlling
these
things,
so
I'm
wondering
whether
our
drive
towards
all
of
this
very
sophisticated
features
towards
controlling
the
processes,
controlling
scheduling.
Aren't
we
what's
your
opinion?
Aren't
we
actually
over
indexing
on
a
very
small
fraction
of
power
users
versus
delivering
something
that
is
going
to
be
more
broadly
power,
powerful
for
for
broader
community.
C
So
at
least
sort
of
at
the
research,
our
users
are
absolutely
terrible
in
knowing
what
they
want.
On
the
other
hand,
they're
using
state-of-the-art
machine
learning,
Frameworks
and
the
authors
of
those
Frameworks
know
exactly
what
they're
supposed
to
need,
and
so
I
think.
That's
where
you
want
to
put
the
smarts
of
figuring
out
exactly
sort
of
what
Newmar
topology
awareness
is
necessary
for
this
particular
application,
and
so
on,
because
I
know
today
they
don't
really
write
normal
general
purpose
codes
for
submitting
into
our
clusters.
A
A
B
I
Thank
you,
yeah
I
wanted
to
go
back
to
something
earlier
that
you
know
it's
a
common
pattern
that
people
use
multi-cluster,
because
one
one
control
plane
can't
take
it.
That
actually
really
speaks
to
the
fact
that
either
we
accept
that
or
and
or
we
invest
to
try
to
make
one
control
plane
larger,
which
is
probably
never
going
to
work,
but
as
a
batch
community.
I
E
E
So
yeah
I
think
that's
that's
a
great
observation
and,
and
something
we've
been
thinking
about
for
for
some
time.
I
do
think
that
at
some
level
we
need
to
start
to
think
about
how
we
do
this
Federation
for
batch
jobs
across
the
the
different
clusters,
but
in
an
upstream
point
of
view,
whether
we
enable
the
ability
to
provision
clusters
just
for
scheduling.
E
In
other
words,
you
you
have
a
scheduling,
sort
of
mini
cluster
there's
some
interesting
work
with
KPC,
which
provides
an
API,
a
kubernetes,
AP
I,
don't
know
if
folks
are
familiar
with
that
which,
which
is
something
we
may
be
able
to
to
look
at,
which
would
be
able
to
put
a
scheduler
behind
that.
That
would
would
Federate
workloads
across
the
different
clusters,
but
I
do
I
think
it
should
be
part
of
the
batch
work
group
agenda,
at
least
to
evaluate
how
we
might
approach
it,
but
I
think
there's
a
lot
more
work.
A
I,
just
have
one
quick
comment
about
this
as
well,
like
all
the
solutions
that
we've
seen
were
mostly
related
to
specific
use
cases
right
like
we
had
the
spark
implemented
for
spark,
we
are
Implement
for
specific
type
workloads
embarrassingly
parallel.
For
example,
the
challenge
is
actually
coming
up
with
something
extremely
genetic
and
acceptable
by
the
community.
D
Up,
let
me
add
on
to
what
Abdullah
said.
Also,
it's
not
just
the
cluster
that
needs
to
scale
then,
but
if
you
use,
for
instance,
the
spark
operator
or
something
else,
all
these
things
need
to
scale
up
and
what
we've
seen
with
our
users
that
have
been
using,
for
instance,
the
spark
operator
it
does
not
scale
up
that
high
either.
So
you've
already
got
multiple
instances
of
the
operator
in
the
cluster
of
a
thousand
nodes.
So
it
is
yeah
you
can.
J
Foreign,
so
quick
question
on
the
math
involved,
so
different
chips
have
different.
Some
of
them
don't
follow
the
IEEE
standard.
So
if
you
look
at
the
cell
architecture,
for
instance,
for
the
PS
PlayStations,
they
don't
follow
it
and
they're,
not
the
only
ones
that
have
different
architecture
for
floating
point.
J
C
The
queue
has
this
really
a
q
project.
That
is
really
nice
feature
which
is
called
node,
flavors
I.
Believe
you
can
comment
on
this
I
guess
you
know
better
me
whereby
you
have
some
set
of
labels.
I
believe
right
and
a
particular
set
of
labels
that
have
the
same
value
denotes
a
particular
node
flavor
and
so,
for
example,
scheduling
could
guarantee
that
all
of
your
nodes
are
scheduled
across
nodes
with
identical
node
flavors,
which
I
think
would
be
an
elegant
solution
to
problems
such
as
this,
at
least,
then.
You
are
consistently
incorrect.
K
K
D
So,
from
a
unicorn
perspective,
we
have
taken
the
approach
that
we
want
to
support
things
as
simple
as
possible:
no
cids,
nothing!
We
allow
you
to
we
schedule
based
on
annotations
that
you
put
on
the
pot
and
that's
it
the
rest.
We
use
the
standard
kubernetes
objects,
so
we've
tried
to
keep
it
as
simple
as
possible
as
close
to
kubernetes,
as
we
can
that's.
That
was
the
doll
setup
that
we
we
started
with
so
and
from
user
perspective,
we're
working
on
use
equalities
at
the
moment
we're
working
on
the
design
dock.
D
C
Yeah,
so
in
our
motherland,
we
have
the
notion
of
a
job
that
we
use
to
start
off,
essentially
a
bag
of
kubernetes
objects,
and
so
you'll
give
us
your
bag
of
kubernetes
objects,
and
you
say
one
of
them
is
my
main
objects
and
whenever
that
object
then
dies
for
whatever
reason,
typically,
support
that
runs
through
completion.
We
just
kill
all
of
the
other
resources
and
user
identities
is
based
entirely
around
queues.
So
when
you
give
us
your
back
objects,
you
tell
them
put
all
of
these
bags.
C
E
A
I
guess
one
like
one
major
difference
between,
of
course,
Services
The
Bachelor,
because
the
services
like
you
have
I,
don't
know
the
user
has
there's
a
service
which
is
a
web
page
right
with
that
the
user
is
actually
trying
to
do
something
with
the
system
directly
they're
trying
to
submit
something,
and
at
least
what
I
observed
in,
for
example,
machine
learning.
They
have
sdks
right
like
they
start
from
an
SDK.
They
Define
everything
in
the
code
in
Python
Etc,
that's
their
API
for
submitting
jobs,
and
then
everything
after
that
is
handled
by
the
SDK.
A
It
creates
kubernetes
objects,
etc,
etc.
I
don't
know
if
that,
like
model
can
translate
to
something
like
with
slurs
the
same
thing
you
have
probably
these
command
line
interfaces
that
people
grow
like
used
to
like
Slayer,
submit
and
all
these
things
and
I
guess
it
depends
on
the
type
of
workload
but
I
feel
with
batch.
A
F
D
Yes,
there's
a
huge
push
for
multi-tenancy
support
and
that's
why
we're
also
looking
at
the
decided
to
use
the
qualities
and
all
that
kind
of
stuff.
So
we
see
that
push
from
from
the
uses
that
we've
got
and
then
multi-tenancy
doesn't
mean
for
for
them
multiple
companies,
but
even
within
companies,
there's
different
groups
that
get
a
quota
assigned
or
different
users
that
have
got
a
quota
assigned
so
yeah.
We
we
do
see
that
and
that's
huge
pressure.
C
Yeah
indeed,
also
in
Armada,
maybe
we
have
many
users
internally
and
there's
a
lot
of
like
it's
very
important,
obviously
to
the
users
that
each
of
them
get
their
fair
share,
because
they
will
come
and
complain
to
us
if
we
don't
give
them
their
fair
share.
And
so
for
this
reason
we
do
have
all
of
these
things.
We
have
sort
of
per
user
quotas.
L
L
E
Haven't
really
looked
at
least
our
group
hasn't
really
looked
at
the
replacing
the
back
end,
but
definitely
the
you
know,
that's
the
big
scare.
That's
the
motivation
that
some
of
our
end
users
have
multi-clusters
is
just
essentially
to
handle
those
limitations.
So
it's
a
good
point
and
I
don't
know
any
if
there's
any
work
around
that,
but
it's
very
valid
and
I
think
you're
right
once
that's
the
biggest
driver
for
multi-clusters.
On
our
end.
B
I
should
I
should
say
that
part
of
the
motivation
for
creating
the
the
flux
operator
was
to
was
to
avoid
a
perceived
limitation
of
FCD
as
well,
in
the
sense
that
we
would
create
this
batch
system
with
inside
of
kubernetes.
It
then
can
subdivide
the
resources
and
handle
the
same
number
of
tasks
that
flux
natively
can
handle,
which
is
tens.
Hundreds
of
thousands,
perhaps
millions
of
tasks.
A
I
mean
this
goes
back
to
the
first
question
they
asked
is
actually
hcd
a
fundamental
part
of
kubernetes
Canada
is
not
right,
like
it's
just
the
storage
solution
that
they
chose
at
the
beginning,
but
it
can
be
replaced
with
something
that
is,
you
know
infinitely
scalable
globally,
scalable
right.
We
have
such
Solutions
in
our
Cloud,
multiple
Cloud
providers.
If
we
just
adopt
the
hcd
API
to
those
storage
solutions
that
are
globally
scalable,
it
will
solve
most
of
these
problems.
M
Hi
I'm
just
wondering
if
there's
an
opportunity
for
other
projects
like
slurm
to
to
actually
work
with
the
kubernetes
community
to
develop.
You
know
cues
and
different
apis,
so
that
slurm
could
actually
do
a
lot
of
the
scheduling
and
resource
control
that
people
are
asking
from
the
batch
Community
but
treat
a
kubernetes
cluster
as
a
resource.
M
A
But
I
guess
is
this
most
of
these
schedulers
they
did,
they
were
not
built
for
kubernetes
I
guess
unicore
was
was
unique,
like
unicorn
is
component
outside
kubernetes
same
thing,
with
with
your
scheduler,
you
just
use
the
scheduler
plugins
to
basically
offload
the
scheduling
or
like
call
call
outside
into
an
external
service,
can
be
done.
The
same
thing
with
slur
so
like
I
mean
what
I'm
trying
to
say
is
that
that
model
is
actually
being
adopted
in
multiple
other
projects.
Other
than
slur.
N
First
I
wanted
to
comment
on
FCB
I.
Remember.
One
of
the
topics
of
the
last
computer
Summit
in
Valencia
was
that
at
CV
was
in
a
crisis
trying
to
find
a
maintainers.
So
I
guess
my
question
is:
are
we
not
doing
enough
as
a
batch
Community
to
you
know,
go
back
to
to
hcd
in
this
case
and
improve
it
for
our
needs
and
same
about
applications
like
spark
or
yarn
and
MPI?
N
Those
are
all
very
old
Frameworks
that
kind
of
assume
that
they
have
like
certain
characteristics
and
in
like
they
don't
know
about
Parts.
They
just
assume
they're
running
in
bare
metal.
Even
they
assume
everything.
Is
you
only
run
when
everything
is
already
set
up,
and
should
we
be
reaching
back
to,
let's
say
open
API,
Intel,
API
or
Spark
maintainers
to
actually
help
them
or
like
help
them
be
more
Cloud
native
to
be
more
resilient
to
failure?
N
I
recently
was
in
a
discussion
also
with
Tim
Hawking,
where
there
was
a
cap
about
discussing
a
startup
order
of
containers
and
Tim
cooking
was
saying
well,
the
containers
should
be
resilient
to
not
having
all
the
setup
already
there,
but
that's
not
the
reality.
Right
MPI
operator
has
that
problem
and
I
don't
know
about
spark
too
much,
but
so
back
to
my
question
like
are
we
doing
enough
reaching
back
to
those
communities?
Are
we
helping
them
move
forward,
so
it
they
simplify
the
things
for
us
as
kubernetes
developers
and
so
on.
B
So
I
just
wanted
to
comment
on
the
the
MPI
at
least
I.
Think
that's
a
really
really
interesting.
Point.
I
I
think
there
are
efforts
in
MPI
to
make
it
more
to
make
the
communicator
more
flexible
in
a
sense
of
dynamically,
adding
and
removing
ranks,
but
I
I,
don't
know
where
that
is
I,
don't
think!
That's
production,
ready,
I
think
that's
mainly
research
at
this
point,
but
I
think
that
could
be
hugely
hugely
transformative.
E
D
Spark
from
its
side
is
pretty
flexible,
so
you
can
tell
it
to
start
up
with
a
minimum
number
of
executors
and
grow
to
a
maximum
number
of
executors
and
do
all
that
kind
of
stuff.
For
you
in
the
background,
but
the
end
user
needs
to
ask
for
it
and
if
they
don't
do
it,
then
you
don't
get
it.
So
it's
it's
partially
also
a
education
of
the
end
users
to
say
this
is
how
you
submit,
and
this
is
what
you
set
up.
That
makes
it
easier
for
you.
D
We
still
see
people
that
run
spark
jobs
and
then
all
the
work
is
done
in
the
driver
but
they're
asking
for
10
other
executors
and
waste
an
enormous
amount
of
resources,
and
then
you
tell
them
and
say
hey.
This
is
what
you're
doing
and
they
say.
Oh,
wasn't
the
way
that
I
was
doing
that
they
don't
have
the
view
they
don't
look
at
it.
My
way
said
hiding
a
lot
of
the
things
and-
and
they
just
want
to
submit
the
job
and
want
to
run
quickly.
Anybody
can
get
the
results
back.
D
D
The
people
just
submit
the
job
they
want
to
get
the
result
back.
They
don't
look
at
how
have
our
submitted
this
and
how
many
resources
database
up
until
you
tell
them
here's
the
bill
in
the
end,
and
then
they
start
complaining
so,
but
they
they
don't
get
that
because
there's
no
use
of
accounting
there's!
No!
So
there's!
If
we
provide
that
information
back
to
them,
we
can
educate
them
and
they
will
run
better.
D
O
We
talked
a
lot
about
scheduling,
but
storage
is
another
problem
that
I'm
really
struggling
with
this
I've
spoken
to
people
here
about
it
already
I'm
kind
of
curious
about
that
idea
of.
Do
we
think,
there's
a
way
to
better
correlate
the
like
scheduling,
user
or
information
about
the
user
into
kubernetes
itself,
or
do
we
feel
like
we
have
to
keep
relying
on
building
external
things
around
it
and
kind
of
parallel
onto
that
like?
C
I
H
Using
the
fact
that
nobody
wants
to
ask
a
question,
I
just
wanted
to
throw
a
comment
defending
at
CD
now
this
is
this
is
as
I'm
working
on
Google
kubernetes
engine
and
our
internet
testing
pipelines,
test
clusters
up
to
15
000
nodes
by
cluster,
the
largest
production
clusters
that
our
customers
have
are
to
have
12
000
nodes,
all
spots,
so
very
heavy
on
like
Recycling
and
preemption.
So
like
this
is
a
very
powerful
tool
like
we
need
to
fix
a
couple
of
problems
like
that.
H
F
F
P
I
was
going
to
do
make
a
similar
comment,
which
is
a
couple
of
months
ago.
People
realized
that
that
city
was
left
with
one
maintainer,
but
that's
that
was
a
problem
with
the
project,
but
it
was
also
a
show
of
the
community
that
has
been
built
around
these
tools
because
as
soon
as
there
was
a
call
for
help,
there
were
a
bunch
of
companies
that
jumped
in
and
actually
volunteered
resources
to
fix
it,
and
maybe
at
CD
is
the
current
problem.
P
But
this
problems
keep
repeating
themselves
with
a
bunch
of
tools
that,
after
the
hype
and
when
they
become
kind
of
stable,
lose
interest
from
a
lot
of
people.
So
we
are
seeing
it
with
etcd,
but
we'll
see
to
it.
Any
number
of
tools
that
will
keep
coming
so
jumping
out
of
a
solution
is
sometimes
not
the
best
option.
Maybe
maybe
there
is
room
to
fix
it
and
then
I
had
another
comment
which
was
regarding
the
the
opportunity
of
running
batch
on
kubernetes.
P
There
are
many
options,
as
we've
heard
today,
but
I
had
a
question
because
I
come
from
an
environment
where
we
were
doing
massive
scale
Computing
before
the
cloud
appeared,
and
we
built
a
bunch
of
custom
tools
for
two
decades
that
we
still
run
and
moving.
Those
to
kubernetes
has
been
a
massive
simplification
and
it
also
democratizes
a
bit
the
access
to
a
large
amount
of
resources
with
the
public,
Cloud
I'm
I,
wonder
how
much
of
an
effort
should
be
in
these
common
apis.
P
P
How
do
you
see
the
the
benefit
of
actually
putting
a
lot
of
effort
into
having
this
single
extensions
to
the
job,
API
and
and
some
sort
of
single
scheduler,
instead
of
just
relying
on
the
base,
kubernetes,
API
and,
and
then
people
choose
whatever
they
choose
on
top?
How
much
effort
do
you
think
we
get
how
much
benefit
we
you
think
we
get
from
this
potential
effort.
E
So,
from
our
point
of
view,
I
think
it's
a
lot
of
benefit.
You
can,
as
you
hear
the
talks
today
and
from
the
other
fellow
panelists,
there's
a
lot
of
common
challenges
that
we
have
and
to
be
able
to
solve.
As
many
of
those
we
can
with
one
code
base
and
have
a
community
that
supports
that
same
code
base
I
think
it
becomes
more
effective,
we're
become
more
efficient
right
and
we
can
evolve.
It
I
think
a
lot
quicker,
so
I
think
there's
a
lot
of
benefit.
E
I
I
guess
a
little
bit
more
on
the
NCD
thing,
I
actually
think
like
for
us.
It's
always
never
been
NCD.
The
API
server
can
go
down
right
because
you
have
like
a
Thundering
Herd,
because
you
have
so
many
nodes,
and
you
know
there's
a
lot
of
these
lessons.
We've
learned
internally,
we've
learned
from
each
other
just
inside
of
our
company.
Unfortunately,
we
mostly
work
proprietary,
I
kind
of
have
a
question
like
this
has
been
an
incredible
Forum,
but
it's
been
like
one
one
day,
maybe
one
two
days
out
of
the
year.
I
B
I'll
just
mention
that
there
are,
there
are
containers
in
HPC
and
orchestrated
containers
in
HBC
workshops
at
the
super
Computing
conference
as
well,
and
that's
starting
to
gain
traction.
We're
starting
to
see
a
lot
more
users
in
that,
so
coming
from
that
direction
would
be
really
interesting
too.
I
think
those
users
that
are
used
to
kind
of
bare
metal,
traditional
HPC
would
benefit
a
lot
from
hearing
some
of
these
presentations
too.
E
I
want
to
apologize
for
laughing,
but
Alex
was
back
there
waving
his
hands,
so
yeah
I
I
I,
want
to
thank
everybody
for
doing
the
presentations
and
coming
in
and
sharing
their
their
thoughts,
ideas,
requests
and
I
again.
I
just
want
to
encourage
you
to
come
to
the
batch
work
group
and
help
us
at
least
try
to
solve
some
of
your
issues
or
note
be
able
to
identify
those.
So
we
can
make
recommendations
or
hopefully
Upstream
some
changes
and
I.
Think
that's
it.
Thank
you.
D
Yeah
last
Mark
also
I
think
also
from
an
Apache
unicorn,
we'll
we're
seeing
way
more
traction
out
of
the
old
big
data
kind
of
workloads,
kind
of
moves
over
onto
the
kubernetes
side.
So
there's
there
is
demand
now
from
that
side.
So
we'll
see
more
of
that
and
I
think
when
I
looked
at
all
the
presentations
today,
we
all
have
the
same
kind
of
limitation
on
either
the
job
API
or
so
it's
it's
not
that
we
want
to
do
different
things.
We
just
need
to
get
together
in
the
group
and
then
move
on.
C
Yeah
so
similar
for
me,
we've
built
Armada
because
we
needed
to
address
what
we
perceive
to
be
limitations
in
kubernetes,
but
over
time,
as
kubernetes
can
natively
sort
of
address
this.
We
would
love
to
come
back
closer
to
just
straight
up,
regular
kubernetes
again
as
close
as
we
possibly
can,
because
we
want
to
use
the
standard
interfaces.