►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Big Data SIG - Erik Erlandson, Red Hat & Yinan Li, Google
This presentation will cover two projects from sig-big-data: Apache Spark on Kubernetes and Apache Airflow on Kubernetes. Kubernetes became a native scheduler backend for Spark in 2.3 and we have been working on expanding the feature set as well as hardening the integration since then. Apache Airflow on Kubernetes achieved a big milestone with the new Kubernetes Operator for natively launching arbitrary Pods and the Kubernetes Executor that is a Kubernetes native scheduler for Airflow. We will give an overview of the current state and present the roadmap of both projects, and give attendees opportunities to ask questions and provide feedback on roadmaps.
To learn more: https://sched.co/Greq
A
A
My
name
is
Eric
and
I
work
at
Red,
Hat
and
my
partner
here
is
a
he
nan
works
at
Google
and
two
of
the
co-chairs
on
the
sig
Big
Data.
We
talk
about
the
sig
today,
so
the
basic
landscape
of
the
talk
I'm
just
going
to
briefly
introduce
what
the
sig
is
about.
I'm
gonna
we're
gonna
cover
a
couple
of
the
longer-term
engagements
we've
had
over
the
last
couple
of
years:
Apache
air
flow
and
Apache
spark.
A
Then
I'm
gonna
circle
back
to
the
cig
and
talk
a
little
bit
about
some
of
the
trajectories
and
the
possible
futures.
And
then
you
can
see
at
the
bottom
here.
I
called
that
audience
dialogue.
So
we
are
hoping
to
leave
a
little
bit
of
extra
time
to
get
feedback
from
you
guys
about
possible
directions.
We
can
take
the
cig
or
new
projects.
A
So
the
the
main
function
of
the
cig,
since
2017,
has
been
to
sort
of
serve
as
a
community
resource
for
advising
other
projects.
In
our
case,
focusing
on
you
know,
big
data
or
data
science
software
projects
and
just
to
serve
as
a
reservoir
of
expertise
and
help
for
how
helping
these
people
integrate
with
kubernetes
and
then
in
turn.
A
Our
other
goal
was
to
you
know,
represent
the
concerns
of
these
users
back
to
kubernetes,
in
the
form
of
you
know,
driving
possible
new
features
or
enhancements,
shepherding,
PRS,
etc,
and
the
text
you
see
above
here
is
actually
the
exact
text
I
submitted
via
this
PR
at
the
bottom,
as
the
official
cig
mission
and
I'll
be
circling
back
to
this
PR
later.
At
the
end,.
A
A
A
Since
then,
we've
actually
moved
the
coop
scheduler
backend
upstream
to
apache
spark
community.
We've
also
done
some
work
on
HDFS
deployments
inside
kubernetes,
and
also
helped
the
apache
airflow
community,
create
operators
and
executor
x'
to
work
with
coop
we've
had
a
pleasingly
diverse
community
involvement
and
the
above
seven
companies
here
are,
like
my
own
scientific
assessment
of
who's
like
been
most
consistent
attending
if
I've
left
anybody
off.
That
is
only
because
I
want
to
try
and
complete
the
talk
on
time,
so
I'm
gonna
go
into
the
first
engagement,
which
was
with
a
patchy
airflow.
A
Airflow
has
kind
of
like
an
opinion
about
its
compute
model
at
these
computation
as
workflows
of
tasks
in
each
task
is
allowed
to
have
dependencies.
Now,
if
anybody
who
comes
out
of
computer
science
will
know
that
the
data
structure
represented,
add
is
a
directed
acyclic
graph,
you
can
see
the
arcs
of
the
graph
are
pointing
from
right
to
left
as
traditional
and,
however,
when
you
actually
run
it
execution
logically
precedes
them
from
left
to
right.
A
So
anybody
also
familiar
with
the
UNIX
make
will
realize
that
this
is
essentially
UNIX
make
after
taking
a
lot
of
steroids
and
doping
and
such
like.
So
the
basic
unit
of
work
for
airflows
called
an
operator
corresponds
to
like
executing
a
single
command
or
a
function
generally
associated
with
a
unique
task.
Id,
and
it
is
intended
to
contain
all
the
parameters
or
other
resources
it
needs
to
execute
operators
come
in
many
flavors.
You
can
literally
run
a
bash
command
invoke.
A
A
A
It
also
is
concept
of
executor.
An
executor
is
really
nothing
but
a
selection
of
an
environment
that
will
actually
run
the
task,
so
the
basic
one
is
literally
just
executing
locally
on
a
machine.
It
also
has
one
for
executing
tasks
instead
of
mesos
and
now
more
recently
inside
of
kubernetes,
so
the
kubrana
is
operator
or
kubernetes.
A
Pod
operator
is
I
guess,
in
my
opinion,
these
sort
of
keystone
of
the
integration
with
Kubb,
and
you
can
see
it
contains
very
common
things-
common
to
all
operators
like
a
name,
a
task
ID,
but
then
it
also
contains
fields
for
container
image
and
a
command
to
run
on
that
image
and,
of
course,
that's
basically
the
core
functionality
that
allows
you
to
run
inside
of
the
cuve
environment.
You
can
see
also
that
there
are
other
kubernetes
features
supported
labels,
secrets
volume
mounts,
so
it's
got
a
fairly
full-featured.
A
And
upcoming,
there
is
going
to
be
support
for
a
airflow
communities
operator,
and
so
it's
in
a
sense,
the
fullest
possible
coop
integration.
It
actually
runs
the
scheduler
and
the
user
interface
and
the
sequel
database
inside
of
kubernetes,
and
so
all
the
components
are
actually
running
in
Kubb
and
then
executing
the
workflows.
Also
inside
of
the
same
cluster.
A
They
can
run
into
the
megabytes
and
a
user
could
have
thousands
of
them
as
part
of
their
workflows
and
so
you're
talking
about
actually
large-scale
data
and
one
of
the
things
that
the
airflow
community
put
a
lot
of
effort
and
design
into
is
supporting
different
ways
to
stage
these
DAGs
to
make
them
available
to
containers
running
coop.
You
know
in
a
scalable
way,
and
here
just
a
few
of
the
ones
that
you
know
you
might
see
yourself
running
into,
and
it's
also
extensible.
A
So
you
can
add
your
own
ways
of
staging
via
hooks
so
that
that's
the
engagement
we
had
with
airflow
and
you
know
what
I
gave
you
is
vastly
incomplete
and
if
you're
actually
interested
in
airflow
I
encourage
you
to
check
out
the
recording
of
Daniel
in
Berman's
and
Barney.
So
the
theorem
I
think
is
his
name
I'm
a
butcher
dad
and
I
apologize,
but
they
gave
that
talk
yesterday
and
they
go
into
much
more
detail
to
the
Anya
hair
flow
on
Cooper
DS
effort
I'm
gonna
hand
it
over
to
a
Ian
on
who's.
B
I
should
before
I
start
just
wanna
get
it
since,
like
you
know
how
many
users
are
Co,
how
many
of
you
are
already
running
a
spark
on
khorinis?
Okay,
it's
quite
a
few
okay
great.
So
before
you
know,
I
dive
into
the
details
of
running
spark
on
kubernetes
I
would
just
want
to
get.
You
know
a
sense
of
what
smart
really
is
like
you
know,
I
want
to
talk
about
a
smart
computer
model,
so
on
a
high
level
like
you
know,
you
have
a
spark
job.
You
have
a
bunch
of
input
data
to
process.
B
This
is
a
really
really
simple
example.
Like
you
know,
you
have
a
bunch
of
array
of
numbers
as
the
input
and
you
tell
like
no
spar
how
many
executors
I
want
in
this
example,
like
you
know,
so
he
wants
reacts
executors.
So
the
spark
task
scheduler
will
try
to
like.
You
know,
even
for
a
distribute
that
that
you
know
the
tasks
among
these
executors,
so
the
driver
is
in
a
position
to
actually
manage.
You
know
the
executors
so
saying,
except
in
this
perfect
example,
you
know
each
exterior
gets
three
numbers
to
to
to
process.
B
So,
let's
say,
like
you
know,
the
compute
compute
we
want
to
do
is
really
simple
I.
Can
you
just
wanted
to
a
doubles
of
the
input
numbers?
So
each
executor
gets
free
numbers
process.
Then
you
know
they
generate
some
output.
This
is
all
orchestrated
by
the
driver
which
is
sort
of
like
you
know
the
master
kind
of
thing.
So
when
you
put
these
things
onto
a
container,
that's
basically
how
everything
works.
B
B
Your
driver
actually
runs
your
so
the
driver
runs
your
your
men
program
and
you
know,
is
responsible
for
actually
creating
executors
and
managing
them,
and
you
know
these
are
being
a
task
to
the
executors.
So
this
is
how
you
know
basically
like
the
entire
flow
of
what
happens
when
you
run,
the
spark
uncle
increment
is
done
on
a
unicloset
mode,
so
the
user
runs
the
spark
some
eight
script.
So
what
happens
like
you
know
when
you
give
a
master
URL?
B
That
starts
with
Kade
s,
so
this
is
what
have
like
you
know,
so
the
sparks
am
a
script,
will
called
kubernetes
specific
submission
client.
So
this
salvation
submission
client
will
actually
try
to
create
a
pass
back
from
initial
prospect
and
goes
through
a
bunch
of
steps
to
actually
configure
the
pass
spec
towards
you
know
the
final
state.
So
in
my
you
know
there
might
be
steps
for
like
in
mounting
wallings,
more
monthly
secrets.
Weren't,
you
know
doing
other
kind
of
things.
So
basically
you
know
so
each
step
modifies
the
prospect
in
certain
way.
B
B
B
For
the
driver,
so
basically
what
the
submission
client
will
also
create
a
headless
terminated
service
for
the
driver,
so
it
you
know,
give
us
it
a
fqdn
name,
okay.
So
this
is
what
we
have
done
so
far
like
you
know
regarding
the
kubernetes
schedule
back
in
for
spark,
so
we
have
the
first
release
in
spark
2.3
with
initial
support
for
cluster
mode.
B
With
you
know,
language
binding
for
Java
and
Scala,
and
also
with
support
using
remote
indices
on,
for
example,
HTTP
servers
were
or
s3
or
HDFS
or,
like
you
know,
Google
Cloud
storage,
and
we
also
support
limited
set
of
customizations
on
the
on
a
pause
on
the
driver
and
extra
parts.
For
example,
you
can
actually
specify
an
environment
variables,
labels
annotations
those
kind
of
things,
so
we
you
know
actually
suggest
months
ago,
spark
2.4
was
released,
so
this
release
actor
came
with
looking
a
more
feature
for
the
kubernetes
schedule:
back-end,
for
example.
B
Now
you
can
actually
run
Python
or
are
smart
jobs
on
kubernetes,
and
this
release
also
comes
with
limited
climate
support.
I
will
talk
about
that
later,
so
why
I
say
it's
limited,
so
in
Sparks
3.0,
which
will
be
the
next
release
of
spark
will
have
more
features
in
it
like
Ana,
so
we'll
have
Kerberos
support,
so
you
can
actually
use
its
security
FS
for
your
input
or
output,
and
we're
also
adding
support
for
using
pod
templates
to
do
pod
customization.
So
we
won't.
That
means
like
a
no.
We
won't
be
adding
new
spark.
B
Config
options
for
like
in
a
doing
communities,
plot,
pod,
customizations,
so
climb
mode
is
actually
pretty
useful
for
a
lot
of
seems,
like
you
know,
you
can
use
Hana
mode
to
run,
spark
shell
or
drew
bitter
notebooks,
for
example,
so
we
actually
support
running
the
drivers.
The
driver,
pods
or
acting
on
did
not
pass
like
driver
both
inside
and
outside
of
clusters.
So
when
the
driver
runs
inside
the
cluster
or
acts
would
be
running
in
a
pod.
B
So
when
you
actually
use
the
climb
mode,
there's
a
few
things
to
take
into
mind
like
you
know,
so,
the
first
of
all.
If
you're
running
the
driver
part
inside
a
if
you're
running
the
driver
in
side
the
cluster,
so
when
you
are
so
you
the
better,
is
that
you
give?
You
know
you
tell
spark
what
is
the
name
of
your
driver
pod?
So
in
our
case
we
actually
set
up
like
a
no
garbage
owner
references
on
the
exterior
parts
created
by
the
driver.
B
So
then
this
will
allow
a
garbage
collection
to
kickin
when
you
know
when
when
the
job
is
done
so
what
world
you
know
when
we
actually
get
rid
of
the
driver
part,
so
the
exterior
parts
will
be
deleted
by
a
garbage
collector.
So
when
you
are
running
the
driver
outside
the
cluster,
it
won't
be
running
apart.
So
now,
what
do
you
need
to
do
is?
Actually
you
want
to
make
sure
that
you
have
connectivity
from
your
executor
powers
running
in
a
cluster
to
the
driver,
which
is
you
know
now
outside
the
cluster?
B
So
we
don't,
like
you,
know
we're
not
really
opinionated
on
how
people
will
do
connectivity.
So,
in
this
case,
it's
actually
on
your
hand,
like
you
know,
so,
if
you're
running
the
driver
inside
apart
inside
the
cluster,
you
want
to
have
a
health
service
for
your
driver,
so
you
have
a
fully
qualified
DNS
name
for
your
driver,
so
because
actually
the
Acura
P
X
series
will
be
using
our
fqdn
name
to
connect
to
the
driver.
So
when
you're
running
your
driver
outside
the
cluster,
well,
it's
you
know
it's
much
more
trickier,
but
still
you're.
B
You
have
to
make
sure
that
you
know
this
connectivity
is
set
up
properly,
so
the
exterior
of
house
can
actually
talk
to
the
driver
yeah.
So
this
animation
basically
shows
like
you
know.
If
you
have
a
driver
inside
a
cluster,
you
want
to
create
a
header
this
service,
for
that,
so
that
you
know,
executors
can
actually
use
that
to
talk
to
the
driver,
so
Kerberos
supports
will
coming
sparks
3.0.
So
this
is
really
you
know,
critical
for
using
secure
HDFS
for
input
or
output.
So
to
use
this
feature,
you
actually
need
two
things.
B
One
is
the
dedication
token.
The
other
is
actually
a
custom
header
configuration
that
works
with
the
way
you
set
up.
You
know
security
efforts,
so
this
feature
doesn't
yet
support
dedicated
totally
new
renewable.
That
means
extra
you
by
default,
dedicated
telecom
will
expire
in
24
hours.
It
means
that
here
you
have
to
make
sure
that
your
job
finish.
You
know
in
24
hours,
because
otherwise
the
dedication
token
will
expire.
You
won't
have
be
able
to
access
your
HDFS
class
or
any
anymore.
B
So
this
there's
actually
two
ways
you
can.
You
know
tell
us
part
what
kind
of
a
so
how
do
how
to
actually
get
access
to
a
dedicated
token
one
ways
you
give
the
sparks
of
summation
client
a
key
tab,
so
the
simply
summation
client
will
actually
use
that
key
tap
to
do
the
logging
and
then
create
a
secret
that
carries
the
dedicated
token
or
optionally.
You
can
also
bypass
this.
You
can
just
tell
spark
okay.
B
This
is
factor
the
secret
I
want
to
use
which
had
already
has
the
dedication
token
stored,
well,
the
same
clumps
for
configuration.
So
one
way
is
like
you
know
so:
yeah
you
basically
set
the
environment.
Verbal
had
took
off,
I
was
sorry,
I
forgot,
what's
the
exactly
the
name
for
that
environment,
verbal,
but
also
a
sectoral
hot
cod
had
took
oh
yeah,
so
it's
actually
had
a
comforter
comforter.
So
you
said:
are
you
burn
but
we're
about
to
a
local,
Hadoop
conf
directory?
B
So
then
the
submission
client
work,
for
you
start
to
read
all
the
config
files
under
that
direction
and
create
a
config
map
to
actress
stores.
You
know
the
the
config
and
then
mount
it
into
the
driver
and
Exeter
pods,
so
also
optionally.
You
can
actually
use
your
own
config
map,
like
you
know,
if
you
have
a
config
that
map
that
stores
your
Hadoop
config.
So
you
can
also
use
that.
So,
in
addition
to
like
the
native
integration
of
spark
with
kubernetes,
we
also
working
on
this,
the
so
kakuni
the
operator
for
spark.
B
So
this
is
actually
like.
You
know
leveraging
operator
pattern.
That
was
that's
actually
becoming
really
popular
these
days.
So
basically,
you
will
have
a
CR
D
and
you
know
a
custom
controller
directory.
One
spark
some
it
for
you,
so
you,
basically
you
define
your
application.
Declaratively
llamo
spec,
for
example,
and
you
actually
just
you
know,
create
CRS
to
trigger
spark
submission,
so
they
also
like.
So
it
actually
has
you
know
a
richer
support
for
part,
because
the
customization
beyond
was
we're
currently
able
to
do.
That
is
actually
done
through
a
mutating
web
hook.
B
So
it
also
has
native
cron
support
for
running
time-based
jobs.
Also,
like
you
know
so,
integration
knitting
integration
with
prometheus
for
metrics.
You
can
actually
export
like
a
knobos
application,
metrics
and
also
driver
or
executors
of
metrics
to
Prometheus
yeah.
So
it
also
comes
with
a
custom
command
line
tool
called,
which
is
called
Smart
control
for
doing
this
things
like
now,
you
can
actually
use
that
to
do
a
port
forwarding
really
easily.
You
don't
even
need
to
know
the
name
of
the
pod
so
because,
actually
so
well,
sport
control
and
will
get.
B
You
know
the
driver
power
name
from
the
CR,
their
customer
resource.
So
then
we'll
be
able
to
actually
to
forwarding
for
you,
for
example.
So
this
is
like
a
no
soul.
I
wrote
a
simple
example
of
one
spark:
a
vacation,
spec,
ok,
so
in
terms
of
roadmap,
like
what
I
just
said
so
in
Sparks
2
points,
3
will
have
support
for
pot
I'm
bleeding
like
an
ax,
so
you
can
actually
supply
a
pala
template
for
the
driver
or
executor.
B
So
you
can
do
things
like
know
like
you
can
set
a
pot
infinity
or
infinity,
or
you
can
actually
use
past
security
context,
for
example,
so
which
are
not
currently
supported.
So
the
mean
structure
will
no
longer
add
more
spark
config
options,
people
well
what
the
reason
is
we
already
have
a
bunch
of
you
know
different
kubernetes,
specific
configure
options.
B
We
don't
want
to
add
more
because
that
you
know
so,
but
the
more
you
have
them
the
more
difficult
to
manage
so
dynamic
resource
allocation
is
also
on
a
road
map
like
so
with
dynamic
resource
allocation.
You
will
be
able
to
actually
you
know
so.
Spark
will
be
able
to
adjust
the
number
of
executors
at
runtime
based
on
the
load
or
than
you
know
the
amount
of
tasks.
So
this
also
requires
the
so
called
external
shuffle
service.
So
we
have
a
new
shelf
of
service
design
in
progress
actually,
so
we
also
plan
to
work
on.
B
You
know
better
support
for
local
application
dependencies.
So
right
now,
the
way
you
do
with
dependencies
is
either
you.
You
know
you
have
a
custom
image
that
you
bake
your
advances
in,
or
you
put
your
dependencies
somewhere
remotely
like
an
oil
on
s3
or
GCS.
So
what
if
like
a
know,
so
you
have
some
dependencies
on
your
local
machine.
You
want
to
use
those
the
first
thing
you
have
to
do
it
like
you
know
you
have
to
stage
somewhere
either
you
know,
make
it
into
the
inter
York
image
or
put
it
somewhere.
B
So
we
want
to
have.
You
know,
provide
better
support
for
these
kind
of
things,
so
you
don't
need
to
worry
about,
like
you
know
the
staging
at
all,
so
the
other
thing
is
like
anaphors
bar
streaming
on
driver
resilience
like
or
org
is
really
important.
So
you
want
to
make
sure
that
your
driver
stays
up
even
though
there's
like
an
ax,
so
fader
you,
there's
restart
and
being
handled
so
we're
also
working
out
foot
but
with
seek
scheduling
to
you
know,
have
better
support
for
scheduling
for
batch.
We're
closing
general,
but
that
also
will
benefit.
B
Spark
spark
is
actually
pretty
unique
in
the
sense
that
if
you
only
have
resource
to
run
the
driver
there's
once
the
driver
star,
you
don't
have
any
resource
to
run
your
executors,
but
the
job
will
be
hunting.
So
basically
job
will
not
make
any
progress,
but
the
driver
is
be,
is
they're
staying.
So
we
want
to
make
sure
that
the
schedule
can
take
this
into
consideration
like
now.
So
you
want
to
make
sure
that
you
have
enough
resources
to
run
at
least
your
driver
plus
one
executor,
so
you
can
actually
make
progress
yeah.
B
So
if
you
want
to
get
involved
in
this
project,
you
can
actually
check
out
the
code
at
you
know.
This
official
spark
repo
what
Dakota
we
enter.
Resource
managers,
slash
kubernetes
and
we
have
you
know:
documentation
zhan
on
a
facial
spark
site
and
spark
because
it's
a
party
project
it
actually
prefers
using
user
and
main
enlisted
floor.
You
know
questions
or
or
general
discussions,
so
we
also
have
you
know
we
also
use
JIRA
for
for
feature
requests
or
bug
reports.
B
A
Thanks
T
non
so
I
think
these
two
projects
have
shown
an
example
of
a
kind
of
common
trajectory,
I've
seen
so
far,
which
is
that
once
you
kind
of
get
over
the
hump
and
get
some
pretty
good
feature
full
integration
with
Kubb,
you
tend
to
see
like
you
know,
the
discussion
move
away
from
our
sig
meetings
and
kind
of
back
upstream
and
I've
been
calling
it.
You
know
graduating
which
some
projects
actually
have
a
concept
of
graduation
there's
no
formal
process
of
that
at
all.
A
But
you
know
it's
part
of
the
lifecycle
and
I'm
actually
pleased
that
it's
happening,
but
it
does
mean,
of
course,
that
we
want
to
keep
a
sort
of
pipeline
of
reaching
out
to
new
projects.
So,
just
a
couple
weeks
ago
we
got
a
demo
of
a
flank
operator
for
kubernetes,
and
this
coming,
Wednesday
hazal
cast
is
going
to
be
demoing,
their
in-memory
data
grid
and
jet
stream
processing
on
Kubb
recently,
as
I
mentioned
earlier
friend,
pull
requests
we
submitted.
A
They
have
formal
charter
for
sig,
big
data,
and
it's
currently
up,
and
you
can
see
the
QR
code
will
take
you
to
that
URL
and
so
far
the
main
feedback
we've
had
is
that
kubernetes
governance
officially
defines
a
sig
as
owning
some
component
or
subsystem
of
a
kubernetes,
and,
as
you
can
see
from
our
description,
we
actually
haven't
owned
code
and
that
way
for
the
sig,
and
so
we're
currently
discussions
with
the
steering
committee
about
what
to
do.
What
can
we
do?
A
We
could
acquire
ownership
of
some
code,
we
might
recategorize
ourselves
as
a
kubernetes
working
group
or
sub
project,
or
possibly
even
a
new
kind
of
a
some
new
kind
of
user
community.
That
fits
the
description
of
what
it
is,
we're
actually
doing
and
then
also
even
I
discussed
with
one
of
the
other
steering
committee
members
grandfathering
the
some
of
the
older
SIG's
that
never
had
this
as
well.
So
all
these
are
possible.
Options
were
probably
sort
them
out.
A
After
all,
the
dust
settles
in
the
new
year
and
we
are
continuing
to
meanwhile
reach
out
to
people.
So
if
you
know
of
a
user
community
or
project
that
wants
to
integrate
better
with
Kubb
we'd,
be
more
than
happy
to
have
you
guys,
attend
and
get
in
touch
with
us
and
there's
our
contact
handles
for
just
email
and
then
on
kubernetes
slack,
comm
and
I'm
gonna
turn
it
over
to
you
in
the
audience.
A
C
So
my
name
is
Katun
I
work
with
lyft
we've
been
working
on
the
spark
operator
within
an
and
it's
been
it's
been
going
great
we
will
be.
Has
yourself,
you
mentioned
flink
operators,
some
work
going
on
on
our
side
and
then
there's
some
other
stuff
that
we
are
actually
doing.
That
kind
of
replaces
air
flow
for
most
of
our
cases,
but
I
think
one
of
the
other
sides
on
big
data
that
we,
it
does
touch
scheduling
a
little
bit
like
six
scheduling.
D
C
How
should
we-
probably
maybe
it's
it's
like
combining
those
two
SIG's
if,
if
at
all
or
keeping
this
sake
and
also
including
like
batch
scheduling
in
this
in
this
sake,
might
be
useful
because
there
are
it's
not
just
job
like
equipment
is
native
jobs
right.
Do
you
like,
for
example,
if
I
want
to
schedule
apart
what
happens
if
I'm
running
out
at
capacity?
What
should
be
the
behavior
of
a
spark
driver?
Should
it
crash,
or
should
it
like
continue
on
in
currently
it
works,
I
think
but,
like
I,
think
we
need
to
think
about
that.
A
Those
are
definitely
open.
Questions
I
mean
behavior
of
behavior
of
certain
categories,
especially
of
spark
jobs
like
long-running
streaming
jobs,
restart
policies
is
still
being
actually
debated
and
there's
the
concept
of
I
believe
you
know
working
group
specifically
designed
to
go
across
cig
like
that.
So
that's
another
possible
direction.
We
could
take
that
yeah.
Thanks
for
your
question,.
E
A
B
What
happens
is
like
so
the
driver
pod
will
always
stay
like
a
nacelle
once
they
drop
completes.
The
drivers.
Pod
would
be
there
well,
even
though
it's
not
taking
computer
resource,
but
it's
after
taking
an
SAT
resource
so
but
extra
deposit.
So,
for
example,
if
if
one
actuator
just
finishes
successfully
will
be
gone
after
be
deleted,
I.
B
D
B
A
that's
a
good
question,
so
we
actually
had
that
featuring
in
our
fork
before
we
have
streamed
on
spark.
You
know
does
so
that
the
scheduler
beckoning
to
the
main
spark
rebel,
so
we
I
think
we
still
need
to
have
that
special.
You
know,
given
we
know
how
am
I
gonna
support
for,
for
example,
for
seeker
sniff
us
cover
support
so
EFSA
also
one
of
the
project
we're
being
engaging
with.
So
that's
really
I
think
that's
really
important
yeah!
D
D
B
First
of
all,
we
are
not
all
a
like
touching
any
of
these
things,
like
you
know
so,
like
forget,
some
law
can
persist
in
water.
That's
really
up
to
the
application
developers.
They
chose
like
analyze
the
kind
of
things
they
want
to
use
if
they
want
to,
like
you
know,
touch
me
a
local
person
bottom
to
their
their
jobs.
They
can
do
that.
But
so,
if
we
didn't
smart,
we
don't
do
anything
special
about.
A
A
A
D
F
G
F
A
F
F
B
We
are
actually
talking
to
sake
scheduling
about
us,
so
they
have
a
project
called,
could
be
bash
buffer.
If
you
guys
have
heard
of
it
or
not
like
that
was
supposed
to
be,
you
know,
doing
gone
scheduling,
but
spark
it's
actually
pretty
unique
in
a
sense
that
you,
the
driver
in
executors.
They
don't
start
at
the
same
time.
They're
not
like
you
know
the
same
they're,
not
the
same,
so
you
can
related
you
treat
them
as
the
same
group.
B
So
the
current
way,
the
coop
about
concept,
doesn't
really
work
for
spark,
even
though
you
know
my
work
for
machine
learning
workloads,
but
now
for
spark
so
actually
I.
Just
how
does
discussion
with
the
lead
on
that
project
harder
today,
so
I
guess
we
will
have
some
plan
to
actually
see
how
we
can
make
work
house
of
work
for
spark
and
or
generally
like.
B
If
you
have
a
master
walker
kind
of
architecture
and
marks
there
and
water,
they
don't
start
at
the
same
time,
how
you
actually
make
that
work
for
these
kind
of
workers
as
well
and
also
it
so
when
it
comes
to
scheduling
I.
Think,
there's
like
like
I
see
it's
definitely,
you
know
increasing
demands
for
things.
Like
the
you
know,
the
capacity
or
first
coordinate
scheduler
from
yarn,
for
example,
in
communities.
Well,
we
don't
currently
have
anything
like
those,
so
that's
still
something
that
we
have
to
work
on.
E
So
is
there
any
plan
to
do
a
kind
of
data
catalog
thing
on
kubernetes,
for
example,
something
like
Adele
brouse,
school
or
hive,
so
you
can,
when
I
run
sprout
job
I
can
reference
that
metadata
to
even
though
I
have
managed
spark
job
so
I
have
a
kind
of
consciousness,
centralized
metadata
store,
so
I
know
where
the
data
is
coming
from.
Is
there
any
plan
for
doing
some
sort
of
protocol
that
we.
A
G
B
B
There's
like
you
know
so
different
ways
of
doing
that
or
you
can
act
like
that,
so
you
can
also
do
it
like
you
know,
for
example,
like
C
get
CS
ad
kind
of
thing.
Like
you
know,
you
have
your
specs
stored
on
github,
for
example.
Then,
once
you
make
a
change,
Walker
automatically
trigger
reapply,
something
like
that
yeah.
So
it's
just
you
know
so
basic
just
customer
resources.
D
B
So
I
think
one
unique
thing
about
this:
sega's
actually
covers.
You
know
not
only
about
computer
but
also
storage
and
scheduling.
So
there's
not
like
a
whole
bunch
of
scenes.
You
have
to
consider
when
you
run
your
data
processing,
workflows
on
kubernetes
yeah.
So
like
what
you
said,
yeah,
that's
a
good
idea.
Actual
can
you
know,
provide
some
kind
of
reference,
so
people
when
people
come
to
us.
Like
a
know,
they
know
what
will
be
the
recommendation
that
we
have
one.
It.
B
A
A
Yeah,
actually,
we
could
do
that
and
of
course
some
of
them
might
actually
want
to
resign
now
upon
the
Apache
spark
docs
or
for
that
are
the
airflow
Doc's
we're
trying
there
it's
on
it.
There
comes
a
point
where
we
try
to
like
make
sure
that,
like
we're
directing
discussions
to
the
right
organization,
because
of
course
the
up
streams
want
to
make
sure
that,
like
you,
know,
they're
in
the
loop
on
anything
that
happens
before
it
becomes
a
PR
that
shows
up
on
there,
mm-hmm.